Skip to content
AI Agent Eval Harness

ADR-0028: Faithfulness construct correction

  • Status: Accepted
  • Date: 2026-06-14
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The first calibration run over the rebalanced corpus (ADR-0027) produced a valid groundedness agreement but a low faithfulness agreement. Root-cause analysis found the low faithfulness number was a construct mismatch on the human side, not a judge defect.

The human faithfulness rubric was mis-specified: it told the rater to penalise a response that “avoids contradicting OR ADDING content beyond the cited context.” But penalising an unsupported ADDITION is exactly what the groundedness construct already does. So the human faithfulness label was measuring a groundedness-like construct, while the production scorer measured contradiction. The two sides were scoring different things, so the low faithfulness kappa was a construct artifact, not a valid measurement of human-vs-judge agreement. Groundedness, by contrast, used a matched construct on both sides (support), so its agreement was valid.

The evidence is clearest on the fail cases. A reply that directly CONTRADICTS a card constraint scores low on both sides. But replies that ADD ungrounded content without flatly contradicting the card were scored faithful by the judge (no contradiction) and unfaithful by the human (unsupported addition). How do we correct the faithfulness construct and re-calibrate it honestly, without gaming the agreement number?

Decision Drivers

Section titled “Decision Drivers”
  • Construct validity. A calibration of faithfulness is meaningful only if the human and the judge score the SAME construct. They were not.
  • No gaming. The design lock (ADR-0026) forbids tuning to inflate agreement. Re-labeling after the judge scores are known would be gaming if done to agree with the judge; the correction must be principled, blinded, and anchored.
  • Preserve the frozen constants. The bins, the gate threshold, the two gated dimensions, the single-rater framing, and the label-anchoring discipline are unchanged. Only the faithfulness rubric’s definition is corrected - it was mis-specified, not re-tuned.

Considered Options

Section titled “Considered Options”
  • Correct the faithfulness rubric to a contradiction-only construct and re-calibrate under blinding (chosen).
  • Leave the rubric as-is and report the low agreement (rejected: it reports a construct artifact as if it were a real disagreement).
  • Re-label toward the judge’s values to raise agreement (rejected and forbidden: that is the exact threshold gaming the design lock prohibits).

Decision Outcome

Section titled “Decision Outcome”
  • Corrected construct - contradiction only. Faithfulness is the absence of claims that CONTRADICT the cited context. A high score means no claim contradicts the cited card; a low score means the response asserts something that directly contradicts it; a partial or borderline contradiction sits in the middle. An unsupported ADDITION that contradicts nothing does NOT lower faithfulness - that is a groundedness concern. Groundedness (support; additions lower it) stays as-is and its valid calibration is unaffected. The construct is completed on the judge side by the faithfulness rubric scorer record.
  • Clean re-calibration. Because the owner had already seen the first judge run’s faithfulness scores, re-labeling cannot be owner-blind. So a blinded expert panel - raters who never saw any judge score - re-labels faithfulness under the corrected rubric; the owner adjudicates only the genuinely-contested cases by applying the contradiction rule, so the rubric, not the judge’s value, decides; the new human labels are committed before a fresh judge run, preserving the label-anchoring order; the judge is re-run and the agreement recomputed. Groundedness labels are carried over unchanged.
  • Anti-gaming safeguards (load-bearing). The correction is legitimate, not threshold gaming, because: (a) the corrected construct is the standard, independently-defined faithfulness construct, chosen for validity, not to maximise agreement; (b) the bulk re-labeling is blinded to the judge scores; (c) any resulting increase in agreement is a consequence of both sides finally measuring the same construct, not of moving a label toward a seen judge value; (d) the bins, the gate, and the dimensions are untouched. If the corrected agreement is still below the gate, that is an honest finding and the only lever is corpus rebalancing (ADR-0027), never boundary or threshold movement.

Confirmation

Section titled “Confirmation”
  • The labeling rubric states the contradiction-only faithfulness construct.
  • The re-labeled human faithfulness is committed before the re-run judge scores, so the label-anchoring order holds for every case.
  • The groundedness human labels are unchanged from the first labeling.
  • The bins, the gate threshold, and the gated dimensions are unchanged; only the faithfulness rubric wording is corrected.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Faithfulness now measures a well-defined construct that matches the production scorer, so its agreement becomes a valid measurement.
  • The groundedness calibration is untouched and remains valid.
  • The two constructs are explicitly disentangled - support versus contradiction - which clarifies the eval surface beyond calibration.

Negative

Section titled “Negative”
  • A re-anchoring and a second judge run are required (modest cost).
  • Re-labeling after a first run carries an APPEARANCE of gaming; the blinded panel, the principled rubric, and the frozen constants are what distinguish this correction from gaming, and they are visible in the record.

Neutral

Section titled “Neutral”
  • The design-lock and corpus constants (bins, gate, dimensions, framing, anchoring, parallel composition) are unchanged.
  • The first run’s faithfulness scores remain in version history as the record of why the construct mismatch was found.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Correct to a contradiction-only construct, re-calibrate blinded (chosen)

Section titled “Correct to a contradiction-only construct, re-calibrate blinded (chosen)”
  • Good, because it makes both sides measure the same construct, so the agreement number becomes valid.
  • Good, because the blinding and the frozen constants keep the correction honest.
  • Bad, because it costs a re-anchoring and a second judge run, and it must carry visible safeguards against the appearance of gaming.

Leave the rubric as-is (rejected)

Section titled “Leave the rubric as-is (rejected)”
  • Good, because it changes nothing and needs no re-run.
  • Bad, because it publishes a construct artifact as though it were a real human-vs-judge disagreement.

More Information

Section titled “More Information”