Skip to content
AI Agent Eval Harness

ADR-0029: Faithfulness rubric scorer

  • Status: Accepted
  • Date: 2026-06-14
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The human-side construct correction (ADR-0028) re-specified faithfulness as contradiction-only, but after a blinded re-label the agreement was still low. The residual gap was not a labeling artifact; it was a metric-suitability problem in the production scorer itself.

The production faithfulness scorer wrapped an off-the-shelf library metric that measures whether the response’s claims contradict the FACTS in the retrieved context. But in this domain the cited context is a knowledge-base card whose body carries not only facts but explicit BEHAVIORAL RULES the agent must not violate - for example, never advise doubling a dose, never advise stopping a medication, do not diagnose. The fail cases that violate a rule without contradicting a fact expose the mismatch: a reply that both contradicts a fact and violates a rule scores low on both sides, but replies that violate a rule (advising a prohibited action) without flatly contradicting a fact were scored faithful by the off-the-shelf metric and unfaithful by the corrected human. The two sides were measuring different constructs - the human scores contradiction of the cited context including its rules; the library metric scores contradiction of the facts only. This residual gap is the mirror, on the judge side, of the human-side mismatch ADR-0028 corrected.

How do we make the production faithfulness scorer measure the same contradiction-of-rules construct the corrected human rubric uses, without gaming the agreement number?

Decision Drivers

Section titled “Decision Drivers”
  • Construct validity on both sides. ADR-0028 fixed the human side. The judge side must measure the same construct or the calibration is meaningless.
  • Domain fit. The cited context is rule-bearing. A faithfulness metric blind to the rules cannot detect the domain’s most important unfaithful responses - advising a prohibited action.
  • No gaming. The scorer change must be a principled construct correction, not a lever pulled to raise agreement.
  • Preserve the frozen constants and the judge invariant. The bins, the gate, the two gated dimensions, the framing, and the anchoring hold; the judge is read-only and unchanged.

Considered Options

Section titled “Considered Options”
  • Replace the off-the-shelf metric with a judge-rubric faithfulness scorer that reads the full card and scores rule violations (chosen).
  • Keep the off-the-shelf metric and accept the low agreement (rejected: it leaves the judge blind to rule violations, the domain’s most important unfaithfulness, and reports a construct mismatch as a real disagreement).
  • Post-process the metric’s output with a rule check bolted on (rejected: a second construct stitched onto a facts-only metric is harder to reason about than one rubric that scores the whole construct).

Decision Outcome

Section titled “Decision Outcome”
  • Replace the off-the-shelf metric with a judge-rubric scorer. The faithfulness scorer now drives the same judge as the groundedness scorer, with a rubric that measures contradiction or violation of the cited context including its explicit behavioral rules. A high score means the response contradicts nothing in the cited context and violates none of its stated rules; a low score means it asserts or advises something that contradicts the context or violates a rule - advising a double dose, advising stopping a medication, declaring a cure, and the like; a partial contradiction or strain sits in the middle. An unsupported ADDITION that contradicts nothing does not lower faithfulness - that is a groundedness concern. This is the exact human construct from ADR-0028, now mirrored on the judge side.
  • Full-context framing is load-bearing. The groundedness scorer reads only the head of each cited card, where the supporting claims sit. But the behavioral rules this scorer must check sit in the card TAIL, past the head cutoff. The faithfulness scorer therefore frames the judge with the FULL cited card text. Truncating to the head would hide every rule and silently score all responses faithful; a uniform-high faithfulness column is the signature of that regression, and a guard on the scoring path catches an all-high column - as well as an all-low one - so the failure cannot bake in silently.
  • Scope. This changes the production faithfulness scorer across the whole eval surface, not only calibration, so both sides now measure contradiction-of-rules. The hallucination metric keeps its off-the-shelf library implementation - a distinct construct, and not a gated calibration dimension. The judge is untouched; the change is entirely in the scorer.
  • Anti-gaming safeguards (load-bearing). The replacement is legitimate, not threshold gaming, because: (a) the new construct is the principled, domain-correct definition and is the same construct the corrected human rubric uses, chosen for validity; (b) the human labels are blinded to the judge; (c) the bins, the gate, and the dimensions are untouched; (d) any resulting change in agreement is a consequence of both sides finally measuring the same construct. If the corrected agreement is still below the gate, that is an honest finding and the only lever is corpus rebalancing (ADR-0027).

Confirmation

Section titled “Confirmation”
  • The faithfulness scorer calls the judge with a contradiction-of-rules rubric and frames it with the full cited card text (no head truncation); tests pin both behaviours.
  • The judge is byte-for-byte unchanged.
  • The re-labeled human faithfulness is committed before the re-run judge scores, so the label-anchoring order holds for every case; groundedness labels and scores are unchanged.
  • The bins, the gate threshold, and the gated dimensions are unchanged; only the faithfulness scorer’s construct is corrected.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Both sides of the faithfulness agreement measure contradiction of the cited context including its rules, so the agreement becomes a valid measurement.
  • The production faithfulness scorer can now detect the domain’s most important unfaithful responses - advising a prohibited action - which the off-the-shelf metric missed.
  • One fewer heavyweight library dependency on a gating dimension: the faithfulness path is now a thin judge-rubric call, like groundedness.
  • The groundedness calibration is untouched and remains valid.

Negative

Section titled “Negative”
  • A re-anchoring and a second judge run are required (modest cost).
  • The production faithfulness numbers on the main eval surface shift, because the metric now penalises rule violations; the committed eval reports regenerate on the next run and the diff is expected.

Neutral

Section titled “Neutral”
  • The design-lock, corpus, and construct constants are unchanged.
  • The hallucination metric and its library adapter remain.
  • The first library-based faithfulness scores remain in version history as the record of why the metric-suitability gap was found.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Judge-rubric faithfulness scorer over the full card (chosen)

Section titled “Judge-rubric faithfulness scorer over the full card (chosen)”
  • Good, because it makes the judge measure the same rule-inclusive construct the human does, so the agreement is valid.
  • Good, because it detects the domain’s most important unfaithfulness - advising a prohibited action - and simplifies the faithfulness path to a thin judge call.
  • Bad, because the production faithfulness numbers shift and a re-anchoring and second judge run are needed.

Keep the off-the-shelf metric (rejected)

Section titled “Keep the off-the-shelf metric (rejected)”
  • Good, because it changes nothing on the judge side.
  • Bad, because it leaves the judge blind to rule violations and reports a construct mismatch as a real human-vs-judge disagreement.

More Information

Section titled “More Information”