Skip to content
AI Agent Eval Harness

ADR-0026: Judge calibration design lock

  • Status: Accepted
  • Date: 2026-06-13
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The eval harness (ADR-0003) scores two of its rubric dimensions - groundedness and faithfulness - with an LLM-as-judge. A judge is only trustworthy once someone has measured how well it agrees with a human on the same cases. The harness therefore adds a calibration gate: for each calibration case a human assigns a label and the judge produces a score, and their agreement is measured per dimension with a weighted Cohen’s kappa.

A calibration metric is only credible if its framing, its bins, its threshold, and its corpus composition are fixed BEFORE any case is labeled or scored. If any of those can move after the numbers are in, the agreement claim becomes unfalsifiable: a low result can always be “fixed” by nudging a bin, lowering the bar, or dropping the inconvenient dimension. That is threshold gaming dressed as tuning, and it hollows out the gate’s entire purpose.

This record runs first, on purpose, so every later phase - the metric and its schema, the human labeling pass, the one-time judge run, and the gate wiring - inherits the same frozen constants and the same naming. How do we lock every framing, metric, bin, threshold, and corpus decision for the calibration gate, in one place, before any labeling or scoring begins, so that no later tuning can retroactively rescue the agreement claim?

Decision Drivers

Section titled “Decision Drivers”
  • Pre-data lock. Every constant that could be tuned after the fact is fixed before the first label is written. The freeze is the credibility, not an afterthought.
  • Honest framing. The metric measures one human rater against the judge. It must be named and described as exactly that, so no reader mistakes it for a multi-rater reliability number it is not.
  • Reproducibility. A publicly replicable agreement number beats a hidden one: anyone who forks the reference implementation should be able to recompute it.
  • Invariant preservation. The gate reads the judge’s outputs only and never alters the judge (ADR-0003); the synthetic-only corpus invariant (ADR-0018) holds.
  • Small-sample realism. The calibration corpus is deliberately small, so every choice is sized for a small-sample regime where an over-tight threshold would simply flap.

Considered Options

Section titled “Considered Options”
  • Framing: single-rater human-vs-judge agreement (chosen) versus inter-annotator reliability with multiple annotators (rejected: there is one owner-rater by design; a multi-annotator pipeline is out of scope and would misdescribe what the number measures).
  • Locale gating: one pooled gate with per-locale diagnostics (chosen) versus a separate gate per locale (rejected: triples the labeling and makes each per-locale estimate small-sample and flap-prone) versus English-only gating (rejected: leaves es-419 and pt-BR ungated).
  • Corpus visibility: a fully public calibration corpus (chosen) versus withholding it (rejected: hides the reproducibility that is the whole value) versus a partial public sample (rejected: an arbitrary split with no clean rationale).
  • Bins: freeze the boundaries now as semantic constants (chosen) versus place them empirically from a pre-labeling judge distribution run (rejected: it delays the lock and invites fitting the bins to the data).
  • Stratification: a production-realistic mix (chosen) versus a balanced or even-thirds mix (rejected: neither reflects the score distribution a real change produces).

Decision Outcome

Section titled “Decision Outcome”

Chosen options, locked before any labeling:

  • Framing - single-rater human-vs-judge agreement. The metric is the agreement between one human rater and the judge, per dimension. It is not inter-annotator reliability and is never described as such: there is a single owner-rater by design, so there is no second annotator to be reliable against. The metric’s output labels carry a name that hard-codes the human-vs-judge framing, so it cannot drift into reliability language on any downstream report.
  • Metric - linear-weighted Cohen’s kappa. Agreement is a linear-weighted Cohen’s kappa (scikit-learn, already present in the evaluation dependencies, so no new runtime dependency). The linear weighting gives partial credit for an off-by-one-bin disagreement, which suits ordered bins.
  • Dimensions. The judge-backed dimensions that gate a change - groundedness and faithfulness - are calibrated. The method is later extended to the safety dimensions, which become launch-blockers rather than advisory signals (see the escalation / self-harm calibration record); this design lock is the method those inherit.
  • Bins - frozen semantic constants. Each dimension’s continuous judge score is mapped into three ordinal bins - unfaithful / partial / solid - whose boundaries are frozen now as semantic constants, aligned with the Landis-and-Koch reading of agreement strength and fixed without any pre-labeling distribution run. A per-bin marginal report produced later is a confirmatory check only; it has no authority to move a boundary.
  • Threshold - frozen pre-data. The gate blocks when a gated dimension’s pooled agreement falls below a fixed threshold, sized for the small-sample regime and held distinct from a slightly higher corpus-mean buffer the corpus is aimed at, so sampling noise does not push a genuinely-calibrated judge under the bar. The threshold is frozen pre-data; raising it is out of scope until more labels exist, because at this sample size an over-tight bar would flap. The operating point itself is calibrated per deployment and provided as part of the engagement.
  • Locale gating - pooled gate, per-locale diagnostics. One pooled per-dimension kappa over the whole cross-locale corpus is the only blocking number. The es-419 and pt-BR per-locale kappas are reported as non-gating diagnostics - noisy-but-directional at their per-locale counts, and labeled as diagnostic, not gating. The direction is deliberate and must not invert. A locale may be promoted from diagnostic to gated in a future milestone once it accumulates enough of its own labeled cases.
  • Corpus - synthetic-only and fully public. The calibration corpus - the cases, the human labels, and the pre-committed judge scores - is synthetic-only (ADR-0018) and fully public, so the agreement is independently reproducible by anyone who forks the reference implementation. Its exact size and stratification are settled in the corpus-rebalance record.
  • The locked kappa-paradox remedy. A production-realistic mix produces skewed marginals, and skewed marginals can yield a low kappa despite high raw percent agreement - the kappa paradox. This risk is accepted, and the remedy is locked: if the committed corpus paradoxes, the ONLY permitted fix is to rebalance the corpus - add borderline and fail cases, re-label, re-score. Lowering the threshold, moving a bin boundary, or dropping a gated dimension is forbidden. The constants stay frozen regardless of the observed number; rebalancing is the only lever.
  • Detection backstop. The calibration report carries, together, the per-dimension point estimate, a bootstrapped confidence interval, the per-bin marginal distribution, and the raw percent agreement. Reading those four together distinguishes a genuine miscalibration from a marginal-skew paradox and routes a paradox to corpus rebalancing rather than to a boundary or threshold edit.
  • Label-anchoring discipline. Human labels are committed in an earlier commit than the judge scores, so the human judgment provably predates the judge run; in every case the human-label timestamp precedes the judge-score timestamp. The labels cannot be back-fitted to the judge.
  • Invariant restatement. The gate reads the judge’s outputs only and makes no change to the judge (ADR-0003); the corpus stays synthetic-only (ADR-0018).

Confirmation

Section titled “Confirmation”
  • The agreement number is falsifiable: every constant that could be tuned after the fact is fixed before the first label, and the leak-check stays green with the public calibration corpus present.
  • The metric adds no new runtime dependency.
  • Downstream phases read the frozen constants and the locked naming from this record; the bins, the threshold, and the dimensions are pinned.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • The agreement claim is falsifiable and credible: a low kappa cannot be silently engineered away, because no constant may move after the data is in.
  • The corpus is publicly reproducible: anyone who forks the reference implementation can recompute the same agreement over the same committed cases, labels, and scores.
  • The naming is locked once and propagates unambiguously across every downstream output surface.
  • The judge is untouched: the gate reads judge outputs only.

Negative

Section titled “Negative”
  • The publicly reproducible corpus has to be re-confirmed against the leak-check rather than relying on a default exclusion.
  • The production-realistic mix carries the accepted kappa-paradox risk, whose only sanctioned response is the comparatively expensive corpus rebalance.
  • Freezing the bins and the threshold pre-data means they cannot be retuned to the observed distribution even if it later looks awkward; the freeze is the point, but it removes that lever on purpose.

Neutral

Section titled “Neutral”
  • The corpus-activation mode (auto-on-present versus an opt-in path) is settled in a later record, not here.
  • Raising the threshold with more labels, and promoting a per-locale diagnostic to a gated locale, are documented future steps, not present commitments.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Single-rater human-vs-judge framing (chosen)

Section titled “Single-rater human-vs-judge framing (chosen)”
  • Good, because it describes exactly what the number measures: one human rater against the judge.
  • Good, because the locked naming keeps the framing legible on every surface.
  • Bad, because a single rater carries that rater’s bias with no second annotator to average it out.

Inter-annotator reliability (rejected)

Section titled “Inter-annotator reliability (rejected)”
  • Good, because multiple annotators would quantify human disagreement directly.
  • Bad, because there is one owner-rater by design, so there is nothing to be reliable against, and the pipeline would misdescribe the number.

Pooled gate, per-locale diagnostics (chosen)

Section titled “Pooled gate, per-locale diagnostics (chosen)”
  • Good, because a pooled gate is feasible to label and stable enough to block on, while each locale still gets a signal.
  • Bad, because a pooled gate can mask a single-locale regression the noisy per-locale diagnostics only hint at.

Fully public calibration corpus (chosen)

Section titled “Fully public calibration corpus (chosen)”
  • Good, because a publicly replicable agreement is more credible than a hidden one, and the corpus is synthetic-only so there is no privacy concern.
  • Bad, because it must be re-confirmed against the leak-check.

Frozen semantic bins (chosen)

Section titled “Frozen semantic bins (chosen)”
  • Good, because the bins are semantic, not distribution-fitted, so they can be fixed without data and foreclose post-hoc tuning.
  • Bad, because a semantic boundary may not align perfectly with where the judge clusters its scores.

More Information

Section titled “More Information”