Skip to content
AI Agent Eval Harness

ADR-0027: Calibration corpus rebalance

  • Status: Accepted
  • Date: 2026-06-14
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The design lock (ADR-0026) froze the calibration gate’s framing, bins, threshold, and dimensions, and left corpus composition as the single open lever - pre-committing corpus rebalancing as the only sanctioned remedy if the corpus could not discriminate. An initial composition chose a production-realistic skew: mostly clean passes, a thin borderline and fail region, one repeated failure mode, and a non-parallel per-locale split.

Before spending the one-time human-labeling pass and the judge run on it, an expert-panel review found that composition structurally degenerate, not cosmetically flawed. The corpus was saturated at the poles - so cleanly constructed that labeling was near-deterministic and blinded raters agreed almost unanimously; the entire low anchor rested on a single failure mode in a single scenario; and the locales were not parallel, which breaks the per-locale diagnostic comparison the gate relies on. The verdict: a human-vs-judge kappa over that corpus would be circular by construction and nearly uninformative - precisely the kappa paradox the design lock anticipated.

A calibration whose corpus cannot discriminate does not measure what it claims to measure. How do we rebalance the corpus so the agreement is genuinely informative, without touching any of the frozen constants that make the agreement claim falsifiable?

Decision Drivers

Section titled “Decision Drivers”
  • Discrimination over surface realism. A kappa’s value is separating a calibrated judge from a miscalibrated one, which needs mass in the borderline region where a human and the judge can genuinely differ - not a corpus of self-evident passes.
  • Apply the remedy, do not game it. Adding borderline and fail cases raises difficulty, which is the opposite of threshold gaming; the boundaries and the gate do not move.
  • Cross-locale parallelism. Per-locale diagnostics are comparable only if the same scenarios appear in every locale.
  • Diverse failure modes. One failure mode cannot anchor the low end or characterise the fail-to-borderline boundary.
  • Pre-data, pre-label. The rebalance happens before any human label is written, so the label-anchoring discipline stays intact.

Considered Options

Section titled “Considered Options”
  • A borderline-weighted, cross-locale parallel, defect-diverse corpus (chosen).
  • Keep the production-realistic, pole-saturated composition (rejected: the panel showed it yields a near-circular, uninformative kappa, with the entire low end resting on a single failure mode and the locales not parallel).
  • Move a boundary or lower the threshold to rescue a low kappa (rejected and forbidden by the design lock: that is threshold gaming; composition is the only permitted lever).

Decision Outcome

Section titled “Decision Outcome”

The corpus composition is set for discrimination; every design-lock integrity constant is retained unchanged.

  • Borderline-weighted. The corpus over-samples the borderline region near the bin boundaries, because the borderline cases are the informative ones for a kappa: their continuous judge score can fall on either side of a boundary, so they are exactly where a human and the judge can genuinely disagree. Clean passes, which both label the same way, carry little agreement information.
  • Cross-locale parallel. The same scenarios are instantiated in en, es-419, and pt-BR - only the language differs; the cited card, the intended stratum, and the defect mode are identical across the three - so the per-locale diagnostics compare like with like.
  • Defect-diverse. The fail cases span distinct failure modes on distinct cards - contradicting an explicit card constraint, fabricating a remedy no card supports, and answering out of scope - rather than one repeated template. The borderline cases likewise span several distinct defect modes, so the borderline stratum is not a single appended-sentence template; template uniformity itself inflates agreement.
  • Everything frozen stays frozen. The bin boundaries, the gate threshold and the corpus-mean buffer, the single-rater human-vs-judge framing and its locked naming, the two gated dimensions, the linear-weighted Cohen’s kappa, the four detection diagnostics, the label-anchoring discipline, and the synthetic-only public corpus (ADR-0018) are all unchanged. Only the composition moves - the one lever the design lock (ADR-0026) left open for exactly this case.
  • Pre-data, pre-label. The composition is set before any human label is written, so the label-anchoring discipline is untouched.

Confirmation

Section titled “Confirmation”
  • The corpus carries the revised composition, scenario-parallel across the three locales, with diverse fail and borderline modes.
  • The calibration module is unchanged: the frozen boundaries, threshold, framing, dimensions, and diagnostics are intact.
  • The leak-check stays green with the rebalanced public corpus present.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • The human-vs-judge kappa becomes genuinely informative: the borderline mass near the boundaries gives the metric the variance it needs to discriminate.
  • Cross-locale parallelism restores comparable per-locale diagnostics; the pooled gate is balanced across locales.
  • Diverse failure and borderline modes characterise the boundaries instead of one repeated template.
  • The human-labeling pass becomes a meaningful exercise of judgment rather than a near-deterministic recording of obvious cases.

Negative

Section titled “Negative”
  • The labeling burden rises modestly.
  • The mix is less surface-realistic; the corpus deliberately over-samples the borderline region, so the pooled gate reflects a harder-than-production distribution - the intended trade for a discriminating metric.
  • A borderline-heavy corpus is itself harder to label consistently; the four detection diagnostics remain the guard for reading a low kappa correctly.

Neutral

Section titled “Neutral”
  • The corpus stays in the small-sample regime the threshold and the buffer were sized for; those are unchanged.
  • Per-locale gating remains deferred (pooled gate, per-locale diagnostics); cross-locale parallelism makes a future per-locale promotion cleaner.
  • The rebalance is pre-data and pre-label, so the label-anchoring discipline is untouched.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Borderline-weighted, parallel, defect-diverse corpus (chosen)

Section titled “Borderline-weighted, parallel, defect-diverse corpus (chosen)”
  • Good, because it gives the kappa the discriminating variance a self-evident corpus lacks.
  • Good, because cross-locale parallelism makes the per-locale diagnostics comparable, and diverse modes characterise the boundaries.
  • Bad, because it raises the labeling burden and is a harder, less surface-realistic mix to label consistently.

Keep the pole-saturated production-realistic composition (rejected)

Section titled “Keep the pole-saturated production-realistic composition (rejected)”
  • Good, because it mirrors the score distribution a real change produces.
  • Bad, because the panel showed it produces a near-circular, uninformative kappa: the low end rests on a single failure mode and the locales are not parallel.

More Information

Section titled “More Information”