Skip to content
AI Agent Eval Harness

ADR-0030: Advisory-by-default calibration gate

  • Status: Accepted
  • Date: 2026-06-14
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The calibration gate (ADR-0026) is wired as an additive check in the eval pipeline: it computes the pooled, per-dimension human-vs-judge kappa over the committed corpus and contributes a failing exit code when a gated dimension falls below the threshold. It reads judge outputs only, is deterministic and key-less, and does not touch the judge. The design lock deferred one sub-decision to the wiring phase: the gate’s activation mode - whether an unset corpus path auto-resolves and hard-gates, or stays unset and skips.

The measured calibration then produced an honest, awkward result. Faithfulness passes comfortably, but groundedness sits right at the gate boundary - statistically indistinguishable from the bar: its confidence interval includes the threshold and one locale passes on its own. The frozen threshold and bins are never a lever, so the below-boundary dimension is accepted as an honest measurement - a systematic judge leniency on the borderline band, not a corpus or truncation artifact. The only sanctioned lever, corpus rebalancing, was declined this cycle: rebalancing right at the line would be composition-tuning, not a principled fix.

That leaves a contradiction. Hard-gating by default would make every change on the default path fail the boundary groundedness gate - turning an honest, known-boundary finding into a permanently-red CI the team would be pressured to silence by exactly the gaming moves the design lock forbids. How should the wired gate be activated by default, given a dimension that passes and a dimension honestly at the boundary, without gaming the frozen constants and without recording the divergence silently?

Decision Drivers

Section titled “Decision Drivers”
  • No gaming. The bins and the threshold are frozen and never a lever; a below-boundary dimension is accepted as honest, not engineered away. The activation mode must not become a backdoor for silencing the gate.
  • Honest divergence record. An earlier intent to hard-gate by default and the shipped behavior differ; the divergence is documented, not absorbed silently.
  • Demonstrable, opt-in enforcement. The gate must be proven to FIRE on the boundary dimension and PASS on the passing dimension when an operator opts in - the mechanism is real, not decorative.
  • Key-less, deterministic CI green by default. The default path - no judge key, no operator-set corpus path - must not turn red on an honest boundary finding.
  • Reversibility. Flipping to enforcing-by-default later must be a one-line config change, not a re-architecture.

Considered Options

Section titled “Considered Options”
  • Advisory / opt-in by default (chosen): an unset corpus path skips the gate (pass-neutral, logged); enforcement runs only when an operator explicitly sets the path; no default-path resolution.
  • Hard-gate by default (rejected): an unset path auto-resolves and gates every default run, which reds CI on the honest groundedness boundary and invites the forbidden gaming.
  • Move the threshold or the bins to make groundedness pass (rejected, forbidden): violates the pre-data freeze; threshold gaming dressed as tuning.
  • Rebalance the corpus this cycle to lift groundedness (rejected this cycle): the one sanctioned lever, declined because the disagreement is systematic judge leniency on the borderline band, not corpus saturation, so rebalancing at the line would be composition-tuning.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: advisory / opt-in by default - the only activation mode that ships a real, proven gate without either gaming the frozen constants or turning an honest boundary finding into permanently-red CI.

  • Advisory by default. With the corpus path unset (the default), the gate skips pass-neutral and CI stays green; the skip is logged, not silent. Enforcement is opt-in: it runs only when an operator explicitly sets the corpus path, at which point the gate FIRES on the boundary dimension and PASSES the other. The gate is wired and key-less-safe; only its default activation is advisory.
  • The earlier hard-gate intent is set aside. An earlier reading of the deferred activation-mode sub-decision would have resolved an unset path to a default and hard-gated by default. That is set aside: on the committed corpus a hard default gate would red CI on a finding statistically indistinguishable from the bar. The divergence is recorded here, not absorbed silently.
  • No-gaming posture (explicit). The bins and the threshold remain frozen and are not touched. The advisory default is NOT a way to make the boundary dimension “pass”; it is a way to ship the gate honestly while that dimension sits at the boundary. The corpus-rebalance lever was declined this cycle as composition-tuning - in contrast to the faithfulness correction (ADR-0028 / ADR-0029), which was a genuine two-sided construct change, not a kappa-chasing move.
  • Flip-to-enforcing trigger. The gate flips to enforcing-by-default only once the boundary dimension clears the gate in a future cycle, and only via the sanctioned corpus-rebalance lever (never a threshold or bin move). Until then, enforcement is the operator’s explicit opt-in.

Confirmation

Section titled “Confirmation”
  • Tests pin the disposition end-to-end: an unset path exits green with the skip logged; a set path exits failing with a per-dimension gate reason on the boundary dimension - proving default-skip / opt-in-enforce on the committed corpus.
  • The config comment and the CI workflow document the gate as additive, key-less-safe, and advisory-by-default.
  • The judge is byte-for-byte unchanged.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • The gate ships honest: a real, proven mechanism that fires on the boundary dimension and passes the other when opted in, without gaming the frozen constants.
  • The default key-less CI stays green on an honest boundary finding, so the team is never pressured into the forbidden gaming moves to un-red the gate.
  • The divergence from the earlier hard-gate intent is recorded openly, so a reader can see why the shipped activation mode differs.
  • Flipping to enforcing-by-default later is a one-line config change.

Negative

Section titled “Negative”
  • The default path does not enforce calibration until an operator opts in, so a regression in the judge’s agreement would not red the default CI until then. The opt-in knob and the committed corpus are the mitigation; the trade is deliberate while the dimension sits at the boundary.
  • Two activation states - default-skip and opt-in-enforce - are a small extra surface a reader must understand.

Neutral

Section titled “Neutral”
  • Only the default activation changed (unset skips rather than resolves-and-gates); the corpus target and the frozen constants are unchanged.
  • The opt-in knob adds no new secret and no new runtime dependency.
  • The design lock’s other decisions are untouched; this narrows only the activation-mode sub-decision.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Advisory / opt-in by default (chosen)

Section titled “Advisory / opt-in by default (chosen)”
  • Good, because it ships a real, proven gate without gaming the frozen bins or the threshold.
  • Good, because the default key-less CI stays green on an honest boundary finding, removing the pressure to silence the gate.
  • Good, because flipping to enforcing-by-default later is a one-line change.
  • Bad, because the default path does not enforce calibration until an operator opts in.

Hard-gate by default (rejected)

Section titled “Hard-gate by default (rejected)”
  • Good, because it would enforce calibration on every default-path run.
  • Bad, because on the committed corpus it reds CI on the honest groundedness boundary and invites the exact threshold or bin gaming the design lock forbids.

More Information

Section titled “More Information”