Skip to content
AI Agent Eval Harness

ADR-0003: Eval harness

  • Status: Accepted
  • Date: 2026-03-18
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The eval harness is the centre of this project. The agent is the vehicle; the harness is the artefact. A reader has to come away convinced that the harness is real: it loads JSONL golden datasets, runs the agent end-to-end, emits per-turn traces, scores those traces across a set of deterministic and judge-backed evaluation dimensions, and produces a PR-gating verdict plus a longer-form report.

None of the off-the-shelf “agent eval platforms” cover all of these dimensions for a multi-turn health-domain agent. DeepEval contributes a strong hallucination metric; an Anthropic Claude Haiku LLM-as-judge scores the rubric-based dimensions (groundedness, faithfulness, tone and empathy); Phoenix gives an OTel backend with tracing UI; Promptfoo is the canonical red-team / OWASP LLM Top 10 runner. A hand-rolled pytest core that orchestrates these pieces is the smallest amount of glue that delivers every dimension while staying portable.

How do we structure the eval harness so that (a) it runs as a normal pytest job in CI, (b) deterministic scorers gate every PR cheaply, (c) LLM-as-judge runs nightly without blowing the free-tier budget, (d) red-team scenarios run nightly out-of-band, and (e) every scorer can be swapped or upgraded without rewriting the runner?

Decision Drivers

Section titled “Decision Drivers”
  • Coverage of the evaluation dimensions the project commits to
  • PR-gate cost ceiling: deterministic scorers only on every PR; LLM-as-judge nightly
  • Reproducibility: a contributor must be able to run the eval suite locally and get the same verdict CI gives
  • Pin the judge model exactly (provider + model + version) so the scoring is stable across runs
  • License hygiene: every scorer library must be permissively licensed (Apache 2 / MIT / ELv2 acceptable for tooling)
  • Avoid a black-box “AI evaluation platform” SaaS; the harness has to be portable and self-hosted

Considered Options

Section titled “Considered Options”
  • Hand-rolled pytest core + DeepEval + an Anthropic LLM-as-judge + Promptfoo + Phoenix with a pinned Anthropic Claude Haiku model as the judge (chosen)
  • One vendor SaaS (Braintrust / Galileo / LangSmith Eval)
  • Pure DeepEval as the entire harness, with its built-in dataset and report runners
  • OpenAI Evals as the runner

Decision Outcome

Section titled “Decision Outcome”

Chosen option: hand-rolled pytest core orchestrating DeepEval, an Anthropic LLM-as-judge, Promptfoo, and Phoenix, with a pinned Anthropic Claude Haiku model as the LLM judge (model id behind an environment flag, version pinned). The harness has three workflows:

  • PR-gating: deterministic scorers only (citation coverage, citation correctness, refusal-template match, escalation-list match, retrieval recall). These need no LLM and no API key, so the gate is cheap and fully reproducible. The judge-backed corpus thresholds are evaluated on the nightly judge workflow, not on the PR gate.
  • Nightly LLM-as-judge (scheduled): the pinned Anthropic Claude Haiku judge scores the rubric-based dimensions (groundedness, faithfulness, and the tone / empathy / medical-appropriateness rubrics), and DeepEval contributes its hallucination metric. This is the only workflow that spends judge tokens.
  • Red-team (separate workflow): two layers. (a) A deterministic, key-free CI gate — a Promptfoo evaluation of the 13 hand-crafted adversarial cases; the shim drives the real LangGraph agent using an offline stub client; the agent’s guardrail layer decides, not the LLM. (b) A separate keyed live measurement — a Promptfoo red-team run with Groq llama-3.3-70b-versatile as the target, OpenAI gpt-4o as the attack generator, and Groq as the grader. Non-blocking; real failures are expected and documented with provenance.

Phoenix is the observability sink during eval runs: every turn emits OTel spans through the project’s existing OpenInference wiring (see ADR-0006), Phoenix collects them, and the harness attaches Phoenix’s trace URLs to the eval report. Inspect AI (UK AISI) is reserved as a bonus capability-eval task, optional for the initial milestone.

This composition gives every eval dimension a concrete home, keeps the PR gate cheap, and never depends on a closed eval platform.

Confirmation

Section titled “Confirmation”
  • The eval suite runs the PR-gating subset locally and in CI; the CI job fails on the configured threshold violations
  • Every scorer is implemented behind a small Scorer Protocol; the runner does not import scorer libraries directly
  • The judge provider is Anthropic and the judge model id is read from a JUDGE_MODEL environment variable; the live judge is constructed with that variable left empty, so a mistyped id defers to the adapter’s pinned Claude Haiku default (fail-closed) instead of failing open on a bad model string
  • A nightly red-team report is published as an artifact

Feature-aware rubrics

Section titled “Feature-aware rubrics”

The judge evaluates five feature-specific rubric dimensions when a golden case carries feature tags in its metadata. The features are: voice, i18n (internationalization), pii (PII redaction), governance (regulatory coverage), and data_layer (data-layer key gating / cost display).

  • A set of feature rubric templates defines the five feature rubric descriptions, each with its own graded scoring criteria.
  • The groundedness scorer conditionally extends its rubric with feature entries when a case carries feature metadata. Cases without feature metadata produce identical scoring to the baseline.
  • Feature score keys follow the pattern feature_{name} (e.g., feature_voice, feature_i18n).
  • A CI feature-coverage gate verifies that all five feature categories have at least one golden case and that no feature dimension returns a malformed rationale on every case (malformed detection). Legitimate zero scores with non-malformed rationales are not flagged.

Judge calibration (human-vs-judge agreement gate)

Section titled “Judge calibration (human-vs-judge agreement gate)”

An LLM judge is only trustworthy to the degree it agrees with a human expert. The harness therefore ships a calibration gate: a held-out set of human-labelled cases is scored by both a domain expert and the judge, and their agreement is measured per dimension. The gate is advisory by default - it reports agreement and never blocks a normal PR - but in the enforcing CI lane it is armed fail-closed on the two safety dimensions, escalation and self-harm, so a judge that drifts out of agreement with the human labels on a safety dimension blocks the launch lane instead of silently mis-scoring it. The remaining dimensions (groundedness, faithfulness, persona stability) calibrate in advisory mode. The gate is deterministic and key-free in CI; the agreement operating point and the labelling protocol are calibrated per deployment and provided as part of the engagement.

This closes the gap most LLM-as-judge harnesses leave open: it turns “the judge scored it high” into “the judge agrees with our expert often enough to be trusted on the dimensions that decide safety.”

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Every eval dimension is produced by a named scorer behind a single Scorer Protocol, so scorers can be swapped or upgraded independently
  • PR gate is cheap and deterministic enough to run on every push without burning free-tier quotas
  • LLM-as-judge stays nightly and uses a pinned Claude Haiku, the lowest-cost-per-token Anthropic model that still scores well on tone / empathy rubrics
  • Phoenix self-hosted gives a real OTel backend without quota pressure; its ELv2 license is acceptable for this use case
  • Promptfoo’s OWASP LLM Top 10 suite covers a red-team surface DeepEval does not, pinned to a public attack catalogue
  • The harness is portable: no SaaS lock-in, every dependency is open source
  • The judge is calibrated against human labels: a per-dimension agreement gate blocks the launch lane if the judge drifts out of agreement on a safety dimension (escalation or self-harm)

Negative

Section titled “Negative”
  • Three external eval libraries mean three upgrade tracks; we mitigate by pinning minor versions
  • The hand-rolled core is real code we maintain; it earns its keep by giving us full control over thresholds and report format
  • Anthropic Haiku judge calls are billed; nightly cadence plus a small token cap keep them inside the $0/month envelope

Neutral

Section titled “Neutral”
  • The harness emits two artefacts per run, a machine-readable JSON report and a Markdown summary
  • Inspect AI is reserved as an optional capability-eval task; the initial milestone does not depend on it
  • Phoenix and Promptfoo run in Docker profiles, not in the main Compose file, to keep the live-demo image small

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Hand-rolled pytest core + DeepEval + an Anthropic judge + Promptfoo + Phoenix

Section titled “Hand-rolled pytest core + DeepEval + an Anthropic judge + Promptfoo + Phoenix”
  • Good, because each piece is best-in-class for its slice (hallucination metric / rubric-scored judge / OTel backend / red-team)
  • Good, because pytest is already the project test harness
  • Good, because every scorer is swappable behind a Protocol
  • Bad, because several moving parts have to be maintained

One vendor SaaS (Braintrust / Galileo / LangSmith Eval)

Section titled “One vendor SaaS (Braintrust / Galileo / LangSmith Eval)”
  • Good, because the dashboard story is excellent out of the box
  • Bad, because the project must run with zero accounts
  • Bad, because the eval definitions live in someone else’s UI

Pure DeepEval

Section titled “Pure DeepEval”
  • Good, because DeepEval’s hallucination metric is strong and well-supported
  • Bad, because it does not cover the rubric-scored judge dimensions (groundedness, faithfulness, tone) or Promptfoo’s red-team surface on its own

OpenAI Evals

Section titled “OpenAI Evals”
  • Good, because the format is well-known
  • Bad, because the runner is OpenAI-centric and does not map cleanly onto multi-turn agent evals

More Information

Section titled “More Information”