Evaluation Pipeline

Reference documentation: not a medical device

This documentation describes a public reference implementation evaluated on 100% synthetic data. It is a capability and readiness reference, not a compliance certification or legal advice, and it is not a medical device. It is not clinically validated and handles no production PHI.

The eval harness reads a pinned JSONL golden dataset, runs each case end-to-end through the same compiled LangGraph agent the production path uses (it builds the graph once per run, with HITL off), dispatches each final agent state to a composable set of scorers, and emits both a markdown summary and a JSON artifact. The CI gate consumes the JSON artifact; a deterministic per-case failure or a judge-backed corpus-threshold breach yields a non-zero exit code and fails the PR check.

The runner default uses two groups of scorers:

• Always-on deterministic scorers (no LLM, no key required) gate every PR: CitationCoverageScorer, CitationCorrectnessScorer, RefusalCorrectnessScorer, EscalationCorrectnessScorer, and RetrievalRecallScorer.
• Judge-backed scorers attach only when a judge client is supplied (an Anthropic API key is set): GroundednessScorer and FaithfulnessScorer (rubric scorers via LLMAsJudge directly) and HallucinationScorer (via DeepEval’s HallucinationMetric). With no judge key the run reports the judge as disabled and the threshold gate runs against the deterministic scorers only. The judge model is Anthropic claude-haiku-4-5.

See ADR-0003 for the threshold policy and the judge-model choice.

A separate calibration gate checks the judge itself. On a held-out human-labelled slice it measures the per-dimension agreement between the judge and a domain expert; it runs advisory by default and armed fail-closed on the two safety dimensions (escalation and self-harm) in the enforcing CI lane, key-free and deterministic.

flowchart LR
  A["Golden dataset<br/>(JSONL, pinned)"] --> B["evals/runner.py<br/>PytestEvalRunner"]
  B --> C["Agent run<br/>(compiled LangGraph + LLMClient + RAG)"]
  C --> D["Final AgentState per case<br/>(response, retrieved context,<br/>guardrail decisions, cost)"]

  D --> E1["Deterministic scorers (always on)<br/>CitationCoverage, CitationCorrectness,<br/>RefusalCorrectness, EscalationCorrectness,<br/>RetrievalRecall"]
  D --> E2["Judge-backed scorers (only when a judge key is set)<br/>Groundedness, Faithfulness (LLMAsJudge rubric),<br/>Hallucination (DeepEval)"]

  E1 --> F["Report writer<br/>(write_reports)"]
  E2 --> F

  F --> G1["Markdown summary<br/>(evals/reports/latest.md)"]
  F --> G2["JSON artifact<br/>(evals/reports/latest.json)"]

  G2 --> H{"CI gate<br/>per-case deterministic gate +<br/>judge corpus thresholds"}
  H -- "pass" --> I["PR check: pass"]
  H -- "fail" --> J["PR check: fail"]

  G2 --> K["Calibration gate (key-free)<br/>human-vs-judge agreement per dimension<br/>advisory; fail-closed on escalation + self-harm"]
  K -- "agree" --> I
  K -- "drift on a safety dim" --> J