AI Agent Eval Harness

Methodology, not fine print

Built and evaluated on 100% synthetic, representative data by design - your PHI never has to touch an unproven system. Deliberately scoped as a wellness reference, not a medical device; what is public here is the reasoning and the shape, not a client’s implementation.

This is the public reference for a measurement-first approach to GenAI in regulated industries: a conversational health agent for medication adherence whose every clinical assertion must cite a verified knowledge-base card - or the agent refuses - paired with a CI-gated evaluation harness that proves it.

For a one-page orientation aimed at decision-makers, start with the executive summary.

The four failure modes, each with an answer

Roughly 70-95% of GenAI pilots stall before measurable impact, and four causes do most of the damage. Each one has a concrete artifact here:

• Data readiness goes to the data-readiness scorecard and a fully documented, licensable, PHI-free corpus.
• Risk and compliance goes to a deterministically-enforced safe envelope (refuse or escalate) and the governance package with the regulatory posture.
• Cost goes to an as-if-paid cost ledger and the cost-at-volume model.
• Value goes to the value model and the opportunity map, including where AI is not the answer.

Two sprints

The work runs as two sprints: Discovery - is this the right opportunity, is the data ready, what is the value and the risk - and PoC-to-Production - a measured, safe, production-shaped system with the evidence to ship it.

What you receive

Opportunity map Why this opportunity was chosen, what was deprioritized, and where AI is not the answer.

Value model The business case on the demo's own case: measured run cost, illustrative outcome, sensitivity.

Data-readiness scorecard A graded self-assessment of the corpus, and the scorecard to run on your own data.

Judge calibration & evals The eval harness and the human-vs-judge calibration gate that blocks launch on safety.

Operations runbook The operator handoff: shipping, watching, production-readiness definition of done, incident posture.

Costed blueprint Demo-to-production across five layers, with a timeline band and cost at volume.

Explore the library

Governance & readiness HIPAA readiness, the EU AI Act, the NIST AI RMF, ISO/IEC 42001, and regional data-protection regimes.

Architecture decisions The ADRs behind the agent graph, retrieval, guardrails, evaluation, and deployment.

Diagrams C4 context, container, and component views, the agent state machine, and the evaluation pipeline.

Reference The regulatory posture, model card, data statement, glossary, and the enterprise agent stack.