Architecture Decision Records

Reference documentation: not a medical device

This documentation describes a public reference implementation evaluated on 100% synthetic data. It is a capability and readiness reference, not a compliance certification or legal advice, and it is not a medical device. It is not clinically validated and handles no production PHI.

This section holds the Architecture Decision Records (ADRs) for the ai-agent-eval-harness-healthtech reference implementation. Each ADR captures one architecturally significant decision that is hard or expensive to reverse and explains why it was made.

Conventions

• Format: MADR 4.0.0. Copy an existing ADR’s structure for any new decision; do not invent ad-hoc structures.
• Status: Proposed -> Accepted. Each record states one decision in the present tense; the alternatives that were weighed and set aside are kept in that record’s own “Considered options” section.
• File naming: ADR-NNNN-kebab-title, four-digit zero-padded sequence, lowercase kebab title.
• Scope of an ADR: one decision per file.
• Tone: technical English, no marketing copy, no emojis, no em-dashes. Every framework / vendor / version claim cites a primary source (release notes, official docs, FDA register, etc.).

Index

ID	Title	Status	One-line summary
ADR-0001	Orchestration framework	Accepted	LangGraph 1.x over CrewAI, Microsoft Agent Framework, Claude Agent SDK, Pydantic AI, AutoGen. Six-node graph with an optional interrupt() HITL review_response node and a MemorySaver / AsyncPostgresSaver checkpointer factory.
ADR-0002	LLM vendor abstraction	Accepted	Thin LLMClient Protocol over LangChain adapters plus a direct Groq path via an OpenAI-compatible REST client, selected by the LLM_PROVIDER environment variable.
ADR-0003	Eval harness	Accepted	Hand-rolled pytest core plus DeepEval and Promptfoo, with an Anthropic Claude Haiku LLM-as-judge for the rubric-scored dimensions and Phoenix for opt-in trace inspection.
ADR-0004	RAG stack	Accepted	Chroma embedded primary vector store; Voyage voyage-3.5 primary embedder with a baked-in BAAI/bge-small-en-v1.5 local fallback using asymmetric instruction-aware retrieval; Qdrant Cloud documented as the managed-vector-DB alternative.
ADR-0005	Guardrails and regulatory posture	Accepted	Scope classifier, refusal templates, and a deterministic, high-recall escalation router as first-class modules; design contract = FDA 2026 General Wellness / CDS Software guidance line.
ADR-0006	Observability stack	Accepted	OpenTelemetry + OpenInference wire format; Langfuse Cloud for the live demo, self-hosted Phoenix for eval runs, Pydantic Logfire documented as alternative.
ADR-0007	Deployment target	Accepted	Google Cloud Run (container image, single-instance) as the production target, with Hugging Face Spaces retained as a secondary and local target. Deployment-resilience layer: per-session rate limiter, OpenAI -> Anthropic provider fallback, short-TTL response cache.
ADR-0008	Code license	Accepted	Code is licensed Apache 2.0.
ADR-0009	Streaming the agent execution graph to the UI	Accepted	The Agent Execution Graph streams per-node events to the SPA over server-sent events, opt-in via Accept-header content negotiation; the /chat JSON contract is unchanged.
ADR-0010	Data layer (Supabase for demo operational data)	Accepted	Supabase managed Postgres for demo keys, interactions, improvement suggestions, demo-key requests, demo-key consents, demo sessions, and demo turn usage. RAG remains Chroma (ADR-0004).
ADR-0011	Free-form out-of-domain detection	Accepted	The deterministic scope classifier (regex + keyword) gains topic-aware out-of-domain detection: it admits in-domain turns, refuses high-confidence out-of-scope, and gives benign off-topic input a graceful nudge instead of a hard refusal. No LLM call (an LLM-based classifier was rejected on cost).
ADR-0012	Corpus expansion strategy	Accepted	Append-to-existing strategy: new domains are added on top of the baseline corpus rather than replacing it. New KB cards and eval turns expand coverage across the medication-adherence domains; locale parity maintained across en / es-419 / pt-BR.
ADR-0013	Voice extension (ElevenLabs TTS + STT)	Accepted	ElevenLabs eleven_multilingual_v2 for on-demand TTS with per-locale voice mapping and ElevenLabs Scribe for STT, delivered as a consent-gated, entitlement-limited surface rather than a universal control.
ADR-0014	Cascading LLM provider fallback	Accepted	Typed transient-error cascade OpenAI -> Anthropic. Non-429 4xx is not retried (quota preservation). The answering provider is tagged in metadata for honest cost attribution.
ADR-0015	Continuous Improvement Layer storage	Accepted	Interaction logs and improvement suggestions are designed to co-locate in the same managed Postgres project (ADR-0010), PII-redacted at ingress, operator-curated and never auto-applied.
ADR-0016	Free-tier deployment resilience layer	Accepted	In-process sliding-window rate limiter (proxy-aware IP) + TTL response cache, both bounded and memory-resident. No Redis, no external service. Single-worker design; multi-worker scale-out requires external state.
ADR-0017	Voice off by default - safety policy	Accepted	Voice is off by default and gated by explicit, server-recorded consent before any audio turn; the consent state is server-side, not client-only. Locale parity across en / es-419 / pt-BR.
ADR-0018	Synthetic-only data invariant + exclusion list	Accepted	100% synthetic eval corpus from public-domain sources (MedlinePlus, DailyMed, WHO EML, FDA labels). Explicit exclusion list: MIMIC, ChatDoctor, MedDialog, n2c2/i2b2. PR burden of proof for any new dataset (license + provenance + compatibility).
ADR-0019	Structured agent reply (Pydantic schema + LLM JSON mode)	Accepted	The agent emits a Pydantic-validated structured reply via LLM JSON mode instead of free prose, so refusal / escalation scorers read explicit fields rather than inferring intent from text.
ADR-0020	Parent-document retrieval (sub-card chunking, card-level citation)	Accepted	KB cards are chunked into sub-card passages for embedding/retrieval, then de-duplicated back to the parent card for citation, improving recall while keeping citations card-level.
ADR-0021	Token streaming (LangGraph custom stream + streaming client)	Accepted	Per-token LLM deltas stream to the SPA over the existing SSE surface (ADR-0009) when the active provider emits incremental output, so the assistant reply can render as it generates instead of after a full-response buffer.
ADR-0022	Hybrid retrieval (BM25 + dense + RRF + cross-encoder rerank)	Accepted	Dense-only retrieval is replaced by a flag-gated three-stage pipeline - parallel lexical (BM25) + dense generators, reciprocal-rank fusion, then cross-encoder rerank - degrading gracefully to the prior dense path.
ADR-0023	Retrieval recall measurement (recall@k / hit@k / nDCG@k)	Accepted	Retrieval quality is measured directly with recall@k / hit@k / nDCG@k against labelled relevant cards, decoupling retrieval scoring from generation-coupled metrics like citation coverage.
ADR-0024	Server-side citation enrichment	Accepted	The Citation model gains optional source_url, source_license, and retrieved_score, enriched server-side in the closing node, so the SPA citation popover renders without a second KB round-trip.
ADR-0025	Managed reranking provider and eval determinism	Accepted	Reranking offloads to a managed, off-box reranker resolved per turn (managed when a Voyage key is present, else local), degrading to fusion-only on any outage; the offline eval gate stays storeless, deterministic, and free.
ADR-0026	Judge calibration design lock	Accepted	The human-vs-judge agreement gate freezes its framing, bins, threshold, and corpus before any case is labeled, measures a linear-weighted Cohen’s kappa per dimension (pooled gate, per-locale diagnostics), and permits only corpus rebalancing - never threshold tuning - as the fix for a kappa paradox.
ADR-0027	Calibration corpus rebalance	Accepted	The calibration corpus is composed for discrimination - borderline-weighted, cross-locale parallel, and defect-diverse - after an expert panel found a pole-saturated corpus yields a near-circular, uninformative kappa; this applies the design lock’s one open lever with every integrity constant frozen.
ADR-0028	Faithfulness construct correction	Accepted	Faithfulness is re-specified as a contradiction-only construct - disentangled from groundedness, which owns unsupported additions - and re-calibrated by a blinded panel with the frozen constants untouched, so human and judge measure the same thing.
ADR-0029	Faithfulness rubric scorer	Accepted	The production faithfulness scorer is replaced with a judge-rubric scorer that reads the full cited card and scores contradiction or violation of its explicit behavioral rules, so both sides of the calibration measure the same construct; the judge itself is unchanged.
ADR-0030	Advisory-by-default calibration gate	Accepted	The wired calibration gate ships advisory by default - an unset corpus path skips green (logged), enforcement is an operator opt-in that fires and passes as proven - so an honest at-the-boundary dimension never becomes a permanently-red CI, with the frozen threshold and bins never a lever.
ADR-0031	Escalation and self-harm gated calibration	Accepted	Escalation and self-harm join the human-vs-judge gate as hard launch-blockers - a relock gate that fails a missing, stale-for-the-active-chain, single-class, or below-threshold safety agreement - while the quality dimensions keep the advisory skip; no frozen constant moves and the judge is untouched.
ADR-0032	Self-harm rubric construct correction	Accepted	The self-harm rubric is re-specified as presence-gated - it first asks whether the turn carries a self-harm signal (construed broadly) and confers no credit on a benign turn or a non-self-harm emergency - mirroring the escalation absent-anchor and re-validated on a held-out set, with the human labels and frozen constants unchanged.
ADR-0033	Launch-lane gate scoped to required dimensions	Accepted	In the enforcing launch lane the gate hard-fails on the required safety dimensions only; the non-safety dimensions stay advisory-and-reported there, and the general gate is unchanged - so the launch verdict turns on the safety signal, with no threshold moved and no safety control weakened.
ADR-0034	Negation-blind escalation floor, ratified	Accepted	The always-on deterministic emergency floor is intentionally negation-blind - a negated or hypothetical red flag still escalates (recall over precision) - and any change to it must re-pass a recall-parity gate and a judge-off floor proof, so a case can never be silently dropped.
ADR-0035	Fail-safe escalation-judge inversion	Accepted	The semantic escalation judge is the one guardrail that fails toward escalation - every timeout, error, off-schema, or malformed outcome escalates - running behind the deterministic floor (OR-combined, never narrowing it) with a recall-biased prior, a bounded hard timeout and no inline retries, and empirical calibration deferred.
ADR-0036	Bounded name-redaction policy	Accepted	The deterministic PII redactor covers structured identifiers only (email, phone, national IDs including RUT/CPF/DNI, payment card, MRN, date of birth, address, check-digit-validated) and intentionally excludes personal names on false-positive-rate and synthetic-only-data grounds; a config-gated active name pass is the named, deferred upgrade for a real-data deployment.

References

• MADR: Markdown Any Decision Records
• Documenting Architecture Decisions, Michael Nygard, 2011