Enterprise agent stack

Purpose

This document maps the gap between the current demo reference implementation and what a production deployment would require. It serves operators planning to adapt the demo into a real deployment, and readers assessing the reference implementation’s production awareness.

The architecture is presented across five layers (Compute/Hosting, Storage/Data, AI/ML, Observability, Security/Compliance) using Mermaid diagrams for visual clarity.

Scope

This covers the medication-adherence conversational agent as deployed on Google Cloud Run, the CI/eval pipeline, and a generic production reference architecture. The demo runs on a specific host (Cloud Run); the production reference does not prescribe a specific cloud vendor - it identifies what a real deployment would need, not which vendor to use.

1. System Context Diagram

The highest-level view: who uses the system, what external systems it depends on, and what data flows between them.

graph TB
    Patient[Patient / Caregiver<br/>Chat + Voice UI]
    Operator[Operator / Reviewer<br/>Demo Key Mgmt + HITL Review]
    Agent[Medication Adherence Agent<br/>FastAPI + LangGraph]

    LLM[LLM Providers<br/>OpenAI + Anthropic<br/>Groq judge fallback]
    TTS[ElevenLabs<br/>TTS + STT]
    KB[Knowledge Base<br/>Chroma Embedded<br/>38 Synthetic KB Cards]
    Obs[Observability<br/>OTel / Langfuse / Phoenix]
    Supabase[(Supabase<br/>Demo Operational Data)]

    Patient -->|Chat / Voice Input| Agent
    Operator -->|Key Mgmt + Review| Agent
    Agent -->|Generate + Judge| LLM
    Agent -->|TTS / STT| TTS
    Agent -->|Retrieve + Cite| KB
    Agent -->|Spans + Metrics| Obs
    Agent -->|Keys + Sessions + Interactions| Supabase
    Agent -->|Response| Patient

Key boundaries:

The agent is the sole trust boundary. All user input enters through FastAPI and passes through the guardrail pipeline before reaching any external system.
LLM providers, ElevenLabs, and observability sinks are external dependencies. The agent degrades gracefully when any of them are unavailable.
Supabase stores operational data (demo keys, interactions, improvement suggestions) but the agent continues serving turns if it is down.

2. Container Diagrams

Three deployment contexts: the live demo, the CI/eval pipeline, and a production reference.

2.1 Demo Container (Current)

graph TB
    subgraph CloudRun["Google Cloud Run<br/>Single Instance (max-instances 1)<br/>us-central1 / scale-to-zero"]
        FastAPI[FastAPI<br/>Single Uvicorn Worker]
        Graph[LangGraph StateGraph<br/>6 Nodes + Optional HITL]
        MemorySaver[MemorySaver<br/>In-Memory Checkpointer]
        Chroma[Chroma Embedded<br/>DuckDB + Parquet<br/>38 Synthetic Cards]
        Resilience[Resilience Layer<br/>Rate Limiter + Fallback + Cache]
        OTel[OTel Exporter]
    end

    subgraph External["External Services"]
        OpenAI[OpenAI<br/>gpt-4o-mini<br/>Primary Completions]
        Anthropic[Anthropic<br/>claude-haiku-4-5<br/>Fallback + Eval Judge]
        Langfuse[Langfuse Cloud Hobby<br/>50K obs/month]
        Supabase[(Supabase Free Tier<br/>500 MB Postgres)]
        ElevenLabs[ElevenLabs<br/>TTS + STT<br/>Entitlement-gated]
    end

    FastAPI --> Graph
    Graph --> MemorySaver
    Graph --> Chroma
    Graph --> Resilience
    Resilience --> OpenAI
    Resilience --> Anthropic
    FastAPI --> OTel
    OTel --> Langfuse
    FastAPI --> Supabase
    FastAPI --> ElevenLabs

Cost: near $0/month at demo scale. Hosting (Cloud Run’s always-free allowance under a single scale-to-zero instance), observability (Langfuse Cloud Hobby, 50K obs/month), and operational data (Supabase free tier) stay within free tiers at demo traffic (50-150 reviewers, 5-10 turns each). LLM completions run on paid providers (OpenAI primary, Anthropic fallback), metered by the as-if-paid cost ledger and capped by a hard daily budget breaker, so demo spend is bounded. The first request after the instance scales to zero pays a container cold start. The same image also runs at genuinely $0 on Hugging Face Spaces as a secondary target.

2.2 CI/Eval Container

graph TB
    subgraph GitHub["GitHub Actions<br/>ubuntu-latest"]
        Build[Docker Build<br/>Same Dockerfile]
        Pytest[pytest Runner<br/>Deterministic Core<br/>Coverage Gate]
    end

    subgraph Eval["Eval Pipeline"]
        Deterministic[Deterministic Scorers<br/>Citation + Refusal<br/>Escalation + Recall]
        Judge[LLM-as-Judge<br/>Anthropic claude-haiku-4-5<br/>Groundedness + Faithfulness]
        DeepEval[DeepEval<br/>Hallucination Metric]
        Promptfoo[Promptfoo Nightly<br/>OWASP LLM Top 10 subset]
    end

    subgraph TestInfra["Test Infrastructure"]
        StubLLM[StubLLMClient<br/>Deterministic, Key-Free]
        Phoenix[Phoenix Self-Hosted<br/>Docker Compose Profile]
        GoldenCorpus[Golden Corpus<br/>315 Cases / 3 Locales<br/>JSONL Format]
    end

    Build --> Pytest
    Pytest --> StubLLM
    Pytest --> Deterministic
    Pytest --> Judge
    Pytest --> DeepEval
    Pytest --> Phoenix
    Pytest --> GoldenCorpus
    Promptfoo --> GoldenCorpus

Determinism guarantee: The CI gate passes key-free via a deterministic stub LLM client. Judge-backed scorers activate only when a judge provider key (Anthropic) is set. The same Docker image that passes CI is the image that ships to Cloud Run.

2.3 Production Reference Container

graph TB
    subgraph Cloud["Cloud Hosting<br/>e.g., AWS / GCP / Azure"]
        LB[Load Balancer<br/>TLS Termination]
        FastAPIProd[FastAPI<br/>Multiple Workers<br/>Behind Reverse Proxy]
        GraphProd[LangGraph StateGraph<br/>Same 6 Nodes]
        PostgresSaver[AsyncPostgresSaver<br/>Durable Checkpointer]
        ManagedVector[Managed Vector Store<br/>e.g., Qdrant Cloud / Pinecone]
        ResilienceProd[Resilience Layer<br/>Redis-backed Rate Limiting<br/>Circuit Breakers]
    end

    subgraph LLMProd["LLM Providers"]
        LLMPrimary[Primary LLM<br/>SLA-backed API]
        LLMFallback[Fallback LLM<br/>Different Vendor]
        LLMJudge[Judge Model<br/>Dedicated Instance]
    end

    subgraph SecurityProd["Security + Compliance"]
        WAF[Web Application Firewall]
        SIEM[SIEM Integration<br/>Audit Log Export]
        SecretsMgr[Secrets Manager<br/>Vault / Cloud Secret Manager]
        KMS[Key Management Service]
    end

    LB --> WAF
    WAF --> FastAPIProd
    FastAPIProd --> GraphProd
    GraphProd --> PostgresSaver
    GraphProd --> ManagedVector
    FastAPIProd --> ResilienceProd
    ResilienceProd --> LLMPrimary
    ResilienceProd --> LLMFallback
    FastAPIProd --> SIEM
    FastAPIProd --> SecretsMgr
    FastAPIProd --> KMS

Production gap: The demo runs on a single scale-to-zero instance with in-memory state. Production needs horizontal scaling behind a load balancer, durable persistence, managed secrets, WAF, SIEM, and SLA-backed LLM providers. The architecture is the same (six-node LangGraph StateGraph) - only the infrastructure layers change.

3. Component Diagram: Agent Graph Internals

The six-node LangGraph StateGraph with the optional seventh HITL node.

graph LR
    Intake[intake<br/>Mark turn start<br/>Seed trace_id<br/>PII redaction]
    GuardPre[guardrail_pre<br/>Scope classification<br/>Escalation detection<br/>Out-of-domain routing]
    Retrieve[retrieve_context<br/>RAG from Chroma<br/>Citation matching]
    Generate[generate_response<br/>LLM generation<br/>Citation enforcement]
    GuardPost[guardrail_post<br/>Output safety check<br/>Refusal verification]
    Closing[closing<br/>Duration calculation<br/>OTel span close]
    Review[review_response<br/>HITL pause<br/>Optional 7th node]

    Intake --> GuardPre
    GuardPre -->|in-scope| Retrieve
    GuardPre -->|out-of-domain| Generate
    GuardPre -->|escalation| Closing
    Retrieve --> Generate
    Generate --> GuardPost
    GuardPost -->|safe| Closing
    GuardPost -->|flagged| Review
    Review -->|approved| Closing
    Review -->|rejected| Generate

Data flow:

intake - marks the turn start, seeds the trace ID, applies PII redaction.
guardrail_pre - runs the scope classifier. In-scope messages proceed to RAG retrieval. Out-of-domain messages bypass retrieval and get a graceful fallback. Escalation-triggering messages short-circuit to closing with a referral.
retrieve_context - queries Chroma embedded for relevant KB cards. If no card matches, the agent refuses rather than hallucinating.
generate_response - calls the configured LLM provider with the retrieved context and a citation-enforcing prompt.
guardrail_post - verifies the generated response does not contain dosing, diagnosis, or other out-of-scope content. Flagged responses route to HITL review.
closing - calculates turn duration, closes the OTel span, and returns the response.
review_response (optional) - a LangGraph interrupt for human-in-the-loop review of flagged turns. Disabled by default in eval mode.

4. Five-Layer Comparison: Demo vs Production

Layer	Demo (Current)	Production Reference
Compute/Hosting	Google Cloud Run; single container instance (`--max-instances 1`); region `us-central1`; scale-to-zero when idle; keyless, tag-gated deploy from the same image as CI	Cloud hosting (any vendor); multiple instances behind a load balancer; auto-scaling; zero-downtime deploys; SLA-backed uptime
Storage/Data	Chroma embedded (DuckDB+Parquet) for RAG; in-memory `MemorySaver` for conversation state; Supabase free tier (500 MB) for demo operational data	Managed vector store (Qdrant Cloud / Pinecone) for RAG; `AsyncPostgresSaver` for durable conversation state; managed Postgres (RDS / Cloud SQL) with backups for operational data; data encryption at rest
AI/ML	OpenAI `gpt-4o-mini` (primary) with Anthropic `claude-haiku-4-5` (fallback) for completions; Anthropic `claude-haiku-4-5` eval judge (Groq judge fallback); Voyage `voyage-3.5` managed embeddings (primary) with a baked-in `BAAI/bge-small-en-v1.5` local fallback; deterministic stub LLM client for CI; synthetic KB cards	SLA-backed LLM API with dedicated throughput; a dedicated judge instance; managed embedding service; expanded knowledge base with clinical review; continuous eval with drift detection
Observability	OTel spans with OpenInference conventions; Langfuse Cloud Hobby (50K obs/month); Phoenix self-hosted in Docker for eval runs; Arize AX as a secondary dashboard; OTLP export	Full OTel stack with collector, sampling, and retention policies; dedicated observability backend (Datadog / Grafana / Honeycomb); alerting on latency, error rate, and cost anomalies; audit log export to SIEM
Security/Compliance	No secrets in repo (gitleaks CI); lockfile pinned; Dependabot enabled; no PHI / no real EHR; FDA General Wellness framing; PII redaction at ingress and egress; Cloudflare edge (Turnstile, origin-lock)	WAF / DDoS protection; managed secrets (Vault / Cloud Secret Manager); BAA with all LLM providers; HIPAA Security Rule compliance; SOC 2 Type II; penetration testing; incident response plan

Production timeline (qualitative)

The five-layer comparison names what changes between the demo and a production deployment; this band sketches when, as qualitative phases rather than committed dates. Each phase is gated on the prior one, and the agent code (the six-node graph and its guardrails) carries across all four unchanged - the work is in the infrastructure and governance layers.

graph LR
    A[Demo today<br/>Cloud Run single instance<br/>synthetic data, near $0] --> B[Hardening<br/>durable state + shared store<br/>managed vector store]
    B --> C[Pilot<br/>clinical KB review<br/>BAA + compliance controls]
    C --> D[Production<br/>multi-instance autoscale<br/>SLA + alerting + SIEM]

The demo already ships the load-bearing hard part - the measured, gated agent behaviour - so the timeline is dominated by infrastructure and compliance work, not by rebuilding the agent.

Cost at volume (illustrative)

The demo runs near $0 / month: scale-to-zero hosting, free-tier observability and operational data, and LLM completions metered by the as-if-paid ledger under a hard daily breaker. Production cost is not that number scaled linearly - it has two parts that scale differently.

Variable cost is LLM inference: tokens per turn times the provider rate. It scales roughly linearly with traffic and is the dominant line at volume. The as-if-paid ledger already meters this per turn, so a real quote starts from a measured basis rather than a guess.
Fixed cost is the infrastructure floor: a warm instance, a managed vector store, a managed Postgres, and an observability backend with retention. It appears as a step once you leave the free tiers and stays largely flat across a wide traffic band.

The shape, as illustrative order-of-magnitude bands - real figures depend on the provider contract, the model, retrieval size, and voice usage:

Monthly turns	What dominates	Cost shape
Demo (hundreds)	free tiers + metered LLM	near `$0`, bounded by the daily breaker
Low thousands	LLM tokens per turn	small variable spend; infra still near-free
Tens of thousands	LLM tokens + warm instance + managed stores	a fixed infra floor appears; variable LLM cost dominates
Hundreds of thousands+	LLM throughput + retention	provider-contract territory - negotiate throughput and rates

Two levers move the variable line without touching the agent: a smaller or cheaper generation model, and retrieval that keeps the prompt tight (fewer tokens per turn). Both are configuration, not a rebuild.

5. Layer Details

5.1 Compute/Hosting

Current state. The demo runs on Google Cloud Run as a single container instance (--max-instances 1, region us-central1), scaling to zero when idle. Deploys are keyless (Workload Identity Federation) and gated on a green CI matrix for the release tag; the same Dockerfile builds in CI, in local development, and on Cloud Run. The single-instance ceiling is deliberate: the rate limiter, response cache, and HITL checkpointer are in-process.

Production gap. A production deployment needs multiple instances behind a load balancer, auto-scaling to handle traffic spikes, zero-downtime deployments, and SLA-backed uptime (typically 99.9% or higher). Cold starts must be eliminated with a warm minimum instance count. The agent code itself is portable - FastAPI with uvicorn works behind any reverse proxy - but the infrastructure layer needs significant investment.

Migration path. The same Docker image can deploy to any Docker-capable host; Hugging Face Spaces is documented as a secondary target (see deploy). Scaling out on Cloud Run means raising the instance ceiling, adding a minimum warm instance, and moving the in-process rate limiter, cache, and checkpointer to a shared store (Redis, Postgres) so they behave consistently across instances.

5.2 Storage/Data

Current state. RAG retrieval uses Chroma embedded (DuckDB+Parquet), which is zero-network and runs entirely within the instance’s memory. Conversation state uses in-memory MemorySaver by default; it is lost on instance restart. Demo operational data (keys, sessions, interactions) is stored in Supabase free tier (500 MB managed Postgres). The knowledge base contains synthetic KB cards in JSONL format.

Production gap. A production deployment needs durable conversation state (AsyncPostgresSaver via a Postgres connection string). RAG retrieval should use a managed vector store (Qdrant Cloud, Pinecone, or pgvector in the operational Postgres) for persistence, larger corpus support, and query performance at scale. Operational data needs a managed Postgres with automated backups, point-in-time recovery, and encryption at rest.

Migration path. The agent already provisions a durable Postgres checkpointer factory via a connection string. Switching from Chroma embedded to a managed vector store requires updating the retriever configuration but not the agent graph itself. The committed JSONL format is already compatible with bulk loading into any vector store.

5.3 AI/ML

Current state. LLM completions run a two-provider cascade: OpenAI gpt-4o-mini as the primary, falling back to Anthropic claude-haiku-4-5 on a transient upstream failure. The evaluation judge is Anthropic claude-haiku-4-5, with Groq llama-3.3-70b-versatile as a judge fallback tier. Embeddings default to the managed Voyage embedder when a Voyage key is present and fall back to BAAI/bge-small-en-v1.5 locally (no API call). The eval harness is a hand-rolled pytest core plus DeepEval (Hallucination metric), an Anthropic LLM-as-judge, and Promptfoo (red-team), run against synthetic golden cases across three locales (en, es-419, pt-BR). The knowledge base covers several medication-adherence domains with 38 synthetic cards.

Production gap. Provider APIs have rate limits and shared infrastructure. A production deployment needs SLA-backed LLM providers with guaranteed throughput and latency. The eval judge should run on a dedicated instance for consistency. The knowledge base would need clinical review and expansion beyond synthetic data. Continuous eval with drift detection is essential for production safety.

Migration path. The client Protocol makes provider swapping a configuration change. Adding a new provider requires implementing the Protocol (at most a handful of methods) and setting the generation cascade via configuration. The eval harness already scores against configurable thresholds and is CI-gated.

5.4 Observability

Current state. OpenTelemetry spans with OpenInference semantic conventions wrap every node, LLM call, retrieval, and guardrail decision. Sinks: Langfuse Cloud Hobby for the live demo (50K observations/month, 30-day retention), Phoenix self-hosted via Docker Compose for eval runs, and Arize AX as a secondary dashboard. No alerting is configured.

Production gap. A production deployment needs a full OTel stack: collector with configurable sampling, a dedicated backend with long retention (90+ days), alerting on latency percentiles, error rates, and cost anomalies, and audit log export to SIEM for compliance. The current spans already carry the right attributes; the gap is in the backend infrastructure, not the instrumentation.

Migration path. The OTel instrumentation is vendor-neutral. Switching backends is an OTLP exporter configuration change. The existing span attributes (interaction.*, llm.*, retrieval.*) are compatible with any OTel-compatible backend.

5.5 Security/Compliance

Current state. No secrets in the repository (enforced by gitleaks in CI). Dependencies are pinned via the lockfile with Dependabot monitoring. No PHI, no real EHR data, no patient-identifiable information (100% synthetic data). The agent operates under FDA 2026 General Wellness / CDS framing (see regulatory posture). PII redaction is applied at both ingress and egress. A Cloudflare edge posture (Turnstile bot check, origin-lock) fronts the service. Demo key fingerprinting uses a pseudonymized (re-identifiable) sha256 fingerprint, not anonymization.

Production gap. A production deployment handling real patient data would need: Web Application Firewall and DDoS protection, managed secrets (Vault, Cloud Secret Manager), Business Associate Agreements with all LLM providers, HIPAA Security Rule compliance (risk assessment, breach notification, minimum necessary access), SOC 2 Type II certification, regular penetration testing, and an incident response plan. The regulatory posture would shift from General Wellness to a full compliance framework.

Migration path. The guardrail architecture is designed for production: deterministic scope classification, auditable refusal templates, and rule-based escalation are all production-grade patterns. The security gap is primarily in infrastructure (WAF, secrets management, encryption) and process (risk assessments, audit schedules), not in the application code.

Cross-References

Topic	ADR
Orchestration framework (six-node StateGraph)	ADR-0001
LLM vendor abstraction (Protocol + adapters)	ADR-0002
Eval harness (pytest + DeepEval + Anthropic judge + Promptfoo)	ADR-0003
RAG stack (Chroma embedded)	ADR-0004
Guardrails (scope + refusal + escalation)	ADR-0005
Observability (OTel + OpenInference)	ADR-0006
Deployment (Google Cloud Run)	ADR-0007
Data layer (Supabase free tier)	ADR-0010
Voice extension (ElevenLabs TTS/STT)	ADR-0013
Streaming architecture (SSE events)	ADR-0009