Skip to content
AI Agent Eval Harness

ADR-0004: RAG stack

  • Status: Accepted
  • Date: 2026-03-18
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The agent grounds every clinical assertion in a small knowledge base of 30 to 50 cards covering drug-interaction summaries, adherence barriers, motivational-interviewing talking points, and escalation criteria. KB sources are restricted to public-domain or properly-attributed material: DailyMed (FDA SPL), MedlinePlus (US-gov), and paraphrased WHO Essential Medicines List entries. The retrieval layer does not need horizontal scaling; it needs to be cheap, reproducible, and self-contained inside the Docker image we ship.

At the same time, this is a reference implementation. It has to show when an embedded vector store is the right call and when a managed vector DB is the right call. The narrative is “start embedded, document the managed path”.

How do we choose a vector store and an embedding model that (a) run at $0 with no external accounts in the default demo, (b) demonstrate managed-vector-DB awareness as an alternative path, (c) match the quality the LLM-as-judge eval will hold us to, and (d) keep deterministic reproducibility for the eval harness?

Decision Drivers

Section titled “Decision Drivers”
  • Zero external services for the default demo path; the vector store must work inside the Docker image
  • Reproducibility: the same KB plus the same embedding model plus the same query must yield the same retrieval, so the eval scorer for groundedness is stable
  • Cost: free at demo scale (50 cards or fewer, hundreds of queries per day), with a documented free-tier managed alternative
  • Embedding quality: the judge eval will penalise weak retrieval through the faithfulness and hallucination gates; the primary embedding model should be a recent strong one, with a baked-in offline fallback if no API key is configured
  • License: every component permissively licensed; embeddings generated for the KB ship inside the image without per-query cost at runtime if the offline fallback is used

Considered Options

Section titled “Considered Options”
  • Chroma embedded (DuckDB+Parquet) + Voyage AI voyage-3.5 as primary embeddings, sentence-transformers BAAI/bge-small-en-v1.5 as baked-in offline fallback (chosen)
  • Qdrant Cloud free tier + Voyage AI voyage-3.5: managed service, generous free tier, but external dependency
  • FAISS as an embedded store: high performance, but metadata story is thinner than Chroma
  • Postgres + pgvector: co-located with the LangGraph Postgres saver, but adds operational surface for a 50-card KB
  • OpenAI text-embedding-3-large as the embedding model

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Chroma embedded as the primary vector store, with Voyage AI voyage-3.5 as the primary embedding model and sentence-transformers BAAI/bge-small-en-v1.5 as the baked-in offline fallback. Qdrant Cloud’s free tier is documented as the managed alternative path; it is the right answer for any reader whose use case has more than ~50K chunks or needs a hosted dashboard.

Voyage AI gives 200 million free tokens on the voyage-3.5 family to new users, which is far in excess of what the KB needs (the entire 50-card corpus embeds in under a million tokens). The sentence-transformers fallback is baked into the Docker image, so the demo runs with zero external API keys if the user prefers; the harness picks the fallback automatically when no Voyage API key is set.

The choice keeps the live demo zero-cost, gives a clean managed-DB alternative for readers who want one, and uses two embedding paths that both score well on retrieval benchmarks.

Confirmation

Section titled “Confirmation”
  • The default Compose file runs Chroma embedded; no external service is required to bring the demo up
  • An optional Compose file declares a Qdrant Cloud configuration with documented free-tier signup steps, exercised in a manual integration test
  • The embedder factory selects Voyage AI if a Voyage API key is set, and falls back to the local sentence-transformers model otherwise; a unit test exercises both branches
  • The KB build writes a manifest with model id, model version, embedding dimension, and SHA-256 of every card, so the eval harness can assert the retrieval surface is the expected one

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Demo runs offline: no external service is required, which keeps the cold-start wake-up path fast and deterministic
  • The eval harness sees a deterministic retrieval surface (Chroma
    • pinned embeddings + manifest hash), exactly what the groundedness scorer needs
  • Voyage AI voyage-3.5 is a recent, strong embedding model (announced 2025-05-20); the 200M free-token tier covers the KB many times over
  • The offline fallback removes the “needs an API key” reading for any reader who wants to clone-and-run
  • Qdrant Cloud as a documented alternative path lets the project signal managed-vector-DB awareness without inheriting the free tier’s suspension risk

Negative

Section titled “Negative”
  • The baked-in sentence-transformers model adds to the Docker image size; accepted because it removes the “embeddings need an internet round-trip” failure mode
  • Chroma embedded scales poorly past hundreds of thousands of chunks; irrelevant for a 50-card KB but worth flagging
  • Two embedding paths mean two retrieval signatures; the manifest hash makes the difference auditable, but eval results must be compared within one embedding path, not across them

Neutral

Section titled “Neutral”
  • The project gains chromadb and voyageai dependencies
  • The image carries the sentence-transformers weights; intentional and documented
  • A future migration to Qdrant Cloud is a Protocol-level swap, not a rewrite: the store abstraction covers both backends

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Chroma embedded + Voyage AI primary + bge-small-en-v1.5 fallback

Section titled “Chroma embedded + Voyage AI primary + bge-small-en-v1.5 fallback”
  • Good, because the default path runs with zero external services
  • Good, because Voyage AI’s 200M-token free tier covers the KB
  • Good, because the offline fallback removes the “needs-a-key” reading
  • Good, because the eval harness sees a deterministic retrieval surface
  • Bad, because the Docker image grows for the baked-in fallback model
  • Bad, because Chroma embedded does not scale to hundreds of thousands of chunks

Qdrant Cloud free tier + Voyage AI

Section titled “Qdrant Cloud free tier + Voyage AI”
  • Good, because the managed dashboard and free tier (1 GB, no card) are generous
  • Bad, because the demo would depend on an external service and Qdrant’s account policy; every reader would have to sign up
  • Kept as a documented alternative

FAISS embedded

Section titled “FAISS embedded”
  • Good, because FAISS is fast and battle-tested
  • Bad, because metadata + filtering ergonomics are weaker than Chroma’s

Postgres + pgvector

Section titled “Postgres + pgvector”
  • Good, because Postgres is already used for the conversation-state saver
  • Bad, because co-locating conversation state and vector storage complicates ops for a 50-card KB, and shipping Postgres for retrieval contradicts the embedded-by-default posture

OpenAI text-embedding-3-large

Section titled “OpenAI text-embedding-3-large”
  • Good, because it is a strong, well-known embedding model
  • Bad, because it would force the demo to require an OpenAI key for retrieval alone, and there is no clean offline fallback with comparable quality outside sentence-transformers anyway

More Information

Section titled “More Information”

As-built embedder and asymmetric retrieval

Section titled “As-built embedder and asymmetric retrieval”

Primary embedder: Voyage voyage-3.5, with a baked-in local fallback. The embedder factory is key-driven: it resolves Voyage voyage-3.5 when a Voyage API key is configured, as in the live deployment, and falls back to the baked-in local BAAI/bge-small-en-v1.5 otherwise. The fallback is a lightweight model, roughly 130 MB, CPU-friendly on a small single instance, so the demo also runs at $0 with no external keys while keeping strong retrieval quality.

Retrieval is asymmetric and instruction-aware. The BGE v1.5 family is instruction-tuned and asymmetric. The shipped code honours that: a query is embedded with the documented BGE retrieval instruction prefix (Represent this sentence for searching relevant passages: ); a passage is embedded with no prefix; every vector is L2-normalized so Chroma’s inner-product search behaves as cosine similarity. A symmetric general-purpose model (for example all-MiniLM-L6-v2) receives no instruction prefix. Used without the asymmetric handling, BGE retrieval quality degrades; the retrieval layer is built to apply it.

Retrieval similarity threshold ships disabled. A retrieval-minimum-similarity setting exists but ships disabled by default. On the single-domain KB corpus a threshold cannot separate a near-miss off-corpus clinical question from a genuine in-corpus one without false-refusing the latter. The agent refuses on zero-hit retrieval; a near-miss off-corpus question is answered against the closest card. The threshold is left in place, disabled, so a broader, more topically diverse corpus can enable it later. See the model card for the full limitation.