Skip to content
AI Agent Eval Harness

ADR-0004: RAG stack

Context and Problem Statement

Section titled “Context and Problem Statement”

The agent grounds every clinical assertion in a small knowledge base of 30 to 50 cards covering drug-interaction summaries, adherence barriers, motivational-interviewing talking points, and escalation criteria. KB sources are restricted to public-domain or properly-attributed material: DailyMed (FDA SPL), MedlinePlus (US-gov), and paraphrased WHO Essential Medicines List entries. The retrieval layer does not need horizontal scaling; it needs to be cheap, reproducible, and self-contained inside the Docker image we ship.

At the same time, this is a reference implementation. It has to show when an embedded vector store is the right call and when a managed vector DB is the right call. The narrative is “start embedded, document the managed path”.

How do we choose a vector store and an embedding model that (a) run at $0 with no external accounts in the default demo, (b) demonstrate managed-vector-DB awareness as an alternative path, (c) match the quality the LLM-as-judge eval will hold us to, and (d) keep deterministic reproducibility for the eval harness?

Decision Drivers

Section titled “Decision Drivers”
  • Zero external services for the default demo path; the vector store must work inside the Docker image
  • Reproducibility: the same KB plus the same embedding model plus the same query must yield the same retrieval, so the eval scorer for groundedness is stable
  • Cost: free at demo scale (50 cards or fewer, hundreds of queries per day), with a documented free-tier managed alternative
  • Embedding quality: the judge eval will penalise weak retrieval through the faithfulness and hallucination gates; the primary embedding model should be a recent strong one, with an offline fallback available in the dev/eval extra when no API key is configured
  • License: every component permissively licensed; embeddings generated for the KB ship inside the image without per-query cost at runtime when the dev/eval offline fallback is used

Considered Options

Section titled “Considered Options”
  • Chroma embedded (SQLite + HNSW) + Voyage AI voyage-3.5 as primary embeddings, sentence-transformers BAAI/bge-small-en-v1.5 as a dev/eval offline fallback in the embeddings-local extra (chosen)
  • Qdrant Cloud free tier + Voyage AI voyage-3.5: managed service, generous free tier, but external dependency
  • FAISS as an embedded store: high performance, but metadata story is thinner than Chroma
  • Postgres + pgvector: co-located with the LangGraph Postgres saver, but adds operational surface for a 50-card KB
  • OpenAI text-embedding-3-large as the embedding model

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Chroma embedded as the primary vector store, with Voyage AI voyage-3.5 as the primary embedding model and sentence-transformers BAAI/bge-small-en-v1.5 as a dev/eval offline fallback in the embeddings-local extra. Qdrant Cloud’s free tier is documented as the managed alternative path; it is the right answer for any reader whose use case has more than ~50K chunks or needs a hosted dashboard.

Voyage AI gives 200 million free tokens on the voyage-3.5 family to new users, which is far in excess of what the KB needs (the entire 50-card corpus embeds in under a million tokens). The sentence-transformers fallback ships in the optional embeddings-local extra used by dev, eval, and key-less local checkouts, so those runs work with zero external API keys if the user prefers; the production image installs embeddings-cloud instead and drops torch, so when no Voyage API key is set it degrades to lexical BM25 retrieval rather than the local model.

The choice keeps the live demo zero-cost, gives a clean managed-DB alternative for readers who want one, and uses two embedding paths that both score well on retrieval benchmarks.

Confirmation

Section titled “Confirmation”
  • The demo runs Chroma embedded inside the Docker image; no external service is required to bring it up
  • The embedder factory selects Voyage AI if a Voyage API key is set, and falls back to the local sentence-transformers model otherwise; a unit test exercises both branches
  • The KB ingest upserts one Chroma row per sub-chunk and prints the upserted count; the embedder is pinned by config, so the eval harness runs against the expected retrieval surface

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Demo runs offline: no external service is required, which keeps the cold-start wake-up path fast and deterministic
  • The eval harness sees a deterministic retrieval surface (Chroma
    • pinned embeddings + the config-pinned embedder), exactly what the groundedness scorer needs
  • Voyage AI voyage-3.5 is a recent, strong embedding model (announced 2025-05-20); the 200M free-token tier covers the KB many times over
  • The offline fallback removes the “needs an API key” reading for any reader who wants to clone-and-run
  • Qdrant Cloud as a documented alternative path lets the project signal managed-vector-DB awareness without inheriting the free tier’s suspension risk

Negative

Section titled “Negative”
  • The sentence-transformers fallback enlarges the embeddings-local extra used by dev and eval, not the production image, which installs embeddings-cloud and drops torch; the extra buys offline, no-round-trip embeddings for those runs
  • Chroma embedded scales poorly past hundreds of thousands of chunks; irrelevant for a 50-card KB but worth flagging
  • Two embedding paths mean two retrieval signatures; the config-pinned embedder makes the difference auditable, but eval results must be compared within one embedding path, not across them

Neutral

Section titled “Neutral”
  • The project gains chromadb and voyageai dependencies
  • The production image does not carry the sentence-transformers weights; they ship only in the embeddings-local extra for dev and eval, intentional and documented
  • A future migration to Qdrant Cloud is a Protocol-level swap, not a rewrite: the store abstraction covers both backends

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Chroma embedded + Voyage AI primary + bge-small-en-v1.5 fallback

Section titled “Chroma embedded + Voyage AI primary + bge-small-en-v1.5 fallback”
  • Good, because the default path runs with zero external services
  • Good, because Voyage AI’s 200M-token free tier covers the KB
  • Good, because the offline fallback removes the “needs-a-key” reading
  • Good, because the eval harness sees a deterministic retrieval surface
  • Bad, because the fallback model enlarges the dev/eval embeddings-local extra, though the production image drops it
  • Bad, because Chroma embedded does not scale to hundreds of thousands of chunks

Qdrant Cloud free tier + Voyage AI

Section titled “Qdrant Cloud free tier + Voyage AI”
  • Good, because the managed dashboard and free tier (1 GB, no card) are generous
  • Bad, because the demo would depend on an external service and Qdrant’s account policy; every reader would have to sign up
  • Kept as a documented alternative

FAISS embedded

Section titled “FAISS embedded”
  • Good, because FAISS is fast and battle-tested
  • Bad, because metadata + filtering ergonomics are weaker than Chroma’s

Postgres + pgvector

Section titled “Postgres + pgvector”
  • Good, because Postgres is already used for the conversation-state saver
  • Bad, because co-locating conversation state and vector storage complicates ops for a 50-card KB, and shipping Postgres for retrieval contradicts the embedded-by-default posture

OpenAI text-embedding-3-large

Section titled “OpenAI text-embedding-3-large”
  • Good, because it is a strong, well-known embedding model
  • Bad, because it would force the demo to require an OpenAI key for retrieval alone, and there is no clean offline fallback with comparable quality outside sentence-transformers anyway

More Information

Section titled “More Information”

As-built embedder and asymmetric retrieval

Section titled “As-built embedder and asymmetric retrieval”

Primary embedder: Voyage voyage-3.5, with a local offline fallback in the embeddings-local extra. The embedder factory is key-driven: it resolves Voyage voyage-3.5 when a Voyage API key is configured, as in the live deployment. The local BAAI/bge-small-en-v1.5 fallback ships in the optional embeddings-local extra used by dev, eval, and key-less local checkouts; the production image installs embeddings-cloud rather than embeddings-local and drops torch, so it does not carry the local model. Without a Voyage key, the production image degrades to lexical BM25 retrieval rather than the local embedder. The fallback is a lightweight model, roughly 130 MB, CPU-friendly on a small single instance, so a dev or eval run also works at $0 with no external keys while keeping strong retrieval quality.

Retrieval is asymmetric and instruction-aware. The BGE v1.5 family is instruction-tuned and asymmetric. The shipped code honours that: a query is embedded with the documented BGE retrieval instruction prefix (Represent this sentence for searching relevant passages: ); a passage is embedded with no prefix; every vector is L2-normalized so Chroma’s inner-product search behaves as cosine similarity. A symmetric general-purpose model (for example all-MiniLM-L6-v2) receives no instruction prefix. Used without the asymmetric handling, BGE retrieval quality degrades; the retrieval layer is built to apply it.

Retrieval similarity threshold ships disabled. A retrieval-minimum-similarity setting exists but ships disabled by default. On the single-domain KB corpus a threshold cannot separate a near-miss off-corpus clinical question from a genuine in-corpus one without false-refusing the latter. The agent refuses on zero-hit retrieval; a near-miss off-corpus question is answered against the closest card. The threshold is left in place, disabled, so a broader, more topically diverse corpus can enable it later. See the model card for the full limitation.

Part of the portfolio of Waldemar Szemat · szemat.pro
GitHub · LinkedIn