Skip to content
AI Agent Eval Harness

ADR-0007: Deployment target

  • Status: Accepted
  • Date: 2026-05-12
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

This is a public reference implementation. A live demo URL is itself a load-bearing signal: a reader is two scrolls away from clicking a link that opens a real multi-turn conversational agent in a browser. That URL must be reachable without a credit card wall, must run the same image the project’s Dockerfile builds locally, and must be deployed in a way that never ships an un-vetted commit. Any divergence between the development image and the deployed image undermines the eval harness story; the whole point is that what runs in CI is what runs in production.

The agent is single-process FastAPI / Uvicorn, with an embedded Chroma store and a baked-in fallback embedder, dispatching LLM calls to an external provider (OpenAI by default; see ADR-0002). No GPU, no model weight to host, no persistent disk beyond the small synthetic KB. Its in-process resilience primitives (rate limiter, response cache, paused HITL checkpointer) are not shared across workers, so the deployment must be able to run a single instance deterministically.

How do we ship a public, always-reachable demo URL of this agent at near-zero cost under demonstration load, from the same Dockerfile the project ships, deployed server-side and gated on a green CI matrix so a red build can never reach production, and rolled back in one operation when something breaks?

Decision Drivers

Section titled “Decision Drivers”
  • Same image everywhere: the deploy must build from the same Dockerfile the project ships. No production-only Dockerfile divergence between a contributor’s laptop, CI, and production.
  • Gated, server-side deploy: production must ship only from a release tag whose full CI matrix went green, decided by the platform, not by an operator running a command by hand.
  • Keyless credentials: no long-lived service-account JSON key stored as a CI secret; the deploy authenticates with a short-lived, federated token.
  • Secret-preserving redeploys: provider and integration credentials live in a managed secret store and survive every redeploy without being re-entered.
  • Near-zero cost at demo scale: the demonstration platform is open-ended; a scale-to-zero, single-instance service keeps real spend inside the host’s always-free allowance at demonstration traffic.
  • Single-instance determinism: the in-process rate limiter, cache, and HITL checkpointer require that one instance serve all traffic; the host must make a single-instance ceiling a first-class setting.
  • One-operation rollback: a bad deploy is reverted by shifting traffic back to the prior healthy revision, with no rebuild.
  • Custom domain: the live demo is reachable at a project-owned domain, not only a platform-generated hostname.

Considered Options

Section titled “Considered Options”
  • Google Cloud Run, single-instance container (chosen): same Dockerfile via a source build, --max-instances 1, region us-central1, keyless Workload Identity Federation deploy gated on a green tag CI matrix, revision-based rollback, custom domain mapping, managed secrets, scale-to-zero.
  • Hugging Face Spaces, Docker SDK, CPU Basic free: the same image on a genuinely zero-cost, always-on host; kept as the documented secondary deploy target. Its edge does not offer the keyless gated pipeline, the custom-domain control, or the managed secret store that the production posture wants, and its deploy path is a force-pushed mirror rather than a gated build.
  • Hugging Face Spaces, Gradio / Streamlit SDK: same host, but the SDK builds the UI; the FastAPI surface the project ships would not match the deployed image.
  • Render Web Service, free tier: same Dockerfile target via $PORT; sleeps after 15 minutes of idle with a 30-60 s cold start; no gated, keyless deploy pipeline out of the box.
  • Fly.io, free tier: the original free tier ended in October 2024, replaced by a $5 / month trial-credit posture.
  • Railway, free tier: free plan withdrawn in August 2023 in favour of $5 / month trial credits; not near-zero cost.
  • Vercel, Hobby plan: serverless function timeout of 10 s on Hobby kills any multi-turn agent stream once the LLM round-trip exceeds the limit.
  • Cloudflare Workers AI: a model-routing platform rather than a generic Docker host; the Python + embedded Chroma stack would need rewriting against the Workers runtime.
  • Streamlit Community Cloud: free, but binds the demo to a Streamlit UI; the FastAPI surface becomes inaccessible.
  • Modal: trial credits, then pay-per-second; steady state is not near-zero cost.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Google Cloud Run, single-instance container, with Hugging Face Spaces, Docker SDK documented as the operator’s secondary target in the deploy reference and reachable from the same image with one port change ($PORT).

The service ai-agent-eval-harness runs in project ai-agent-eval-harness, region us-central1, capped at --max-instances 1, and is reached at the custom domain agent.szemat.pro. Cloud Build builds the image from the repository’s Dockerfile via a source-based deploy, so the same image runs in three places: a contributor’s laptop, the eval harness, and the public demo. One image, one mental model, one set of behaviours under test.

The deploy is server-side, keyless, and gated. A release tag matching vX.Y.Z triggers the tag-guard workflow, which re-runs the full CI matrix against the tagged commit. Only when that run concludes success does the deploy workflow fire (via workflow_run), authenticate to Google Cloud with Workload Identity Federation (a short-lived OIDC token, no stored JSON key), and run a secret-preserving Cloud Run source deploy. A red CI matrix means the deploy job is skipped and nothing ships. The source rebuild preserves the service’s existing environment variables and Secret Manager bindings; a failed revision keeps traffic on the prior healthy one.

The single-instance ceiling is load-bearing: the in-process rate limiter, response cache, and HITL checkpointer are not shared across instances, so the service runs exactly one instance by design. Cloud Run scale-to-zero keeps real spend inside the host’s always-free allowance at demonstration traffic; the container hosts no local LLM, so it only handles RAG retrieval, orchestration, and HTTP, dispatching completions to an external provider.

Confirmation

Section titled “Confirmation”
  • A green tag-guard CI matrix on a vX.Y.Z tag, followed by a green deploy workflow run that authenticated keyless via WIF.
  • A reachable public URL at https://agent.szemat.pro.
  • The service answers GET /health with 200 OK.
  • The service answers POST /chat in demo mode against an offline stub client (no caller-side keys required for the public demo).
  • The deploy reference documents the operator bootstrap (WIF pool / provider + least-privilege deployer service account, Secret Manager bindings, first deploy).

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • Gated, server-side deploys: only a published-green release tag can reach production; an operator cannot hand-ship an un-vetted commit.
  • Keyless credentials: no long-lived service-account key exists or is stored; the deploy uses a short-lived federated token.
  • Same Dockerfile everywhere: development, CI, production.
  • Secret-preserving redeploys: the managed secret bindings survive every source deploy.
  • One-operation rollback: shift traffic back to the prior healthy revision; no rebuild.
  • Custom domain: the live demo is served at agent.szemat.pro.
  • No model-weight hosting: avoids LFS and any storage tier on the host; keeps the image small.

Negative

Section titled “Negative”
  • Cold start with scale-to-zero: the first request after an idle window pays a container cold start (dependency import, first-boot KB ingest, embedder load).
  • Single-instance ceiling: --max-instances 1 caps horizontal concurrency; scaling out would require moving the in-process primitives to a shared store (Redis, Postgres), out of scope for the demo.
  • One-time cloud wiring: the keyless deploy needs a WIF pool / provider and a least-privilege deployer service account configured once before the pipeline can authenticate.
  • CPU-only: the container cannot host a local LLM; the demo depends on an external provider for completions. By design (see ADR-0002).

Neutral

Section titled “Neutral”
  • A tag-guard gate and a deploy workflow become part of the repository’s CI layout.
  • Repository variables (WIF_PROVIDER, WIF_SERVICE_ACCOUNT) become required for the live deploy; contributors without deploy rights work locally without them.
  • The same image is kept deployable to Hugging Face Spaces as a secondary target, so the deploy reference documents two hosts.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Google Cloud Run, single-instance container

Section titled “Google Cloud Run, single-instance container”
  • Good, because the deploy is gated on a green CI matrix and runs server-side, keyless via WIF.
  • Good, because a source build runs the project’s Dockerfile as-is and preserves managed secrets across redeploys.
  • Good, because rollback is a one-operation traffic shift to the prior revision.
  • Good, because --max-instances 1 makes the required single-instance posture a first-class setting, and scale-to-zero keeps demo cost near zero.
  • Good, because the service maps to a project-owned custom domain.
  • Bad, because scale-to-zero means a cold start on the first request after idle.
  • Bad, because horizontal scale-out needs a shared store for the in-process primitives.

Hugging Face Spaces, Docker SDK, CPU Basic free

Section titled “Hugging Face Spaces, Docker SDK, CPU Basic free”
  • Good, because the same image runs unchanged at genuinely zero cost, always on, which makes it an ideal secondary demo target.
  • Good, because the huggingface.co domain signals “AI reference implementation” at a glance.
  • Bad, because its edge offers no keyless gated deploy pipeline, no managed secret store, and no custom-domain control; the deploy path is a force-pushed mirror rather than a gated build.
  • Bad, because the Space-card YAML front-matter requires a deploy-only swap of the root README.

Hugging Face Spaces, Gradio / Streamlit SDK

Section titled “Hugging Face Spaces, Gradio / Streamlit SDK”
  • Good, because the SDKs ship an opinionated, hosted UI.
  • Bad, because the demo surface diverges from the FastAPI surface the project ships; the deployed thing stops being the same thing as the locally-built thing.

Render Web Service, free tier

Section titled “Render Web Service, free tier”
  • Good, because the same Dockerfile deploys with one environment-variable change ($PORT).
  • Bad, because the 15-minute idle sleep is more aggressive than scale-to-zero on a warmed service, and there is no gated, keyless deploy pipeline out of the box.

Fly.io / Railway free tiers

Section titled “Fly.io / Railway free tiers”
  • Bad, because the historical free tiers were withdrawn (Fly.io October 2024, Railway August 2023) in favour of trial credits; steady state is not near-zero cost.

Vercel Hobby

Section titled “Vercel Hobby”
  • Bad, because the 10-second function timeout kills any multi-turn agent stream once the LLM round-trip exceeds it.

Cloudflare Workers AI

Section titled “Cloudflare Workers AI”
  • Bad, because the Python + embedded Chroma stack would have to be rewritten against the Workers runtime.

Streamlit Community Cloud

Section titled “Streamlit Community Cloud”
  • Bad, because the UI binds the demo to Streamlit; the FastAPI surface becomes inaccessible.
Section titled “Modal”
  • Bad, because trial credits expire; steady state costs money.

More Information

Section titled “More Information”

Deployment-resilience layer

Section titled “Deployment-resilience layer”

Three per-process primitives ship to make the demo degrade gracefully under load rather than surface raw upstream errors. All three are driven by enable flags so they are off by default for deterministic tests and on for the live service:

  • Per-session rate limiter. A per-key (per client IP) sliding-window limiter. An over-limit caller receives an HTTP 429 with a Retry-After header instead of a raw 502. Tunable via maximum requests and window-seconds settings. A dependency was deliberately not added; the in-process limiter keeps the framework-free posture.
  • Provider fallback chain. A wrapper around the LLMClient Protocol: a transient OpenAI failure (HTTP 429, 5xx, or a bare transport failure) cascades to Anthropic before any error reaches the frontend. A non-429 4xx is a genuine client error and is re-raised unchanged. Consistent with the ADR-0002 vendor abstraction - the fallback is a Protocol-level wrapper, not a node-level change.
  • Short-TTL response cache. A bounded, short-TTL in-process cache keyed on the normalized (input, locale, model) tuple, so the SPA “load example” clicks are served without hitting the provider.

Single-instance consequence. All three primitives are per-process, as is the in-memory HITL checkpointer (see ADR-0001). The service therefore runs a single instance by design (--max-instances 1); a second instance would not share the limiter, the cache, or the paused-thread state. A multi-instance deployment would need a shared store (Redis, Postgres), which is out of scope for the demo. This is documented in the deploy reference and the Dockerfile comment.

Baked-in embedder. The baked-in local fallback embedder is BAAI/bge-small-en-v1.5, CPU-friendly on the single-instance service; the primary embedder is Voyage voyage-3.5 when a Voyage key is configured. See ADR-0004 for the embedder decision; the deployment posture here is unaffected by the model choice.