Skip to content
AI Agent Eval Harness

ADR-0016: Free-Tier Deployment Resilience Layer

  • Status: Accepted
  • Date: 2026-05-27 (recorded retroactively; the resilience layer shipped in an earlier release)
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The demo runs on Google Cloud Run as a single scale-to-zero instance (ADR-0007); the same image also runs on the Hugging Face Spaces CPU Basic secondary target. The public demo URL is shared openly. A short burst of curious visitors — a post that gets picked up, a conference mention — can spike traffic faster than the agent can answer.

Without protection, the instance queues turns until either the event loop falls behind and timeouts cascade, or the platform edge proxy returns 502 Bad Gateway. Both outcomes turn a public moment into a broken demo.

How do we protect the instance from concurrent-request bursts while keeping the near-$0 operational cost at demo scale and the single-instance simplicity, without introducing Redis?

Decision Drivers

Section titled “Decision Drivers”
  • Near-$0 operational budget at demo scale (ADR-0007): no Redis, no managed rate-limit service.
  • Single-instance simplicity: the deployment design is one instance (--max-instances 1); sharing rate-limit state across instances is not a current concern.
  • Transparent retry semantics: the client (the app) must know when to retry. Retry-After is the load-bearing header.
  • Demo survives identical-input bursts: when ten viewers click the same example scenario, the instance should compute the answer once.

Considered Options

Section titled “Considered Options”
  • Option A: No protection. The instance handles every request until it falls over.
  • Option B: Redis-backed sliding-window rate limiter + response cache.
  • Option C: In-process sliding-window rate limiter + in-process TTL response cache, both keyed on a normalised request signature. Bounded memory.
  • Option D: Use a CDN-level rate limit (Cloudflare in front of the hosting platform).

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Option C — an in-process resilience layer with a sliding-window rate limiter and a TTL response cache, both bounded and memory-resident. The single most load-bearing reason is the near-$0 budget: any external dependency for rate limiting violates the deployment posture. The in-process design also matches the single-instance truth — there is no second instance to coordinate with.

Behaviour:

  • The rate limiter is keyed on the client IP (proxy-aware, so the X-Forwarded-For header is honoured behind the platform reverse proxy). A sliding window with a bounded request count per window rejects bursts with HTTP 429 and a Retry-After hint.
  • The response cache is keyed on a normalised (text, locale, model) tuple. A short TTL keeps memory bounded. Identical inputs within the TTL window return the cached answer, avoiding redundant LLM spend.
  • The cache is NEVER used for a paused (HITL) turn — a paused draft is not a final answer and must not be served to a different session as if it were.

Confirmation

Section titled “Confirmation”
  • The rate limiter and cache are implemented in a dedicated resilience module.
  • Unit tests cover: the rate-limit decision under burst, a cache hit on identical input, a cache miss on a different locale or model, no cache on paused turns, and Retry-After propagation.
  • The limiter is applied before agent work, and the cache is applied around the agent invocation.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • The demo survives traffic bursts without external infra.
  • Identical-input traffic (the “everyone clicked the same example” pattern) costs zero LLM spend on repeats within the cache TTL.
  • The HTTP 429 + Retry-After semantics are transparent and client-friendly.
  • The near-$0 budget at demo scale is preserved.

Negative

Section titled “Negative”
  • The protection is per-instance. A horizontal-scale-out deployment (multiple instances behind a load balancer) would need to externalise the limiter and cache to Redis. The design is honest about this limitation, recorded as a production-readiness gap in the performance report.
  • Instance restart loses cache state. Cold-start traffic pays full LLM cost until the cache warms.
  • The IP-based limiter is coarse: a corporate NAT presents the whole organisation as one IP. The bounded-window count must be tuned not to penalise legitimate NATted traffic; current tuning is intentionally lenient.

Neutral

Section titled “Neutral”
  • The rate-limit decision adds one in-process check per request. Latency cost is sub-millisecond.
  • The cache adds in-memory state. A bounded TTL keeps the memory footprint manageable within the instance memory.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Option A: No protection

Section titled “Option A: No protection”
  • Good, because zero infrastructure.
  • Bad, because the instance falls over under burst traffic.
  • Bad, because there is no 429 + Retry-After semantics — the client sees a 502 from the platform proxy.

Option B: Redis-backed limiter + cache

Section titled “Option B: Redis-backed limiter + cache”
  • Good, because it survives multi-instance scale-out.
  • Bad, because it adds an external dependency (Redis instance, secret management, network hop).
  • Bad, because it conflicts with the near-$0 budget.
  • Bad, because it is over-engineered for the single-instance reality.

Option C (chosen): In-process limiter + cache

Section titled “Option C (chosen): In-process limiter + cache”
  • Good, because no external dependency.
  • Good, because it matches the single-instance truth.
  • Good, because it is explicit, testable, and self-contained.
  • Bad, because it does not scale beyond one instance (acknowledged).

Option D: CDN-level rate limit

Section titled “Option D: CDN-level rate limit”
  • Good, because it offloads protection to Cloudflare’s edge.
  • Bad, because it does not cover the response-cache need.
  • Bad, because on the free secondary target (Hugging Face Spaces) traffic routes directly with no DNS / domain control to put Cloudflare in front.

More Information

Section titled “More Information”