Skip to content
AI Agent Eval Harness

ADR-0025: Managed reranking provider and eval determinism

  • Status: Accepted
  • Date: 2026-06-08
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The hybrid retrieval pipeline (ADR-0022) ends by re-scoring the fused candidate set with a cross-encoder reranker. Running that cross-encoder locally makes it the single dominant cost of a turn: on the production host (Google Cloud Run, small CPU allocation, scale-to-zero) the per-node timing wired by the observability stack (ADR-0006) measured the local reranker as the overwhelming majority of a warm turn’s latency, and it holds a large model resident in memory. An earlier scoping had kept paid rerankers out on cost grounds; once the local reranker became the headline latency defect, that trade no longer held.

Two questions follow. First, which reranker preserves - or improves - retrieval quality while removing the local inference from the turn? Second, how does the evaluation gate (ADR-0003) stay deterministic once a managed, network-backed reranker sits in the retrieval path, without ever calling a paid API inside the gate and while still catching vendor model drift?

Decision Drivers

Section titled “Decision Drivers”
  • Bring the warm turn back under the latency budget without sacrificing retrieval quality; the rerank must be sub-second and hosted.
  • Multilingual ranking parity: the demo serves en / es-419 / pt-BR, so the reranker must rank non-English query-document pairs well.
  • $0 at demo scale, with no quota surprise.
  • The offline eval gate (ADR-0003) stays deterministic, offline, and free: no live paid-API call inside the gate.
  • Reversible and degradation-safe, honouring the graceful-degradation contract of the hybrid pipeline (ADR-0022).

Considered Options

Section titled “Considered Options”
  • A managed, off-box reranker (Voyage rerank-2.5), selected per turn (chosen).
  • Keep the local cross-encoder and accept the dominant warm-turn latency.
  • Quantize / ONNX-compile the local model to speed CPU inference.
  • A different managed vendor (for example Cohere or Jina) off-box.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: offload reranking to a managed, off-box reranker (Voyage rerank-2.5), resolved per turn by a provider setting with three values - automatic (the default), managed, and local.

  • Provider resolution. The automatic default resolves to the managed reranker when a Voyage API key is present, and to the local cross-encoder otherwise. The deployed demo resolves to the managed path; a key-less checkout reranks locally. Production never loads the local cross-encoder, which is what sheds the resident model weight the offload exists to remove.
  • Why a managed quality model, not the local model off-box. The local cross-encoder is not offered by the managed vendor, which ships its own models; hosting it would mean a third vendor. The chosen managed model is multilingual with a long context window and sub-second hosted latency, so the move trades latency down without trading quality down.
  • Async interface. Reranking is an async interface, so a hosted round-trip does not block the event loop; the managed adapter reuses the bounded-retry, no-retry-on-client-error async HTTP pattern already established for the managed embedder (ADR-0004), and the local reranker is wrapped to present the same async surface. The ranking result is identical across both; only scheduling differs.
  • Degradation. A managed non-success or transport failure is raised as a typed rerank error, caught at the retrieval boundary, and degraded to the fusion-only ranking - the same degraded path a local load failure produces (ADR-0022). A rerank outage never fails the turn.
  • Cost. The managed reranker is billed per token with a large free allowance; at the demo’s per-turn rerank volume this leaves the demo comfortably inside the free allowance ($0 at demo scale), and beyond it the marginal per-turn cost is a fraction of a cent, metered server-side in the cost ledger (ADR-0006).

Eval determinism strategy

Section titled “Eval determinism strategy”

The retrieval path now contains a network-backed reranker. Determinism in the gate is preserved on layers, not by trusting the vendor:

  1. The offline gate is storeless. The canonical pull-request gate runs a stub LLM with no vector store wired, so the retrieval node is not in the graph at all and no reranker - local or managed - is ever called inside the gate. Managed-off in the gate is structural, not a flag that could be forgotten.
  2. The store-backed eval pins the local reranker. When an eval run does wire a store (retrieval-quality and integration runs), it builds the local reranker so the scored ranking is deterministic and offline. No gating run ever issues a live managed call.
  3. Recorded replay and non-gating live parity (specified, key-gated). For production-fidelity scoring, a separate evaluation key records real managed responses into cassettes the store-backed eval replays deterministically, alongside a scheduled, non-gating live-parity run that calls the real service to catch vendor model drift early. Live calls stay out of every gating run. This layer is specified here and provisioned off the developer machine.

Confirmation

Section titled “Confirmation”
  • The offline gate stays green and free of any managed-reranker call.
  • Unit tests exercise both providers through a mocked HTTP transport - including the response-envelope variants and the client-error typed-error path - and the provider resolution.
  • Live verification on the production host confirms the warm turn drops under target and the trace’s rerank component reflects the off-box latency (ADR-0006).

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • The dominant local rerank leaves the turn; the warm turn meets its latency goal with equal-or-better multilingual ranking.
  • Production stops carrying the reranker weight in memory, which unlocks a separate memory re-tune.
  • The async interface unblocks concurrent turns: the retrieval path no longer blocks the event loop during local inference.

Negative

Section titled “Negative”
  • Production gains a network dependency on the managed reranker for the full rerank path, mitigated by the fusion-only degradation contract and full reversibility to the local path.
  • The production-fidelity replay / live-parity layer needs a separate evaluation key, so it runs outside the developer machine.

Neutral

Section titled “Neutral”
  • The provider setting and the managed model name join the configuration surface.
  • The local cross-encoder remains the supported offline and eval reranker; it is no longer loaded in production, not removed.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

A managed, off-box reranker (chosen)

Section titled “A managed, off-box reranker (chosen)”
  • Good, because it removes the dominant local inference from the turn and brings the warm turn under budget.
  • Good, because the chosen model is multilingual and sub-second hosted, so latency drops without a quality trade.
  • Good, because it stays $0 at demo scale on the vendor free allowance.
  • Bad, because it introduces a production network dependency, accepted because the pipeline degrades to fusion-only on any rerank outage.

Keep the local cross-encoder

Section titled “Keep the local cross-encoder”
  • Good, because it has no network dependency and no vendor.
  • Bad, because it is the single dominant latency of a warm turn and holds a large model resident in memory.

Quantize / ONNX-compile the local model

Section titled “Quantize / ONNX-compile the local model”
  • Good, because it keeps reranking on-box with no vendor.
  • Bad, because CPU inference on a large cross-encoder stays far above the sub-second target even quantized, and it adds a build-time compilation path to maintain.

A different managed vendor

Section titled “A different managed vendor”
  • Good, because it is the same architectural move to a hosted reranker.
  • Bad, because it offers no decisive quality or cost advantage here and still introduces a second reranking vendor.

More Information

Section titled “More Information”