Operations runbook

The operator handoff: how the system is shipped, run, and watched, described at the architecture level. The exact operator commands live in the private operator command reference; this page is the public reference for the operating shape, and each section links to the page that documents it in depth.

Operator responsibilities

The system is built so that no single release step depends on an operator remembering to do the right thing by hand.

Shipping is automated, not manual. A release is a vX.Y.Z tag; a green CI matrix on that exact tag is the gate, and the deploy runs server-side only after the matrix passes. No operator ships a production release by hand. See Deploy.
Watching is continuous. Every node, model call, retrieval, and guardrail emits a trace, and cost and latency budgets are enforced per turn. See Observability.
Responding is a traffic operation. Rollback shifts traffic to the prior healthy revision instantly, with no rebuild; a failed revision never takes traffic.
Operator-applied changes are additive and explicit. The few settings an operator applies - enabling durable human review, or trading scale-to-zero for a warm instance - are additive, never a wholesale environment reset.

Production-readiness: definition of done

“Production-ready” here is an evidenced checklist, not a feeling. A release is ready when:

The full CI matrix is green on the exact tagged commit - this is the deploy gate; a red matrix means nothing ships.
The deploy is keyless (a short-lived federated token, no stored key) and secret-preserving (the source rebuild keeps the service’s existing environment and secret bindings). See Deploy.
The public documentation passes its fail-closed leak check, so no internal path, private identifier, or tuned constant reaches the public site.
The evaluation gate passes: deterministic scorers on every change, the judge-backed corpus thresholds on the nightly run, and the judge-calibration gate armed fail-closed on the two safety dimensions (escalation and self-harm).
The safety floor holds: the deterministic acute-emergency categories short-circuit before any model call and are never gated behind human review.
The frontend accessibility and visual-regression gates pass.
The security rails are enforced on the running revision: edge origin-lock, the bot check, per-key consent, and the rate and concurrency limiters. See Security.

Run and observe

Deploy cadence. Tag, then the CI gate re-runs the full matrix on that tag, then the deploy runs only on green, then traffic shifts once the new revision is healthy.
What to watch. OpenTelemetry spans across every node and model call; the judge-health signal on the nightly eval; the drift signals in the Drift detection plan; and the trail in the Audit logging plan.
Keep-warm. An external scheduler pings a constant-time keep-warm route to soften cold starts without holding an always-on instance.

Cost and SLOs

Steady-state cost is a design target. The service runs scale-to-zero, so demo-scale traffic stays inside the host’s always-free allowance and steady-state spend stays near $0 / month. This is a deliberate portfolio posture; a production SLA and multi-tenant scaling are out of scope for the public demo.
Cost is metered as if paid. Per-turn token and compute cost is priced at real pay-as-you-go rates and server-metered; a hard daily spend breaker and a monthly ceiling stop runaway spend and reset on a fixed cadence.
Per-turn budgets are enforced. Token and latency budgets bound each turn; a turn that would exceed them is stopped rather than served unbounded.
A production UX path exists. An operator who needs to remove cold start can run a warm, always-on instance for a small fixed monthly cost; it is an additive, operator-applied setting, never auto-applied to the demo. The costed blueprint is in the enterprise agent stack.

Human-in-the-loop and durability

Human review of a high-risk-but-not-acute draft is an opt-in capability, off by default. It compiles into the agent graph only when enabled, and enabling it in production requires a durable checkpointer (a Postgres-backed conversation store) and a warm instance, so a paused turn survives a scale-to-zero recycle. Acute emergencies never depend on it: the deterministic acute categories are handled before the model runs and bypass human review entirely.

Incident and rollback posture

Rollback is instant and rebuild-free: shift traffic to the prior healthy revision.
Roll forward a fix by reverting the offending commit and cutting a new tag, which re-runs the gate and, on green, redeploys.
Spend breakers fail safe: they trip on the metered spend and reset on a fixed cadence, so a cost incident is bounded automatically.
The public topology endpoint carries no secrets: it returns only the compiled graph’s node names, edges, and feature booleans, and a standing CI test locks that contract.