ADR-0013: Voice extension (ElevenLabs TTS + STT)

Status: Accepted
Date: 2026-05-24
Decision-makers: Waldemar Szemat

Context and Problem Statement

The demo is a channel-agnostic medication-adherence agent that demonstrates how to build LLM-based systems for regulated industries with verifiable citation, compliance, and cost transparency. A text-only surface does not exercise the voice modality that regulated workflows (telehealth, patient hotlines, accessibility) require. Adding voice input (STT) and voice output (TTS) demonstrates that the agent’s processing pipeline — retrieval, citation enforcement, refusal logic, audit — is genuinely channel-independent, not coupled to a text-only surface.

The extension must respect the constraints the existing ADRs lock:

Google Cloud Run single scale-to-zero instance, near-$0 baseline at demo scale (ADR-0007); Hugging Face Spaces CPU Basic as the genuinely-$0 secondary target.
OpenAI / Anthropic / Groq LLM providers (ADR-0002).
The /chat JSON response contract is frozen; any new surface must not break existing consumers.
Supabase free tier for operational data (ADR-0010, ADR-0015).

The voice layer must be an additive extension: off by default, unlocked only for entitled keys and only after explicit consent, with zero resource consumption until then, and cleanly separable from the text-only path so the agent works identically without it.

How do we add voice I/O to the demo in a way that (a) proves the processing pipeline is channel-agnostic, (b) keeps the baseline at $0/month, (c) does not break the locked /chat contract, and (d) tracks cost against the existing Supabase accounting schema?

Decision Drivers

Channel-agnostic proof: voice must be a first-class surface, not a bolted-on afterthought, to demonstrate that the processing pipeline is genuinely channel-independent.
$0/month baseline: the text-only demo remains free. Voice consumption is opt-in and budgeted separately.
Contract stability: the /chat JSON schema is frozen (ADR-0009). Voice metadata must ride alongside, not inside, that schema.
Cost observability: per ADR-0010 and ADR-0015, every billable unit (LLM tokens, TTS chars, STT seconds) must be tracked in Supabase for post-demo cost accounting.
Minimal vendor surface: the project already depends on OpenAI, Anthropic, Groq, Hugging Face, and Supabase. Voice should add one vendor (ElevenLabs), not two.
Latency budget: the single scale-to-zero instance already has a cold start on the first request after idle. Voice must not add unbounded latency to the warm-path chat experience.
Privacy: audio of healthcare-related conversations is sensitive. Raw audio must not be persisted; only the transcribed text is logged (after guardrails redaction per ADR-0005).
Entitlement and consent: voice is a paid capability, not a free default. It must unlock only on keys provisioned with the voice entitlement (the extended and reviewer tiers), and every served audio turn must be gated behind explicit, server-recorded consent.

Considered Options

TTS (text-to-speech)

ElevenLabs eleven_multilingual_v2, on-demand (chosen): per-request TTS triggered by the user clicking a play button on a completed assistant turn. Language-appropriate voice mapping: Sarah for es-419, Matilda for en-US, Bella for pt-BR.
Always-on TTS streaming: server-push audio on every assistant token. Rejected: each streamed turn consumes characters whether the user listens or not, blowing through the free-tier quota under light demo load and adding latency to the SSE event stream.
Browser-native SpeechSynthesis API: no vendor dependency, but voice quality is inconsistent across browsers and OSes; the healthcare context demands clarity that native TTS cannot guarantee.

STT (speech-to-text)

ElevenLabs realtime WebSocket STT (chosen): a persistent WebSocket streams microphone audio to ElevenLabs and returns incremental transcripts, from the same vendor, one API key, consistent billing surface.
ElevenLabs batch STT (blob-forward): record the full utterance, POST the blob, receive one transcript. Simpler, but adds end-of-utterance latency and does not exercise the streaming voice path; retained as a fallback transcription mode.
Browser-native Web Speech API: no vendor dependency, but quality is too low for healthcare-context utterances (medical terminology, drug names, accented Spanish/Portuguese). Errors in transcription would propagate into the RAG pipeline and produce unsafe responses.
OpenAI Whisper (API): high quality, but adds a second vendor dependency and a second billing surface. The demo already showcases ElevenLabs for TTS; using a different provider for STT fragments the narrative.

Per-key tier entitlement + server-recorded consent (chosen): voice unlocks only on keys provisioned with the voice entitlement (the extended and reviewer tiers); the server records consent per key and returns a 403 consent_required until it is granted, and a 403 voice_not_entitled for keys without the entitlement.
A universal client-side toggle: expose voice to every visitor via a UI switch persisted in localStorage. Rejected: it turns a paid, consent-sensitive capability into a free client-controlled default, cannot be enforced on the server, and puts synthesized audio one click away for unentitled, non-consenting visitors.

Voice agent (full-duplex)

Full-duplex voice agent simulation via ElevenLabs Conversational AI: real-time bidirectional audio. Deferred: the cost model is per-minute of connected time, the implementation complexity is substantial, and the click-to-play + push-to-talk pattern covers the channel-agnostic proof of concept. Documented as a future opt-in extension.
No voice at all: rejected. A text-only surface does not exercise the voice modality that regulated workflows demand.

Decision Outcome

Chosen option: ElevenLabs voice extension with on-demand TTS (eleven_multilingual_v2, click-to-play), realtime WebSocket STT, unlocked as a per-key tier entitlement behind server-recorded consent, with full-duplex conversational voice deferred to a future milestone.

TTS: ElevenLabs `eleven_multilingual_v2`, on-demand

Each completed assistant turn is rendered into audio only when the user clicks the play button on that turn. The TTS request is a background call to the ElevenLabs API; the resulting audio URL (or base64 blob) is returned to the app for playback. No audio is generated proactively.

Voice mapping by locale:

Locale	Voice ID	Name
en-US	`XrExE9yKIg1WjnnlVkGX`	Matilda
es-419	`yoZ06kpGZMiJkInNR0Gt`	Sarah
pt-BR	`tiBZYpS5hJTFXbHm5CwK`	Bella

The mapping is configurable and is overridable by the operator without a code change.

STT: ElevenLabs realtime WebSocket

The browser captures microphone audio (access prompted by the browser) and streams it over a WebSocket to the backend, which relays it to ElevenLabs’ realtime speech-to-text channel and receives incremental transcripts. The final transcript replaces what would have been the typed input and enters the normal /chat pipeline (RAG retrieval, guardrails, LLM generation). Audio is processed in transit and never retained; only the transcribed text is logged, subject to the same guardrail redaction as typed input (ADR-0005). The server clamps the maximum STT hold duration; any client-side recording cap is a UX affordance only and is never trusted for billing. A batch blob-forward transcription mode is retained as a fallback.

Voice is a paid capability, not a universal default. It unlocks only on demo keys provisioned with the voice entitlement (the extended and reviewer tiers). Every voice request passes a server-side gate before any audio is served or transcribed: the server checks recorded consent first (returning 403 consent_required until the key holder has explicitly consented, a decision recorded server-side and persisted across sessions), then the voice entitlement (returning 403 voice_not_entitled for keys without it), then the cost breaker and a provider health probe, before transcribing or synthesizing. Consent and entitlement are enforced on the server; neither is a client-side flag the browser can set.

Audio metadata transport: SSE sidecar, not schema mutation

The ChatResponse JSON schema is frozen. TTS audio metadata (audio URL, duration, character count) is carried in a dedicated SSE event type (voice_audio) emitted alongside the existing response event. Consumers that do not opt into voice ignore the event type entirely. The /chat JSON response (when SSE is not negotiated) remains unchanged; audio is only available over the SSE channel with voice enabled.

This is consistent with the ADR-0009 pattern: new event types extend the SSE surface without mutating the base JSON contract.

Cost model and tracking

ElevenLabs free tier: 10,000 characters/month TTS. Scribe STT has its own free-tier allocation.
For demo loads that exceed the free tier, the operator supplements with a paid ElevenLabs key. The baseline demo (text-only) is unaffected.
Per ADR-0010 and ADR-0015, TTS character counts and STT seconds are tracked in the demo turn usage table in Supabase alongside LLM token counts. This enables per-session and per-key cost accounting without a separate billing system.

Dependency: ElevenLabs SDK

The ElevenLabs Python SDK (version 2.49.0, already installed in the project) is the sole new runtime dependency. No additional vendor SDKs are introduced.

UI defaults

Voice is off by default. A key without the voice entitlement is never offered the voice affordances at all. An entitled key keeps voice off until its holder grants consent through the disclosure gate; until then, and whenever voice is off, no TTS or STT calls are made, no audio UI is active, and the chat behaves identically to the text-only build. This preserves the text-only baseline for every session that does not explicitly unlock and consent to voice.

Full-duplex voice agent: deferred

A real-time voice agent using ElevenLabs Conversational AI (full-duplex WebSocket) is architecturally compatible with this extension but is deferred to a future milestone. The click-to-play TTS and push-to-talk STT provide sufficient evidence of channel-agnostic processing without the cost and complexity of a persistent audio WebSocket.

Confirmation

A key without the voice entitlement is never offered the voice affordances; an entitled key is offered voice only after server-recorded consent.
A voice request without recorded consent is rejected server-side with 403 consent_required; a request from an unentitled key is rejected with 403 voice_not_entitled.
With voice unlocked and consented, each assistant turn shows a play button; clicking it triggers a TTS call and plays the returned audio.
With voice unlocked and consented, a microphone button streams audio over a WebSocket to the backend for realtime STT; the transcript enters the /chat pipeline.
The ChatResponse JSON schema is unchanged when SSE is not negotiated.
SSE consumers see a new voice_audio event type for TTS; consumers that ignore it are unaffected.
The demo turn usage table in Supabase records TTS characters and STT seconds per turn.
A session that never unlocks and consents to voice produces zero ElevenLabs API calls.
The ElevenLabs SDK is the only new runtime dependency.

Consequences

Positive

Channel-agnostic validation: voice I/O proves the processing pipeline works identically across text and audio surfaces, not just in theory but in a runnable demo.
Baseline preserved: voice is off by default and gated by entitlement and consent; sessions that never unlock voice cost nothing on the ElevenLabs side.
Contract stability: the /chat JSON schema is untouched. Audio rides on a separate SSE event type, following the ADR-0009 extension pattern.
Single new vendor: ElevenLabs handles both TTS and STT. No second voice vendor, no OpenAI dependency for Whisper.
Cost observability: TTS characters and STT seconds are tracked in the existing Supabase demo turn usage table, enabling per-session cost reporting alongside LLM costs.
Privacy by design: audio is streamed and processed in transit and never retained. Only transcribed text is logged, subject to guardrail redaction (ADR-0005).
Future-ready: the on-demand TTS + STT pattern is a stepping stone to full-duplex Conversational AI without an architectural rewrite.

Negative

ElevenLabs free-tier ceiling: 10,000 characters/month TTS is roughly 2,000 words. Sustained demo use (evaluator sessions, conference demonstrations) will exhaust it. The operator must supplement with a paid key for high-traffic periods.
Latency on first TTS call: the on-demand TTS request adds 1-3 seconds of latency per turn. This is acceptable for click-to-play (the user expects a wait) but would not be acceptable for always-on streaming.
New runtime dependency: the ElevenLabs SDK adds to the dependency tree. The SDK is well-maintained and the API surface is narrow (TTS generate, STT transcribe), but it is one more package to track for security updates.
Browser microphone permission: STT requires the user to grant microphone access. Some corporate networks and browsers restrict this; the text input fallback is always available.

Neutral

A new voice module is added for the ElevenLabs client, TTS rendering, and STT transcription.
For an entitled, consented key the app gains a play button per turn and a microphone button for input. When voice is off or the key is not entitled, these UI elements are hidden.
The SSE event stream gains a voice_audio event type. Existing SSE consumers that do not handle this event type are unaffected (forward-compatible by design).

Pros and Cons of the Options

ElevenLabs `eleven_multilingual_v2`, on-demand (chosen)

Good, because click-to-play generates audio only when consumed, keeping TTS character usage proportional to actual listens.
Good, because eleven_multilingual_v2 handles en-US, es-419, and pt-BR with a single model, avoiding per-language model routing.
Good, because the per-language voice mapping (Sarah, Matilda, Bella) provides locale-appropriate tone for the healthcare context.
Bad, because the free tier (10K chars/month) is easy to exhaust under sustained demo load.
Bad, because on-demand generation adds 1-3 s latency per play action.

Always-on TTS streaming

Good, because the user hears audio immediately without an extra click.
Bad, because every assistant turn generates a full audio render whether the user listens or not, multiplying character consumption by the number of turns.
Bad, because streaming audio interleaved with the SSE event stream increases the latency budget and the complexity of the client-side event handling.
Bad, because the cost model is unpredictable under demo load.

Browser-native `SpeechSynthesis`

Good, because there is no vendor dependency and no quota.
Bad, because voice quality varies across browsers and OSes; healthcare-context clarity is not guaranteed.
Bad, because it does not exercise the ElevenLabs integration, leaving the voice modality unproven.

ElevenLabs realtime WebSocket STT (chosen)

Good, because it is purpose-built by the same vendor, one API key, one billing surface.
Good, because streaming transcription keeps latency low and exercises the realtime voice path end to end.
Good, because it handles medical terminology and multilingual input (en-US, es-419, pt-BR) better than browser-native alternatives.
Bad, because it requires an ElevenLabs API key and consumes the STT quota, and a persistent WebSocket is more moving parts than a single batch POST.

Browser-native Web Speech API for STT

Good, because there is no vendor dependency and no quota.
Bad, because transcription quality is too low for healthcare utterances (drug names, conditions, accented multilingual input).
Bad, because errors propagate into the RAG pipeline and can produce unsafe or nonsensical responses.

OpenAI Whisper (API) for STT

Good, because Whisper is a well-known, high-quality STT model.
Bad, because it introduces a second vendor dependency (OpenAI) for a single capability, fragmenting the billing surface and the narrative.
Bad, because it fragments the voice vendor surface: the demo uses ElevenLabs for TTS, so STT should come from the same provider to keep the billing surface and integration surface unified.

Full-duplex voice agent (ElevenLabs Conversational AI)

Good, because it is the most impressive demo of ElevenLabs capabilities.
Good, because real-time bidirectional audio is the production-grade pattern for voice agents.
Bad, because the cost model is per-minute of connected time, which is harder to control under demo load.
Bad, because the implementation complexity (WebSocket management, interruption handling, VAD) is substantial for a reference implementation.
Bad, because the click-to-play + push-to-talk pattern already proves channel-agnostic processing at lower cost and complexity.

No voice at all

Good, because it adds no cost, no dependency, no complexity.
Bad, because a text-only surface does not exercise the voice modality, leaving the channel-agnostic claim unsupported by runnable evidence.

Good, because a paid, consent-sensitive capability is gated on the server, where it cannot be bypassed by a client flag.
Good, because consent is recorded per key and persists across sessions, so the consent context is not re-litigated on every visit.
Bad, because it adds server-side gating state (entitlement lookup, consent record) that a simple client toggle would not need.

A universal client-side toggle

Good, because it is the simplest possible affordance, with no server state.
Bad, because it makes a paid capability a free default and cannot be enforced server-side.
Bad, because it puts synthesized audio one click away for every visitor, including unentitled, non-consenting ones.

More Information

LLM vendor abstraction: ADR-0002
Deployment target and $0/month constraint: ADR-0007
Streaming architecture and SSE extension pattern: ADR-0009
Data layer and cost tracking (Supabase): ADR-0010
Continuous improvement and usage storage: ADR-0015
Guardrails and privacy redaction: ADR-0005
ElevenLabs API documentation: https://elevenlabs.io/docs/api-reference
ElevenLabs eleven_multilingual_v2 model: https://elevenlabs.io/docs/speech-synthesis/models
ElevenLabs Scribe STT: https://elevenlabs.io/docs/capabilities/speech-to-text
ElevenLabs pricing and free-tier limits: https://elevenlabs.io/pricing
MADR 4.0.0: https://adr.github.io/madr/