Multilingual Safety Parity Assessment

Reference documentation: not a medical device

This documentation describes a public reference implementation evaluated on 100% synthetic data. It is a capability and readiness reference, not a compliance certification or legal advice, and it is not a medical device. It is not clinically validated and handles no production PHI.

Documents the safety parity posture of the ai-agent-eval-harness-healthtech reference implementation across its three supported locales: English (en), Latin-American Spanish (es-419), and Brazilian Portuguese (pt-BR). This assessment evaluates whether users in all three locales receive equivalent safety protections.

Read alongside the data statement, the model card, and the OWASP ATLAS threat model.

Locale Coverage

Dimension	en	es-419	pt-BR
Refusal templates	Yes (10 slugs)	Yes (10 slugs)	Yes (10 slugs)
Scope classifier	Yes (regex patterns)	Yes (regex patterns)	Yes (regex patterns)
Escalation templates	Yes (2 templates)	Yes (2 templates)	Yes (2 templates)
Eval corpus cases	105	105	105
KB cards	38 (English)	uses English KB	uses English KB
Demo disclaimer	Yes	Yes	Yes
Voice TTS voices	Sarah	Matilda	Bella

Safety Parity Analysis

Refusal Template Parity

The refusal templates are fully localized for all three locales (10 slugs each, full override, no English inheritance), covering:

1. Dosing-advice refusal (out-of-scope-dosing)
2. Diagnosis refusal (out-of-scope-diagnosis)
3. Lab/imaging/device interpretation refusal (out-of-scope-interpretation)
4. PII-handling refusal (pii-blocked, out-of-scope-pii)
5. Malformed-input refusal (input-malformed)
6. No-verified-source refusal (no-context)
7. Self-disclosure / role-play meta refusal (out-of-scope-meta)
8. Generic out-of-scope refusal (default)
9. Out-of-domain graceful fallback (out-of-domain)

Each template follows the same structure across locales: a clear refusal, redirection to the appropriate healthcare professional, and the locale-appropriate footer disclaimer. The eval harness verifies refusal correctness as a CI-enforced PR-gate with published pass/fail semantics across all locale slices, so the deterministic guardrails fire identically regardless of locale.

Eval Threshold Parity

Metric	Gate	Applied to Locales
Faithfulness	CI-enforced PR-gate (published pass/fail semantics)	en, es-419, pt-BR (identical)
Hallucination	CI-enforced PR-gate (published pass/fail semantics)	en, es-419, pt-BR (identical)
Refusal correctness	CI-enforced PR-gate (binary per case)	en, es-419, pt-BR (identical)
Escalation correctness	CI-enforced PR-gate (binary per case)	en, es-419, pt-BR (identical)

All three locale slices are held to the same gates in the eval harness. A locale-specific regression fails the CI eval job (a signal, not a hard merge gate). The deterministic CI floor runs all 315 cases (105 en + 105 es-419 + 105 pt-BR) on every code change: the non-English slices are full translations of the English master with identical behavioural labels and gold card-ids, so every locale exercises the same scenarios.

PII Redaction Parity

The PII redaction module covers locale-specific identifier patterns:

Identifier Type	en (US)	es-419 (Chile)	pt-BR (Brazil)
Email	Yes	Yes	Yes
Phone numbers	US formats	Chilean formats	Brazilian formats
National ID	SSN detection	RUT detection (check-digit validated)	CPF detection (check-digit validated)
Credit card	Check-digit validated	Check-digit validated	Check-digit validated
Health identifiers	MRN, DOB	RUT-context, clinical-record identifiers	CPF-context, clinical-record identifiers

PII redaction is applied at both input and output stages, regardless of locale. The redaction patterns for all three locales are tested in the unit test suite.

Known Parity Gaps

The following parity gaps are acknowledged honestly:

1. KB cards are English only: The KB cards are written in English. es-419 and pt-BR users interact with the agent in their language, but the underlying knowledge base content is English. This means the agent may retrieve and cite an English-language card while responding in Spanish or Portuguese. The quality of cross-lingual retrieval depends on the embedder’s multilingual capability.

2. Synthetic data has US-English vocabulary bias: The eval corpus, while held to identical thresholds, was generated with a known US-English clinical vocabulary bias. The producer-critic loop partially corrects this, but residual bias is documented in the data statement rather than claimed solved.

3. Eval corpus is symmetric across locales: the en, es-419, and pt-BR slices each carry the same 105 scenarios with identical behavioural labels and gold card-ids, so no locale is under-sampled relative to English. Cross-lingual retrieval quality still depends on the embedder’s multilingual coverage (see gap 4), but scenario coverage is equal.

4. Embedder language coverage: The primary embedder is Voyage voyage-3.5, which is multilingual. The baked-in local fallback (BAAI/bge-small-en-v1.5, used on the no-key offline path) is English-focused, so on that path cross-lingual retrieval for es-419 and pt-BR relies on the embedder’s ability to match non-English queries to English KB cards; a multilingual embedder is preferred whenever the offline fallback is in use.

5. Voice TTS language coverage: The TTS voices (Sarah for EN, Matilda for ES, Bella for PT-BR) provide language-specific synthesis, but the quality and naturalness may vary across voices.

Current State

The reference implementation maintains safety parity across three locales through the following mechanisms:

• Identical eval thresholds: All three locale slices are scored under the same thresholds on every CI run. A locale-specific regression is a build failure.
• Locale-aware refusal templates: All ten refusal slugs have native templates in en, es-419, and pt-BR, following the same structure and enforced by the same eval dimensions.
• Locale-aware PII redaction: Identifier patterns for US, Chile, and Brazil are detected and redacted in the same pipeline stage.
• Locale-aware escalation: Red-flag escalation templates are available in all three locales (two templates: emergency-medical and mental-health-crisis), and the deterministic escalation floor fires identically across locales.
• es-419 and pt-BR eval cases: Dedicated eval corpus slices test locale-specific behaviour on every code change.

The deterministic eval gate proves guardrail parity: refusal correctness and escalation correctness pass identically across all three locales on every run, enforced as CI PR-gates. This means the safety guardrails fire identically regardless of the user’s locale.

The honest assessment is that safety parity is achieved at the guardrail layer (deterministic, testable, reproducible) and at the eval-coverage layer (a symmetric 105-case corpus per locale), but not fully at the model layer (probabilistic, locale-dependent) or the knowledge layer (the KB cards remain English).

Production Path

Achieving full multilingual safety parity in a production deployment:

1. Localised knowledge base: KB cards written in each locale’s language by qualified medical translators, not machine-translated; clinical review for each locale to ensure medical terminology is accurate and culturally appropriate

2. Native-language safety review: Safety evaluation conducted by native-language reviewers for each locale, not just by running the same eval harness against translated test cases; identification of locale-specific safety concerns (e.g., medication naming conventions, emergency service numbers, cultural health beliefs)

3. Balanced eval corpus sizes: Comparable corpus sizes across locales to ensure equal representation of failure modes; targeted generation of adversarial cases specific to each locale’s language and cultural context

4. Multilingual embedder: Embedder with strong multilingual performance for cross-lingual retrieval; evaluation of retrieval quality per locale

5. Locale-specific clinical escalation paths: Emergency services guidance tailored to each locale’s healthcare system (e.g., 911 for US, 131 for Chile, 192/SAMU for Brazil); locale-specific crisis resources and helplines

6. Cultural adaptation beyond translation: Agent behaviour adapted to cultural norms around health discussions (e.g., directness of medical advice, family involvement in healthcare decisions, attitudes toward medication); not just translated text but culturally appropriate interaction patterns

7. Continuous locale monitoring: Separate performance monitoring per locale; automated alerting on locale-specific regressions; regular review of locale performance data by native-language analysts

See Also

• Data statement — synthetic dataset card with locale methodology
• Model card — CHAI Applied Model Card, Fairness section
• Voice consent and deepfake policy — voice consent policy
• PII redaction — PII redaction per locale
• Guardrails design — guardrails design