PII Redaction

Reference documentation: not a medical device

This documentation describes a public reference implementation evaluated on 100% synthetic data. It is a capability and readiness reference, not a compliance certification or legal advice, and it is not a medical device. It is not clinically validated and handles no production PHI.

Documents the PII (Personally Identifiable Information) redaction capabilities of the ai-agent-eval-harness-healthtech reference implementation. Covers the detection patterns, pipeline integration, and limitations of the current PII redaction module.

Read alongside the HIPAA readiness assessment and the Chile Ley 19.628 mapping.

PII Redaction Architecture

Pipeline Integration

PII redaction is applied at two points in the agent pipeline:

1. At-ingress (input): User input is scanned for PII patterns before being processed by the agent pipeline. Detected PII is replaced with typed placeholder tokens, one per identifier class. The redacted text is what the agent processes.

2. At-egress (output): Agent output is scanned for PII patterns before being returned to the user. This catches cases where the LLM inadvertently generates PII (e.g., echoing a phone number from the conversation context).

The redaction runs in the guardrail layer and is integrated into the guardrail_pre and guardrail_post nodes of the LangGraph pipeline.

Redaction Events in Telemetry Spans

PII redaction events are recorded as telemetry span attributes:

• pii.redacted: boolean indicating whether redaction occurred
• pii.pattern_type: the class of identifier that was matched (for example, an email address or a locale-specific national-ID pattern)

User message text is never included in the span attributes (privacy invariant).

Detection Patterns

Currently Detected PII Types

PII Type	Locale	Detection and validation
Email addresses	Universal	Deterministic regex
Phone numbers	en (US), es-419 (Chile), pt-BR (Brazil)	Locale-specific number formats
US SSN	en (US)	Deterministic regex (format-based)
Chilean RUT	es-419 (Chile)	Deterministic regex with check-digit validation
Brazilian CPF	pt-BR (Brazil)	Deterministic regex with check-digit validation
DNI	es-419 (regional)	National identity document detection
Credit card numbers	Universal	Deterministic regex with check-digit validation
Medical Record Numbers (MRN)	en (US)	Alphanumeric healthcare-identifier patterns
Dates of Birth (DOB)	Universal	Common date formats
Street addresses	Universal	Deterministic address patterns

Pattern Matching Methodology

PII detection uses deterministic regex patterns, not LLM-based detection. This ensures:

• Reproducibility: The same input always produces the same redaction result
• Determinism: PII redaction runs in the guardrail layer, which is tested by the deterministic CI gate (a stub LLM client, no API keys required)
• Low latency: Regex matching is fast and adds negligible overhead to the pipeline
• Auditability: Pattern definitions are version-controlled and reviewable

Validation Checks

Some PII types include structural validation beyond pattern matching:

• Credit card numbers: Validated with a check-digit checksum to reduce false positives on arbitrary digit sequences
• RUT (Chile) and CPF (Brazil): Validated with their respective national-ID check-digit algorithms

National identifiers without a check digit (for example, the US SSN) are detected by format alone, so structural validation does not apply to them. The specific checksum implementations are deterministic and version-controlled.

Limitations

Known Limitations of Regex-Based Detection

1. Context-dependent PII: Regex patterns cannot detect PII that is implied by context rather than formatted in a recognisable pattern. For example, “my name is John and I live at the corner of Main and Oak” contains PII (name, location) that no regex can reliably extract.

2. Novel formats: New phone number formats, ID number formats, or address formats not covered by the existing patterns will pass through undetected.

3. International formats: While the module covers US, Chilean, and Brazilian formats, PII from other jurisdictions may not be detected. The coverage is intentionally aligned with the three supported locales.

4. False positives: The regex patterns may occasionally redact non-PII that happens to match a pattern format. The check-digit validations mitigate this for identifiers that carry a checksum.

Scope of redaction

Redaction targets structured identifiers. Personal names are retained by policy, not redacted: reliable name detection without an NER model produces unacceptable false-positive rates, so name retention is a deliberate, documented policy choice (see the name redaction policy decision) rather than an oversight. Redaction is not anonymization and not differential privacy: it removes direct identifiers deterministically but offers no mathematical privacy guarantee, and the system does not collect or process biometric data.

Current State

The reference implementation provides deterministic PII redaction covering the most common identifier types across its three supported locales (US, Chile, Brazil). The redaction module runs in the guardrail layer, is tested by the deterministic CI gate, and logs redaction events to telemetry spans.

Key strengths:

1. Deterministic and reproducible: Same input, same redaction, every time. No dependency on LLM behaviour for PII detection.
2. Multi-locale coverage: Patterns for US, Chilean, and Brazilian identifiers, aligned with the three supported locales.
3. Structural validation: check-digit validation for credit cards and for the Chile/Brazil national IDs that carry a check digit — reducing false positives on the most sensitive patterns.
4. Pipeline integration: Redaction at both input and output stages, integrated into the guardrail nodes that are tested by the eval harness.
5. Audit trail: Redaction events logged in telemetry spans with pattern type, enabling monitoring of redaction frequency and pattern distribution.

Production Path

Production-grade PII redaction would require:

1. Expanded pattern coverage: Additional PII types (names via NER, IP addresses, vehicle identifiers, biometric data patterns, medical record formats from specific EHR systems); coverage for additional jurisdictions beyond US/Chile/Brazil

2. Promote LLM-based residual detection: An optional LLM-based residual PII pass already exists in the codebase as an annotation-only check, disabled by default; production would promote it to an enforcing second stage after the deterministic redaction, to catch context-dependent PII, novel formats, and identifiers the regex patterns miss

3. Redaction quality metrics: Automated measurement of redaction precision and recall against a labelled test set; tracking of false-positive and false-negative rates; regression testing when patterns are updated

4. Periodic pattern review: Regular review of PII patterns against emerging data formats, new identifier types, and evolving privacy regulations; update cadence tied to regulatory review cycle

5. Domain-specific patterns: If deployed in specific healthcare or financial contexts, custom patterns for institution-specific identifier formats (e.g., specific EHR MRN formats, insurance ID formats, account number formats)

6. Redaction logging for compliance: Beyond the current telemetry span attributes, production would need dedicated redaction audit logs with retention policies, access controls, and compliance reporting

7. Data subject request handling: Mechanisms for data subjects to request information about what PII was detected and redacted; deletion procedures for any stored redaction metadata

See Also

• HIPAA readiness assessment — HIPAA readiness assessment
• Chile Ley 19.628 mapping — Chile data protection mapping
• Audit logging plan — audit logging plan
• Guardrails design — guardrails design