ISO 42001 and SOC 2 Readiness Assessment

Maps the ai-agent-eval-harness-healthtech reference implementation against ISO/IEC 42001:2023 (AI Management System) and the SOC 2 Trust Service Criteria. This is an honest readiness assessment: the system is a reference implementation and has not been audited against either standard. The assessment evaluates which controls exist, which are partially addressed, and what a formal certification process would require.

Read alongside the NIST AI RMF mapping and the audit logging plan.

ISO/IEC 42001:2023 — AI Management System

ISO 42001 is the first international standard for AI management systems. It provides a framework for establishing, implementing, maintaining, and continually improving an AI management system within an organisation. The standard follows the harmonised structure of ISO management system standards (Plan-Do-Check-Act).

Clause-by-Clause Mapping

Clause 4: Context of the Organisation

Requirement	Current Implementation	Gap
4.1 Understanding the organisation and its context	The repository scope (reference implementation), invariants, and intended stakeholders (engineers, governance reviewers) are described in the repository overview and the security disclosure process	Context covers the repository; a formal ISMS would need organisational context including business strategy, market, and regulatory landscape
4.2 Understanding the needs and expectations of interested parties	Intended users defined in the model card: engineers, governance reviewers; regulatory posture defined in the regulatory posture	No formal stakeholder analysis; single-author project has limited stakeholder diversity
4.3 Determining the scope of the AI management system	Scope defined in the system specification and the regulatory posture: medication-adherence wellness coach, patient-facing, no clinical functions	Scope is repository-level; an ISMS scope would cover the entire AI lifecycle within the organisation
4.4 AI management system	Eval harness as measurement system; architecture decision records as decision records; guardrails as controls; OpenTelemetry as monitoring; the regulatory posture as policy	Demonstrates the pattern; not a formal management system with defined procedures, responsibilities, and review cycles

Clause 5: Leadership

Requirement	Current Implementation	Gap
5.1 Leadership and commitment	Author commitment to governance demonstrated by an extensive set of governance documents and architecture decision records, the eval harness, and honest limitation statements	Single author; no top-management commitment structure; no resource allocation process
5.2 Policy	The regulatory posture serves as the regulatory policy; the operational decision record captures operational policies	No formal AI policy document; policies are distributed across multiple artefacts
5.3 Organisational roles, responsibilities, and authorities	Module ownership documented (guardrails, evals, observability); decision-record ownership per decision	No formal role definitions; no authority delegation; no separation of duties

Clause 6: Planning

Requirement	Current Implementation	Gap
6.1 Actions to address risks and opportunities	The OWASP / ATLAS threat model maps threats; the NIST AI RMF mapping maps risk functions; the eval harness detects regressions; the nightly red-team exercises adversarial risks	No formal risk register; no risk treatment plan; no opportunities register
6.2 AI objectives and planning to achieve them	The system specification defines objectives; eval thresholds define measurable targets; release history is published	Objectives are repository-level; no organisational AI objectives or resource plans
6.3 AI risk assessment	Threat model (OWASP/ATLAS), NIST AI RMF mapping, honest limitation statements in the model card	Assessment is qualitative and documented; no quantitative risk scoring; no residual risk acceptance

Clause 7: Support

Requirement	Current Implementation	Gap
7.1 Resources	$0/month cost target; free-tier infrastructure; existing tooling (pytest, OpenTelemetry, Promptfoo)	No resource planning process; no budget allocation; no capacity management
7.2 Competence	Author demonstrates domain competence through architectural decisions, regulatory mapping, and governance documentation	No competence framework; no training records; no competence assessment process
7.3 Awareness	The regulatory posture change control requires a decision record for scope changes; the security disclosure process	No formal awareness programme; no awareness metrics
7.4 Communication	Public governance docs; the model card; the regulatory posture; architecture decision records	Communication is passive (published documents); no active stakeholder communication plan
7.5 Documented information	Architecture decision records, governance docs, the system specification, the regulatory posture, the data statement, the model card	Documentation is comprehensive; version-controlled; no formal document management system

Clause 8: Operation

Requirement	Current Implementation	Gap
8.1 Operational planning and control	Eval harness gates every code change; guardrails enforce scope; OpenTelemetry monitors execution; cost/latency budgets enforced	Operational controls exist; no formal operational plan; no change management process
8.2 AI risk assessment (operational)	Scope classifier runs on every input; escalation detection runs on every input; PII redaction runs on every input/output	Real-time risk assessment for safety-critical functions; no broader operational risk assessment
8.3 AI risk treatment	Guardrails-before-LLM architecture; refusal on out-of-scope; escalation on acute red flags; citation enforcement	Risk treatments are implemented and tested; no formal treatment plan with owner, timeline, and acceptance criteria

Clause 9: Performance Evaluation

Requirement	Current Implementation	Gap
9.1 Monitoring, measurement, analysis and evaluation	Eval harness (7 dimensions, 315 cases); OpenTelemetry spans; cost/latency gates; locale parity enforcement	Monitoring is comprehensive for the reference implementation; no management-review-level performance reporting
9.2 Internal audit	No formal internal audit	No audit programme; no audit criteria; no audit findings tracking
9.3 Management review	No formal management review	No review cycle; no review inputs/outputs; no action items from review

Clause 10: Improvement

Requirement	Current Implementation	Gap
10.1 Continual improvement	Red-team findings folded into the seed bank; eval regression drives code fixes; release history is published	Improvement is reactive and within-repository; no formal continual improvement programme
10.2 Nonconformity and corrective action	Eval regression fails the CI eval check (a signal, not a hard merge gate - no enforced branch protection); the security disclosure process	No formal nonconformity management; no root cause analysis process; no corrective action tracking

SOC 2 Trust Service Criteria Mapping

SOC 2 (System and Organization Controls 2) evaluates service organisations against five Trust Service Criteria: Security, Availability, Processing Integrity, Confidentiality, and Privacy.

Common Criteria (Security) — Required for All Engagements

Control	Current Implementation	Gap
CC6.1 Logical access controls	No runtime authentication for demo API; repository-level access; no user identity management	Production would need authentication, authorisation, and access review processes
CC6.2 Physical access controls	Inherited from the cloud provider (Google Cloud Run; no customer-managed physical infrastructure)	Production with on-premises components would need physical access controls
CC6.3 Data encryption	HTTPS on Google Cloud Run (platform-provided); no encryption at rest (no persistent storage)	Production would need TLS 1.3 minimum; AES-256 at rest for any stored data
CC7.1 Change management	Architecture decision records document changes; the eval harness gates changes; semantic versioning; release notes	No formal change management process with approval workflows and rollback procedures
CC7.2 Risk mitigation	Guardrails, eval harness, red-team, PII redaction, OpenTelemetry monitoring	Controls exist; no formal risk mitigation programme with tracking and reporting
CC7.3 Incident response	The security disclosure process; secret scanning	No formal incident response plan; no severity classification; no notification procedures
CC8.1 Change management (detailed)	Change-based workflow; eval regression testing; version control	No formal change approval board; no pre-deployment validation checklist

Availability

Control	Current Implementation	Gap
A1.1 System availability	Google Cloud Run single scale-to-zero instance (scale-to-zero, cold start on the first request after idle; near-$0 at demo scale); no SLA	Production would need availability targets, monitoring, alerting, and incident escalation
A1.2 Backup and recovery	No backup needed (no persistent data); eval corpus and KB in version control	Production with persistent data would need backup procedures, recovery testing, RTO/RPO targets

Processing Integrity

Control	Current Implementation	Gap
PI1.1 Processing integrity	Eval harness validates output quality (faithfulness, hallucination, citation); guardrails enforce scope; deterministic CI gate	Processing is measured against defined thresholds; no end-user-visible processing integrity controls
PI1.2 Input processing	Scope classifier validates input; PII redaction sanitises input; input length limits	Input validation exists for safety; no broader input quality controls

Confidentiality

Control	Current Implementation	Gap
C1.1 Confidentiality protections	No PHI/PII in storage; user text excluded from OpenTelemetry spans; PII redaction at input/output; secret scanning prevents secret exposure	Confidentiality is maintained by not storing sensitive data; production would need active confidentiality controls for stored data

Privacy

Control	Current Implementation	Gap
P1.1 Privacy notice	Demo disclaimer on every response; the model card; the regulatory posture	No formal privacy notice; no data collection disclosure; no consent management
P2.1 Consent management	Voice consent (disclosure modal, opt-in toggle, “Audio NOT retained” notices)	Consent exists for voice only; no consent management for general data processing
P3.1 Data collection	No personal data collected; 100% synthetic	Production would need data collection policies, purpose limitation, and minimisation procedures
P4.1 Data access and deletion	No personal data to access or delete	Production would need data subject access and deletion request procedures
P6.1 Data retention	No persistent data; no retention needed	Production would need retention schedules, secure deletion procedures, and retention monitoring

Current State

The reference implementation demonstrates a governance posture that aligns with both ISO 42001 and SOC 2 principles, implemented at the depth a single-author reference project can reasonably achieve:

ISO 42001 alignment: Decision-record-based traceability (Clause 7.5), eval harness as measurement system (Clause 9.1), guardrails as risk treatment (Clause 8.3), honest risk assessment (Clause 6.3), transparent documentation (Clauses 4, 7.4)
SOC 2 alignment: Eval harness as processing integrity control (PI1.1), guardrails and PII redaction as confidentiality controls (C1.1), privacy-by-design (P1-P6), OpenTelemetry monitoring as security control (CC7.2), decision-record-based change management (CC7.1)

The honest assessment is that this is a governance-aware reference implementation, not a certified management system. No ISO 42001 audit has been conducted. No SOC 2 engagement has been performed. The value is in demonstrating that the governance patterns are understood and implemented at a level that would accelerate formal certification should it be pursued.

Production Path

Achieving ISO 42001 certification or SOC 2 Type II attestation:

Formal ISMS (ISO 42001): Documented AI management system with defined scope, leadership commitment, risk register, objectives, operational plans, performance evaluation, and continual improvement programme; internal audit programme; management review cycle
SOC 2 Type I preparation: Documented controls for all applicable Trust Service Criteria; evidence collection for control existence; readiness assessment by a qualified CPA firm
SOC 2 Type II attestation: 6-12 months of operating evidence for all controls; independent audit by a CPA firm; attestation report covering the observation period
Evidence collection automation: Automated collection and organisation of audit evidence (eval run logs, change records, access logs, incident reports, training records); evidence repository with retention policies
Continuous monitoring: Real-time monitoring of control effectiveness; automated alerting on control failures; periodic control testing; dashboard for control status visibility
Vendor management: Third-party risk assessments for all service providers (LLM providers, observability platforms, hosting providers); BAA where applicable; contractual security requirements

The repository’s governance patterns — decision-record traceability, eval harness gating, guardrails-first architecture, OpenTelemetry instrumentation, honest documentation — provide the technical foundation. The gap is in the organisational, procedural, and audit-readiness layers.