Skip to content
AI Agent Eval Harness

ADR-0015: Continuous Improvement Layer storage (Supabase)

  • Status: Accepted
  • Date: 2026-05-24
  • Decision-makers: Waldemar Szemat

Context and Problem Statement

Section titled “Context and Problem Statement”

The Continuous Improvement Layer requires persistent storage for two concerns:

  1. Interaction logs: every demo turn, redacted at ingress via PII redaction, with dedup hashes, compliance flags, latency, cost, and citation status. Pending interactions are clustered by semantic similarity and analyzed by a batch script to produce improvement suggestions. Retention: 90 days raw, aggregated metrics 1 year.

  2. Improvement suggestions: operator-curated proposals (new KB cards, card refinements, eval cases, prompt adjustments, guardrail refinements, corpus gaps) with a human-review workflow (pending -> approved -> integrated). Retention: indefinite for audit trail.

ADR-0010 establishes the Supabase free tier as the managed Postgres backend for demo operational data (keys, sessions, consents). The interaction and suggestion tables are part of the same operational data domain and naturally co-locate with the demo key and session tables already provisioned there.

The continuous improvement layer has specific requirements:

  • PII-redacted at ingress (never raw text stored)
  • Operator-curated (never auto-applied)
  • Batch processing (operator-triggered, not cron)
  • Audit trail (who approved what, when, which commit)
  • Pseudonymized dedup (sha256 hash of redacted input)

Where should the interaction logs and improvement suggestions live?

Decision Drivers

Section titled “Decision Drivers”
  • Colocation with demo operational data: the interaction logger is designed to write after each turn, and the demo key and session tables already live in Supabase (ADR-0010). Cross-table queries (e.g., “show all interactions for this demo key”) are natural in a single database.
  • PII redaction at ingress: the storage backend must never receive raw PII. Redaction runs before any persistence, so only already-redacted data would ever reach the storage layer.
  • Operator-curated workflow: the improvement-suggestions table enforces a status check constraint (pending review -> approved -> rejected -> integrated). Only the operator transitions status. The batch script proposes, never applies.
  • $0/month hosting: consistent with ADR-0007 and ADR-0010.
  • Queryable dashboard: the operator reviews pending suggestions in the Supabase dashboard, the same interface used for demo key management.
  • Regulatory anchors: GDPR Art. 25 (Privacy by Design), HIPAA Safe Harbor (18 identifiers), Chile Ley 19.628 + Reforma 21.719 (PII-redacted with an improvement purpose). The storage choice must support these requirements.

Considered Options

Section titled “Considered Options”
  • Supabase (same project as ADR-0010) (chosen): the interactions and improvement-suggestions tables co-located with demo keys, demo sessions, etc. in the same free-tier Postgres.
  • SQLite local on Hugging Face persistent storage: zero-vendor, but persistent storage is not guaranteed for Docker SDK Spaces, no dashboard, no cross-table queries with demo keys.
  • Neon (separate project): would fragment the operational data across two managed Postgres instances for no benefit.
  • Firestore (NoSQL): poor fit for relational schema (status check constraints, FK to demo keys, JSONB for compliance flags).
  • CSV/JSONL files on Hugging Face persistent storage: append-only, no query capability, no dashboard, no status transitions for suggestions.

Decision Outcome

Section titled “Decision Outcome”

Chosen option: Supabase free tier, same project established in ADR-0010, with the interactions and improvement-suggestions tables deployed alongside the demo operational tables.

The interaction logger hooks into the agent execution graph after the audit-emit node. Its designed per-turn ingestion contract is to:

  1. Receive the full turn context (input, response, citations, compliance flags, latency, cost).
  2. Apply PII redaction on both the user input AND the response text (defense in depth).
  3. Compute a sha256 hash of the redacted input for dedup.
  4. Persist the redacted record to the interactions table.

Redaction and hashing are wired today; the persistence write in step 4 is the designed sink for this contract and is not yet active in the deployed demo. When the write is enabled and Supabase is unreachable, the logger is designed to degrade to a local log and a UI warning and never block the agent flow.

The batch improvement script runs on the operator’s local machine (not on the Space). It reads pending interactions, clusters by semantic similarity, generates suggestions via LLM analysis, and persists them in the improvement-suggestions table with status “pending review”. The operator reviews in the Supabase dashboard and approves/rejects manually.

The improvement-suggestions status check constraint enforces: pending review | approved | rejected | integrated. Only the operator can transition status. The batch script only inserts at “pending review”. No automated status change exists.

Confirmation

Section titled “Confirmation”
  • The interactions table has PII-redacted columns, a dedup hash, and a compliance-flags JSONB column.
  • The improvement-suggestions table has a status check constraint and operator review fields.
  • A foreign key links each interaction to its demo key (cross-table query: “all interactions for this key”).
  • A foreign key links demo turn usage to its interaction (cost tracking linked to the interaction log).
  • The batch script runs locally and connects to Supabase via a service-role key from an environment variable.
  • If improvement logging is disabled for a key, the logger skips that key entirely.

Consequences

Section titled “Consequences”

Positive

Section titled “Positive”
  • All operational data (keys, sessions, interactions, suggestions) co-located in one database. Cross-table queries are natural.
  • The operator uses a single dashboard for all review workflows (key management + improvement suggestions).
  • PII redaction at ingress means the database is designed never to receive raw PII: redaction runs before the write, so the redacted input column cannot carry an unredacted marker by construction.
  • Free tier (500 MB) is sufficient for low-volume demo (50-150 reviewers x 5-10 turns = ~1000 rows, well under the cap).
  • pgvector is available for semantic clustering of interactions in the batch script (reusing the same embedding model as RAG).

Negative

Section titled “Negative”
  • Adds two more tables to the Supabase migration surface.
  • The batch script requires a service-role key with write access to both the interactions and the improvement-suggestions tables.
  • Once the persistence write is enabled, a Supabase outage degrades interaction logging gracefully (local log + warning) with the data for those turns lost rather than blocking the turn.

Neutral

Section titled “Neutral”
  • The interactions table has a 90-day retention policy enforced by the operator (manual or via a scheduled script). The Supabase free tier does not auto-enforce retention.
  • Improvement suggestions are retained indefinitely as an audit trail.
  • The batch script is operator-triggered, never automated. This is by design for regulated AI: online learning amplifies bias without governance.

Pros and Cons of the Options

Section titled “Pros and Cons of the Options”

Supabase (same project as ADR-0010)

Section titled “Supabase (same project as ADR-0010)”
  • Good, because colocation enables cross-table queries (keys + sessions + interactions + suggestions)
  • Good, because a single dashboard serves all operator workflows
  • Good, because PII redaction at ingress is verifiable in one place
  • Good, because pgvector is available for semantic clustering
  • Good, because $0/month, consistent with ADR-0007 and ADR-0010
  • Bad, because it adds migration surface for two more tables
  • Bad, because the batch script needs a service-role key

SQLite local on Hugging Face persistent storage

Section titled “SQLite local on Hugging Face persistent storage”
  • Good, because zero-vendor
  • Bad, because persistent storage is not guaranteed for Docker SDK
  • Bad, because no dashboard for operator review
  • Bad, because no cross-table queries with demo keys
  • Bad, because concurrent write risks

Neon (separate project)

Section titled “Neon (separate project)”
  • Good, because managed Postgres
  • Bad, because it fragments operational data across two instances
  • Bad, because no benefit over co-location

Firestore (NoSQL)

Section titled “Firestore (NoSQL)”
  • Good, because Google-managed
  • Bad, because poor fit for relational schema and status transitions

CSV/JSONL on Hugging Face persistent storage

Section titled “CSV/JSONL on Hugging Face persistent storage”
  • Good, because simplest possible append-only storage
  • Bad, because no query capability
  • Bad, because no status transitions for suggestions
  • Bad, because no dashboard

More Information

Section titled “More Information”