One authority per data class: five 'reads zero' bugs and a hard 500 were all the same split-brain

2026-08-08

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "GET /v1/audit/events", "GET /v1/audit/verify", "GET /v1/audit/export", "GET /v1/audit/retention", "PUT /v1/audit/retention", "GET /v1/governance/completion-audit", "GET /v1/governance/lineage", "GET /v1/governance/compliance/coverage", "GET /v1/billing/plan", ] sdk_methods_updated: [ "client.audit.events()", "client.audit.verify()", "client.audit.export()", "client.audit.getRetention()", "client.audit.setRetention()", "audit.events()/verify()/export_events()/get_retention()/set_retention() (py, sync+async)", ] mcp_tools_updated: ["none"] ---

What We Built

For nearly every read surface in the governance and billing stack there were two stores: a process-local in-memory one the API read, and a durable Postgres one that traffic wrote. They were never bridged. That one defect produced five independently-reported symptoms — an empty audit export next to 68 committed requests, a valid: true chain verdict over chain_length: 0, a lineage list whose records the lineage detail route could not resolve, an evidence-coverage metric that scored itself 100% from 0/0, and a billing page that reported zero usage for every tenant.

This ships the rule: one authority per data class. Evidence and usage live in Postgres. Coordination lives in Redis. In-memory is a read-through cache and never a store of record. Every affected surface was repointed at the authority, and every surface that can fail to read it now says so instead of returning a flattering default.

It also fixes GET /v1/governance/completion-audit, which returned HTTP 500 to every caller that had data — the single most damaging item, because a shipped Governance Artifact cites that exact endpoint as its own SOC 2 CC7.2 evidence source. The control was mapped to something an auditor could not read.

Why It Matters

Governance software is only worth what an auditor can independently check. An empty audit export is worse than an error: it reads as an affirmative all-clear. A boundRatio of 1 computed from 0/0 is a perfect evidence-coverage score awarded for having recorded nothing. Both would have survived a demo and failed an audit.

The new rule is that absence must be distinguishable from a measured pass. Where a value is genuinely unknown, these endpoints now return null plus a source / degraded marker, never a plausible zero.

How It Works

/v1/audit/ now reads completion_audit. src/security/audit-store.ts used to be a Map with exactly one writer, recordAuditEvent — which had no caller on any traffic path (only setRetention in the same file, plus its unit test). The Map is gone. The module is now a pure projection of committed Postgres rows, with total coming from a real count() over the same predicate as the page, so an export reconciles 1:1 with the tenant's request count. X-Audit-Row-Count, X-Audit-Total and X-Audit-Truncated headers let a consumer prove completeness without trusting the body length. Retention policy — configuration, not evidence — moved to the per-tenant Postgres config store, so it survives restarts and is identical across replicas.

Chain verification is three-state. /v1/audit/verify delegates to the shared verifyTenantAuditChain, the same implementation behind the governance route and the dashboard badge, and mirrors its chain_state verbatim (verified | broken | not_applicable). An empty chain is not_applicable with a reason — reporting true claims a pass nobody computed, and reporting false would render every brand-new tenant as BROKEN.

The 500 was a BigInt. completion_audit.tenant_seq is bigint(… { mode: "bigint" }), i.e. a JS BigInt. The handler returned raw drizzle rows, and JSON.stringify throws TypeError: Do not know how to serialize a BigInt. The throw happens inside the Hono JSON serializer, _after_ the handler's own try/catch has returned, so the route's error handling never saw it. Rows are now projected through a pure, snake_case shapeCompletionAuditRow — which additionally makes the route match the published SDK CompletionAuditEntry type for the first time (it was camelCase on the wire and snake_case in the SDK).

Coverage reconciles against traffic. boundRatio is null when total is 0, and the response now carries traffic.auditRows (the real completion_audit count for the same window), traffic.recordedRatio and traffic.unrecorded. The production shape now reads "0 of 68 recorded", not "total 0, 100% bound".

Lineage list and detail share one authority. The unfiltered GET /v1/governance/lineage used to fall through to the ring buffer while GET /v1/governance/lineage/{request_id} read Postgres only. The list now reads the same durable table; the ring buffer requires an explicit ?live=true and is labelled source: "live".

Billing usage is measured. getPlanUsage counts usage_events for the current calendar month (matching the requests_per_month limit's unit), non-revoked api_keys, and agent_registry identities in one round trip. PlanUsage fields are number | null; an unknown counter contributes nothing to the upgrade threshold rather than being silently treated as 0%.

Lineage persistence is enabled. BR_LINEAGE_PERSIST and BR_CHAIN_PAYLOAD_V2 are now set in the Fargate task definition. The resolver fails safe to off when the variable is _absent_, which it was — that single unset variable is why coverage divided by an empty table, why the WHY route answered lineage: null, and why the Governance Artifact's decision field was null (the assembler embeds answer.lineage, which carries selected_model / selected_provider; the shape was right, the table was empty).

The Numbers

• 6 read surfaces repointed from a process-local store to the Postgres authority.
• 1 endpoint that returned HTTP 500 on 6/6 live probes now returns 200.
• 3 dishonest defaults removed: boundRatio: 1 from 0/0, usage: {0,0,0}, and

valid: true over chain_length: 0.

• 0 new tables and 0 migrations — every fix reads already-committed rows.

Competitive Edge

Portkey and OpenRouter can tell you what a request cost. A documentation-and- registry governance tool (Credo, Holistic AI) can tell you what policy you wrote down. Neither sits on the request path, so neither can produce a number like "of 68 completions in this window, N carry cryptographically bound, verifiable evidence, and here is the export that reconciles to it row for row." That reconciliation — and the discipline of reporting null instead of a flattering default when it cannot be computed — is what makes the evidence usable in an actual audit rather than in a slide.

Lockstep Checklist

• [x] API Routes: capability handlers under src/api/capabilities/ updated

(this surface migrated off src/api/routes/).

• [x] TS SDK: packages/sdk-ts/src/resources/audit.ts fully typed;

types.ts gains EvidenceCoverage, PlanUsageStatus, and the corrected CompletionAuditEntry. Generated resources are rebuilt by pnpm gen:contract (that tree is gitignored).

• [x] Python SDK: packages/sdk-py/.../resources/audit.py updated, sync +

async, with the new filters and the three-state verify semantics.

• [x] MCP Schemas: no agent-facing tool added or changed.
• [ ] Master Record: docs/architecture/master-capability-record.mdx — not

touched; no new capability was introduced, only existing ones repointed.