Stochastic review round 5: a fix wave regressed observability — caught, root-caused, and closed

2026-08-28

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "POST/PUT/DELETE /v1/observability/* (503 write_not_persisted instead of false 2xx)", "POST /v1/chat/completions + /v1/messages (body.model now canonical provider/model)", "auto-routing-exhausted 503 (service_unavailable/routing_exhausted, Retry-After)", "POST /v1/governance/sovereignty/policy (403 on foreign org, was 201)", "GET /v1/audit/events (now surfaces cross-tenant denial rows)", "POST /v1/chat/completions (400 misplaced_compliance_field on top-level dataPolicy/zdr/…)", ] sdk_methods_updated: ["none — no new routes; response-shape/error-code changes only (documented)"] mcp_tools_updated: [ "agents.json now emits inputSchema for all 120 tools", "br_route_completion / completion body.model canonical", ] ---

What We Built

Round 5 re-scored the live build after the round-4 fix wave. It did the most valuable thing a review can do: it caught a regression the fix wave itself introduced. Mean fell 8.2 → 7.5, anchored by platform-sre dropping 6 → 4 — the observability write path had become a silent success. This entry covers that finding and the 10-defect wave that closed it and the round's other residuals.

Every _targeted_ round-4 fix was independently re-verified live and held (the failure artifact names the attempted model, evidence_mode is enforced at the gateway, budget spend is real, canonical ids in explain, honest /test, the 405 handler, deploy-delta, purge org-validation). The score drop was not those fixes failing — it was (a) one regression and (b) each persona probing a layer deeper and finding the next defect.

Why It Matters

The observability regression is the textbook case the whole method exists to catch. The round-4 change "return a fresh config object" removed an in-memory singleton that had been _masking_ a write that never persisted. POST /destinations returned 201 with a UUID and PUT /config {enabled:true} returned 200 — while nothing was stored. An operator wiring a Datadog/Langfuse sink would see success and believe telemetry flowed; none would. A product that sells provable behavior cannot ship a write that lies about persisting.

How It Works — the 10 closed defects

Observability (platform-sre 4): the root cause was double silent-success — saveBroadcastConfig returned as if it succeeded when the config manager was unavailable, and loadTenantBroadcastConfig swallowed real read errors as "empty." The read path already reads through to shared Postgres, so the fix is to make writes prove themselves: saveBroadcastConfig now throws + reads back via getSection to verify before returning, and the handlers return 503 write_not_persisted rather than a false 2xx. The config-key enum gained the missing posthog/s3 types so valid destinations persist instead of 503-ing.

Routing (router-scientist 7): body.model on /v1/chat/completions and /v1/messages (streaming + non-streaming) is now the canonical provider/model id (so explain → compare_models chains); an auto-routing-exhausted 503 is reclassed service_unavailable/routing_exhausted with Retry-After; and the BYOK cascade now actually cascades — the re-entry was excluding only the pre-BYOK pick, re-hammering the executed endpoint; it now falls through to the next scored candidate with constraints re-applied. (The circuit-breaker fast-fail is a documented residual — a hot-path change deferred rather than half-done.)

Tenant isolation (tenant-security 7): the sibling sovereignty/policy route now rejects a foreign organization_id (403, sharing a generalized ownership error with the purge fix), and cross-tenant denials now surface on /v1/audit/events for the actor tenant (metadata-only, unchained).

Data-protection & discovery: a misplaced top-level compliance field (dataPolicy, zdr, retention_max_days, …) now fails closed with 400 misplaced_compliance_field naming the correct nesting; and agents.json now emits an inputSchema for all 120 MCP tools (39KB → 116KB) so agents stop brute-forcing arguments.

Dashboard (dashboard-user 8): /dashboard/favicon. serve real images (SPA catch-all rewrite fixed), /auth/ CORS echoes the allowlisted origin instead of * across the namespace, and the Evaluations view defers its datasets fetch until the session provisions (no cold-boot 404).

The Numbers

• Round 5 mean 7.5 (round 4: 8.2); platform-sre 6 → 4 (regression),

cost-owner/tenant-security/router-scientist 9 → 7 (deeper probing).

• Auditor: honesty 9, calibration 9, no grade inflation; the regression was

independently confirmed live.

• 10 defects closed; unit suite 10027 passing, 0 failures.

Competitive Edge

A self-grading loop that catches its _own_ regression the same day — and refuses to score a lying write above a 4 — is the moat. The value is not a high number; it is a number you can trust because an adversarial auditor tried to break it.

Lockstep Checklist

• [x] API Routes: no new routes; response-shape/error-code changes on

observability (503), completions (body.model, 503 class), sovereignty policy (403), audit/events (denial rows), data-protection (400).

• [x] TS/Python SDK: no new methods (no route additions); error-code/shape

changes documented for SDK error handling.

• [x] MCP / discovery: agents.json regenerated with per-tool inputSchema;

canonical body.model. Tool count unchanged (120).

• [x] llms / routes: llms-full.txt + routes.json regenerated (677 routes).

Open Follow-ups (carried to round 6 / beyond)

• Circuit-breaker fast-fail (consult breaker before pinned/BYOK dispatch;

threshold tuning) — deferred hot-path change.

• Cache attribution in the JSON usage body (a hit is reported as a miss; no

aggregated savings) — cost-owner residual, not yet closed.

• Evidence provenance-binding + coverage ratios (compliance) — deeper work.
• Verify live whether the observability manager is available in prod (writes then

persist for real) or writes now 503 honestly — either removes the hard cap.

• External: /attestation verified:false (cosign/ECR), PyPI token.