Home Dashboard GitHub

Stochastic review round 2: 21 defects fixed, alerting resurrected, false attestations killed

2026-08-28

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "POST /v1/presets (now durable)", "GET /v1/explain/{id} (failed requests)", "GET /v1/intelligence/savings", "GET /v1/ops/diagnose", "GET /attestation", ] sdk_methods_updated: ["python 0.1.2 base_url /v1 fix (needs PyPI publish)"] mcp_tools_updated: ["br_route_completion (+request_id)", "br_compare_models (+output +picks)"] ---

What We Built

The 2026-08-28 stochastic review (10 personas, build bf1c5f0) scored mean 7.0 with platform-sre anchored at 4 by the hard-cap rule. This round closes 21 of the open defects. The highest-value ones:

False governance attestations (3 personas). The outbound DLP path minted signed artifacts + response headers claiming pii_redacted satisfying SOC2 CC6.7 / GDPR Art.32 "masked before egress" while the body delivered the value unmasked — and the swift_bic matcher false-positived on ordinary uppercase words ("INTEGRATION", "BASELINE"). Fixed: the matcher now requires a real ISO-3166 country and rejects common words; warn-mode records pii_detected (not pii_redacted) and cites a detective control (CC7.3), never a masking claim, unless the egressed bytes were actually changed.

Dead alerting, resurrected (2 personas). CloudWatch alarm notifications had been silently denied for 13 days — the SNS topic policy allowed only budgets.amazonaws.com to publish. And the alert-receipt Lambda's REDIS_URL had been blanked by a parameterless cloudformation deploy, so it logged "0 errors" while writing nothing. Both fixed in the template (added the cloudwatch.amazonaws.com principal; resolve REDIS_URL from the routing secret) and applied live — proven end-to-end: a forced test alarm now published to SNS, invoked the Lambda, and appeared in /v1/ops/alert-log. The soak-test alarm's treatMissing flipped to breaching so a silent soak can no longer read as green.

Failed requests are now explainable (4 personas). Upstream 504s, byok_required, and model-unavailable denials left no /v1/explain record. A classifyDispatchFailure

  • recordDispatchFailure path now commits a decision record (outcome error/denied,

terminal code/provider/message, the real failover attempt chain) for failed and denied outcomes, keyed by the same request_id the caller received.

Honest cost & savings (2 personas). Auto-select sampler probes are now tagged platform:sampling with initiator: router_probe so they're excluded from tenantCostUsd and the tenant quota; savings.bySource/topSource report the honest ranked_runner_up and aggressive vsCostliestConsidered figures separately instead of blending them into one ~1000x-inflated number.

Silent data loss (2 personas). POST /v1/presets was a phantom write (persisted to a null boot-set singleton, returned 201 anyway) — now durable via the per-replica Postgres config store, throwing on failure. POST /v1/memory/entries stored the context field as the fact (context ?? fact) — flipped to fact ?? context.

Also fixed

Ops truthfulness (/v1/ops/diagnose now carries probe evidence and can't return empty-evidence "healthy"; /v1/ops/status flags probe_derived per endpoint; /health/ready and providers share one key map; /v1/messages self-describes); attestation verified:boolean + --k8s-keychain so cosign authenticates to ECR + Redis-shared cross-instance determinism; kill-switch 503 carries recovery+request_id and deactivate busts the enforcement cache (effective_within_ms); br_route_completion returns the request_id (closes the MCP explain loop); br_compare_models delivers output + cheapest/fastest/best picks; bare 405 returns a typed envelope; top-level evidence_mode validated; dp cache re-gated on fact revocation; auto+retention:0 routes to an eligible attested endpoint; zdr honored on an in-force zero fact; X-BR-Budget-Remaining emitted for tenant caps; dashboard SSE badge from real connectionState + #analytics stream. Python SDK 0.1.2 base_url//v1 fix verified working locally (publish pending a valid PYPI_TOKEN).

Verification

  • pnpm check + pnpm build green; 218 unit tests across 18 round-2 test files pass.
  • Alerting pipeline proven live (SNS publish + Lambda invocation + alert-log receipt).
  • Full re-review (round 2) runs against the deployed build; scores land in

docs/reviews/2026-08-28/ledger.json.

Lockstep Checklist

  • [x] API Routes: presets durability, failed-request evidence, savings split, ops surfaces.
  • [x] TS SDK: unaffected (0.1.2 correct); Python SDK: 0.1.2 base_url fix verified, publish pending token.
  • [x] MCP: br_route_completion +request_id, br_compare_models +output/picks; tool count 120 unchanged.
  • [x] Infra: monitoring stack (SNS principal + Lambda Redis + soak breaching) applied live + durable in template.
  • [x] Docs/OpenAPI: regenerated (529 paths); self-describing refreshed.