Home Dashboard GitHub

Stochastic review round 3: honest savings, closed explain loops, first-class model substitution

2026-08-28

routinggovernancecost-attributionobservabilitymcptenant-onboardingdashboard

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "GET /v1/intelligence/savings", "GET /v1/explain/{id}", "GET /v1/governance/completion-audit", "GET /v1/observability", "GET /v1/ops/diagnose", ] sdk_methods_updated: ["ts: presets.create sends slug + surfaces issues/recovery; README memory block"] mcp_tools_updated: ["br_route_completion (platform req_ id)"] ---

What We Built

Round 3 of the 2026-08-28 review (round-2 mean 7.8). Closes 14 residuals across the 7-at-the-bottom personas.

Cost/savings, honest and reconcilable. The savings summary was internally inconsistent (avg × trackedRequests ≠ total). Now avgSavingsPerRequest × headlineSavingRequests == totalSavingsUsd (the denominator is displayed), the aggressive vsCostliestConsideredUsd moved under hypothetical: {billable: false} (out of the headline), a prompt_cache savings source credits cache-hit avoided cost, and usage/summary.totalCostUsd now equals tenant-billable cost with platform overhead a separate labeled field.

Explain loops closed. br_route_completion (MCP) returned the bare upstream completion UUID; it now mints and threads the platform req_ id so the tool result resolves in br_explain_request / /v1/explain. Failed and denied requests now carry the pre-failure decision detail (scored alternatives_considered, complexity, exploration, and the ordered failover_path), and pre-dispatch validation rejections (unknown_model) resolve with outcome:"rejected" instead of 404.

First-class model substitution. When a tenant lacks the capability-matched winner's provider key, the BYOK cascade substitutes another model — now recorded as a structured selected_model / served_model / substitution_reason triple in /v1/explain, so the divergence is intentional and traceable rather than reading as if capability-matching didn't run.

Governance audit surfaces. GET /v1/governance/completion-audit?request_id= now actually filters (404 on no match, shaped like the artifacts route); audit/verify is .strict() so an unknown top-level field (event_hash) 400s instead of a benign incomplete.

Data-protection. The two zero-retention expressions are unified — an operator-declared static-zero endpoint now satisfies both route.dataPolicy:"zero" and data_protection.retention_max_days:0 (unknown/standard still fail closed).

Prompts durability. POST /v1/prompts was a phantom write (null boot-set singleton, 201 anyway) — now durable via the per-replica Postgres config store, the same fix presets got.

Truthful ops + polish. /attestation top-level status is now signed_unverified when cosign hasn't verified (no green over a failed check); X-BR-Budget-Remaining emits for per-key budgets; a typed 404/405 envelope with a "did you mean /v1/…?" hint covers non-/v1 known suffixes; GET /v1/observability aggregate + GET /v1/ops/diagnose alias restore the surfaces that read as absent; the dashboard SSE reconnects with a fresh token and the Settings nav link is wired.

Review-environment: openai + google BYOK keys were added to all 10 review tenants so frontier-tier routing serves the real capability-matched winner (verified: a hard prompt now routes to openai/o3, not a forced-deepseek 504).

Verification

  • pnpm check + pnpm build + dashboard build green; 155 unit tests across 14 round-3

test files pass; openapi 530 paths.

  • Round 4 review runs against the deployed build; scores in the ledger.

Known external blocker

Python SDK 0.1.2 (base_url /v1 fix) is verified working locally but PyPI still serves the broken 0.1.0 — twine 403 Forbidden, the GitHub PYPI_TOKEN is invalid. Caps api-integrator until the operator refreshes the token. Not fixable autonomously.

Lockstep Checklist

  • [x] API Routes: savings, explain (served/selected + failure detail + rejection), completion-audit filter, audit/verify strict, observability/diagnose GET, prompts durability.
  • [x] TS SDK: presets.create slug + structured errors; README block. Python SDK: publish still token-blocked.
  • [x] MCP: brroute_completion platform req id.
  • [x] Docs/OpenAPI: regenerated (530 paths); self-describing refreshed.