Home Dashboard GitHub

Stochastic review round 4: fully-audited re-score (7.5→8.2), then 13 defects closed

2026-08-28

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "GET /v1/usage (new alias of /v1/usage/summary)", "GET /v1/ops/build (+deploy sub-object)", "POST /v1/observability/test/{id} (delivery receipt, 502 on failure)", "GET /v1/observability/destinations (+recent_test_results)", "POST /v1/register (409 tenant_already_exists on dup slug, was 500)", "POST /v1/governance/sovereignty/purge (403 on org mismatch, was 500)", "GET /v1/governance/artifacts/{request_id} (decision non-null on failure)", "GET /auth/eval/datasets (new dashboard JWT bridge)", ] sdk_methods_updated: ["usage.get() / usage.get (TS+Py sync+async) for the /v1/usage alias"] mcp_tools_updated: [ "br_route_completion (canonical provider/model id)", "br_explain_request (canonical served/selected model id)", "br_get_budget_status (spent_usd reflects real usage)", ] ---

What We Built

The first fully-audited stochastic-review re-score after the round-2/3/4-fix wave reached production, followed immediately by a 13-defect remediation. Round 4 scored the live build 398d205 with 11 full-tier keyed tenants (real deepseek+groq providers — no stub mode, the blocker that sank earlier waves), ten neutral-prompt personas in one wave, and an eleventh auditor that re-probed every ≥9 claim live. Mean 8.2, up from round 3's 7.5; no grade inflation (all four ≥9s reproduced against the live system, not asserted).

The headline movement: data-protection 4 → 9. Round 3's hard-cap — a route.dataPolicy:"zero" false guarantee that served 200 on an unattested endpoint — was verified closed with a positive control (bare policy now returns 403 data_protection_ineligible; all five adversarial checks fail closed). api-integrator (6→9), agent-consumer (6→8), and router-scientist (8→9, its 503-retains-decision gap closed) also climbed.

Deeper probing then found new real defects in the personas that had scored 9 on shallower passes — which is the point of the round. Those became the fix wave.

Why It Matters

A product whose differentiator is _provable_ behavior cannot ship checks that report success without running. Round 4 confirmed one live: observability POST /test returned {status:"sent"} for a destination with a bogus API key — a green with no observed delivery. It also found a governance artifact that named no model on a failed request, a data-residency purge that let a tenant stamp a foreign org id into the tamper-evident chain, and an evidence_mode directive that leaked to the upstream provider (Groq 400'd on it) instead of being enforced at the gateway. Each is the kind of gap that makes a broken thing look fixed.

How It Works — the 13 closed defects

platform-sre (was 6):

  • Observability /test now performs a real attemptDelivery and returns

{status:"delivered", accepted:true} only on an observed upstream 2xx, else HTTP 502 {status:"failed", upstream_status, reason} — with a persisted, Redis-backed delivery receipt provable afterward via recent_test_results.

  • The cross-task destination flip (200↔404 by load-balancer routing) was a

shared mutable DEFAULT_CONFIG singleton mutated per task; now a fresh object per call, reads reflect only the shared datastore.

  • The /v1/ops/* 405 handler no longer fabricates Allow: GET for paths where

GET 404s — findAllowedMethodsForPath now proves existence via a truly- registered handler, and a garbage ops path honestly 404s for every method.

  • GET /v1/ops/build gained a deploy sub-object (previous vs current

commit/digest + changed), honest as changed:null when the prior is unknown.

compliance-officer (was 8):

  • evidence_mode (the top-level alias) added to the provider-adapter reserved-

field strip, so the gateway governance directive never reaches the provider.

  • On provider failure, a bound route/error decision-lineage row is now

written from the same decisionTrace /v1/explain uses, so the artifact's decision names the attempted model+provider instead of null.

tenant-security (was 8):

  • sovereignty/purge validates the caller-supplied organization_id and

rejects a foreign org (403 sovereignty_org_mismatch) before any chain record is built; a reusable recordCrossTenantDenial emitter logs the refusal actor-scoped and metadata-only.

agent-consumer (was 8):

  • Route/explain/br_route_completion now emit the canonical

provider/model id, so an agent can chain explain → compare_models.

  • br_get_budget_status.spent_usd now reads the same usage_events source

br_get_usage uses (was a 0 enforcement counter never incremented on the MCP path).

dashboard-user (was 8) + claim-auditor (was 8):

  • GET /auth/eval/datasets bridge implemented (real backend existed); /v1/usage

bare alias added; register dup-slug now a typed 409 not a 500; /auth/api-keys CORS echoes the allowlisted origin; a real favicon; and llms.txt's header enumeration is honest (budget-remaining hedged, the non-existent memory-facts-injected header removed).

The Numbers

  • Round 4 mean 8.2 (round 3: 7.5). data-protection 4→9, api-integrator

6→9, router-scientist 8→9, agent-consumer 6→8.

  • Auditor: honesty 9, calibration 9; corrected once, in the harsh direction

(compliance-officer 7→8, under-credited crypto-verify evidence).

  • 13 defects closed across 6 subsystems; unit suite 10004 passing, 0 failures.

Competitive Edge

The moat is not that a feature works but that the platform can prove it did — and survive an adversarial re-probe of that proof. This round's method (keyed tenants, neutral prompts, a live-re-probing auditor that rejects unearned 9s) is itself the differentiator: Portkey/OpenRouter do not publish a self-graded, audited readiness ledger that hunts its own false-confidence surfaces.

Lockstep Checklist

  • [x] API Routes: new /v1/usage, /auth/eval/datasets; shape changes on

ops/build, observability/test+destinations, register (409), sovereignty purge (403), governance artifact.

  • [x] TS SDK: usage.get() added for /v1/usage.
  • [x] Python SDK: usage.get / async get added for /v1/usage.
  • [x] MCP Schemas: br_route_completion / br_explain_request canonical id;

br_get_budget_status real spend. Tool count unchanged (120).

  • [x] Master Record / llms: llms-full.txt + routes.json regenerated on

build (677 routes); llms.txt header enumeration corrected at source.

Open Follow-ups (carried to round 5)

  • Cross-tenant denial audit is wired at the sovereignty chokepoint; the

replay-returns-404 read routes (explain/artifacts/api-keys) are architecturally an empty tenant-scoped result, not an explicit denial — auditing those is a design decision, not a quick fix.

  • BYOK-denied dispatch-failure branch still yields decision:null (only the

provider-error branch was closed).

  • 9→10 residuals: router 503 error-class mislabel, null structured explain

fields, api-integrator generic recovery.* on 400s, 2-sig-fig usage rounding.

  • External blockers unchanged: /attestation verified:false (cosign/ECR),

PyPI publish token.