Stochastic Review Round 2 — 10 personas, audited mean 5.4 on a pinned build
2026-08-08
LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["none"] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---
What We Built
This entry records a full, valid run of the Stochastic Review harness against production build ff6f275 — the first complete 10-persona round (Round 1 measured only 5 personas and was invalidated when a mid-round push changed x-br-build). Ten personas each did a real job against the live gateway on their own fresh sandbox tenant, an eleventh agent audited every score for inflation and deflation, and the result was written to docs/reviews/2026-08-08/ledger.json.
The audited mean Provable Readiness was 5.4 / 10, range 4–8. The auditor accepted all ten scores with zero corrections, having verified the top scores' positive controls are real and every capping defect on the low scores against both source and the live gateway. The round did not converge (convergence requires every persona ≥9 and mean ≥9).
Getting to a _valid_ round was itself most of the work. A terraform apply had rolled production back to a 27-day-old image (:latest) and dropped the build-provenance env vars; that was corrected (task-def rev 35, image repinned to ff6f275, provenance + governance env restored) and the service converged to a single revision before any persona ran. Each mutating persona then needed a tenant with a _real_ provider key — a fresh sandbox tenant returns a canned brainstorm/sandbox stub, and grading a stub would measure nothing. The per-IP sandbox cap (working as designed) allowed only 3; the rest were provisioned via an invite code minted in-container over ECS Exec on the same build.
Why It Matters
The differentiating claim of this platform is not that a feature works but that the platform can prove it worked. Round 2 shows that thesis is half-true in a very specific, honest way: the evidence core is excellent — three independent personas verified the Ed25519 governance artifact signature offline against the published JWKS, one with its own RFC 8785 canonicalizer, and tampering one byte broke verification — while the routing-intelligence and cost surfaces have truthfulness gaps that several personas found independently. Knowing exactly where the line is, with reproducible evidence, is worth more than a flattering number.
How It Works
Each persona authenticated with its own br_live_ key, exercised its surfaces over live HTTPS, and returned the JSON contract with how fields naming the build. Isolation claims required a positive control on the same route (owner 200 before attacker 404). The auditor re-verified selectively against the gateway. Build stability was checked at round start, by every persona, and at round end — all ff6f275 (Rule 0 satisfied).
The Numbers
| Persona | Score | One-line |
|---|---|---|
| tenant-security | 8 | No cross-tenant leaks; every refusal has a positive control; rotate() fix holds |
| compliance-officer | 7 | Signed artifact verified offline; capped by evidence_mode not being a control + cache-hit gaps |
| dashboard-user | 7 | Real signup→key→call→logs loop; capped by dead Settings link + /auth/events CORS storm |
| api-integrator | 6 | Gateway excellent; both SDK quickstarts 405 (default baseURL omits /v1) |
| router-scientist | 5 | Routing intelligence genuinely in-path; auto 400s ~half the time, fallback won't cross providers |
| cost-owner | 5 | Accounting exact; savings fabricated, advise% off 16–47x, cross-tenant intelligence data |
| platform-sre | 4 | Alarms fire but reach no human (SNS PendingConfirmation); readiness guard can't fail |
| claim-auditor | 4 | Grants API (README headline) 403; memory not injected; explicit pin silently substituted |
| agent-consumer | 4 | 4 advertised MCP tools return success-shaped "not configured" |
| data-protection | 4 | provider.only/ignore inviolable; data_protection field inert; dataPolicy:"zero" discarded |
Mean 5.4. Auditor: honesty 9, calibration 8, corrections 0.
Three systemic defects, each found independently by multiple personas and confirmed by the auditor:
- BYOK-cascade silent model substitution (high) — explicit pin served by a different vendor while
x-br-model-contract: strict. - Prompt cache bypasses governance/evidence/coverage/ZDR and the ledger (high) — cache hits have no audit row or artifact.
/v1/intelligence/savings&/benchmarkserve cross-tenant aggregates to a tenant key (medium) — metadata, not content.
Competitive Edge
Portkey and OpenRouter ship runtime _controls_; Credo AI governs _pre-deployment_. This review demonstrates the thing none of them can: a self-grading harness that runs real jobs against the running gateway and produces cryptographically verifiable evidence of what happened — and is honest enough to score itself 4 where the evidence does not hold. The next rounds close the three systemic defects and re-measure.
Lockstep Checklist
> No API routes, SDKs, or MCP tools changed in this entry — it is a measurement record plus the > ledger. Remediation of the defects above will carry its own lockstep updates.
- [x] API Routes: none changed.
- [x] TS SDK: none changed.
- [x] Python SDK: none changed.
- [x] MCP Schemas: none changed.
- [x] Master Record: no new capability introduced; defects tracked in the ledger.