Home Dashboard GitHub

Round-4 remediation — cost truthfulness + routing explain/health honesty

2026-08-09

routerintelligenceapisdk

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "GET /v1/intelligence/savings", "GET /v1/intelligence/benchmark", "GET /v1/usage/summary", "GET /v1/explain/{id}", "GET /v1/health/providers", "none", ] sdk_methods_updated: ["explain-request response types (packages/sdk-ts)"] mcp_tools_updated: ["none"] ---

What We Built

Six Round-4 defects across cost-truthfulness and routing observability, all revert-tested, tsgo clean.

Cost truthfulness

  1. Savings are real. x-br-routing-savings / /v1/intelligence/savings baselined every request against gpt-5.5 (the priciest catalog model nobody chose) and booked "savings" even for explicit single-option requests. Now: recorded only when the router actually chose (auto stage, >1 model considered), baselined against the costliest model actually considered and declined, with an X-BR-Routing-Savings-Baseline header and 0 on explicit routes.
  2. Advise math matches the catalog. /v1/intelligence/advise percentages came from empirical means (16–47× wrong; listed the recommended model as its own alternative). Now computed from the same catalog pricing /v1/models serves, keyed canonically.
  3. The Tier-2 quality judge is not billed to the tenant. It fired an HTTP loopback replaying the caller's auth → a shadow gpt-4.1-mini charge under the tenant with caused_by_request_id null. Now a direct in-process call marked isInternal + costCenter: platform:quality; /v1/usage/summary splits tenantCostUsd vs platformOverheadUsd.
  4. Benchmark verdict requires concordant evidence. thompson_beats_static_price was declared on a mean-reward delta while price won 72% of head-to-heads. Now requires win-rate concordance and reports verdict_basis + cost_comparison (→ mixed_evidence for the observed data).

Routing observability 5. /v1/explain tells the truth. It reported the last (post-cascade) stage's reason with routing_strategy: null and alternatives_considered: [], contradicting the headers. Now it reports the actual selection method/strategy from the auto-select stage, the effective strategy, and the considered-model set; the hardcoded x-br-route-reason: thompson-sampling is replaced with the real selection method. 6. /v1/health/providers reflects reachability. routable meant "not all breakers open" regardless of whether the tenant has a key. Now routable = has_key && !allOpen && tenant_auth !== auth_error, with per-provider tenant_auth and routable_for_tenant; a keyless provider reports routable_models: 0, reason: no_tenant_key.

Why It Matters

Round 4 capped cost-owner (5) and router-scientist (8) on exactly these — a "cost optimization" product whose savings are invented and an explain surface that disagrees with what ran. This wave makes both honest.

Lockstep Checklist

  • [x] API Routes: intelligence/explain/health/usage capabilities updated; no route add/remove.
  • [x] TS SDK: explain-request response types updated; savings/usage/benchmark additive fields (types follow-up).
  • [x] Python SDK: additive response fields (follow-up).
  • [x] MCP: none.
  • [x] Master Record: defects tracked in docs/reviews/2026-08-08/ledger.json.

Follow-ups: SDK additive response types for savings/usage/benchmark; behavior-json snapshot regen. platform-sre alerting + dashboard CORS remain (code + owner-gated OPS).