Learning loop closed at all three joints — truthful labels, streaming trace, outcome taxonomy

2026-08-27

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "POST /v1/chat/completions (streaming: real X-BR-Selection-Method/X-BR-Route-Reason, persisted decisionTrace, lineage)", "POST /v1/feedback/{requestId} (reward now reaches the live selector)", "GET /v1/routing-stream (new kind=outcome events with success + latency_ms)", "GET /v1/explain/{request_id} (non-empty for streamed requests)", ] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---

What We Built

The 2026-08-27 cross-repo audit found the "routing that learns from verified outcomes" claim broken at three joints, none of them on the review board:

1. The self-label was a constant. The only writer into ModelAutoHistory

— the store selectCapabilityMatched reads its learned quality term from — recorded a literal success: true on non-streaming successes only. EWMA of a constant meant every used model converged to successRate 1.0 and permanently outranked unused priors (rich-get-richer lock-in, offset only by the ε≥0.02 exploration floor). Failures recorded nothing; streaming recorded nothing.

1. /v1/feedback computed a real reward into a dead-end store. The

endpoint wrote to ModelPerformanceTracker, which feeds read surfaces (/v1/recommend, leaderboard) but is never consulted by the live selector.

1. The routing stream had no failure taxonomy. Route-time decision events

were the only kind; a downstream consumer (the Brainstorm harness's learned-strategy observer) recorded every decision as success=true.

All three are now closed:

• The completion label derives from scoreValidity().outcomeStatus

(LEARNING_FAILURE_STATUSES: failed / refused / tool_invalid / timeout / rate_limited; degraded deliberately counts as success). Model-attributable execution failures (not 401/403/404/429 — tenant-key, catalog, and capacity signals never masquerade as quality) record success: false. The streaming wrapper records a validity-labeled sample at stream end and a failure sample on mid-stream error.

• /v1/feedback bridges its computed reward into a model-global evidence

channel (ModelAutoHistory.recordModelOutcome, reserved shape key __model_global__ — persists through model_performance with zero schema changes). selectCapabilityMatched blends it into qEst with capped weight (GLOBAL_FEEDBACK_EVIDENCE_CAP = 10 against shrinkage K = 5): byte-identical to the legacy formula until feedback exists, decisive when it does.

• The routing stream gained completion-time kind: "outcome" events carrying

the truthful success label and real latency_ms, emitted from the non-streaming success/failure paths and the streaming wrapper. Backward compatible: kind is optional; absent means route-time decision.

Streaming transparency shipped alongside: RouteStreamResult now carries routingStages and a decisionTrace (built at route time via buildDecisionTrace, timing finalized in-place at stream end), the streaming route emits the REAL X-BR-Selection-Method / X-BR-Route-Reason / complexity headers from stages, persists the trace into the usage row's metadata, and wrapStreamWithIntelligence captures completion lineage — so a streamed request's X-BR-Explain link finally resolves to a real explanation with a non-null selection reason and scored alternatives.

Why It Matters

The learning term in the exact expression that routes (qEst = (n·measured + n_g·global + K·prior)/(n + n_g + K)) is now falsifiable: a model that keeps failing loses standing, reported outcomes are observable by the next selection of any request shape, and agent traffic — which streams — teaches the posterior instead of bypassing it. This converts "learned routing" from copy into the data asset the positioning claims.

Proof

src/tests/invariants/learning-loop-closure.invariant.test.ts — five permanent regression pins: a control (successes record), the three joints, and the acceptance test: two same-price same-tier models, repeated verified-failure reports through the REAL feedback handler flip the next selection away from the failing model. All assertions target ModelAutoHistory, never ModelPerformanceTracker — the fake-pass where dashboards move while routing stays blind is designed out.