Telemetry truthfulness: three fields that reported things that weren't true

2026-08-04

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "GET /v1/intelligence/sampling-audit", "GET /v1/explain/{request_id}", "GET /v1/routing/decisions", ] sdk_methods_updated: ["ExplainRequest.get() response type (routing_strategy field)"] mcp_tools_updated: ["none"] ---

What We Built

Three telemetry fields that reported things that were not true, fixed after one of them caused a false P0 in a production review. GET /v1/intelligence/sampling-audit was printing hardcoded statsBefore: {successRate: 0, sampleCount: 0} / statsAfter: {successRate: 1, sampleCount: 1} on every single row, regardless of what actually happened — a reviewer read that and concluded Thompson sampling could never accumulate toward thompson_min_samples: 50. It could; the audit log just never looked at the real posterior. GET /v1/explain/{request_id} reported model_requested from the routing STRATEGY label ("price"/"quality") instead of the literal model string the caller sent — three reviewers independently found requests sent as "auto", "auto:best", and "openai/gpt-4o-mini" all reporting back model_requested: "price". And GET /v1/routing/decisions persisted routingMethod from the caller's strategy preference (not the method the router actually used) and never wrote candidateCount / score at all, so every row silently read them back as 0 / null — while the live X-BR-Selection-Method / X-BR-Models-Considered headers for the SAME request reported complexity-based / 28.

Why It Matters

Telemetry that looks real but isn't is worse than telemetry that's honestly absent — it gets trusted, and trusted-but-wrong data produces wrong conclusions (a false P0, in this case) and unanswerable chargeback disputes (a customer disputing a bill cannot be shown what model they actually asked for if the record was overwritten with an internal routing label). The fix follows the same rule as the health check's not_configured vs ok: a field must carry a real observed value or be absent — never a hardcoded or mislabeled value that reads as real.

How It Works

• Sampling audit: model-router.ts now snapshots the real per-model+shape posterior

from autoSelector.history — the SAME ModelAutoHistory store that gates history-based selection (thompson_min_samples) — immediately before and after the mutating recordOutcome() call, instead of printing hardcoded constants. When no prior sample exists, successRate is recorded as null (genuinely unknown), not 0 (a measured rate of zero).

• Explain model_requested: completions/index.ts now captures the caller's literal

model string BEFORE virtual-model resolution / community-tier override / canary / preset substitution, threads it through recordUsageEvent's metadata.requestedModel, and explain-request.ts reads it back. The routing strategy is preserved too, under its own routing_strategy field — never conflated with what the caller asked for.

• Decision trace: buildDecisionTrace now derives routingMethod / candidateCount

/ score / explorationBonus from the SAME autoSelectMeta / bandit-candidate lookup the X-BR-Selection-Method / X-BR-Models-Considered response headers use, so the persisted decision log can no longer contradict the live headers for the same request. usage-queries.ts's getRoutingDecisions now reads routingMethod from the decision trace instead of the unrelated routing_strategy DB column.

The Numbers

Three fields fixed; zero fields fabricated to "make tests pass." Two were populated with real data from existing stores, one field pairing (model_requested / routing_strategy) was split so neither concept overwrites the other.

Competitive Edge

Not customer-facing directly — but a governance/observability platform that ships wrong telemetry is a liability, not a feature. This closes three specific false-signal paths a reviewer or customer could hit.

Lockstep Checklist

• [x] API Routes: src/api/routes/explain-request.ts, src/api/usage-queries.ts —

response shape additions only (routing_strategy, corrected model_requested, corrected routingMethod/candidateCount/score), no new routes.

• [x] TS SDK: packages/sdk-ts/src/resources/explain-request.ts — added

routing_strategy field, documented model_requested semantics.

• [ ] Python SDK: not needed — ExplainRequest.get() returns an untyped Any in

packages/sdk-py, so no type update is required.

• [ ] MCP Schemas: not applicable — no MCP-facing schema change.
• [ ] Master Record: no new capability; existing entries unchanged.