auto routing rebuilt: capability-matched, learning, and on for every tier
2026-08-11
LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["none"] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---
What We Built
A raw, fully-logged bake-off proved our "intelligent routing" claim was false: model:"auto" served one cheapest model (deepseek) on 15/15 prompts, labeled explicit, while a real competitor spread across four models matched to each prompt. We went back to the foundation and rebuilt the path.
auto now runs a single capability-matched selector: it classifies the prompt's required capability tier (economy → standard → premium → frontier), filters to models that are capable, affordable, and routable, then scores each with a learned-and-prior-blended reward and explores. A hard prompt reaches a strong model; an easy one stays cheap; the choice is recorded and the router gets better with every completion. And it is on for every tier — the community pin that rewrote auto to a fixed cheapest model before the router ever ran is gone.
Why It Matters
The product's entire promise is intelligent, per-prompt routing. We were shipping a cheapest-sort wearing that label. This makes the promise true — and true for free-tier traffic too, which is where most first impressions happen. A developer who sends a hard reasoning prompt on a community key now gets a model that can actually answer it, bounded by the tenant's budget killswitch and output clamp.
How It Works
Selection scores each capable candidate as:
score = qEst − λ·logNormCost − μ·tierDistance
qEst = (n·measuredReward + k·qualityPrior) / (n + k) // shrinkage, k=5- Capability priors (
provider-catalog-intelligence.ts) give every model a tier and a 0–1
quality prior, with a price/reasoning/context heuristic fallback so the map is never a hard dependency.
classifyRequiredTierturns the (previously discarded) contentintent+ complexity +
structural signals into a required tier — a hard floor for capability, and a soft target via μ·tierDistance so routing genuinely varies by difficulty instead of collapsing onto one cost-optimal model.
- Shrinkage means the static prior drives cold-start (capability-matched from request #1) and
measured reward takes over by n≈20 (learned). The learning loop was also repaired: outcomes now record for every completion, not just the auto path, ending a self-reinforcing cold start.
- The selector reuses the existing
ModelBanditfor exploration and ships behind an
adaptiveSelection flag — on by default in the running router, off in the unit-construction default so the legacy cascade stays under test until it is deleted post-soak.
The Numbers
An in-process proof (model-auto-selector-proof.test.ts) drives a difficulty-spanning workload through the real selector and asserts the routing decisions directly (no assumption about model output quality):
trivial → gpt-4.1-nano [economy] moderate → deepseek-v4-flash [standard]
easy → gpt-4.1-nano [economy] hard → deepseek-v4-pro [premium]
expert → deepseek-v4-pro [premium]3 distinct models, tier monotonic in difficulty, cheap stays cheap, hard reaches premium — versus the old cascade's one model for everything. 9766 unit tests pass.
Competitive Edge
Portkey and OpenRouter route on price, latency, or a static config; their "auto" is a fixed rule. Ours classifies each prompt's required capability, matches a model to it, and then _learns_ per tenant from real outcomes — cost-efficiently (routing a proof to a $2/M premium reasoner instead of a $90/M frontier model is the thesis working, not a shortcut). It is the memory + bandit + capability-catalog stack doing something a generic gateway structurally cannot.
Lockstep Checklist
- [x] API Routes: no new/changed routes — behavior change in the router + community middleware only.
- [x] TS SDK: no surface change (routing is transparent to callers).
- [x] Python SDK: no surface change.
- [x] MCP Schemas: no change.
- [x] Master Record: routing capability already recorded; behavior updated, not a new capability.