Comparative router harness — capability + performance vs Portkey / OpenRouter / Cursor / DigitalOcean

2026-08-26

What We Built

A two-layer comparative harness plus a synthesis into a ranked BR backlog:

1. Performance bake-off (extended scripts/bench-routers.ts) — now captures response

headers and post-hoc introspection (/v1/explain, OpenRouter /generation), so cost/ latency/quality carry a disclosure column, not just a number.

1. Capability probe matrix (scripts/router-capability-probes.ts +

src/router/intelligence/capability-probes.ts) — live-probes six dimensions across all platforms, grading full/partial/none with a required evidence string per claim.

1. Synthesis (this run) — mechanical scores from the matrix + curated best/worst-practice

analysis → the ledger and this entry.

Platforms compared: brainstormrouter, cursor, digitalocean, openrouter, portkey.

Scoreboard

Dimension	BR	Leader(s)	vs BR	Recommendation (impact/effort)
Routing intelligence	full	brainstormrouter, openrouter	parity	Expose BR routing policies as addressable model:"br:" ids (reuse the auto-selector + Thompson bandit behind a named object). (high/medium)
Inline cost attribution	full	brainstormrouter, openrouter	parity	Keep X-BR-Actual-Cost always-on AND mirror it into the usage body object for OpenAI-SDK callers who never read headers. (medium/low)
Caching	partial	brainstormrouter, digitalocean, portkey	parity	Make BR's semantic cache a headline, generally-available feature with a caller-tunable similarity threshold and X-BR-Cache-Similarity always disclosed. (high/medium)
Reliability / failover	full	brainstormrouter	win	Promote BR's circuit-breaker to a headline, observable feature: always emit X-BR-Failover-Path on any hop and expose breaker state on /v1/ops. (high/low)
Governance / data policy	full	brainstormrouter, cursor, portkey	parity	Lead with the per-request data_protection obligation + X-BR-Data-Policy proof (already shipped) and publish BR's own retention plainly, one toggle. (medium/low)
Decision rationale	full	brainstormrouter	win	Bank the lead: make the decision trace a marketed differentiator, and surface a compact rationale inline (X-BR-Route-Reason + a link to /v1/explain) on every routed response. (high/low)

Result: BR leads 2 dimension(s), trails on 0.

The Headline

The competitor survey's core finding held up under live probing: no competitor exposes a per-request routing rationale. BR's /v1/explain returns selection_reason plus priced alternatives_considered — WHY, not just which. That is the field's weakest axis and BR's strongest; the backlog's top move is to bank and publicize it, not build it.

Best Practices Worth Stealing

• Routing intelligence (from DigitalOcean): Router-as-model-id: a named, dashboard-managed policy invoked by one OpenAI-compatible model:"router:" string — the caller changes nothing else.
• Inline cost attribution (from OpenRouter): Per-request usage.cost + cost_details.upstream_inference_cost returned in the response body — log cost per call with zero extra API round-trips.
• Caching (from Portkey): Semantic cache with tunable similarity + TTL + namespaces (cosine, config-driven).
• Reliability / failover (from Cursor): Typed error taxonomy (409 agent_busy, 410 stream_expired) + HMAC-signed statusChange webhooks + resumable streams.
• Governance / data policy (from DigitalOcean / Cursor): Explicit, enforced no-training + zero-retention posture stated plainly (DO), ZDR on-by-default and enforced (Cursor).
• Decision rationale (from BrainstormRouter): /v1/explain returns a real decision trace: selection_reason, priced alternatives_considered, rejection reasons — WHY, not just which.

Worst Practices To Avoid

• Routing intelligence (OpenRouter / Portkey): Routing config smeared across body objects or headers (models[], provider{}, x-portkey-config) — powerful but caller-coupled and hard to reuse.
• Inline cost attribution (Portkey / DigitalOcean): Cost lives only in a dashboard (Portkey) or is attributed out-of-band (DO) — you cannot budget on what you cannot read inline.
• Caching (OpenRouter / DigitalOcean): No own-side semantic cache (OpenRouter) or prefix/KV-only (DO); Portkey paywalls semantic behind Enterprise — so it rarely fires for most users.
• Reliability / failover (all four): None documents a true circuit-breaker (retries + fallback ≠ breaker), and failover hops are hidden from the caller — degradation is invisible until it is an outage.
• Governance / data policy (OpenRouter): Own-service data stance left ambiguous — provider policy "has no bearing on OpenRouter's own policies" with no detail. User-hostile opacity.
• Decision rationale (all four competitors): The single biggest shared gap: every competitor discloses which model served, none exposes a per-request 'why this one' trace.

Lockstep Checklist

• [x] Harness: scripts/bench-routers.ts, scripts/router-capability-probes.ts, scripts/router-competitive-synthesis.ts.
• [x] Core (tested): src/router/intelligence/router-benchmark.ts (+disclosure), capability-probes.ts.
• [x] Master Record: docs/reviews/2026-08-26/router-competitive-ledger.json.
• [ ] Follow-ups: the high-impact recommendations above become tracked BR work.