A 30s timeout was turning healthy models into 91-second 503s

2026-08-29

routermodel-executorobservabilityguardrails

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["POST /v1/chat/completions", "POST /v1/observability/destinations", "PUT /v1/guardrails/config"] sdk_methods_updated: ["DestinationCreateParams.enabled (ts)", "add_destination enabled (py)"] mcp_tools_updated: ["none"] ---

What We Built

Three defects found while developing against the live API, all of which made the gateway lie about whose fault a failure was.

The 91-second 503. A long generation on a healthy reasoning model was aborted by our own 30s execution budget, retried twice more against the _same endpoint with the same budget_, and surfaced ~91s later as 503 model_unavailable — advising the caller to switch models. Reproduced live against deepseek-v4-pro: 91.57s. The same model answers a short prompt in 2.0s. Nothing was unavailable; we simply did not wait, then waited three times.

Guardrail config 500. PUT /v1/guardrails/config documents enabled as the only required field, and its own handler validates exactly that. It then forwarded the partial body to a schema requiring mode and providers, so the documented example — { "enabled": true } — returned a 500.

Observability enabled: false ignored. The create schema had no enabled field (only the update schema did), so zod stripped it before the handler, which hardcoded enabled: true. Adding any destination also force-enabled tenant-wide broadcast. Staging a destination started broadcasting to it.

Why It Matters

Each one sends the caller to fix the wrong thing. model_unavailable with a switch_model hint is actively misleading when the model is healthy — the developer changes models, sees the same 91s failure, and concludes the gateway is broken. A 500 on a malformed body tells a client to retry something that can never succeed. And a destination that ignores enabled: false is a safety control that silently does the opposite of what was asked.

The retry amplification is the sharpest edge: a 30s budget produced a 91s failure, so the timeout the operator configured was not the timeout the caller experienced, and every timeout cost 3× the provider quota it should have.

How It Works

The execution budget now scales with the requested output length, because generation time tracks output tokens rather than being one flat number for a 32-token reply and a 6000-token essay:

resolveNonStreamingTimeoutMs(maxTokens) = min(90_000, 60_000 + maxTokens * 25);

The ceiling is set by Cloudflare, not the ALB — a correction found by testing the fix in production rather than assuming. api.brainstormrouter.com is proxied, and Cloudflare's origin-response timeout (100s by default) fires before the ALB's 180s idle timeout: the first post-deploy run of the repro returned an opaque Cloudflare 524 at ~125s, which is a worse caller experience than the 503 it replaced. 90s keeps our typed 504 model_timeout ahead of Cloudflare's 524 under every plan.

Raising the budget requires raising Cloudflare's proxy_read_timeout first, then the ALB idle timeout. A generation that genuinely needs longer than this cannot complete non-streaming through the proxy chain at all — which is exactly why the recovery hint points at stream: true.

Retry semantics now distinguish _our_ abort from an upstream 504. Both carry status 504, but they need opposite handling, so ModelExecutionError gained an isClientTimeout flag:

isRetryable → false for a client timeout. The retry would carry an

identical request and budget, so it fails identically.

isFallbackCandidate → true, listed explicitly. A timeout on this

endpoint says nothing about a faster one, so auto-routing still cascades.

A genuine upstream 504 keeps retrying exactly as before.

Finally, timeouts surface as model_timeout with a recovery hint pointing at the two things that actually help — stream: true (a 300s budget, bounded only by the connection phase) or a lower max_tokens — instead of model_unavailable / switch_model.

The status stays 503, not the semantically-correct 504, for a reason worth recording: Cloudflare replaces the body of an origin 502/504 with its own branded error page unless Origin Error Page Pass-thru (Enterprise) is enabled. Shipping 504 was measured live and reached the caller as content-type: text/plain, error code: 504, no x-br-* headers, and no recovery envelope — strictly worse than the misleading 503 it replaced. 503 passes through intact, so the caller actually receives the remedy. On a Cloudflare-proxied origin, an error contract that uses 502 or 504 does not exist as far as the client is concerned.

The Numbers

Worst-case timeout failure: 91.6s → ~30s sooner, and 2 wasted upstream

calls per timeout eliminated.

Non-streaming budget: max_tokens unset → 60s; 1000 → 85s; 1200+ → 90s (capped).
Full unit suite green: 1040 files / 10053 tests, pnpm check clean.
14 new regression tests pinning the budget curve, the ALB ceiling, and the

retry/cascade split.

Competitive Edge

Routers that treat every 504 as retryable turn one slow provider into three, and every timeout into a multiple of the configured budget. Distinguishing "the provider timed out" from "we stopped waiting" is what lets BR retry the first, cascade the second, and tell the caller the truth about which happened — with a recovery contract that names the fix rather than guessing at it.

Lockstep Checklist

[x] API Routes: src/api/capabilities/ updated; docs/openapi.yaml

regenerated via scripts/merge-openapi.ts.

[x] TS SDK: DestinationCreateParams.enabled added and documented.
[x] Python SDK: add_destination docstring documents enabled.
[x] MCP Schemas: no tool surface changed.
[x] Master Record: no new capability introduced.

Known Follow-ups

The 90s ceiling is bounded by Cloudflare's proxy_read_timeout. Generations

legitimately longer than that must stream; there is no non-streaming path for them through the proxy chain by construction. Raising it is a Cloudflare zone change first, an ALB change second, and only then a constant in this repo.

timeoutMultiplier (model-router-request.ts:169) is still a no-op unless

config.retry.timeoutMs is explicitly set, because the multiplier is only applied to a configured base. The reputation-driven multiplier plumbed from _reputationTimeoutMultiplier therefore does nothing in the default config.