Load shedding: 100-VU 504 cascade → cheap 503 + Retry-After at the edge

2026-05-22

What We Built

In-process load shedding for the BrainstormRouter API. The middleware tracks concurrent in-flight requests per Node.js process; when the count exceeds a configurable ceiling (default 80, env BR_MAX_INFLIGHT_REQUESTS), it returns HTTP 503 with Retry-After: 2 and a structured code: "load_shed" error envelope — instead of accepting work that will sit in the event-loop queue and time out at Cloudflare's edge (~100s).

Health/readiness/ops/metrics paths bypass the shed so liveness probes always report the truth (which is the signal autoscaling needs to add capacity).

The in-flight count, high-water mark, and total-shed count are surfaced on /v1/ops/status.capacity so operators can see whether the gateway is backpressuring without leaving the API.

Why It Matters

The 2026-05-22 k6 100-VU load test produced a 32.19% error rate, every single failure HTTP 504 from Cloudflare — even /health failed 22.7%. Root cause: a single ECS task accepts as many concurrent connections as the kernel allows, then serializes them through the event loop. Slow requests saturate the loop, fast ones (health checks) back up behind them, Cloudflare times out the lot.

A 504 cascade looks like an outage to consumers. A 503 with Retry-After looks like backpressure. Agent SDKs and clients with retry logic handle the latter automatically; the former triggers escalation.

This is also the kind of failure that hides in load tests below the knee. At 50 VU the gateway returned 3.66% errors. At 100 VU it returned 32%. Without explicit backpressure the failure mode is non-linear; with it, the response is graceful and observable.

How It Works

// Middleware order in src/api/server.ts:
app.use("*", bodyLimitMiddleware()); // Reject oversized payloads (cheapest)
app.use("*", loadShedMiddleware()); // Reject when overloaded (also cheap)
app.use("*", requestIdMiddleware); // Now we're committed; track this one

Implementation: a module-level counter (safe because Node's event loop is single-threaded). Increment on entry, decrement in finally so the count survives handler exceptions. Path-prefix allowlist for health/ops endpoints so they're never shed:

const NEVER_SHED_PREFIXES = ["/health", "/ready", "/metrics", "/v1/ops/status", "/.well-known/"];

if (inflight >= ceiling) {
  c.header("Retry-After", String(retryAfter));
  c.header("X-BR-Load-Shed", "1");
  return c.json(
    {
      error: {
        message: `Gateway temporarily over capacity (in-flight ${inflight}/${ceiling})...`,
        type: "service_unavailable",
        code: "load_shed",
        recovery: { retry_after_seconds: retryAfter, retryable: true },
      },
    },
    503,
  );
}

Operators can tune the ceiling per ECS task size via the BR_MAX_INFLIGHT_REQUESTS env var. Default 80 matches the empirical knee between 50 and 100 VU on the current task definition.

The Numbers

100-VU k6 baseline (pre-shed, 2026-05-22):

Metric	Value
Error rate	32.19%
Failures	235/730
Status code mix	100% HTTP 504
/health error rate	22.7%
p95 latency	27.84s
Sustained RPS	5.70

Expected post-shed behavior under the same 100-VU load:

• ECS task accepts up to 80 in-flight, sheds the rest with 503 + Retry-After
• /health continues to return 200 (bypass list)
• Cloudflare 504 count drops to ~0 (event loop is no longer starved)
• ECS target tracking gets a clean overload signal via the 503 count metric

Validation under load happens after deploy (separate session).

Competitive Edge

Portkey, Helicone, OpenRouter, and LiteLLM all let the underlying runtime's default behavior pass requests through unbounded; the failure mode at peak load is opaque 5xx with no Retry-After or code field. BR's code: "load_shed" + recovery.retry_after_seconds gives agent consumers a deterministic retry strategy without a human in the loop.

Lockstep Checklist

• [x] API Routes: No new routes. /v1/ops/status adds capacity.inflight_requests, capacity.inflight_high_water, capacity.total_shed (additive).
• [x] TS SDK: No SDK method changes — new fields are read-through.
• [x] Python SDK: Same.
• [x] MCP Schemas: No tool changes.
• [x] Ship Log: This entry.