Production hardened, stress-tested, and audit-reviewed — infra idle under 90 req/s flood, $0/mo added

2026-08-26

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["GET /v1/explain/{request_id}"] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---

What We Built

Three linked pieces of work to get production ready for real load, then prove it: (1) brought the live fargate-minimal stack up to a codified standard, (2) ran a full stochastic review against a fully-keyed tenant pool, and (3) ran a k6 stress ladder to find the binding constraint.

Hardening. The live production values existed only in tfstate — a bare terraform apply would have destroyed the ALB. We added scripts/ops/reconstruct-fargate-tfvars.py (rebuilds the tfvars from state) and codified the drift that had accumulated live: RDS deletion-protection, 7-day PITR, ECS-exec, and ignore_changes on desired_count/task_definition/ minimum_healthy_percent so terraform never fights the CloudFormation autoscaler or the deploy workflow. terraform plan now reports No changes. Added AWS Budgets (monthly $150, daily $15) to the monitoring CFN stack on the existing SNS→Lambda alert path, and rewrote verify-alarms.sh to match the nine live alarms (9/9 green) and the fargate-minimal README as a production runbook documenting the three-owner split (terraform / CloudFormation / ecs-deploy).

Review. Provisioned 11 full-tier tenants (one invite code minted via ECS-exec into the gateway task) with real DeepSeek+Groq keys and per-key budget caps, then ran all ten personas plus the eleventh auditor against held build 2c53613. This was the first audited wave where router-scientist, cost-owner, data-protection, agent-consumer and claim-auditor exercised real routing, cost and governance instead of sandbox stubs.

Stress. A k6 ladder (baseline → 50 → 75 → 100 VU, plus a 12-tenant multi-key run) with a live CloudWatch telemetry watcher, inside an announced no-deploy window.

Why It Matters

The stress test answered the question the whole "forty-dollar" proposal was built around — _is the micro datastore tier the bottleneck?_ — and the answer is no. Under a 90 req/s single-tenant flood, the binding constraint is BR's own per-tenant in-flight concurrency quota (15), which sheds excess load gracefully with a retryable 503 load_shed + retry-after. The datastores never noticed: Postgres peaked at 21 of ~85 connections, Redis had zero evictions, the service never scaled past 2 tasks, and it was fully healthy the instant load stopped. Total provider spend for ~24,000 attempted requests: $0.006 — the rate limiters protected the budget as effectively as they protected the infra.

The practical conclusion: the utilitarian steady state holds. Nothing broke that buying bigger hardware would fix, so the permanent monthly cost change is $0.

How It Works

The convergent review finding (7 of 10 personas hit it independently) was that /v1/explain's selection_reason named a model the tenant had no key for (e.g. openai/gpt-4.1-nano) while the served model was deepseek/*. Root cause: a fresh tenant has no recorded provider-unavailability, so the auto-selector picks the top-scored (unreachable) model, then a downstream byok_cascade stage remaps to a reachable one — but the frozen auto_select reason string was never updated. The fix chains the cascade's own reason onto selection_reason only when the served model differs from the one the reason names:

capability-matched: openai/gpt-4.1-nano (tier≥economy) → BYOK cascade: tenant has no openai key, using deepseek

When they match (the common case) the reason is untouched. Verified live post-deploy: consistent picks stay clean, remapped picks now carry the full provenance.

The Numbers

• Stress: peak 21 DB connections (of ~85), 0 Redis evictions, 2 tasks (no scale-out), 0

infra alarms; 503 load_shed backpressure at the per-tenant 15-in-flight quota; instant recovery; $0.006 total spend vs a $40 ceiling.

• Review: audited mean 7.8 (tenant-security 9 accepted; router-scientist 9→7 corrected by

the auditor for an unverifiable epsilon-exploration claim); 1 convergent bug fixed live, 19 findings ledgered at docs/reviews/2026-08-26/ledger.json.

• Competitive re-run: BR holds 2 sole-leader dimensions (reliability/failover, decision

rationale), 0 gaps, 4 parity; 51/52 bench tasks pass, p50 1586 ms. Ledger at docs/reviews/2026-08-26/router-competitive-ledger.json.

• Cost: AWS baseline unchanged at ~$121/mo; permanent change $0.

Competitive Edge

The stress test is itself a differentiator: BR shed a single-tenant flood with a _typed, retryable_ load_shed response carrying retry_after_seconds, not an opaque 500 or a silent drop — the same "prove what happened" posture as the decision trace. No surveyed competitor (Portkey, OpenRouter, Cursor, DigitalOcean) documents a per-tenant in-flight quota with a machine-actionable backpressure contract.

Lockstep Checklist

• [x] API Routes: src/api/routes/explain-request.ts — selection_reason provenance fix (behavior of an existing route; no new route).
• [x] TS SDK: none — response-body/field-value change, no method or type change.
• [x] Python SDK: none — same.
• [x] MCP Schemas: none — /v1/explain shape unchanged.
• [x] Master Record: no capability added; existing router.explain behavior corrected.

Open Follow-ups (from the stress test + review)

• load_shed returns 503, tripping br-prod-high-5xx-rate and counting against the 5xx SLO.

Intentional backpressure arguably should be 429, so genuine server errors stay distinguishable. (The alarm fired during the ladder as expected — no deploy was in flight, so no rollback; it self-cleared after load stopped.)

• True infra ceiling is unmeasured: a single load-gen IP is capped by per-IP edge rate limiting

(Cloudflare), so reaching the gateway's real compute ceiling needs a CF allowlist or a distributed source. Deliberate follow-up, not blocked infra.

• The full 19-item review backlog (SDK /v1 baseURL, streaming chunk order, audit/verify receipt

check, cost-attribution of probe traffic, cosign ECR auth, dashboard console errors, …) lives in the ledger.