One missing env var was 503-ing every config write in production

2026-08-29

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["none"] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---

What We Built

Nothing new — this removes a production defect that had been misdiagnosed once already, and closes two monitoring gaps found alongside it.

Every tenant-scoped config write in production was returning 503 write_not_persisted. POST /v1/observability/destinations and PUT /v1/guardrails/config both refused to persist, correctly declining to report a false success. The cause was a single unset environment variable.

getWorkspaceStorageMode() (src/infra/workspace-storage.ts) defaults to "file" when WORKSPACE_STORAGE is unset — there is no inference from DATABASE_URL being present. File mode makes boot-db.ts:34 (dbBoot.available && dbBoot.workspaceStorage !== "file") skip its entire block, so setGlobalConfigServiceManager() at boot-db.ts:61 was never called. The request path then found a null manager and — correctly — refused to claim the write had landed.

Why It Matters

The blast radius was wider than the two 503ing endpoints. The same skipped block also silently drops the DB session store, the Postgres workspace store, tenant config pre-warm, and the DB-backed router paths at boot-router.ts:85/89/113. The service looked completely healthy throughout: /health returned db:true redis:true, both ECS tasks were HEALTHY, and ALB targets were green. Only a write attempt revealed it.

This was misdiagnosed on 2026-08-29 as a bundler chunk-splitting bug. Commit ff547ee re-anchored the config manager singleton on globalThis via a Symbol.for slot, reasoning that mixed static/dynamic imports had split the module across chunks with separate bindings. That change is harmless and arguably more correct, but it was never the cause: boot never reached the setter under any bundling. The lesson is that the hypothesis was never tested against the boot log — gateway: DB-backed ConfigServiceManager initialized was absent the whole time, which falsifies "boot set one copy" directly.

How It Works

Three variables were added to the task definition. The deploy workflow reads the _current_ task definition and only replaces the image plus BR_BUILD_* attestation (SECRET_FILTERS is empty), so these persist across future deploys rather than needing to be re-applied.

{ name = "WORKSPACE_STORAGE",  value = "db" }
{ name = "BR_SOAK_TENANT_ID",  value = var.soak_tenant_id }
{ name = "BR_SOAK_API_KEY",    valueFrom = "<routing-secret>:BR_SOAK_API_KEY::" }

Diagnostic for next time: if gateway: DB-backed ConfigServiceManager initialized is missing from the boot log, WORKSPACE_STORAGE is unset. That single line distinguishes this failure from every other config-write problem.

br-prod-soak-test-failure had been in ALARM since 2026-08-28 10:48, paging SNS. It was not reporting a failing soak test — it was reporting that no soak test existed. BR_SOAK_API_KEY gates the cron seed at server-cron.ts:383, so nothing ever published BrainstormRouter/SoakTest TestsFailed, and the alarm is treatMissingData = breaching. A dedicated developer-role key (not an admin key) now backs it, covering exactly the paths the suite hits: /health, /v1/models, /v1/ops/status, /v1/chat/completions.

The Numbers

• Config write success: 0% → 100%, verified live on both surfaces

(201 + read-back for observability, 200 "enforced":true for guardrails).

• write_not_persisted events since deploy of task definition :99: 0

(last occurrence 19:37:18Z, deploy at 19:46Z).

• Deploy was config-only — identical image digest to :98 — which

isolates the fix to the environment change.

• Doc drift gate: 4 failures → 0. README claimed 45 models (catalog has 40);

docs/concepts/mcp.mdx claimed 116 MCP tools (manifest has 120). This gate had been failing CI on ff547ee while that commit deployed anyway.

Competitive Edge

The failure mode is the point: BR returned 503 write_not_persisted rather than a cheerful 200 over a write that went nowhere. A gateway that quietly accepts guardrail config it never applied is worse than one that refuses, because the operator believes enforcement is active when it is not. The write_not_persisted contract — and the test at observability-persistence.test.ts:110 that pins "never a false 2xx" — is what turned a silent governance gap into a loud, greppable one.

Lockstep Checklist

• [x] API Routes: no src/api/routes/ changes — infrastructure + docs only.
• [x] TS SDK: not applicable, no API surface change.
• [x] Python SDK: not applicable, no API surface change.
• [x] MCP Schemas: not applicable, no tool changes.
• [x] Master Record: no new capability introduced.

Known Follow-ups

Found during this work, not fixed here:

1. PUT /v1/guardrails/config returns 500 on a schema-invalid body

(missing mode / providers) where it should return 400. The error envelope advises retry for what is a permanent client error.

1. POST /v1/observability/destinations ignores enabled: false in the

request body and creates the destination enabled, flipping the tenant's broadcast flag to true as a side effect.

1. Perplexity ingestor has been failing hourly with

HTTP 401 insufficient_quota — a billing issue, not code. The GTM discovery pipeline is down until the account is funded.

1. mTLS is misconfigured for an ALB-fronted deployment:

gateway.tls.mtls.behindAlb is OFF, so issued certs cannot authenticate downstream.