Six checks that reported success while doing nothing — 27 days of undeployed code, recovered

2026-08-08

infradeploysecuritymedia

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: ["none"] sdk_methods_updated: ["none"] mcp_tools_updated: ["none"] ---

What We Built

Nothing new, again. This is the second consecutive entry whose value is subtraction — and where 0A removed claims the platform could not keep, this removes checks that could not fail.

Production had been serving a 2026-07-11 image for 27 days. Three workstreams were waiting on "the deploy". main was green the entire time. The deploy could not have succeeded: the workflow targeted an ECR repository named brainstorm-router and a cluster named brainstorm-router-production, and the fargate-minimal migration had created brainstormrouter and brainstormrouter-minimal. Every push built an image and died at docker push.

Behind that sat three more failures, each invisible because the one in front of it never let execution reach them. A block of ~15 hardcoded secret ARNs built from ${{ secrets.AWS_ACCOUNT_ID }} — a GitHub secret never set on this repo — rendered as arn:aws:secretsmanager:us-east-1::secret:... and failed at _task start_ rather than at deploy time, so the workflow reported success and the service then could not place tasks. packages/governance-sdk-ts was in the workspace but absent from pnpm-lock.yaml; --frozen-lockfile passed and pnpm prune --prod failed, deep in the Docker build. ZAI_BASE_URL was code-configurable and never configured, so a valid GLM Coding Plan key hit the pay-as-you-go path and returned 429 code 1113 "Insufficient balance" — a 429, not a 401, which reads as an unfunded account rather than a wrong endpoint.

Fixing those exposed a fifth. Terraform pins runtime_platform.cpu_architecture = "ARM64" (Fargate Graviton), and its comment explains why: _"The gateway image is built on Apple Silicon."_ True while images were pushed by hand from a Mac. CI runs on ubuntu-latest, and docker/build-push-action defaults to the builder's native platform — so it pushed amd64 into an ARM64 task definition. Containers pulled, started, and died instantly with exec /usr/local/bin/docker-entrypoint.sh: exec format error, exit 255. ECS surfaced only _"Essential container in task exited"_; the cause was visible only in CloudWatch. This bug was _created by fixing the first one_ — it could not fire while no image ever reached ECR.

The sixth took dashboard signin down and is the purest specimen. supabase-keepalive.yml exists precisely to stop the free-tier auth project from pausing, and it decided liveness from curl's exit status:

code=$(probe || echo "000")          # BUG
if [ "$code" != "000" ]; then echo "Project responding"; exit 0; fi

curl prints %{http_code} — 000 on NXDOMAIN — and exits non-zero, so || echo "000" appended a second value. code became 000\n000, which never equalled the "000" sentinel, so the failure branch and the auto-restore hanging off it were unreachable. The workflow logged Project responding (HTTP 000000) and went green daily while .supabase.co did not resolve and both Google and GitHub signin were down.

Why It Matters

Every one of the six failed in a way that looked like success. That is the single common thread, and it is the same failure the compliance work is built to prevent one layer up: absence of evidence is never evidence of compliance. A green workflow is not a deploy. A 200 is not proof no outbound call happened. A probe that cannot parse its own input is not a health check.

BrainstormRouter's differentiator is governance you can prove — evidence receipts, decision lineage, artifact projection. That thesis is only as good as the platform's own operational honesty. A system whose monitoring reports success on a dead dependency has no standing to sell attestation to anyone else. Today's work is the same principle applied inward.

The customer-visible outcomes: dashboard signin restored, the routing engine deploying again, and the media pipeline no longer about to start rejecting partial uploads.

How It Works

Ownership is now split so there is exactly one writer per concern:

Concern	Owner	Never touched by
Image tag	ECS Deploy workflow	terraform
Env vars, secrets, CPU/memory, roles	terraform	the workflow
Service scale, ALB, networking	terraform	the workflow

The workflow reads the current task definition, replaces only the image, and inherits everything else. Secret injection was removed outright — it was a second, competing source of truth for the task definition, which is what broke task placement.

Two guards now observe the _effect_ rather than a step's exit code. The architecture guard reads the live runtimePlatform and refuses a mismatched image in seconds instead of after a ten-minute stabilize wait:

task definition wants: ARM64 (arm64)   image provides: arm64

The keepalive now swallows curl's exit status inside probe() so exactly one value is ever emitted, and classifies it with an explicit arm whose default case is failure:

classify() {
  case "$1" in
    [1-5][0-9][0-9]) printf 'yes' ;;
    000 | "")        printf 'no'  ;;
    *) echo "::warning::Unparseable probe result '$1'" >&2; printf 'no' ;;
  esac
}

One classify() serves both the initial probe and the post-restore re-probe. They previously disagreed — liveness accepted any response while the restore check demanded exactly 200, and this project's PostgREST root answers the anon key with 401, so a restore that genuinely succeeded still exited 1.

Separately, pnpm audit reported 18 high/critical advisories across 12 packages, one critical (GHSA-23hp-3jrh-7fpw, tar decompression DoS). Most were existing overrides whose floors had gone stale rather than missing entries. Two carry deliberate upper bounds — ajv@8 declares fast-uri ^3.0.1 and markdown-it@14 declares linkify-it ^5.0.0, so an unbounded floor resolves a breaking major. nanoid is scoped rather than global because postcss is on the v3 line and node-llama-cpp on v5.

The package.json#pnpm block was deleted. pnpm ≥10.23 ignores it (packageManager pins 10.23.0), it warned on every command, and it had already drifted from pnpm-workspace.yaml — so a future CVE fix applied there would have produced a clean-looking diff that did nothing. The same failure shape as the other six, sitting in the config that governs security patches.

sharp 0.34 → 0.35 needed real code changes. Its ESM types no longer expose a callable namespace, and failOnError was removed in favour of failOn. The second is not cosmetic: the unknown option is ignored and sharp reverts to its default failOn: "warning", which throws on truncated uploads the media pipeline previously decoded. Verified against 0.35.3 rather than assumed:

truncated to 90% (46795/51995 bytes)
   failOn:"none" -> OK
   default       -> THREW: VipsJpeg: premature end of JPEG image

image-ops.truncated-input.test.ts locks that behaviour, and was confirmed to fail with the old option name before being accepted with the new one.

The Numbers

Metric	Value
Days production served a stale image	27 (image dated 2026-07-11)
Independent silent failures	6 (4 latent, 1 uncovered by fixing them, 1 in a second workflow)
High/critical CVEs cleared	18 across 12 packages, incl. 1 critical
CVEs remaining	2, both pre-existing documented ignores
Unit suite	9360 passing, 0 failures
Task definition	`:10` (rolled back) → `:15` → `:16` → `:17`
Production downtime	zero — the ECS circuit breaker held throughout

Production never went down. The deployment circuit breaker rolled back two failed rollouts correctly and kept :10 serving. The outage was of _change_, not of availability — which is exactly why nothing alerted.

Competitive Edge

Portkey and OpenRouter ship runtime controls; Credo AI governs pre-deployment with no runtime. Everyone in the category is selling assurance. The distinguishing question is not whether a platform has controls — it is whether it can tell the difference between a control that held and a control that never ran.

Six times today, this codebase could not. The fix is not more checks; it is checks whose default branch is failure and whose assertion is about an observed effect — a task-definition revision, a reset uptime, an image's actual architecture — rather than a step's exit code. That discipline is the same one mayDerive / mayPersist / evidence receipts encode for customer data, and it is the honest basis for selling governance to anyone else.

Postscript — why this entry exists

This ledger now holds 266 entries and ~152,000 words going back to 2026-02-15. The conversation transcripts for this project only reach 2026-07-15; everything before that was lost with the machine. The Ship Log convention is the reason the history survived at all.

Entry volume has fallen sharply — 121 entries in March, 88 in May, 19 in June, 7 in July. Entries have grown longer, so the record per session has not thinned, but the practice has. The convention that preserved this project's memory is the one being exercised least. This entry is a deliberate correction to that, on a day whose work would otherwise have existed only in a transcript.

Lockstep Checklist

> _You MUST check these boxes [x] and verify the corresponding files are updated BEFORE committing this log._

[x] API Routes: no src/api/routes/ change — infrastructure, dependencies and media internals only.
[x] TS SDK: n/a — no API surface change.
[x] Python SDK: n/a — no API surface change.
[x] MCP Schemas: n/a — no agent-facing surface change.
[x] Master Record: n/a — no new capability; docs/reference/deployment.md is the authoritative record for this work.