GA P0 bundle: dashboard members can't mint admin keys, cached tokens billed once, alarms reach a human

2026-09-22

LOCKSTEP TRACEABILITY MATRIX --- api_endpoints: [ "PUT /v1/admin/tenants/{id}/plan (new, platform operator)", "PUT /v1/billing/plan (now platform-operator only; tenants get 403 contact_support)", "POST /v1/ops/circuit/reset (now platform.admin)", "POST /auth/api-keys (optional role, capped at the caller's membership)", ] sdk_methods_updated: ["admin.setTenantPlan() (TS)", "admin.set_tenant_plan() sync + async (Python)"] mcp_tools_updated: ["none"] ---

What We Built

A GA-readiness audit on 2026-09-22 found six production defects. None were visible from the outside, and every one was confirmed in source before this change was made.

1. Privilege escalation. RBAC middleware is mounted on /v1/* only, so the

permission field on every dashboard (/auth/*) capability was never enforced. POST /auth/api-keys hard-coded scopes: ["admin"], which meant any tenant _member_ could mint a full-admin API key. The capability adapter now enforces each dashboard capability's permission against the caller's membership role. Writes are denied by default: members keep reads, invoke/feedback actions, and an explicitly reviewed set of day-to-day writes (own keys, prompts, tasks, memory, workspace files). A registry test snapshots that member-reachable set, so widening it is a deliberate review item. Dashboard keys never outrank their minter: members get developer keys, and owners and admins may choose a role.

1. Free self-upgrade. With no payment integration, PUT /v1/billing/plan let any

config.write key set its own tenant to enterprise. Plan changes are now platform-operator only, via the new PUT /v1/admin/tenants/{id}/plan (with dry_run).

1. Operator permissions no role could hold. admin.* and platform.admin were granted

by no RBAC role, so /v1/admin rate-limit, invite-delete and XDR always returned 403. RBAC now satisfies them from the platform-admin allowlist, which is read only in src/security/platform-admin.ts. The cross-tenant circuit-breaker reset moved from router.admin (any tenant admin) to platform.admin. Suspending a tenant now evicts its cached key resolutions instead of letting traffic run for up to 30s.

1. Metering.

• OpenAI and Gemini include cached tokens in the input count, yet computeCost billed

them at the full input rate _and_ the cache rate. Extractors now flag inputIncludesCacheRead, so each token is billed once.

• DeepSeek cache hits (prompt_cache_hit_tokens) are now recognised.
• Gemini thinking tokens (thoughtsTokenCount) were unbilled; they are now billable output.
• Anthropic stream output was summed, but message_delta usage is cumulative per the docs.
• Eleven conversions in eight files used 1e8 instead of the registry's 1e9: image and audio

rows were stored 10x low, admin spend was shown 10x high, and the swarm estimate was 10,000x low. A source-scan test now blocks 1e8 conversions.

• Usage rows now persist cache_read_tokens/cache_write_tokens (always 0 before), plus

reasoning_tokens in metadata.

• A stream that ends without the provider's usage item (mid-stream failure, disconnect,

sever) now records a flagged estimated row instead of nothing.

• DeepSeek silently ignores max_completion_tokens, so a caller's max_tokens never

reached DeepSeek. Only OpenAI gets that field now; every other OpenAI-compatible provider gets max_tokens.

1. Alerting. The production alerts topic's only subscriber was the alert-receipt Lambda,

so no human was notified. infra/monitoring-stack.yml gains AlertEmail. br-prod-soak-test-failure had been in ALARM since 2026-08-28 because the soak suite hard-coded deepseek/deepseek-chat (pruned from the catalog) and openai/gpt-4.1-nano (403: the soak tenant holds no OpenAI key). The suite now picks the cheapest model the tenant can actually run from /v1/catalog/runnable, pins a configurable current model, and runs once per slot across the fleet (Redis claim) instead of once per task. /v1/catalog/runnable stopped counting platform env keys, which tenant traffic never uses, so it now agrees with the completion path.

1. A test runner that looked green. With vmForks, vitest exited 0 after its workers ran

out of memory and printed no summary, so pnpm test reported success while the main unit group had crashed. The runner now verifies each group's JSON report, and a missing or unsuccessful report fails the run. It also caps vm worker memory (--vmMemoryLimit), which removed the crash. Three test isolation bugs that the crash had hidden are fixed: leaked agent-dir env, leaked flag state, and fixed-millisecond timing assertions.

1. Prices on served models were corrected against the providers' rate cards

(separate commit): claude-sonnet-5 dropped from $3/$15 to $2/$10, gpt-5.6-sol from $5/$30 to $4/$20, cache-read rates changed on gemini-3.5-flash, 3.1-flash-lite and grok-4.6, and deepseek-v4 moved to the off-peak list price.

Adversarial review (clean-context agent) and what it changed

• Fixed: cached tokens were discounted even when the endpoint had no cache-read price,

which made them free on groq gpt-oss and kimi. Now they are discounted only when priced.

• Fixed: the stream fallback wrote its row but never committed spend to the budget

ledger. It now commits the estimate. If the provider's measured cost was already committed before the stream failed, the row carries the measured figures and no second commit happens.

• Fixed: the fallback now honours Zero Completion Insurance (no visible output means no

charge, flagged insurance_waived).

• Fixed: the output-cap field is keyed on the wire API (openai-compatible → max_tokens)

instead of the provider name, so Copilot and BYOK OpenAI-API endpoints keep max_completion_tokens.

• Fixed: pnpm test -- also verifies its report. A missing report directory or a

report with zero suites now fails.

• Fixed (lockout guard): users with no membership row resolve with no membership role, which

the gate treats as a plain member. The legacy users.role = "admin" is now carried over, so a legacy tenant owner is not locked out of their own settings.

• Open (low, carried to Wave 2):
• Perplexity streams (supportsStreamUsage:false) still record $0.
• Quality scoring counts Gemini thinking tokens toward conciseness.
• Moonshot's top-level cached_tokens is not read.

Why It Matters

For a governance gateway these are GA blockers: a member-to-admin escalation, invoices that overcharge cached traffic, and a production alarm that told no one for 25 days.

How It Works

• Gate: evaluateDashboardPermission(permission, membershipRole) runs in makeRouteHandler

for auth: "supabaseJwt" capabilities, before admission and parsing.

• Cost: computeCost bills input - cacheRead at the input rate when

usage.inputIncludesCacheRead. input is still echoed to clients unchanged as prompt_tokens, preserving OpenAI-compatible semantics.

Lockstep Checklist

• [x] API Routes: capabilities updated (system/admin.ts, system/billing.ts,

system/ops.ts, auth/api-keys.ts, system/catalog.ts); docs/openapi.yaml regenerated.

• [x] TS SDK: admin.setTenantPlan().
• [x] Python SDK: admin.set_tenant_plan() sync + async.
• [ ] MCP Schemas: none. No operator plan tool yet (admin MCP tools cover

list/approve/reject only), tracked for the Wave 4 control-plane work.

• [ ] Prod verification (post-deploy):
• Member JWT → POST /auth/api-keys returns a developer key; asking for role:"admin"

→ 403 insufficient_role.

• Tenant key → PUT /v1/billing/plan → 403 plan_change_requires_operator.
• Tenant admin key → POST /v1/ops/circuit/reset → 403.
• DeepSeek completion with max_tokens:8 → completion_tokens ≤ 8.
• A cached OpenAI request's usage_events row has non-zero cache_read_tokens.
• br-prod-soak-test-failure returns to OK with TestsFailed=0.
• The SNS topic lists a confirmed email subscription.