Home Dashboard GitHub

2026-05-20-invariant-test-layer

2026-05-20 — Invariant test layer (OpenClaw Phase 5)

Summary

Adds a new test category: cross-component invariants. Three invariants land in this PR:

  1. status-implies-content.invariant.test.ts — every HTTP 200 chat completion has usable content (text OR tool_calls).
  2. cache-quality-parity.invariant.test.ts — cached responses uphold the same contract as fresh responses (no null content, no contract violations).
  3. response-waf-no-false-positive.invariant.test.ts — WAF response-phase patterns produce zero critical/high false positives on a 30-sample corpus of legitimate model output.

Why this matters

OpenClaw 2026-05-20 caught three classes of regression that slipped through both unit and e2e tests:

  1. A model returning content: null passed every unit test (the response was OpenAI-shaped) and every e2e test (the gateway returned 200), but violated the OpenAI completions contract from the caller's perspective.
  2. The cache layer returned a previously-stored broken response that satisfied the request-similarity threshold but violated the current request's contract.
  3. A WAF security rule fired warn/block on legitimate model output because the rule's positive set was tuned against user input, not model output.

Each was a missing cross-cutting invariant: a property that must hold across modules, but that no single module's tests checked end-to-end.

This layer pins those invariants explicitly. The tests live in src/tests/invariants/ with a README explaining what an invariant is, when to add new ones, and when NOT to.

How

  • New runner: pnpm test:invariants (config: vitest.invariants.config.ts). Runs only *.invariant.test.ts files. Default mode is offline; live-mode tests are gated on BR_INVARIANT_LIVE=1.
  • PR-level CI: invariants are excluded from test:fast (kept offline + fast for PR-level CI). They run in pnpm test:invariants and will be wired to nightly CI in Phase 6.
  • Each test documents its property + failure mode: the README explains the philosophy; each file's header comment explains what regression it pins.

Coverage

InvariantMock-mode testsLive-mode tests (gated)
status-implies-content123 (one per major provider)
cache-quality-parity92 (free-text + json_object)
response-waf-no-false-positive34 (corpus of 30 + 4 meta-assertions)0

Total: 55 mock-mode tests passing, 5 live-mode tests gated.

Verification

$ pnpm test:invariants
✓ 55 passed, 5 skipped (live-mode), 0 failed
$ pnpm test:fast
✓ test:fast excludes invariant tests (unchanged behavior)

Files

  • src/tests/invariants/README.md (NEW) — what an invariant is + conventions
  • src/tests/invariants/status-implies-content.invariant.test.ts (NEW, 230 LOC)
  • src/tests/invariants/cache-quality-parity.invariant.test.ts (NEW, 215 LOC)
  • src/tests/invariants/response-waf-no-false-positive.invariant.test.ts (NEW, 250 LOC)
  • vitest.invariants.config.ts (NEW) — dedicated runner
  • vitest.unit.config.ts — excludes src/tests/invariants/**
  • package.json — test:invariants script added

Findings from writing the invariants

Two assumptions in the original Phase 5 plan turned out to be wrong, and the invariant suite surfaced them:

  1. Cache admission strips fences before validating JSON. The Phase 5 plan assumed \\\json\n{...}\n\\\`would be rejected from the cache. In fact,cache-admission-gate.tsstrips common markdown fences before JSON.parse — intentional symmetry with the response-WAF fence-strip (Phase 3, #371). Fenced JSON satisfies thejson_object` contract; the invariant now reflects this (and pins the symmetry).
  1. Inline backtick + relative path with ../ legitimately triggers WAF. The plan's corpus included ` ../../scripts/helper.sh as "legitimate" output. WAF (post-Phase-3) correctly flags this because: (a) sh is a shell verb inside backticks → cmdi_backtick, (b) ../ is path-traversal → path_dotdot`. Both fires are correct. The sample was replaced with the same prose without backticks/dotdot.

These are the kind of findings the invariant layer is designed to surface — implicit assumptions about cross-module behavior become explicit through the corpus.

Lockstep

  • [x] No API surface change.
  • [x] SDKs unaffected.
  • [x] MCP tools unaffected.
  • [x] Ship-log (this file).

Deferred to Phase 6

  • Wire pnpm test:invariants into fortress-nightly.yml. Currently runnable but not scheduled.
  • Live-mode tests require a staging tenant API key — Phase 6 will provision and gate it.

Plan reference

Phase 5 of .claude/notes/openclaw-stress-fixes-phased-plan-2026-05-20.md. Independent of Phase 4 (#376). Phase 6 wires this into CI; Phase 7 runs everything against R33.