2026-05-20-phase-8-3-contract-verifier

2026-05-20 — Contract verifier (Phase 8.3)

Summary

Five deterministic verifiers for the contract-aware-routing arc. Each takes a contract verdict (from Phase 8.1's detectContract) and a model response and returns passed | failed | not_applicable.

Verifier	Verdict.kind	What it checks
JSON parse	json_object / json_format	Body parses as JSON (single markdown fence stripped first)
Tool args	tool_args	Each tool_calls[].function.arguments parses as JSON
Exact integer	exact_integer	Body content is a bare integer (no trailing prose)
Exact string	exact_string	Body matches one of the quoted strings in the prompt
Enum	enum	Body matches one of the "one of: X, Y, Z" values in the prompt

What's intentionally NOT here

• No arithmetic evaluator. The original Phase 8.3 plan included a side-evaluator for prompts like (17*19)+23 to catch wrong-integer answers. Both available paths to evaluate arithmetic in JS carry risk (dynamic code eval is rejected by the codebase's security hook; a hand-rolled recursive-descent parser is worth building but not in this PR). Wrong-integer responses where the model returns a valid-but-incorrect number will be caught downstream by the Phase 8.4 reward path: per-model task-class competence will demote models that repeatedly return parseable-but-wrong integers.
• No model verifier. Phase 9+ if we ever build it.
• No routing decisions. Downstream of the verdict.
• No side effects. verifyContract is pure.

Why it's a separate module from contract-detect

The detector knows WHAT contract the caller declared. The verifier knows WHETHER the response satisfied it. Keeping them separate lets the Phase 8.2 cache-skip path use only the detector (cheap; decision is "skip cache or not"), while the Phase 8.4 reward path uses both (verifier output drives bandit V2 reward).

API surface

verifyContract({
  verdict,   // ContractVerdict from detectContract()
  response,  // Model response in OpenAI completion shape
  messages,  // Optional — needed for exact_string and enum to extract references
}): {
  status: "passed" | "failed" | "not_applicable",
  reason?: string,
  kind?: ContractKind,  // echoed from the verdict
}

not_applicable fires when:

• The verdict says no contract was detected
• A verifier needs reference data it can't find (e.g., enum kind but no "one of: …" list in the prompt)

Callers should treat not_applicable as "skip" — same downstream behavior as no contract.

Verification

$ npx vitest run --config vitest.unit.config.ts src/router/contract-verifier.test.ts
✓ 35 passed
$ pnpm tsgo
✓ no errors

Coverage:

• All 5 verifiers (passing + failing + edge cases)
• Dispatcher: not_applicable on missing verdict
• not_applicable on missing reference data
• Fenced JSON passes (Phase 2 symmetry)
• Tool args with malformed JSON fails
• Integer with trailing prose fails ("346 because…" → fail)
• Enum is case-insensitive; supports "or" / "and" separators

Files

• src/router/contract-verifier.ts (NEW, ~280 LOC)
• src/router/contract-verifier.test.ts (NEW, 35 tests)

No wiring changes — this is a pure module. Phase 8.4 will wire it into the bandit reward path.

Lockstep

• [x] No API surface change.
• [x] SDKs unaffected.
• [x] MCP tools unaffected.
• [x] Ship-log (this file).

Plan reference

Phase 8.3 of the contract-aware-routing arc. Stacks on Phase 8.1+8.2 (PR #380, in flight at commit). Phase 8.4 (reward hook) wires verifier output into evaluateOutcomeV2. Phase 8.5 (MCP gating) uses the verifier to gate tool calls.