2026-05-20-bandit-reward-v2

2026-05-20 — Bandit reward V2: content-aware outcome classification (OpenClaw Phase 4)

Summary

OpenClaw Phase 4 of 7. Adds a feature-flagged content-aware reward signal for the Thompson-sampling bandit. The previous V1 reward derived validity from scoreValidity(), which checked structural shape (hasContent, errorCode, tool JSON) but did NOT distinguish:

HTTP 200 + content: null + no tool_calls → counted as success
HTTP 200 + content: "" + finish_reason: "length" → counted as success
HTTP 200 + response_format: json_object requested + fenced/free-text body

→ counted as success

These were the exact failure modes that polluted Thompson posterior weights: a model that returned nothing at zero cost looked better than one that actually produced output. Phase 1 (PR #365) stopped the response from ever reaching the client; Phase 2 (#366) stopped it from being cached. Phase 4 removes its imprint on the bandit signal.

Why it matters

After Phase 1 + 2, the failure path is:

Executor sees content: null → throws ModelExecutionError(502, "null_content")
Cascade engages → next-tier model tried
Cascade succeeds OR exhausts → response written
Reward recorded ← this is the V1 blind spot

If the executor's null-content detection fires, the bandit never sees a "successful" null-content row. But:

If a future code path bypasses the executor check (e.g., a streaming

bug, a new provider adapter), V2 catches it.

If the model returns _structurally valid_ content but violates the

json_object contract, the executor lets it through (correctly — it's a successful response) but the bandit should still demote.

V2 is defense-in-depth, not redundancy.

How V2 classifies

evaluateOutcomeV2({ response, request, httpStatus, executorNullContent }) returns one of 7 classifications:

Classification	Validity	When
`upstream_error`	0	HTTP 5xx
`null_content`	0	content:null, no tool_calls (Phase 1 predicate)
`empty_truncated`	0	content:"" + finish_reason:length
`json_invalid`	0.3	response_format=json_object + body fails JSON.parse
`refused`	0.2	start-of-response matches refusal pattern
`tool_use_success`	1.0	content:null + tool_calls present (correct OpenAI shape)
`success`	1.0	usable content, contract satisfied

The validity score feeds into computeRewardV1 as the V1 validity slot, preserving cost/latency normalization. V2 is purely a validity-input sharpening; it doesn't change the reward-version contract or weights.

Feature flag

Off by default. Set BR_BANDIT_REWARD_V2=1 on the ECS task definition to enable. When off, V1 is the sole reward source and V2 is not computed.

When on, V2's validity only overrides V1 when more pessimistic:

const effectiveValidity = v2.validity < v1.validity ? v2.validity : v1.validity;

This prevents V2 from optimistically upgrading V1's partial-credit signals (e.g., V1 might score 0.65 for a tool-shape edge case V2 doesn't model; V2's 1.0 success wouldn't replace that).

Wiring

Three reward call sites in model-router.ts now route through a new private helper maybeApplyRewardV2():

L692 — non-streaming success path → shadowComparator
L1634 — scoreAndRecordValidity() → recordQuality (audit chain)
L1786 — streaming success path → shadowComparator

The helper is fail-soft: any error in V2 evaluation falls back to V1 silently with a warning log. V2 is an optimization, never a correctness gate.

Files

src/router/bandit-reward-v2.ts — NEW (228 LOC). Predicate + flag.
src/router/bandit-reward-v2.test.ts — NEW (31 tests).
src/router/model-router.ts — wired at 3 call sites + helper added.

Tests

$ pnpm test:fast -- src/router/bandit-reward-v2.test.ts
✓ 7526 passed, 0 failed (31 new)

Coverage:

All 7 classifications produce the expected validity
JSON-contract check accepts both raw and fenced JSON
JSON-contract check fires only when caller requested json_object
Refusal detection is start-anchored (not mid-response)
Tool-use responses with content:null are NOT penalized
Feature flag respects multiple truthy strings + env override
Priority ordering: upstream_error > executor-flagged > null > json > refusal

Deferred to follow-up

bandit_reward_v2_delta metric emission (needs metric registry plumbing)
24-hour canary deploy with V2 enabled on one of two ECS tasks
Comparison dashboard for V1 vs V2 reward distributions

These are runtime/ops concerns, not code changes; tracked separately in the OpenClaw stress-fix plan as Phase 4.6.

Lockstep

[x] No API surface change.
[x] SDKs unaffected (V2 is internal routing optimization).
[x] MCP tools unaffected.
[x] Ship-log entry (this file).

OpenClaw Plan Reference

Phase 4 of .claude/notes/openclaw-stress-fixes-phased-plan-2026-05-20.md. Depends on Phase 1's null-content-detect.ts (#365 merged). Stacks cleanly on Phase 3 (#371). Phases 5–7 to follow.