Thompson Sampling

UCB1 → Gaussian Thompson: how BrainstormRouter learns its model leaderboard and consensus rankings from measured performance.

Scope: this page describes the leaderboard learner, not the per-request router. When a request specifies model: "auto", the pick is made by the capability-matched auto-selector (src/router/model-auto-selector.ts) — a required-tier floor plus shrinkage-blended learned quality minus cost, with epsilon-greedy exploration. Live responses report it as X-BR-Selection-Method: capability-matched. A request with an explicit model touches no bandit at all. The UCB1 → Thompson bandit on this page is an async learning signal: it powers the model leaderboard and consensus-voting rankings, and its learned reward statistics inform those surfaces — it is not a per-request routing stage.

Why not static rankings

Most AI gateways rank models by static rules: cheapest, lowest latency, or manual priority lists. Rules don't learn. If model A starts outperforming model B for your specific workload, a static ranking never notices.

BrainstormRouter uses Thompson Sampling, a Bayesian approach to the multi-armed bandit problem, to learn its model leaderboard. The bandit maintains a posterior distribution for each model endpoint and samples from it to balance exploration (crediting underexplored models) with exploitation (ranking proven performers on top).

The full implementation is in src/router/model-bandit.ts.

Algorithm overview

%%{init: {'theme': 'dark', 'themeVariables': {'primaryColor': '#d97706', 'lineColor': '#9494a8', 'primaryTextColor': '#e8e8ee'}}}%%
flowchart TD
    Start([Rank models]) --> Floor{Hard floor<br/>gating}
    Floor -->|"validity < 0.5<br/>(≥10 samples)"| Exclude[Exclude arm]
    Floor -->|"quality < 0.3<br/>(when data exists)"| Exclude
    Floor -->|Pass| Eligible

    Eligible --> Zero{Any arms with<br/>0 samples?}
    Zero -->|Yes| Random[Random pick<br/>among unexplored]
    Zero -->|No| SampleCheck{Total samples<br/>≥ 500?}

    SampleCheck -->|No| UCB1["UCB1 Selection<br/>score = mean + C × √(ln(total)/n)"]
    SampleCheck -->|Yes| Thompson["Thompson Sampling<br/>sample ~ N(mean, var/n)"]

    UCB1 --> Pick([Pick highest score])
    Thompson --> Pick
    Random --> Pick

    style Exclude fill:#7f1d1d
    style Random fill:#1e3a5f
    style UCB1 fill:#4a3728
    style Thompson fill:#2d4a1e

Phase 1: UCB1 (cold start)

When total samples across all arms are below 500, the bandit uses Upper Confidence Bound (UCB1). This is a deterministic algorithm that balances exploitation and exploration:

ucb_score = reward_mean + C × √(ln(total_samples) / arm_samples)

From model-bandit.ts:131-156:

private ucb1Select(arms: BanditArm[], totalSamples: number): BanditSelection {
  // C decays from initialC to minC as total samples grow
  const decayFactor = Math.min(1, totalSamples / 1000);
  const C = this.config.initialC - (this.config.initialC - this.config.minC) * decayFactor;

  let bestArm = arms[0];
  let bestScore = -Infinity;
  let bestBonus = 0;

  for (const arm of arms) {
    const explorationBonus = C * Math.sqrt(Math.log(totalSamples) / arm.sampleCount);
    const ucbScore = arm.rewardMean + explorationBonus;

    if (ucbScore > bestScore) {
      bestScore = ucbScore;
      bestArm = arm;
      bestBonus = explorationBonus;
    }
  }
  // ...
}

The exploration coefficient C decays from 1.5 to 0.5 as total samples grow toward 1000. This means:

• Early (few samples): C=1.5, heavy exploration — tries underexplored models aggressively
• Later (many samples): C=0.5, mild exploration — mostly exploits the best model, occasionally probes others

Arms with zero samples are explored first via random selection, before UCB1 kicks in.

Phase 2: Gaussian Thompson Sampling (steady state)

Once total samples reach 500, the bandit switches to Gaussian Thompson Sampling. Instead of a deterministic formula, it samples from each model's posterior distribution:

sampled_reward ~ N(mean, variance / sample_count)

From model-bandit.ts:159-180:

private thompsonSelect(arms: BanditArm[]): BanditSelection {
  let bestArm = arms[0];
  let bestSample = -Infinity;

  for (const arm of arms) {
    // Sample from posterior N(mean, var / n)
    const stddev = arm.sampleCount > 0
      ? Math.sqrt(Math.max(arm.rewardVar, 0.001) / arm.sampleCount)
      : 0.5;
    const sampled = arm.rewardMean + gaussianRandom() * stddev;

    if (sampled > bestSample) {
      bestSample = sampled;
      bestArm = arm;
    }
  }
  // ...
}

The beauty of Thompson Sampling: a model with high mean and low variance (consistent performer) will almost always win. But a model with high variance (inconsistent) gets occasional chances to prove itself. The exploration is probability-weighted, not coefficient-tuned.

The gaussianRandom() function uses the Box-Muller transform for standard normal samples (model-bandit.ts:218-228).

Hard floors

Before any selection runs, arms are filtered by hard floors:

Floor	Threshold	Condition	Source
Validity	≥ 0.5	Only after ≥ 10 samples	model-bandit.ts:64
Quality	≥ 0.3	Only when quality data exists and ≥ 10 samples	model-bandit.ts:65

If no arms pass the hard floors (cold start scenario), all arms are allowed as a fallback.

Reward signal

The reward is a composite score derived from:

• Validity — Did the response complete successfully? (0 or 1)
• Quality — Tool-call accuracy tracked via EWMA
• Cost — Relative to the cheapest available option
• Latency — Relative to the fastest available option

Statistics are maintained in a 7-day sliding window using Welford's online algorithm for numerically stable variance computation. The mergeDailyStats() function (model-bandit.ts:187-215) implements parallel Welford merge for combining daily buckets.

Configuration

All thresholds are configurable per-instance:

const bandit = new ModelBandit({
  minValidity: 0.5, // Hard floor: validity
  minQuality: 0.3, // Hard floor: quality
  initialC: 1.5, // UCB1 exploration start
  minC: 0.5, // UCB1 exploration floor
  thompsonThreshold: 500, // Samples before switching to Thompson
});

What this means in practice

For a tenant sending 100 requests/day across 4 model endpoints, the leaderboard learns like this:

• Day 1-2: UCB1 credits all 4 models roughly equally (C=1.5)
• Day 3-5: UCB1 starts favoring the best performer in its rankings but still probes (C decaying)
• Day 5+: Passes 500 samples, switches to Thompson Sampling
• Steady state: the posterior concentrates — the best model wins ~80-90% of leaderboard/consensus draws, with ~10-20% of the probability mass still crediting alternatives
• If a model degrades: Thompson detects it within ~50 samples as the posterior shifts, and the leaderboard reorders

The leaderboard learns your workload without configuration. You don't set weights. You don't pick strategies. The math does it. And because every response also updates the shared learned-quality signals that the capability-matched auto-selector blends into its scores, what the leaderboard learns shows up in model: "auto" picks too — through the selector's scoring, not through this bandit.