Home Dashboard GitHub

SSE Streaming Heartbeat — zero-cost connection keep-alive

2026-03-26

gatewaystreaming

What We Built

Periodic SSE comment heartbeats (: heartbeat\n\n) during streaming responses. When a provider is thinking (e.g., Anthropic extended thinking, 30+ seconds), the gateway now emits a heartbeat comment every 5 seconds of silence. Per the W3C SSE spec, lines starting with : are comments — conforming clients (EventSource, AI SDKs) silently ignore them.

The heartbeat is wired into all three streaming paths: main completions, agentic mode, and swarm mode.

Why It Matters

CDNs and reverse proxies have idle timeouts (AWS ALB: 60s, Cloudflare: 100s). During long model thinking pauses, the TCP connection can be killed by these intermediaries. Clients then see a connection reset instead of the eventual response. This is especially painful for reasoning models that can think for 30+ seconds before emitting the first token.

How It Works

A startHeartbeat() utility creates a timer that fires every 5 seconds. If no real SSE data was sent since the last tick, it writes : heartbeat\n\n (a raw SSE comment). Every writeSSE() call resets the timer. The heartbeat stops on stream end, governance sever, or client disconnect (abort signal).

The Numbers

  • Heartbeat interval: 5 seconds
  • Payload overhead: 0 bytes (SSE comments carry no data payload)
  • Timer overhead: one setInterval per active streaming request
  • Cleanup: automatic via clearInterval on stream end + abort signal listener

Competitive Edge

Most AI gateways don't handle the "long thinking pause" problem. They either let connections die or require clients to set long timeouts. BrainstormRouter keeps connections alive transparently at the transport layer — no client changes needed, no SDK updates required.

Lockstep Checklist

  • [x] API Routes: src/api/routes/completions.ts updated (heartbeat wired into 3 streaming paths).
  • [n] TS SDK: N/A — SSE comments are invisible to SDK per W3C spec.
  • [n] Python SDK: N/A — SSE comments are invisible to SDK per W3C spec.
  • [n] MCP Schemas: N/A — MCP does not expose streaming.
  • [ ] Master Record: No new capability — transport-level improvement.