Evaluation Datasets

Manage test case collections for prompt and model evaluation.

Endpoints

Method	Path	Auth	Description
POST	/v1/eval/datasets	API Key	Create dataset
GET	/v1/eval/datasets	API Key	List datasets
GET	/v1/eval/datasets/:id	API Key	Get dataset
PUT	/v1/eval/datasets/:id	API Key	Update dataset
DELETE	/v1/eval/datasets/:id	API Key	Delete dataset + items
POST	/v1/eval/datasets/:id/items	API Key	Add items (batch, max 100)
GET	/v1/eval/datasets/:id/items	API Key	List items
DELETE	/v1/eval/datasets/:id/items/:itemId	API Key	Delete item
POST	/v1/eval/datasets/:id/import-from-logs	API Key	Import from prod logs
GET	/v1/eval/datasets/:id/export	API Key	Export CSV or JSON

Create dataset

curl -X POST https://api.brainstormrouter.com/v1/eval/datasets \
  -H "Authorization: Bearer br_live_..." \
  -H "Content-Type: application/json" \
  -d '{"name": "Customer Support", "description": "Test cases for support prompts"}'

Add items

curl -X POST https://api.brainstormrouter.com/v1/eval/datasets/{id}/items \
  -H "Authorization: Bearer br_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"input": {"topic": "billing"}, "expected_output": "I can help with billing."},
      {"input": {"topic": "refund"}, "expected_output": "To process a refund..."}
    ]
  }'

Maximum 100 items per batch.

Import from production logs

curl -X POST https://api.brainstormrouter.com/v1/eval/datasets/{id}/import-from-logs \
  -H "Authorization: Bearer br_live_..." \
  -d '{"model": "anthropic/claude-sonnet-5", "min_tokens": 100, "limit": 50}'

Imports request metadata from the completion_audit table as dataset items.

Export

# JSON
curl ".../v1/eval/datasets/{id}/export?format=json"

# CSV
curl ".../v1/eval/datasets/{id}/export?format=csv" > dataset.csv

CSV columns: input, expected_output, metadata.