Testing New Models

When a new model ships, you want to know: would it hold up on my traffic, and what would it save? valite answers this by re-sending your captured request bodies to the candidate — your workflow is never re-run and your users are never exposed. Sampling is deterministic (seeded), so a replay is reproducible.

Only callsites shortlisted by triage are replayed; quality anchors are excluded by construction. Cross-provider replays (e.g. OpenAI traffic against a Claude model) are translated automatically, except for tool-calling requests, which stay same-provider.

Each candidate is measured per callsite on four axes:

Compliance — for schema-strict callsites (JSON mode, tool calls), does the output actually parse and validate? For prose, is it non-empty? Quality — a pairwise comparison against your captured baseline output (see the judge below). Latency and cost per call — measured directly, with live pricing pulled from provider catalogs and cached for six hours. Models without published pricing are flagged, never silently treated as free.

Deterministic graders can replace or supplement the judge per workflow: json_valid, nonempty, or regex:<pattern> — and you can plug in your own eval command that scores runs however you like.

Quality comparison uses a blinded pairwise LLM judge: model names are hidden, both response orders are tried, and the two votes must agree — if they don't, the verdict degrades to a tie. Verdicts are A, B, TIE, or BAD(both unusable). The judge must come from a different model family than both the incumbent and the candidate; valite warns if it doesn't. With no judge configured, a token-overlap F1 score is used as a proxy instead.

Judge scores screen candidates in or out of the shortlist. The gate — the thing that decides whether a routed configuration is acceptable — is always your own metric via the quality bar.

Before trusting the judge on your traffic, run the calibrate job. It regenerates sampled calls with your own incumbent model and has the judge compare captured vs. fresh — the true difference is zero by construction, so any preference the judge shows is bias. Calibration passes when arm deviation stays within 0.25 and position-swap agreement is at least 0.5; fewer than 10 judged pairs marks the result underpowered.

The bar job freezes a non-inferiority bar from your baseline runs: bar = mean − k·σ of your own run-level scores (default k = 1.0). A candidate configuration passes when the 95% bootstrap confidence interval on its score difference stays within that margin. The job flags saturated metrics (zero variance), high variance, and underpowered baselines — and requires 5 complete runs before it will freeze anything.