Testing New Models

When a new model ships, you want to know: would it hold up on my traffic, and what would it save? valite answers this by re-sending your captured request bodies to the candidate — your workflow is never re-run and your users are never exposed. Sampling is deterministic (seeded), so a replay is reproducible.

Replay runs on the frozen golden set — a fixed, representative sample of each callsite's traffic — so verdicts are comparable across models and across time. Cross-provider replays (e.g. OpenAI traffic against a Claude model) are translated automatically, except for tool-calling requests, which stay same-provider.

Each candidate is measured per callsite on three axes:

Quality— the judge compares the candidate's fresh response against your captured baseline for the exact same call: extremely_worse, similar, or better, per item. The number that gates routing is the worse rate — the share of judged items rated extremely worse. Latency and cost per call — measured directly, with live pricing pulled from provider catalogs and cached for six hours. Models without published pricing are flagged, never silently treated as free.

Quality is decided only by a blinded pairwise LLM judge: model names are hidden, both response orders are tried, and the two votes must agree — if they don't, the verdict degrades to similar. Verdicts are WORSE (extremely, clearly worse), SIMILAR, or BETTER.

The judge is required and must come from a third model family— never the incumbent's or a candidate's. valite enforces this before spending anything: a family collision fails the replay job rather than producing self-preferring verdicts.

Before trusting the judge on your traffic, run the calibrate job. It regenerates sampled calls with your own incumbent model and has the judge compare captured vs. fresh — the true difference is zero by construction, so any preference the judge shows is bias. Calibration passes when arm deviation stays within 0.25 and position-swap agreement is at least 0.5; fewer than 10 judged pairs marks the result underpowered.

The goldensetjob freezes a fixed sample of captured calls per callsite (default 50), stratified over input length, language script, and response shape — so short JSON extractions and long multilingual prose are both represented. Every replay screens candidates against exactly these items, and the plan accepts a route only when the candidate's judged worse rate stays at or under the ceiling (default 5%) over a minimum number of judged calls (default 5).