Cost Optimization

Most LLM spend hides in a few callsites: structured extraction, classification, and short transformations running on a frontier model because that's what the team started with. These are exactly the callsites where smaller models tend to hold quality — and where replay can prove it cheaply, because outputs are short and the judge's comparisons are cheap.

The profilejob makes this visible: cost per run, decomposed by callsite, from your real traffic. Every profiled callsite is then screened by replay — quality is gated purely by the pairwise judge's verdicts, not by heuristics.

Run the pipeline per workflow, in order:

1. profile    know your cost per run and its breakdown
2. goldenset  freeze a representative sample per callsite
3. calibrate  A/A-test the judge is unbiased on your traffic
4. replay     test candidates within a fixed budget, judged pairwise
5. plan       accept only routes with evidence attached
6. canary     apply at 10–25% of runs, hashed per run
7. validate   judge routed calls vs fresh incumbent regenerations
8. active     promote — the tripwire keeps watching

Because every call logs both requested_model and routed_model, realized savings are computed from actual traffic, not projections. The plan's projection gives you the expected number (savings_pct); validation gives you the realized cost per routed run; the report ties them together with the quality outcome.

Pricing comes from live provider catalogs (cached six hours). Models without published pricing are flagged in the profile's pricing notes rather than silently costed at zero.

A healthy plan often routes only some callsites. Common reasons a candidate is rejected — all recorded in the plan notes: the judged extremely-worse rate above the ceiling, too few judged calls, latency ratio above your cap, or clearing quality while saving nothing. An empty plan is a valid, useful result: it means the evidence says your current models are earning their cost.