Routing Recommendations

The plan job reads the profile and replay artifacts and, for each profiled callsite, picks the cheapest candidate that clears every threshold: a judged extremely-worse rate at or under the ceiling (default 5%) over a minimum number of judged calls, and an optional latency ratio cap. Candidates that fail are recorded in the plan's notes with the reason, so a rejected route is as auditable as an accepted one.

Routes stay within the provider the callsite already uses; cross-provider routes are not routable in v1 — a cheaper cross-provider candidate is recorded in the notes instead.

The plan is a YAML config where each route embeds its proof:

routes:
  - callsite_id: cs_9f2ab1
    from_model: gpt-4o
    to_model: gpt-4o-mini
    evidence:
      judged: 10
      worse: 0
      similar: 9
      better: 1
      worse_rate: 0.0
      judge_model: gemini-2.5-flash
      savings_per_run_usd: 0.0312
      replay_sampled: 10
      baseline_cost_per_run_usd: 0.041
      latency_ms: 640
      baseline_latency_ms: 812
projection:
  baseline_cost_per_run_usd: 0.041
  routed_cost_per_run_usd: 0.0098
  savings_pct: 76.1

The projection summarizes what the whole plan saves per run. The report job renders the same evidence as a human-readable scorecard for review.

Routing is opt-in and explicit: the plan config is activated on the gateway, which then rewrites the model field for matched callsites — nothing else in the request is touched, and unmatched calls pass through exactly as before. Requests record both requested_model and routed_model in the call log, so attribution is never ambiguous.

Shadow — production is served by the baseline model unchanged; the candidate is mirrored out-of-band after the response has been fully delivered, giving you live comparison data at zero user risk.

Canary — a configurable percentage of runs get the route. Assignment is hashed on the run ID, so a run is always entirely canary or entirely baseline — no mixed runs muddying per-run metrics.

Active — the route applies to all matched traffic.

Set VALITE_DISABLE_ROUTING=1 on the gateway and all routing stops instantly — checked per request, no restart needed.

Independently, an automatic per-callsite tripwire watches routed traffic: once at least 20 routed calls have been observed for a callsite, it reverts to passthrough if its routed error rate exceeds 10% absolute and3× the baseline error rate. The revert is persisted so every replica converges, and that callsite's traffic falls back to the original model until you intervene.

After running routed traffic (canary or active), the validate job re-generates a sample of the routed calls with the incumbent model and has the judge compare them head-to-head — the exact same requests, so the comparison is apples-to-apples — and reports pass/fail plus realized live cost and latency. The report job assembles the final scorecard: what was tested, what passed, what it saves.