Routing Recommendations

The plan job reads the profile, triage, and replay artifacts and, for each shortlisted callsite, picks the cheapest candidate that clears every threshold: compliance (100% for schema-strict callsites, 98% for prose), a quality floor, and an optional latency ratio cap. Candidates that fail are recorded in the plan's notes with the reason, so a rejected route is as auditable as an accepted one.

Routes stay within the provider the callsite already uses; cross-provider routing is planned but conservative by design.

The plan is a YAML config where each route embeds its proof:

routes:
  - callsite_id: cs_9f2ab1
    from_model: gpt-4o
    to_model: gpt-4o-mini
    evidence:
      compliance: 1.0
      proxy: 0.94
      savings_per_run_usd: 0.0312
      replay_sampled: 10
      schema_strict: true
      baseline_cost_per_run_usd: 0.041
projection:
  baseline_cost_per_run_usd: 0.041
  routed_cost_per_run_usd: 0.0098
  savings_pct: 76.1

The projection summarizes what the whole plan saves per run. The report job renders the same evidence as a human-readable scorecard for review.

Routing is opt-in and explicit: the plan config is activated on the gateway, which then rewrites the model field for matched callsites — nothing else in the request is touched, and unmatched calls pass through exactly as before. Requests record both requested_model and routed_model in the call log, so attribution is never ambiguous.

Shadow — production is served by the baseline model unchanged; the candidate is mirrored out-of-band after the response has been fully delivered, giving you live comparison data at zero user risk.

Canary — a configurable percentage of runs get the route. Assignment is hashed on the run ID, so a run is always entirely canary or entirely baseline — no mixed runs muddying per-run metrics.

Active — the route applies to all matched traffic.

Set VALITE_DISABLE_ROUTING=1 on the gateway and all routing stops instantly — checked per request, no restart needed.

Independently, an automatic tripwire watches routed traffic: once at least 20 routed calls have been observed, routing disables itself if the routed error rate exceeds 10% absolute and 3× the baseline error rate. Traffic falls back to the original models until you intervene.

After running routed traffic (canary or active), the validate job scores the routed runs against the frozen quality bar — the same bar, so the comparison is apples-to-apples — and reports pass/fail plus realized cost per run. The report job assembles the final scorecard: what was tested, what passed, what it saves.