When teams talk about optimizing AI costs, they almost always mean one thing: swapping the model. It is the most visible decision, the one with a name on the invoice. But a production API call carries dozens of other decisions with it — the prompt revision, the provider, the pricing tier, the decoding parameters, whether the call is cached, batched, or even made at all.
Each of those decisions is set once, usually at integration time, usually by whoever was shipping under deadline. Then they fossilize. The result is that most production workloads run on a policy nobody chose deliberately — an accident of defaults that quietly compounds at scale.
Treat the whole boundary as one policy
Our framing is that everything decidable at the API boundary is a single execution policy, and the policy — not the model — is the unit of optimization. Model swap is the baseline rung, and it is a good one. But the rungs above and below it are just as real: prompt compression against a quality floor, provider selection for the same open model, batch and off-peak tiers for latency-tolerant traffic, caching for repeated context, fallback chains for reliability.
Individually each rung looks like a modest single-digit or double-digit improvement. The reason to treat them as one policy is that they compound, and that they interact — the cheapest model on a fast tier is often more expensive than a better model on a batch tier, and a compressed prompt changes which model clears the quality bar.
Why this can't be a one-time audit
The other property of the execution stack is that it moves. New models ship monthly, providers reprice weekly, and new tiers and serving options appear without ceremony. A policy that was optimal in March is stale by June — not because your workload changed, but because the frontier under it did.
That is why Valite is built as a continuous system rather than a consulting exercise: it tracks new prompts, models, providers, and pricing as they appear, replays them against your captured traffic, and surfaces the ones that beat your current policy within your quality, latency, and reliability constraints. The workload stays yours; the search runs all the time.
The teams that internalize this stop asking “which model should we use?” and start asking “what is our policy, and what is it worth to keep it current?” In our experience, the answer to the second question is a large fraction of the AI budget.