How valite Works

The gateway is a transparent reverse proxy. The first path segment of each request names the provider (/openai/…, /anthropic/…), and everything after it is forwarded to that provider byte-for-byte — streaming, tool calls, and all. Because nothing is re-implemented, anything the provider's API can do works through valite unchanged.

Logging happens off the hot path: the response is relayed to you first and parsed for the log afterwards, so the gateway adds zero time-to-first-token. Token counts and the served model come from the provider's own usage metadata, including on streamed responses.

A callsite is a distinct place in your code that calls a model — the same prompt template, tools, and output shape, with different values filled in. valite discovers callsites by fingerprinting request bodies: volatile values are masked, and the stable structure that remains identifies the callsite.

Callsites are the unit of every decision valite makes. A candidate model is proven (or rejected) per callsite, and a routing rule only ever rewrites the model for callsites where the evidence passed. Callsites with long, open-ended outputs are marked as quality anchors— "do not touch" — and are never routed.

A workflow is a unit of work in your product — an agent task, a document pipeline, a support reply. A run is one execution of it. valite groups calls into runs so it can measure what actually matters: cost per run and quality per run, not per isolated API call.

You attribute calls to workflows via a path segment in the base URL, an x-valite-workflow header, or W3C traceparent propagation — see Workflows & Runs. Once a workflow has around five complete runs, the pipeline has enough baseline to work with.

Analysis runs as eight jobs over the call log, each writing a durable artifact. Later stages read earlier artifacts, so the chain of evidence from raw traffic to routing decision is inspectable at every step:

profile   → what does this workflow cost today, per run?
triage    → which callsites are candidates to move? which are anchors?
bar       → freeze the quality bar from your baseline (mean − k·σ)
calibrate → A/A-test the judge on your own traffic before trusting it
replay    → re-send captured requests to candidate models, score them
plan      → pick the cheapest candidate that clears the bar, per callsite
validate  → check the routed configuration against the frozen bar
report    → human-readable scorecard of the whole run

Each job is started and polled over HTTP — see the Jobs API. Replay and calibrate spend real provider money, so they are budget-gated: without an explicit budget the job fails fast with a cost projection instead of running.

Load balancers make reactive, per-request decisions against live traffic. valite makes deterministic, pre-proven decisions: routing rules are computed offline from replay evidence, reviewed by you, and applied explicitly. On the data plane a rule is a simple lookup — matched callsites get their model field rewritten, everything else passes through untouched.