Disclosure: BlueBear builds an AI agent platform and an MCP gateway, so we have a commercial interest in this topic. The implementation details below are drawn from our own gateway source, and we have marked what is a contract in the code, what is a default, and what we have not measured.
What shadow-mode routing is, in one paragraph
Shadow-mode routing runs the model-selection decision against real production traffic and records the answer without acting on it. The request goes to the router. The router returns a recommendation. Production continues executing whatever it was already going to execute, and the recommendation is written down beside it. You end up with paired records — what was planned, what actually ran, what each would have cost — for your own traffic rather than for a benchmark.
This matters because the alternative is guessing. Every model-routing business case starts with an estimate: "if we sent the classification steps to a smaller model we would save maybe 40%." That estimate is built from an assumed traffic mix, and traffic mixes are consistently not what people assume. Shadow mode replaces the assumption with a measurement, and the measurement costs one extra planning call per request rather than a production incident.
The three states, and why there are exactly three
In BlueBear's gateway the mode is a three-value type. The literal definition, from src/services/modelRouting/modelRouting.service.ts:
export type RoutePlanningMode = "off" | "shadow" | "active";Two states would not be enough, and that is the whole design argument. With only off and active, the first time your router's judgement meets production traffic is also the first time it changes a customer's answer. Those are two different risks and they should be taken separately. Shadow is the state that lets you take the first one alone.
| Mode | Planner runs? | Recommendation recorded? | Execution changes? | What it is for |
|---|---|---|---|---|
off | No — the call is short-circuited before any network request | No | No | The default. Nothing to debug, nothing to pay for. |
shadow | Yes | Yes, in a separate field | No, except for a configured canary sample | Building the evidence that a change is worth making. |
active | Yes | Yes | Yes — the recommendation becomes the model that runs | The change itself, after the evidence exists. |
The trap: shadow mode is not unconditionally a no-op
This is the part that most descriptions of shadow routing get wrong, including descriptions we could have written more carefully ourselves. It is worth being exact.
In our implementation, the returned plan in shadow mode keeps the previously-configured model as the selection and parks the recommendation in a separate field. The relevant expression is a conditional, not an unconditional:
selectedModel: mode === "shadow" && canaryExecution?.executed !== true
? legacyModel
: validated.selectedModel,
shadowSelectedModel: mode === "shadow" ? validated.selectedModel : null,Read that carefully. In shadow mode the recommendation is normally parked and legacyModel — the model the caller already asked for, or the first eligible catalog entry — is what comes back. But if a canary executed, the recommendation is what comes back. Shadow mode is where the canary lives.
That is a deliberate design and it is bounded, but the operational consequence is real: "we are only in shadow mode" is not by itself a guarantee that nothing can change. The guarantee comes from the canary configuration, which in our gateway requires three separate environment values to be set together — a sampling rate, a tenant allowlist, and a model allowlist — and is disabled if any is missing. There is also a hard ceiling in the source, with its rationale stated in the file header:
const MAX_CANARY_SAMPLING_RATE = 0.1;The comment above it explains why the cap exists rather than being clamped silently: it "prevents an env typo from silently turning shadow routing into broad active execution." A rate above the cap does not get quietly reduced to 10% — it invalidates the whole canary configuration, and the canary does not run at all. Failing loudly on a misconfiguration is the correct behaviour for a control that changes what a customer receives.
If you are building your own shadow mode, decide explicitly whether it is inert or whether it carries a canary, and put that decision somewhere an operator will read it. A shadow mode that sometimes executes and does not say so is worse than no shadow mode.
BlueBear Shadow-Mode Routing Evidence Contract
A shadow run is only worth what you recorded while it was running. This is the minimum record that makes a shadow dataset answerable, taken from the fields BlueBear's gateway already emits. It is an implementation contract, not a measurement — no figures below are drawn from production traffic.
| Field | Why it is load-bearing | Recoverable afterwards? |
|---|---|---|
Planning mode in force (off, shadow, active) | Without it, a divergence between planned and executed models is uninterpretable. | No |
| The model that would have been chosen, separate from the model that ran | Two fields, never one. Overwriting the executed model with the recommendation destroys the comparison you are running the experiment to make. | No |
| Router strategy and selection confidence | A low-confidence recommendation and a high-confidence one are different evidence, and only one of them should be promoted. | No |
| Candidate scores, including the ones not selected | The rejected candidates and their reasons are how you find out the router never even considered the model you assumed it would. | No |
| Cost projection with its derivation source | A projection built from an explicit token demand is defensible; one built from a message-size heuristic needs a wider interval. Blending them silently overstates confidence. | No |
| Fallback reason when no recommendation was produced | A shadow dataset computed over only the requests that produced a plan is computed over the easy requests. | No |
| Whether a canary executed | Shadow mode is where the canary lives. A canary execution is a real change to a real user and must never be counted as a shadow observation. | No |
Illustrative candidate record
The field names below are the real ones from RoutePlanCandidateScore. The values are an example for review; they are not production measurements.
{
"profile": "chat-default",
"model": "example-small",
"backend": "remote",
"selected": true,
"eligible": true,
"confidence": 0.86,
"estimatedCostUsd": 0.0041,
"estimateSource": "observed",
"costProjection": {
"source": "explicit_token_demand",
"estimatedInputTokens": 3200,
"estimatedOutputTokens": 380,
"inputCostPer1m": 0.0,
"outputCostPer1m": 0.0,
"calibration": {
"scope": "tenant_model_task",
"sampleCount": 0,
"inputTokenMultiplier": 1.0,
"outputTokenMultiplier": 1.0
}
},
"reasons": ["eligible", "within_cost_ceiling"]
}Prices are held per million tokens with the unit as an explicit field rather than a convention, so nothing downstream has to infer whether a number is per-1K or per-1M. Zero values above are placeholders, not a claim about any model's price.
Two invariants worth enforcing
- Exactly one candidate may be marked selected, and its confidence must equal the plan's selection confidence. A plan with two selected candidates, or a mismatch between the two numbers, is a malformed plan and should be rejected rather than recorded.
- A recommendation must be re-validated against the live catalog before it is used. A plan built from a stale snapshot can name a model that is no longer active, no longer available to the workspace, or has no usable credential. Rejecting late is better than executing a model your policy no longer permits.
What shadow mode can and cannot tell you
Shadow data answers one question well and a second question not at all, and conflating them is the most common way teams get burned.
| Question | Can shadow answer it? | Why |
|---|---|---|
| What would routing have cost? | Yes | Cost is projectable. Token counts are estimable from the request and prices are published, so a plan can carry a cost projection without the model ever running. |
| Which requests would have been routed differently? | Yes | You have the planned and the executed model on the same request. |
| How often does the router decline to recommend anything? | Yes, and this is underrated | Fallbacks, timeouts, and confidence rejections are all recorded with reasons. |
| Would the cheaper model have produced an acceptable answer? | No | It never ran. There is no output to grade. |
| Would latency have been acceptable? | No | Same reason. A latency target in the plan is a constraint, not an observation. |
So shadow mode is a cost and coverage instrument. The quality question needs a separate gate — offline evaluation against a held-out set, or a canary that genuinely executes. Is a cheaper model good enough? covers the acceptance method, and canary rollout covers the execution gate.
The cost projection, and why its provenance is a field
A cost number without provenance is not evidence. In our route-plan candidate records, the cost estimate carries how it was derived. The candidate score type includes estimatedCostUsd alongside estimateSource, which is one of three literals:
estimateSource?: "default" | "request" | "observed";And the projection itself records the token model it used:
costProjection?: {
source: "explicit_token_demand" | "message_heuristic";
estimatedInputTokens: number;
estimatedOutputTokens: number;
inputCostPer1m: number;
outputCostPer1m: number;
calibration?: {
scope: "tenant_model_task";
sampleCount: number;
inputTokenMultiplier: number;
outputTokenMultiplier: number;
};
}The distinction between explicit_token_demand and message_heuristic is the difference between a number you can defend and a number you should hedge. A heuristic estimate over a message payload is a guess about tokenisation; an explicit token demand is not. When you build a savings model from shadow data, split the total by projection source — if most of your projected saving rests on heuristics, your confidence interval is much wider than a single number implies.
The optional calibration block is the correction for that: a per-tenant, per-model, per-task multiplier derived from a stated sampleCount. A projection carrying a calibration with a sample count of four is not the same evidence as one with a sample count of four thousand, and the field exists so you do not have to take it on faith.
Prices themselves are held per million tokens in a single projection with an explicit unit, so nobody has to guess whether a number is per-1K or per-1M:
{ schemaVersion: "bluebear-model-token-pricing/v1",
currency: "USD", unitTokens: 1_000_000,
inputCostPer1m, outputCostPer1m,
source: "gateway-model-catalog" }Routing must not become the latency problem
A router that adds meaningful latency to every request has traded a cost problem for a worse one. Our planning call is bounded by a constant:
const DEFAULT_ROUTE_PLAN_TIMEOUT_MS = 750;Be precise about what that is: it is a client-side abort on exactly one HTTP request — the planner call — implemented with an AbortController. It is not an end-to-end routing SLA, and it does not bound the model call that follows. It is overridable by environment variable and by injected dependency, with the constant as the last fallback.
What happens on timeout matters more than the number. The planner call fails open: the catch branch returns a result with source: "fallback", the selection set to legacyModel, no routing metadata, and a fallbackReason of "route_plan_timeout". A non-2xx response produces route_plan_http_<status>; a missing planner base URL produces inference_os_base_url_missing.
Two design points worth stealing regardless of what you build on. First, fail open to the model the caller already wanted, not to a hardcoded default — falling back to a "safe default model" silently changes behaviour at exactly the moment you have least information. Second, make every fallback carry a distinct reason string. When your shadow dataset shows a 30% fallback rate, "the planner was slow" and "the planner was misconfigured" require completely different responses, and one string tells you which.
Running it: a four-week shape
- Turn planning on in shadow, with the canary explicitly unconfigured. Verify the recommendation is being recorded and that execution is unchanged, by checking that planned and executed models differ on some requests and that user-visible behaviour did not move.
- Watch the fallback rate before looking at savings. If a meaningful share of requests never produced a recommendation, your savings estimate is computed over a biased subset — probably the easy requests. Fix the fallbacks first.
- Compute the projected saving, split by projection source and by task type. A single blended percentage hides the thing you need: routing usually pays enormously on a few task types and nothing on the rest. Which agent steps need a frontier model covers where that line tends to fall.
- Take the top task type to a quality gate. Only now does the cheaper model actually run, on a held-out set or a capped canary.
Step 2 is the one people skip and the one that most often invalidates the result.
How this fits the rest of the routing decision
Shadow mode is the evidence stage of a larger sequence: eligibility (which models are even permitted for this tenant, this compliance class, this workspace), then ranking (which permitted model is best for this request), then fallback (what happens when the preferred route is not available). Those are three different decisions with three different owners, and collapsing them into one opaque score makes incidents unreconstructable. The routing policy guide works through the policy fields in full.
One point of precision about our own product, because the category is loose about it and precision is the only thing that makes this article worth more than a competitor's. Our gateway does not make the inference call from a route plan — plans go out to a planner service and come back as recommendations. But in active mode a returned recommendation does become the model the runtime is told to execute, and the gateway also resolves the provider credential for it. It is accurate to say the gateway plans and the runtime executes. It would not be accurate to say plans are only ever recorded and never applied. Route planning ships disabled by default in our deployment configuration; we are describing a contract in the code, not a fleet-wide measurement.
Do this next
Before configuring anything, answer one question about your current stack: if the router recommended a different model right now, would you be able to tell afterwards whether it was actually used? If the answer is no, the recording gap is a bigger problem than the routing gap — a shadow dataset you cannot join to execution is not evidence. Did your router honour the plan? is the piece on closing exactly that gap.
Questions people actually search for
- what is shadow mode in llm routing
Shadow mode runs the routing decision on real production traffic without acting on it. The router receives the same request the production path receives, produces a recommendation, and the recommendation is recorded next to what actually executed. Nothing about the user's response changes. After a few days you have a paired dataset — planned model versus executed model, on your own traffic — and you can compute the saving the router would have produced instead of estimating it.
- how do I test a cheaper model without risking production
Shadow mode answers the cost half of the question and only the cost half. It tells you what the router would have picked and what that would have cost, because cost can be projected from token counts and published prices. It cannot tell you whether the cheaper model would have produced an acceptable answer, because the cheaper model never ran. For the quality half you need either offline evaluation on a held-out set or a small canary that actually executes. Treat them as two separate gates.
- is shadow mode routing safe
Only if you check one thing: whether shadow mode is genuinely inert in your implementation. In BlueBear's gateway it is not unconditionally inert — shadow mode is also where the canary lives, so a configured canary can cause a shadow-mode request to execute the recommendation on a small deterministic sample. That is deliberate and it is capped, but it means "we are only in shadow" is not by itself a statement that nothing can change. Read your own configuration before you assume.
- how long should I run shadow mode
Long enough to cover your traffic's real variety rather than a fixed number of days. The failure is running it over a quiet week and missing the month-end batch, the enterprise customer who sends different work, or the retry storm. A useful heuristic is to run until the distribution of task types in your shadow dataset matches the distribution in your last full billing period.