Disclosure: BlueBear builds an AI agent platform and an MCP gateway, so we have a commercial interest in this topic. Implementation details are quoted from our own gateway source. Route planning ships disabled by default in our deployment configuration, so these are contracts in code rather than fleet-wide measurements.
The short answer: the canary is the only stage where the router's judgement meets a real user
A model-routing rollout has three stages and they answer different questions. Shadow mode answers "what would this have cost?" — cheaply, on real traffic, without changing anything. An activation gate answers "do recent outcomes justify trusting this?" A canary sits between them and answers the one question neither can: when this router's recommendation is actually executed, is the result acceptable?
That question cannot be answered without executing, and executing is the risky part. So the entire design problem of a routing canary is making the blast radius small, bounded, reproducible, and — the part most implementations miss — explainable after the fact. A canary you cannot reconstruct is a change you cannot defend.
This assumes you already have a routing policy to roll out. If you do not, the routing policy guide covers the policy fields — eligibility, constraints, fallback, audit mode — that a canary is putting into production, and the pix2code routing case study covers what feeding outcomes back into the decision looks like end to end.
Deterministic sampling, not random
Our canary derives its sample from a hash of the request identifier: SHA-256 over the request ID, the first four bytes read as a big-endian integer, divided by 232 to give a stable float in [0, 1). The canary executes when that value is below the configured sampling rate.
The statistical properties are the same as random sampling. What you gain is reproducibility, and it is worth more than it sounds:
- A user reports a bad answer. You have the request ID. You can determine with certainty whether they were in the canary — no guessing from timestamps.
- The same request replayed lands in the same arm, so a bug is reproducible rather than intermittent.
- Nothing has to be stored to know the assignment. The identifier is the assignment, which means the property survives a restart, a redeploy, and a cache flush.
If you take one thing from this article, take this one. It is a few lines of code and it converts an entire class of unfalsifiable incident reports into answerable questions.
Configuration that fails loudly
Three environment values are required together — a sampling rate, a tenant allowlist, and a model allowlist. Miss any one and the canary is disabled. And there is a hard ceiling in source with the reasoning attached:
const MAX_CANARY_SAMPLING_RATE = 0.1;"Canary execution is disabled unless rate + tenant allowlist + model allowlist are all configured. The hard cap prevents an env typo from silently turning shadow routing into broad active execution."
The important detail is what happens when someone configures 0.5. It is not clamped to 0.1. It invalidates the configuration and the canary does not run.
Clamping is the instinct — it feels forgiving. It is the wrong instinct for a control that changes what a customer receives. A clamped value means someone who typed 0.5 intending 50% gets 10% and no signal that their intent was overridden; they will believe they are running a large canary. Refusing to run produces a visible zero, and a visible zero gets investigated. For safety controls, the forgiving behaviour is the dangerous one.
The configuration record reflects this by separating two different states that a single boolean would conflate: nothing configured at all is disabled but valid; partially configured is invalid. "Deliberately off" and "someone tried to turn this on and got it wrong" should never be the same value on a dashboard.
The eight conditions that stop a canary, in order
Each is evaluated in sequence and each produces a distinct recorded reason. That granularity is the difference between "the canary is not running" and knowing why.
| # | Condition | Recorded reason | Why it stops execution |
|---|---|---|---|
| 1 | Sampling rate invalid or above the cap | canary_rate_invalid | A misconfiguration must not become a broad rollout. |
| 2 | Configuration incomplete | canary_configuration_missing | Partial configuration is a mistake, not an intention. |
| 3 | Tenant not allowlisted | canary_tenant_not_allowed | Rollout is per customer, deliberately. Not every tenant is an acceptable first. |
| 4 | Recommended model not allowlisted | canary_model_not_allowed | The set of models you are willing to execute is narrower than the set the planner may recommend. |
| 5 | Recommendation equals what would have run | canary_same_as_legacy | Nothing to test. Counting these as canary executions inflates your denominator and dilutes the comparison. |
| 6 | Explicit user override present | canary_explicit_override | A user who chose a model gets that model. Overriding an explicit choice to run an experiment is not a defensible trade. |
| 7 | Hard model constraint | canary_hard_model_constraint | A warm runtime that cannot switch, or a locked workspace default. Executing here would produce a record claiming a model that did not run. |
| 8 | Confidence below the minimum | canary_confidence_below_minimum | If the planner is not confident, a live user is the wrong place to find out. |
Condition 5 is the one most likely to be missing from a hand-rolled implementation, and it quietly corrupts results: if a third of your "canary executions" recommended the model that was already going to run, your quality comparison is a third noise.
Condition 7 connects to a constraint the gateway resolves before planning, whose source comment states the reasoning better than we can paraphrase it: "Warm runtimes that cannot switch must keep their current model as a hard constraint so route metadata cannot claim a model that the runtime did not execute." That is the same concern as plan-honoured accounting, caught one stage earlier — prevent the divergence where you can, record it where you cannot.
Where the canary lives, and why that surprises people
In our implementation the canary is constructed only in shadow mode. It does not exist in off and it does not exist in active, where the recommendation is applied anyway.
That has a consequence worth stating plainly, because it inverts a common assumption: shadow mode is not unconditionally inert. Shadow is where a small, capped, deterministic slice of traffic actually executes the recommendation. If you describe shadow mode to a stakeholder as "nothing can change", check your canary configuration first, or you will have said something untrue in a room where it mattered. We cover this in more depth in shadow-mode routing.
Rollback, honestly
There is no automatic rollback loop in our gateway. It is worth saying that plainly rather than implying an autonomous safety system that does not exist.
What does exist is one in-band revert: if the credential for the canary-selected model turns out to be unavailable at execution time, the canary is marked not executed with the reason canary_selected_model_key_unavailable and the legacy model is used. Beyond that, operator rollback is unsetting the configuration — which, given the configuration is three environment values that must all be present, is a single decisive action rather than a delicate one.
There is also a readiness check that compares the canary's model allowlist against the live catalog and reports which allowlisted models are missing. It warns; it does not disable. That is a defensible split — a coverage check that silently switched off a rollout would itself be an unexplained change — but it means the check is only as useful as the person reading it.
The activation gate: the stage after the canary
The canary produces outcomes. Something has to decide whether those outcomes justify going active, and in our implementation that decision is not left to whoever is editing the tenant settings.
Moving a tenant's routing policy to active triggers an evaluation, and the transition fails closed. If the evaluation cannot be reached, the change is rejected with a service-unavailable error. If it is reached and returns not-approved, the change is rejected with a conflict error. The evaluation carries an approval boolean, a map of thresholds, a summary, and a list of failed reasons — so a rejection tells you which threshold was missed rather than just refusing.
Two honest caveats. The threshold names and values live in the planner service, not in our gateway, so we are not going to invent examples of them. And the evaluation is a gate at the moment of transition, not a continuous monitor — it does not automatically revert a tenant that degrades later. Continuous monitoring is a separate thing you still need, and it lives in the outcome data described in plan-honoured accounting.
Do this next
Whatever you are rolling out, write down the list of conditions under which it must not execute, and give each one a distinct string. Do that before writing the sampling logic. The sampling is ten lines; the refusal taxonomy is the part that makes the rollout explainable six weeks later when someone asks why a specific customer got a specific answer.
Questions people actually search for
- how do I safely roll out llm model routing
In three stages, each with its own gate. Shadow mode to prove the cost case without changing anything. A capped canary that executes the recommendation on a small, deterministic slice of traffic to prove the quality case. Then an activation gate — an explicit evaluation of recent outcomes that must approve before the tenant moves to active. Skipping the middle stage means the first real execution of the router's judgement happens at full traffic.
- what percentage of traffic should a routing canary use
Small enough that a bad recommendation is an inconvenience rather than an incident, and large enough to accumulate outcomes in a reasonable window. Our implementation caps it in source at 10% and refuses to run at all if a higher value is configured. The right number for you depends on volume — the question to answer is "how many outcomes do I need before the quality comparison is meaningful", and then pick the smallest rate that reaches it in an acceptable time.
- should canary sampling be random
No — deterministic. Hash a stable request identifier and compare the derived fraction to the sampling rate. Random sampling means the same request replayed lands differently, which makes a bug report unreproducible and makes it impossible to tell whether a user saw the canary. A hash of the request ID gives you the same statistical properties with reproducibility for free.
- when should a routing canary not run
Whenever executing the recommendation would violate something the caller already asked for. Concretely: an explicit user model override, a hard model constraint from a runtime that cannot switch, a recommendation identical to what would have run anyway, a model or tenant outside the allowlists, or a confidence score below the configured minimum. Each should be a distinct recorded reason, not a silent skip.