Did Your Router Actually Honour the Plan? The Gap That Breaks LLM Savings Models

Disclosure: BlueBear builds an AI agent platform and an MCP gateway, so we have a commercial interest in this topic. The field names and contracts below are quoted from our own gateway source. Where a value comes from a service outside this codebase, or where we have not measured something, we say so.

The short answer: a plan is a recommendation, and recommendations get declined

Every model-routing savings model rests on an assumption that nobody states out loud: that the model the router selected is the model that ran. When you compute "we routed 40% of requests to a cheaper model, therefore we saved X", you have quietly assumed a 100% plan-honoured rate.

It is never 100%. Planning and execution are different systems with different information, and the executor legitimately declines. The runtime was already warm on another model and cannot switch mid-session. The recommended model was deprecated at the provider after the catalog snapshot was taken. The credential for it was unavailable. A user or workspace override outranked the recommendation. Every one of those is the system working correctly — and every one of them turns a projected saving into no saving at all, silently.

The failure is not that plans get declined. It is that a declined plan and an executed plan produce identical records unless you deliberately record the difference.

Why this corrupts more than the cost number

The cost distortion is the obvious problem and it is the smaller one. The larger one is that plan-honoured data feeds back into whether routing gets switched on at all.

Our gateway's own source states this in the doc comment above the type, and it is the clearest statement of the problem we have:

"Whether the caller actually executed the recommended plan. Without this a plan the caller declined for safety reports identically to one it executed, so routingActivationEvaluation counts refusals as acceptances."

Follow the consequence through. An activation evaluation looks at recent outcomes and decides whether routing is behaving well enough to be trusted in active mode. If refusals are counted as acceptances, the evaluation is reading a population that includes every request where the router's advice was ignored — and concluding, from outcomes the router did not cause, that the router is safe to enable. That is not a slightly noisy metric. It is a control loop reading the wrong signal, in the direction of turning something on.

The contract: a boolean and a reason

The shape is deliberately minimal. From modelRouting.service.ts:

interface RouteOutcomePlanHonoured {
  honoured: boolean;
  reason: RouteOutcomePlanHonouredReason | null;
}

And the reason codes, quoted in full and in source order:

const ROUTE_OUTCOME_PLAN_HONOURED_REASONS = [
  "not_active",
  "model_deprecated",
  "model_not_executable",
  "already_requested",
  "no_selected_model",
  "execution_disabled",
  "provider_unavailable",
];

Two things about this list are worth borrowing even if you never touch our platform.

First, a boolean alone is nearly useless. "Not honoured" covers both "the runtime physically could not" and "policy said no", and those have opposite implications: one is an engineering constraint you might remove, the other is a governance decision you must not. A closed set of reason codes is what makes the boolean actionable.

Second — and this is a discipline point rather than a design one — we are quoting these literals rather than explaining what each means, because there is no in-repository documentation defining them individually. The names are self-describing and nothing more. If you build a taxonomy like this, write down what each code means at the point of definition, because in two years the person reading your dashboard will not be the person who chose the strings.

Normalisation: lenient where it should be, strict where it matters

The parsing behaviour encodes a judgement worth copying. The normaliser returns null unless honoured is a genuine boolean — a missing or malformed honoured flag produces no record at all, rather than a default. But an unrecognised reason is coerced to null rather than rejecting the whole record.

That asymmetry is right. The boolean is the load-bearing fact and a guessed default would poison the arithmetic. The reason is diagnostic colour, and losing an unknown code is better than losing the record it was attached to — which matters when the executor is deployed independently and may be running a newer version than the receiver.

Classifying the mismatch: five outcomes, one alarm

Recording whether a plan was honoured is half the job. The other half is deciding whether a divergence was expected. Our gateway classifies every planned-versus-executed comparison into one of five parity values:

The design decision underneath is the interesting one. When the classification comes out as unexpected_mismatch, our gateway refuses to forward the outcome at all, returning a report failure with the reason route_outcome_unclassified_model_mismatch.

That is a deliberate choice to lose data rather than record data you cannot interpret. It is the right default for anything that feeds an automated decision: an outcome you cannot explain, admitted into a training or activation dataset, is worse than a gap you can see. If you are building this yourself, decide explicitly which way you want to fail — and if you choose to admit unexplained records, quarantine them rather than blending them.

Who reports it, and the honest limit of what we know

Plan-honoured is reported inward. It cannot be inferred by the planner, because the planner does not observe execution; it has to be asserted by whatever actually ran the model, and it arrives on the outcome-reporting endpoint alongside the executed model, token counts, observed cost and latency, and whether execution succeeded.

Being precise about what we can and cannot verify: in our codebase the field is read from the inbound outcome payload on an internal route, from a trusted source system. We can point at the receiving code. We are not going to claim a specific named service writes it, because we could not demonstrate that from the source, and an article whose whole value is precision should not blur the one thing it cannot check.

Note also a real gap, which we would rather state than have someone find: the ingestion adapter that normalises stored observation rows into outcome records does not currently carry the plan-honoured field. So the completeness of your plan-honoured coverage depends on which path an outcome arrives by. If you implement this pattern, check every ingestion path, not just the one you wrote first — a field that is present on the primary path and absent on the replay path produces a dataset that looks complete and is not.

BlueBear Route Outcome Plan-Honoured Contract

An outcome record is what turns routing from an opinion into an audit trail. This is the payload BlueBear's gateway emits after a routed request, with the field names it actually uses. The values are an illustrative example for review; they are not production measurements.

{
  "schemaVersion": "bluebear-route-outcome/v1",
  "outcomeId": "example-outcome-id",
  "requestId": "example-request-id",
  "routePlanId": "example-plan-id",
  "tenantId": "example-tenant",
  "selectedModel": "example-small",
  "executedModel": "example-large",
  "routePlanningMode": "active",
  "canaryExecuted": false,
  "executionSucceeded": true,
  "routerConfidence": 0.86,
  "qualityStatus": "accepted",
  "observedLatencyMs": 0,
  "observedCostUsd": 0.0,
  "inputTokens": 0,
  "outputTokens": 0,
  "userRegenerated": false,
  "toolCallFailed": false,
  "planHonoured": { "honoured": false, "reason": "model_not_executable" }
}

Read the example carefully: selectedModel and executedModel differ, and planHonoured.honoured is false with a stated reason. That is the entire point of the contract. Without the last field this record is indistinguishable from one where the plan was followed, and any savings model built over it will count a refusal as an acceptance.

The declared refusal reasons

These are the literal values in ROUTE_OUTCOME_PLAN_HONOURED_REASONS. They are quoted rather than glossed: the codes carry no in-repository definitions, and inventing meanings for them would be exactly the kind of confident guess this contract exists to prevent.

not_active
model_deprecated
model_not_executable
already_requested
no_selected_model
execution_disabled
provider_unavailable

Parity classification

Three numbers to report, not one

1. Projected saving — over every request that produced a recommendation, assuming all were followed.
2. Realised saving — over exact_match outcomes only.
3. Recoverable saving — the difference, broken down by refusal reason, which is the engineering roadmap.

Reporting all three is also more persuasive than reporting one. A proposal that says "projected 34%, realised 19%, and here is where the other 15 points went" survives scrutiny in a way a single percentage does not.

What a usable outcome record contains

The payload we emit carries, among other fields: an outcome and request identifier, the plan identifier, the selected model, the executed model, the planning mode, whether a canary executed, an optional evaluation pair identifier and role, whether execution succeeded, router confidence, a quality status, observed latency and cost, input and output tokens, whether the user regenerated, whether a tool call failed, and the plan-honoured object.

Three of those are the ones teams usually lack and later wish they had:

• userRegenerated — the cheapest quality signal in existence. A user pressing regenerate is telling you the answer was not good enough, and it costs nothing to record. A routing change that reduces model cost by 20% and raises regeneration by 30% has made things worse, and cost dashboards alone will call it a win.
• toolCallFailed — the failure mode that separates models most sharply. See tool calling on a budget.
• The evaluation pair identifier and role — emitted only as a pair, baseline and recommendation together. A comparison whose two halves cannot be joined is not a comparison.

Rebuilding the savings model correctly

Once plan-honoured exists, the arithmetic changes shape. Instead of one number you have three, and the gap between them is the finding:

1. Projected saving — computed over every request that produced a recommendation, assuming all were followed. This is the number most routing business cases actually report.
2. Realised saving — computed only over exact_match outcomes. This is the honest one.
3. Recoverable saving — projected minus realised, broken down by refusal reason. This is the roadmap: model_not_executable is an engineering problem, execution_disabled is a configuration problem, not_active is expected, and provider_unavailable is a reliability problem that may be someone else's.

Reporting all three is also simply more persuasive than reporting one. A routing proposal that says "projected 34%, realised 19%, and here is where the other 15 points went" is credible in a way that "34%" is not.

Fold the result into cost per accepted outcome rather than cost per call — retries and human rework belong in the denominator. The cost-per-outcome calculator and cost per completed workflow cover that framing.

Do this next

Take last week's routing data and compute one number: the share of requests where the executed model equals the planned model. If you cannot compute it because you never recorded the executed model next to the planned one, that is the finding, and it is a two-field change to fix. Every routing decision you make afterwards will be built on it — including whether to run shadow mode at all.

Questions people actually search for

why does my llm router pick a different model than the one that ran

Because planning and execution are separate systems, and the executor has information and constraints the planner did not. The common causes are a warm runtime that cannot switch models mid-session, a model that has been deprecated at the provider since the catalog was built, a credential for the recommended model that is unavailable, and an explicit user or workspace override that outranks the recommendation. All four are correct behaviour. The defect is not recording which one happened.

how do I measure actual llm routing savings

Compare planned cost against executed cost on the same request, and use only the requests where the plan was actually honoured to attribute savings to routing. A savings model built on planned models alone answers "what would we have saved if every recommendation had been followed", which is a different and always larger number than "what did we save". The size of the difference is your plan-honoured rate.

what should a route outcome record contain

At minimum: a request identifier, the plan identifier, the model that was selected, the model that actually executed, the mode the planner was in, whether execution succeeded, observed cost and latency, input and output tokens, and an explicit boolean for whether the plan was honoured with a reason when it was not. The last field is the one almost everyone omits and the one that makes the rest interpretable.

planned model vs executed model mismatch

A mismatch is normal and expected in several modes — in shadow mode the executed model is supposed to differ from the recommendation, and a fallback is supposed to diverge. The thing to alarm on is an unexplained mismatch: planned and executed differ and no mode, fallback, or declared refusal accounts for it. That is either a bug in the executor or a bug in your recording, and both matter.