Disclosure: BlueBear builds an AI agent platform and an MCP gateway, so we have a commercial interest in this topic. This article is a method. It contains no model comparison results, because publishing a benchmark we have not run at defensible scale would be exactly the kind of unverifiable claim it warns against.
The short answer: define acceptance first, or the answer is decided before you start
"Is model B good enough?" is not answerable as stated, and the failure to notice that is why most model-substitution decisions are made on vibes. Good enough for what, at what error rate, judged by whom, on what traffic?
The single most important discipline is ordering: write down the acceptance criterion before you look at any output. Once you have seen the cheaper model's answers, you will find reasons they are fine or reasons they are not, depending on what you were hoping. This is not a failure of character; it is how evaluation works when the criterion is negotiable.
The method, in six steps
1. Pick one step, not the agent
Evaluate one model call in isolation — extraction, or classification, or the final response. Evaluating "the agent" mixes several substitutions and you will not know which one moved the result. Which steps need a frontier model covers which to start with; a closed-output, machine-checkable, non-compounding step is the right first target.
2. Write the acceptance criterion down
It must be checkable by someone who was not in the room. Three shapes work:
- Exact match against a known-correct answer. Only for genuinely closed tasks.
- Rubric — a short list of conditions the output must satisfy. Best for open outputs. Keep it under five conditions or your labellers will disagree with each other.
- Downstream success — whether the next step worked. The right criterion for compaction, query rewriting, and anything whose quality is only visible later.
Then state the tolerance: "no more than 2 percentage points of acceptance lost." That number is a business decision, not a technical one, and it should be made by whoever owns the outcome.
3. Build the set from real traffic
Sample from production logs, not from examples you write. Invented examples are systematically cleaner than reality — that is precisely why they are easy to write — and a set of clean examples will tell you the cheap model is fine when it is not.
Stratify deliberately. If 10% of production traffic is the hard document type, make it 10% of your set, or oversample it and weight the result. Record the sampling window and method alongside the set; an evaluation whose provenance is unknown cannot be re-run.
4. Run both models over the same set
Same inputs, same prompt, same tools, same temperature settings. Change one variable. If you also "improved the prompt for the small model", you have not measured a model swap and cannot attribute the result.
Keep the outputs. When someone asks in three months whether the swap was safe, the paired outputs are the answer and they are cheap to store.
5. Grade blind
The grader must not know which model produced which output. If a human grades, shuffle and strip labels. If a model grades, calibrate it first against human labels on a sample so you know its agreement rate, and never use a judge from the same family as either candidate — self-preference is a documented bias in LLM-as-judge setups.
6. Report three numbers, not one
- Acceptance rate for each model, with the difference and its uncertainty.
- Correction rate — how often a human would have to intervene. This is the number that converts quality into money.
- Cost per accepted output — not cost per call. A model that is 30% cheaper and needs correction twice as often may be more expensive in total.
The equivalence test: the right frame for a downgrade
Most people run a superiority test by reflex: is B better than A? For a downgrade that is the wrong question, and it produces the wrong conclusion twice over.
You are not trying to show the cheaper model is better. You are trying to show it is not worse by more than a margin you can afford. That is an equivalence — or more precisely a non-inferiority — test, and it has a different structure:
- Declare the margin first. "We will accept up to 2 percentage points of acceptance loss for this saving." Written down, dated, agreed by the person who owns the outcome.
- Run the paired comparison on the same inputs.
- Ask whether the observed loss is confidently within the margin — not whether the two models differ. Two models can be statistically indistinguishable on a small set purely because the set is small, and "no significant difference" is not evidence of equivalence. It is often evidence of an underpowered test.
That last distinction is the one that matters and the one most commonly got backwards. A test that fails to detect a difference has not demonstrated there is none.
Note also that the margin is a rate and rates hide distributions. A model that loses 2 points spread evenly across all traffic is a different product from one that loses 2 points concentrated entirely in your largest customer's document type. Break the result down by input class before accepting it.
The four ways this gets rigged by accident
| Failure | What it looks like | Fix |
|---|---|---|
| Criterion set after seeing outputs | "Actually this is fine for our use case." | Write and date the criterion first. |
| Clean test set | The cheap model scores far better in the test than in production. | Sample real traffic, stratified by input class. |
| Two variables changed | Prompt tuned for the new model "to be fair". | Change one thing. Tune afterwards, and re-measure. |
| Cost per call, not per accepted output | A 40% saving that evaporates in review time. | Put corrections in the denominator: the cost-per-outcome calculator. |
A fifth, subtler one: evaluating on the traffic you already handle well. If your set is drawn from successful runs, you have excluded the population where the models most differ. Sample from all runs, including the failures.
What shadow mode can and cannot contribute
If you have a routing layer in shadow mode, it will give you a projected cost saving on real traffic — genuinely useful, and it costs nothing to collect. What it cannot give you is the quality half, and the reason is simple: in shadow mode the cheaper model never ran, so there is no output to grade.
Treat them as two separate gates that both have to pass. Shadow answers "is the saving big enough to be worth an evaluation?" This method answers "is the quality loss small enough to take the saving?" Neither substitutes for the other, and running the evaluation first on a step where the saving turns out to be trivial is a common waste of a fortnight. Shadow-mode routing covers the first gate.
Recording the decision so it survives
A model decision that exists only in someone's memory will be re-litigated, usually by someone who was not there. Record, in one place: the step, the models compared, the acceptance criterion and margin, the test set's provenance and size, the three result numbers, the date, and the person accountable.
Then record the review trigger, which most teams forget. Model versions change under stable aliases, providers deprecate, and your traffic mix drifts. A decision taken on a set sampled in March is a claim about March. Setting an explicit re-test date — and re-running the same set — turns a one-off evaluation into something you can still rely on.
This is the same discipline that makes routing evidence usable rather than decorative: an outcome record you cannot interpret later is not evidence, which is why our own route outcomes carry the mode, the confidence, the fallback reason and an explicit flag for whether the plan was honoured rather than just the models involved. Plan-honoured accounting covers why.
Do this next
Pick your highest-volume classification or extraction step. Write one sentence defining acceptance and one number for the tolerance you will accept, before you look at anything. Then sample 200 real requests from the last month. That is the whole first day, and everything after it is mechanical.
Questions people actually search for
- how do I know if a cheaper model is good enough
Write down what "good enough" means before you look at a single output, build a test set from real production traffic rather than examples you invent, run both models over it, and count how often a human would have to correct the result. If the cheaper model's correction rate is within your stated tolerance, it is good enough for that step. Any evaluation that defines acceptance after seeing the outputs will find whatever the person running it hoped to find.
- how many examples do I need to test a model
Enough that a difference you care about is distinguishable from noise, which depends on your base rate and your tolerance. Practically: if you would act on a 5-point difference in acceptance, a few dozen examples will not tell you anything reliable and a few hundred usually will. More useful than chasing a number is making sure the set covers the hard cases in the same proportion production does — a large, easy set is worse than a smaller representative one.
- what is equivalence testing for llm model swaps
Instead of asking "is B better than A", equivalence testing asks "is B no worse than A by more than a margin I have declared acceptable". You state the margin first — say, no more than 2 percentage points of acceptance lost — and the test either clears it or does not. This is the correct framing for a downgrade, because you are not trying to prove the cheaper model is better; you are trying to prove the loss is small enough to be worth the saving.
- can I use an llm to judge model quality
Yes, with two conditions. Calibrate the judge against human labels on a sample first, so you know its agreement rate rather than assuming it. And never judge a model with itself or with a sibling from the same family, because self-preference is a documented bias. A judge is a measuring instrument; an uncalibrated instrument produces confident numbers that mean nothing.