Disclosure: BlueBear builds an AI agent platform and an MCP gateway, so we have a commercial interest in this topic. This article gives a decision framework and a method; it deliberately contains no model benchmark numbers, because any such table would be out of date within weeks and we have not run one at a scale that would justify publishing it.
The short answer: three properties predict it
The question "which model should we use?" has no useful answer at the agent level, because an agent is not one task. It is six to fifteen model calls doing quite different jobs — one request, many model calls pulls that apart — and those jobs have genuinely different quality requirements.
Rather than a leaderboard, score each step on three properties. They predict cheap-model viability far better than any benchmark, and unlike a benchmark they do not expire.
| Property | Cheap model likely fine | Cheap model risky |
|---|---|---|
| Output space | Closed — one of N labels, a fixed schema, a boolean | Open — free prose, a plan, code |
| Error detectability | Machine-checkable — schema validation, enum membership, a downstream call that fails | Only a human can tell, and only sometimes |
| Compounding | Self-contained — a wrong answer affects one field | Feeds later steps — a wrong plan wastes the whole run |
Three yeses in the left column is a strong cheap-model candidate. Three in the right is where the strongest model available earns its price. Mixed is where you have to measure — and the measurement method is is a cheaper model good enough?
Step by step
Extraction against a known schema — strong candidate
Pulling an invoice number, a date, and a total out of a document. Closed output space, machine-checkable against the schema, and errors are contained. This is the most consistently successful cheap-model substitution in agent work, and constrained decoding or structured-output modes narrow the gap further by making malformed output impossible rather than unlikely.
The caveat is not the model, it is the fields. Extracting a date is easy; extracting "the effective date, unless a later amendment supersedes it" is a judgement task wearing an extraction costume. If a field's definition contains the word "unless", it belongs in the open-output column.
Classification into known labels — strong candidate
Intent detection, sentiment, triage category, "is this in scope". Closed output space, checkable against the label set, self-contained. Guardrails belong here too, which matters because guardrails run on every turn and are commonly configured on the same model as the agent they protect — roughly doubling per-turn cost for a check a much smaller model would perform just as well.
The place classification does get hard is when labels are close together or the taxonomy is under-specified. That is a taxonomy problem and a larger model will paper over it rather than fix it.
Routing and branching — strong candidate, with a caveat
Deciding which of four sub-workflows handles a request. Closed and checkable — but compounding, since a wrong branch wastes everything downstream. Two of three properties favourable. Worth a cheap model, worth an explicit measurement, and worth recording branch accuracy as its own metric rather than folding it into overall success.
Query rewriting for retrieval — reasonable candidate
Turning a user question into a search query. Semi-open output, and the error is detectable indirectly through retrieval quality. Evaluate it by what retrieval returns, not by reading the query.
Tool selection — measure it, and measure it at your real tool count
Choosing among the tools you have bound. Closed output space and immediately checkable, but strongly compounding — a wrong tool call costs the call, the result, and at least one more iteration.
This is where the naive test misleads most. Tool selection degrades sharply with tool count, and a demo with three tools tells you nothing about production with twenty. Test at the real count, with the real descriptions.
Summarisation and context compaction — depends on the consumer
For a human reader, open-ended generation and quality shows. For an agent's own working memory it is compression, and the failure mode is quiet: a summary that drops the single constraint the user stated reads perfectly well and breaks the run three steps later.
Evaluate compaction by the success rate of the step after it. Never by reading the summary and finding it reasonable — of course it reads reasonably, that is what the failure looks like.
Multi-constraint planning — usually needs the strong model
Decomposing a request into steps under several interacting constraints. Open, hard to check, maximally compounding. All three properties unfavourable.
The productive move here is usually not a cheaper model but less planning: many agents plan dynamically over a workflow whose shape is fixed. If your agent produces the same plan every time, that plan is a constant and you are paying a model to rediscover it. Making it a coded workflow with model calls at the genuinely variable steps is a bigger saving than any model swap, and it removes a failure mode rather than trading one.
Final user-facing response — the last thing to touch
Open, judged by humans, and the part your customer actually experiences. Even where a cheaper model is adequate on average, the variance is what gets noticed — the tenth-percentile answer is the one that generates a complaint.
Not never. But last, with a real evaluation and a staged rollout, not first because it is the largest line item.
The decision table
| Step | Output | Error detectable? | Compounds? | Verdict |
|---|---|---|---|---|
| Schema extraction | Closed | Yes | No | Cheap model |
| Classification / guardrails | Closed | Yes | No | Cheap model |
| Routing / branching | Closed | Yes | Yes | Cheap, measured |
| Query rewriting | Semi-open | Indirectly | Somewhat | Cheap, measured |
| Tool selection | Closed | Yes | Yes | Measure at real tool count |
| Context compaction | Open | Downstream only | Yes | Measure by the next step |
| Multi-constraint planning | Open | No | Strongly | Strong model, or remove the planning |
| Final response | Open | Human only | Terminal | Strong model until proven otherwise |
Why this maps onto eligibility, not ranking
A step-level model policy is not the same thing as a router picking the cheapest adequate model per request. They are complementary and they answer different questions.
Eligibility is governance: which models are permitted at all for this step, this tenant, this compliance class. That is a decision a human makes and writes down. Ranking is optimisation within the permitted set, and it can be automated. Fallback is exception handling when the ranked choice is unavailable. Collapsing all three into one opaque score is what makes an incident unreconstructable six weeks later — the routing policy guide works through the policy fields that keep them separate.
The table above is an eligibility input. It says which models may be considered for each step. What happens inside that permitted set is a separate, safer problem.
Do this next
List every model call in your highest-volume agent and score each on the three properties. You will typically find that a majority of calls are closed-output, machine-checkable and non-compounding — and that they are not the calls you were planning to optimise. Take the highest-volume one to a real acceptance test: is a cheaper model good enough?
Questions people actually search for
- which parts of an ai agent need the best model
The ones where the output space is open and an error is not detectable by machine: the final user-facing response, multi-constraint planning, and any step whose mistakes will not surface until later. Steps with a closed output space and a machine-checkable result — classification into known labels, extraction against a schema, routing to one of N branches — are where smaller models most reliably hold up, because a wrong answer is caught rather than shipped.
- can a small model do tool calling
Frequently yes for a small, well-described tool set, and this degrades faster than any other capability as the tool count grows. Tool calling is really two skills — choosing the right tool, and emitting arguments that satisfy the schema. Smaller models tend to keep the second and lose the first. If you plan to use a cheap model for tool selection, test it with your real tool count, not with three toy tools. See tool calling on a budget.
- should I use a cheap model for summarisation
It depends entirely on what happens next. Summarising a document for a human reader is an open-ended generation task and quality differences show. Summarising a conversation so an agent can keep working is compression with a downstream consumer — and its failure mode is subtle, because a summary that silently drops the one constraint the user stated does not look wrong. Test summarisation by the success of the step after it, not by reading the summary.
- how do I decide which model for which task
Score each step on three properties: is the output space closed or open, is an error machine-detectable, and does the step compound into later steps. Steps that are closed, detectable and non-compounding are cheap-model candidates. Steps that are open, undetectable and compounding are where the strongest model earns its price. Most agents have more of the first kind than their authors expect.