Disclosure: BlueBear builds a control plane and an MCP gateway, so we have a commercial interest in these distinctions. The concessions below - where a framework or a plain gateway is genuinely sufficient - are the part of this article worth reading first.
The short answer: three words, three different jobs, usually all three eventually
These terms get used interchangeably in vendor conversations, which makes evaluations circular. They are not synonyms and they are not competitors. Each answers a different question:
| Term | The question it answers | Where it sits | Bought by |
|---|---|---|---|
| Orchestrator | What runs next? | Inside the workload | The team building the workflow |
| Gateway | Everything goes through one door — which door, and what happens at it? | At the edge of the workload | A platform or infrastructure owner |
| Control plane | Is this permitted, by whom, and what is recorded? | Above the workload | An operations, risk or platform owner |
A gateway is frequently how a control plane is enforced, because one choke point is the easiest place to evaluate a policy. But a gateway with no policy behind it is a proxy, and a control plane with no enforcement point is a document.
Orchestrator: sequencing, and nothing about authority
An orchestrator decides the order of work. It handles branching, retries, parallelism, and passing a result from one step to the next. Agent frameworks contain one; workflow engines are one; a hand-written loop is one.
What it deliberately does not do is decide whether a step is allowed. That is not an oversight — an orchestrator that also owned permissions would be enforcing policy inside application code, which is precisely the arrangement that becomes unreviewable once a second team appears.
An orchestrator alone is sufficient when one team owns the whole workflow, the tools it touches are read-only or low-consequence, and nobody outside the team needs to reconstruct what happened. That describes a large share of useful AI work, and adding layers to it is cost with no buyer.
Gateway: one door, and two very different doors in AI
A gateway is a single point every request passes through, which makes it the natural home for anything you want applied consistently: authentication, rate limits, key custody, logging, spend.
In AI systems there are two distinct gateways and they are routinely confused, which is the most consequential mix-up in this whole vocabulary.
| AI gateway (also: LLM gateway, model gateway) | MCP gateway (also: tool gateway) | |
|---|---|---|
| Governs | Traffic to model providers | Traffic to tools and systems of record |
| Decides | Which model, which provider, what fallback | Which connection, which operation, whose credential |
| Protects you from | Key sprawl, provider outages, runaway spend, lock-in | Over-broad access, unattributable actions, credentials in prompts |
| Typical record | Model, tokens, latency, cost, plan honoured | Agent, connection, operation, policy evaluated, result |
| Fails as | An expensive proxy, if no policy sits behind it | An expensive proxy, if credentials still reach the agent |
The short version: an AI gateway governs what the agent thinks with; an MCP gateway governs what it can touch. The Model Context Protocol — the open standard for describing a tool connection and its operations — is what makes the second one tractable, because it turns "the agent has database access" into "the agent may call these named operations". MCP gateway vs. API gateway covers why a conventional API gateway does not cover this by default.
A plain gateway alone is sufficient when your problem is genuinely about keys, spend and provider choice, and the actions your agents take are not consequential enough to need an authority record. Plenty of teams are in exactly that position and should not buy more.
Control plane: authority and evidence
The control plane is the layer that decides what is permitted and records what happened, kept separate from the runtime that does the work. It is defined properly in the control plane definition; the relevant point here is what makes it distinct from the other two.
An orchestrator knows the order. A gateway knows the traffic. Only the control plane knows the authority: which human or team stands behind this agent, what boundary it belongs to, which policy was evaluated for this specific step, and what must be retained afterwards. That is why the evidence question resolves here and not in the other two layers — the gateway can record that a call happened, but only the control plane can record that it was allowed.
A decision table
| If this is your live problem | What you need | What you do not need yet |
|---|---|---|
| The workflow has too many steps to manage by hand | An orchestrator or a framework | A gateway or a control plane |
| API keys are in four codebases and nobody rotates them | An AI gateway | A control plane |
| The bill is a single line and you cannot attribute it | An AI gateway with per-caller attribution | A control plane, unless the attribution question is also a compliance question |
| An agent holds credentials that reach far more than its job | An MCP or tool gateway with scoped connections | An orchestrator change |
| Someone asked what an agent did and it took two days to answer | A control plane with an evidence record | A different model |
| Three teams run agents and no one person can list what they reach | A control plane | More frameworks |
| You resell agents and need a per-customer boundary in a contract | A control plane with tenant isolation | Anything else, until that exists |
If you are a platform or engineering lead
The layering matters more than the labels. Two rules hold up well: enforcement belongs at a boundary the agent cannot bypass, and policy belongs somewhere it can be read without reading application code. Anything that satisfies both is a control plane regardless of what it is called; anything that satisfies neither is configuration regardless of what it is called.
The second thing worth deciding early is whether your plan and your execution are the same event. In our own gateway they are not: a route is planned, then a runtime executes, and the system records whether the plan was honoured — because a preferred model can be unavailable, a planner can exceed its 750 ms window, and a fallback can fire. Keeping those as two recorded events rather than one is what makes an incident reconstructable. The plan-honoured gap is the detailed version.
If you operate in a regulated industry
Buy the layer that produces the evidence, and buy it before the layer that produces the efficiency. An efficiency layer added first can be re-pointed later; evidence not recorded at the time cannot be reconstructed at all. What an audit trail must capture lists what to specify.
If you resell this to customers
The load-bearing question is which layer owns the customer boundary. If it is the orchestrator, the boundary lives in application code and one mistake removes it. If it is the control plane, the boundary is a property of the request. Multi-tenant isolation for buyers covers what to ask a vendor to demonstrate.
Do this next
Write down the one sentence that describes your actual problem — "we cannot attribute spend", "an agent can reach too much", "nobody can explain what it did" — and match it against the decision table above. The layer follows from the sentence. Then read what a control plane actually contains if the answer was the third row, or how to find your most expensive LLM calls if it was the second.
Questions people actually search for
- ai gateway vs mcp gateway
They govern different traffic. An AI gateway sits between your application and the model providers - it handles which model runs, API key custody, rate limits, retries and spend. An MCP gateway sits between your agents and the tools and systems they act on, using the Model Context Protocol, and handles which connections an agent may use, which operations within them, whose credentials are supplied, and what evidence is retained. One governs what the agent thinks with; the other governs what it can touch. Most organisations need both, and they are usually different products.
- difference between orchestrator and control plane
An orchestrator answers "what runs next" - it sequences steps, passes results, handles branching and retries. A control plane answers "is this permitted, with whose credentials, and what must be recorded". An orchestrator can run happily with no control plane, and that is the normal starting point. The two only need separating when a decision has to be reviewable by someone who was not in the room.
- do I need an ai gateway if I already use langchain
Not necessarily. A framework gives you the agent loop, tool abstractions and model clients, and for a single team running a single workload that is often enough. A gateway earns its place when you need something the framework deliberately does not own: keys held outside application code, spend attributed per customer or per workflow, a model change made without a deploy, or a single record of every model call across several teams and several codebases. If none of those is a live problem, adding one is premature.
- is an api gateway enough for ai agents
For traffic management, yes - routing, authentication, rate limiting and TLS work the same for agent traffic as for any other. What a conventional API gateway does not model is the thing that makes agent traffic different: a single logical action can be many calls, the action taken is chosen at run time rather than by the caller, and the evidence you need is about authority over a business action rather than about a request. You can extend an API gateway to cover that. You cannot get it for free.
- what sits between an ai agent and a database
Ideally a described connection with named operations and scoped credentials, rather than a database driver holding a shared password. That is what an MCP gateway or an equivalent tool layer provides: the agent asks for an operation it is permitted to call, the gateway supplies the credential and executes it, and the record names the agent, the connection, the operation and the outcome. The agent never holds the credential, which is what makes the permission revocable and the action attributable.