Disclosure: BlueBear sells software that addresses several of the problems below, so we have a commercial interest. Three of the five changes described here are architecture decisions you make regardless of what you buy, and the article says where nothing needs buying.
The short answer: five properties change, and none of them are model quality
Teams prepare for production by testing accuracy. That is the wrong preparation, because accuracy is the property that changes least when an agent goes from drafting to doing. Five other things change on the same day:
| What changes | Before: the agent drafts | After: the agent acts |
|---|---|---|
| Authority | A person reads the output and decides. The person is the actor. | The agent is the actor, on someone's behalf. Somebody real stands behind every action. |
| Blast radius — how far the damage from one mistake can reach | Bounded by what a person will act on | Bounded by what the credential can reach, which is usually far more than the job needs |
| Reversibility | Nothing happened; discard the draft | A record changed, a message went out, money moved. Some of that cannot be undone. |
| Attribution | Irrelevant — a person did it | Essential, and usually broken, because the agent inherited the application's credential |
| Exceptions | A person hesitates and asks someone | The system continues. Hesitation is not a behaviour software has unless you build it. |
Each of these is worth a paragraph, because the mitigations are different and only one of them is a purchase.
1. Authority: the agent is now acting as someone
When an agent writes to a system of record, that write carries authority. Somebody's permissions allowed it, and somebody is accountable for it. That is a governance fact rather than a technical one, and it is the reason this whole class of problem exists.
The practical consequence is that "the agent did it" must resolve to a real chain: this agent, acting for this human or this team, inside this boundary, permitted by this rule. If any link in that chain is missing, you have an action nobody owns. What AI agent identity is covers the first link in plain terms.
2. Blast radius: what the credential reaches, not what the job needs
This is the change with the worst ratio of likelihood to consequence, and it is almost always an accident of convenience.
An agent usually inherits its access. It runs inside an application, so it uses the application's database connection or API key — and that credential was scoped for the application, which does many things the agent does not. The agent's job might be "look up an order and update its shipping address". The credential's reach might be the entire customer table.
The important point is that this is not a hypothetical prompt-injection scenario. It is a standing condition. Any bug, any confusion, any successfully manipulated instruction, expands to fill whatever the credential permits. Narrowing that set is the single highest-value security change available, and it is architecture rather than a product: named operations on named connections, with the credential supplied at the boundary rather than held by the agent. Least privilege for AI agents covers the design.
3. Reversibility: the question nobody asks in the demo
Sort every action the agent can take into three buckets: freely reversible, reversible with effort, and not reversible. Sending an email to a customer is not reversible. Posting a journal entry is reversible with effort and a paper trail. Updating a draft record is freely reversible.
That sort takes half an hour and it changes the design, because the right control differs by bucket. Freely reversible actions need recording and nothing else. Actions reversible with effort need recording plus the ability to find every one of them quickly. Irreversible actions need a human in the path, or a delay window, or both.
A related practical trap: agents retry. More than the systems they call expect. If a write is not idempotent — if running it twice produces two effects rather than one — an ordinary timeout becomes a data-integrity incident. Our own tooling treats this as a first-class property: an evidence record for an action carries an idempotency block with required, key, scope and a status that is one of claimed_by_gateway, missing or not_required, precisely so that "this action needs an idempotency key and does not have one" is a recorded, blocking condition rather than a surprise. Data integrity at enterprise integration points covers this properly.
4. Attribution: usually broken before you start
If the agent uses the application's credential, then in every downstream system everything the agent does looks exactly like everything the application does. You cannot answer "which actions were the agent's", cannot revoke the agent without breaking the application, and cannot scope its permissions separately.
The fix is a credential issued to the agent as an actor in its own right. In our gateway, tenant API keys carry a subject_type of tenant or user, are stored as a hash rather than as the key, and are revoked by a status transition scoped to the tenant — so revoking one actor is a local operation rather than a redeploy. The design principle transfers regardless of what you use: an identity you cannot revoke on its own is not an identity.
5. Exceptions: hesitation is not a default behaviour
When a person meets a case that is not quite right, they hesitate and ask someone. Software does not do this unless it is built to. An agent meeting an ambiguous case does what it was going to do anyway, with a plausible explanation.
Designing an exception path is therefore an explicit act: which conditions stop the run, who they go to, what that person is shown, and what happens if nobody responds. The last of those is the one most often skipped and the one that causes the outage. Human-in-the-loop approval patterns is the existing guide, and healthcare AI agents need exception queues is the operational treatment.
The failure that matters throws no error
This deserves emphasis because it inverts normal monitoring instincts.
The dangerous failure with an agent is not a crash. It is a permitted action that should not have been taken: the agent updated the wrong record, cancelled the wrong subscription, emailed the wrong customer. Everything succeeded. Every status code was 200. Every dashboard is green.
Log-based alerting is tuned for exceptions and will not see this. What sees it is a record of authority — what was requested, what was permitted, what changed — reviewed against expectation, plus an outcome recorded by whoever notices in the business. That is the argument for an audit trail as a security control rather than only a compliance one. What an AI agent audit trail is covers what the record contains.
If you operate in a regulated industry
Two additions. First, the exception path is usually itself a control you will be asked to evidence, which means the approval record needs to capture what the approver was shown, not merely that they clicked. Second, reversibility bucketing maps directly onto materiality, so it is worth doing with your risk function in the room rather than presenting to them afterwards. Audit trails for financial services covers what a review will ask for.
If you are a platform or engineering lead
Sequence: the credential first, then the reversibility sort, then the recording. The credential change is the one that shrinks the problem — every subsequent control is cheaper against a narrow surface. The security evaluation checklist is the vendor-facing version of the same questions.
If you resell this to customers
Your blast radius question has an extra dimension: one mistake can cross customers. That makes the boundary a contractual property rather than an engineering preference, and it is worth being able to demonstrate rather than assert. Multi-tenant isolation for buyers covers what to demonstrate.
Do this next
List every action your agent can take and sort it into the three reversibility buckets. It takes half an hour, it is the input to every other decision here, and it usually reveals that the irreversible list is short enough to gate individually. Then read least privilege for AI agents for the credential change, or the breach response playbook for what to have ready before you need it.
Questions people actually search for
- what happens when an ai agent gets access to production systems
Five things change at once, and none of them are about the model. The agent gains authority, so its actions carry someone real behind them. Its blast radius becomes whatever its credentials can reach rather than whatever its job needs. Its mistakes may not be reversible. Its actions become hard to attribute if it inherited the application credential. And exceptions that used to be a person hesitating become a system continuing. Quality is the thing everyone tests and the thing that changes least.
- why do ai agents work in testing but fail in production
Because testing exercises the happy path with clean inputs and no consequences, and production supplies awkward inputs and permanent effects. The specific gap is that a test asks "did it produce the right answer" while production asks "was it allowed to do that, can we undo it, and can we tell who did it". A system can pass every accuracy test and fail all three, because accuracy and authority are unrelated properties.
- what is the biggest security risk of ai agents
Over-broad standing access, by a wide margin. An agent that inherits a credential reaching far more than its job requires converts any successful prompt manipulation into a large incident rather than a small one. It is a more consequential risk than the model saying something wrong, because the model saying something wrong is contained by whatever the agent can actually do - and if that set is unbounded, so is the damage.
- can an ai agent be tricked into doing something harmful
Yes, and the design assumption should be that it will be. Instructions arriving inside data an agent reads - a document, a ticket, an email, a web page - can influence what it does next, because a model does not reliably distinguish content from instruction. That is why constraints belong at the execution boundary rather than in the prompt: a rule the agent evaluates can be argued with, and a rule evaluated at the point the call leaves your system cannot.
- should ai agents have write access to production
Sometimes, and the useful framing is not whether but under what containment. Write access is reasonable when the action is reversible or a human approves it, the credential is scoped to the specific operations the job needs, every action is attributed and recorded, and there is a defined exception path for the cases the agent should not decide. Where any of those four is missing, the honest answer is to fix that first rather than to accept the risk implicitly.