Disclosure: BlueBear sells an AI agent platform and therefore has a commercial interest here — we are one of the vendors this checklist is aimed at. It is written to be used against us. Where we give an example of a good answer we quote our own implementation, and where our own answer is weak — the outcome records — we say so.
The short answer: ask questions whose answers are artefacts
Security evaluations in this category go wrong in a predictable way. The buyer asks capability questions — "do you have audit logging", "is the platform multi-tenant", "do you support least privilege" — and the honest answer to almost all of them is yes. Everyone passes, nothing is learned, and the differences that matter surface six months later.
The fix is to ask for artefacts instead. An artefact question has one answer: it is produced or it is not. Below are twenty, grouped by what they are actually testing, each with the answer that should worry you.
Group 1 — Identity: who is acting, and inside what
| Ask | Answer that should worry you | Why it matters |
|---|---|---|
| Where is the tenant or workspace resolved from? | Anything involving a field in the request body | A request that can nominate its own boundary can nominate someone else's |
| How are API keys stored? | "Encrypted at rest" without further detail | Encrypted is reversible. A hash is not. Ask which. |
| Can an agent credential be revoked without a deploy? | "You would redeploy the agent" | This is your containment time in an incident |
| Can one machine credential create another? | Yes, or an unclear answer | A key that can mint keys is a privilege escalation path |
| Can a credential act across tenants? | Anything other than a firm no with a mechanism | The single worst failure mode in a multi-customer system |
| Show me what a caller can learn about its own credential | Nothing available | If a caller cannot introspect its own scopes, neither can you audit them |
A concrete example of what a good answer sounds like, from our own gateway: keys are minted with the prefix bbk_ followed by 96 hexadecimal characters from 48 random bytes; what is stored is a SHA-256 digest, so a database read cannot recover a usable key; the plaintext is shown once, which is why regeneration rather than retrieval is the recovery path; keys carry a subject_type of tenant or user; there is a database-level partial unique index enforcing one active key per tenant and subject rather than relying on an application check that can race; an API key can never manage keys at all, which returns a specific error rather than a generic denial; and cross-tenant minting is refused unless a human platform administrator is behind the session. Whether or not you buy ours, that is the shape of an answer worth accepting: specific, mechanical, and checkable.
Group 2 — Authority: where the decision is made
| Ask | Answer that should worry you | Why it matters |
|---|---|---|
| Where is a tool permission enforced? | Anything mentioning the system prompt | A prompt is a request to the agent, not a control over it |
| Can permissions be expressed per operation, not per system? | "Per integration" | Per-system access is the blast radius problem restated |
| Does the agent ever hold the credential? | Yes, or "it depends" | If it does, revocation is a deploy and the actor is also the witness |
| What happens when the policy service is unavailable? | "It continues" | Fail-open is a decision. Make sure it is one somebody made. |
| How is an unknown permission treated? | Permissively | Fail-safe defaults are cheap and reveal the design philosophy |
| Can a specific action class require human approval? | Only globally, or not at all | All-or-nothing approval gets switched off within a month |
| Is there a distinction between granted and effective permissions? | The question is not understood | Implied permissions are where quiet over-granting lives |
Two of these have concrete answers in our implementation that illustrate what to look for. Our permission registry defines a fixed set of named scopes with a risk tier attached, reads sitting at a lower tier than management and execution — and a permission that is not in the registry is treated as high risk rather than unknown-and-therefore-allowed. Granted scopes and effective scopes are distinct concepts throughout, because one permission can imply another and the difference is exactly where over-granting hides. On the second axis, our tenant privacy resolution deliberately fails closed: a tenant classified as privacy-sensitive with no resolvable policy baseline raises an error rather than defaulting to a permissive path.
Group 3 — Evidence: what you can prove afterwards
| Ask | Answer that should worry you | Why it matters |
|---|---|---|
| Show me a stored evidence record from a real failed run | A schema, a screenshot, or a successful run | Failure records are where systems are thinnest and incidents start |
| Is the record typed and versioned? | Free text, or "it is JSON" | JSON is not a contract. Ask for the version field. |
| Are records sampled? | Yes, or the question is unexpected | Correct for telemetry, fatal for evidence, and rarely volunteered |
| Is the policy decision recorded with its reason? | Only the decision | Without the reason you cannot tell a working control from a misfiring one |
| Show me the read path and what enforces the tenant filter | "The application filters by tenant" | A filter in application code is one refactor from not being there |
| What is retained of prompts and tool arguments, by default? | Everything | Retaining everything turns an audit control into a data-protection liability |
| Can a customer's records be exported on their own? | "We would write a query" | This becomes a contractual obligation the moment you resell |
Our own answers, including the weak one. Evidence records carry explicit schema versions — the dependency evidence a plan produces is versioned, and the security posture evidence a tenant can pull is versioned separately — so a consumer can tell which contract it is reading. Content retention is off by default: the columns for request and response content exist on the invocation records and are deliberately not written, so enabling retention later is a write-path change rather than a migration, and the policy is a tenant decision rather than a platform default. And the weak answer: our action-outcome store is built, tested and mounted, and no producer currently writes to it, so it is empty. If you asked us for a stored outcome row today, we could not produce one. That is exactly the question we think you should ask everyone.
Three probes that are worth more than the twenty questions
- Ask for a failed run, end to end. Not a demo of the product working. Every system looks governed on the happy path.
- Ask what happens when the policy layer is unavailable. The answer reveals whether anyone has thought about the boundary as a dependency rather than as a feature.
- Ask them to revoke something while you watch, and time it. This is the single most predictive test of incident readiness, and it takes four minutes.
If you operate in a regulated industry
Add three: who inside the vendor can read the evidence store directly; what the process is for changing its schema; and what the stated retention is, as a number. These are control questions rather than capability questions and they are frequently answered badly by products built for engineering teams. Security posture evidence for buyers covers the artefacts to collect.
If you are a platform or engineering lead
Run this checklist against your own system before you run it against a vendor. The exercise usually reveals that two or three items are things you can fix in a week and were going to pay someone else to solve. What breaks when agents get production access covers the underlying changes, and the existing enterprise AI agent platform evaluation scorecard covers the non-security half of the same review.
If you resell this to customers
Your customers will eventually run a version of this checklist against you, so the useful move is to prepare the artefacts now: a sample evidence record, the read-path explanation, the retention number, and a per-customer export. White-label brand workspaces explained covers the boundary those artefacts describe.
Do this next
Send the three probes to whichever vendor is furthest along in your process, and give them a week. The responses will separate the shortlist faster than any scoring matrix. Then read security posture evidence for buyers for the artefact list, or data integrity at enterprise integration points if your review is specifically about connecting agents to systems of record.
Questions people actually search for
- ai agent security checklist
Group the questions into three: identity, authority and evidence. Identity - is the boundary established from the credential or from something in the request, how are keys stored, how fast can one be revoked. Authority - where is the permission decision made, can it be expressed per operation, what happens when a policy is unavailable. Evidence - what is recorded, is it typed, is it complete or sampled, and can you see a stored record from a failed run. Every one of those has a demonstrable answer.
- what to ask an ai vendor about security
Ask questions whose answers are artefacts. "Show me a stored evidence record from a failed run" is a better question than "do you have audit logging". "Show me the read path and what enforces the tenant filter" is better than "is the platform multi-tenant". A vendor answers capability questions optimistically because the honest answer to almost any capability question is yes; artefact questions have only one answer and it is either produced or it is not.
- how do you evaluate ai agent platform security
Test the boundary rather than reading the feature list. Three probes settle most of it. Ask where the tenant or workspace is resolved from - if any part of it comes from the request body rather than the credential, stop there. Ask where a tool permission is enforced - if the answer involves the system prompt, it is a request rather than a control. And ask to see a failed run end to end - failure records are where systems are thinnest and where every incident starts.
- what security certifications should an ai platform have
Certifications establish that a process exists and was audited, which is genuinely worth something and answers a different question from the one you are asking. A certification does not tell you where the permission decision is made, how an agent credential is scoped, or whether failure records are written. Treat certification as a gate on the vendor as a company and the questions in this article as the gate on the product, and do not let one substitute for the other.
- is prompt injection a real risk for enterprise ai agents
Yes, and the useful question is not whether it can be prevented but what it can reach when it succeeds. Instructions arriving inside data an agent reads can influence what it does next, and no filtering approach reliably eliminates that. So the design assumption is that manipulation will sometimes work, and the control that matters is the one that bounds the consequence: scoped credentials and permission decisions evaluated at the execution boundary rather than in the prompt.