Disclosure: BlueBear is a vendor this list is designed to be used against, so we have a commercial interest in how it is framed. We have tried to make it uncomfortable for us as well: two of the eight artefacts are ones where our own answer is partial, and we say which.
The short answer: collect artefacts, not assurances
A security review that collects assurances produces a file. A review that collects artefacts produces a decision. The difference is whether the thing you receive would be different if the vendor were worse.
"We follow security best practices and encrypt data at rest" is true of every vendor in the market, so it carries no information. "Keys are stored as a hash, the plaintext is shown once, and an API key cannot manage keys at all" is a claim that could be false, which is what makes it worth having.
Here are eight artefacts. Ask for them in writing, in one message, early.
The eight
| # | Artefact | What it proves |
|---|---|---|
| 1 | A machine-readable security posture report for a tenant, with a version | Control state is generated from the system rather than described in a document |
| 2 | A stored evidence record from a failed run | The evidence layer covers the path incidents actually take |
| 3 | The record schema and its version field | Records are typed, and can be validated and migrated rather than only read |
| 4 | The read path, and what enforces the tenant filter | Isolation is a property of the request, not a convention in application code |
| 5 | The permission catalogue, with risk tiers and implied permissions | Least privilege is expressible and reviewable, not aspirational |
| 6 | Retention, as a number, per data class | Retention is a decision someone made rather than a default nobody set |
| 7 | A timed revocation, demonstrated | Containment is minutes rather than a release cycle |
| 8 | A per-customer export of one complete run | Your obligations to your own customers are servable |
1. A security posture report, generated rather than written
The distinction that matters: a description of what the product can do is marketing; a generated statement of what the account's controls currently look like is evidence.
A good one is versioned and machine-readable, so it can be pulled repeatedly and compared — which turns a point-in-time review into ongoing assurance without a second procurement exercise.
Concretely, ours reports per tenant across four control areas — tenant API keys, tenant model keys, connection credentials and cloud accounts — with a count of what is active or connected and what needs attention in each; a set of named controls each marked ok or attention_required; and findings carrying a code, a severity of low, medium or high, a count, a summary and a recommendation. The finding codes are fixed strings rather than prose, covering things like keys expiring soon, stale keys, expired keys, high-risk key permissions, and connection credentials needing attention. It is served as JSON by default or as Markdown on request, and an unrecognised format is rejected rather than guessed at. The shape is the point: fixed codes and a schema version mean two reports six months apart are comparable.
2. A stored record from a failed run
The most discriminating single request in a security review, and the one most often deflected into a demo of the product working.
Every system looks governed on the happy path. Failure is where the recording is thin, and failure is where every incident starts. Ask for a real one, from a real failed run, with the fields visible.
Ours is one of the two places our own answer is partial, so it is worth stating plainly: the last_error column on our session records is populated in none of the error rows in our production database — the sessions are marked failed and the field that would say why is empty. The schema is right, the producer does not write it. We would rather tell you that than show you a successful run and let you infer the rest, and we would encourage the same question of every vendor including us.
3. The schema, and its version
"It is JSON" is not a contract. A record with a declared schema and a version can be validated when written and migrated when the shape changes; free text can only be read by someone who already knows the format, which after two years is nobody.
Ask for the version field by name. In our system the evidence contracts carry explicit versions — the dependency evidence attached to a plan and the security posture report each have their own — so a consumer can tell which contract it is reading rather than inferring it from the fields present.
4. The read path, and what enforces the filter
Nearly every multi-tenant system has a tenant column. The question is what happens if a query forgets it.
"The application filters by tenant" means the isolation is one refactor from not being there. Better answers involve the boundary being a property of the request rather than a parameter a query supplies, uniqueness constraints that make cross-boundary collisions impossible at the database level, and a test that fails if a read path is added without the filter.
Worth asking in the same breath: where is the tenant resolved from? If any part of the answer involves the request body rather than the credential, stop and pursue that. Workspace boundary access control covers what a good answer looks like.
5. The permission catalogue, with risk tiers
Ask for the list of named permissions, not a description of the permission model. Two properties tell you whether least privilege is real:
Granularity — are permissions expressible per operation, or only per integration? Per-integration access is the blast radius problem restated as a feature.
Implied permissions — does granting one permission confer others? This is where quiet over-granting lives, and a vendor who distinguishes granted scopes from effective scopes has thought about it. Ours does, with a fixed registry of named scopes, a risk tier on each, and an unregistered permission treated as high risk rather than as unknown-and-therefore-allowed.
6. Retention, as a number
Per data class: evidence records, prompts and tool arguments, model outputs, telemetry. A single retention answer covering all four is a sign that nobody has separated them, which matters because they have genuinely different obligations — evidence needs years, prompt content usually needs to be absent.
The good pattern is retention off by default for content, with the shape reserved so enabling it later is a write-path change rather than a migration, and the decision belonging to the customer rather than the platform.
7. A timed revocation
Ask them to revoke a credential while you watch, and time it. Four minutes of the meeting, and it is the most predictive single test of incident readiness in this list — because it measures the thing that actually determines the size of an incident.
The follow-up is: can that be done without a deploy? If revocation is a code change, containment time is the length of the release process, and that number belongs in your risk assessment.
8. A per-customer export
The artefact that becomes a contractual obligation the moment you resell, and the one most often bolted on late. "We would write a query" is a failing answer, because it means the capability is a person rather than a feature.
The second place our own answer is partial
Since this article demands candour from vendors, here is the other one. Our action-outcome store — the record of whether work was accepted, corrected or rejected in the business — is built, tested and mounted, and no producer currently calls it, so it is empty. If you asked us for a stored outcome row today we could not produce one.
We flag it because it is the exact failure this article is about: a pipeline existing is not the same as a record being written, and the only question that distinguishes them is "show me a stored row". Ask it of everyone. Proving AI agent value beyond demos covers why that particular record matters commercially.
If you operate in a regulated industry
Add three control questions to the eight artefacts: who inside the vendor can read the evidence store directly, what the process is for changing its schema, and whether records are sampled. All three are frequently answered badly by products built for engineering teams, and all three surface during examination rather than during procurement. Audit trails for financial services covers what a review asks for.
If you are a platform or engineering lead
Produce these eight for your own system first. It is a genuinely useful internal exercise and it usually shows that two or three are a week of work rather than a purchase. The audit trail requirements list is the specification for artefacts 2 and 3, and the existing production AI agent readiness checklist places them in the wider go-live gate.
If you resell this to customers
Artefacts 1, 6 and 8 will be asked of you by your own customers, and having them ready shortens your sales cycle rather than lengthening it. Multi-tenant isolation for buyers covers the boundary claims that sit behind them.
Do this next
Send artefacts 2, 4 and 7 to your current shortlist in one email, with a week's deadline. Those three separate a shortlist faster than any scoring matrix, because they cannot be answered by a person who does not know the system. Then read the twenty-question security checklist for the interview, or how to evaluate audit trail software if the review is specifically about the evidence layer.
Questions people actually search for
- what security evidence should I request from an ai vendor
Eight artefacts. A machine-readable security posture report with a version. A stored evidence record from a failed run. The record schema and its version. The read path and what enforces the tenant filter. The permission catalogue with risk tiers. The retention policy, as a number, per data class. A timed revocation demonstration. And a per-customer export of one complete run. Each of these either exists or does not, which is why they are more useful than any questionnaire.
- what is a security posture report
A generated statement of the current control state of an account or tenant - which credentials are active, expiring or stale, which connections are healthy, which permissions are high risk, and what needs attention - rather than a description of what the product can do. The useful ones are versioned and machine-readable so they can be pulled repeatedly and compared over time, which turns a point-in-time review into ongoing assurance.
- how do you review the security of an ai agent platform
Collect artefacts rather than answers, and test three boundaries. Where is the tenant resolved from - the credential or the request. Where is a tool permission enforced - a boundary or a prompt. And what does a failed run leave behind. Those three, plus a timed revocation, discriminate between products far better than a two hundred line questionnaire, which every serious vendor can complete without revealing anything.
- does a soc 2 report cover ai agent risks
It covers the vendor as an organisation - that controls were described and tested against criteria over a period - which is real and worth having. It does not tell you where a permission decision is made inside the product, how an agent credential is scoped, whether failure records are written, or what happens when the policy layer is unavailable. Those are product questions and they need product artefacts. Treat the report as a gate on the company and this list as the gate on the software.
- what does good look like in an ai vendor security answer
Specific, mechanical and checkable. "Keys are stored hashed, the plaintext is shown once, and here is the error you get if you try to manage keys with a machine credential" is a good answer. "We follow security best practices and encrypt data at rest" is not, because it is true of everyone and describes nothing. A useful heuristic: if the sentence would be equally true of every vendor in the market, it has told you nothing.