Disclosure: BlueBear builds an AI agent platform and has a commercial interest in this topic. The method below is deliberately vendor-neutral, and the arithmetic examples are illustrative constructions, not measurements of any customer or of our own platform.
The short answer: value is proved by a population, not by a demonstration
A demo is a single run, chosen after you know it worked. That is not a criticism of demos — it is what a demo is for. But it means a demo can never answer the question a finance owner is actually asking, which is not "can this work" but "what does this cost, per result that we can use, across all the times it runs".
Proving value means replacing one selected run with a defined population of runs, and replacing an impression with an acceptance standard written down before the runs happened. There are four steps and none of them require a vendor.
| Step | What you produce | The failure mode it removes |
|---|---|---|
| 1. Define acceptance, in writing, first | A one-page standard: what a good result looks like, who judges it, what disqualifies one | Judging output after the fact, which reliably grades the good runs and forgets the rest |
| 2. Fix the population before you run it | A named set of real cases, including the awkward ones | Cherry-picking, usually unconsciously, by re-running the hard cases until they pass |
| 3. Record an outcome on every run | Accepted, corrected, rejected, abandoned — one value per run | Counting completions and calling them successes |
| 4. Load the cost fully | Model spend + retries + human review and correction time | A cheaper model that looks better per call and costs more per result |
Steps 1 and 2 cost an afternoon and are where nearly all the value is. Steps 3 and 4 are where instrumentation matters, and where a platform can legitimately help.
Step 1: acceptance has to be written before the runs, or it is not a standard
An acceptance standard answers three questions in plain language. What does a usable result contain? — the specific fields, the tone, the required citation, the format. Who decides? — a named role, not "the team". What disqualifies a result outright? — the failures you will not trade away at any price, such as a wrong figure in a customer-facing document.
Written afterwards, this document is a rationalisation. Written first, it is the thing that makes a disappointing result informative rather than embarrassing. It is also the artefact that lets a risk owner sign off, because it converts "the AI seems good" into "the AI met a standard someone accountable set".
Step 2: a fixed population, chosen for difficulty as well as volume
Pick a set of real cases — thirty is usually enough to be informative, a few hundred to be persuasive — and freeze it before the first run. Include the cases that are awkward: incomplete inputs, unusual formats, the ones your best person has to think about. A population that contains only clean cases measures your data quality, not your agent.
Then run the whole population, once, and keep every result. Including the ones that failed. Especially the ones that failed, because the failure rate is the input that changes the arithmetic most and is the one most often left out.
Step 3: an outcome on every run is the part that has to be built in
This is the step where the honest limits of tooling show up, and where we will be specific about our own.
An outcome is a value attached to a run saying what happened to it in the business: accepted, corrected by a person, rejected, or abandoned. It is not the same as a status code. A run can return HTTP 200, produce well-formed output, and be rejected by the person who reads it. Systems that record only technical success are, from a value-measurement standpoint, recording nothing.
Recording an outcome requires a producer — something at the point where a human accepts or corrects the work that reports the fact back. That is a product decision as much as an infrastructure one, and it is commonly the missing piece. In our own gateway the ingestion path and the storage for outcomes exist and are wired up, and at the time of writing no producer calls them, so the outcome table is empty. We are stating that plainly because it is exactly the gap this article is about: the pipeline being built is not the same as the outcome being recorded, and if you are evaluating any tool in this category, that is the distinction to test rather than to accept.
What this means practically: when you ask a vendor "can you measure outcomes", the useful follow-up is "show me a stored record with an outcome on it, from a real run". A schema is a necessary condition. A populated row is the sufficient one.
Step 4: cost per accepted outcome, and the two costs everyone forgets
The measure that survives scrutiny is total loaded cost divided by accepted outcomes. Two components are almost always missing from the numerator:
- Retried and failed work. Every attempt is billed. A run that timed out, returned malformed output, hit a guardrail or was abandoned at an iteration cap cost full price and produced nothing.
- Human correction time. If a person spends four minutes fixing a result, that is the dominant cost in most back-office workflows, and it is invisible in every model-spend dashboard.
An illustrative example
The following figures are constructed to show the arithmetic. They are not measurements of any customer, of BlueBear, or of any model.
Suppose two configurations process the same fixed population of 1,000 cases.
| Configuration A | Configuration B | |
|---|---|---|
| Assumed model cost per attempt | $0.08 | $0.02 |
| Assumed attempts per case (including retries) | 1.1 | 1.6 |
| Assumed acceptance rate, first pass | 90% | 62% |
| Assumed human correction, minutes per corrected case | 3 | 5 |
| Model spend | $88 | $32 |
| Correction time | 5 hours | 31.7 hours |
| Cost per accepted outcome, model only | $0.098 | $0.052 |
| Cost per accepted outcome, loaded at an assumed $40/hour | $0.32 | $1.32 |
Configuration B is four times cheaper per call and four times more expensive per usable result. The point of the example is not the numbers, which are invented; it is that the two measures point in opposite directions, and that only one of them is the business's actual cost. The cost-per-outcome calculator is the reusable version of this arithmetic, and is a cheaper model good enough is the method for the quality half.
What a demo can and cannot tell you
| A demo can honestly show | A demo cannot show |
|---|---|
| That the capability exists at all | How often it works on your inputs |
| The shape of the workflow and where a human sits in it | What a failure looks like, or how it is contained |
| What is recorded — if you ask to see the record, not the result | The cost of the runs that did not appear in the demo |
| Where the boundaries are, if you ask what it is not allowed to do | Anything about your data quality, which is usually the real variable |
The most useful thing you can do in a vendor demo is ask to see a run that failed, and then ask what the system recorded about it. The answer separates products that have an evidence layer from products that have a happy path. Why AI agent demos break when real operations begin covers the operational half of this gap.
If you operate in a regulated industry
Your proof has a second audience: a reviewer who was not in the room. That changes what you retain rather than what you measure. Keep the acceptance standard as a dated document, keep the population definition, and keep the per-run records including the failures — a proof that only retains successes is not evidence, it is marketing with a spreadsheet. What an audit trail must capture lists the fields that make a run reconstructable months later.
If you are a platform or engineering lead
The single highest-leverage thing to build is not a dashboard, it is the outcome field and the identifier that ties a run to a business result. Both must be attached at the time of the run; neither can be reconstructed afterwards from provider invoices, which know an API key and not a workflow. Everything else — charts, alerts, allocation — is derived from those two and can wait. What to trace in an AI agent covers the wider recording decision.
If you resell this to customers
You need the same measure per customer, because a blended cost per outcome hides the account that is destroying your margin. That is an attribution requirement at call time, and it is the same requirement as the one above with a tenant identifier added. White-label unit economics works the margin case through.
Do this next
Write the acceptance standard for one workflow — one page, three questions, a named judge — before you run anything else. It is the cheapest step and it is the one that makes every subsequent number mean something. Then read how to design a proof of value that survives production review for the programme structure around it, and what an AI agent platform is if you still need the category framing to bring colleagues along.
Questions people actually search for
- who proves ai product value beyond demos
Nobody proves it for you, and that is the honest answer. A vendor can show you a demo and a reference; neither is evidence about your work. The proof has to be produced by whoever owns the business process, using a population of your own runs measured against an acceptance standard written down before the runs happened. What a vendor can legitimately supply is the instrumentation that makes the measurement possible - an outcome recorded against each run, a cost attributed to it, and a record you can re-read.
- how do you measure ai agent roi
Divide the fully loaded cost by the number of outcomes that were actually accepted, and compare that to what the same outcome cost before. Fully loaded means model spend plus retries plus the human time spent reviewing, correcting and re-running. Accepted means it met a standard written down in advance. Both halves are usually missing: teams count model spend but not review time, and count completions but not corrections. The gap between cost per call and cost per accepted outcome is the whole subject.
- why do ai pilots fail to move to production
Most commonly because the pilot never defined what success meant, so at the end there is a pile of impressive runs and no way to say whether it worked. The second most common reason is that the pilot measured only the runs that finished, so the failure rate, the retries and the human corrections were never in the denominator. A pilot that measures a population and states its acceptance standard up front can fail honestly and quickly, which is more valuable than one that succeeds ambiguously.
- what evidence should I ask an ai vendor for
Ask for artefacts, not outcomes. A record schema you can read. A worked example of a single run end to end, including one that failed. The list of fields retained and for how long. How cost is attributed to a workflow or a customer. Whether an outcome can be recorded against a run at all, and by what mechanism. Percentages in a case study tell you about someone else business; a record schema tells you whether you will be able to produce your own numbers.
- what is cost per successful outcome
Total spend divided by the number of results that were accepted, rather than by the number of calls made. It is the only cost measure that moves in the right direction when quality changes: a cheaper model that needs two attempts and a human correction raises cost per successful outcome even as it lowers cost per call. Retries, abandoned runs and human review all belong in the numerator or the denominator, and most dashboards put them in neither.