AI agent reliability is the probability that a versioned workflow produces an accepted outcome, inside its authority and safety boundaries, within its time and cost objectives. HTTP uptime alone misses model, tool, approval, and business-result failures.
Define outcome-level indicators
| Indicator | Example definition |
|---|---|
| Accepted outcome rate | Business-accepted outcomes divided by initiated workflows. |
| Unsafe action rate | Executed actions outside policy or approval; target should be zero. |
| Time to accepted outcome | Elapsed time including queues, retries, approvals, and repair. |
| Recovery success | Failed runs reaching a reconciled terminal state without duplicate effects. |
| Evidence completeness | Runs with required identity, policy, tool, approval, and outcome records. |
| Cost per accepted outcome | Model, tool, runtime, retry, review, and exception cost divided by accepted outcomes. |
Build the failure budget around harm
Do not trade unsafe actions for feature velocity. Use separate budgets: availability and latency may consume a conventional error budget; unauthorized execution, cross-tenant access, or lost audit evidence should trigger an immediate stop condition.
Replay test suite
- Normal workflow with an accepted downstream result.
- Model timeout and approved fallback.
- Tool timeout before and after a side effect.
- Expired credential and revoked workload identity.
- Prompt injection in retrieved content.
- Approval changed after the action fingerprint was created.
- Duplicate delivery with the same idempotency key.
- Telemetry collector outage during a high-impact action.
Operate by workflow version
Segment indicators by workflow, agent version, model route, tool, tenant, and risk tier. Aggregate averages conceal regressions. Freeze or roll back the smallest affected version, preserve the evidence chain, and compare recovery against the defined SLO.
Build a reliability evidence chain for one BlueBear workflow.