One user request can create many model, retrieval, and tool operations. Capacity planning must model amplification and external quotas, not only frontend requests per second.
Capacity worksheet
For each workflow record arrival rate, concurrent sessions, steps per session, model calls per step, tool calls, token volume, service time, retries, approval wait, and terminal-state rate. Segment by tenant and priority.
Find the real bottleneck
| Boundary | Signal | Control |
|---|---|---|
| Model | rate/token quota, latency, fallback | admission, routing, batch, reserved quota |
| Tool | API limit, lock, downstream saturation | per-tool concurrency and circuit breaker |
| Runtime | CPU, memory, pod startup, queue depth | autoscaling and selective pre-warm |
| Human | approval age and exception backlog | risk routing, staffing, expiry |
Safe overload
Prioritize critical workflows, reject early with a visible reason, cap retries, preserve idempotency, and prevent one tenant from consuming shared capacity. Test model slowdown, tool outage, burst traffic, and approval backlog together.