Concurrency Capacity Planner
Size concurrency and instances for a target request rate.
With the default inputs, 60 concurrent requests needed — ≈ 6 instances at 10 concurrent each. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- Target throughput
- 20 req/s
- Avg latency
- 3 s
- Concurrency (Little's law)
- 60
- Instances
- 6
Use Little's law to size your deployment: enter target requests/second and average latency to see the concurrency needed, and how many instances that requires.
How this is calculated
By Little's law, required concurrency = target requests/second × average latency. Instances needed = required concurrency ÷ concurrent requests per instance, rounded up. To serve 20 req/s at 3s each you need ~60 requests in flight.
Is this a good result? What to do next
This sizes steady-state capacity; you still need headroom for spikes and failures. If instance count looks high, latency is usually the culprit — cutting it reduces concurrency requirements proportionally.
Typical planning ranges
- Concurrency
- throughput × latency
- 20 req/s × 3s
- ~60 in flight
- Headroom
- add for spikes and failover
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Reduce latency to cut required concurrency.
- Raise per-instance concurrency where the runtime allows.
- Autoscale on queue depth, with spare headroom.
Common mistakes
- Sizing to exact steady state with no spike headroom.
- Using peak latency inconsistently with average load.
When to use a different approach
To check provider TPM/RPM limits, use the rate limit planner. For GPU counts, use the GPU count calculator.
Worked example (defaults)
With the default inputs above, here is the result:
- Target throughput
- 20 req/s
- Avg latency
- 3 s
- Concurrency (Little's law)
- 60
- Instances
- 6
- concurrency = throughput x latency
Frequently asked questions
What is Little's law?+
Concurrency = throughput × latency. To serve 20 req/s at 3s each, you need ~60 requests in flight.
How many instances do I need?+
Divide required concurrency by how many concurrent requests each instance handles, then round up.