GPU Count for Throughput Calculator
Find how many GPUs you need for a target tokens/sec.
With the default inputs, 7 GPU(s) needed — for 500 tok/s at 80 tok/s each. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- Target throughput
- 500 tok/s
- Per-GPU throughput
- 80 tok/s
- GPUs needed
- 7
Enter your target total throughput and the tokens/sec a single GPU delivers to see how many GPUs your inference workload needs.
How this is calculated
GPUs needed = ceil(target total throughput ÷ per-GPU throughput). Enter your target tokens/sec and what one GPU delivers (benchmarked or estimated) to size the fleet for inference.
Is this a good result? What to do next
The estimate is only as good as your per-GPU tokens/sec, which batching can raise substantially — so measure under realistic load before buying. Add headroom for spikes and failover on top of the raw count.
Typical planning ranges
- GPUs
- ceil(target ÷ per-GPU tok/s)
- Batching
- raises per-GPU throughput
- Headroom
- add for spikes/failover
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Raise per-GPU throughput via batching and better serving stacks.
- Quantize for faster generation.
- Autoscale rather than provisioning for peak.
Common mistakes
- Using single-stream throughput to size a batched fleet.
- Provisioning exactly to target with no headroom.
When to use a different approach
To estimate per-GPU speed, use the local LLM throughput calculator. To size concurrency, use the concurrency capacity planner.
Worked example (defaults)
With the default inputs above, here is the result:
- Target throughput
- 500 tok/s
- Per-GPU throughput
- 80 tok/s
- GPUs needed
- 7
- GPUs = ceil(target tok/s / per-GPU tok/s)
Frequently asked questions
How do I find per-GPU throughput?+
Benchmark your model on one GPU, or estimate it with our local LLM throughput calculator.
Does batching change this?+
Yes — batching raises aggregate tokens/sec per GPU, reducing the count needed. Measure under realistic load.
Related calculators
Answering a real question?
This calculator powers these problem-solving guides: