Local LLM Throughput Calculator
Estimate tokens/sec for a model on your GPU.
With the default inputs, ~250 tokens/sec — 8B model at int4 on 1,000 GB/s. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- • Rough upper bound; real speed is lower due to compute and batching effects.
- Model size
- 8 B
- Precision
- int4
- Memory bandwidth
- 1,000 GB/s
- Estimated speed
- 250 tok/s
Estimate the token generation speed of a model on your hardware using a memory-bandwidth-bound approximation — a quick sanity check before running models locally.
How this is calculated
For single-stream generation, tokens/sec ≈ GPU memory bandwidth (GB/s) ÷ model size in bytes (parameters × bytes per parameter at your precision). Each token reads all weights, so generation is usually memory-bandwidth-bound — this is an optimistic upper bound.
Is this a good result? What to do next
Treat the result as a ceiling: real speed is lower due to compute, attention and framework overhead. Quantizing shrinks the bytes read per token, so it raises this bandwidth-bound estimate as well as cutting VRAM.
Typical planning ranges
- tok/s
- ≈ bandwidth ÷ model bytes
- Memory-bound
- single-stream generation
- Reality
- below the estimate
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Quantize to reduce bytes read per token.
- Use higher-bandwidth GPUs.
- Batch requests to raise aggregate throughput.
Common mistakes
- Treating the estimate as achievable real-world speed.
- Ignoring batching, which changes aggregate throughput.
When to use a different approach
For VRAM/fit, use the GPU VRAM calculator. To size a fleet for target throughput, use the GPU count calculator.
Worked example (defaults)
With the default inputs above, here is the result:
- • Rough upper bound; real speed is lower due to compute and batching effects.
- Model size
- 8 B
- Precision
- int4
- Memory bandwidth
- 1,000 GB/s
- Estimated speed
- 250 tok/s
- Memory-bound estimate: tok/s ≈ bandwidth / (params x bytes/param)
Frequently asked questions
Why memory bandwidth?+
Single-stream token generation is usually memory-bound: each token reads all model weights, so tok/s ≈ bandwidth ÷ model bytes.
Is this exact?+
No — it's an optimistic upper bound. Real speed is lower due to compute, attention and framework overhead.