GPU VRAM Calculator for LLMs
Estimate the VRAM needed to run a model locally.
With the default inputs, ~168 GB VRAM — ≈ 3 x 80 GB GPU(s) to serve. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- Parameters
- 70 B
- Precision
- fp16
- Weights + overhead
- 168 GB
- GPUs needed
- 3
Estimate how much GPU memory a model needs based on its parameter count and precision, and how many GPUs it takes to serve — essential before running models locally.
How this is calculated
VRAM ≈ parameters (billions) × bytes per parameter (4 fp32, 2 fp16, 1 int8, 0.5 int4) × overhead factor for KV cache and activations. GPUs needed = ceil(VRAM ÷ per-GPU memory). A 70B model at fp16 is ~168 GB — so 2–3 80 GB GPUs; int4 drops it to ~42 GB.
Is this a good result? What to do next
The single biggest lever is precision: dropping from fp16 to int4 quarters the weight memory and often decides whether a model fits one GPU or needs several. Leave overhead for KV cache, which grows with context length and batch size.
Typical planning ranges
- 70B @ fp16
- ~168 GB (2–3× 80 GB GPUs)
- 70B @ int4
- ~42 GB
- Overhead
- ~15–30% for KV cache/activations
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Quantize (int8/int4) to fit fewer/cheaper GPUs.
- Reduce context length and batch size to shrink KV cache.
- Use a smaller model variant.
Common mistakes
- Counting only weight memory and ignoring KV/activation overhead.
- Assuming quantization is free of quality impact.
When to use a different approach
To compare precisions, use the quantization memory savings calculator. For speed, use the local LLM throughput calculator.
Worked example (defaults)
With the default inputs above, here is the result:
- Parameters
- 70 B
- Precision
- fp16
- Weights + overhead
- 168 GB
- GPUs needed
- 3
- VRAM ≈ params(B) x bytes/param x 1.2 (KV-cache/activation overhead)
Frequently asked questions
How much VRAM does a 70B model need?+
About 168 GB at float16 (2 bytes/param + overhead), so 2–3 high-end 80 GB GPUs. int4 quantization drops it to ~42 GB.
What's the overhead for?+
KV cache, activations and framework overhead on top of the raw weights, roughly 15–30%.