LLM Throughput and Latency: Planning for Load
Cost is not the only constraint — you also need enough throughput to serve peak traffic without users waiting. This guide connects latency, throughput and concurrency into a capacity plan.
Latency, throughput, concurrency
Latency is how long one request takes; throughput is how many tokens per second you can push; concurrency is how many requests run at once. Peak load divided by per-instance throughput tells you how much capacity you need.
Plan for the peak, not the average
Sizing for average traffic guarantees slowdowns at the peak. Estimate concurrent requests at your busiest moment and provision — or set rate limits — so the worst case still meets your latency target.
Calculators for this topic
Frequently asked questions
How do I plan LLM capacity?+
Divide peak concurrent load by per-instance throughput to find how many instances or how high a rate limit you need.
Should I plan for average or peak load?+
Peak — sizing for the average leaves you slow exactly when traffic is highest.