LLM Latency Calculator
Find the tokens/sec needed to hit a latency target.
With the default inputs, Need ~192 tokens/sec — to return 500 tokens within 3s. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- Target latency
- 3 s
- Time to first token
- 400 ms
- Output tokens
- 500 tokens
- Required speed
- 192 tok/s
Have a latency budget? Enter your target response time, output length and time-to-first-token to see how fast the model must generate tokens.
How this is calculated
Required tokens/sec = output tokens ÷ (target response time − time-to-first-token). Given a latency budget, it tells you how fast the model must generate to return the full response in time.
Is this a good result? What to do next
If the required tokens/sec is higher than your model delivers, you won't hit the budget — the fix is a shorter output, a faster model, or a lower TTFT. Long outputs are the usual reason a latency target is missed.
Typical planning ranges
- Required speed
- output ÷ (target − TTFT)
- Long outputs
- hardest to fit a budget
- TTFT
- eats into the time budget
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Cap output length and stream responses.
- Pick faster models; reduce TTFT with warm instances.
- Split long generations into progressive chunks.
Common mistakes
- Ignoring TTFT in the time budget.
- Setting a budget too tight for the output length.
When to use a different approach
For observed throughput, use the LLM throughput calculator. To size capacity, use the concurrency capacity planner.
Worked example (defaults)
With the default inputs above, here is the result:
- Target latency
- 3 s
- Time to first token
- 400 ms
- Output tokens
- 500 tokens
- Required speed
- 192 tok/s
Frequently asked questions
Why does output length affect latency?+
Tokens are generated sequentially, so more output means more time at a given tokens-per-second rate.
How do I reduce latency?+
Cap output length, stream responses, pick faster models, and reduce time-to-first-token with warm instances.