LLM Throughput Calculator
See response time and throughput at a given tokens/sec.
With the default inputs, 8.73s per response — ~6.9 responses/minute per stream. Enter your own numbers below to recompute instantly; the full step-by-step math is shown under the worked example.
Results update automatically as you type.
- Time to first token
- 400 ms
- Generation time
- 8.33 s
- Total per response
- 8.73 s
- Responses / minute
- 6.9
Enter output length, generation speed and time-to-first-token to see how long a response takes and how many responses per minute a single stream can produce.
How this is calculated
Total response time = time-to-first-token + output tokens ÷ tokens per second. Responses per minute for one stream = 60 ÷ response time. It shows how generation speed and TTFT translate into latency and how many responses a single stream can produce.
Is this a good result? What to do next
For chat UIs, TTFT dominates perceived responsiveness while tokens/sec sets total time for long outputs. A single stream's throughput is low — real capacity comes from batching and running multiple replicas.
Typical planning ranges
- Response time
- TTFT + output ÷ tok-per-sec
- TTFT
- drives perceived responsiveness
- Single stream
- low throughput; scale with replicas
Ranges are typical planning figures to sanity-check your result, not authoritative benchmarks. Your numbers will vary with use case, volume, and vendor.
How to improve this number
- Cap output length to cut total time.
- Use faster/smaller models or better hardware.
- Batch requests and add replicas for aggregate throughput.
Common mistakes
- Ignoring TTFT when judging responsiveness.
- Assuming one stream's throughput equals system capacity.
When to use a different approach
For a latency target, use the LLM latency calculator. To size instances, use the concurrency capacity planner.
Worked example (defaults)
With the default inputs above, here is the result:
- Time to first token
- 400 ms
- Generation time
- 8.33 s
- Total per response
- 8.73 s
- Responses / minute
- 6.9
- time = TTFT + output tokens / tokens-per-second
Frequently asked questions
What is time to first token?+
The delay before the first output token appears. It matters a lot for perceived responsiveness in chat UIs.
How do I get more throughput?+
Batch requests, use faster/smaller models, or add more replicas. See our concurrency planner.