How Chunk Size Affects RAG Quality and Cost
Chunk size is a small setting with outsized impact: it shapes how well retrieval works and how many tokens each query drags into the prompt. Getting it right saves money and improves answers.
Bigger chunks, bigger prompts
Large chunks capture more context per hit but push more tokens into every query's input. If you retrieve several big chunks per question, per-query cost climbs quickly — often for context the model never uses.
Tune chunks and top-k together
Smaller, focused chunks plus a modest number of retrieved results usually beats a few huge chunks on both cost and precision. Tune chunk size and how many you retrieve as one decision, not separately.
Calculators for this topic
Frequently asked questions
Does chunk size affect cost?+
Yes — larger chunks mean more tokens per retrieved result, raising per-query input cost.
What is a good chunk size?+
Small enough to stay focused, large enough to keep meaning — tune it with the number of chunks you retrieve.