RAG Cost Guide: Embedding, Storage and Query Economics
Retrieval-augmented generation (RAG) grounds an LLM in your data, but its cost has three distinct parts that scale differently: embedding your documents, storing the vectors, and answering each query. This guide breaks down RAG economics and where the money goes.
The three cost components
First, a mostly one-time cost to embed your corpus (plus re-embedding when documents change). Second, ongoing vector storage, which scales with the number of vectors and their dimensions. Third, per-query cost: embedding the question, then generating an answer over the retrieved context.
Retrieved context is the per-query lever
The biggest per-query cost is usually the retrieved chunks you stuff into the prompt. Retrieving fewer, higher-quality chunks — and capping how many you include — cuts input tokens without hurting answer quality much. This is where most RAG cost optimization happens.
RAG vs long context vs fine-tuning
When only a fraction of your knowledge is relevant per query, RAG beats stuffing a giant context window. For stable behavior at high volume, fine-tuning can be cheaper than resending examples. The right choice depends on how often your data changes and your query volume.
Calculators for this topic
Worked examples
Frequently asked questions
What makes RAG expensive?+
Usually the retrieved context stuffed into each query's prompt — retrieving fewer, better chunks is the main lever.
Is embedding a one-time cost?+
Mostly — you embed the corpus once and re-embed only when documents change.