A team considers dropping their RAG pipeline entirely now that a newer model supports a 1-million-token context window, reasoning they could simply paste their whole private document collection into every prompt instead of retrieving a handful of relevant passages. Independent benchmarking comparing this long-context approach against RAG on the same workload found the long-context approach answered correctly about as often, but was far more expensive and far slower per query. What best explains why RAG can still be the better choice even when a model's context window is large enough to fit the whole corpus?
- A larger context window causes the model to ignore the retrieved information entirely and answer purely from its own training data instead, regardless of what is included in the prompt
- Feeding a full corpus into every single prompt means paying to process and re-process a huge number of tokens on every query, which research measuring this trade-off found costs and takes many times longer than retrieving and sending only a handful of relevant passages, so at meaningful query volume the long-context approach's cost and latency scale far worse even when its answer quality is comparable
- Context windows above roughly 100,000 tokens are not actually supported by any current model despite vendor claims, so the long-context approach would fail outright rather than merely being slower
- RAG pipelines are the only approach capable of producing an answer that cites which source passage it came from, so long-context prompting can never support citations under any circumstances
Why B? And why not the others?
Correct answer: B. Feeding a full corpus into every single prompt means paying to process and re-process a huge number of tokens on every query, which research measuring this trade-off found costs and takes many times longer than retrieving and sending only a handful of relevant passages, so at meaningful query volume the long-context approach's cost and latency scale far worse even when its answer quality is comparable
Pasting an entire document collection into the prompt for every single query means the model has to process that huge number of tokens over again on every call, and comparisons measuring this directly found the long-context approach costing many times more and taking many times longer per query than retrieving and sending only a handful of relevant passages, while answering about as accurately on the same workload; at any meaningful query volume, that cost-and-latency gap compounds because it is paid again on every single request, which is why RAG can remain the better engineering choice even once a model's context window is technically large enough to fit everything. The claim that a larger context window makes a model ignore retrieved information is not an accurate description of what was being compared or observed; the actual finding was about cost and speed, not about the model refusing to use provided context. The claim that context windows above roughly 100,000 tokens don't actually work is false; current models do support windows at that scale and well beyond, which is exactly what makes the whole-corpus-in-one-prompt approach possible in the first place, just expensive. And citations are not exclusive to RAG pipelines; a long-context prompt can also be instructed to cite which part of the pasted text supports its answer, so the ability to cite sources does not by itself distinguish the two approaches.
Source: Towards Data Science, 'Kimi K3's 1M Token Context Window vs. RAG: Cost, Latency and Answer Quality,' https://towardsdatascience.com/kimi-k3s-1m-token-context-window-vs-rag-cost-latency-and-answer-quality/; Redis, 'RAG vs Large Context Window: Real Trade-offs for AI Apps,' https://redis.io/blog/rag-vs-large-context-window-ai-apps/