A RAG system retrieves 8 relevant passages for a query and inserts all of them into a single long prompt in an arbitrary order before generation. Research studying how language models use long contexts documented a failure mode directly relevant here. What is that failure mode, and what does it suggest about how the 8 passages should be arranged in the prompt?
- Models cannot process context windows longer than a few thousand tokens at all, so several of the 8 passages would simply be truncated and never reach the model regardless of their order
- Models weight every position in the context equally when generating an answer, so the order in which the 8 passages appear has no measurable effect on the result
- Models process the context strictly from the last token backward, so only the very last of the 8 passages in the prompt has any influence on the generated answer
- Performance on tasks that require using information from within a long context follows a U-shaped curve, staying strongest for information near the very beginning or the very end of the context and degrading for information placed in the middle, so the most relevant of the 8 passages should be placed near the start or end of the prompt rather than buried in the middle
Why D? And why not the others?
Correct answer: D. Performance on tasks that require using information from within a long context follows a U-shaped curve, staying strongest for information near the very beginning or the very end of the context and degrading for information placed in the middle, so the most relevant of the 8 passages should be placed near the start or end of the prompt rather than buried in the middle
Studies of how language models use long input contexts found a consistent U-shaped performance curve on tasks that require locating and using a specific piece of information: accuracy is highest when the needed information sits near the beginning or the end of the context and drops noticeably when that information is placed in the middle, a pattern attributed to a mix of primacy and recency effects rather than uniform attention across all positions; the practical implication for a RAG prompt built from several retrieved passages is to place the passages judged most relevant near the start or end rather than in the middle of the stack. The option describing a hard length ceiling of only a few thousand tokens conflates this positional effect with a completely different constraint, a fixed context-length limit, when the scenario already assumes all 8 passages fit inside the prompt. The option claiming position has no effect is the opposite of what was documented. The option claiming only the very last passage matters overstates the finding, since the documented curve favors both the beginning and the end, not the end alone.
Source: Liu et al., 'Lost in the Middle: How Language Models Use Long Contexts' (2023), arXiv:2307.03172