A team indexes a knowledge base by embedding small, single-idea chunks (roughly one or two sentences each) so that similarity search can pinpoint the exact passage that answers a narrow question. But when they inspected the passages actually sent to the LLM, they found these tiny chunks often lacked enough surrounding context for the model to interpret them correctly on their own. Rather than switching to embedding larger chunks (which would blur the precision of the similarity search), what technique keeps the small chunks for search while fixing the context problem?
- Re-embed every chunk twice, once at the small size and once at a larger size, and always return whichever of the two embeddings scores higher for the query, discarding the other
- Increase the number of small chunks retrieved to the maximum the context window allows, without changing which chunks are retrieved or how much surrounding text accompanies each one
- Fine-tune the embedding model specifically on the small chunks so each one independently encodes more surrounding context inside its own vector
- Keep the small chunks as the unit that gets embedded and searched over, but once a small chunk is retrieved, expand it by pulling in its neighboring text, up to the surrounding paragraph, page, or even the whole source document, and pass that expanded window to the LLM instead of the bare small chunk
Why D? And why not the others?
Correct answer: D. Keep the small chunks as the unit that gets embedded and searched over, but once a small chunk is retrieved, expand it by pulling in its neighboring text, up to the surrounding paragraph, page, or even the whole source document, and pass that expanded window to the LLM instead of the bare small chunk
This 'chunk expansion' (small-to-big) approach deliberately decouples the unit used for search from the unit delivered to the LLM: the small, precisely-scoped chunk stays the thing that gets embedded and matched against the query, preserving the similarity search's precision, but once that small chunk is identified as relevant, the surrounding text around it is pulled in and expanded up to a paragraph, page, or whole document before being handed to the LLM, restoring the context the bare fragment was missing. The option describing embedding every chunk at two sizes and picking whichever scores higher is wrong because it still forces a single tradeoff between search precision and context richness on the search side, rather than separating the two concerns as chunk expansion does. The option describing simply retrieving more small chunks is wrong because piling on more disconnected small fragments does not restore the specific surrounding context of any one retrieved chunk. The option describing fine-tuning the embedding model to pack more context into a small chunk's own vector is wrong because a fixed-length vector has limited capacity, and this approach does not address the fact that the chunk's actual text, not just its vector, is what the LLM needs more of.
Source: Pinecone, 'Chunking Strategies for LLM Applications,' https://www.pinecone.io/learn/chunking-strategies/