passdrill
AI & LLM Engineering · RAG & Embeddings · Card 019/022 medium

A RAG system built on ordinary vector similarity search answers narrow, fact-lookup questions about a large private document collection well, but performs poorly on a broad question like 'what are the main recurring themes across this entire collection,' because no single retrieved passage or small handful of passages contains a synthesis of the whole corpus. GraphRAG (Microsoft Research, 2024) was designed to address this class of question. What does it do differently from standard vector-similarity RAG?

  1. It increases the number of passages retrieved for every query to the maximum the context window allows, on the theory that more raw passages will eventually contain the needed synthesis
  2. It replaces the embedding model with a larger one that has more parameters, so that individual passage embeddings become more informative on their own
  3. It fine-tunes the language model directly on the entire document collection so that broad thematic questions can be answered from the model's own updated parameters instead of from any retrieval step at all
  4. It first uses a language model to extract entities and relationships from the corpus into a knowledge graph, partitions that graph into communities of closely related entities, and pre-generates a summary for each community, so a broad question can be answered from these higher-level community summaries instead of depending on any single retrieved passage to contain the whole synthesis
Next card → Shuffle