passdrill
AI & LLM Engineering · RAG & Embeddings · Card 028/032 easy

A RAG system sometimes retrieves passages that are only weakly related to the query, and when that happens, the generated answer still depends entirely on those weak passages because nothing in the pipeline checks how good the retrieval was before generation runs. Corrective Retrieval Augmented Generation (CRAG, Yan et al., 2024) adds a step to address this. What does it do?

  1. It adds a lightweight retrieval evaluator that scores the quality of the retrieved documents before generation, and when that score indicates the retrieval is poor, it triggers a corrective action such as falling back to a web search, rather than generating directly from documents already judged to be weak
  2. It has the language model generate special reflection tokens as part of its own output, deciding token by token whether the passages it was given were worth using at all
  3. It removes the retrieval step from the pipeline entirely whenever the query is judged to be a broad, corpus-wide question rather than a narrow factual one
  4. It re-embeds every document in the corpus using a larger embedding model whenever a low-quality retrieval is detected, then re-runs the same query against the newly re-embedded corpus
Next card → Shuffle