Stage 1: Retrieval Methods

This section covers methods for the first stage of the RAG pipeline: efficiently retrieving candidate documents from large corpora.

Stage 1 Topics:

• Sparse Retrieval Methods
  • BM25: The Classic Baseline
  • When to Use BM25
  • Implementation
  • Modern Sparse Methods
  • Recommendations
  • Resources
  • Next Steps
• Dense Baselines & Fixed Embeddings
  • The Dense Retrieval Revolution
  • Dense Passage Retrieval (DPR)
  • Fixed Embeddings: RepBERT
  • Comparison: DPR vs RepBERT
  • When to Use Each
  • Modern Successors
  • Implementation Recommendations
  • Next Steps
• Hard Negative Mining
  • The Hard Negative Problem
  • Evolution of Hard Negative Strategies
  • Hard Negative Mining Papers
  • Key Implementation Strategies
  • Bird’s Eye View: Which Strategy to Choose?
  • Recommended Path for Practitioners
  • Common Pitfalls
  • Next Steps
• Late Interaction (ColBERT)
  • Short Answer
  • Long Answer: Why Late Interaction is Special
  • The ColBERT Architecture
  • ColBERT Literature
  • Other Multi-Vector Methods
  • How ColBERT Does Both Stages
  • Performance Comparison
  • When to Use ColBERT
  • Implementation Example
  • The Index Size Problem
  • Advanced Topic: MaxSim Operation
  • Comparison with Cross-Encoders
  • Poly-Encoders: Another Middle Ground
  • Research Directions
  • Next Steps
• Hybrid Dense-Sparse Methods
  • The Complementarity Principle
  • Hybrid Methods Literature
  • Implementation Patterns
  • When to Use Hybrid
  • Empirical Results
  • Implementation Resources
  • Best Practices
  • Next Steps
• Pre-training Methods for Dense Retrievers
  • Why Pre-training Matters for Retrieval
  • Pre-training Methods Literature
  • Comparison: Fine-tuning vs Pre-training
  • When to Use Pre-trained Models
  • Recommended Practice
  • Implementation Example
  • Next Steps
• Joint Learning of Retrieval and Indexing
  • The Motivation
  • Joint Learning Literature
  • Why This Matters
  • JPQ Implementation Strategy
  • When to Use Joint Learning
  • The Future: Learned Indexes
  • Next Steps
• Literature Survey: Retrieval Methods
  • Hard Negative Mining: A Deep Dive for Building a Unified Mining Library
  • Overview
  • Featured Papers
  • Topics Covered
  • Contributing

Overview

Stage 1 retrieval must balance two competing demands:

• Speed: Process millions of documents in milliseconds

• Recall: Don’t miss relevant documents

The solution is to use architectures that allow pre-computation and efficient similarity search.

Evolution of Stage 1 Methods

Historical Evolution

Era	Method Type	Key Innovation	Representative Papers
Pre-2020	Sparse (BM25)	Inverted index, TF-IDF	Traditional IR
2020	Dense Baselines	Dual-encoder with BERT	DPR, RepBERT
2021	Hard Negatives	Dynamic mining, denoising	ANCE, RocketQA, ADORE
2021-2022	Late Interaction	Token-level representations	ColBERT, ColBERTv2
2022-2023	Sampling Strategies	Curriculum, score-based	TAS-Balanced, SimANS
2023-2024	LLM Integration	Synthetic negatives, prompting	SyNeg, LLM embeddings

Key Dimensions

When evaluating Stage 1 methods, consider:

Architecture

• Dual-Encoder: Independent encoding (fastest)

• Late Interaction: Token-level matching (more accurate)

• Hybrid: Combines sparse and dense

Training Strategy

• Negative Mining: How to find informative negatives?

• Knowledge Distillation: Learn from cross-encoder teachers

• Curriculum Learning: Progressive difficulty

Index Structure

• Dense Vector: Single vector per document

• Multi-Vector: Multiple vectors (e.g., ColBERT)

• Learned Index: End-to-end optimized structures

Quick Navigation

• Sparse Retrieval Methods - BM25 and traditional IR methods

• Dense Baselines & Fixed Embeddings - DPR, RepBERT (foundational papers)

• Hard Negative Mining - Core focus: Advanced negative mining strategies

• Late Interaction (ColBERT) - ColBERT and token-level methods

• Hybrid Dense-Sparse Methods - Combining sparse and dense

• Pre-training Methods for Dense Retrievers - Pre-training strategies for dense retrievers

• Joint Learning of Retrieval and Indexing - Jointly optimizing retrieval and indexing

The Central Challenge: Hard Negative Mining

The quality of Stage 1 retrievers depends critically on the negative examples used during training. This is explored in depth in Hard Negative Mining, which covers:

• The hard negative problem

• Evolution from static to dynamic mining

• Denoising strategies

• Curriculum learning

• LLM-based generation

This is the primary bottleneck in dense retrieval research today.