Learning path · Retrieval & Ranking · 45
Reranking
Second-stage model that scores query-passage pairs with richer interaction than bi-encoder retrieval alone.
Why it matters
- Lifts Precision@K before chunks enter the LLM context.
- Mitigates lost-in-the-middle by ordering best evidence last.
- Adds latency—usually worth it for high-stakes answers.
Key ideas
- Cross-encoder
- Top-K refinement
- Latency tradeoff
Bi-encoder retrieval is fast but shallow; cross-encoder rerankers read query and passage together for accurate relevance scores. Retrieve generously (50–200), rerank to 5–10, then assemble context. ColBERT-style late interaction offers a middle ground for large candidate sets. Monitor reranker timeouts—they are often the p95 bottleneck in RAG paths. Set aggressive timeouts and fallback to bi-encoder ordering rather than blocking user requests when rerankers saturate GPU pools. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path