Learning path · Production RAG · 56
REFRAG
Retrieval compression pattern that keeps many candidate chunks as compact embeddings and expands only the ones the decoder needs back into tokens.
Why it matters
- Cuts wasted tokens from stuffing oversized retrieval sets into the prompt.
- Preserves broad recall while controlling latency and cost at decode time.
- Fits production stacks already paying for large context windows.
Key ideas
- Compressed retrieval
- Selective expansion
- Compute efficiency
REFRAG-style pipelines separate "what might be relevant" from "what must be readable tokens." Candidates stay cheap as embeddings until a selector expands a smaller subset into the generator prompt. That reduces the Lost-in-the-Middle tax and the bill for stuffing twenty near-duplicates into every call. Implement with explicit budgets: max expanded chunks, max tokens, and a fallback when the selector is uncertain. Measure faithfulness before celebrating cost wins—compression that drops the supporting span is a silent regression. Log which chunks expanded versus stayed compressed so ops can debug wrong answers.
Updated 2026-08-09 · Full learning path