Learning path · Foundations · 03
Parametric Memory Limits
Knowledge compressed into model weights is static, approximate, and cannot be updated per request—unlike retrieval or tool-backed non-parametric memory.
Why it matters
- Explains hallucinations on niche or time-sensitive facts.
- Justifies RAG, knowledge bases, and agent tool layers.
- Sets realistic fine-tuning goals: behaviour tuning, not live fact storage.
Key ideas
- Static weights
- Training cutoff
- Non-parametric complement
Everything the model 'knows' without external input was absorbed during training into billions of parameters. That memory is broad but fuzzy: it confuses similar entities, invents plausible citations, and drifts from your ground truth. Updating weights is slow and expensive; injecting documents at query time is fast and auditable. Mature teams assume parametric memory is a prior, not a source of record, and wire explicit retrieval and citation paths for anything compliance-sensitive. Run periodic audits where experts compare model answers against authoritative systems of record to quantify parametric drift on your domain vocabulary.
Updated 2026-08-09 · Full learning path