Learning path · Context Engineering & Caching · 25
Prompt Caching
Reusing computed prefix states across requests with identical early prompt segments to cut latency and input-token cost.
Why it matters
- Major lever for RAG and agent systems with stable instructions.
- Provider behaviour differs: explicit cache breakpoints vs automatic prefix matching.
- Requires stable prompt prefixes to realize savings.
Key ideas
- Prefix reuse
- Cache breakpoints
- TTL and invalidation
Prompt caching avoids recomputing attention over repeated prefixes. Anthropic exposes explicit cache breakpoints—you mark stable segments (system policy, tool schemas, static manuals) so later requests reuse KV state. OpenAI applies automatic prefix caching when consecutive requests share identical leading tokens; you do not set breakpoints manually, but benefit when prefixes match byte-for-byte. Design prompts with stable-first, dynamic-last ordering. Measure cache hit rate alongside p95 latency; reported industry case studies cite substantial savings when prefixes stay fixed across high-volume traffic. Instrument cache hit ratio per route and correlate with finance dashboards; savings disappear quickly when dynamic content precedes static prefixes.
Updated 2026-08-09 · Full learning path