// The optimization that makes AI responses fast (and cheap)
💡 The core idea: Every time the AI processes text, it does heavy math on every word. Without a cache, if you send 1000 words + 1 new word, it redoes math on all 1001 words. With KV cache, it saves the math results for the first 1000 words. Next turn: only compute 1 new word. Like caching a webpage instead of rebuilding it from scratch every request.
KV Cache in action
Scenario: A 10-message conversation. Each turn adds new tokens. Watch what gets recomputed vs cached.
Turn 0
COMPUTE COST PER TURN
Cached (free)New (computed)
K, V — what do these letters mean?
In the transformer's attention mechanism, every token creates three vectors: Q (query), K (key), V (value). To process a new token, the model needs the K and V vectors of every previous token. The KV cache stores these so they don't have to be recomputed. K = "what do I contain?", V = "what info do I carry?"