// What actually happens on the GPU between your prompt and the first word of response
💡 The core idea: AI generation happens in two phases. Prefill: the AI reads and processes your entire prompt in one batch (fast). Decode: the AI generates one token at a time, feeding each new word back in to predict the next (slower). This is why short prompts feel faster and why the AI can't generate all tokens at once.
The two phases of LLM inference
Phase 1: Prefill
⚡
Read and process all your prompt tokens at once. Compute KV cache for each token. This is parallelized — all tokens computed in one GPU pass.
Fast, parallel, compute-bound
Phase 2: Decode
🔄
Generate one token at a time. Each new token gets fed back in. KV cache grows. Must be done sequentially — can't parallelize because each token depends on the previous one.
Slow, sequential, memory-bound
0 / 6
Simulate a streaming response — one token at a time, as a real LLM API sends it:
Prompt: "In one sentence, explain what gravity is."
Click "stream response"
This is exactly how the OpenAI API works with stream: true — it sends each token as a Server-Sent Event (SSE). Your UI renders tokens as they arrive, making responses feel faster even though total time is the same.
Why is generation sequential?
Each token is generated based on ALL previous tokens. To generate token 50, the AI needs tokens 1-49 as input. You can't generate token 50 without first having 49, which requires 48, and so on. This sequential dependency is fundamental to the autoregressive (next-token-prediction) design of all major LLMs.