Transformer Architecture

transformer_arch

// The building block of every modern AI — how it's structured inside

💡 The core idea: Every AI you use — ChatGPT, Claude, Gemini, Llama — is built on the "transformer" architecture invented in 2017. Think of it as an assembly line: your text goes in one end, gets processed by many specialized stations, and a prediction comes out the other end. Click "step" to visit each station.

Step 0 / 0

Ready — click "step" to begin

Select a step

Click "step" or a block to learn about each component of the transformer architecture.

Transformer Architecture

The transformer (Vaswani et al., 2017) replaced RNNs with attention. GPT-style models use a decoder-only stack: each token can only attend to previous tokens (causal masking). BERT uses encoder-only: bidirectional attention to understand text.