chapter 06 · part 1
The transformer block
the hook
GPT, Claude, Gemini, Llama — all of them are this one block, copy-pasted.
the idea
Before any of that though, there's a problem: attention treats tokens as a bag, not a sequence — it has no built-in sense of 'first' or 'third'. Feed it 'dog bites man' and 'man bites dog' and, without help, it sees the same 3 tokens either way. The fix: add a small 'position vector' to every token's embedding before anything else happens — literally encoding 'you are token #0', 'you are token #2', right into the numbers.
GPT, Claude, Gemini, Llama — strip away the branding and they're all the same 4 steps, copy-pasted dozens of times: attention, add the result back in, a small feedforward network, add that back in too. That's a transformer block.
'Add the result back in' is called a residual connection — instead of replacing the input, you add the new computation ON TOP of it. This one trick is a big part of why you can stack dozens of these blocks without training collapsing. Without it, every block would completely REPLACE its input with something new — stack enough replacements and the original signal gets diluted into mush a few layers in. Residual connections keep a direct path back to the original input at every block, which is a big part of why you can stack 96 of these and still train the thing successfully.
Because every block outputs the exact same shape it received (same number of tokens, same number of dimensions per token), blocks stack like lego — the output of block 1 slots directly into block 2, no adapter needed.
First, add positional encoding, then put the 4 steps in the right order below. Then stack a few blocks and watch a real (tiny) set of token vectors flow through — shape identical at every stage, values transformed every time.
playground — your hands now
what: put the 4 steps of a transformer block in order, then watch real token vectors flow through a stack of them.
why: this exact 4-step block, repeated dozens of times, is the entire architecture behind every major LLM.
how: drag the 4 steps into order, then use the step buttons to watch values flow through the stack.
start here → order the 4 steps, then step through the block stack below.
step 0 · positional encoding
what: add real position vectors to token embeddings, then compare 'dog bites man' vs 'man bites dog'.
why: attention treats tokens as an unordered set — position vectors are the only thing telling the model where each word actually sits.
how: toggle positional encoding on/off and watch whether 'dog' looks different depending on where it sits.
dog (pos 0)
bites (pos 1)
man (pos 2)
watch 'dog's card — its colors change depending on which position it lands in, because position 0 and position 2 get added different position vectors.
step 1 · order the block
step 2 · stack blocks, watch shapes flow
what: stack real transformer blocks and step through them, with residual+norm on or off.
why: residual connections are the plumbing that stops deep nets from dying — turn them off and watch the original signal disappear.
how: pick a block count, toggle residual+norm, then click through each step and watch 'signal retained' below.
shape: 4 tokens × 6 dims
signal retained (similarity to original input): 100%
same shape at every step (4 × 6) regardless of the toggle — only "signal retained" and the cell colors tell you whether the original input is still in there.
try this
Stack 4 blocks and step through one at a time. Watch the shape label — does it ever change? Now imagine doing this 96 times, the way GPT-3 does. Same trick, just more of it.
checkpoint
1. What's a residual connection, in one line?
2. Why does it matter that every transformer block outputs the exact same shape it received?
recap card — keep this
transformer block = attention + feedforward + residuals. LLMs = this block × dozens.