chapter 05 · part 1
Attention
the hook
In 'the trophy didn't fit in the suitcase because it was too big' — how does the model know what 'it' means?
the idea
Read this: 'the trophy didn't fit in the suitcase because it was too big.' You instantly know 'it' means the trophy, not the suitcase. How? You quietly checked which nearby word 'it' relates to. That checking process, done by every token for every other token, is attention.
Concretely: each token asks a question ('what am I looking for?') and offers an answer ('here's what I contain') — called a Query and a Key. Multiply a token's Query against every other token's Key, and you get a relevance score per pair.
Turn those scores into percentages (softmax) and you get attention weights — how much each token should 'listen to' every other token. High weight = highly relevant, near zero = ignore.
Below is a real attention weight matrix — computed with actual dot products, not decoration. Click a row to see exactly which words 'it' (or 'she', or 'cat') is paying attention to.
playground — your hands now
what: click a word and see exactly which other words it's paying attention to.
why: this single trick — every token scoring every other token — is what lets models track meaning across a whole sentence.
how: pick a sentence, click a word in the grid, read the highlighted weights below it.
start here → click the word 'it' (or 'she') and see who it's really talking about.
thetrophydidn'tfitinthesuitcasebecauseitwastoobig
'it' is paying attention to:
try this
Switch sentences. Find which word the pronoun ('it' or 'she') attends to most in each one — then check: does it match what YOU think the pronoun refers to? Real attention, in real trained models, gets this right for the same underlying reason our hand-placed numbers do: related words end up with similar vectors, pointing in similar directions.
checkpoint
1. In plain terms, what are a token's Query and Key?
2. Why turn raw attention scores into a softmax distribution?
recap card — keep this
every token asks every other token: 'how relevant are you to me?' — that's attention.