chapter 03 · part 1
Embeddings
the hook
How can math know that 'king' minus 'man' plus 'woman' is 'queen'?
the idea
Tokens get ID numbers, sure — but IDs alone don't carry meaning. Token 50257 isn't 'more related' to 50258 just because they're neighbors in a list. Embeddings are the fix.
An embedding turns a token's ID into a point in space — usually hundreds of dimensions, but the idea is identical in 2: similar meaning, nearby points. 'Dog' and 'wolf' end up close together. 'Dog' and 'orange' don't.
This is where the famous trick comes from: king − man + woman ≈ queen. Not because a computer 'understands' royalty — because the direction from 'man' to 'woman' and the direction from 'king' to 'queen' point roughly the same way in the space. Subtract, add, land near the answer.
Drag the points below and watch what 'wrong' looks like. Then hit the arithmetic demo and watch the math find queen anyway — unless you've dragged things too far.
playground — your hands now
what: drag word-points around a 2D map and run the king − man + woman demo.
why: this is literally how models represent meaning — as position, not as text.
how: drag a point, then press the arithmetic button and read the closest match.
start here → drag 'king' around and watch the queen distance change.
predict first: which word do you think will be closest to king − man + woman?
king(22,16) queen(40,24) prince(14,24) princess(20,31) man(18,70) woman(36,78) dog(75,22) wolf(68,14) cat(82,30) apple(66,74) orange(79,82) banana(72,90)
try this
Drag 'apple' out of the fruit cluster and into the royalty cluster. Run the king − man + woman demo again — does it still land near queen, or does it now land near your misplaced apple? Meaning lives entirely in position. Wreck the positions, and you wreck the meaning — that's the whole reason training an embedding space carefully matters.
checkpoint
1. What does the DISTANCE between two embedding points represent?
2. In king − man + woman ≈ queen, what is actually being computed?
recap card — keep this
every token becomes a point in space. meaning = position. similar meaning = nearby points.