chapter 02 · part 1
Tokenization
the hook
Why does the model think 'strawberry' has two r's? Because it never sees letters.
the idea
Tokenization is the very first thing that happens to your text — before attention, before any layer of the network, before anything. Get this step wrong and everything downstream inherits the mistake.
The dumbest tokenizer just splits on spaces: `"hi there".split()` → `["hi", "there"]`. Works fine until punctuation shows up, or a word the tokenizer has never seen before does.
Real tokenizers keep a fixed vocabulary of known chunks and greedily match the longest one they can find at each position — exactly what ch1's playground did in JavaScript. Now you'll write that same logic yourself, in real Python.
The interesting part isn't matching known words. It's what happens when a word ISN'T in the vocabulary — that's where 'strawberry' gets shredded into pieces.
playground — your hands now
what: build a tokenizer from scratch, in two steps.
why: you'll never look at 'AI is just autocomplete' the same way once you've written the chopping logic yourself.
how: run the warm-up cell, then write the real tokenizer below it.
start here → run the warm-up cell first, then write the real tokenizer.
warm up · the dumb way
what: split a sentence into words using .split().
why: this is the laziest possible tokenizer — you need to feel its limits before building a real one.
how: fill in the method name, press run.
predict first: once you fix the blank, what will tokens look like?
fill in the method that splits a string on spaces, then run.
now write a real one
what: write a greedy longest-match tokenizer function.
why: this exact logic — minus the training — is what turns your messages into tokens before any model sees them.
how: finish tokenize(), press run — stuck? tap 'need a hint?' up to 3 times for the full answer.
predict first: once tokenize() works, what will tokenize("strawberry") return?
finish tokenize() — greedily match the longest chunk in VOCAB that the remaining text starts with. no match? peel off one character as its own token. stuck? there's a hint button.
try this
Run the tokenizer below on a word that's fully in the vocab, then on one that isn't. Watch the unknown word get chopped into single characters — that fallback is exactly why models sometimes 'miscount' letters. They're not looking at letters, they're looking at whatever chunks survived matching.
checkpoint
1. A greedy tokenizer scans a word and matches the longest known chunk it can find first. Why longest, not shortest?
2. What happens when a tokenizer hits a chunk of text that matches nothing in its vocabulary?
recap card — keep this
text becomes tokens (chunks), tokens become numbers. models only ever see numbers.