chapter 21 · part 4
Scaling — from 10K to 1T params
the hook
Your ch9 model and Claude are the same species. One just ate more.
the idea
Everything you've built in this app scales up, not sideways. Same architecture (transformer block × dozens), same training loop, same tokens — just more of everything: more parameters, more data, more compute.
Bigger models don't just get 'a bit better' — certain abilities seem to switch on abruptly at certain scales (this is called emergent behavior). A 10M-param model can barely form a sentence. A 100B+ param model can reason through multi-step problems it's never seen phrased that way before.
But bigger also means heavier. A model's raw memory footprint scales directly with its parameter count — which is exactly the problem quantization solves. It shrinks each number's precision (fp32 → fp16 → int8 → int4) instead of shrinking the model itself. Same parameter count, way less memory — that's why a 7B model can run on your laptop instead of needing a server rack.
One more scaling trick, in one minute: images aren't fundamentally different from text to a transformer. Chop an image into small patches, turn each patch into a token-like vector, and feed it into the exact same architecture alongside text tokens. That's most of how 'multimodal' models work.
playground — your hands now
what: drag a parameter-count slider from 10K to 1T and watch memory, data needs, and abilities change in real numbers.
why: this is why bigger models cost more, need more data, and can suddenly do things smaller ones simply can't.
how: drag the slider, then check the quantization example below to see why smaller numbers (not fewer parameters) is what fits big models on small hardware.
start here → drag the slider from 10K to 1T parameters and watch the numbers change.
training data (illustrative)
20B tokens
~20 tokens per parameter (Chinchilla-style ratio)
memory footprint (fp16)
2.0 GB
2 bytes × parameter count
abilities at this scale
few-shot learning, basic reasoning
everything unlocked so far:
bonus · quantization
what: see the same 7B-parameter model's memory footprint at 4 different precisions.
why: this is the literal reason a model can run on your laptop instead of needing a data center — fewer bits per number, not fewer numbers.
how: compare the 4 cards — a typical laptop has 8-16GB of RAM.
fp32
28.0 GB
needs a server
fp16
14.0 GB
needs a server
int8
7.0 GB
fits a laptop
int4
3.5 GB
fits a laptop
same 7-billion-parameter model, every time — only the number of bits per parameter changes.
bonus · multimodality in one minute
what: see how an image gets chopped into patches that become tokens, just like words.
why: this patch-embedding trick is most of how 'multimodal' models see images at all.
how: look at the grid — each colored patch becomes one token, fed into the same transformer as text.
16 patches → 16 tokens, mixed right in with your text tokens in the same sequence.
try this
Find the slider position where 'multi-step reasoning' first appears in the abilities list. The exact threshold here is illustrative, but the real phenomenon is genuine and well-documented — some abilities really do seem to appear abruptly at scale, not gradually.
checkpoint
1. What does quantization actually reduce?
2. In the patch-based approach to multimodal models, how does an image become something a transformer can process?
recap card — keep this
same architecture + more data + more compute = emergent abilities. that's the whole secret.