chapter 09 · part 2
The training loop ★
the hook
Paste YOUR text. Train a real tiny GPT on it, live. Watch gibberish become your writing style in 2 minutes.
the idea
Everything in this app has been building to this one loop: predict the next character, measure how wrong you were, nudge the weights a little, repeat. Thousands of times. That's training. That's all training ever is.
This is a real neural net, training in your browser, on YOUR text, right now — not a video, not a simulation. Small (a context window of a few characters, one hidden layer), but the loop is identical to what trains GPT-4.
Two things to watch: the loss curve (should trend down — the model is getting less surprised by your text) and the sample text (should slowly stop being random noise and start echoing your text's patterns — common words, common letter combinations, your rhythm).
Before you get the live trainer, two quick unlocks — the exact two pieces this training loop is built from.
playground — your hands now
what: unlock and run a real tiny neural net that trains on YOUR pasted text, live, in your browser.
why: the loop below — predict, measure loss, backprop, nudge weights, repeat — is exactly the loop that trains GPT-4. only the size differs, not the process.
how: pass both unlock cells, paste your own text (or keep the default), hit train, and watch the loss curve and sample text evolve.
what: one-hot encode a character index.
why: this is literally how the trainer below turns each character of your text into numbers the network can multiply.
how: build a list of zeros the size of the vocab, with a 1 at char_index, press run.
predict first: what will one_hot look like for vocab_size=5, char_index=2?
given char_index and vocab_size, build the one-hot list. stuck? there's a hint button.
what: compute cross-entropy loss for a prediction, one more time.
why: this is the exact loss the trainer below computes, every single step, thousands of times.
how: use math.log() to compute -log(probs[true_index]), press run.
predict first: probs=[0.05, 0.85, 0.1], true_index=1 — small loss or large loss?
given probs and true_index, compute the loss. same formula as ch7 — you'll need it constantly from here on.
🔒 pass both cells above to unlock the live trainer.
try this
Replace the default text with something of yours — lyrics, notes, a chat export. Train for at least 1500 steps. Then compare the sample text at step 50 against step 1500 — that transformation, gibberish to pattern, is the entire miracle of training, compressed into about a minute.
checkpoint
1. What does the loss curve trending DOWN actually mean?
2. Why does the sample text get more coherent as training continues?
recap card — keep this
loop: predict → measure loss → backprop → nudge weights → repeat. same loop as GPT-4, just tiny.