chapter 07 · part 2
Loss — how wrong was the guess?
the hook
Training needs a single number that says 'you were THIS wrong.' You'll write that number.
the idea
Training needs a single number that says 'you were THIS wrong' — that number is called the loss. Get a good number, training works. Get a bad number, training optimizes the wrong thing entirely.
For predicting the next token, the model outputs a probability for every word in its vocabulary. Cross-entropy loss looks at ONE thing: what probability did the model assign to the actual correct word? Then it takes -log of that probability.
Why -log? Because it punishes confidence in the wrong answer HARD, and rewards confidence in the right answer gently. Predict the right word with 99% confidence — tiny loss. Predict it with 1% confidence — loss shoots up fast.
Below, tune 4 raw prediction scores and watch cross-entropy do exactly this: reward correct confidence, punish wrong confidence, brutally.
playground — your hands now
what: tune 4 prediction scores and watch cross-entropy loss punish (or reward) your confidence.
why: this single number — how surprised the model was by the right answer — is what every training step tries to shrink.
how: drag the sliders, or hit a preset button, and watch the loss number react.
start here → hit a preset button, then drag the sliders and watch the loss number.
correct answer: cat
predict first: which gets punished HARDER: being confidently wrong, or just unsure (uniform)?
cross-entropy loss: 1.386
what: write the cross-entropy loss formula in python.
why: this exact formula — negative log of the correct class's probability — is what every model minimizes during training.
how: use math.log() to compute -log(probs[true_index]), press run.
predict first: with probs=[0.7, 0.1, 0.1, 0.1] and true_index=0, will the loss be small or large?
given probs (a list of probabilities) and true_index, compute the loss. stuck? there's a hint button.
try this
Push the true class's slider all the way down and another class all the way up. Watch how fast the loss explodes compared to just being 'a little wrong'. That's -log doing its job — cross-entropy hates confident wrongness way more than it hates plain uncertainty.
checkpoint
1. What does cross-entropy loss actually look at?
2. Why does a confidently WRONG prediction get punished so much harder than an uncertain one?
recap card — keep this
loss = wrongness score. all of training = make this number go down.