chapter 08 · part 2
Backprop & gradient descent
the hook
Drop a ball on a hilly landscape. Control one dial. Too high — it flies forever. Too low — it never arrives.
the idea
Gradient descent has exactly one dial: the learning rate. It controls how big a step you take downhill after each gradient check. Below, you'll turn that dial yourself and watch what happens.
The gradient (the slope at your current position) always points in the direction of steepest INCREASE. To go downhill — to reduce loss — you step in the OPPOSITE direction. That's the entire 'descent' in gradient descent.
Learning rate too small: each step barely moves you. You'll eventually get there, but painfully slowly — imagine tip-toeing down a hill in the dark.
Learning rate too large: you leap right over the valley and land higher up on the other side than where you started. Do this enough times and the ball doesn't descend — it launches.
playground — your hands now
what: pick a learning rate, then watch a ball perform real gradient descent down a loss curve.
why: learning rate is the single most important dial in all of machine learning — too big or too small and training simply doesn't work.
how: set the learning rate slider, hit 'run', and watch where the ball ends up.
start here → set the learning rate, hit run, watch the ball.
predict first: with THIS learning rate, will the ball converge, creep too slowly, or fly off the landscape?
try this
Find 3 learning rates: one that creeps so slowly it barely moves in 50 steps, one that glides smoothly to the bottom, and one that sends the ball flying off the landscape entirely. Same hill, same starting point — only the dial changed.
checkpoint
1. Which direction does the gradient point, and which direction do you actually step?
2. What tends to happen with a learning rate that's too large?
recap card — keep this
gradients point uphill; we step downhill. learning rate = step size, the #1 dial in all of ML.