chapter 04 · part 1
Neural nets 101
the hook
You'll hand-tune 9 neurons to classify points. You will lose to gradient descent. That loss is the lesson.
the idea
A neuron does three things: multiply, add, squash. Take some numbers, multiply each by a weight (a tunable number), add them up plus one more tunable number (a bias), then squash the result into a small range so it doesn't explode.
Stack a few neurons side by side — that's a layer. Stack a few layers — that's a network. Every 'neural net' you've ever heard of, from a 3-neuron toy to GPT-4, is this pattern, repeated.
The squashing function here is tanh — it takes any number and squeezes it into [-1, 1]. Below, positive output means 'inside', negative means 'outside'.
Training means tuning the weights and biases so the squashed outputs land on the right side for as many points as possible. You're about to try doing that by hand. Good luck — you'll need it.
playground — your hands now
what: hand-tune 9 numbers to classify points, then race gradient descent on the same task.
why: this is the entire 'training' problem in miniature — before trusting GD to tune millions of numbers, tune 9 yourself.
how: drag the 9 sliders, watch the left canvas change, then press 'run gradient descent' to compare.
start here → drag any slider and watch the left canvas repaint.
your weights
accuracy: 56% · loss: 1.183
gradient descent
press the button below →
neuron 1
neuron 2
neuron 3
predict first: will gradient descent beat your hand-tuned accuracy?
try this
Get your hand-tuned accuracy above 80% before hitting 'run gradient descent'. Then look at how few steps GD needed to beat your best score, and how little effort it took compared to your dragging and squinting.
checkpoint
1. What does a single neuron actually compute?
2. Why squash (tanh) the output instead of leaving it as a raw number?
recap card — keep this
a neural net = weights + multiply + add + squash, stacked. training = auto-tuning weights.