chapter 12 · part 3
Sampling & temperature
the hook
One slider separates 'boring corporate email' from 'unhinged poetry'. It's yours now.
the idea
A model doesn't output ONE word — it outputs a probability for every word in its vocabulary, every single time. Sampling is the process of turning that giant list of probabilities into one actual choice.
Temperature controls how adventurous that choice is. Low temperature (near 0): always grab the highest-probability word — safe, repetitive, predictable. High temperature: probabilities flatten out, rare words get a real shot — creative, occasionally unhinged.
Greedy decoding (temperature → 0) always produces the exact same output for the exact same input. Sampling with temperature > 0 introduces real randomness — ask the same question twice, get two different answers.
One more thing you've definitely noticed: chat replies appear word by word instead of all at once. That's streaming — the model generates one token at a time regardless, streaming just ships each one the moment it's ready instead of making you wait for the whole reply.
Below: 8 candidate next-words, real probabilities, a temperature dial you control. Watch the bars morph as you turn it, then hit sample and watch it actually pick one.
playground — your hands now
what: turn a temperature dial and watch 8 word probabilities morph, then sample real words into a growing sentence.
why: temperature is the one setting that separates a boring corporate chatbot reply from unhinged creative writing — same model, same weights, different dial.
how: drag the temperature slider, predict, then hit sample and watch the sentence build — compare it to the greedy (always-safe) version running alongside.
start here → drag temperature, predict, then hit sample a few times and watch the sentence build.
next word after: "the weather today is"
predict first: at THIS temperature, will the sampled word probably match the greedy (safe) pick?
sampled (temperature 1.00)
the weather today is …
greedy (always safe)
the weather today is …
bonus · streaming
what: watch the same response arrive two ways — streamed token-by-token, or dumped all at once.
why: the model generates one token at a time either way — streaming just decides WHEN you get to see each one.
how: pick a mode, hit generate, and time how long it feels before you see anything.
what: implement temperature scaling in numpy: logits / temperature, then softmax.
why: this exact 2-line trick — divide, then softmax — is the real implementation behind the slider you just played with.
how: scale logits by temperature first, then apply the standard softmax formula.
predict first: at temperature=0.5, will the top candidate's probability be higher or lower than at temperature=1.0?
given logits and temperature, compute probs. round probs[0] to 2 decimals when printing. stuck? there's a hint button.
try this
Set temperature to 0.1 and sample 5 times — notice you get the exact same sentence, every time. Now set it to 2.0 and sample 5 times — watch it go somewhere different every single time. That's the entire creativity dial of every chatbot you've used.
checkpoint
1. What does temperature actually control?
2. Why can a model start streaming text back to you almost immediately, even though the full response isn't finished?
recap card — keep this
models output probabilities. sampling picks. temperature = how adventurous the pick is.