chapter 11 · part 2
RLHF
the hook
Why does Claude feel helpful instead of unhinged? Humans voted.
the idea
A finetuned chat model still isn't done — it needs to learn WHICH of its own possible responses humans actually prefer. RLHF (Reinforcement Learning from Human Feedback) is how that preference gets baked in.
Step one: show human raters 2 possible responses to the same prompt, ask 'which is better?' Do this thousands of times. Step two: train a small 'reward model' to predict those preferences automatically. Step three: use that reward model to nudge the actual LLM toward responses it would score highly.
The raters aren't grading facts, they're grading preference — tone, length, confidence, structure. That's a big part of why RLHF shapes a model's personality and style far more than it shapes its knowledge.
Below, you ARE the human rater. Pick your preferred response across 6 rounds, and watch your own reward model get revealed at the end.
playground — your hands now
what: rank 6 pairs of AI responses by preference — your picks train a tiny visible reward score.
why: this exact ranking process, at massive scale, is literally how models like Claude and ChatGPT got their personalities — not from facts, from millions of preference votes like yours.
how: read each pair, click whichever you prefer, and see your revealed 'reward model' at the end.
start here → read each pair, click whichever response you prefer — 6 rounds total.
round 1 / 6 — "explain what a for loop does"
try this
Go back and deliberately pick the OPPOSITE of your natural preference in a couple of rounds, then check your revealed reward model again. Notice it changes to match. That's the exact mechanism by which inconsistent or biased raters can bake bias into a real model's personality.
checkpoint
1. What does a reward model actually learn to predict?
2. Why does RLHF shape a model's personality/style more than its factual knowledge?
recap card — keep this
humans rank outputs → reward model learns taste → model optimizes for it.