chapter 20 · part 4
Evals
the hook
'It seems good' is not engineering. How do you PROVE your AI works?
the idea
'It seems good' is not engineering. An eval is a test suite for AI behavior — a fixed set of inputs with known-good expected outputs, run automatically, every time something changes.
Without evals, you find out your model got worse when a user complains. With evals, you find out the moment you make the change — before anyone else sees it.
Below: a tiny fake 'model', a suite of test cases checking its behavior, and a regression — watch what happens when the model changes and the eval suite catches exactly what broke.
This is the same pattern behind every serious AI product's CI pipeline: define expected behavior, test automatically, block bad changes before they ship.
playground — your hands now
what: write test cases for a tiny fake model, run them, then watch a regression get caught when the model changes.
why: evals are the entire difference between 'I think it's better' and 'I can prove it didn't break anything.'
how: run the eval suite, add your own test case, then switch the model version and run again.
start here → run the eval suite, add a test case, then switch model versions and run again.
the "model": summarize(text) → first 3 words + "..."
input: "the quick brown fox jumps"
expected: "the quick brown..."
input: "artificial intelligence is transforming the world"
expected: "artificial intelligence is..."
input: "python is a great programming language"
expected: "python is a..."
what: write an assert-based eval in python.
why: this exact pattern — assert actual == expected — is the core of every real eval suite, just with more test cases.
how: call summarize(), compare it to the expected string with assert, print a message if it passes.
predict first: if summarize() has the 2-word regression, will this assert pass or fail?
finish the assert so it checks summarize() gives the right output. stuck? there's a hint button.
try this
Add a test case designed to specifically catch the regression you're about to see (hint: the bug drops from 3 words to 2) — then watch your own test be the one that fails.
checkpoint
1. What's the main purpose of running the SAME eval suite before and after a model/code change?
2. Why is an assert-based test more useful than eyeballing a single output?
recap card — keep this
evals = test cases for AI. define good, measure it, track it on every change.