part 4 boss 👾
Part 4 Boss: Is This AI Ready to Ship?
the hook
A fictional 'ready to ship' report, full of confident numbers. 3 of them are real red flags. Find all 3 — everything else genuinely checks out.
the idea
'Ready to ship' reports look authoritative — numbers, percentages, charts. That doesn't mean they're trustworthy. The report below has 3 real red flags hidden among statements that sound just as confident.
This is the last mile of everything you've learned: hallucination awareness (ch19), evals (ch20), scaling/cost awareness (ch21) — all converging into one skill: can you tell if an AI system is actually ready, or just LOOKS ready?
Find all 3. The rest of the report is genuinely fine — don't flag things just because they're numbers you don't immediately understand.
playground — your hands now
what: read a fictional ship-readiness report and find the 3 real red flags among several unrelated (fine) statements.
why: this exact skill — reading past confident-sounding numbers to what's ACTUALLY being measured — is what separates a real ship/no-ship decision from rubber-stamping a report.
how: flag whichever statements you think are red flags, then submit and see what you caught.
start here → flag whichever statements you think are red flags, then submit.
CustomerBot v2 — Ship Readiness Report
try this
For each flag you correctly caught, think through what SPECIFIC eval or process change would actually fix it — that turns 'this is a mistake' into an improvement plan.
checkpoint
1. Why is a tiny, non-independent eval set (written by the same engineer who built the feature) a red flag, even at 98.7% accuracy?
2. Why does testing on the same data used for finetuning inflate an eval score meaninglessly?
recap card — keep this
confident numbers aren't the same as trustworthy numbers — check the sample, check for contamination, check what wasn't tested at all.
🎉 that's every chapter live so far — you've reached the end of what's built. more landing soon.