chapter 14 · part 3
Context windows
the hook
Why does the model 'forget' the start of a long chat? Watch its memory physically run out.
the idea
Every model has a hard limit on how much text it can 'see' at once — the context window. Once your conversation crosses that limit, the oldest parts don't get summarized or compressed, they just vanish from what the model can see.
This isn't the model 'forgetting' emotionally — it's a hard technical wall. Old messages get pushed out, and there's no traceback, no partial memory. Gone means gone.
There's also a cost side to this: every token you keep in context has to be reprocessed (or at least re-referenced) on every new turn, which is why long conversations get slower and pricier as they grow — more on that in the KV cache section below.
Fill the memory tank below and watch it happen — add messages, watch older ones get pushed out, then try asking about something that already fell out.
playground — your hands now
what: add messages to a fake chat and watch the token tank fill up, then overflow and push out old messages.
why: this is exactly why long chats 'forget' the start — it's not memory loss, it's a hard token budget with nowhere for old tokens to go.
how: add a few filler messages until the tank overflows, then ask about the very first message and see if it still works.
start here → add filler messages until the tank overflows, then ask about the very first message.
tank capacity: 40 tokens (word count, as a stand-in for real subword tokens from ch2)
11 / 40 tokens used
predict first: if you ask about my dog's name RIGHT NOW, will the model still know it?
bonus · the KV cache
what: drag a conversation-length slider and watch the compute cost with vs without caching.
why: this quadratic-vs-linear gap is the literal reason long chats get slower and pricier.
how: drag the slider, watch both numbers, read the multiplier at the bottom.
without KV cache
≈ 2,500 compute units / new token
re-attends over every past token, every step
with KV cache
≈ 50 compute units / new token
reuses cached keys/values, only computes the new token
at 50 tokens of conversation, caching saves roughly 50× the compute per new token — but the cache itself still holds all 50 tokens in memory, which is why cost never fully flattens out.
try this
Raise the stakes: imagine putting an important fact in your SECOND message instead of the first, then filling the tank. Would it survive longer than the first message did? By how much? That's context budgeting — the same instinct behind why system prompts usually go first.
checkpoint
1. When a conversation exceeds the context window, what actually happens to the oldest messages?
2. Why do long conversations get slower and more expensive, even with KV caching turned on?
recap card — keep this
context = the model's entire working memory, measured in tokens. outside it = doesn't exist.