Context Length: Why the Model Forgets, and What It Costs You

People are often surprised the first time a chatbot forgets what they said three messages ago. It feels like a bug. It isn't. Understanding why comes down to one idea that explains a huge amount of how these models behave: the context window.

The model has no memory

Here's the thing that trips everyone up. A language model doesn't remember your conversation. Between one message and the next, it retains nothing. What creates the illusion of memory is that the application re-sends the entire conversation so far, every single time. When you send message ten, the model is actually handed messages one through ten as one big block of text, and it responds to all of it fresh.

That block — everything you send plus everything the model generates — has to fit inside a fixed limit called the context window, measured in tokens. The model can only see what's inside that window on any given call. Nothing outside it exists, as far as the model is concerned.

Why it forgets

So the reason a long conversation loses its early details is mechanical, not mysterious. Once the running conversation grows past the context window, something has to give, and the usual answer is that the oldest content falls off the front to make room. The model didn't forget the beginning of your chat — the beginning was simply no longer in the window when it answered. It never saw it.

This is also why pasting a giant document and asking questions sometimes works beautifully and sometimes fails: if the document plus your question fits the window, great; if it doesn't, part of it silently isn't there.

Why bigger windows aren't a free win

Context windows have grown enormously, and it's tempting to think a big enough one makes all of this go away. Just put everything in. Two catches.

First, you pay by the token, on every call. Re-sending a long conversation or a huge document each turn isn't free — it's a cost you incur again and again, and it grows with the length of the exchange. A long-running chat quietly gets more expensive as it goes, because the model is re-reading more each time.

Second, more context isn't automatically better answers. Even when a model technically accepts a huge window, its ability to use the middle of it well tends to degrade as you fill it. Burying the one relevant sentence inside a hundred thousand irrelevant ones doesn't help the model find it. It can hurt.

What this means in practice

Once the context window clicks, a lot of model behaviour stops being surprising. Give the model the context it needs rather than assuming it remembers. Keep what you send relevant rather than dumping everything in and hoping. And when a long conversation starts drifting or forgetting, know that you're bumping against the window, not against a flaw you can prompt your way out of.

The model isn't thinking about your whole conversation. It's reading a fixed-size block of text, once, and answering. Everything it knows, you handed it — and everything you didn't fit, it never had.