Max Tokens: The Setting That Quietly Cuts Off Your AI Mid-Sentence
It's a length limit and a cost lever — and the reason your output sometimes just stops.
If you've ever watched an AI response stop dead in the middle of a sentence, you've met the max tokens setting without knowing its name. It's one of the least glamorous knobs in working with these models, and one of the few genuinely worth understanding, because it controls two things you actually care about: how long the output can be, and how much it costs.
What a token is, quickly
Models don't read or write in words. They work in tokens — chunks of text that are often a whole word, sometimes part of one, sometimes just a punctuation mark. A rough rule is that a token is about four characters of English, so a hundred tokens is roughly seventy-five words. You don't need the exact ratio; you need the idea that the model counts in tokens, and everything — pricing, limits, memory — is measured in them.
What max tokens actually does
Max tokens is a cap on how many tokens the model is allowed to generate in its response. That's the whole feature. Set it low and you get a short answer — or worse, one that gets cut off mid-thought when it hits the ceiling before it was finished. Set it high and you give the model room to write at length, but you're also authorising a bigger bill and a longer wait.
The cut-off-mid-sentence problem is the one that surprises people. The model didn't fail or run out of ideas; it hit the limit you (or the tool's default) set, and generation simply stopped. If your outputs keep ending abruptly, this setting is almost always the culprit.
Why it's a cost and speed lever, not just a length one
Here's the part worth internalising. You pay per token, and longer generations take longer to produce. So max tokens isn't only "how long can the answer be" — it's a direct handle on cost and latency. Capping it stops a runaway response from generating, and charging you for, far more than you needed. Raising it removes that guardrail.
There's also a shared budget to respect: output tokens and input tokens both count against the model's context window. A very long prompt leaves less room for a long answer. Send a huge input and also demand a huge output, and you can run into the ceiling from both sides at once.
The practical version
Set max tokens a bit above the longest reasonable answer you actually expect — not to the maximum "just in case." Too low and you truncate real answers; too high and you quietly pay for headroom you never use and occasionally let a response ramble. If answers keep getting chopped off, raise it. If the bill on a high-volume endpoint is creeping up, this is one of the first dials to check.
It's a small setting that people ignore until it bites them. But it's the whole difference between an answer that finishes its thought and one that just stops.
