Rate Limits: The Wall You'll Hit Right When Things Start Working
The API caps how fast you can call it. Plan for the 429, because you will meet it.
Rate limits are the thing you never think about while you're building and slam into the moment you go to production. Everything works beautifully in testing — you're making a handful of calls — and then real traffic arrives, the API starts returning 429s, and you're learning about rate limits the hard way. Better to learn them the easy way.
What a rate limit is
A rate limit is a cap on how much you can call the API in a given window of time. It's not one number; it's usually a few, measured along different axes — requests per minute and tokens per minute being the common ones. Exceed any of them and the API stops serving you and returns an error (the famous 429, "Too Many Requests") until the window resets.
The reason they exist is mundane and reasonable: shared infrastructure. The provider is protecting the service from any single customer — or a runaway loop in your code — overwhelming it. It's the same reason every serious API has them.
Why tokens-per-minute trips people up
The requests-per-minute limit is intuitive. The tokens-per-minute one catches people out, because it means your effective capacity depends on how big your calls are, not just how many. A hundred tiny requests and ten enormous ones can hit the same token ceiling. If you're sending large prompts or asking for long outputs, you'll exhaust the token budget long before the request budget — and it won't be obvious why you're throttled while making "only a few" calls.
The failure mode, and the fix
Here's how it usually goes wrong. You build something that fires requests in a tight loop — processing a list, backfilling data, handling a spike — and you blow through the limit almost instantly. The naive version just crashes or drops work.
The grown-up version expects the 429 and handles it. Two habits cover most of it. First, retry with backoff: when you get throttled, wait a moment and try again, increasing the wait each time rather than hammering the door. Second, control your own send rate so you stay under the ceiling in the first place, instead of sprinting into it and recovering. Any client library worth using has this built in; the mistake is assuming you'll never need it.
The mindset
Treat rate limits as a normal operating condition, not an error. The 429 isn't a bug in your code or theirs — it's the API telling you to slow down, and a well-built integration listens. If your volume genuinely outgrows the limits, that's a conversation with the provider about higher tiers, not something to brute-force around.
The teams that get surprised by rate limits are the ones who tested with ten requests and shipped for ten thousand. The ones who don't are the ones who assumed, from day one, that the wall was there — and built to live comfortably on the right side of it.
