LLM API Rate Limits: Which One You Are Actually Hitting

The first 429 usually arrives with a confident diagnosis attached: we are sending too many requests. Then someone halves the request rate, the errors continue, and the real cause turns out to be a token ceiling nobody was tracking. Rate limits are enforced across several dimensions at once, and whichever one you exhaust first is the one that stops you. Requests per minute is frequently not that one.

Which limit are you actually hitting?#

Several run in parallel and the strictest binds. OpenAI enforces requests per minute, tokens per minute, requests per day, tokens per day, and images per minute, noting that whichever limit is reached first applies. Anthropic measures the Messages API in requests per minute plus separate input and output token rates per model class.

Whichever is exhausted first applies. Averaging them into one number hides the one that is binding.

That split between input and output tokens matters more than it looks. A retrieval-heavy workload stuffing long context into every call exhausts an input-token budget while barely touching the output one. A generation workload writing long documents does the reverse. Averaging them into a single "tokens per minute" number hides which side is actually saturated, and the two respond to completely different fixes.

The practical first move is to stop guessing. OpenAI returns the current state on every response in headers covering the request and token limits, what remains of each, and when each resets, alongside a Retry-After value when you are blocked. Source Anthropic returns a 429 that describes which limit was exceeded along with a retry-after header indicating how long to wait. Source Log those. A dashboard showing remaining capacity per dimension turns this from a debugging exercise into a reading.

Why does a limit of 60 per minute reject your tenth request?#

Because the limit is not a bucket that refills at the top of each minute. Anthropic documents its rate limiting as a token bucket algorithm where capacity replenishes continuously rather than resetting, and notes that a 60-requests-per-minute limit might be enforced as 1 request per second, so short bursts can exceed it.

Capacity refills gradually rather than resetting, so the average rate is not what gets measured.

This is the single most common reason a workload that looks well under its limit still fails. Firing 30 requests in 2 seconds and then idling for the rest of the minute averages out to 30 per minute, and gets rejected anyway, because the bucket never held 30 requests' worth of capacity at any instant. The fix is not a lower rate. It is a smoother one.

There is a related mechanism worth knowing about before it surprises you. Anthropic also documents acceleration limits, which can produce 429s when an organization's usage increases sharply, and advises ramping traffic gradually and maintaining consistent usage patterns. A launch that goes from near-zero to full volume can therefore hit limits its steady-state quota would comfortably support.

Why does max_tokens spend quota you never use?#

Because token limits are calculated against estimated usage, not actual usage. The provider has to reserve capacity before generating, so it assumes your request could produce the full output you allowed. Set a ceiling of 4,000 tokens for a reply that runs 200, and the other 3,800 were charged against your budget anyway.

OpenAI states this directly, recommending you set the max token value as close to your expected response size as possible, because limits are calculated against estimated token usage. Source

Modeled for a workload whose replies run 200 tokens. Reservation happens before generation, so the gap is pure waste.

The modeled figures above show why this is the cheapest fix available on most rate-limited workloads: it costs 1 configuration change and no architecture. It is also the one people resist, because a generous ceiling feels like insurance against truncated responses. Size it from your actual output distribution instead. Look at the 95th percentile length your workload really produces, add headroom, and set it there rather than at a round number chosen for comfort. The same token math you would use to estimate cost tells you what that number is.

What should retry logic actually do?#

Honor the Retry-After header first, then fall back to exponential backoff with jitter. Both providers return that header on a 429 and both recommend backoff over immediate retry. Most importantly, check whether your SDK already does this, because retry logic layered on top of built-in retries produces surprising delays.

The order matters: an explicit retry-after value beats any delay you would have calculated.

That last point catches teams regularly. Anthropic's official SDKs automatically retry transient failures including connection errors, rate limits, and 5xx server errors, with exponential backoff, twice by default, honoring the retry-after header when present. OpenAI notes the same behavior in its SDKs. If you wrap that in your own 3-attempt retry loop, a single request can now make 9 attempts and take far longer to fail than anyone intended. Configure the SDK's retry count rather than adding a second layer.

Jitter is the part hand-rolled implementations usually miss. If every worker that got a 429 waits exactly 1 second and retries, they all retry simultaneously and collide again. Randomizing each delay spreads the retry storm out. And a retry loop needs a ceiling, because some errors are not transient: a quota or billing failure will never succeed on retry, and OpenAI's guidance explicitly distinguishes those from temporary rate-limit errors that need manual intervention rather than patience.

The subtler failure is retrying work that already happened. A request that timed out on your side may have completed on theirs, so a naive retry can bill you twice and, if the call had side effects, run them twice. Rate-limit rejections are safe to retry because nothing ran, but timeouts and connection errors are not the same case, and treating every failure as identically retryable is how duplicate work enters a pipeline. Distinguish the two in your handler rather than wrapping everything in the same block.

How do you stop hitting limits at all?#

Cap concurrency instead of reacting to rejections. A fixed ceiling on in-flight requests, with work queued behind it, converts a bursty caller into a steady one and eliminates most 429s at the source. Retry logic then handles the exceptions rather than carrying the whole load.

Measurement comes first: every wrong fix here starts with tuning a dimension that was never the constraint.

The queue is doing 2 useful things at once. It smooths bursts into a rate the token bucket can absorb, and it gives you a place to apply priority, so an interactive request is not stuck behind a batch job. That second property matters more as the workload grows, because it is the difference between degrading gracefully under load and failing uniformly.

Anything that does not need an immediate answer should leave the synchronous path entirely. Both providers offer batch endpoints designed for exactly this, and the economics are usually favorable on top of the capacity relief, which is the case made in the batch API discount. Moving nightly summarization or bulk classification off your interactive quota often solves the problem outright, without any tuning.

There is also a request-shape lever worth knowing. OpenAI notes that when you are hitting request-per-minute limits while token capacity remains, batching several prompts into a single request increases throughput, since the constraint is the number of calls rather than the work inside them. That only helps in the specific case where requests, not tokens, are binding, which is why identifying the right dimension comes first.

What should you build first?#

Instrument before you optimize. Log the remaining-capacity headers on every response, chart them by dimension, and find which one actually saturates first under real traffic. Nearly every wrong fix in this area comes from tuning a dimension that was never the binding constraint, which is why the measurement step pays for itself immediately.

Then work in order of cost. Right-sizing your output ceiling is a configuration change and often recovers substantial headroom immediately. Capping concurrency is a small amount of code and removes most bursts. Moving deferrable work to a batch endpoint is a bigger change but takes entire workloads off the interactive budget. Only after those does it make sense to ask for a quota increase, which is the slowest lever and the one providers grant based on the usage history the earlier steps produce.

Two things are worth building before you need them. A queue with a concurrency ceiling is far easier to add while the system is calm than during an incident, and it is the same infrastructure that makes retries and failure handling durable rather than best-effort. And if you serve traffic that can tolerate a substitute, routing overflow to a second provider or a smaller model turns a hard ceiling into a soft one, which is the underlying argument for a gateway in front of your model calls. Neither is urgent on day 1. Both are much cheaper to add before the launch that makes them necessary than during it.