AI Gateway

Limits, budgets & errors

Rate limits, spend caps, and the error envelope.

Every error uses the OpenAI-style envelope, so SDK error handling just works:

Error shape

{
  "error": {
    "message": "Rate limit exceeded for key (60 req/min). Retry in 12s.",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded",
    "param": null
  }
}

What can limit a request

Status	Code	Meaning
`429`	`rate_limit_exceeded`	Requests/min, tokens/min or concurrency cap hit, or the model is busy. Honor the `retry-after` header.
`402`	`budget_exceeded`	A hard spend cap (key, project or account) is used up for the period.
`413`	`request_too_large`	Body over the inference size limit (10MB).
`400`	`bad_request`	A document could not be read, or is over a document limit. The `param` names the content part.
`422`	`document_too_complex`	A document took too long to read. Split it, or send fewer pages at a time.
`422`	`guardrail_blocked`	Your account's data guardrails blocked the request (e.g. credit card numbers headed to an external model).
`502`	`upstream_error`	The model could not be reached, including after failover. Safe to retry.

The limits you start with

Limit	Default	Applies to
Requests per minute	600	Each API key, separately — including the key each hosted app gets
Tokens per minute	None	Set per account on request
Concurrent requests	None	Set per account on request
Account-wide requests per minute	None	Set per account on request, as a ceiling across all your keys
PDF documents	5 per request, 8 MB each, 100 pages, 10 scanned pages	Every chat request — see Documents

The Limits section of your console's Settings shows the numbers that apply to your account. If you need more than the defaults, ask us and we will raise them.

Separately, and rarely, a model can be busy. You then get a 429 with code rate_limit_exceeded and a message beginning "This model is busy right now", straight away rather than after a long wait. When it comes from us it carries retry-after: 1. It means an abnormal surge has reached the most we can serve at once, and we turn the excess away immediately instead of making every request wait. That capacity is shared fairly between accounts, so another account's surge cannot crowd yours out; if you send far more at once than everyone else, your extra requests are the ones turned away. Retry after the wait. When the busy answer comes from further along and we know how long until the model can be tried again, retry-after carries that wait; when we do not, the header is absent rather than guessed.

Seeing a limit before you hit it

Every gateway response — streamed included — carries the ceilings we measured for that call, using the same header names the OpenAI SDKs already understand:

Header	Meaning
`x-ratelimit-limit-requests`	Your requests/min ceiling
`x-ratelimit-remaining-requests`	Requests left in the current window
`x-ratelimit-reset-requests`	Time until the allowance is back to full, e.g. `12s`
`x-ratelimit-limit-tokens`	Your tokens/min ceiling, if one is set
`x-ratelimit-remaining-tokens`	Tokens left this minute
`x-ratelimit-reset-tokens`	Time until the token allowance refills

Pace against remaining and you should not see a 429 at all.

A missing header means we are not measuring that limit for you — never that the limit is zero. If no tokens/min ceiling is set on your account, the -tokens headers are absent rather than reported as 0. Treat absence as "no ceiling of this kind", and read the values that are present.

Two things that can refuse a request are deliberately not reported here, because neither has an honest reading in these units:

Concurrency — the cap on requests in flight at once. There is no OpenAI header for it; you will see a 429 with retry-after if you exceed it.
Budgets and credit balance — measured in dollars, on your billing period rather than a per-minute window, and returned as 402, not 429. Mapping a balance onto a token ceiling would be a guess, and you would pace against it. Use GET /client/budget for the real number.

Budgets

Spend caps can be set per key, per project, and account-wide — the tightest one wins. A cap is either blocking (requests get 402 at the limit) or alert-only (email warning, requests keep flowing). Check live budget status with GET /client/budget using a key with usage:read.

A batch settles hours after you submit it, so it is checked against a blocking cap at submission using an estimate of its cost, and that estimate counts against your budget until the job finishes. A batch too large to fit is refused up front rather than discovered halfway through.

Retries

Idempotent handling on your side plus exponential backoff on 429 and 502 is all you need — the gateway has already failed over and retried before a 502 reaches you.