Rate limits & quotas
RPM and concurrency limits, key quotas, account balance, and payload size caps.
The gateway applies several layers of limits to every request, from the outside in: rate (RPM/concurrency) → key spending limit → account balance → payload size. Understanding which error each layer produces is what makes retry logic correct.
RPM & concurrency
Requests-per-minute (RPM) and concurrent-session caps are configured per account. Exceeding them returns 429:
rpm_exceeded: too many requests per minute; the error message includes the limit. Back off and retry.concurrent_exceeded: too many requests in flight at once. Wait for in-flight requests to finish, or lower client concurrency.
Note: metadata requests such as Models also count toward RPM — don't poll them in a tight loop.
Key spending limits
Each key can have a USD spending limit, optionally resetting daily/weekly/monthly. When exhausted, requests return 402 (key_quota_exhausted) until the reset or a higher limit. Use this to give each workload or environment its own budget.
Account balance
An insufficient balance returns 402 (balance_exhausted); requests resume after topping up.
Billing is post-paid: a request's cost is only known once the upstream reports token counts, so the balance can dip slightly negative; the debt is settled on your next top-up. Enable low-balance alerts so workloads don't hit 402 unexpectedly.
Payload size
- 413 (
body_too_large): the request body exceeds this endpoint's size cap. Chat endpoints allow inlined media; text-only endpoints (such as embeddings) have a smaller cap. Compress inlined media or use an endpoint sized for large payloads. - 503 (
server_busy): the gateway's total in-flight body volume hit its ceiling — your request isn't too big, the gateway is momentarily busy. Retry shortly.
Practical advice
- Retry 429/502/503 with exponential backoff; alert (don't retry) on 402 — see Error handling & retries.
- Monitor how often you see 429 and 402: the former means lower concurrency or ask for higher limits, the latter means top up or adjust budgets.
- Leave generous timeouts for long and streaming requests so clients don't cut them off early (see Streaming).
- Split keys per workload with separate limits, so one workload overspending never affects the others.