A client application calling an LLM API in a tight loop starts receiving HTTP 429 responses. What is the generally recommended way to handle this, per common LLM provider API documentation?
- Immediately retry the exact same request as fast as possible in a loop, since 429 responses are transient and will resolve within milliseconds if retried aggressively
- Switch to a different, unrelated API endpoint entirely, since a 429 on one endpoint indicates that the provider's entire platform is unavailable
- Reduce the `max_tokens` parameter on the failing request, since 429 responses indicate the requested output would be too long to generate
- Back off and retry after a delay that increases with each subsequent failure (exponential backoff), typically with some added random jitter, rather than retrying immediately or at a fixed short interval
Why D? And why not the others?
Correct answer: D. Back off and retry after a delay that increases with each subsequent failure (exponential backoff), typically with some added random jitter, rather than retrying immediately or at a fixed short interval
A 429 status code signals that the client has exceeded a rate or usage limit (such as requests per minute or tokens per minute), and the standard remediation across REST-style LLM provider APIs is exponential backoff: wait progressively longer between retries as failures continue, with random jitter added so that many concurrent clients don't all retry at the exact same moment and cause a new burst of failures. Retrying immediately and aggressively is the opposite of what is recommended, since it worsens the very condition causing the 429s rather than relieving it. A 429 on one endpoint reflects that specific endpoint's or account's usage limit, not a platform-wide outage, so switching to an unrelated endpoint does not address the underlying limit at all. Reducing `max_tokens` addresses a different failure mode entirely -- a request that would generate too much output or exceed a context-length limit -- not a rate-limit error, which is about request or token volume over time rather than the size of any single request.
Source: OpenAI, 'Rate limits' guide, https://platform.openai.com/docs/guides/rate-limits; Anthropic, 'Rate limits' documentation, https://platform.claude.com/docs/en/api/rate-limits