A 429 response, a 503 response, and a timeout describe different conditions. They require different limits, delays, and routing decisions. Applying the same immediate retry to all three can increase load, duplicate token usage, and turn a short provider event into an application-wide incident.
1. Interpret the Signal Before Taking Action
429: Too Many Requests. A 429 normally indicates an admission or rate-limit decision. The account may have exceeded RPM, TPM, or concurrency, or the provider may be protecting capacity. Retrying immediately against the same route usually repeats the condition.
Honor Retry-After when available. Reduce the sending rate, place delay-tolerant work in a bounded queue, and check whether another approved route has verified spare capacity. If the application exceeded its own allocation, provider failover may only move the same uncontrolled traffic elsewhere.
503: Service Unavailable. A 503 usually represents temporary service unavailability or overload. A bounded retry with exponential backoff and full jitter may succeed. Repeated 503 responses should reduce traffic through a circuit breaker. Failover is reasonable only when the fallback route is healthy and has capacity for the additional load.
Timeout. A timeout is a client observation, not one provider condition. It may occur while connecting, waiting for the first token, waiting between streamed tokens, or waiting for the complete response. The provider may still be processing the request after the client stops waiting.
Timeout recovery should depend on the phase and remaining user deadline. A connection timeout may be safe to retry. A timeout after generation begins may create duplicate token usage. Agent actions require idempotency so a repeated model call cannot cause an irreversible tool operation twice.
2. Put Every Recovery Attempt Inside a Budget
A useful policy can be summarized as follows:
| Signal | Default interpretation | First response | Retry or failover condition |
|---|---|---|---|
| 429 | Rate or capacity limit | Slow admission; honor Retry-After |
Route only to verified spare capacity |
| 503 | Temporary unavailability | Backoff with full jitter | Retry within budget or open circuit |
| Connect timeout | Route not established | Check deadline and route health | Limited retry may be safe |
| TTFT/read timeout | Slow or stalled request | Cancel if possible; mark route degraded | Retry only if deadline and budget remain |
The table is a starting point, not a substitute for provider-specific documentation.
Set one total attempt limit for the user operation. If a request can be tried twice, that means two attempts across all providers—not two attempts on each route. Pass the operation deadline through gateways, workers, and SDKs so a downstream component does not start a retry that cannot finish in time.
Use randomized backoff and a global retry limiter. Prioritize new high-value work over repeated low-value calls. When every route is constrained, shed or queue traffic instead of assuming that another attempt is harmless.
Record the response code, request ID, route, phase, latency, and final outcome for each attempt. Token Forge Cloud exposes structured errors and X-Request-Id, which helps connect client behavior to individual requests without treating every failure as the same event.
3. Calibrate the Policy With Normal Performance
Timeouts and capacity decisions need a healthy baseline. Token Forge Cloud observed the following results for its tested short DeepSeek workload:
| Model | Stable tested load | Success rate | TTFT p50 | TTFT p95 |
|---|---|---|---|---|
deepseek-v4.1-flash |
5,000 RPM | 100% | 1.36 s | 1.97 s |
deepseek-v4-pro |
2,500 RPM | 100% | 2.15 s | 2.82 s |
kimi-k3 |
5,000 RPM | 100% | 0.96 s | 1.26 s |
glm-5.3 |
4,200 RPM | 99.95% | 1.89 s | 5.62 s |
glm-5.3-flash |
2,400 RPM | 100% | 2.01 s | 5.29 s |
The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.
These percentiles help distinguish a normal response from a meaningful delay for that workload. They should not be copied directly into every application's timeout configuration. Long contexts and outputs require different deadlines.
A good timeout is long enough to cover expected tail latency but short enough to leave time for an alternative action. A good retry policy spends that remaining time only on errors likely to recover. A good failover policy knows the receiving route's current load before moving traffic.
The essential rule is to preserve the meaning of the signal. A 429 asks the client to reduce pressure. A 503 suggests temporary unavailability. A timeout says the client stopped waiting and may not know the final outcome. Responding to each correctly is more reliable than adding a generic retry loop.