The danger comes from additional work. A retry creates another request. Failover may leave the original request running while a second provider starts generating. A circuit breaker can move a large traffic stream from one constrained route to another. Recovery logic must reduce pressure, not only search for another endpoint.
1. Retry Only Failures That May Succeed on Another Attempt
Invalid authentication, unsupported parameters, and malformed requests do not become valid when repeated. Retrying them wastes capacity and can hide an application bug.
Transient transport failures, some 503 responses, and selected timeouts may justify another attempt. Even then, the retry needs a deadline, a randomized backoff, and a maximum attempt count. Honor Retry-After when it is provided.
A 429 response requires special care. It may indicate an account limit, a provider capacity limit, or a temporary admission policy. Immediate retries against the same endpoint usually increase the overload. The client should slow its sending rate, queue eligible work, or route a controlled portion to capacity that is already known to be healthy.
Timeouts are ambiguous. The client may stop waiting while the provider continues generating. Starting another call can duplicate token usage and downstream actions. Separate connection timeout, TTFT timeout, inter-token timeout, and total request deadline so the recovery action matches the phase that failed.
Every request should carry an identifier that lets logs and billing records connect attempts belonging to the same user operation. For tool-using agents, irreversible actions also need application-level idempotency.
2. Scope Circuit Breakers Narrowly and Recover Gradually
A circuit breaker stops sending requests to a route when recent failures cross a threshold. The common mistake is using one global breaker for every model, traffic class, and error type.
If only long-context requests are timing out, opening the breaker for short chat traffic removes healthy capacity. If one credential reaches its rate limit, marking the complete provider unavailable may send too much traffic elsewhere. Breakers should be scoped to the smallest meaningful route: provider, model, region or client path, and sometimes workload class.
The breaker should also measure failures that indicate provider health. Client-side validation errors should not open it. Rate limits may need a separate capacity signal. Slow TTFT can be treated as degradation before requests begin failing completely.
Recovery is another risk. When a route becomes healthy, sending all traffic back immediately can overload it again. A half-open state should admit a small number of representative requests, then increase traffic gradually if latency and success remain within target.
Failover needs the same restraint. A backup route should have a known traffic ceiling. The router must consider its current load before moving more requests. Otherwise, a correct circuit-breaker decision on the primary can trigger a second incident on the fallback.
3. Use One Coordinated Reliability Policy
Token Forge Cloud's DeepSeek load test demonstrates why recovery rules need a capacity baseline:
| Model | Stable tested load | Success rate | TTFT p95 | Generation speed |
|---|---|---|---|---|
deepseek-v4.1-flash |
5,000 RPM | 100% | 1.97 s | 71 token/s |
deepseek-v4-pro |
2,500 RPM | 100% | 2.82 s | 41 token/s |
kimi-k3 |
5,000 RPM | 100% | 1.26 s | 169 token/s |
glm-5.3 |
4,200 RPM | 99.95% | 5.62 s | 73 token/s |
glm-5.3-flash |
2,400 RPM | 100% | 5.29 s | 40 token/s |
The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.
These are tested operating points for a defined short-request workload. A failover controller should know comparable limits for every route and avoid sending recovery traffic beyond them.
Retries, breakers, and routing should share one budget. A user operation might be allowed two total attempts, not two attempts per provider. A global concurrency limiter should count original and retry calls together. When the system is overloaded, low-priority work should wait or fail quickly rather than multiplying.
A safer sequence is:
- Classify the error and confirm that another attempt can help.
- Check the user's remaining deadline and the global retry budget.
- Select a route with recent health data and available capacity.
- Apply randomized backoff where waiting is appropriate.
- Record every attempt under one operation identifier.
Reliability mechanisms work when they coordinate demand. They make incidents worse when each component independently creates more traffic in response to the same failure.