All insights

Routing and Reliability

How to Prevent a Retry Storm From Overloading an LLM Application

A retry is intended to recover one failed request. A retry storm turns a temporary problem into sustained overload.

When many API calls slow down or fail together, thousands of clients may repeat them at nearly the same time. Original requests remain active, retries consume additional capacity, queues grow, and more requests cross their timeouts. Without a coordinated budget, the application creates its own traffic spike.

1. Understand How Retries Multiply Load

Suppose an application normally sends 2,000 requests per minute. A provider slowdown causes 20% of requests to time out. If every timeout is retried once, the application adds roughly 400 RPM. If some retries also time out and the client permits three attempts, offered traffic can grow again while the provider has less capacity than before.

Agent workflows amplify the effect. One user operation may create several model calls. Retrying the complete workflow after one step fails can repeat steps that already succeeded. Parallel agent branches can each apply their own retry logic, producing more attempts than the top-level service expects.

Fixed retry delays make synchronization worse. If every client waits one second, they return together. Exponential backoff spreads attempts across larger intervals, but identical schedules can still align. Randomized jitter is needed to distribute recovery traffic.

Timeout selection also affects the storm. A deadline shorter than normal TTFT creates unnecessary retries. A very long timeout leaves calls active while fallback attempts begin. Use observed latency percentiles to set phase-specific timeouts and cancel work when the provider supports cancellation.

2. Control Retries as a Shared Capacity Budget

Retries should be limited at the application boundary, not independently in every library and service.

Assign one attempt budget to the complete user operation. If the gateway retries, an internal worker and SDK should not each add their own hidden attempts. Count provider failover as another attempt within the same budget.

Use a retry token bucket or similar global limiter. During normal operation, successful requests replenish the budget. During an incident, retries consume it faster than it refills, automatically limiting amplification. Original high-priority traffic should usually have priority over retries.

Before retrying, classify the failure:

  • Do not retry invalid requests or authentication failures without changing the request.
  • Slow down after 429 responses and honor Retry-After.
  • Retry selected 503 and transport failures with bounded backoff and jitter.
  • Treat timeouts as ambiguous and check the remaining user deadline.

Queue only work that remains valuable later. Interactive requests may be better served by a fast error than by a response that arrives after the user has left. Batch DeepSeek jobs can wait behind a concurrency limiter.

Circuit breakers should reduce calls to a degraded route, while load shedding protects the remaining routes. If every backup is near its tested capacity, the safe response is not another retry.

3. Set Retry Policy From Measured Capacity

Token Forge Cloud's fixed-pacing DeepSeek test established the following short-request baselines:

Model Stable tested load Success rate TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

A recovery plan should leave headroom below a tested operating point. If normal traffic already consumes nearly all stable capacity, even a small retry percentage can push the route into queueing or rejection.

Monitor retry amplification as a first-class metric:

retry amplification = total provider attempts ÷ original user operations

A value of 1.0 means no additional attempts. Rising values during a latency event show that recovery logic is adding load. Break the metric down by response code, model, route, and workload class.

Also record the final outcome. A retry that succeeds after the user's deadline consumed capacity without restoring the product experience. Cost per successful response should include every attempt.

The purpose of retry logic is not to make every request continue indefinitely. It is to spend a small, controlled amount of spare capacity on failures likely to recover. Backoff, jitter, deadlines, shared budgets, and load shedding keep that spare capacity from becoming the incident.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us