All insights

Routing and Reliability

Provider Failover Is Not Free: Balancing Reliability, Latency, and Cost

Provider failover improves the chance that an LLM request eventually completes. It also adds latency, consumes additional tokens, and can change output behavior.

A production DeepSeek architecture should treat failover as a limited reliability budget rather than an unlimited sequence of alternative endpoints. The correct policy depends on the value of the request, its remaining deadline, and the verified capacity of each route.

1. Every Additional Attempt Has a Price

The most visible failover cost is time. A fallback cannot begin until the application detects a failure or degradation. If the primary waits five seconds before timing out, the second route starts with five fewer seconds in the user's deadline.

Token cost can also be duplicated. A timed-out provider may continue processing after the client disconnects. The fallback then generates another answer. Depending on billing rules and cancellation support, the application may pay for both attempts.

Agents add downstream risk. If the first attempt triggered a tool or external action before the client lost the response, repeating the workflow can perform the action twice. Application-level idempotency is required for payments, messages, database updates, and other irreversible operations.

There is also a quality cost. Two providers offering the same DeepSeek model may use different configurations, update aliases at different times, or support parameters differently. The fallback response may not behave exactly like the primary. Structured outputs and task-specific evaluations should be tested on every approved route.

Finally, standby capacity has an operational price. Keeping routes warm, monitoring them, maintaining credentials, validating model changes, and running regular load tests requires engineering work even when few requests fail over.

2. Fail Over Only When the Expected Benefit Is Positive

A useful decision compares the value of another attempt with its probability of finishing before the deadline.

Retrying an invalid request has almost no expected benefit. Retrying a transient connection failure on a healthy alternative may have high value. Starting a long DeepSeek Pro call with two seconds remaining in the user deadline is unlikely to help, even if the route is available.

Classify requests by importance and delay tolerance. High-value interactive calls may receive one fast fallback. Batch jobs can wait and retry later. Optional agent steps may be skipped during an incident. Low-priority traffic may be rejected to preserve capacity for critical work.

Set one attempt budget across all providers. Include prior latency and cost when choosing the next route. The router should know:

  • Remaining user deadline
  • Tokens already consumed or potentially in progress
  • Recent route success and tail latency
  • Available RPM, TPM, and concurrency
  • Expected price and quality of the alternative

Hedged requests—starting a second call before the first fails—can reduce tail latency, but they deliberately duplicate load. Reserve them for high-value operations and use a delay so the second call starts only when the primary is unusually slow. Cancel the losing request when possible.

3. Measure Reliability per Completed Outcome

Token Forge Cloud's DeepSeek load test established these short-request baselines:

Model Stable tested load Success rate TTFT p95 Generation speed
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s 71 token/s
deepseek-v4-pro 2,500 RPM 100% 2.82 s 41 token/s
kimi-k3 5,000 RPM 100% 1.26 s 169 token/s
glm-5.3 4,200 RPM 99.95% 5.62 s 73 token/s
glm-5.3-flash 2,400 RPM 100% 5.29 s 40 token/s

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

A failover route needs an equivalent baseline. Moving traffic to an endpoint without verified spare capacity can increase both latency and failure rate.

Evaluate the complete user operation rather than individual attempts. Useful metrics include final success rate, time to first useful response, total attempts, retry amplification, duplicated tokens, effective cost per valid result, and the percentage of fallbacks completed within the original deadline.

The cheapest failover is the one never needed, but eliminating fallback entirely accepts more failed outcomes. The goal is not zero failover cost. It is to spend that cost where it produces a meaningful reliability improvement.

For many applications, the balanced design is a warm alternative route, a small total attempt budget, gradual traffic movement, and strict deadline propagation. This provides recovery without allowing one failed DeepSeek request to become several expensive requests that all finish too late.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us