Neither strategy is universally correct. Proactive routing improves readiness and spreads load, while reactive failover can preserve a preferred provider's economics and output behavior. Most productionapplications benefit from a controlled hybrid.
1. Reactive Failover Is Simple but Leaves Cold Backups
In a primary-backup design, one provider receives normal traffic. The router selects another route only after an error, timeout, or circuit-breaker event.
The advantage is consistency. Most responses use the same model configuration, latency profile, billing path, and output behavior. Volume commitments or a lower primary price can also make this design economical.
The weakness is that the fallback path is rarely exercised. Credentials can expire. API compatibility can drift. A model alias can change. Capacity that appeared available during onboarding may not exist when a large incident sends traffic to the route months later.
Reactive failover also adds latency. The user waits for the primary attempt to fail before a second request begins. If the first call times out after several seconds, the fallback may no longer fit within the user's total deadline.
The largest risk is simultaneous migration. When a primary provider fails, the entire traffic stream can arrive at the backup at once. If that route was sized for occasional errors rather than full load, failover converts one outage into two.
2. Proactive Splitting Keeps Routes Warm but Adds Variability
Active-active routing sends a portion of normal traffic to more than one provider. Even a small share keeps authentication, connections, model parameters, telemetry, billing, and content validation exercised.
The router also collects current performance data. It can compare TTFT, error rate, and available capacity using real requests rather than synthetic health checks. Traffic weights can change gradually when one route degrades.
The cost is greater complexity. Providers may differ in output quality, context handling, token accounting, and model update timing. The application needs a common request contract and task-level evaluation. Users may notice behavioral differences when consecutive requests take different routes.
Active routing can also chase noise. If weights react to short-term latency changes, traffic may oscillate between providers. Use smoothed metrics, minimum observation windows, and limits on how quickly weights change.
Not every request should be freely distributed. High-value or specialized workloads may need a preferred route. Delay-tolerant batch work can absorb spare capacity, while interactive chat receives the lowest-latency healthy route.
3. Use a Hybrid With Warm Capacity and Controlled Failover
Token Forge Cloud measured stable DeepSeek capacity under fixed pacing:
| Model | Stable tested load | Success rate | TTFT p95 | Token throughput |
|---|---|---|---|---|
deepseek-v4.1-flash |
5,000 RPM | 100% | 1.97 s | >1.21M TPM |
deepseek-v4-pro |
2,500 RPM | 100% | 2.82 s | >617K TPM |
kimi-k3 |
5,000 RPM | 100% | 1.26 s | ~5M TPM |
glm-5.3 |
4,200 RPM | 99.95% | 5.62 s | ~5.4M TPM |
glm-5.3-flash |
2,400 RPM | 100% | 5.29 s | ~3.0M TPM |
The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.
The same method should establish a traffic envelope for every provider route. A router cannot make a safe failover decision from health status alone; it needs to know the healthy route's remaining RPM, TPM, and concurrency.
A practical hybrid uses a primary route for most traffic and assigns a small continuous share to each approved backup. The share should be large enough to detect compatibility and performance changes. During degradation, traffic shifts in steps rather than all at once. The router stops increasing a route's weight when its tail latency, rejection rate, or capacity budget approaches the defined limit.
Keep a total retry budget across all routes, and include previous attempts in the user's deadline. If no route has enough capacity, queue delay-tolerant work and reject low-priority requests instead of overflowing the entire provider pool.
Proactive splitting is best for readiness and capacity discovery. Reactive failover is useful for containing normal traffic on a preferred route. A hybrid combines both: backups remain warm, and large traffic movements happen only when measurements show they can be absorbed.