All insights

Routing and Reliability

Multi-Provider LLM Routing: Proactive Traffic Splitting or Failover After an Error?

A multi-provider router can use alternatives in two basic ways. It can split normal traffic across routes, or it can keep backups idle until the primary fails.

Neither strategy is universally correct. Proactive routing improves readiness and spreads load, while reactive failover can preserve a preferred provider's economics and output behavior. Most productionapplications benefit from a controlled hybrid.

1. Reactive Failover Is Simple but Leaves Cold Backups

In a primary-backup design, one provider receives normal traffic. The router selects another route only after an error, timeout, or circuit-breaker event.

The advantage is consistency. Most responses use the same model configuration, latency profile, billing path, and output behavior. Volume commitments or a lower primary price can also make this design economical.

The weakness is that the fallback path is rarely exercised. Credentials can expire. API compatibility can drift. A model alias can change. Capacity that appeared available during onboarding may not exist when a large incident sends traffic to the route months later.

Reactive failover also adds latency. The user waits for the primary attempt to fail before a second request begins. If the first call times out after several seconds, the fallback may no longer fit within the user's total deadline.

The largest risk is simultaneous migration. When a primary provider fails, the entire traffic stream can arrive at the backup at once. If that route was sized for occasional errors rather than full load, failover converts one outage into two.

2. Proactive Splitting Keeps Routes Warm but Adds Variability

Active-active routing sends a portion of normal traffic to more than one provider. Even a small share keeps authentication, connections, model parameters, telemetry, billing, and content validation exercised.

The router also collects current performance data. It can compare TTFT, error rate, and available capacity using real requests rather than synthetic health checks. Traffic weights can change gradually when one route degrades.

The cost is greater complexity. Providers may differ in output quality, context handling, token accounting, and model update timing. The application needs a common request contract and task-level evaluation. Users may notice behavioral differences when consecutive requests take different routes.

Active routing can also chase noise. If weights react to short-term latency changes, traffic may oscillate between providers. Use smoothed metrics, minimum observation windows, and limits on how quickly weights change.

Not every request should be freely distributed. High-value or specialized workloads may need a preferred route. Delay-tolerant batch work can absorb spare capacity, while interactive chat receives the lowest-latency healthy route.

3. Use a Hybrid With Warm Capacity and Controlled Failover

Token Forge Cloud measured stable DeepSeek capacity under fixed pacing:

Model Stable tested load Success rate TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

The same method should establish a traffic envelope for every provider route. A router cannot make a safe failover decision from health status alone; it needs to know the healthy route's remaining RPM, TPM, and concurrency.

A practical hybrid uses a primary route for most traffic and assigns a small continuous share to each approved backup. The share should be large enough to detect compatibility and performance changes. During degradation, traffic shifts in steps rather than all at once. The router stops increasing a route's weight when its tail latency, rejection rate, or capacity budget approaches the defined limit.

Keep a total retry budget across all routes, and include previous attempts in the user's deadline. If no route has enough capacity, queue delay-tolerant work and reject low-priority requests instead of overflowing the entire provider pool.

Proactive splitting is best for readiness and capacity discovery. Reactive failover is useful for containing normal traffic on a preferred route. A hybrid combines both: backups remain warm, and large traffic movements happen only when measurements show they can be absorbed.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us