All insights

Load Testing and Capacity Planning

Why a Short Traffic Spike Does Not Prove Stable LLM Capacity

An LLM API can absorb a large burst for several seconds and still fail under sustained traffic.

Queues, spare workers, cached prompts, connection buffers, and temporary compute headroom can make a short peak look healthy. Stable capacity asks a different question: can the system continue completing work at the offered rate while latency, errors, and backlog remain controlled?

Peak throughput and sustainable throughput should be measured separately.

1. Bursts Can Borrow Capacity From the Future

When requests arrive faster than they can be processed, a queue absorbs the difference. During a short test, most requests may be accepted before the queue becomes visibly unhealthy. The test ends, arrivals stop, and the system finishes the backlog afterward.

If the benchmark counts accepted requests rather than completed requests, it may report a high RPM even though the work was not completed at that rate. End-to-end latency and queue depth reveal the borrowed capacity.

The same effect appears through active concurrency. A service can open many streams quickly, but long generations keep them active. If new traffic continues, memory and connection use accumulate until admission control or timeouts begin.

Caching can support a temporary peak as well. Repeated prompts or prefixes may avoid full prefill work. Warm connections and preallocated workers can improve the first interval. Neither is invalid, but both must match the workload the result claims to represent.

A burst test remains useful. It shows how the service responds to sudden demand and whether it rejects, queues, or processes excess traffic. It simply does not establish the steady rate the service can maintain.

2. Stable Capacity Requires Time and a Backlog Check

At each load level, compare offered work with completed work. If requests arrive at 5,000 RPM but completions stabilize at 4,000 RPM, the remaining 1,000 RPM becomes a growing backlog or an error stream.

Watch TTFT p50 and p95 over time rather than reporting only one value for the whole run. A steadily rising TTFT is a queue signal even when the final success rate remains high. Track active requests, incomplete streams, timeouts, and 429 responses alongside it.

The test should run long enough to observe several times the normal request duration and to expose periodic system behavior. Then add a longer soak test at the intended production level. Soak tests can reveal performance drift, recurring contention, connection churn, and backlog that a short step misses.

After arrivals stop, measure drain time. A healthy stable level should not leave a large queue that requires a long recovery period. The service should also return to its baseline latency before the next step begins.

Define two numbers when useful: burst capacity for a short, named duration and sustained capacity for a longer interval with no growing backlog. This prevents one peak figure from being used for every planning decision.

3. Read Short Load Levels as a Baseline, Not a Soak Test

Token Forge Cloud tested DeepSeek in 90-second steps with fixed pacing, unique prompts, and a combined success and TTFT requirement:

Model Highest stable tested load Success rate TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

These levels establish a current stable baseline for the tested short-request workload. They identify immediate saturation more reliably than a one-time burst because requests were continuously paced and both success and tail latency were evaluated.

They should not replace a long-duration validation for a specific production workload. A team planning continuous 5,000 RPM Flash traffic should repeat the test with its prompt distribution and run a soak test at or below the intended operating level. It should leave headroom for traffic variation and longer-than-average outputs.

Capacity planning works best with several time scales: seconds for bursts, minutes for load steps, hours for soak testing, and ongoing production monitoring for long-term change. A high short-term peak is useful evidence. Stable capacity is the rate that remains healthy after the buffers and initial headroom have stopped hiding the difference between arrivals and completed work.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us