All insights

Load Testing and Capacity Planning

How to Design a Credible LLM API Load Test

A credible LLM API load test does more than send many requests. It applies a defined workload at a controlled rate, measures the complete response path, and states the conditions under which the result is valid.

Without those controls, a benchmark can produce an impressive RPM number that says little about production capacity. Prompt length, output length, cache behavior, streaming mode, traffic pacing, and success criteria all change the result.

DeepSeek is a useful example because the same endpoint may serve short chat, long documents, and multi-step agents. Each workload needs a different capacity test.

1. Define the Workload Before Choosing the Load

Start with the application rather than the provider's published limit. Build a versioned prompt set that represents real traffic. Separate workload classes such as chat, extraction, code generation, and long-context analysis instead of averaging them into one synthetic request.

Record the distributions of input tokens, requested output limits, and actual output tokens. Fix streaming mode, reasoning settings, sampling parameters, tool definitions, and structured-output configuration. Use the same model alias and client region throughout a comparison.

Decide whether the test represents cold inference or a cache-heavy production workload. For a cold test, use unique prompts and verify that repeated prefixes do not trigger caching. For a warm test, publish the expected cache hit rate and report cached and uncached requests separately.

Define the traffic model too. Real-time applications may have short bursts and daily peaks. Batch systems may produce a sustained queue. Agents create several correlated calls per user action. The test should reproduce arrival patterns as well as average RPM.

Finally, make the client capable of generating the intended load. Insufficient connections, CPU, network bandwidth, or event-loop capacity can make the load generator the bottleneck before the API is tested.

2. Apply Controlled Load and Measure the Full Response

Use open-loop, fixed pacing when the goal is to measure stable provider capacity. The generator should send requests at the target rate without waiting for earlier requests to complete. In a closed-loop test, slow responses automatically reduce new traffic and can hide saturation.

Increase load in steps—for example, 500, 1,000, 2,500, and 5,000 RPM. Hold each level for a defined interval, allow the system to reach a steady state, and include a longer soak test for the expected production level.

Collect per-request measurements:

  • Dispatch time, first-token time, and completion time
  • HTTP status and stream completion state
  • Input and output tokens
  • 429, 5xx, timeout, and transport errors
  • Active concurrency
  • Provider request ID and client operation ID
  • Billed cost when available

Define success as a complete usable response, not merely an HTTP 200 status. Set the acceptance rule before viewing the data. A test might require at least 99% completed requests and TTFT p95 below a workload-specific threshold.

Report p50, p95, and, when the sample is large enough, p99. Averages are useful summaries but can hide queueing in the tail. Report offered RPM as well as completed RPM so rejected or unfinished work does not disappear.

3. Publish a Result That Can Be Reproduced

Token Forge Cloud's DeepSeek test used one organization and API key, unique prompts, approximately 100 input tokens, about 140 output tokens, fixed pacing, and 90-second load levels. Stability required at least 99% success and TTFT p95 no higher than five seconds.

Model Highest stable tested load Success rate TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

The table is meaningful because the workload and acceptance rule are attached to the numbers. It is a capacity baseline for that short-request test, not a result for every context length or application.

A reproducible report should include the test date, client location, prompt-set description, request parameters, traffic schedule, duration, cache policy, success definition, percentile method, sample size, and error categories. Raw per-request data or calculation code makes review stronger when it can be shared safely.

Run the same test after model-alias changes, routing changes, major application releases, and meaningful traffic growth. Capacity is not a permanent property of a model name. It is an observation of a complete system under defined conditions.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us