All insights

Load Testing and Capacity Planning

Eight Common Mistakes When Load Testing an Open-Source Model API

An LLM load test can generate thousands of requests and still answer the wrong question.

The mistakes fall into three groups: the test runs an easier workload than production, the measurements hide saturation or failure, or the result is too narrow to support a capacity decision. They apply to any open-source or open-weight model API. So keep an eye on these questions before carrying out your LLM load test.

1. Mistakes That Change the Workload

1. Comparing different model configurations. Two endpoints with the same model family may use different aliases, reasoning settings, output limits, quantization, or supported parameters. Record the exact request and verify that both routes return comparable behavior.

2. Using one tiny prompt for every request. A repeated short prompt does not represent chat history, document analysis, or agent traffic. Build prompt groups with realistic input and output distributions.

3. Allowing cache hits without labeling them. Repeated prompts and shared prefixes can reduce prefill work. Use unique content for a cold-inference test, or measure warm and cold traffic separately.

These errors make the test easier than production. The reported performance may be real for the synthetic request but irrelevant to the application.

2. Mistakes That Hide Saturation or Failure

4. Using only closed-loop traffic. If each worker waits for a response before sending another request, slow latency reduces offered load. Use fixed pacing when measuring the capacity of a target RPM.

5. Reporting only averages. Average latency hides slow-tail behavior. Report TTFT and total latency at p50 and p95, plus p99 when the sample is large enough.

6. Counting HTTP 200 as success. A stream may start and then terminate, return malformed structured output, or finish after the product deadline. Separate transport success, complete response, and task validity.

These mistakes make an overloaded system look healthy. The client silently sends less traffic, the average hides queueing, or incomplete work is counted as successful.

3. Mistakes That Make the Result Impossible to Use

7. Testing only one short peak. A burst can be absorbed by queues and temporary headroom. Increase load in steps, compare offered and completed work, inspect backlog, and add a soak test.

8. Failing to verify the load generator. Client CPU, connections, network throughput, logging, or event-loop limits can cap the test before the provider is stressed. Monitor the generator and, when needed, distribute load across several machines.

Token Forge Cloud's DeepSeek test avoided several of these errors by using unique prompts, fixed pacing, explicit token lengths, multiple load levels, and a predefined stability rule:

Model Stable tested load Success rate TTFT p50 TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.36 s 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.15 s 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 0.96 s 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 1.89 s 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 2.01 s 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

The results remain bounded by their conditions: approximately 100 input tokens, about 140 output tokens, reasoning disabled, and 90 seconds per load level. Publishing those limits is part of a credible benchmark.

Before accepting any open-source model performance claim, ask eight corresponding questions:

  • Was the configuration identical?
  • Were prompts representative?
  • Was caching controlled?
  • Was offered load independent of latency?
  • Were percentiles reported?
  • Did success require a complete valid response?
  • Was the test sustained?
  • Could the client generate the target load?

If the answer to any of these is unknown, the result may still be interesting. It is not yet enough for production capacity planning.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us