All insights

Performance Metrics

From 500 RPM to 5,000 RPM: How to Find an LLM API's Stable Capacity

A provider may publish a rate limit of 5,000 requests per minute. That does not automatically mean the API can complete 5,000 requests per minute with acceptable latency.

A rate limit is an admission rule. Stable capacity is an observed operating range. The first tells you how much traffic an account is allowed to send. The second tells you how much traffic the complete system can process before success rate or latency begins to degrade.

The difference matters whenever an application moves from a demo to sustained production traffic.

1. Increase Load in Steps Instead of Testing One Peak

A useful capacity test is a load curve, not a single large burst.

Start below the expected limit, then increase traffic in controlled steps—for example, 500, 1,000, 2,500, and 5,000 RPM. Hold each level long enough to collect a meaningful sample. Keep the workload consistent so that a change in performance is more likely to come from load rather than from different prompts.

The request generator should also use fixed pacing. If it sends a new request only after the previous request finishes, slow responses reduce the offered load. That creates a feedback loop: the system looks stable partly because the client stopped applying pressure. An open-loop test continues sending requests at the intended rate and makes queueing easier to see.

For LLM APIs, control at least these variables:

  • Input and output length distributions
  • Streaming or non-streaming mode
  • Reasoning and tool-use settings
  • Cache behavior
  • Number of API keys and organizations
  • Test duration and traffic pacing

Unique prompts are useful when the goal is to measure uncached inference. Repeated prompts may produce a valid test for a cache-heavy application, but the result should be labeled accordingly.

2. Find the Point Where the System Stops Being Predictable

Stable capacity is not simply the highest level at which some requests succeed. It is the highest level that satisfies a predefined performance rule.

A practical rule might combine:

  • A minimum success rate
  • A maximum TTFT p95
  • A maximum end-to-end latency p95
  • A minimum sustained output-token rate
  • Limits on 429, 5xx, and timeout responses

Why combine them? Because each metric can hide failure in another dimension. A system may return 100% successful responses but make users wait in a long queue. Another system may preserve low latency by rejecting a large share of traffic immediately. Neither is stable for an application that needs both availability and responsiveness.

Watch how the distribution changes as load rises. A gradual increase in median latency is usually manageable. A rapidly increasing p95, growing timeout count, or widening gap between offered and completed requests indicates saturation. This transition is sometimes called the “knee” of the load curve.

Do not treat the first bad level as the capacity number. The useful baseline is usually the last level that met every criterion. Then leave operational headroom below it. Traffic is rarely perfectly uniform, and real prompts may be heavier than benchmark prompts.

3. Interpret a Capacity Result Within Its Test Conditions

Token Forge Cloud tested DeepSeek using one organization and one API key. Requests contained approximately 100 input tokens and produced about 140 output tokens on average, with reasoning disabled. Each load level ran for 90 seconds, and requests were sent at fixed intervals without waiting for earlier requests to complete.

The stability rule was at least 99% success and TTFT p95 no higher than 5 seconds. Under those conditions, the highest stable levels were:

Model Highest stable tested load Sustained token throughput Success rate TTFT p95
deepseek-v4.1-flash 5,000 RPM >1.21M TPM 100% 1.97 s
deepseek-v4-pro 2,500 RPM >617K TPM 100% 2.82 s
kimi-k3 5,000 RPM ~5M TPM 100% 1.26 s
glm-5.3 4,200 RPM ~5.4M TPM 99.95% 5.62 s
glm-5.3-flash 2,400 RPM ~3.0M TPM 100% 5.29 s

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

RPM shows request volume, while TPM shows the amount of token work completed. Reporting both prevents a short-output workload from looking equivalent to a long-generation workload.

The table should be read as a tested baseline, not as a universal guarantee. Longer prompts, larger outputs, a different concurrency pattern, or tool calls can move the capacity boundary. A 90-second load level is useful for identifying immediate saturation, but it does not replace a long-duration soak test for memory growth, periodic contention, or daily traffic cycles.

The most useful capacity result is therefore reproducible and qualified: a specific model, workload, measurement boundary, duration, and acceptance rule. “5,000 RPM” alone is a limit. “5,000 RPM at 100% success with TTFT p95 below two seconds for the tested workload” is an engineering result.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us