All insights

Performance Metrics

RPM, TPM, and Concurrency: Three LLM API Capacity Metrics That Are Easy to Confuse

LLM API capacity is often summarized with three numbers: requests per minute (RPM), tokens per minute (TPM), and concurrency. They are related, but they measure different constraints.

Confusing them can produce surprising results. An application may remain far below its RPM limit and still be throttled by TPM. Another may stay within both limits but create a large queue because too many long-running requests are active at once.

Understanding the difference is the first step toward realistic capacity planning.

1. RPM Measures Request Arrival Rate

Requests per minute (RPM) measures how many requests an account or system can accept during a one-minute window.

If an API allows 1,000 RPM, that is an average of about 16.7 request arrivals per second. It does not mean that each request finishes within one second, or that the service can process any possible 1,000-request workload with the same latency.

RPM is most useful when requests are relatively similar. A classification endpoint that produces five output tokens per request may be constrained mainly by RPM. The same RPM can represent a much heavier workload when every request includes a long document and generates several thousand tokens.

The measurement window also matters. Some services use a fixed one-minute window. Others use a rolling window or a token bucket that allows short bursts. Two limits labeled “1,000 RPM” can therefore behave differently during a sudden traffic spike.

Finally, an RPM limit is not a capacity benchmark. It is an admission policy for an account. Stable capacity must be measured under a defined workload, including latency and success-rate requirements.

2. TPM Measures Work, While Concurrency Measures Requests in Flight

Tokens per minute (TPM) measures the amount of token processing associated with requests over time. Depending on the service, it may include input tokens, output tokens, or both.

The basic relationship is:

TPM = RPM × average tokens per request

At 1,000 RPM, a workload averaging 200 total tokens per request requires roughly 200,000 TPM. If the average rises to 2,000 tokens, the same request rate requires 2,000,000 TPM. The RPM did not change, but the computational workload increased by a factor of ten.

This is why TPM is essential for applications with variable context or output lengths. Document processing, code generation, and agent workflows can reach a TPM limit even when their request count looks modest.

Concurrency is the number of requests that are active at the same time. It depends on both arrival rate and request duration. A useful approximation is:

concurrency ≈ requests per second × average request duration in seconds

Suppose an application sends 600 RPM, or 10 requests per second. If each request takes two seconds, average concurrency is about 20. If each request takes twenty seconds, it rises to about 200 even though RPM is unchanged.

Concurrency matters because every active request consumes resources such as connections, streaming slots, memory, and queue capacity. Long generations can therefore create a concurrency bottleneck before the application reaches its nominal RPM or TPM limit.

These metrics answer different questions:

  • RPM: How often do requests arrive?
  • TPM: How much token work is being processed?
  • Concurrency: How many requests remain active at once?

3. Plan Capacity With All Three Metrics

A useful capacity estimate starts with the real workload rather than the provider's largest published number.

Measure request arrival patterns, input and output token distributions, and request duration. Use percentiles as well as averages. A few long generations may have little effect on average RPM but a large effect on peak concurrency. Then test the expected combination under controlled load and check whether success rate and latency remain within the application's requirements.

Token Forge Cloud's DeepSeek load test used fixed pacing, unique prompts, approximately 100 input tokens, and about 140 output tokens per successful request. Reasoning was disabled, and each load level ran for 90 seconds. The results show why RPM and TPM are most useful when reported together:

Model Highest stable tested load Sustained token throughput Success rate TTFT p95
deepseek-v4.1-flash 5,000 RPM >1.21M TPM 100% 1.97 s
deepseek-v4-pro 2,500 RPM >617K TPM 100% 2.82 s
kimi-k3 5,000 RPM ~5M TPM 100% 1.26 s
glm-5.3 4,200 RPM ~5.4M TPM 99.95% 5.62 s
glm-5.3-flash 2,400 RPM ~3.0M TPM 100% 5.29 s

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

These values are a baseline for the tested workload, not a universal guarantee. Longer prompts and outputs would increase TPM at the same RPM. Slower or longer-running requests would also increase concurrency.

For production planning, leave headroom in every dimension. Traffic arrives unevenly, output length is difficult to predict, and a temporary latency increase can cause requests in flight to accumulate. Monitoring only RPM may miss both token saturation and queue growth.

The practical rule is straightforward: use RPM to describe traffic, TPM to describe token volume, and concurrency to describe simultaneous pressure. Stable LLM capacity exists only where all three remain manageable.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us