Requests per minute (RPM) limits how many API calls are admitted during a provider-defined time window, tokens per minute (TPM) limits the volume of tokens processed during that window, and concurrent requests limits how many calls can be in flight at the same time. An AI workload must remain within every applicable limit; whichever threshold it reaches first becomes the immediate capacity bottleneck.
The Short Answer: Three Limits Measure Different Parts of API Capacity
RPM, TPM, and concurrency are related, but they are not interchangeable. RPM and TPM are generally throughput limits measured over time. Concurrency is an in-flight capacity limit determined by how many requests remain active simultaneously.
| Limit | What it measures | Common bottleneck pattern | Relevant response |
|---|---|---|---|
| Requests per minute (RPM) | Number of API calls admitted within a provider-defined time window | Many small calls, polling, burst traffic, or retry storms | Pace requests, add queues, consolidate eligible work, and apply retry backoff |
| Tokens per minute (TPM) | Token volume processed or reserved under the provider’s accounting rules | Large prompts, long completions, context-heavy agents, or high-volume generation | Control context and output size, route work appropriately, and evaluate caching where applicable |
| Concurrent requests | Number of requests currently in flight | Slow responses, streaming sessions, tool-using agents, or long-running generation | Manage request duration, cancellations, backpressure, and admission of new work |
An RPM limit usually counts calls independently of request size unless the provider says otherwise. A short classification request and a long generation request may each count as one request, even though their token use differs substantially.
TPM instead concerns token volume. Depending on the provider, accounting may include input tokens, generated output, estimated output, reserved output capacity, or a combined value. Tokens are not equivalent to words or characters, so teams should use the relevant model tokenizer and current provider rules when estimating demand.
Concurrency counts active requests rather than requests started during a minute. A request may occupy an in-flight slot while waiting for generation, streaming output, invoking tools, or completing other processing, depending on how the service defines request state.
Exact quotas, enforcement windows, token-accounting methods, rejected-request treatment, and quota scopes vary. Always verify these details in the provider’s current official documentation.
How Throughput Limits Differ from In-Flight Capacity
Throughput limits answer questions such as “How many calls can start during this window?” and “How much token volume can be processed during it?” Concurrency answers a different question: “How much work can remain active at once?”
A useful planning approximation is:
estimated in-flight requests ≈ requests per second × average request duration in seconds
This relationship helps explain why concurrency is not another name for requests per second or RPM. Two applications can submit requests at the same average rate but consume different levels of concurrency if one has longer response times.
For example, a short classification call may release its slot quickly. A streaming assistant can hold a slot while tokens are delivered, and an agentic workflow may remain active across model generation and tool execution. Longer duration increases in-flight pressure without necessarily increasing RPM.
The approximation is only a starting point. Production planning should use request-duration distributions rather than averages alone because tail latency can create more overlap than the average suggests. Provider definitions also matter: queued, streaming, cancelled, and retried requests may not be handled identically across services.
Which Limit Binds for Small, Token-Heavy, Bursty, and Long-Running Requests
The binding limit depends on request shape, traffic timing, and service behavior. Common tendencies include:
- Many small requests: A classification, extraction, or moderation workload with short prompts and outputs may reach RPM before TPM because each small operation still consumes a request.
- A few token-heavy requests: Document analysis, long-context generation, and large batch prompts may reach TPM even when request count is modest.
- Bursty traffic: A workload can exceed a provider’s enforcement threshold during a short spike even if its hourly average appears manageable. Queues and traffic pacing are therefore relevant to capacity design.
- Long-running or streaming requests: Calls that remain open longer can consume concurrent capacity while staying below RPM and TPM limits.
- Agentic workflows: One user action can trigger multiple model calls, retries, and tool-driven steps. The external user-session count may therefore understate request and token demand.
These are planning tendencies, not fixed rules. A token-heavy request can also be long-running, while a burst of small requests can pressure both RPM and concurrency. The application must stay within all applicable limits at the same time.
Token Forge Cloud treats latency-sensitive chat, batch enrichment, and agentic workflows as different serving-policy problems. That distinction matters because each workload creates a different combination of arrival rate, token volume, duration, and cost exposure.
Hypothetical Example: Estimating the First Capacity Constraint
The following values are fictional, provider-neutral planning inputs—not Token Forge Cloud limits or measured customer results.
Assume an API has limits of 120 RPM, 120,000 TPM, and 8 concurrent requests. A workload sends 90 requests per minute, uses an average of 1,000 combined tokens per request under the assumed accounting method, and takes four seconds per request on average.
The rough comparison is:
- Request demand: 90 RPM, which is below the assumed RPM threshold.
- Token demand:
90 × 1,000 = 90,000 tokens per minute, which is below the assumed TPM threshold. - In-flight demand:
90 ÷ 60 × 4 = 6 active requests on average, which is below the assumed concurrency threshold but leaves relatively limited headroom.
Under these averages, concurrency is the closest threshold. If average duration rises while arrival rate and request size remain unchanged, concurrency demand rises even though estimated RPM and TPM do not. If completion length doubles instead, token pressure may become the first constraint. A short traffic burst could also exceed RPM or concurrency despite acceptable minute-level averages.
This estimate should not be treated as guaranteed capacity. Real deployments need headroom for variable completion lengths, long-tail latency, retries, streaming behavior, and correlated bursts. Providers may also calculate TPM using reserved or estimated output rather than final observed output.
What to Measure Before Setting Production Capacity
Production planning works best with distributions and peaks, not a single average. Measure the workload at the model-call level and separate results by model, route, endpoint, and workload class where practical.
Key inputs include:
- Request arrival rate: Track typical, peak, and burst arrival patterns. User sessions are not a substitute for model-call counts when one session can create multiple calls.
- Input and output token distributions: Record representative percentiles, not only averages. Long prompts and unusually verbose completions can dominate TPM demand and cost.
- Request duration: Measure end-to-end duration and time spent in relevant processing stages. For streaming, distinguish time to first output from total time until the request releases in-flight capacity.
- Burstiness and queue depth: Observe how quickly demand arrives and how long queued work can wait before it loses business value.
- Retries and failures: Retries can add both requests and token consumption. Use bounded retries, backoff, and idempotency controls where appropriate. Whether rejected attempts count against a limit depends on provider policy.
- Batching behavior: Batching can change request count, token shape, latency, and resource utilization. Its effects depend on the API and workload rather than being uniformly beneficial.
- Quota scope: Confirm whether limits apply by account, project, model, region, endpoint, deployment, or another provider-defined unit.
- Cost signals: Connect token volume, model selection, cache behavior, retries, and unused reserved capacity to the financial view of the workload.
Operational dashboards should make the binding constraint visible. Useful signals include quota utilization, throttled calls, token distributions, active requests, queue delay, request duration, retry volume, completion rates, and cost by workload. Teams should also retain enough history to distinguish sustained growth from isolated bursts.
Token Forge Cloud Managed Model APIs provides model access and usage data for teams validating demand before considering private deployment. That usage history can support workload characterization, while exact capacity decisions should still reflect provider documentation and application-level measurements.
Architecture Responses for Each Type of Bottleneck
The appropriate response depends on what is actually constrained. Increasing concurrency does not resolve TPM pressure, and shortening prompts does not necessarily solve an RPM burst.
When RPM is the bottleneck
Use client-side pacing, admission control, and queues to smooth bursts. Exponential backoff with jitter can reduce synchronized retry storms. Where the API and use case permit it, consolidating eligible operations or batching work may reduce call count—but teams should evaluate the resulting latency and token shape rather than assume batching always lowers quota consumption.
When TPM is the bottleneck
Examine prompt construction, retrieved context, conversation history, and completion controls. Routing requests to models suited to their complexity may help align token demand and economics. Caching can be relevant when work is safely reusable, but whether cached content changes billed or quota-counted tokens depends on the serving system and provider rules.
When concurrency is the bottleneck
Control how much new work is admitted, cancel abandoned requests, set appropriate timeouts, and apply backpressure before downstream systems become saturated. Separate interactive and asynchronous queues when they have different latency requirements. For streaming and agentic workflows, monitor total request lifetime rather than only time to first token.
When operating private inference
Self-deployed serving shifts more capacity responsibility to the operator. Routing, batching, quantization, and GPU scheduling can be relevant to serving-layer efficiency and control, but their effects depend on model behavior, hardware, latency objectives, workload mix, and implementation choices. Private inference does not remove the need for admission control, observability, or cost measurement.
How Token Forge Cloud Fits API Access and Private Inference Planning
Token Forge Cloud supports two practical stages of enterprise inference planning. Token Forge Cloud Managed Model APIs offers an API-first path for teams that want model access, usage data, and a way to validate demand before private deployment. This can help teams characterize request shapes and identify whether RPM, TPM, duration, or cost is driving the next architecture decision.
For workloads that become predictable enough to evaluate greater serving-layer control, teams can consider Token Forge Cloud Private LLM Inference for private deployment and serving-layer optimization. Relevant controls can include caching, routing, batching, quantization, and GPU scheduling. Their value should be assessed against the workload’s model mix, latency targets, token distributions, traffic variability, and operating constraints.
Private routing, policy-aware access, and telemetry under enterprise control may also be important when organizations are evaluating AI sovereignty and operational governance. Managed API access and private inference are not automatically substitutes: some organizations may use managed access for validation or variable demand while evaluating private serving for stable, sensitive, or operationally distinct workloads.
Neither deployment model eliminates the need to understand capacity constraints. Third-party quotas still apply when traffic uses third-party endpoints, while private deployments require teams to define and operate their own safeguards around throughput, concurrency, resource scheduling, and cost.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.