All insights

Load Testing and Capacity Planning

When Is the Bottleneck Not the Model, but the Gateway, Database, or Billing System?

An LLM API request passes through more than a model server.

Before inference begins, a gateway may terminate TLS, authenticate the key, enforce limits, reserve funds, and write request metadata. During and after generation, the system streams tokens, records usage, calculates cost, and updates logs. Any stage can limit throughput or increase latency even when the model has spare compute.

A useful load test measures the end-to-end path and then isolates each stage.

1. Map the Request Into Observable Phases

A simplified AI model request path can include:

  1. Client connection, DNS, and TLS
  2. Gateway admission and authentication
  3. Rate-limit or quota evaluation
  4. Request and billing metadata operations
  5. Upstream dispatch and queueing
  6. Model prefill and token generation
  7. Streaming through proxies to the client
  8. Usage finalization, cost calculation, and logging

Time to first token combines most stages before generation becomes visible. End-to-end latency includes the stream and finalization visible to the client. A single latency number cannot identify which stage is slow.

Add timestamps around boundaries the application controls: request arrival, completion of authentication, upstream dispatch, first upstream byte, first client byte, final upstream byte, and request finalization. Preserve a request ID across services so one slow call can be reconstructed.

Resource metrics should match each phase. Gateway CPU, open connections, event-loop delay, database latency, connection-pool saturation, queue depth, and log-pipeline backpressure can explain behavior that model token speed cannot.

2. Use Symptoms and Isolation Tests to Locate the Limit

Different bottlenecks leave different patterns.

If TTFT increases while upstream dispatch is delayed, the problem is before the model. If dispatch remains fast but first upstream token slows, provider queueing or model prefill is more likely. If upstream tokens arrive normally but the client receives them irregularly, inspect gateways, proxies, buffering, and network paths.

A gateway bottleneck often affects short and long requests similarly at admission. A model bottleneck usually varies more strongly with context, output length, and active generation. A metadata or billing bottleneck may appear as rising pre-dispatch latency, transaction contention, or slow request finalization.

Run isolation tests:

  • Replace the upstream model with a controlled streaming stub to test the gateway path.
  • Bypass optional synchronous metadata work in a non-production test environment.
  • Compare streaming and non-streaming requests.
  • Increase prompt length while keeping RPM constant, then increase RPM while keeping tokens constant.
  • Separate admission latency from upstream TTFT.

Change one variable at a time. If a fast stub cannot reach the target RPM, the model is not the current bottleneck. If the gateway remains healthy while longer contexts sharply increase upstream TTFT, attention should move toward inference capacity.

3. Plan Capacity for the Slowest Required Stage

Token Forge Cloud's DeepSeek benchmark measured the complete customer-facing path under fixed pacing:

Model Stable tested load Success rate TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

The end-to-end result proves that every required stage completed the tested short-request workload together. It does not identify the internal utilization of any one component, and it should not be interpreted as evidence of a particular internal bottleneck.

For production diagnosis, Token Forge Cloud exposes request-level model, status, token usage, latency, final cost, and request ID. Application teams can combine these fields with their own gateway and client timestamps to distinguish API behavior from local queueing or network delay.

Capacity must be provisioned across the complete chain. A model route capable of one million TPM is not useful if admission can process only a fraction of the required RPM. A fast gateway is not enough if synchronous logging stalls completion. Each mandatory stage needs headroom, bounded queues, timeouts, and degradation behavior.

The right question is not simply, “Is the model slow?” It is, “At which boundary does work begin to accumulate?” Once that boundary is measured, the capacity fix becomes specific instead of speculative.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us