If a load test repeats the same prompt thousands of times, the provider or application may reuse prompt-processing work. Time to first token improves, token charges may change, and the system can handle more requests than it would with unique inputs. The numbers are real, but only for a cache-heavy workload.
A credible benchmark should control cache behavior and label it explicitly.
1. Understand Where Reuse Can Enter the Request Path
Caching can occur at several layers. An application may return a previously stored complete response. A gateway may deduplicate identical requests. A provider may cache repeated prompt prefixes and reuse prefill state. Network and connection reuse can also improve later requests, although that is normally considered warm transport rather than model caching.
Complete-response caching removes most inference work. Prompt-prefix caching still requires the model to generate new output, but it can reduce input processing and TTFT. The effect grows when prompts share a long static system message, tool definition, document prefix, or conversation history.
This means that merely changing a user name or request ID may not create a cold prompt. Most of the token prefix can remain identical and eligible for reuse. A benchmark generator should vary meaningful prompt content while preserving the same length and difficulty distribution.
Cache state can also leak between test stages. A low-load warm-up may populate entries used during the high-load run. Provider comparisons become unfair if one endpoint is tested cold and another after repeated prompts.
Before testing, decide which question the benchmark should answer:
- Cold capacity: performance when prompts cannot reuse cached model work
- Warm capacity: performance at a measured production cache hit rate
- Mixed capacity: a documented combination of cached and uncached traffic
2. Design Separate Cold and Warm Experiments
For a cold AI model test, generate unique prompts or prefixes for every request. Preserve realistic token lengths and task complexity. Random strings can defeat caching but may not exercise the model like natural language, so use a versioned corpus or parameterized real-world prompts.
Randomize test order and isolate measurement windows. If the provider exposes cached-token usage, record it per request. If it does not, look for clusters in TTFT and input-token billing that may indicate reuse, but do not assume that every fast request was cached.
For a warm test, control the cache-hit ratio intentionally. For example, select a fixed percentage of requests from a repeated prefix set and send the rest with unique content. Report hit and miss results separately before showing a blended number.
Measure more than TTFT. Record end-to-end latency, generation speed, completed RPM, TPM, success rate, and billed input categories. Prompt caching primarily changes input processing; long output generation may still dominate total time.
Also test invalidation behavior when relevant. Model alias changes, prompt edits, provider routing, or cache expiry can turn a warm workload cold. An application that depends on caching needs enough uncached capacity for those transitions.
3. Publish the Cache Policy Beside the Result
Token Forge Cloud used unique prompts in its DeepSeek load test to avoid cache-hit inflation. Under the tested short-request conditions:
| Model | Stable tested load | Success rate | TTFT p95 | Token throughput |
|---|---|---|---|---|
deepseek-v4.1-flash |
5,000 RPM | 100% | 1.97 s | >1.21M TPM |
deepseek-v4-pro |
2,500 RPM | 100% | 2.82 s | >617K TPM |
kimi-k3 |
5,000 RPM | 100% | 1.26 s | ~5M TPM |
glm-5.3 |
4,200 RPM | 99.95% | 5.62 s | ~5.4M TPM |
glm-5.3-flash |
2,400 RPM | 100% | 5.29 s | ~3.0M TPM |
The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.
Because prompts were unique, these figures are intended as an uncached inference baseline for the stated workload. They do not estimate the additional benefit a cache-friendly application might receive.
Every benchmark result should state whether complete-response caching was enabled, whether prefixes repeated, how uniqueness was generated, whether cached tokens were reported, and whether warm-up traffic was excluded from measurement. If a blended cache rate is used, publish that percentage.
Caching is valuable when production requests genuinely share reusable content. A warm benchmark can therefore be highly relevant. The problem begins only when a cache-assisted result is presented as general model capacity.
The safest approach is to publish both cold and warm results. Cold performance establishes the fallback capacity when reuse disappears. Warm performance shows the optimization available to the real application. Together, they turn caching from a hidden benchmark advantage into a measurable engineering feature.