Imagine that nine requests finish in two seconds and one takes twenty seconds. The average is 3.8 seconds. That number describes none of the actual requests particularly well. Most users experienced a fast response, while one user waited ten times longer. A single average compressed both experiences into a value that nobody saw.
For LLM APIs, this problem is especially important. Request times naturally vary with prompt length, output length, model behavior, provider load, and network conditions. Percentiles make that variation visible.
1. What p50, p95, and p99 Actually Mean
A percentile answers a simple question: what latency did a given percentage of requests finish at or below?
- p50, or the median, is the middle request. Half of requests were faster and half were slower.
- p95 is the threshold that 95% of requests met. The slowest 5% took longer.
- p99 is the threshold that 99% of requests met. It focuses on the tail that averages usually hide.
These numbers describe different user experiences.
p50 represents the typical path. It is useful when optimizing the experience most users see most of the time. p95 is often more relevant for product reliability because a slowest-five-percent event is not rare at scale. At 100,000 daily requests, 5% means 5,000 requests.
p99 examines more extreme tail behavior. It can expose queueing, retries, noisy infrastructure, or occasional upstream stalls. But it also needs enough data. With only 100 requests, p99 is effectively determined by one observation. A larger sample and repeated test windows make it more meaningful.
Percentiles are not percentages of the average. If TTFT p95 is 2 seconds, it does not mean that requests are “95% fast.” It means 95% received their first token within 2 seconds under the measured workload.
2. Read the Metric Boundary Before Reading the Number
“Latency” is not one universal measurement. Before comparing percentiles, check what the timer includes.
For a streaming LLM API, useful boundaries include:
- Time to first token (TTFT): from sending the request until the first generated token arrives.
- Generation time: from the first token until the response completes.
- End-to-end latency: from sending the request until the complete response arrives.
A provider can have a good TTFT p50 but a weak TTFT p95. That usually feels fast in a demo and unpredictable in production. Another provider may start consistently but generate long answers slowly. Both can report a single “latency” number, even though the user experience is different.
Workload context matters just as much. Prompt length, output length, streaming mode, reasoning settings, concurrency, region, and cache behavior can all change the distribution. A p95 from short chat prompts should not be compared directly with a p95 from long document analysis.
This is also why percentiles should be inspected at each load level. A healthy system often has similar p50 and p95 values at moderate traffic. As it approaches saturation, p95 may rise sharply while p50 changes only slightly. The widening gap is an early sign of queueing.
3. Use Percentiles to Find Predictability, Not Just Speed
Token Forge Cloud tested its DeepSeek endpoints with fixed pacing, unique prompts, streaming responses, approximately 100 input tokens, and about 140 output tokens per successful request. Each load level ran for 90 seconds. A level was considered stable when success rate remained at least 99% and TTFT p95 remained at or below 5 seconds.
At the highest stable level observed for each model, the results were:
| Model | Stable tested load | Success rate | TTFT p50 | TTFT p95 | Generation speed |
|---|---|---|---|---|---|
deepseek-v4.1-flash |
5,000 RPM | 100% | 1.36 s | 1.97 s | 71 token/s |
deepseek-v4-pro |
2,500 RPM | 100% | 2.15 s | 2.82 s | 41 token/s |
kimi-k3 |
5,000 RPM | 100% | 0.96 s | 1.26 s | 169 token/s |
glm-5.3 |
4,200 RPM | 99.95% | 1.89 s | 5.62 s | 73 token/s |
glm-5.3-flash |
2,400 RPM | 100% | 2.01 s | 5.29 s | 40 token/s |
The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.
The useful signal is not only the median. The p95 values stayed within the test's stability threshold while all requests succeeded. That combination says more about the tested user experience than an average TTFT would.
These results are a capacity baseline for this specific workload, not a universal latency promise. A production team should repeat the same analysis with its own prompt and output distributions. It should also monitor p50 for the common experience, p95 for routine reliability, and p99 for rare but costly slowdowns.
The practical rule is simple: use averages for rough summaries, but use percentiles to make engineering decisions. A fast median is pleasant. A controlled tail is what makes performance dependable.