All insights

Performance Metrics

Why 100 Tokens per Second May Feel Slower Than 50

Tokens per second is one of the most visible LLM performance numbers. It is also one of the easiest to overinterpret.

A model generating 100 tokens per second is objectively producing tokens faster than one generating 50—if both numbers use the same definition. But the faster generator does not necessarily deliver the faster user experience. The request may spend several seconds waiting before token generation begins, and many applications need only a short answer.

1. Users Experience Waiting Time, Not an Isolated Throughput Number

Consider two APIs generating a 100-token answer.

  • API A has a 3-second time to first token and generates at 100 token/s.
  • API B has a 0.5-second time to first token and generates at 50 token/s.

Using a simplified estimate, API A completes in about four seconds: three seconds of TTFT plus one second of generation. API B completes in about 2.5 seconds: half a second of TTFT plus two seconds of generation.

API B has half the token speed, but it begins and finishes earlier. In an interactive chat, it will usually feel faster as well.

Output length can reverse the result. For a 500-token answer, API A takes roughly eight seconds, while API B takes about 10.5 seconds. The fixed TTFT becomes a smaller share of total time, and the higher generation rate eventually wins.

This creates a crossover point. Short answers favor a fast start. Long answers give generation throughput more time to matter. There is no meaningful “fastest provider” without a workload length.

Perceived speed also depends on presentation. A smoothly streamed response gives the user something to read while generation continues. A non-streaming client hides all output until completion, turning TTFT and generation time into one wait. The same backend can therefore feel different in two interfaces.

2. Token-Speed Numbers May Use Different Definitions

Before comparing 100 token/s with 50 token/s, check what time interval each number includes.

A post-first-token generation rate divides output tokens by the time between the first token and the end of the response. It isolates decoding speed.

An end-to-end token rate divides output tokens by the entire time from request dispatch to completion. It includes TTFT and will be lower for the same request.

Some measurements also differ in how they count tokens, whether cached requests are included, which output lengths qualify, and whether failed or rate-limited requests enter the distribution. Model configuration matters too. Reasoning, tools, sampling settings, and quantization can all affect performance.

This is why a leaderboard number is best treated as an observation of a measurement system, not a universal property of the model. For an apples-to-apples comparison, use the same prompt set, output limits, streaming behavior, traffic level, region, and timing boundary.

Percentiles matter as well. A median of 100 token/s says little about the slowest 5% of requests. A provider with a slightly lower median but a tighter p95 may produce a more predictable product experience.

3. Choose Performance for the Workload, Not the Headline

Token Forge Cloud's DeepSeek test separated TTFT from post-first-token generation speed. It used unique prompts, approximately 100 input tokens, about 140 output tokens on average, reasoning disabled, and fixed pacing at each load level.

Model Stable tested load TTFT p50 TTFT p95 Post-first-token generation speed Success rate
deepseek-v4.1-flash 5,000 RPM 1.36 s 1.97 s 71 token/s 100%
deepseek-v4-pro 2,500 RPM 2.15 s 2.82 s 41 token/s 100%
kimi-k3 5,000 RPM 0.96 s 1.26 s 169 token/s 100%
glm-5.3 4,200 RPM 1.89 s 5.62 s 73 token/s 99.95%
glm-5.3-flash 2,400 RPM 2.01 s 5.29 s 40 token/s 100%

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

These figures should be read together. The token rate describes the stream after it begins; TTFT describes the wait before it begins. The stable load and success rate show the traffic conditions under which those measurements were observed.

  • For chat, autocomplete, classification, routing, and short structured outputs, optimize TTFT and tail latency first.
  • For reports, code generation, and long-form content, generation speed and sustained TPM become more important.
  • For agents, measure the whole workflow: many short sequential calls can make TTFT more costly than a single long generation.

The best evaluation uses representative prompts and reports time to first token, time to completion, output length, token throughput, and success rate at the expected load. Cost should be evaluated on the same workload because a small speed difference may not justify a large price difference.

One number cannot describe the whole experience. A 100 token/s endpoint may be excellent for long generation and still feel slow in chat. A 50 token/s endpoint may feel immediate and finish short tasks first. The faster system is the one that reaches the user's required outcome sooner and more consistently.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us