All insights

Performance Metrics

TTFT, Generation Speed, and Total Latency: Which Metric Affects User Experience?

TTFT strongly affects interactive products. In a chat interface, users see no progress until the first token arrives. A low TTFT makes the system feel responsive even when the complete answer takes longer.

“How fast is this model?” sounds like a simple question. For a streaming LLM API, it has at least three different answers.

Time to first token (TTFT) describes how quickly a response begins. Generation speed describes how quickly it continues. Total latency describes when the complete answer is available. Each metric affects the user differently, and none should be interpreted without knowing where its timer starts and stops.

1. A Streaming Response Has More Than One Phase

Time to first token (TTFT) is the time between sending a request and receiving the first generated token. It includes network transit, request admission, queueing, prompt processing, and the start of model generation.

Generation speed, or Throughput usually reported in tokens per second (tok/s), measures how quickly tokens arrive after generation begins. It affects reading flow and the completion time of longer outputs. However, providers do not always use the same boundary. Some calculate output tokens divided by the time from first token to completion. Others divide by the entire request time, including TTFT.

Total latency, or end-to-end latency, is the time from sending the request until the full response completes. For a simplified streaming request, it can be estimated as:

total latency ≈ TTFT + output tokens ÷ generation speed

The formula is approximate because token intervals are not perfectly uniform, but it is useful for understanding the relationship. TTFT is a largely fixed cost per request. Generation time grows with output length.

2. The Most Important Metric Depends on the Product

For conversational interfaces, TTFT is often the first performance constraint. Users can begin reading while later tokens are still arriving. A fast start with a moderate generation rate may feel better than a delayed start followed by a very fast stream.

For background workloads, the priority changes. A batch extraction pipeline cannot use a partially generated result. End-to-end latency and sustained TPM may matter more than first-token speed.

Long-form generation makes token speed increasingly important. If a model produces 2,000 output tokens, a difference of 25 tokens per second can change completion time substantially. For a 20-token classification result, the same difference may be barely noticeable because TTFT dominates the request.

Agents require a mixture of metrics. Each step may generate only a short tool call, making TTFT important, but the workflow may contain many sequential calls. Small delays accumulate. A single slow tail event can also block the entire chain, so p95 and p99 may matter more than the median.

When comparing providers, match at least these conditions:

  • Similar input and output lengths
  • The same streaming mode
  • The same reasoning and tool settings
  • The same latency boundary
  • Comparable load and request pacing
  • Similar percentile windows

Without those controls, two accurate numbers can still support an invalid comparison.

3. Read TTFT and Token Speed Together

Token Forge Cloud tested its Open-Source Model endpoints with approximately 100 input tokens, about 140 output tokens on average, reasoning disabled, unique prompts, and fixed request pacing. The generation-speed measurement covered the period after the first token.

At the highest stable tested load for each model, the results were:

Model Stable tested load TTFT p50 TTFT p95 Post-first-token generation speed Success rate
deepseek-v4.1-flash 5,000 RPM 1.36 s 1.97 s 71 token/s 100%
deepseek-v4-pro 2,500 RPM 2.15 s 2.82 s 41 token/s 100%
kimi-k3 5,000 RPM 0.96 s 1.26 s 169 token/s 100%
glm-5.3 4,200 RPM 1.89 s 5.62 s 73 token/s 99.95%
glm-5.3-flash 2,400 RPM 2.01 s 5.29 s 40 token/s 100%

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

The table describes two parts of the experience. TTFT shows how long users waited for visible output. Generation speed shows how quickly the response continued after it started. The 100% success rate confirms that these observations were not achieved by completing only a subset of requests at the tested stable levels.

These figures are a capacity baseline for a specific workload, not a latency promise for every prompt. Large contexts, long outputs, tool calls, and different traffic patterns can change all three measurements.

The right metric is therefore the one that matches the moment your user is waiting for. Measure TTFT when the user needs immediate feedback, generation speed when the stream itself matters, and total latency when work cannot continue until the response is complete. In many applications, the correct answer is to monitor all three.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us