All insights

Performance Metrics

Success Rate Is Not Enough: How to Define a Useful SLO for an AI API

An LLM API can report a 99.9% success rate and still deliver a poor product experience.

The reason is simple: an HTTP 200 response says that a request completed. It does not say that the first token arrived quickly, that streaming continued without stalling, or that the full response finished within a useful time. It also does not say whether the application received enough capacity during a traffic spike.

A useful service-level objective, or SLO, must describe what a good request looks like from the application's point of view.

1. Define a Good Event Before Choosing a Percentage

An SLO is a target for a measurable service-level indicator, or SLI. For a conventional API, the SLI may be the percentage of non-error responses. For an LLM API, one indicator is rarely enough.

A streaming chat application might define a good request as one that:

  • Is accepted rather than rate-limited
  • Returns a valid first token within three seconds
  • Continues streaming without a long inter-token stall
  • Completes successfully within the application's total time budget

The exact thresholds depend on the product. Interactive chat is sensitive to TTFT because the user is waiting for visible feedback. A background extraction job may tolerate a slower start but require a predictable completion deadline. A coding agent may value completion reliability more than a very fast first token.

Request eligibility also needs a clear definition. Client errors caused by invalid authentication or malformed parameters usually should not count against provider availability. Provider timeouts, upstream failures, and unexpected stream termination usually should. Rate limiting requires an explicit policy: if traffic is within the contracted limit, a 429 may be a service failure; if the client exceeds its allocation, it may be expected behavior.

Without these definitions, two teams can publish the same success rate while counting different events.

2. Combine Availability, Latency, and Capacity

A practical AI API SLO often needs at least three dimensions.

Availability measures whether eligible requests complete successfully. This catches transport errors, provider failures, and incomplete streams.

Latency measures whether successful requests are useful in time. TTFT p95 is a good indicator for interactive streaming. End-to-end p95 is more relevant when the application must wait for the entire answer. Output length should be controlled or segmented, because longer answers naturally take longer.

Capacity measures whether the first two targets remain true at the required traffic level. A latency objective measured at 10 RPM says little about behavior at 2,000 RPM. The SLO should therefore name the expected RPM, TPM, or concurrency range.

For example, an internal objective could say: “During each rolling 30-day window, 99.5% of eligible chat requests complete successfully; 95% receive a first token within three seconds; and these targets apply while traffic remains within the agreed RPM and TPM envelope.” This is only an example, not a Token Forge Cloud commitment. The correct numbers should come from product requirements and measured capacity.

Model quality belongs nearby but is usually measured separately. An API can be operationally healthy while returning an answer that fails a task-specific evaluation. Reliability SLOs should not be used as a substitute for accuracy, safety, or instruction-following tests.

3. Connect the SLO to Measurements and an Error Budget

Token Forge Cloud's DeepSeek load test used a simple stability criterion: success rate of at least 99% and TTFT p95 no higher than 5 seconds. That rule was applied at each traffic level, making it possible to identify the highest level that met both availability and responsiveness requirements.

Model Stable tested load Success rate TTFT p50 TTFT p95 Token throughput
deepseek-v4.1-flash 5,000 RPM 100% 1.36 s 1.97 s >1.21M TPM
deepseek-v4-pro 2,500 RPM 100% 2.15 s 2.82 s >617K TPM
kimi-k3 5,000 RPM 100% 0.96 s 1.26 s ~5M TPM
glm-5.3 4,200 RPM 99.95% 1.89 s 5.62 s ~5.4M TPM
glm-5.3-flash 2,400 RPM 100% 2.01 s 5.29 s ~3.0M TPM

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

This criterion is appropriate for comparing load levels in the test. It is not, by itself, a full production SLO or an uptime guarantee. A production objective needs a longer measurement window, eligibility rules, monitoring coverage, and a plan for missing or delayed telemetry.

It also needs an error budget. If the availability target is 99.5%, the remaining 0.5% is the allowed budget for bad events during the measurement window. Teams can use budget consumption to decide when to prioritize reliability work, reduce rollout speed, or investigate a provider route.

Useful telemetry should make each bad event diagnosable. Token Forge Cloud exposes request-level status, token usage, latency, cost, and request IDs, along with structured error responses. Those fields help separate invalid requests from rate limits, upstream timeouts, and other service failures. They also make it possible to calculate an application-specific SLI instead of relying on a provider-wide headline number.

The central principle is that an SLO should match a user outcome. “The request returned 200” is a system event. “The user saw a first token quickly and received the full answer within the expected traffic envelope” is a service objective.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us