All insights

Provider Selection

Do Not Look Only at Token Price: Seven Metrics for Choosing an Open-Source Model Provider

Token price is easy to compare because it fits into one table. Production performance does not.

An inexpensive DeepSeek endpoint may become costly if requests are frequently retried, wait in long queues, or cannot reach the traffic level the application needs. A slightly more expensive endpoint may cost less per completed task if it produces valid responses consistently.

A useful provider comparison needs seven metrics. None is sufficient alone, but together they describe the service an application will actually receive.

1. Measure Responsiveness and Completed Work

1. Time to first token (TTFT). For streaming applications, TTFT determines when the user first sees progress. Track p50 for the typical experience and p95 for consistency. An average can hide a slow tail.

2. Generation speed. Tokens per second describes how quickly the response continues after it begins. Confirm whether the metric excludes or includes TTFT. The same request will show a higher post-first-token rate than an end-to-end rate.

3. End-to-end latency. Some applications cannot act until the full response is complete. Batch extraction, structured output, and agent tool calls should therefore measure total completion time, segmented by input and output length.

These metrics answer different questions. A short chat response may feel fast because TTFT is low even if token generation is moderate. A long report may benefit more from high generation speed. A provider comparison should not collapse both experiences into one “latency” number.

2. Measure Reliability, Capacity, and Output Quality

4. Success rate. Count only fully completed, usable responses. A stream that begins and then terminates unexpectedly should not be considered successful. Record 429, timeout, 5xx, transport, and invalid-response failures separately.

5. Stable capacity. Measure the highest RPM, TPM, and concurrency at which the provider continues to meet the success and latency requirements. A published account limit is not proof that the backend can sustain that workload.

6. Task success rate. HTTP success is not the same as application success. Evaluate DeepSeek responses using the task's real acceptance rule: schema validity for extraction, tests for code, rubric scores for support answers, or human review for complex writing.

Capacity without quality produces more unusable answers. Quality without capacity may work in evaluation but fail when traffic grows. The provider must support both at the same model configuration and traffic level.

Token Forge Cloud tested DeepSeek with approximately 100 input tokens, about 140 output tokens, fixed pacing, and reasoning disabled:

Model Stable tested load Success rate TTFT p95 Generation speed
deepseek-v4.1-flash 5,000 RPM 100% 1.97 s 71 token/s
deepseek-v4-pro 2,500 RPM 100% 2.82 s 41 token/s
kimi-k3 5,000 RPM 100% 1.26 s 169 token/s
glm-5.3 4,200 RPM 99.95% 5.62 s 73 token/s
glm-5.3-flash 2,400 RPM 100% 5.29 s 40 token/s

The Kimi K3 and GLM rows come from separate tests, the Kimi K3 load test report and the GLM load test report: about 1,000 input and up to 300 output tokens per request, Kimi K3 with reasoning off, and GLM at low reasoning (it cannot be turned off). GLM is held to a TTFT p95 of 8 seconds rather than 5.

This table covers operational performance, not task quality. A buyer should add its own evaluation set before choosing between Flash, Pro, or another provider implementation.

3. Calculate Cost per Valid Result

7. Effective cost. The useful cost metric is not price per million tokens in isolation. It is the total spend required to produce a successful result that passes the application's validation.

A simple formula is:

effective cost per valid result = total billed cost ÷ number of valid completed results

The numerator should include input and output tokens, retries, fallback calls, and any provider-specific charges. The denominator should exclude failed responses and outputs that do not pass the task's acceptance rule.

For example, suppose Provider A has a lower token price but only 90% of requests complete and 80% of completed outputs pass validation. Provider B costs more per token but achieves 99% completion and 95% task validity. Provider B may have the lower effective cost after retries and invalid results are included.

Cost also has a capacity dimension. If an endpoint cannot sustain the required RPM, the application may need additional providers, routing infrastructure, queueing, and operational support. Those costs do not appear on the token price page.

The seven metrics form a practical scorecard: TTFT, generation speed, total latency, success rate, stable capacity, task success, and effective cost. Weight them according to the workload rather than forcing one universal ranking. Real-time DeepSeek chat and overnight DeepSeek extraction can reasonably select different providers from the same measurements.

Read the full Token Forge Cloud DeepSeek load test report.

Contact us