Kimi K3 supports prompt caching for repeated prefixes, which can reduce the repeated work of stable instructions and documents. Caching helps only when the prompt is designed to reuse the same prefix and measured under the application's real traffic.
1. Longer Input Adds Work Before the First Token
Before generating an answer, the model must process the input prompt. This prefill stage contributes to time to first token along with network transit, admission, queueing, and the start of generation.
A short chat turn and a large document therefore create different performance profiles even if both request 300 output tokens. Longer inputs also consume more memory for the request state. At shared load, that can reduce batching efficiency or the number of requests that fit concurrently.
Do not apply a short-context TTFT result to a long-document workload. Token Forge Cloud's Kimi K3 load test used about 1,000 input tokens and produced a 0.96-second median and 1.26-second p95 TTFT at 5,000 RPM. Those figures establish a baseline for that request shape, not for 15,000-token or 100,000-token prompts.
Production dashboards should bucket TTFT by input-token count. Otherwise, a shift toward longer conversations can look like endpoint degradation even when the service has not changed.
2. Cache Stable Prefixes, Not Entire Dynamic Prompts
In compatibility testing, a repeated Kimi K3 prompt contained 15,013 input tokens, and 15,006 were reported as cached on repetition. The usage data appeared in the OpenAI-compatible field prompt_tokens_details.cached_tokens.
That result shows that repeated long prefixes can be recognized. It does not mean any similar prompt will receive the same cache hit rate. Prompt caches usually depend on exact token reuse in an eligible prefix. A timestamp, user identifier, or unique instruction placed near the beginning can break the reusable sequence that follows it.
Structure prompts from stable to dynamic content:
- Stable system instructions and tool definitions
- Reusable reference material
- Conversation or task context
- The current user's unique request
Keep formatting and serialization deterministic. Reordering equivalent JSON keys or changing whitespace may change tokens and reduce reuse. Track cached-token ratios in production rather than assuming the cache is active.
Caching should not be used to hide irrelevant context. Even when repeated input receives a billing or processing benefit, unnecessary material can complicate retrieval, reduce answer focus, and create stale-information risks.
3. Context Changes TPM Even When RPM Does Not
Requests per minute count calls, not the amount of text processed. A simple estimate is:
total TPM = RPM × (average input tokens + average output tokens)
At 1,000 RPM, increasing the average input from 1,000 to 10,000 tokens changes the pre-cache workload from roughly one million to ten million input tokens per minute. RPM remains unchanged while compute, memory, and billable usage can change dramatically.
The Kimi K3 report described 5,000 RPM as about five million input tokens per minute because the test used roughly 1,000 input tokens per request. A workload with 10,000-token prompts at the same request rate would be around 50 million input tokens per minute before considering cache effects.
Output length creates a separate constraint. At 5,000 RPM, Kimi K3's response start stayed stable, but median output speed fell from about 285 to 169 tokens per second. Longer outputs would hold streams open longer and increase active concurrency further.
4. Build a Context-Length Test Matrix
Test the prompt sizes the product actually uses. Useful buckets might include short chat, medium retrieval context, and long-document analysis. For each bucket, run both cold and repeated-prefix cases.
Record:
- Input, cached input, reasoning, and output tokens
- TTFT p50, p95, and p99
- Total latency and output speed
- Successful RPM and active concurrency
- Cost per valid result
- Quality with and without retrieved context
Keep output length and reasoning effort controlled when isolating the effect of input context. Kimi K3 defaults to medium reasoning if the setting is omitted; set none explicitly when comparing against the published no-reasoning load test.
Applications can reduce context through retrieval, conversation summarization, chunking, and removal of duplicated history. The goal is not the smallest possible prompt. It is the smallest prompt that preserves the information required for a correct answer.
Prompt caching is most valuable when the application has a genuinely stable, long prefix. Treat it as an optimization with observable hit rates, not as permission to send every available token on every request.