An AI gateway should normalize provider usage through versioned adapters that populate a canonical usage envelope, preserve the original usage payload, and label each value by status and provenance. Input, visible output, reasoning, cached input, and cache-write usage should remain distinct, while provider-reported totals and gateway-derived totals should be stored separately. Normalization creates a stable data contract; it does not make token counts inherently equivalent across models, providers, or tokenizers.
The Recommended Pattern: A Canonical Usage Envelope With Explicit Provenance
The safest internal contract represents usage as a set of typed measurements rather than a flat collection of integers. Each measurement should answer three questions:
- What category of usage does this value represent?
- Where did the value come from?
- What is known about its availability and relationship to other values?
A practical envelope can include request identity, provider context, normalized measurements, provider-reported totals, derived values, pricing references, and the untouched native usage object. This gives application teams a stable interface without erasing provider-specific details needed by engineering and FinOps teams.
Core fields for input, output, reasoning, cache reads, cache writes, and totals
At minimum, consider separate normalized measurements for:
- Input tokens: Tokens associated with the request input under the provider’s accounting rules.
- Visible output tokens: Tokens returned as user-visible or application-consumable output.
- Reasoning tokens: Internally generated reasoning usage when the provider exposes it.
- Cached-input tokens: Input tokens served through a cache or identified as cache reads.
- Cache-write tokens: Tokens processed to create or populate a cache entry, if separately reported.
- Provider-reported total: The total supplied by the provider without modification.
- Gateway-derived total: A calculation made by the gateway under a documented rule.
Each measurement should carry metadata rather than relying on the field name alone. An illustrative internal representation could look like this:
``json { "metric": "reasoning_tokens", "value": 180, "status": "present", "provenance": "provider_reported", "relationship": "included_in_output", "derivation_method": null } ``
Useful provenance values include provider_reported, gateway_derived, and estimated. Useful status values include present, absent, null, zero, and unsupported. Keeping provenance and status as separate dimensions prevents an estimate, an explicit zero, and an unavailable value from being treated as the same fact.
The full envelope should also record operational context when known:
- Logical request ID and individual attempt ID
- Provider and model identifiers
- Provider request ID
- Tokenizer or tokenizer version
- Native schema version and adapter version
- Normalization version
- Event and response timestamps
- Pricing-version reference and currency
These fields make it possible to explain how a normalized record was produced and to reprocess historical events when an adapter or accounting policy changes.
Normalize reasoning without double counting output
Reasoning-token accounting requires an explicit provider-specific relationship. Depending on the native schema, reasoning usage may be:
- Included within reported output tokens
- Reported separately from visible output
- Reported as an output detail without clarifying whether it is additive
- Omitted entirely
An adapter should therefore record both the reasoning value and its relationship to output. If reasoning is included in the provider’s output count, a gateway must not add it to that output count again. If the provider defines reasoning as additional usage, a derived total may include it—but only under that documented rule.
When the relationship is unknown, the gateway should preserve the reported values and mark the relationship as unknown. It is better to leave a derived total unavailable than to create a precise-looking number from an unsupported assumption.
Keep cache reads and cache writes separate
“Cached tokens” can refer to different events. A cache read or cached-input measurement represents input that was satisfied through a cache mechanism. A cache-write measurement represents work associated with creating or populating reusable cache state. These categories can have different accounting and pricing treatment.
For every cached-input value, the adapter should also record whether those tokens are already included in the reported input total. This allows downstream systems to avoid calculations such as adding cached input to an input total that already contains it.
Cache telemetry can help teams analyze workload behavior, but a cache-token count alone does not establish savings. The financial result depends on the provider’s billing rules, the applicable pricing version, cache eligibility, and whether cache activity changes other categories of usage.
Why normalized counts are not automatically comparable across tokenizers
A canonical field name provides structural consistency, not semantic identity. Two models can tokenize the same text differently, apply different context-processing rules, or report internal usage under different definitions. A record of 1,000 input tokens for one model should not automatically be interpreted as equivalent work, content volume, or cost to 1,000 input tokens for another.
Cross-model analysis should retain the model and tokenizer context and use token counts as model-specific metering facts. For broader workload comparisons, teams may also need business-level measures such as completed requests, documents processed, conversations handled, or successful task outcomes.
Map Each Native Payload Through a Versioned Provider Adapter
Provider-specific fields should terminate at an adapter boundary. Downstream applications, dashboards, allocation systems, and routing services should consume the canonical envelope rather than embed assumptions about every native response shape.
A versioned adapter should perform several bounded tasks:
- Parse the native payload without silently coercing missing values.
- Map recognized measurements into canonical fields.
- Attach status, provenance, and relationship metadata.
- Preserve provider totals independently from derived calculations.
- Retain unknown fields and the original usage object.
- Emit the adapter and normalization versions used for the mapping.
Versioning is important because provider schemas and gateway interpretation rules can change independently. Historical records should continue to show which logic produced them.
Retain the unmodified usage payload for audits and debugging
The normalized envelope should not replace the source payload. Store the original usage object alongside the normalized result, subject to the organization’s data-handling and retention policies.
Raw retention supports practical engineering tasks:
- Investigating discrepancies between native and normalized values
- Reprocessing records after an adapter correction
- Detecting newly introduced provider fields
- Comparing schema versions
- Explaining why a field was marked unavailable or estimated
Normalized fields should be additive. The gateway should not overwrite the provider’s total, remove unknown properties, or rewrite a native value to make it satisfy an internal arithmetic invariant.
Example: mapping two different native payload shapes into one envelope
The following fictional examples illustrate mapping behavior only. They do not represent any named provider’s schema.
Provider A reports input and output totals, with reasoning included as an output detail:
``json { "usage": { "prompt_units": 900, "generated_units": 300, "generated_details": { "thinking_units": 180 }, "all_units": 1200 } } ``
Provider B reports input, visible output, and cache-read usage, but does not expose reasoning:
``json { "metering": { "in": 900, "out_visible": 120, "cache_read": 400 } } ``
A canonical representation can preserve their differences:
``json { "provider_a": { "input_tokens": {"value": 900, "status": "present", "provenance": "provider_reported"}, "visible_output_tokens": {"value": 120, "status": "present", "provenance": "gateway_derived"}, "reasoning_tokens": {"value": 180, "status": "present", "provenance": "provider_reported", "relationship": "included_in_output"}, "cached_input_tokens": {"value": null, "status": "unsupported", "provenance": null}, "provider_total_tokens": 1200 }, "provider_b": { "input_tokens": {"value": 900, "status": "present", "provenance": "provider_reported"}, "visible_output_tokens": {"value": 120, "status": "present", "provenance": "provider_reported"}, "reasoning_tokens": {"value": null, "status": "absent", "provenance": null}, "cached_input_tokens": {"value": 400, "status": "present", "provenance": "provider_reported", "relationship": "included_in_input"}, "provider_total_tokens": null } } ``
Provider A’s visible output is derived in this example because its native output includes reasoning. The derivation method should be recorded explicitly. Provider B’s missing reasoning field is not converted to zero, and its cached input is not added to input because the adapter records that it is already included.
Reconcile totals without destroying conflicting evidence
There is no safe universal rule that every total must equal input plus visible output plus reasoning plus cache usage. The appropriate arithmetic depends on how the native schema defines each component.
A stable reconciliation policy should:
- Preserve the provider-reported total as its own value.
- Calculate a component sum only when all required inputs and relationships are known.
- Store that sum as gateway-derived rather than replacing the provider total.
- Compare the two values and emit a discrepancy state when they conflict.
- Record the normalization rule and version responsible for the calculation.
For display and allocation, teams can define a precedence policy—for example, using the provider-reported total as the primary source while showing a reconciliation warning. The important point is that precedence controls consumption; it should not erase the conflicting measurement.
Attribute streaming, retries, fallbacks, and multi-model calls by attempt
Usage accounting becomes unreliable when one logical application request is assumed to equal one provider call. A request may produce multiple attempts because of streaming, retries, failures, fallback routing, tool use, or multi-model orchestration.
Use two levels of identity:
- A logical request ID groups all work performed for one application-level operation.
- An attempt ID identifies each individual provider or model invocation.
Streaming usage may arrive in a final response, in incremental events, or through a delayed event. The gateway should treat usage records as idempotent events with stable event IDs where possible. Replayed terminal events must not be added twice.
Failed and superseded attempts should remain visible because they may still consume billable resources. Fallback routing should create a new attempt under the same logical request rather than overwriting the failed route. Multi-model workflows should retain a separate record for each model call before calculating a logical-request aggregate.
Represent Missing Values, Zeroes, Estimates, and Unsupported Metrics Differently
A normalized field needs more than a nullable number. The gateway must preserve what is actually known about the value:
- Absent: The field did not appear in this payload.
- Null: The provider included the field but supplied no value.
- Zero: The provider or a valid derivation explicitly established a value of zero.
- Unsupported: The metric is not exposed under the applicable schema or integration.
- Estimated: The gateway calculated an approximation rather than receiving a reported value.
- Present: A value is available, with provenance indicating whether it was reported or derived.
These states have different operational meanings. An explicit zero can participate in arithmetic. An absent or unsupported value generally cannot. A null may indicate delayed data, conditional reporting, or another provider-defined condition. None should be silently converted to zero.
Label estimates and derived values without overwriting reported usage
When an estimate is operationally useful, store its method and inputs. For example, a gateway may estimate usage using a known tokenizer when the native response omits a count. That estimate should remain distinguishable from provider-reported billing data.
Recommended metadata includes:
- Derivation method and implementation version
- Tokenizer identity and version when known
- Input data used for the estimate
- Confidence or limitation note
- Timestamp at which the estimate was generated
If provider-reported usage arrives later, retain both records or supersede the estimate through a traceable event. Do not silently mutate the historical estimate into a reported value.
Keep token normalization separate from price calculation
The usage layer should describe measured or derived consumption. A separate pricing layer should convert those measurements into monetary values using versioned rates and billing rules.
This separation matters because prices can change while historical token facts remain the same. Providers may also price input, output, reasoning, cache reads, and cache writes differently or apply conditions that cannot be represented by a simple per-token multiplier.
A price calculation should reference:
- The immutable normalized usage record
- Model and provider identifiers
- Pricing version and effective period
- Currency
- Category-specific rates and billing rules
Updating a price catalog should permit recalculation without rewriting the underlying usage event. Likewise, correcting an adapter should produce a new normalization result without concealing the earlier interpretation.
Validate adapters with fixtures, invariants, and schema-change observability
Adapter tests should use representative native fixtures for every supported schema version and usage state. Coverage should include:
- Present, absent, null, zero, unsupported, and estimated values
- Reasoning included in output versus separately additive reasoning
- Cached input included in input versus reported outside it
- Conflicting provider totals and component sums
- Streaming events arriving out of order
- Duplicate terminal or delayed usage events
- Retries, failed attempts, fallback routes, and multi-model requests
- Unknown fields and provider schema changes
Arithmetic invariants should be conditional. For example, an invariant may assert that a derived visible-output value cannot be negative, but it should not assert a universal total formula across every provider.
Unknown native fields should be retained where practical and surfaced through logs, metrics, or alerts. An unmapped-field signal gives engineering teams an opportunity to review schema changes before data disappears silently from downstream reporting.
Connecting Normalized Usage to Serving-Layer Operations
Normalized telemetry can become a common input for model routing, caching analysis, cost allocation, capacity planning, and operational reporting. It helps teams ask consistent questions while preserving the provider and model context needed to interpret the answers.
For example, engineering teams can examine per-attempt usage when evaluating fallback behavior, while FinOps teams can aggregate logical-request usage against a versioned pricing catalog. Platform teams can analyze cache reads and writes separately rather than treating every cache-related token as the same event. These analyses can inform decisions, but normalization alone does not prove savings or guarantee agreement with an invoice.
At Token Forge Cloud, we focus on enterprise LLM inference cost control and serving-layer operations. Token Forge Cloud Managed Model APIs provide an API-first path for model access and usage data, while Token Forge Cloud Private LLM Inference supports private deployment and serving-layer optimization through caching, routing, batching, quantization, and GPU scheduling. This guide presents a recommended design pattern; it does not document specific Token Forge Cloud telemetry fields or provider adapters.
Next Step
A useful usage model should remain explainable as model access expands from initial API evaluation to private inference operations. Teams should define the canonical contract, attribution rules, pricing boundary, and adapter test strategy before normalized data becomes embedded in dashboards and financial workflows.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.