A billing system can identify orphaned reservations by selecting overdue reservations that remain pending, then correlating lease expiry, worker ownership, heartbeat or attempt metadata, job status, durable usage or completion records, and financial-ledger state. An expired lease is a reason to investigate—not proof that no billable work occurred. Recovery should depend on the strongest available evidence: release work that did not execute, finalize recorded usage idempotently, retry recoverable work, and quarantine ambiguous cases.
An orphaned reservation is a reservation that remains pending or held after its owning worker stops progressing or its completion signal is not recorded. The key engineering challenge is distinguishing a crashed worker from a completed job whose final event was delayed, duplicated, or lost.
Identify Orphans by Correlating Lease, Worker, Usage, and Ledger Evidence
Age alone cannot identify an orphan safely. Long-running work may still be active, a worker may have completed execution immediately before crashing, or a completion message may have failed after durable usage was recorded.
A robust detector evaluates several categories of evidence:
- Reservation evidence: Current state, creation time, lease deadline, owner, attempt ID, and expected settlement path.
- Worker evidence: Last heartbeat, process or node identity, active-job registry, retry history, and lease-renewal history.
- Execution evidence: Job state, output status, provider response, or another durable indication that work started or completed.
- Usage evidence: Metered units associated with the reservation or attempt, such as tokens, requests, compute time, or other billable measures.
- Settlement evidence: Commit or release records, invoice-line status, and financial-ledger entries.
These records have different purposes and should not be conflated. A reservation represents held capacity, credit, or budget. A usage event records measured consumption. An invoice line expresses a billable amount. A ledger entry records a financial movement. Finding one does not automatically prove the state of the others.
A useful classification rule is:
> Treat pending + expired lease as an orphan candidate. Classify it only after checking worker, execution, usage, completion, and settlement evidence.
This approach prevents a cleanup process from silently releasing a hold when billable work actually completed.
Model Reservation Ownership and Settlement as Durable State
Detection becomes easier when ownership and settlement are represented by a durable state machine rather than inferred from scattered timestamps. State names vary by implementation, but an illustrative model could include:
- Pending: Funds, credits, or capacity are reserved while work is outstanding.
- Committed: Authoritative usage has been validated and the reservation has been settled.
- Released: The hold has been returned because the work did not create a valid charge.
- Expired: The reservation exceeded its allowed ownership period and awaits or has completed recovery.
- Disputed: Available evidence conflicts or requires operator review.
A simplified lifecycle is:
```text create reservation | v pending <--- lease renewal while work remains active / | \ / | \ commit release expire
| | | v v v committed released reconciliation | finalize / release / retry / dispute ```
Terminal states should normally be changed only through explicit correction procedures. For example, a delayed completion event should not overwrite a released reservation without validating whether a correction, dispute, or new ledger transaction is required.
Use bounded leases for worker ownership
A worker can claim a reservation for a bounded lease period and renew that lease only while the associated work remains active. The lease limits how long a failed worker can retain ownership indefinitely.
The lease record should identify the owner and attempt, not merely contain an expiry timestamp. This lets the system reject a renewal from an old attempt after another worker has taken over. The appropriate lease duration and renewal cadence depend on workload duration, heartbeat behavior, retry policy, and financial risk; there is no universal timeout.
The reservation state must also survive worker failure. Keeping critical state only in worker memory defeats the recovery mechanism because the evidence disappears with the process.
Query for Overdue Reservations Using Multiple Failure Signals
A periodic reconciliation job can begin with a narrow candidate query and then gather additional evidence. The following SQL-like example is conceptual and must be adapted to the system’s schema, source-of-truth hierarchy, and transaction model:
``sql SELECT r.reservation_id, r.attempt_id, r.owner_worker_id, r.lease_expires_at, w.last_heartbeat_at, j.status AS job_status, u.usage_id, c.completion_id, l.ledger_entry_id FROM reservations r LEFT JOIN workers w ON w.worker_id = r.owner_worker_id LEFT JOIN jobs j ON j.attempt_id = r.attempt_id LEFT JOIN usage_records u ON u.reservation_id = r.reservation_id AND u.attempt_id = r.attempt_id LEFT JOIN completion_events c ON c.reservation_id = r.reservation_id AND c.attempt_id = r.attempt_id LEFT JOIN ledger_entries l ON l.reservation_id = r.reservation_id WHERE r.state = 'pending' AND r.lease_expires_at < CURRENT_TIMESTAMP; ``
The query finds candidates; it does not decide their outcome. Reconciliation logic should interpret combinations of signals rather than treating any one condition as conclusive.
| Candidate condition | Supporting evidence to inspect | Likely interpretation | Conservative next action |
|---|---|---|---|
| Expired lease and stale heartbeat | Worker registry, job state, attempt history | Worker may have crashed | Check execution and usage before release or retry |
| Expired lease but active job | Current owner and lease version | Renewal may be delayed or racing | Recheck, renew conditionally, or defer |
| No completion event and no usage | Execution logs and provider status | Work may not have run | Release, expire, retry, or escalate under policy |
| Usage exists but no finalized charge | Usage validity and ledger state | Completion or settlement event may be lost | Finalize idempotently if evidence is authoritative |
| Completion exists but ledger entry is absent | Completion validity and prior settlement attempts | Downstream settlement may have failed | Retry guarded finalization |
| Conflicting usage or attempt records | Ownership history, deduplication keys, corrections | Outcome is ambiguous | Quarantine or send for manual review |
Operationally useful filters may include an expired lease, stale heartbeat, missing active worker, absent completion record, or usage present without a finalized charge. A confidence score can help prioritize investigation, but financial actions should still follow explicit rules rather than an opaque score alone.
Reconcile Worker Crashes Differently from Lost Completion Events
A worker crash and a lost completion event can produce the same visible symptom—a pending reservation past its lease deadline—but they call for different responses.
Likely worker crash with no execution evidence
Suppose the lease expired, the worker stopped heartbeating, no replacement attempt is active, and authoritative systems contain no execution, completion, or usage record. Depending on business policy, the system may:
- release or expire the reservation;
- retry the work under a new attempt ID;
- wait through an additional observation window; or
- escalate if the reservation is financially significant.
Even here, destructive deletion is a poor recovery action. Retaining the original reservation and transition history preserves the ability to explain what happened if delayed evidence appears later.
Likely lost completion or settlement event
If durable, validated usage exists for the matching reservation and attempt, releasing the hold merely because the completion message is absent could understate consumption. The safer pattern is generally to reconstruct or retry finalization idempotently, subject to the organization’s billing rules.
The reconciler should confirm that the usage belongs to the correct attempt, has not already been settled through another path, and meets the criteria for billing. Usage telemetry should not become a charge automatically without validation.
Choose recovery from authoritative evidence
| Evidence pattern | Potential action |
|---|---|
| No execution or usage evidence | Release, expire, retry, or defer |
| Valid usage with incomplete settlement | Finalize idempotently |
| Recoverable active attempt | Renew or transfer ownership conditionally |
| Conflicting ownership, usage, or ledger data | Quarantine |
| High-value or policy-sensitive ambiguity | Escalate for manual review |
This decision matrix should reflect the organization’s declared sources of truth. For example, the system should define whether provider-acknowledged execution, internal usage records, or another durable record controls settlement when events disagree.
Make Recovery Safe Under Retries and Late Events
Distributed systems should be designed for duplicate, retried, delayed, and reordered messages. Assuming exactly-once delivery transfers hidden risk into billing logic.
Use stable identities and idempotency keys
Every reservation should have a stable reservation ID. Each execution attempt should also have its own attempt ID so that a retry is not mistaken for the original execution. Finalization requests can carry an idempotency key derived from the reservation, attempt, action, and settlement version.
Deduplication should answer two separate questions:
- Has this event already been processed?
- Has its intended financial effect already occurred through another event or path?
Checking only an event ID may miss semantically duplicate messages with different delivery IDs.
Guard every state transition
Conditional updates—such as compare-and-swap operations—can prevent a reconciler and a stale worker from settling the same reservation independently. Conceptually:
``text update reservation set state = committed, settlement_version = settlement_version + 1 where reservation_id = :id and state = pending and settlement_version = :expected_version; ``
Only the process that observes the expected state and version succeeds. Other processes must reload the current record and decide whether their action is now a no-op, a correction, or a dispute.
When a reservation transition and a ledger entry must represent one financial action, use an appropriate transactional boundary or a durable transactional-outbox pattern. Idempotency reduces duplicate effects, but it does not by itself guarantee correct billing.
Define a late-event policy
Consider a completion event arriving while the reconciler is releasing a reservation. The system should define outcomes for at least these races:
- Equivalent settlement already occurred: Treat the event as an idempotent no-op.
- Release won, but valid usage later appears: Apply the approved correction or dispute workflow rather than silently rewriting history.
- A newer attempt owns the reservation: Reject changes from the stale attempt unless policy explicitly permits attribution.
- Evidence remains inconsistent: Quarantine the record and alert an operator.
A late event should never be discarded solely because it arrived after a timeout. Its identity, attempt, evidence, and current settlement state still need evaluation.
Operate Reconciliation as an Auditable Control Loop
Reconciliation works best as a repeatable control loop rather than an occasional cleanup script:
- Select candidates using overdue pending state and lease criteria.
- Collect evidence from worker, job, usage, completion, and ledger systems.
- Classify the failure using explicit, versioned policy rules.
- Apply a guarded action such as release, finalize, retry, quarantine, or escalate.
- Record the decision and the evidence used to make it.
- Verify the resulting state and retry safely if a downstream operation failed.
- Alert or escalate when records remain unresolved beyond operational thresholds.
For each reconciliation attempt, retain the prior and resulting states, reservation and attempt IDs, evidence references, reason code, policy version, actor or process identity, retry count, and timestamps. Manual-review outcomes should feed back into monitoring and policy refinement.
Useful operational measures include the number and age of unresolved candidates, outcomes by reason code, repeated reconciliation failures, late-event corrections, and reservations stuck in disputed status. Alert thresholds, retention periods, reconciliation frequency, and manual-review criteria should be selected according to workload behavior and financial exposure.
The control loop should never silently delete unresolved records. Quarantine preserves evidence and limits unintended financial effects when automation cannot reach a defensible outcome.
Apply the Pattern to AI Inference Metering and Serving Telemetry
AI inference creates a practical setting for this design because request execution, model serving, usage measurement, and billing settlement may occur in different components. A billing integration may need to correlate a reservation and attempt with serving telemetry before resolving an overdue hold.
The correlation model should account for workload behavior. Latency-sensitive chat, batch enrichment, and agentic workflows can have different execution lengths, retry patterns, and completion semantics. Caching, routing, batching, and retries can also complicate attribution, making stable request and attempt identities important at integration boundaries.
Token Forge Cloud offers Private LLM Inference for private deployment and serving-layer optimization for enterprise AI workloads, including caching, routing, batching, quantization, and GPU scheduling. Token Forge Cloud Managed Model APIs offer model access and usage data as an API-first entry point for teams validating demand before moving toward private deployment.
Operational usage data and financial records should still remain distinct. Teams designing an AI billing flow should determine how serving telemetry maps to their own reservations, rating rules, invoice lines, and ledger entries. They should also define how cached responses, batched execution, retries, interrupted generations, and multi-step agent activity are attributed before automated settlement occurs.
This architectural pattern does not require Token Forge Cloud to serve as the financial ledger. The relevant integration question is how available request and usage data can participate in the enterprise’s broader metering and cost-control design.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.