A client disconnect should not automatically cancel, release, or reuse the reservation. If the upstream provider may still be processing, move the reservation into an explicit detached or pending-reconciliation state, preserve the upstream operation identifier, and wait for an authoritative outcome or apply a defined lease-expiry policy. Treat the network connection, reservation, and upstream operation as separate stateful objects.
The Short Answer: Detach the Reservation From the Client Connection
A transport connection tells you whether the client is currently reachable. It does not establish whether the client intended to cancel the business operation, whether an upstream provider received the request, or whether reserved capacity is still being consumed.
When a connection closes before the upstream outcome is known, a practical design is to:
- Record the disconnect without marking the upstream operation as cancelled.
- Preserve the reservation and its association with the upstream operation.
- Change the reservation to a state such as
detachedorpending_reconciliation. - Request upstream cancellation only if the provider supports it and the business policy calls for cancellation.
- Reconcile the operation through a status lookup, callback, durable event, or another authoritative signal.
- Release or finalize the reservation after confirmed completion, confirmed cancellation, failure, or defined lease-expiry handling.
This avoids two dangerous assumptions: that a disconnected client means the provider stopped working, and that a local timeout proves the reserved resource is no longer in use.
The exact policy depends on what is being reserved. A logical concurrency slot, GPU scheduling allocation, prepaid balance hold, provider quota, and customer-visible job slot may require different release and settlement rules. Provider billing semantics, cancellation behavior, expected workload duration, and the cost of temporary over-reservation should all influence the design.
Track the Connection, Reservation, and Upstream Operation Separately
The system should maintain independent state for three related but distinct concerns:
| Concern | Example states | What it answers |
|---|---|---|
| Client connection | Connected, disconnected, reconnected | Can the original transport still receive a response? |
| Reservation | Active, detached, cancellation requested, finalized, expired | Is capacity, quota, credit, or another resource still held? |
| Upstream operation | Submitted, processing, completed, cancelled, failed, unknown | What is the provider actually doing? |
These states can change independently. A client may disconnect after the provider accepts the operation. The provider may complete while a cancellation request is in flight. A client may reconnect after the result has already been stored. The reservation may also reach a local deadline while the provider's status remains unknown.
Use stable identifiers to maintain the relationship among these records:
- A client request or idempotency key for safe submission retries
- A reservation identifier for the held resource
- A stable internal operation identifier
- The provider's operation or job identifier, when available
- A result reference that survives the original connection
Do not rely on a socket, process-local object, or request handler as the authoritative record. Those objects can disappear while the business operation continues. Durable state should capture the last known provider status, reservation state, relevant deadlines, and transition history.
Stable identifiers and idempotent transitions make repeated handling safer, but they do not guarantee exactly-once execution by the provider. Submission races still require explicit handling—for example, when a request times out before the system learns whether the provider accepted it.
A Practical Reservation Lifecycle After Disconnection
The following lifecycle is an illustrative architecture pattern. State names and transition rules should be adapted to the provider protocol, resource type, and business policy.
| Illustrative state | Meaning | Typical next action |
|---|---|---|
active | Client is connected and upstream work is pending or running | Continue normal processing |
detached | Client disconnected, but the upstream outcome is unresolved | Preserve the hold and start reconciliation |
cancellation_requested | A cancellation request was sent or accepted for processing | Wait for confirmation or a terminal provider status |
completed | Upstream completion was authoritatively reported | Store the result and finalize settlement |
cancelled | The provider confirmed cancellation | Release or settle the reservation under policy |
failed | The operation reached a confirmed failure | Release, retry, or settle according to failure rules |
expired | The local lease or reconciliation deadline was reached | Apply the configured ambiguity and escalation policy |
compensated | A later business action reversed an effect that could not simply be undone | Record the compensating outcome and close the workflow |
Example disconnect path
When the server detects a disconnect, it should first persist the event and retain the identifiers needed to find the upstream work. It can then mark the reservation as detached and schedule reconciliation independently of the original request process.
If upstream cancellation is both supported and appropriate, issue the request using the provider operation identifier. Record that cancellation was requested, but do not treat the request as proof that execution stopped.
Reconciliation should continue until the system receives an authoritative terminal outcome or reaches its local deadline. Depending on the provider, that signal may be:
- A successful status lookup showing completion, cancellation, or failure
- A verified callback associated with the provider operation identifier
- A durable completion event
- A synchronous cancellation response whose documented semantics confirm termination
If the provider completes after the client has gone, store the result or a durable result reference. A reconnecting client can then retrieve the outcome using its stable request identifier, subject to the application's retention and authorization rules. This prevents useful completed work from being discarded merely because the original response channel disappeared.
Cancellation Requested Is Not Cancellation Confirmed
A cancellation request is an instruction or intent. Cancellation confirmation is an authoritative statement about the upstream outcome. Keeping these states separate is essential because cancellation may be best-effort, asynchronous, or too late to stop work that is already completing.
For example, an API may acknowledge receipt of a cancellation request without confirming that computation has ended. During that interval, the provider might still consume capacity, produce a result, or create a billable event. Releasing the reservation immediately could therefore make the same local resource appear available twice.
A robust transition model should account for at least three outcomes after cancellation is requested:
- Cancellation confirmed: Finalize the operation as cancelled and release or settle the hold according to policy.
- Completion wins the race: Record the completed result and apply the completion settlement rule.
- Outcome remains unknown: Preserve a bounded detached state and continue reconciliation until the deadline or escalation path is reached.
Cancellation support should also be evaluated by operation phase. Some providers may permit cancellation while a job is queued but not once execution starts. Others may return a terminal status without guaranteeing that every downstream side effect was reversed. The integration should use the documented meaning of each provider response rather than assigning a universal interpretation to an HTTP success code.
Use Leases and Deadlines to Resolve Unknown Outcomes
A detached reservation should not remain open forever. Attach a bounded lease or reconciliation deadline so unresolved work eventually enters a defined handling path. The deadline limits indefinite holds; it does not prove the provider stopped processing.
Choose the lease policy based on factors such as:
- Expected and maximum practical workload duration
- Whether the reservation represents physical capacity, logical concurrency, quota, or funds
- Provider cancellation and status-lookup capabilities
- Provider billing and settlement behavior
- The cost of holding capacity versus releasing it too early
- The consequences of a late completion
At the deadline, the system needs an explicit policy. It might retain a reduced accounting hold, quarantine the reservation from immediate reuse, escalate the operation for investigation, or locally expire the reservation while continuing to watch for a late provider outcome. The appropriate choice depends on whether double allocation, delayed availability, or incorrect settlement creates the greater business risk.
Plan for late completion
A provider can complete after the local lease expires. The completion handler should therefore recognize both the stable operation identifier and the previous expiry transition. It should not silently reopen or overwrite settled business state.
Possible responses include storing the late result, applying an adjusted settlement rule, notifying an operator, or initiating compensation. Compensation is a new business action—not a perfect rewind. If a completed operation triggered an external side effect, the compensating step may have its own cost, failure modes, and audit history.
Timeout-boundary races should be tested deliberately. Important cases include a completion arriving just before expiry, just after expiry, during a retry, or after an operator has already resolved the reservation manually.
Make Reconciliation Durable, Idempotent, and Observable
Reconciliation must continue even if the original application process restarts. Persist enough information to reconstruct the operation: stable identifiers, reservation type, last known provider status, cancellation state, lease deadline, retry schedule, result reference, and transition history.
Callbacks, polling responses, retries, and reconnects may be duplicated or arrive out of order. Design each state transition to be idempotent and conditional on the current state. For example, a duplicate completion callback should return the already-recorded outcome rather than finalize the same reservation twice. An older processing event should not replace a newer terminal completed state.
Useful controls include:
- Uniqueness constraints around operation and provider identifiers
- Compare-and-set or versioned state updates
- Deduplication of provider events
- Retry schedules with bounded backoff
- A reconciliation worker independent of client connections
- A durable transition history for operational review
- A defined path for reconnecting clients to retrieve status or results
Idempotency makes repeated event handling safer, but it does not eliminate duplicate execution at an upstream provider. Submission retries should follow the provider's idempotency protocol where one exists, and ambiguous submissions should be reconciled before a new operation is created whenever possible.
Operational signals to monitor
Operators should be able to identify detached operations before they become permanent orphans. Monitor reservation age, the number of detached reservations, reconciliation failures, cancellation-confirmation latency, late completions, expired-but-unresolved operations, and repeated provider callbacks.
Alerts should focus on actionable conditions, such as a growing detached backlog, status lookups failing beyond the retry window, or reservations exceeding their expected age. Logs should connect the client request, reservation, internal operation, and provider operation identifiers without depending on the original connection context.
Regular orphan scans provide a second recovery path when callbacks are missed or workers fail. Manual actions should use the same guarded state transitions as automated reconciliation so an operator response cannot accidentally duplicate finalization.
Choosing a Policy for Inference-Serving Reservations
Inference-serving reservations may represent concurrency, queued work, GPU scheduling capacity, provider quota, or an accounting hold. The right disconnect policy depends heavily on whether the upstream service supports cancellation and authoritative status lookup.
| Provider capability | Practical policy |
|---|---|
| Cancellation and status lookup | Request cancellation when appropriate, retain the detached reservation, and verify the terminal outcome through status reconciliation |
| Cancellation but no status lookup | Record cancellation as requested, wait for any authoritative completion signal, and use a bounded ambiguity policy if confirmation never arrives |
| Status lookup but no cancellation | Keep the reservation detached while processing continues, poll within defined limits, and store the result for later retrieval |
| Neither capability | Use a conservative lease based on expected workload duration, avoid immediate reuse where double allocation matters, and define late-completion and escalation handling |
Workload type also matters. Token Forge Cloud treats latency-sensitive chat, batch enrichment, and agentic workflows as different serving-policy problems. A disconnected interactive chat request may call for a different cancellation threshold than a long-running batch job whose completed result remains useful. Agentic workflows may also require attention to external side effects and compensation rather than only compute-resource release.
Token Forge Cloud Private LLM Inference supports private deployment and serving-layer optimization for enterprise AI workloads. Its serving-layer capabilities include caching, routing, batching, quantization, and GPU scheduling. When teams design reservation handling around this broader serving environment, they should define disconnect, cancellation, timeout, reconciliation, and settlement semantics as part of the workload policy rather than treating them as transport behavior.
For teams validating model demand before private deployment, Token Forge Cloud Managed Model APIs provide an API-first entry point. That validation phase can also reveal operation-duration distributions, retry patterns, cancellation behavior, and status-lookup needs that should inform a later private inference design.
Next Step
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.