When an early result has independent value, the usual default is explicit partial-success settlement: preserve the useful result, record every subcall outcome, and retry or compensate only the affected work. Use all-or-nothing handling instead when the partial result would be unsafe, misleading, contractually invalid, or inconsistent with a required business invariant. The final choice depends on API semantics, client expectations, dependency relationships, external side effects, and consistency requirements.
The Short Answer: Settle Explicitly as Partial Success
A logical request does not always fit a binary success-or-failure model. It may complete its primary task while failing an optional enrichment, secondary model call, notification, or downstream update. If the completed work remains useful on its own, marking the entire request as failed can hide value from the client and encourage unnecessary repetition.
An outcome-aware settlement model addresses that problem by representing both facts:
- The request produced a usable result.
- One or more subcalls failed, remain pending, or require follow-up.
The overall state might therefore be partially_succeeded or completed_with_errors, while each subcall retains its own state. This is more informative than treating an HTTP success code, an error code, or a single Boolean field as the complete settlement decision.
Why a useful independent result should not automatically be rolled back
Consider an AI workflow that generates a valid answer and then calls separate services for citation enrichment, classification, archival, or analytics. If citation enrichment fails but the answer still meets the API contract, discarding the answer may provide no business benefit. It may also cause the client to repeat expensive or side-effecting work.
The correct test is not simply whether any subcall failed. Ask whether the early result:
- Satisfies a defined part of the client contract.
- Can be interpreted correctly without the missing output.
- Remains valid if later work never succeeds.
- Can be exposed without breaking a business or consistency invariant.
If those conditions hold, preserving the result and reporting the incomplete work is generally clearer than failing the entire request.
The conditions that make partial-success settlement appropriate
Explicit partial success is often suitable when later subcalls are optional, independently retryable, or used for read-only enrichment. Common examples include:
- A primary model response succeeds, but an optional metadata classifier fails.
- One model in a multi-model request returns a useful result while a secondary comparison model times out.
- A batch enrichment job completes most independent items but cannot process several records.
- A fallback path produces an acceptable output after another route fails.
- A response is delivered while nonessential logging, indexing, or notification work continues asynchronously.
All-or-nothing settlement is usually preferable when an incomplete result would be unsafe or deceptive, when every step is contractually required, or when partial completion would violate a business invariant. For example, a workflow should not report that a resource has been successfully reserved if the required reservation operation failed, even if an earlier availability check succeeded.
Define the Request Contract and Its Outcome States
Partial-success handling works only when the API contract explains what “partial” means. The contract should identify required and optional work, define when a result becomes usable, and tell clients whether incomplete work will be retried, reconciled, or abandoned.
Separate the overall request state from each subcall state
Use two levels of state:
- Logical-request state: What can the client conclude about the request as a whole?
- Subcall state: What happened to each model call, tool invocation, data operation, or downstream action?
The overall state should be derived from documented terminal-state rules rather than from whichever subcall finished last. A required subcall failure may make the logical request unsuccessful, while the same failure in an optional enrichment step may produce a partially successful result.
The following response envelope is an illustrative design example, not a product API contract:
``json { "request_id": "req_8f21", "state": "partially_succeeded", "result": { "answer": "A usable primary result" }, "steps": [ { "name": "primary_inference", "state": "succeeded", "required": true }, { "name": "citation_enrichment", "state": "retryable", "required": false, "error_code": "DEPENDENCY_TIMEOUT" }, { "name": "archive_update", "state": "pending", "required": false } ], "follow_up": { "status_url": "/requests/req_8f21" } } ``
The envelope lets the caller consume the valid answer without assuming that every associated operation completed.
Represent succeeded, failed, pending, skipped, and retryable work
A useful state taxonomy distinguishes outcomes that require different client or platform behavior:
- Succeeded: The step reached its defined completion condition.
- Failed: The step did not complete and no automatic retry is planned.
- Pending: Work has been accepted but has not reached a terminal state.
- Skipped: Policy or dependency logic determined that the step should not run.
- Retryable: The attempt failed, but policy permits another attempt.
Avoid using retryable without defining who owns the retry, when it may occur, and how the final state becomes visible. A client that retries while the platform is also retrying can create duplicate execution unless the contract coordinates those behaviors.
Error details should help the caller distinguish permanent rejection from transient dependency failure without exposing sensitive internal information. For long-running work, a stable request identifier and status endpoint can provide a clearer contract than holding the original connection open.
Use completed-with-errors or partially-successful terminal states
A request can be terminal even when some subcalls failed. Completed_with_errors communicates that no more platform-managed work is planned, while partially_succeeded communicates that usable output exists but the complete requested outcome was not achieved. Teams may choose different names, but their meanings and transition rules should remain precise.
Do not make HTTP status alone carry all of this meaning. An API might return a successful transport response with an explicit partial state, or it might use another documented status convention. Either way, clients need the response body and request contract to understand result usability, failed work, and follow-up behavior.
Likewise, an all-settled-style programming construct can collect the outcomes of several promises, but it does not by itself create durable workflow state, coordinate distributed retries, reverse external effects, or reconcile work after a process restart.
Match Failure Handling to Side Effects and Dependencies
The settlement decision becomes more consequential when subcalls change external state. Read-only enrichment and side-effecting operations should not share an undifferentiated failure policy.
A failed read-only classifier, retrieval call, or secondary inference typically leaves nothing to reverse. The platform may report the missing enrichment, retry it selectively, or complete without it if the contract permits.
A side-effecting operation may reserve inventory, create a record, initiate billing, publish a message, or update an external system. In those cases, the workflow must determine whether the completed effect can remain, must be reconciled, or needs a business-level compensating action.
Dependency structure also matters. If step C requires outputs from both A and B, failure of B may require C to be skipped even when A succeeded. That does not necessarily invalidate A. The response can preserve A while showing that B failed and C was skipped because its dependency was unavailable.
Apply retries selectively and design for repeated execution
Retry only failures that policy classifies as transient and only when the operation can tolerate another attempt. Timeouts are especially ambiguous: the caller may not know whether the downstream system completed the operation before the response was lost.
Where repeated execution could create duplicate effects, use idempotency keys or deduplication controls. These controls can reduce the consequences of retries, but they should not be presented as exactly-once execution guarantees. Their behavior depends on key scope, retention, operation semantics, and the systems participating in the workflow.
Retry policy should define:
- Which error classes are eligible.
- Whether the client or service owns retries.
- How attempts are correlated with the original operation.
- When retrying stops and the step becomes terminal.
- How clients learn that a previously pending step has completed or failed.
Use compensation only when a business invariant requires it
Compensation is a business action that reverses or offsets an earlier external effect. It is not equivalent to rolling back a database transaction, and it may not restore the world to an identical prior state.
For example, compensation might release a hold, cancel a provisional record, or create an offsetting entry. Each action can fail independently and may require its own retry and reconciliation policy.
A saga-style workflow can be useful when several side-effecting operations form a long-running business process and completed steps require coordinated compensation after a later failure. It is not automatically necessary when failed subcalls are read-only, optional, or free of externally visible effects.
Treat an already delivered result as committed communication
Once a useful result has been returned to a caller, the system cannot credibly describe a later action as retroactively rolling back the caller’s receipt of that result. Later work should instead be framed as:
- Asynchronous completion of pending steps.
- Reconciliation of inconsistent external state.
- A status update that changes the request’s recorded outcome.
- Compensation for reversible business effects.
- A correction or invalidation notice when the contract allows it.
Clients need to know whether a returned result is final, provisional, or subject to later correction. That distinction should be explicit before failures occur.
Compare the Main Settlement Models
The following table summarizes the principal options. These models can be combined—for example, a request may return partial success while starting asynchronous reconciliation for one failed side effect.
| Model | Best fit | Treatment of useful early result | Failure handling | Principal tradeoff |
|---|---|---|---|---|
| Explicit partial success | Early output has independent value and incomplete work can be disclosed | Preserve and return it with per-step outcomes | Selective retry, skip, or report failure | Clients must understand richer state semantics |
| Fail-fast atomic handling | Every required step must succeed for the result to be valid | Withhold the result or classify the request as failed | Abort remaining work; roll back only where true transaction boundaries permit | Can discard otherwise useful work and may not span distributed systems |
| Saga with compensation | Multiple side-effecting steps form a long-running business process | Preserve or withhold it according to the business contract | Run compensating actions for completed effects that must be reversed or offset | Compensation adds workflow and operational complexity |
| Asynchronous reconciliation | Useful output can be delivered before noncritical or recoverable work finishes | Return it as final or provisional, as defined by the contract | Continue processing, reconcile state, and publish status changes | Requires durable status tracking and clear expectations about eventual outcomes |
Use fail-fast handling when every subcall is mandatory or the result is meaningless without complete execution. Use a saga when business-level side effects require coordinated reversal. Use asynchronous reconciliation when immediate response value outweighs the need to finish all work in the original request window.
For many enrichment and inference pipelines, explicit partial success is the clearest starting point because it preserves independent value without pretending that failed work succeeded. It should still be replaced by stricter handling whenever the request contract or business invariant demands it.
Build Observability Around the Logical Request
Partial settlement becomes difficult to operate if telemetry exists only for individual network calls. Operators need to reconstruct the full logical request across retries, routes, and asynchronous work.
Useful observability fields generally include:
- A logical-request correlation ID shared across subcalls.
- A stable identifier for each step and attempt.
- Per-step status, timestamps, and dependency relationships.
- Retry history and the policy reason for another attempt.
- A distinction between transient, permanent, and policy-driven outcomes.
- Terminal-state rules for the overall request and each subcall.
- Events for compensation, reconciliation, correction, or abandonment.
Telemetry should make several operational questions answerable: Was the primary result delivered? Which optional step failed? Did an external effect occur before a timeout? Is another retry scheduled? Has compensation reached its own terminal state?
Audit events and application logs should record state transitions rather than relying only on free-form error messages. Retention, access, and data-handling policies should reflect the sensitivity of prompts, model outputs, identifiers, and downstream business records.
Apply the Pattern to AI Inference and Serving-Layer Workflows
Multi-model routing, fallback, tool use, and enrichment can turn one user-visible AI request into several subcalls. The settlement contract should therefore sit above any individual model invocation. It should define which output is primary, which enhancements are optional, and what happens when a routed model, fallback, or downstream tool does not complete.
Token Forge Cloud Private LLM Inference supports private deployment and serving-layer optimization around caching, model routing, batching, quantization, and GPU scheduling. These serving concerns can participate in multi-step inference architectures where teams must define their own request-level outcome semantics, retry controls, telemetry, and business invariants.
Token Forge Cloud also offers Token Forge Cloud Managed Model APIs as an API-first path for teams validating model demand before committing to private serving capacity. Managed API and private deployment workflows may have different ownership boundaries, so architects should document which layer controls retries, request state, and client-visible outcomes.
For private deployment paths, models, prompts, and telemetry can remain in the customer’s controlled environment. Teams can align request correlation and operational telemetry with their surrounding architecture, while separately implementing the durable workflow or application logic required for partial settlement, compensation, and reconciliation.
The practical design sequence is:
- Define when the primary inference result becomes independently usable.
- Classify each later subcall as required, optional, or conditional.
- Identify which operations are read-only and which create external effects.
- Define per-step and overall terminal states.
- Assign ownership for retries, deduplication, compensation, and reconciliation.
- Expose enough status information for clients and operators to interpret the outcome correctly.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.