Treat webhook delivery as at least once and make the financial effect idempotent. Give each billable job phase one immutable settlement key, enforce that key with a database or ledger uniqueness constraint, and commit the job transition, settlement record, and ledger entry atomically. Retries should return the original result rather than create another debit.
The Direct Answer: Make the Financial Effect Idempotent
An asynchronous completion webhook should be treated as a notification that settlement work may now be eligible—not as an instruction to debit a wallet immediately. Webhooks can be duplicated, delayed, reordered, or delivered while another handler is processing the same job.
The core invariant should be:
> For each tenant, job, and billable settlement phase, no more than one immutable debit may commit.
A practical design combines four controls:
- A stable internal job ID that follows the workload from submission through completion and usage accounting.
- An immutable settlement key for each billable phase.
- A durable uniqueness constraint that prevents two handlers from committing the same settlement.
- An atomic transaction that records the state transition and financial entry together.
This aims for an exactly-once financial effect within the defined transaction and recovery model. It does not require—and should not assume—exactly-once webhook delivery.
If the same valid completion event arrives repeatedly, each attempt should resolve to the same recorded settlement result. If two workers process duplicates concurrently, the datastore must decide which one commits. An in-memory cache, webhook retry counter, or distributed lock can reduce unnecessary work, but none should be the sole financial safeguard.
The design also needs operational assumptions. The settlement datastore must provide the consistency required by the selected transaction and uniqueness mechanism. Recovery workers must retry incomplete work, and reconciliation must identify records that diverge because of failures outside the atomic boundary.
Give the Job, Event, and Settlement Separate Durable Identities
Job identity, delivery identity, and financial identity solve different problems. Reusing one identifier for all three can create subtle errors when providers emit multiple events for a job or when a job has more than one legitimate billing phase.
Internal job ID
Create an immutable internal job_id when the asynchronous job is accepted. Use it to correlate:
- The original request and tenant
- Model and workload metadata
- Provider-side job references
- Completion and usage records
- Pricing inputs
- Reservations, captures, releases, and adjustments
- Ledger and reconciliation records
Do not make a mutable status or an external provider reference the only durable correlation point.
Provider event ID
Store the provider's unique event ID when one is available. If the provider does not supply one, derive a deterministic event key from stable payload fields after authenticity and schema validation.
This event key helps identify duplicate deliveries of the same notification. It is not, by itself, the financial uniqueness boundary. A provider may legitimately send a progress event, a completion event, a corrected usage event, and a replay for the same job. Different event IDs must not create multiple final charges for one billing phase.
Settlement key
Create an immutable key representing the financial effect. An illustrative key is:
``text settlement_key = tenant_id + job_id + charge_type + settlement_version ``
For example, charge_type might distinguish capture, release, or adjustment. A settlement_version can support a controlled correction model without allowing a retry to masquerade as a new debit. The exact fields depend on the commercial model, but every field used in the key should have stable semantics.
A compact illustrative data model could include:
```text jobs job_id, tenant_id, state, model_ref, usage_status, settlement_state
webhook_events event_key, provider_ref, job_id, payload_hash, received_at, processing_state
settlements settlement_key, job_id, charge_type, pricing_snapshot, amount, status UNIQUE(settlement_key)
ledger_entries ledger_entry_id, settlement_key, wallet_id, entry_type, amount, created_at UNIQUE(settlement_key, entry_type)
wallet_balances -- optional materialized projection wallet_id, balance, version ```
Store either the computed amount or the complete pricing snapshot used to derive it. That snapshot may include measured usage, pricing version, currency, rounding policy, and applicable commercial terms. A retry should not recalculate the debit using a newer price or changed input.
Commit the Job Transition and Ledger Debit Atomically
The unsafe implementation is a check followed by an insert:
``text if no settlement exists: insert debit ``
Two concurrent handlers can both complete the check before either inserts. Application logic may look correct during normal testing while still permitting duplicate debits under concurrency.
Instead, enforce uniqueness in durable storage and perform the related writes in one transaction where the architecture permits:
- Lock the job record or perform a compare-and-swap against its settlement state.
- Insert the settlement using its immutable settlement key.
- Insert the corresponding ledger debit.
- Update the job's settlement state.
- Update a materialized wallet balance, if one is maintained.
- Commit all changes together.
The unique constraint is the final concurrency control. If two workers attempt the same settlement, one insert commits and the other receives a uniqueness conflict. The losing worker should load the existing record, verify that it represents the expected job and phase, and return that recorded result.
Use a locking or consistency mechanism appropriate to the datastore, such as row locking, compare-and-swap, serializable transaction logic, or an equivalent atomic conditional write. A distributed lock may be useful for reducing contention, but the financial invariant should still be protected by the datastore or ledger.
Prefer an append-only ledger
A mutable wallet.balance field alone is a weak financial record because it does not explain how the current balance was produced. Prefer immutable debit, credit, release, and adjustment entries. If an incorrect debit has committed, add a compensating credit rather than rewriting history.
A wallet balance can be:
- Derived from ledger entries when needed; or
- Maintained as a materialized projection updated atomically with the ledger entry.
If the balance update and ledger insert cannot share one transaction, use a recoverable projection process and make the ledger the authoritative financial record. Reconciliation should detect and repair projection drift without creating another settlement.
Separate Webhook Receipt from Retryable Settlement Work
Webhook receipt and wallet settlement do not have to run in the same request. Separating them is often safer when processing includes external lookups, pricing resolution, queueing, or other work that may exceed the sender's timeout.
The receipt path should:
- Verify the webhook signature or other authenticity mechanism.
- Validate the payload schema and event type.
- Resolve the event to the expected internal job and tenant.
- Reject mismatched wallet, model, usage, currency, or amount-related data.
- Persist the event durably using its event key.
- Enqueue settlement work through a durable queue, transactional outbox, or comparable recoverable handoff.
Do not charge directly from an unverified amount in the webhook. Treat the event as input to an internal settlement decision. Compare its identifiers and usage data with the job information your system already holds, and apply the pricing snapshot associated with the intended billing phase.
A valid duplicate can usually receive a successful HTTP response after the receiver confirms that the event was already persisted. This stops needless provider retries while retaining an audit record of the duplicate attempt.
An HTTP success response should have precise semantics. It may mean durably accepted or already accepted; it does not necessarily mean downstream settlement completed synchronously.
Keep external calls outside the settlement transaction
Never hold the financial database transaction open while calling an inference provider, payment service, pricing service, or another external dependency. Network calls have uncertain latency and failure behavior, which can leave locks open and make transaction outcomes harder to reason about.
If external data is required, persist the event and move the job through durable intermediate states such as received, usage_validated, and ready_to_settle. A retryable worker can obtain the external information, preserve the resulting pricing inputs, and then open a short transaction for the final settlement writes.
A worker crash after event persistence is recoverable because the event remains queued or discoverable. A crash after transaction commit is also recoverable because the next retry encounters the same settlement key and returns the previously committed result.
Define Reservation, Hold, and Completion Rules for Conflicting Events
Wallet-funded asynchronous jobs often need more than a single pending or complete status. Model reservation, capture, release, adjustment, and reversal as distinct billing phases with explicit transition rules.
One illustrative lifecycle is:
``text job accepted -> funds reserved -> work running -> success -> reservation captured -> failure or cancellation -> reservation released -> corrected usage -> separate adjustment or reversal ``
Each phase should have its own settlement identity. For example, a capture must not reuse a release key, and a later adjustment should not overwrite the original capture. At the same time, two completion events must not create two capture entries.
Define deterministic policies for common edge cases:
Success after an earlier failure event
Decide whether the failure was terminal or provisional. If a later success can be valid, the state machine should specify which event has authority and whether the previous release must be followed by a new reservation, capture, or adjustment. Do not silently mutate the prior ledger history.
Cancellation racing with completion
A cancellation request and completion event may arrive concurrently. Use an atomic state transition to determine which transition wins under the commercial rules. The losing path should observe the committed terminal state and avoid creating an incompatible financial entry.
Partial completion
If partially completed work is billable, define the measurable unit, allowed amount, and settlement phase before implementation. Store the accepted usage and pricing snapshot so retries produce the same result. If partial work is not billable, the release policy should be equally explicit.
Delayed or reordered events
A late progress or failure event should not reverse a settled completion merely because it arrived later. Validate an event against the current state and the permitted transition graph rather than applying events in arrival order.
Corrected usage
Do not edit the original debit. Create a separately keyed adjustment or compensating credit that references the original settlement. This preserves the audit trail and prevents a correction from reopening the original capture key.
These rules depend on the organization's commercial, accounting, and customer-credit policies. The important engineering principle is to encode them as a deterministic state machine rather than leave them to handler timing.
Sequence and Pseudocode for an Idempotent Settlement Handler
The following flow is illustrative and should be adapted to the selected datastore, queue, wallet model, and billing rules.
```text Provider Webhook receiver Durable store/queue Settlement worker
| | | |
| completion event | | |
|--------------------->| | |
| | verify authenticity | |
| | validate payload | |
| | resolve internal job | |
| | persist event/outbox | |
| |---------------------->| |
| 2xx: accepted/already accepted | |
|<---------------------| | |
| | | deliver retryable work |
| | |------------------------>|
| | | begin transaction |
| | | lock/CAS job state |
| | | insert settlement |
| | | insert ledger entry |
| | | update job/balance |
| | | commit | ```
Illustrative receiver pseudocode:
```pseudo function receiveWebhook(request): verifiedPayload = verifyAuthenticity(request) event = validateAndNormalize(verifiedPayload) job = resolveExpectedJob(event.internalJobId)
assert event.tenantId == job.tenantId assert event.providerJobRef == job.providerJobRef assert event.modelRef == job.modelRef
eventKey = event.providerEventId ?? deriveDeterministicEventKey(event)
result = persistEventAndOutboxAtomically(eventKey, event, job.jobId)
if result == ALREADY_PERSISTED: recordDuplicateAttemptWithoutSensitivePayload(eventKey)
return successResponse("accepted") ```
Illustrative worker pseudocode:
```pseudo function settleCompletion(eventKey): event = loadPersistedEvent(eventKey) job = loadJob(event.jobId) validatedUsage = validateUsageAgainstJob(event, job) pricingSnapshot = loadOrCreateImmutablePricingSnapshot(job, validatedUsage)
settlementKey = composeKey( job.tenantId, job.jobId, "capture", pricingSnapshot.settlementVersion )
begin transaction currentJob = lockJobOrCompareAndSwap(job.jobId) assert transitionAllowed(currentJob.state, event.type)
settlement = insertSettlementWithUniqueKey( settlementKey, pricingSnapshot, pricingSnapshot.computedAmount )
insertImmutableLedgerDebit( settlementKey, currentJob.walletId, pricingSnapshot.computedAmount )
updateJobSettlementState(currentJob, "settled") updateMaterializedBalanceIfUsed(currentJob.walletId) markEventProcessed(eventKey, settlementKey) commit transaction
return settlement ```
The worker must handle a uniqueness conflict as an expected idempotency outcome:
``pseudo on UniqueConstraintConflict(settlementKey): rollback transaction existing = loadSettlement(settlementKey) verify existing.jobId == event.jobId verify existing.chargeType == "capture" return existing ``
Do not catch every database error and label it a duplicate. A uniqueness conflict on the intended settlement key is materially different from a timeout, unavailable database, serialization failure, or validation error. Ambiguous commit outcomes should be resolved by reading the settlement key before retrying the write.
Reconcile and Test Settlement for Asynchronous Inference Jobs
Idempotency and atomic persistence are the primary duplicate-charge controls. Reconciliation is the integrity backstop that detects missing, orphaned, or inconsistent records after operational failures.
A scheduled reconciliation process should compare:
- Terminal jobs against settlement records
- Settlements against immutable ledger entries
- Ledger entries against any materialized wallet balances
- Persisted completion events against worker outcomes
- Accepted usage and pricing snapshots against provider usage records
Useful exception classes include a completed job without settlement, a settlement without a ledger entry, a captured reservation that was also released, an event stuck in processing, and a materialized balance that does not match ledger-derived value. Repair operations should reuse existing settlement identities or create explicitly keyed compensating entries; they should not bypass normal uniqueness controls.
Test the design with failure injection rather than only happy-path webhook calls. Important scenarios include:
- The same event delivered repeatedly and concurrently
- Different completion event IDs for the same job
- A worker crash immediately before or after transaction commit
- Database timeout with an initially unknown commit outcome
- Delayed success, reordered failure, and cancellation races
- Partial usage, corrected usage, and pricing-version changes
- Queue redelivery and replay during incident recovery
- Ledger projection failure after the authoritative entry commits
For asynchronous AI inference, completion and final usage may become known only after submission. Latency-sensitive chat, batch enrichment, and agentic workflows can also require different serving and accounting policies. The settlement state machine should therefore reflect the workload's actual completion and usage semantics rather than assuming every request is billed identically.
Token Forge Cloud Private LLM Inference supports private deployment and serving-layer optimization for enterprise AI workloads, including approaches involving caching, routing, batching, quantization, and GPU scheduling. Token Forge Cloud Managed Model APIs provides an API-first path for teams validating model demand before moving predictable workloads toward private deployment.
The wallet, ledger, webhook, and settlement pattern in this guide belongs in the surrounding application and billing architecture; it should not be interpreted as a native payment or exactly-once webhook feature of those products. When connecting inference usage to financial settlement, keep serving telemetry, validated usage, pricing snapshots, and ledger records distinct—and define which system is authoritative for each.
The practical conclusion is straightforward: accept webhook duplication as normal, anchor settlement to the job and billing phase, enforce financial uniqueness durably, make related writes atomic, and reconcile the result. That combination substantially reduces duplicate-charge risk without relying on an unrealistic exactly-once delivery guarantee.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.