Reservation and settlement logic should be tested by defining business invariants first, modeling every allowed state transition, forcing deterministic concurrency, retrying operations with stable identities, injecting failures around each durable-effect boundary, disrupting event order and delivery, and triggering database failover during active work. After every scenario, reconcile authoritative records and external side effects. A test passes only when the declared invariants hold and the system reaches an acceptable recoverable state—not merely because an API returned a successful status.
The exact expected outcome depends on the system’s business rules, transaction isolation, locking or compare-and-set strategy, uniqueness constraints, database failover behavior, message-delivery semantics, and external-provider guarantees. Confirm those assumptions before selecting scenarios or declaring a result correct.
Begin With Invariants, Not API Responses
A reservation endpoint can return success before an event is published, a client can time out after a durable commit, and a settlement provider can complete an operation while the calling process loses the response. HTTP status codes are therefore observations, not complete proof of business correctness.
Start by defining properties that must remain true across normal operation, temporary disruption, retries, and recovery. These properties become test assertions and reconciliation rules.
Prevent over-reservation and duplicate settlement
Typical safety invariants include:
- Reserved value or inventory never exceeds the amount legally available under the applicable business rules.
- One logical settlement does not create multiple financial or inventory effects.
- A reservation cannot be confirmed, released, cancelled, expired, or settled through a prohibited transition.
- Repeating the same logical command does not silently create a new operation.
- Reusing an idempotency key with a materially different payload is rejected or handled according to an explicit conflict policy.
- Every durable command outcome can be linked to its operation identity, reservation, resulting records, emitted events, and external effects.
“No duplicate settlement” must be defined at the business-effect level. Deduplicating an API request does not automatically deduplicate a message consumer, ledger write, provider call, notification, or other downstream action. Each effect needs its own identity and protection strategy where duplication matters.
Conserve value or inventory across every state
Conservation checks should cover both stable and transitional states. For an inventory workflow, a simplified reconciliation equation might be:
available + actively reserved + consumed + otherwise accounted for = authoritative total
For a balance workflow, the categories may instead include available, held, released, settled, refunded, or adjusted amounts. The correct equation is domain-specific, but unexplained creation or disappearance of value should always be visible.
A practical invariant register can look like this:
| Invariant | Records involved | Permitted temporary inconsistency | Observation method | Failure condition |
|---|---|---|---|---|
| No over-reservation | Availability record, active reservations | Only what the declared isolation and recovery model permits | Query authoritative rows after concurrent commits | Active reservations exceed reservable capacity |
| No duplicate settlement | Settlement record, ledger entries, provider operation | A pending reconciliation state may be allowed | Compare logical operation IDs and external references | More than one business effect for one logical settlement |
| Conservation | Balance or inventory, holds, releases, settlements | Explicitly documented in-flight difference | Recompute totals from authoritative records | Unexplained gain, loss, or negative availability |
| Valid transitions | Reservation history and current state | None unless an intermediate state is modeled | Replay the transition history | A prohibited edge or terminal-state mutation occurs |
| Auditable linkage | Commands, state changes, events, external effects | Delayed event linkage may be allowed temporarily | Trace by operation and reservation IDs | An effect cannot be attributed to its initiating operation |
Temporary inconsistency must be bounded by a defined recovery mechanism. “Eventually consistent” is not a pass criterion unless the team specifies what may diverge, what repairs it, and how tests determine that convergence occurred.
Record the assumptions that determine expected behavior
Before running fault tests, document the architecture-specific semantics that affect the result:
- Which database operation makes a reservation or settlement durable?
- What isolation level is used, and which anomalies can it permit?
- Does contention use row locks, advisory locks, compare-and-set updates, version checks, uniqueness constraints, serialization, or another mechanism?
- What does the database client report if the connection disappears during commit?
- What happens to in-flight transactions during a leader change?
- Can reads after failover reach a lagging replica?
- Is message delivery at-most-once or at-least-once, and can events be reordered?
- How are external operations identified, queried, retried, and reconciled?
- Which states are terminal, and which can be repaired or compensated?
These assumptions determine whether a result is correct. For example, an ambiguous commit should not automatically be retried as a new settlement. The client may need to recover the original operation by its stable identity and inspect the authoritative state before taking further action.
Model Every Reservation, Hold, Release, and Settlement Transition
A state model turns broad reliability goals into executable scenarios. Use a running reservation lifecycle—creation, confirmation, expiration, cancellation, release, and settlement—and adapt it to the actual business rules.
Map active, terminal, expired, and compensating states
The model should identify the initiating command, permitted source state, durable destination state, emitted event, external effect, and prohibited alternatives. One illustrative model is:
| Command | Allowed source state | Expected destination | Possible event or effect | Transitions to reject or resolve explicitly |
|---|---|---|---|---|
| Create reservation | No existing reservation | Active | Reservation-created event | Duplicate creation with a new identity for the same exclusive unit |
| Confirm | Active | Confirmed | Confirmation event | Confirm after release or cancellation |
| Expire | Active and past expiry policy | Expired | Release of held capacity | Expire after settlement |
| Cancel | Active or confirmed, if permitted | Cancelled | Release or compensation | Cancel a terminal settlement without a modeled reversal |
| Release | Active or confirmed, if permitted | Released | Capacity becomes available | Release the same hold twice |
| Settle | Confirmed, or another declared state | Settled | Ledger or provider effect | Settle expired, cancelled, or already settled work |
| Compensate | Declared failed or completed state | Compensated | Reversal or adjustment | Compensation without an attributable original effect |
This is an example rather than a universal lifecycle. Some systems combine confirmation and settlement; others allow partial settlement, extension, split fulfillment, reversal, or manual review. Every supported branch should appear in the model, including terminal states and recovery transitions.
Link commands, state changes, events, and external effects
Give each logical operation a stable identity that survives timeouts, process restarts, redelivery, and failover. Record enough information to connect:
- The incoming command and normalized payload.
- Its idempotency or operation key.
- The reservation and state version it targeted.
- The transaction outcome, including an unknown or ambiguous result.
- Events created or published because of the command.
- Consumer processing attempts and deduplication decisions.
- External calls and provider-side operation references.
- Any repair, reconciliation, or compensation action.
This chain lets a test distinguish “the client did not receive a response” from “the operation did not happen.” It also makes duplicate effects and stranded intermediate records observable.
Force concurrency instead of hoping for a race
Timing-dependent sleeps are weak tools for reproducing transaction races. Prefer barriers, latches, controlled transaction pauses, scheduler hooks, or fault-injection proxies that place two operations at a precise boundary.
Useful contention scenarios include:
- Multiple reservation attempts competing for the final available unit.
- Two holds attempting to consume the same available balance.
- Confirm racing with expiration.
- Settlement racing with cancellation or release.
- Two workers processing the same reservation or event.
- An update using a stale state version racing with a current update.
- A retry arriving while the original request is still executing.
Pause both transactions after they have read the same starting state, then release them together. Repeat the test using the real isolation and conflict-handling configuration. Verify the database-level behavior—such as a uniqueness violation, failed version check, lock wait, serialization failure, or successful conditional update—and the resulting business state.
The test should also verify how conflicts are surfaced. A rejected stale update, for example, may be correct only if the caller subsequently reads the authoritative outcome rather than treating the conflict as permission to create another reservation.
Test idempotency and retry behavior as separate concerns
Idempotency requires more than attaching a key to a request. A retry-safe design normally needs a stable operation identity, durable storage of the outcome, clear payload-conflict rules, and controlled handling of downstream effects.
Exercise at least these cases:
- The same key and identical payload are sent sequentially.
- The same key and identical payload arrive concurrently.
- A key is reused with a conflicting amount, item, account, or reservation.
- The first request commits, but its response is lost.
- The process restarts after receiving the request but before returning the outcome.
- A retry occurs while the first attempt is still pending.
- A client retries after database failover produces an ambiguous commit result.
- A previously completed operation is retried after normal deduplication retention has elapsed.
Assert the final business effect, not just whether the response was replayed. Also define what callers should do when the original result remains unknown: query by operation identity, wait for reconciliation, or enter a review state rather than issuing a new uncorrelated command.
Inject failures around every durable-effect boundary
Create named fault points around each transition where the system can cross from “not done” to “possibly done.” Important boundaries include:
- Before the database transaction begins.
- During the transaction, before commit.
- During commit, when the client may lose certainty about the result.
- After commit but before the API response is delivered.
- Before and after an event or outbox record is created.
- Before and after event publication.
- Before and after consumer processing.
- After the business update but before consumer acknowledgment.
- Before an external-provider call.
- After the provider accepts the operation but before the response is stored.
- After a local record is updated but before the related external effect completes.
At each point, crash the relevant process, cut its network connection, terminate the worker, delay the response, or force the dependency to return a retriable error. Restart the component and observe whether replay, reconciliation, or compensation reaches the declared final state.
Where the architecture uses a transactional outbox, test crashes between the business update, outbox insertion, publication, and publication marking steps. Where it uses a consumer inbox or deduplication store, test crashes before and after recording receipt, applying the business update, and acknowledging the message. These patterns are not universal requirements; the test must match the system actually deployed.
Disorder delayed and redelivered events
A message broker’s delivery behavior and the application’s business-effect behavior are separate. Tests should deliberately produce:
- Duplicate delivery before and after the first attempt completes.
- Delayed delivery after a reservation has expired.
- A confirmation event arriving after cancellation.
- An older state event arriving after a newer one.
- Missing events that require replay or reconciliation.
- Consumer restart after applying an effect but before acknowledgment.
- Replayed historical events after data restoration or failover.
Assert how the consumer uses aggregate identity, operation identity, state version, sequence information, or current authoritative state. An obsolete event may be ignored, recorded as superseded, or routed for investigation according to the declared model. It should not silently move a terminal reservation backward unless that transition is explicitly supported.
Do not assume exactly-once delivery. Instead, identify the intended semantics:
- At-most-once: an operation or message is attempted no more than once, so loss may be possible under some failures.
- At-least-once: retries reduce loss risk, but duplicate delivery is expected and consumers must handle it.
- Effectively-once: duplicates may occur in transport or execution, but stable identity and deduplication are intended to produce one accepted business effect within a defined domain.
Tests should verify both transport observations and resulting business effects. A broker reporting one acknowledgment does not by itself prove that an external side effect occurred only once.
Trigger database failover during active work
Run failover tests while transactions are reading, locking, writing, and committing—not only while the system is idle. Trigger connection loss and leader change at controlled points, then test client reconnection and replay.
Cover scenarios such as:
- A transaction is known to have rolled back.
- A commit is known to have succeeded.
- The client receives no definitive commit result.
- The new leader does not immediately expose a recently committed record under the documented database semantics.
- A retry reaches a different node while the original operation is recovering.
- Workers reconnect and replay pending records after failover.
- A reconciliation process reads stale data and later sees the authoritative state.
For an ambiguous commit, preserve the original operation identity. The recovery path should determine whether that operation committed before deciding whether any further mutation is appropriate. Verify how the driver, connection pool, transaction library, and database actually report these conditions rather than assuming all connection errors mean rollback.
Replica lag also needs explicit treatment. If a write is followed by a read from another replica, test whether the application can tolerate stale state, route the read appropriately, or wait for the required consistency condition. The correct choice depends on the database and business workflow.
Use a repeatable scenario matrix
A scenario matrix makes coverage reviewable and exposes untested boundaries:
| Initial state | Concurrent operations | Failure point | Retry behavior | Expected durable state | Permitted temporary inconsistency | Recovery mechanism | Final invariant | Evidence collected |
|---|---|---|---|---|---|---|---|---|
| One unit available | Two create-reservation commands | Both paused after reading availability | Retry losing operation with its original identity | One active reservation | Pending responses may differ temporarily | Conflict handling and authoritative read | No over-reservation | Transaction trace, rows, state history |
| Confirmed reservation | Settlement command | Commit response lost | Retry same operation identity | One settlement outcome | Client may temporarily report unknown | Lookup and reconciliation | No duplicate business effect | Settlement record, ledger, provider reference |
| Active reservation | Confirm and expire | Event publication delayed | Redeliver events out of order | State allowed by declared precedence rule | Event stream may lag database | Replay or reconciliation | Valid terminal transition | State versions, event log, reconciliation output |
| Pending settlement | Database leader change | During commit | Reconnect and recover original operation | Committed, rolled back, or explicitly unresolved | Unknown status during recovery | Operation lookup or review workflow | No uncorrelated second settlement | Driver errors, database records, operation history |
Run deterministic cases first, then add concurrency stress, property-based testing, model-based command generation, and repeated randomized schedules. Random exploration is most useful when every failure includes the seed, operation history, fault schedule, and state snapshot needed for reproduction.
Reconcile records and define business-level pass criteria
After each scenario, compare all relevant sources of truth:
- Reservation records and transition history.
- Available balances or inventory.
- Holds, releases, cancellations, and expirations.
- Settlement and ledger records where applicable.
- Pending, published, and acknowledged events.
- Outbox or inbox records where those patterns are used.
- External-provider references and side effects.
- Repair, compensation, and manual-review records.
Track correctness-oriented outcomes such as invariant violations, duplicate effects, stranded reservations, unresolved ambiguous operations, unprocessed outbox records, reconciliation differences, and time to reach the expected recoverable state. Thresholds should come from the system’s business and operational objectives rather than a universal benchmark.
A passing run demonstrates that the tested schedule preserved the declared invariants or entered a permitted temporary state that recovered through the documented mechanism. It does not prove that every possible failure schedule has been eliminated. Keep the suite repeatable, expand it with incidents and design changes, and rerun it against production-equivalent database, broker, client, and failover configurations.
Next Step
This guidance is intended for teams building and validating transactional systems. It does not describe reservation or settlement functionality in Token Forge Cloud products.
Contact Token Forge Cloud to discuss API access, private deployment, and LLM inference cost control.