Transaction recovery

Transaction recovery is the second half of Transactions. It finishes a trade whose server died or whose saves failed, using the trade’s one durable journal decision. DataDriver runs it inside load, before a participant’s profile is handed to any code, so nothing in the game ever sees a record that is halfway through a trade. The defining rule is that the journal decides and recovery never guesses: a committed trade is applied, an aborted one is dropped, and a trade still marked Preparing is aborted through an atomic update of the same journal record the coordinator commits through, so exactly one of them decides.

The journal

A trade writes one journal record in the profiles’ own backend, under the key CKTxn_<GUID>:

Field	Meaning
Kind	"CKTransaction".
Token	A GUID. Must match the participant’s pointer, so a stale pointer can never settle against the wrong journal.
Id	The trade id.
State	"Preparing", "Committed" or "Aborted".
Entries	{ [participantKey] = { Data, Seen } }: each participant’s post-trade data, already encoded for storage, and their replay ids including this trade.
Settled	{ [participantKey] = true } once a participant’s pointer is cleared and acknowledged.
Meta.UserIds	Both owners’ UserIds. The journal holds both players’ data, so the DataStore backend tags it for GDPR erasure. Before 0.9.0 journals were written without them.

Each participant’s record points at the journal through Meta.PendingTransaction = { Key, Token }. The prepare saves write the pointer and the settle saves clear it.

How recovery runs

1. A load’s transform checks the lock first (a held lock still wins), then finds Meta.PendingTransaction and aborts without writing. Recovery needs yielding reads, which cannot run inside an UpdateAsync transform.
2. One atomic update on the journal fixes the decision. A missing journal or a token mismatch fails recovery. A journal still in Preparing is moved to Aborted; Committed and Aborted stay as they are.
3. One atomic update on the participant’s record settles it. If the pointer is already gone, there is nothing to do. If another server holds the record’s lock, recovery refuses, because a live acquisition won the race. The check is the load’s own ownership rule (lock age, and a holder whose presence stopped), so recovery settles exactly when the load that found the pointer could take the record. Otherwise a Committed decision writes the journal’s Data for this participant and its Seen ids into Meta.TransactionIds, an Aborted decision keeps the record’s pre-trade data, and the pointer is cleared.
4. The participant is marked settled in the journal, and when every participant is settled the journal is deleted through the backend’s remove. A crash can clear a peer’s pointer before the peer acknowledged, and that peer will never revisit the journal, so recovery reads each unsettled peer’s record through the store’s authoritative update path and marks it settled when its pointer is gone. A failed acknowledgement only leaves a journal to clean up later.
5. The load runs again, on a record with no pointer.

After a server crash, the joining server takes the lock through lock handoff once the crashed server’s presence stops, and recovery passes the same check, so a participant whose trade was interrupted loads about 15 s after the crash with the trade settled. Before 0.9.1 recovery checked lock age alone (younger than LockTtlSeconds meant owned), and that participant was kicked until the crashed server’s lock was 3 minutes old.

When recovery fails, the load fails with transaction recovery failed: <reason>:

Reason	Cause
malformed pending transaction	The pointer is not a { Key, Token } pair of strings.
transaction journal is missing or invalid	No journal at the key, or its kind, token, entries or state do not match. A backend error is reported instead when the read itself failed.
transaction participant is missing or invalid	The journal has no usable entry for this key.
transaction participant is still owned	Another server holds the record’s lock under the load’s ownership rule. The load had just passed that check, so another server took the lock in between.

This is not a lock error, so an attached DataDriver does not wait on it: the joining player is kicked with the load-failure message and can rejoin.

Outcomes by interruption point

A trade makes six durable writes. The spec suite interrupts after each one, throws the server away and loads both participants on a fresh one:

Interrupted after	Journal	Next load of each participant
1, journal created	Preparing	Neither record has a pointer: both load normally with pre-trade data.
2, A prepared	Preparing	A’s recovery aborts the journal and A keeps its pre-trade data. B loads normally.
3, B prepared	Preparing	The first recovery aborts; both keep pre-trade data.
4, decision	Committed	Both get the committed data and the trade id.
5, A settled	Committed	A is final. B’s recovery applies the committed data.
6, B settled	Committed	Both are final and load normally.

The item is conserved in every case: it ends on exactly one side.

Blocked participants

When a trade ends with a participant’s outcome still unsettled (PendingRecovery or ConcurrentMutationNeedsRecovery), that profile is marked TransactionBlocked. Since 0.9.0 only a participant whose settle save failed is blocked; the other one’s record is final and it keeps playing.

A blocked profile refuses ordinary saves and trades. Its release is metadata-only: it removes the lock and stamps ReleasedToken, and leaves the data and the pointer as stored, so the uncertain live copy never overwrites the record that points at the journal. An attached DataDriver ends the blocked player’s session when Commerce.TransactionEnded fires, with the kick message “Your trade is being finished. Please rejoin.”, and their next load, on any server, runs recovery. Before 0.9.0 a blocked participant kept playing on a profile that could no longer save.

Backend requirements

Recovery is only as safe as the backend’s update. The journal decision is a single point of contention between the coordinator and recovery, and it depends on a real compare-and-swap:

• The DataStore backend gets that from UpdateAsync.
• The Http backend requires the API to enforce If-Match on an ETag for existing records and If-None-Match: * for new ones, atomically. A record without an ETag is refused; missing preconditions fail closed.
• remove is optional. Without it (on Http, a DELETE route), settled journals stay in storage.

API reference

The module is internal: DataDriver calls it during load, and Transactions calls it while settling. The members exist for tests and custom tooling.

Member	Description
Recovery.resolve(driver, key, pending, heldByOther?) -> (ok, err?)	Decides the journal, settles participant key and acknowledges it. heldByOther(envelope) -> boolean is the caller’s ownership rule; DataDriver passes its load rule. Without it, recovery treats a lock younger than LockTtlSeconds as owned. heldByOther added in 0.9.1.
Recovery.acknowledge(driver, journalKey, token, participant)	Marks participant settled and deletes the journal once every participant is.
Recovery.update(driver, key, transform) -> (ok, value, err?)	backend:update through the driver’s retry ladder, protected against errors.
Recovery.valid(journal, token) -> boolean	Whether a record is a well-formed journal for token.
Recovery.cleanup(driver, journalKey)	Deletes the journal when the backend has remove.

Gotchas

Recovery keeps the journal’s copy

A committed recovery writes the journal’s post-trade data for the participant. Anything the server changed on a blocked participant after the trade was never saved, because a blocked profile does not save, and is not in that copy.

• A crash right after the journal is created, or after both settle saves and before the delete, leaves a journal that no record points at. It is never read again and stays in storage, still tagged with both owners’ UserIds.
• Both participants must load through a driver with the same configuration as the one that traded: the journal’s data is already encoded in that driver’s storage format.

Status

TransactionRecovery.spec.luau interrupts the protocol after each of its six durable writes and loads both participants on a fresh driver 300 s later, checking both balances, item conservation, cleared pointers, removed journals and committed replay ids. It also loses the commit response on purpose (PendingRecovery with an unknown decision, both blocked) and checks that release and reload apply the committed trade. The datasafety runner checks the kick of a blocked participant, recovery winning over a same-value write, and a server that crashes before the decision write: its participant loads on another server in under 16 s with the trade aborted. All of it uses storage doubles; interrupting real DataStore writes needs a published place.