Durable execution is the ability for a workflow to survive time, retries, failures, restarts, and long-running operations without losing state. A durably executed workflow that crashes at step 7 does not start over at step 1. It resumes at step 7, with steps 1 through 6 intact.
It is a property, not a library. A workflow has it when every completed step is persisted before the next one begins, so recovery can tell the difference between work already done and work still outstanding. Agents need this for the same reason payment systems do: the steps have real effects and repeating them is not free. An agent twenty minutes and eleven tool calls into a task should not lose all of it because a deploy rolled the pod.
Key takeaways
- Durable execution is the guarantee that a workflow resumes without losing completed work. Checkpointing is one technique that provides it, not a synonym for it.
- The mechanism is a persisted record per completed step, plus recovery logic that skips any step with a recorded result instead of running it again.
- Replay-based recovery requires deterministic control flow, so nondeterministic values, model output included, must be recorded as step results rather than regenerated.
- Side effects are the hard part. The crash window between making an external call and recording it is why every effectful step needs an idempotency key.
- Exactly-once side effects are not achievable in general. What you can build is at-least-once execution plus deduplication, which behaves like exactly-once from outside.
How it works
Two implementations dominate, and they differ in what gets persisted.
Step-record replay. The runtime writes a durable record for each completed step: its identity, its inputs, and the value it returned. After a crash the workflow function runs again from the top, and when execution reaches a step that already has a record, the runtime returns the recorded value instead of doing the work. Replay races through completed steps and stops at the first one with no record, which becomes the resume point.
Snapshot restore. The runtime periodically writes the full state of the run, then loads the most recent snapshot after a crash and continues forward. This is checkpointing. It avoids re-running the workflow function, but loses whatever happened between the last snapshot and the crash.
Both give resumability. Replay gives finer granularity and a complete history at the cost of stricter constraints on your code. Snapshots are simpler and coarser. Many systems use both.
The determinism constraint
Replay only works if re-running the workflow function takes the same path it took the first time. Anything that can return a different value on the second pass has to be recorded, not recomputed. In practice the workflow body cannot call random, read the current time, generate a UUID, or perform I/O outside a recorded step.
For agents, the largest source of nondeterminism is the model. The same prompt can produce a different plan on replay, sending the resumed run down a branch the original never took. The fix is the same as for any nondeterministic value: treat the model call as a step and persist its output, so replay returns the recorded completion. Worth saying plainly, this is a real constraint. Durable workflows put rules on how you write code, and those rules are easy to violate by accident.
Why side effects need idempotency keys
Persisting a step is a separate operation from performing it, and the gap between them is where correctness is won or lost.
The sequence for an effectful step is: call the external system, get a response, write the step record. A crash after the call but before the write leaves the effect done and unrecorded. Recovery sees no record and runs the step again, so the refund is issued twice. Reordering does not help, because writing the record first creates the opposite failure, a step marked done that never happened. Two systems cannot commit atomically without a protocol both participate in, and most third-party APIs do not participate.
The practical answer is an idempotency key. Derive a stable key from the run and step identifiers, send it with the request, and let the external system recognize the retry and return the original result rather than acting twice. The key must be deterministic, since a fresh random key on retry defeats the mechanism.
When the external system offers no idempotency support, the options get worse and you should pick deliberately:
- Write an intent record before the call, then reconcile. Recovery finds the intent, queries the external system to see whether the action landed, and completes or retries. Works only when the effect is queryable.
- Make the step naturally idempotent. Writing to a known key or setting a field to a fixed value can be repeated safely. Appending cannot.
- Mark the step non-retryable. On recovery, halt and escalate rather than guess.
Test it by killing the process between the call and the record, then resuming, and confirm the external system shows one effect rather than two. This is one of the failure classes conventional tests miss, because nothing in the happy path exercises it.
What it costs
Durability is not free. Every step boundary is a durable write, adding latency and storage. Step inputs and outputs must be serializable, so open connections, file handles, and closures cannot live in workflow state. Replay also makes local debugging less intuitive, since a stack trace may come from a replayed execution rather than a fresh one.
The judgment call is scope. A single-turn chat completion does not need durable execution. A run that pauses for approval, spans a deploy, or spends real money at step 9 does. Many teams apply it at the workflow boundary and leave the inner tool-use loop of the agent harness non-durable, keeping the fast path fast while the expensive boundaries stay recoverable.
One operational detail catches people out: resumed runs fragment telemetry. If the resumed execution starts a new trace, one logical run appears as several unrelated ones and the cost and step counts stop adding up. Carrying the run identifier across resumption keeps agent observability coherent for workflows that restart.
FAQ
Is durable execution the same as checkpointing?
No. Durable execution is the guarantee that a workflow survives failure and resumes without redoing completed work. Checkpointing provides that guarantee by periodically saving state you can restore from. The other common mechanism is a persisted record per step combined with replay. When someone says a system is durable, the follow-up question is which mechanism it uses, because the tradeoffs differ.
Does replay call the model again?
It should not. If the model call is a recorded step, replay returns the stored completion and the run follows its original path. If the call is not recorded, replay re-invokes the model, pays for it again, and may get an answer that diverges from the branch the run already took. That divergence is a confusing bug, because the resumed run looks fine while doing something the original never did.
Do I need a workflow engine to get durable execution?
No, but you need what one provides: durable step records, deterministic recovery, timers, and retry with backoff. Small systems get a workable version from a state table and careful step boundaries. Hand-rolled versions usually break on multi-day waits and concurrent branches, which are harder than they look.
Can durable execution undo an action that already happened?
No, and this is the limit worth being clear about. Restoring state rewinds what the agent knows, not what it did. An email that was sent stays sent. Undoing a step takes a compensating action that reverses it, and some actions have no compensation available.
