Checkpointing is the practice of saving an agent’s state at defined points so the workflow can resume, audit, retry, or roll back after an interruption. A checkpoint might store plan state, completed tool calls, retrieved documents, intermediate outputs, and policy approvals. The test of a good one: if the process died now, could a fresh worker load this record and continue as if nothing happened.
Checkpointing is a reliability primitive for long-running agents, and it is one technique for providing durable execution rather than a separate idea. Durable execution is the guarantee that a workflow survives failure and resumes without losing completed work. Checkpointing delivers it by snapshotting state you can restore from. The other common implementation persists a record per completed step and replays, skipping finished steps instead of restoring a snapshot.
Key takeaways
- A checkpoint is only useful if it is sufficient. If restoring it still needs something that lived in process memory, the resume fails in a confusing way.
- Version stamp every checkpoint with the prompt, tool schema, and code version, so resuming into changed code is detected rather than discovered later.
- Frequency trades write cost against how much work you are willing to redo. Semantic boundaries beat timers.
- Restoring state does not undo side effects. The email is still sent. Resumption safety comes from idempotency keys and compensating actions.
- A checkpoint is also a debugging fixture: a state you can fork, replay, and test a change against without reproducing the whole run.
What goes into a checkpoint
The contents follow from what a new worker needs in order to continue:
- Position. The current step, and which planned steps are complete, pending, or failed.
- Completed tool calls. Arguments and results, so a resumed run recognizes finished work.
- Accumulated context. The message history, or on long runs the compacted summary plus recent turns verbatim.
- External identifiers. Ticket numbers, order IDs, created file paths. These cannot be regenerated, and losing them is unrecoverable in a way that losing a summary is not.
- Control counters. Steps, tokens and dollars spent, consecutive errors, deadline. A resumed run that resets its budget counter has no budget.
- Pending obligations and version stamps. The approval the run is waiting on, plus the prompt, tool schema, and code versions it started under.
Store retrieved documents as references rather than content. Nothing unserializable belongs in the record, and neither do secrets, since a checkpoint long outlives the run.
Where to place them
Checkpoint at boundaries that mean something rather than at arbitrary intervals.
After each completed step is the common default. It bounds lost work to one step and pairs with a persisted step record.
Before an expensive or irreversible action. If the next call spends money or changes an external system, write the intent before making it. That record is what recovery reads to determine whether the action may already have landed.
At human approval gates. A run pausing for review is idle for an unpredictable stretch, so persist it and release the process.
Per-step checkpointing is a convention, not a rule, and the right default when steps are expensive relative to a write. Where steps are cheap and frequent, the writes become the dominant cost and coarser placement is defensible. Frequency comes from two numbers you can measure: what a write costs, and what redoing the work since the last one costs.
The cost side
Every checkpoint is a serialization pass and a durable write. Synchronous writes add latency to every step; asynchronous writes remove that latency and reintroduce a window where a crash loses recent state. Size is the second cost, since snapshotting the full message history each step writes a larger copy every time, which turns quadratic unless you store deltas or keep bulky artifacts outside the record. Retention is the third, because thousands of long runs with per-step checkpoints is a storage bill and a compliance question. Choosing where the boundaries go is ordinary agent harness engineering, the same judgment as setting transaction boundaries in any distributed system, except that agent steps are expensive enough to push the economics toward checkpointing more often.
The part checkpointing does not solve
Restoring a checkpoint restores what the agent knows. It does not restore the world. If the agent sent a message, created a ticket, moved money, or deleted a file, that happened. Rewinding to a state where the agent believes it has not yet acted is worse than useless, because now it acts again, which is why “just resume from the checkpoint” is incomplete advice.
The dangerous window is narrow and unavoidable: a step performs its external effect and the process dies before the result is recorded, so recovery sees an unfinished step and retries it. Three defenses are worth having.
Idempotency keys. Derive a stable key from the run and step identifiers and send it with the request, so the external system recognizes the retry and returns the original result instead of acting twice. The key must be deterministic across retries, which is why it comes from the run rather than being generated fresh.
Intent records and reconciliation. On recovery, read the intent written before the call, query the external system to see whether the action landed, then complete or retry. This depends on the effect being queryable.
Compensating actions. When an effect cannot be deduplicated, the only correction is a second action reversing the first. Some effects have no compensation, and for those the honest design is a human approval gate plus a non-retryable step, so recovery escalates instead of guessing.
Resuming into a system whose prompts or tool schemas have changed is a subtler hazard, producing a run that is half old behavior and half new. Version stamps exist so recovery detects the mismatch and refuses.
The benefit nobody plans for
A checkpoint is a reproducible starting state, useful well beyond crash recovery. Fork a run from the checkpoint just before it went wrong, change the prompt or a tool description, and re-run only the tail. Against reproducing a nine-hour run from scratch, that is a large difference in iteration speed. With per-step traces, it is what makes evaluating and optimizing agent skills from execution records practical rather than a matter of reasoning about final outputs, and why OpenTelemetry span and trace structure matters for replay.
FAQ
How often should an agent checkpoint?
Often enough that the work you would redo costs less than the writes you would save. Checkpointing after each completed step is the common convention and a reasonable default, especially when a step involves a model call or an external tool. If steps are cheap and rapid, coarser boundaries are the better trade. Measure both sides rather than adopting someone else’s number.
What is the difference between checkpointing and memory?
A checkpoint is an execution snapshot meant to be restored, scoped to one run, written for a machine that needs to continue work. Memory is knowledge kept across runs, curated rather than snapshotted. You would never load a checkpoint to answer a question.
Can you roll back a checkpoint after a tool call?
You can restore the state that existed before the call. You cannot undo the call. If the action needs reversing, that takes a compensating action. If it must not repeat on resume, that takes an idempotency key. Checkpointing alone provides neither.
What is the difference between checkpointing and durable execution?
Durable execution is the property you want: the workflow survives failure and resumes without losing completed work. Checkpointing is one implementation, based on saving and restoring state. Step-record replay is the other, re-running the workflow and short-circuiting any step with a recorded result. Both deliver resumability, with different granularity and different constraints on your code.
