What happens when a worker dies mid-run
A worker holds a 30 second lease and renews it every 10 seconds. If it dies, the lease expires, another worker claims the run, and the run resumes from the last checkpoint.
- Runtime
- Ops

A worker can die at any point in a run. A deploy restarts it, a node runs out of memory, a machine goes away. What the host does next decides whether the run is lost, started over, or picked up where it stopped.
In Linea it is picked up. This post walks through how, using what the execution worker actually does: a lease, a heartbeat, a checkpoint per node, and a claim rule that lets a second worker take over.
A run is leased, not owned
When a worker claims an execution, it takes a lease on it. The lease lasts 30 seconds. While the worker is alive it renews that lease on a heartbeat every 10 seconds, so a slow tick or two does not expire it.
A run can be claimed in only two situations. Either it is queued and nobody has started it, or it is marked running and its lease has already expired. A run with a live lease cannot be claimed by anyone else.
What gets written as the run goes
After each node completes, the worker writes the step and its checkpoint in a single transaction. The step holds the node's input and output. That write also checks that the worker still owns the lease. If another worker has reclaimed the run, the write is rejected with a lease lost error instead of landing.
That check matters. It means two workers can never both record progress for the same run. Whoever holds the lease is the only one whose checkpoints count.
When the lease expires
Suppose a worker finishes three nodes of a six node graph and then dies. Its heartbeat stops. Thirty seconds after the last renewal, the lease is stale and the run is claimable again.
A second worker claims it and does not start from the top. It loads the resume state from the checkpoints already written, records a resume event on the execution, and carries over the token and cost totals so far. If the run fails later, the usage it reports includes the work done before the crash, not only the work done by the second worker.
Then it runs the remaining nodes. The completed nodes are not executed again. The execution worker has a test that does exactly this: a six step workflow, three steps done, the first worker dies, and a second worker finishes the run without re-executing the completed steps.
What does run again
One step does run again: the one that was in flight. It never wrote a checkpoint, so as far as the host knows it did not finish. The resumed run calls it from the start.
That is the right behavior, and it is why a node that calls something outside the graph should be written with a repeat in mind. If a node sends a message or charges a card, the second attempt must not do it twice. Linea's connector layer handles this for its own side effects, and its tests cover a terminal write that arrives after a lease has expired. For your own HTTP calls the safe habit is the same: send a request that is fine to receive twice, or one that carries a key the receiver can use to deduplicate.
What the old worker does when it is not actually dead
Not every expired lease means a dead worker. A worker can stall, miss its heartbeats, and wake up after another worker has taken over. Linea handles that case too.
When the stalled worker's next heartbeat tries to renew the lease and finds it was reclaimed, it stops heartbeating and aborts the node handler that is in flight. The code comment gives the reason plainly: so a duplicate HTTP mutation or a billed AI completion does not run in parallel with whoever reclaimed the lease. And if the abort comes too late, the stale worker's checkpoint write is still refused, because it no longer owns the lease.
What to take from this
Two numbers and one habit.
- 30 seconds is how long a run can sit without a live worker before another one may take it.
- 10 seconds is how often a live worker proves it is there.
- One checkpoint per completed node is how little a crash can cost you: at most the step that was running.
The habit is to treat the step in flight as one that may run twice, and to write it so that is safe. Everything before it is kept.


