An approval pauses the run, not a workerA paused run releases its lease and holds no worker. It waits until someone decides, then the approval node returns the decision and the run continues from that step.Why every execution is pinned to one versionAn execution records its workflow version once, when it starts, and never follows a live pointer. That is why resume and replay still mean something weeks after a graph has changed.Memory has a subject, a namespace, and an expiryEvery memory fact lives under a key, a subject and a namespace, and it always expires: 90 days by default, one year at most. Here is how writes, reads and sharing work.What happens when a worker dies mid-runA worker holds a 30 second lease and renews it every 10 seconds. If it dies, the lease expires, another worker claims the run, and the run resumes from the last checkpoint.Which Linea client to use, and which key goes whereThe TypeScript SDK has a server client, an environment client for production starts, an end-user client for browsers, and a webhook verifier. Each one takes a different credential.Linea Labs is not Consensys LineaLinea Labs builds Linea, an open-source AI agent runtime at getlinea.app. It is not the Consensys Linea blockchain, and not the Linea Sketch iPad app.What is an AI agent runtime?A framework gives you the agent loop. A runtime runs it in production: durable steps, checkpoints, approvals, crash recovery, and replaying one failed step.Why replay needs the checkpointYou cannot re-run step four without the exact input it saw. That input only exists if the host executed the run and kept the checkpoint.One graph, not two productsThe canvas, the workflow JSON, and the node registry describe one thing. A second editor is fine. A second graph is a second product.The half frameworks do not shipFrameworks solve the loop. You still host it, retry it, store its memory, and keep its checkpoints. That second half is the product.Memory is not a side questFacts have to survive the run. If you stand up the store yourself, you are still doing the ops half. We think the graph should ask for memory and the host should keep it.Sandbox the code nodeA code node is an escape hatch. It still has to run somewhere that cannot take down the host. Linea does not ship one yet, and this is why.Triggers are not the productStarting a graph is a solved problem. The product is what has to stay up after it starts: the worker, the checkpoint, the wait, and the memory.
Toggle menu

What happens when a worker dies mid-run

A worker holds a 30 second lease and renews it every 10 seconds. If it dies, the lease expires, another worker claims the run, and the run resumes from the last checkpoint.

Linea Labs
  • Runtime
  • Ops

A worker can die at any point in a run. A deploy restarts it, a node runs out of memory, a machine goes away. What the host does next decides whether the run is lost, started over, or picked up where it stopped.

In Linea it is picked up. This post walks through how, using what the execution worker actually does: a lease, a heartbeat, a checkpoint per node, and a claim rule that lets a second worker take over.

A run is leased, not owned

When a worker claims an execution, it takes a lease on it. The lease lasts 30 seconds. While the worker is alive it renews that lease on a heartbeat every 10 seconds, so a slow tick or two does not expire it.

A run can be claimed in only two situations. Either it is queued and nobody has started it, or it is marked running and its lease has already expired. A run with a live lease cannot be claimed by anyone else.

A worker claims a run and renews its lease. When it dies, the lease expires and a second worker resumes after the last checkpoint.

What gets written as the run goes

After each node completes, the worker writes the step and its checkpoint in a single transaction. The step holds the node's input and output. That write also checks that the worker still owns the lease. If another worker has reclaimed the run, the write is rejected with a lease lost error instead of landing.

That check matters. It means two workers can never both record progress for the same run. Whoever holds the lease is the only one whose checkpoints count.

When the lease expires

Suppose a worker finishes three nodes of a six node graph and then dies. Its heartbeat stops. Thirty seconds after the last renewal, the lease is stale and the run is claimable again.

A second worker claims it and does not start from the top. It loads the resume state from the checkpoints already written, records a resume event on the execution, and carries over the token and cost totals so far. If the run fails later, the usage it reports includes the work done before the crash, not only the work done by the second worker.

Then it runs the remaining nodes. The completed nodes are not executed again. The execution worker has a test that does exactly this: a six step workflow, three steps done, the first worker dies, and a second worker finishes the run without re-executing the completed steps.

What does run again

One step does run again: the one that was in flight. It never wrote a checkpoint, so as far as the host knows it did not finish. The resumed run calls it from the start.

That is the right behavior, and it is why a node that calls something outside the graph should be written with a repeat in mind. If a node sends a message or charges a card, the second attempt must not do it twice. Linea's connector layer handles this for its own side effects, and its tests cover a terminal write that arrives after a lease has expired. For your own HTTP calls the safe habit is the same: send a request that is fine to receive twice, or one that carries a key the receiver can use to deduplicate.

What the old worker does when it is not actually dead

Not every expired lease means a dead worker. A worker can stall, miss its heartbeats, and wake up after another worker has taken over. Linea handles that case too.

When the stalled worker's next heartbeat tries to renew the lease and finds it was reclaimed, it stops heartbeating and aborts the node handler that is in flight. The code comment gives the reason plainly: so a duplicate HTTP mutation or a billed AI completion does not run in parallel with whoever reclaimed the lease. And if the abort comes too late, the stale worker's checkpoint write is still refused, because it no longer owns the lease.

What to take from this

Two numbers and one habit.

  • 30 seconds is how long a run can sit without a live worker before another one may take it.
  • 10 seconds is how often a live worker proves it is there.
  • One checkpoint per completed node is how little a crash can cost you: at most the step that was running.

The habit is to treat the step in flight as one that may run twice, and to write it so that is safe. Everything before it is kept.

The Linea newsletter

What we learn from every run.

New releases and field notes on running agents in production.

One email when something ships. Unsubscribe anytime.