I Killed an Agent 120 Times to See What Durable Execution Actually Buys You

Durable execution has become the default answer to “what happens when my agent crashes halfway through?” I wanted to know what that answer actually covers, so I wrote the smallest possible version and tried to break it.

The test is a scripted incident-response agent: five model calls, five tool calls, three of which write to the outside world (open a ticket, page on-call, post a status update). I killed it with a real SIGKILL at every step boundary, in three different windows per step, under four persistence policies. That’s 30 kill points per policy and 120 crashes in total.

Policy	Kills causing duplicate writes (of 30)	Worst duplicates in one run	Mean extra tokens	Worst extra tokens
No persistence	20	3	17,967	37,000 (+100%)
Checkpoint per turn	6	1	6,167	10,400 (+28%)
Step journal	3	1	1,233	10,400 (+28%)
Journal + idempotency keys	0	0	1,233	10,400 (+28%)

The token figures are from a scripted cost model, with context growing each turn and a clean run costing 37,000 tokens. The duplicate counts come from real process deaths against a separate SQLite database standing in for the outside world.

The journal alone doesn’t reach zero. To see why, look at the three windows.

Three windows, one of which you can’t close

Every step has the same shape, and the crash can land in any part of it:

         "before"            "effect"              "recorded"
  ──────────┬───── do the thing ──┬───── write to journal ──┬──────
            ▲                     ▲                         ▲
       nothing happened     it happened,            it happened,
       safe to redo         nobody knows            and we know

Crashes in the first and third windows are easy. The middle one is the problem: the ticket exists, and the journal doesn’t know. On restart there’s no record, so the step runs again and you have two tickets. Swap the order and record first, and a crash leaves a recorded step that never happened, which is worse.

No amount of cleverness on the agent’s side fixes this, because it’s the classic two-generals shape: two systems, no shared transaction. The fix has to come from the other side. Send a key that’s stable across retries, and let the receiving system deduplicate on it. It’s the same idempotency key you’d use for a payment API.

Temporal’s documentation is unusually direct about this. It recommends that Activities be idempotent, and says that even when completion is observed exactly once, the Activity “may be executed multiple times.” It suggests building the key from the Workflow Run ID and the Activity ID. My test does the same thing with the run ID and step number.

So “durable execution” is two separate guarantees that are easy to conflate. It gets you progress that survives a crash. No duplicate effects comes from the idempotency key, and only for systems that honour one.

The whole mechanism

It’s small enough to read in full. This is the core of the journal:

def step(self, name, fn, *args):
    seq, self.seq = self.seq, self.seq + 1
    digest = hashlib.sha256(json.dumps([name, args], sort_keys=True).encode()).hexdigest()
    row = self.db.execute(
        "SELECT name, input_hash, output FROM steps WHERE run_id=? AND seq=?",
        (self.run_id, seq),
    ).fetchone()
    if row:
        if (row[0], row[1]) != (name, digest):
            raise NonDeterminism(f"step {seq}: history has {row[0]!r}, code asked for {name!r}")
        return json.loads(row[2])

    key = f"{self.run_id}:{seq}"
    output = fn(*args, key)
    self.db.execute("INSERT INTO steps VALUES (?,?,?,?,?)",
                    (self.run_id, seq, name, digest, json.dumps(output)))
    return output

After a crash, the process just runs the agent again from the top. Recorded steps return instantly, and execution goes live at the first gap. You don’t write recovery code at all. That’s the appeal.

What the other rows tell you

Checkpointing per turn is the trap. It’s what most people build first: save the conversation after each turn. It looks durable and cut the damage a lot, but a crash inside a turn still loses both the model call and the tool call that followed it, so the tool runs again. It produced twice the duplicates of a step journal.

You pay for the call you lost. Even with keys, a crash after the model responds but before the journal write loses that response. Those are the 1,233 mean extra tokens, and the worst case, 10,400, is the last and largest turn. Idempotency keys protect the outside world, and they don’t get your tokens back. Long-context agents make this window more expensive, not less.

The journal is nearly free. A durable SQLite commit took 3.02 ms at the median and 4.78 ms at p99 with fullfsync on, on an Apple M3 laptop. Without that pragma, macOS reported 0.04 ms, because plain fsync there stops at the drive’s cache. If your durability benchmark looks too good, check that first.

The hazard you inherit

Replay has a cost that doesn’t show up in any of those columns: the code has to still match the history. If you deploy a change that reorders two steps while runs are in flight, those runs replay against a journal that describes different code.

My version refuses, raising NonDeterminism when a step’s name or inputs don’t match what was recorded. A real system also needs a way through: workflow versioning, or draining old runs before deploying. Agents make this more awkward than ordinary workflows, because prompts and tool lists change often and the model’s next decision depends on them. Treat a prompt change as a code change for anything in flight.

Build it or buy it

This is about 80 lines because it’s one process, one writer and one machine. It omits everything that makes the real systems worth paying for:

• Multiple workers and task routing
• Durable timers, and waiting days for a human approval
• Compensation when step 7 of 9 fails for good
• Workflow versioning across deploys
• Visibility into what’s running and why it’s stuck

If you need those, use Temporal, Restate, DBOS or similar rather than growing this into one. If you have a single-process agent that needs to survive a restart, a table with five columns will do it, and you’ll understand exactly what you’ve got.

Either way, the question for your next design review is the same: for each tool that writes, what happens if it runs twice? If the honest answer is “two tickets,” no framework you add will change that.

Limits

SIGKILL tests process death, not power loss or disk failure. The agent, model and tools are scripted, so the decisions are deterministic and the token numbers are from a cost model. Real model output varies between runs, which is exactly why you record it instead of regenerating it.

Everything, including the crash matrix you can run in about 15 seconds: ajinb/durable-agent-loop.