Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is durable execution? Crash-proof workflows, explained

Durable execution records every step of a workflow so it can resume after a crash without redoing work. How replay works, with a runnable Python demo.

What is durable execution? Crash-proof workflows, explained

Durable execution is a way of running code so that a crash, deploy or lost machine does not lose its progress: the runtime records the result of every step in a persistent log, and when the process comes back it replays the code against that log, skipping work that already finished and continuing from the first step that did not. You write an ordinary function; the platform makes it behave as if it never stopped. Temporal, Restate and DBOS are the best known engines that do this.

Restate’s docs put it in one line: durable execution is “the ability to make any code execution reliable and fault-tolerant without changing how you write business logic” (Restate). The part after “without” is the selling point. The alternative is the state machine you hand-build out of queues, cron jobs, status columns and retry tables, which every team writes once and regrets.

How does durable execution work?

Every engine I’ve read splits your code into two kinds of thing.

  1. The workflow: the orchestration logic. It decides what happens next. It must be deterministic, meaning that given the same inputs and the same recorded results, it makes the same decisions.
  2. Steps (Temporal calls them Activities): anything with a side effect or an unpredictable result. Charging a card, calling an API, querying a database, calling an LLM.

When a step finishes, the engine writes its result to durable storage before the workflow moves on. Temporal calls that log the Event History, “a complete, ordered log of everything that has already happened in a Workflow” (Temporal docs). Restate calls it a journal. DBOS writes checkpoints into Postgres: “While your application runs, DBOS checkpoints those workflows and steps to a Postgres database” (DBOS architecture).

Recovery is replay. A new worker picks up the interrupted workflow and runs the function again from the top. Each time the code reaches a step, the engine checks the log. If the step already has a recorded result, it returns that result without running the side effect. DBOS states the rule plainly: “Once a step completes and is checkpointed, it is never re-executed.” When the code reaches the first step with no record, it runs for real, and from there the workflow is live again. Restate describes the same thing as replaying the journal, “skipping completed steps and resuming from exactly where it left off.”

Temporal does not snapshot memory. Its docs describe the goal as bringing the workflow “back to the exact same state it was in before the pause occurred” by re-running the code against history. That is why determinism matters: replay only reconstructs the right state if the code “make[s] the same decisions when given the same history.”

What does durable execution look like in code?

Here is the mechanism without any framework, in Python 3 standard library. The event history is a JSON-lines file. Every side effect goes through step(), which either replays a recorded result or runs the function and appends the result to the log. The first attempt crashes after reserving stock; the second attempt is a fresh call to the same function.

# durable.py: durable execution in 40 lines. An event log, a crash mid-workflow,
# and a replay that skips completed steps. Python 3 stdlib only.
import json, os, tempfile

LOG = os.path.join(tempfile.gettempdir(), "order-42.jsonl")  # the "event history"
calls = {"charge_card": 0, "reserve_stock": 0, "send_email": 0}
CRASH_AFTER = "reserve_stock"  # simulate the worker dying after this step

class Crash(Exception): pass

def run_workflow(order_id, crash):
    history = [json.loads(l) for l in open(LOG)] if os.path.exists(LOG) else []
    cursor = iter(history)

    def step(name, fn):
        event = next(cursor, None)
        if event is not None:                      # replay: result already recorded
            assert event["step"] == name, f"non-determinism: expected {event['step']}, got {name}"
            print(f"  replay  {name:<14} -> {event['result']}")
            return event["result"]
        calls[name] += 1                           # first time: run the side effect
        result = fn()
        with open(LOG, "a") as f:                  # record before moving on
            f.write(json.dumps({"step": name, "result": result}) + "\n")
        print(f"  execute {name:<14} -> {result}")
        if crash and name == CRASH_AFTER:
            raise Crash("worker killed")
        return result

    # The workflow: ordinary code. Every side effect goes through step().
    payment = step("charge_card", lambda: f"pay_{order_id}_ok")
    stock = step("reserve_stock", lambda: "sku-7:reserved")
    step("send_email", lambda: f"receipt for {payment}, {stock}")
    return "done"

if os.path.exists(LOG): os.remove(LOG)
print("attempt 1")
try:
    run_workflow(42, crash=True)
except Crash as e:
    print(f"  !! {e}")
print("attempt 2 (new worker, same history)")
print("  result:", run_workflow(42, crash=False))
print("side effects executed:", calls)
assert calls == {"charge_card": 1, "reserve_stock": 1, "send_email": 1}

Output (Python 3.13.11):

attempt 1
  execute charge_card    -> pay_42_ok
  execute reserve_stock  -> sku-7:reserved
  !! worker killed
attempt 2 (new worker, same history)
  replay  charge_card    -> pay_42_ok
  replay  reserve_stock  -> sku-7:reserved
  execute send_email     -> receipt for pay_42_ok, sku-7:reserved
  result: done
side effects executed: {'charge_card': 1, 'reserve_stock': 1, 'send_email': 1}

The card was charged once. The local variables payment and stock on the second attempt hold the same values as on the first, because they came out of the log. That is the whole trick: state is rebuilt from recorded results, so it doesn’t matter which machine runs the function or how many times it starts.

The assert inside step() is a tiny version of what Temporal does for real. On replay, the commands the code emits “are compared with the existing Event History,” and a mismatch fails the execution with a non-determinism error (Temporal workflow definition). Reorder two steps in the code while a workflow is in flight and replay no longer lines up.

What the toy leaves out: a server that owns the log and hands work to workers, timers that survive restarts, retries with backoff, signals from outside, and the window between a side effect finishing and its result being written. Crash in that window and the step runs again on replay. That is why Temporal recommends that an Activity “be idempotent, so retries can be processed without duplicate side effects” (Temporal Activities). Durable execution moves the problem; it does not give you exactly-once side effects for free.

Where did durable execution come from?

The lineage runs through three companies. Temporal’s about page says its founders launched the first public version of Amazon’s Simple Workflow Service, that Samar Abbas “created the Azure Durable Task Framework at Microsoft,” and that in 2015 the two reunited at Uber and co-created Cadence, “a transformative workflow engine that powered many of Uber’s services and applications.” Temporal is the company and open-source project that came out of Cadence. Newer engines took the same idea in other directions: Restate as a separate runtime with a journal, DBOS as a library that stores checkpoints in the Postgres you already run.

Why does durable execution matter for AI agents?

An AI agent is a loop of model calls and tool calls that can run for minutes or hours. Without durability, a crash on step 40 means paying for steps 1 to 39 again, and repeating any tool call that changed something in the world. With durability, each model call and each tool call is a step, and a restart replays them from the log.

Temporal and OpenAI shipped an integration for exactly this. Temporal’s announcement (July 30, 2025) says “every agent invocation is executed through a Temporal Activity,” and a note on the post says the Python integration became generally available on March 23, 2026. The same post shows a hand-written loop with the model call and the tool call as separate Activities, which is the shape I’d reach for when I want finer recovery than one Activity per agent run.

There is a fit problem worth seeing early. Model calls are the most non-deterministic thing you can put in a program. That’s fine, because they live in steps and their outputs get recorded; replay reads the old answer instead of asking the model again. What breaks determinism is putting branching logic on things the log didn’t record: the wall clock, a random number, a config value read at runtime. Temporal’s docs say a workflow “can not have inline logic that branches” on local time or a random number, and to use the SDK’s replay-safe APIs instead.

The other link worth drawing is to human-in-the-loop designs. An approval step that waits two days is trivial in a durable workflow (a signal plus a timer) and miserable to build on a request-response server.

Where does durable execution break?

History grows. Every step adds events, and long-running workflows hit limits. Temporal’s limits page says the Event History “is limited to 51,200 Events or 50 MB and will warn you after 10,240 Events or 10 MB.” An agent loop with hundreds of tool calls, each returning a large payload, can get there. The usual move is to start a fresh execution that carries over only the state it needs, and to keep big payloads in object storage with a pointer in the history.

Code changes under running workflows. If a workflow started on Monday and you deploy new code on Tuesday, replay runs Tuesday’s code against Monday’s history. Temporal offers patching and Worker Versioning for this and calls Worker Versioning “the recommended way to handle versioning.” Either way, deploys now need a plan for in-flight executions, which is new work for teams used to stateless services.

Side effects at the edges. As above, at-least-once is the honest guarantee for a step. Payment APIs with idempotency keys make this easy; email and SMS providers often don’t.

The log is data. Every input and output of every step is persisted. From the privacy seat I had at Walmart, this is the line I’d circle: if a step returns a customer’s address or an LLM prompt contains personal data, it now lives in your workflow history with its own retention. Decide what goes into payloads, encrypt them, and know how long the history is kept before a regulator asks.

Durable execution vs a job queue: what’s the difference?

A queue with retries gives you durability per job. If one job is one step, that’s enough, and I would not add Temporal for it. The gap shows up when a process has several steps with dependencies between them: charge, then reserve, then ship, then wait three days, then email. On a queue, you store the progress yourself, write a status column, chain jobs, and handle partial failure at every link. Durable execution keeps that progress for you and lets the process read as one function. It’s the same split as workflows vs agents at a lower level: who owns the control flow, and where its state lives.

What I would do

  1. If the job is one step, use a queue with retries and idempotency keys. Don’t adopt a workflow engine for single calls.
  2. If the process has three or more dependent steps, waits on people or timers, or runs an agent loop longer than a request timeout, use durable execution. I’d start with DBOS if Postgres is already the system of record and the team wants a library, and Temporal if many services and languages share the workflows.
  3. Put every model call and every tool call in its own step. Keep workflow code free of clocks, randomness and I/O.
  4. Make every step idempotent. Pass an idempotency key derived from the workflow ID and step name to anything that charges, sends or writes.
  5. Keep payloads small and non-sensitive in history; store blobs elsewhere and reference them.
  6. Write the versioning plan before the first production deploy, not after the first non-determinism error.

FAQ

What is durable execution in simple terms?

Code that remembers how far it got. The runtime saves the result of each step, so after a crash the function restarts, reuses the saved results, and continues from the step that hadn’t finished.

Is durable execution the same as Temporal?

No. Temporal is one implementation. Restate, DBOS and Uber’s Cadence implement the same idea with different storage and programming models.

Why must workflow code be deterministic?

Because recovery works by re-running the code against the recorded history. If the code makes a different decision on replay, the recorded results no longer match the steps being asked for, and the engine can’t rebuild the state.

Does durable execution guarantee exactly-once?

For the workflow’s decisions, effectively yes. For side effects in steps, no: a step can run again if the worker dies after the side effect but before the result is recorded. Make steps idempotent.

Do AI agents need durable execution?

Short agents that finish within one request don’t. Agents that run for minutes, wait for approvals, or call tools that change things benefit, because a crash no longer means repeating paid model calls and side effects.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.