> ## Documentation Index > Fetch the complete documentation index at: https://smithers.sh/llms.txt > Use this file to discover all available pages before exploring further. # Durability > How a Smithers run survives a process crash, a reboot, a provider outage, or an approval that arrives fourteen hours later — and exactly what is persisted, replayed, and re-executed. A coding agent that runs for six hours will, at some point, meet a six-hour-shaped failure. The laptop sleeps. The terminal closes. Claude's API returns 529s for forty minutes. A quota resets at midnight. A reviewer who was supposed to approve the diff at 5pm approves it at 9am. None of these are exotic — they are the *normal* operating conditions of background agent work. The question is only what happens next: does the run resume at the right step with all its completed work intact, or does it start over and re-bill every token it already spent? Most agent tooling answers "start over." A chat session holds its orchestration state in process memory; kill the process and the plan, the completed subtask results, and the position in the loop are gone. The transcript may survive — that is memory, not durability. Durability means the *execution* survives: which steps completed, what they produced, what was in flight, and what the working tree looked like at each point. Smithers treats durability as the substrate, not a feature. Every completed task is a database row the moment it finishes; every scheduling decision is derived from those rows rather than from anything held in memory. This page is the deep version of the crash-resume demo in [the durable orchestration essay](/why/durable-open-orchestration): what exactly is persisted, the precise resume contract, the two-layer durability model, and how each failure class is handled — with an honest comparison against the systems that invented this discipline. ## How the industry does it Durable execution is a solved problem with two dominant mechanisms and a few outliers. What differs is where the journal lives, what granularity it records, and what discipline it demands from your code. | System | Mechanism | Constraint / tradeoff | | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Temporal](https://docs.temporal.io/workflow-execution/event#event-history) | Event-sourced history + deterministic re-execution: on recovery, workflow code re-runs from the top and replay serves recorded activity results, timers, and signals | Workflow code must be fully deterministic; incompatible edits against open histories throw non-determinism errors, fixed via [`workflow.patched()` versioning](https://docs.temporal.io/develop/python/versioning); \~50K-event history limit forces continue-as-new | | [Cadence](https://cadenceworkflow.io/docs/concepts/workflows) | Same event-history + replay model (Temporal's ancestor at Uber) | Same determinism constraint; versioning via `workflow.GetVersion()` markers | | [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html) | Service-persisted declarative state machine; Standard runs up to 1 year with exactly-once state transitions; failed executions can be [redriven](https://docs.aws.amazon.com/step-functions/latest/dg/redrive-executions.html) from the failed state | Logic lives in JSON/ASL, not general code; 25K-event and 256KB-payload limits | | [Azure Durable Functions](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-orchestrations#reliability) | Event sourcing into Azure Storage; orchestrators replay from scratch at every await, activity results served from the history table | Strict [determinism constraints](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-code-constraints) (no `DateTime.Now`, no direct I/O); versioning via side-by-side deploys | | [Restate](https://docs.restate.dev/concepts/durable_execution/) | Durable RPC journal: every side-effecting step is written to a replicated log before its result is returned; re-execution serves journaled results instantly | Code between journaled steps must be deterministic; Restate is also your RPC broker — a large architectural buy-in | | [DBOS](https://docs.dbos.dev/architecture) | Library-only Postgres checkpointing: workflow inputs and each step's output written to Postgres tables; recovery re-runs pending workflows, reusing stored step outputs | Workflow functions must be deterministic; tied to Postgres; one checkpoint write per step | | [Inngest](https://www.inngest.com/docs/learn/how-functions-are-executed) | Step memoization: each `step.run` result is persisted server-side; on resume the function re-executes and memoized steps return stored results | Code between steps re-runs on every resume; changing step order mid-flight mismatches memoized state | | [Trigger.dev](https://trigger.dev/docs/how-it-works) | CRIU-style process checkpointing: on waits, the entire container (memory, registers, file descriptors) is frozen to disk and restored later, possibly on another machine | No determinism requirement, but checkpoints are heavyweight, Linux-specific, and infra-coupled | | [Resonate](https://docs.resonatehq.io/evaluate/how-it-works) | Durable promises: each invocation is a write-once promise stored server-side; recovery re-invokes functions whose promises are pending | Coarser than per-step journaling — re-executed code up to the next promise must be safe to repeat | | [Dapr Workflows](https://docs.dapr.io/developing-applications/building-blocks/workflow/workflow-features-concepts/) | Durable Task Framework: event-sourced history in the Dapr state store, orchestrator replay | Same determinism rules; code changes vs running instances require registering new workflow names | | [Golem Cloud](https://learn.golem.cloud/v1.5/operate/persistence) | WASM oplog: every host-function interaction recorded; workers deterministically replayed to the exact interrupted instruction | Code must compile to WASM components; determinism enforced by the sandbox | | [Cloudflare Workflows](https://developers.cloudflare.com/workflows/build/workers-api/) | `step.do()` memoization backed by Durable Object SQLite storage; `step.sleep`/`waitForEvent` hibernate at zero compute | Step return values must be serializable and size-limited; code outside steps re-executes | | [LangGraph](https://docs.langchain.com/oss/python/langgraph/persistence) | Super-step checkpointing: full graph state saved by a checkpointer (Postgres/SQLite) at every node boundary under a `thread_id`; pending writes preserve outputs of successful parallel siblings | Durability granularity is the node; no deterministic replay, so nodes must be idempotent | | [OpenAI Agents SDK sessions](https://openai.github.io/openai-agents-python/sessions/) | Conversation persistence only (SQLite/Conversations API) — the transcript survives, the execution position does not | Not durable execution; OpenAI's own answer for that is the [Temporal integration](https://docs.temporal.io/develop/python/integrations/openai-agents) | | Claude Code [Dynamic Workflows](https://code.claude.com/docs/en/workflows) | In-session orchestration held in the live process; session resume restores the transcript, not workflow step state | No crash-resume of the workflow graph | Two universal patterns fall out of this survey. First: orchestration logic is made *effectively-once* everywhere (via replay or memoization), while side-effecting work is **at-least-once** everywhere — step idempotency is the industry-wide contract, with exactly-once claimed only for service-internal transitions. Second: code changed under an in-flight run is the universal second-order pain, solved by patch markers (Temporal), side-by-side versions (Durable Functions, Dapr), or pinned versions (Cloudflare, Trigger.dev). Smithers has an answer to both, below. ## What Smithers persists, when, and where Smithers is in the **checkpoint family**, not the replay family. It does not re-execute your workflow code against an event log; it re-*renders* the workflow tree against persisted state, and the scheduling plan is a pure function of that state (the full loop is in [How It Works](/how-it-works)). What makes that safe is the write set. Everything lives in one database — `bun:sqlite` by default, with PGlite and Postgres backends behind the same adapter (`smithers migrate` moves a legacy SQLite db over): | Table / store | What it holds | Written when | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `_smithers_runs` | Run status, workflow path + **workflow hash**, VCS type/root/revision, `heartbeat_at_ms`, `runtime_owner_id`, pause/cancel/hijack request timestamps | Run start; heartbeat and lifecycle transitions | | `_smithers_nodes` | Per-node state keyed `(run_id, node_id, iteration)`, last attempt, output table name | Every node state transition | | `_smithers_attempts` | Every attempt: state, timings, per-attempt heartbeat + heartbeat data, error JSON, response text, and a **jj pointer + worktree cwd** for filesystem state | Attempt start, heartbeats during execution, completion/failure | | Per-schema output tables | Validated task outputs as typed rows — the data `ctx.outputs` reads | The moment a task's output validates | | `_smithers_frames` | A snapshot of the rendered workflow tree per frame (`xml_json`, hash, mounted task ids) | Every render frame — this is what `timeline`, `fork`, and `rewind` operate on | | `_smithers_events` | Append-only run/node lifecycle event log `(run_id, seq, type, payload)` | Continuously; this is observability, **not** a replay source | | `_smithers_approvals`, `_smithers_signals`, `_smithers_human_requests` | Durable suspension state: what the run is waiting for and who answered | On suspension and resolution | | `_smithers_workspace_checkpoints` / `_smithers_workspace_states` | Filesystem checkpoints: jj commit id per `(run, node, iteration, attempt, seq)`, with tier and source | During agent execution (next section) | The event log exists for `smithers logs`, scoring, and debugging. Resume never replays it — resume reads current state. That is the core design difference from Temporal: no replay means no determinism requirement on workflow code, at the cost of coarser recovery granularity (task attempts, not individual awaits). ## Two layers: engine state and worktree checkpoints Database rows recover the *orchestration*. But a coding agent's real product is a working tree, and an agent killed 40 minutes into an edit session has filesystem state worth saving too. So durability runs at two layers. **Layer 1 — engine state (always on).** The tables above. A completed task's outputs, the frame history, suspension state. This layer alone gives you crash-resume of the workflow. **Layer 2 — workspace durability snapshots (jj worktrees).** While an agent works in a worktree, `startDurability.js` runs a filesystem watcher that commits the tree via jj on settle (**Tier 2**, `source: "watch"`), and agents that expose tool hooks get strict per-tool-call snapshots (**Tier 1**): the Claude Code plugin's PostToolUse hook runs `smithers snapshot-hook`, which hits a local snapshot socket and captures the tree immediately after each mutation, labeled with the `tool_use_id`. In-process SDK agents get the same via an ambient tool-context wrapper (`source: "wrap"`). The snapshot service serializes captures per worktree, dedups by jj commit id, and assigns a monotonic `seq` per attempt. A capture that fails does not fail the agent — it is recorded as a **gap** in an NDJSON spool outside the worktree (`$TMPDIR/smithers-durability/.gaps.ndjson`), so you can always tell the difference between "nothing changed" and "we couldn't look." The two layers join at the attempt row: `_smithers_attempts.jj_pointer` records where the tree ended up, and `_smithers_workspace_checkpoints` records every intermediate step of how it got there. `smithers snapshots ` lists them; `smithers restore --node [--seq N]` puts a worktree back to any of them via `jj restore --from `. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} smithers snapshots r1 # every checkpoint: node, attempt, seq, tier, age smithers restore r1 --node implement # worktree back to the node's latest checkpoint smithers restore r1 --node implement --seq 3 # or a specific mid-attempt state ``` ## The resume contract ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} smithers up workflow.tsx --run-id r1 --resume true ``` Resume is legal from any of: `running` (stale), `waiting-approval`, `waiting-event`, `waiting-timer`, `waiting-quota`, `paused`, `cancelled`, `finished`, `failed`. The contract, precisely: 1. **Completed tasks never re-execute.** Their output rows exist; on re-render the node is already satisfied and is skipped. Effectively-once, enforced by state, not by replay. 2. **In-flight attempts re-run.** An attempt whose heartbeat is stale (the engine abandons in-progress attempts older than 15 minutes, `STALE_ATTEMPT_MS`) is marked abandoned and the task runs again as a fresh attempt. Task execution is therefore **at-least-once** — same as Temporal activities, LangGraph nodes, and every other system in the table. If your task has external side effects (posts a comment, sends a webhook), make it idempotent or gate it behind an [``](/how-it-works). 3. **Task ids must be stable across renders.** Resume matches nodes by `(node_id, iteration)`. `id={`review-\${file}`}` resumes; `id={Math.random()}` orphans every completed row. 4. **The workflow source must not have changed.** This is the workflow-hash constraint. ### RESUME\_METADATA\_MISMATCH At run start Smithers stores a hash of the workflow **entry file** and of the **entire imported module graph** in `_smithers_runs.workflow_hash` plus durability metadata in the run config. Resume recomputes both and compares, along with the workflow path and VCS root. Any drift throws `RESUME_METADATA_MISMATCH` rather than resuming a run whose completed rows were produced by different code — the same hazard Temporal's non-determinism errors guard, caught by content hash instead of replay divergence. The hash is over file content, not git; no commit is required to trip it, and reverting the edit un-trips it. Your options when it fires, in order of preference: * **Carry the edit forward on a fork:** `smithers fork workflow.tsx --run-id r1 --frame N` branches a new run from a persisted frame under the new code. * **Resume this run anyway:** `--accept-workflow-change` waives *workflow-hash* mismatches only (never path/VCS-root drift). You are asserting the edit is compatible with the persisted state; Smithers cannot check that for you — it has no replay to check it with. * **Revert the file** and resume normally. Unattended resumes (the supervisor, the gateway's timer sweep) that hit a mismatch **fail the run** instead of throwing invisibly — an earlier design silently hot-looped forever on stale runs with drifted source (issues #494, #1361). Interactive `--resume` mismatches throw without failing the run, so you keep all three options. ## Failure taxonomy The run row still says `running` but its heartbeat stops. `smithers supervise ` (or `--all`) polls for runs whose heartbeat is older than the stale threshold (default 30s), claims the run by stamping `runtime_owner_id` with `supervisor:` so two supervisors can't double-resume, and resumes it in a detached child. Completed work is skipped per the contract; the interrupted attempt re-runs. Identical to a crash from the database's perspective — the db is on disk, the heartbeat is stale. Run `supervise` under your init system (launchd/systemd) and reboots become a resume, not a loss. The essay's claim that a suspended run "costs nothing while it waits" is literal: a waiting run is rows, no process. Per-task, `agent={[claude, codex]}` fails over between harnesses. When every option is exhausted the run parks as `waiting-quota` with the provider's reset time in `error_json.resetAtMs`; `runsDueForQuotaResume` wakes it automatically once the reset elapses (via supervisor or the gateway's timer sweep). Quota parks with no known reset time wait for a human — guessing would burn the new quota re-failing. `` moves the run to `waiting-approval` and the process exits. The pending request is a durable row answerable from CLI (`smithers approve` / `deny`), web, or MCP at any hour; resolution triggers resume at exactly the gated node. Nothing replays, because nothing needs to — the state the run suspended with is the state it wakes with. `smithers pause ` sets `pause_requested_at_ms`; the driver stops scheduling new tasks, lets in-flight attempts drain, and parks the run as `paused` — resumable later, cleanly, with zero abandoned attempts. `smithers cancel` propagates abort to running agents and marks the run `cancelled`; even that is resumable, because cancellation deletes no state. ## Guarantees and limits **Guaranteed:** completed tasks are never re-executed; validated outputs, frames, events, approvals, and worktree checkpoints survive any process death once written; suspension costs no compute; every resume path is heartbeat-guarded against double-execution by two processes. **Not guaranteed, honestly:** * **No exactly-once side effects.** An attempt that posted a webhook and then died will post it again on retry. Temporal has the same property for activities; nobody in the table above escapes it. Idempotency keys are your job. * **Coarser recovery than replay systems.** Temporal resumes a workflow *between two awaits*; Smithers re-runs the interrupted *task attempt*. For an agent task that means re-paying that attempt's tokens (Tier 1/2 checkpoints preserve the filesystem work, and `` `fork` can reuse a prior session's context, but the interrupted agent conversation itself restarts). If your steps are minutes of pure compute with strict budgets, Temporal's determinism contract buys you finer resume than we do — see [Smithers vs Temporal](/why/vs-temporal). * **No automatic code versioning.** Temporal lets patched code coexist with old histories; Smithers makes you choose explicitly (fork, accept, or revert). Simpler, but a fleet of long-lived runs across frequent deploys is more manual here. * **Tier 2 snapshots are settle-based**, so the final keystrokes before a hard kill can miss the last checkpoint window; the attempt re-run covers correctness, and the gap spool records that a window was missed. Tier 1 requires an agent harness with tool hooks. * **One database, one write path.** Postgres gets you off the laptop, but there is no multi-region replicated log à la Restate or Temporal Cloud. ## Operating it ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} smithers ps # active, paused, and recently finished runs smithers up wf.tsx --run-id r1 -d # start detached smithers status r1 && smithers why r1 smithers pause r1 # drain in-flight, park resumably smithers up wf.tsx --run-id r1 --resume true smithers supervise --all # auto-resume anything with a stale heartbeat smithers snapshots r1 # filesystem checkpoints per node/attempt smithers restore r1 --node implement --seq 3 smithers timeline r1 # every persisted frame smithers fork wf.tsx --run-id r1 --frame 4 # branch a timeline (incl. under edited code) ``` Every command has an MCP twin (`get_run`, `list_snapshots`, `restore_checkpoint`, `replay_run`, …), so the agent operating a run gets the same durability surface you do. ## See also * [How It Works](/how-it-works#durability--resume) — the render loop, frames, and the resume walkthrough * [The open, durable orchestration layer](/why/durable-open-orchestration) — why this is the layer that doesn't change * [Smithers vs Temporal](/why/vs-temporal) — the replay-vs-checkpoint tradeoff in full * [Background agents](/why/background-agents) — the workload shape that makes all of this necessary