Skip to main content
A coding agent that runs for six hours will, at some point, meet a six-hour-shaped failure. The laptop sleeps. The terminal closes. Claude’s API returns 529s for forty minutes. A quota resets at midnight. A reviewer who was supposed to approve the diff at 5pm approves it at 9am. None of these are exotic — they are the normal operating conditions of background agent work. The question is only what happens next: does the run resume at the right step with all its completed work intact, or does it start over and re-bill every token it already spent? Most agent tooling answers “start over.” A chat session holds its orchestration state in process memory; kill the process and the plan, the completed subtask results, and the position in the loop are gone. The transcript may survive — that is memory, not durability. Durability means the execution survives: which steps completed, what they produced, what was in flight, and what the working tree looked like at each point. Smithers treats durability as the substrate, not a feature. Every completed task is a database row the moment it finishes; every scheduling decision is derived from those rows rather than from anything held in memory. This page is the deep version of the crash-resume demo in the durable orchestration essay: what exactly is persisted, the precise resume contract, the two-layer durability model, and how each failure class is handled — with an honest comparison against the systems that invented this discipline.

​
How the industry does it

Durable execution is a solved problem with two dominant mechanisms and a few outliers. What differs is where the journal lives, what granularity it records, and what discipline it demands from your code. Two universal patterns fall out of this survey. First: orchestration logic is made effectively-once everywhere (via replay or memoization), while side-effecting work is at-least-once everywhere — step idempotency is the industry-wide contract, with exactly-once claimed only for service-internal transitions. Second: code changed under an in-flight run is the universal second-order pain, solved by patch markers (Temporal), side-by-side versions (Durable Functions, Dapr), or pinned versions (Cloudflare, Trigger.dev). Smithers has an answer to both, below.

​
What Smithers persists, when, and where

Smithers is in the checkpoint family, not the replay family. It does not re-execute your workflow code against an event log; it re-renders the workflow tree against persisted state, and the scheduling plan is a pure function of that state (the full loop is in How It Works). What makes that safe is the write set. Everything lives in one database — bun:sqlite by default, with PGlite and Postgres backends behind the same adapter (smithers migrate moves a legacy SQLite db over): The event log exists for smithers logs, scoring, and debugging. Resume never replays it — resume reads current state. That is the core design difference from Temporal: no replay means no determinism requirement on workflow code, at the cost of coarser recovery granularity (task attempts, not individual awaits).

​
Two layers: engine state and worktree checkpoints

Database rows recover the orchestration. But a coding agent’s real product is a working tree, and an agent killed 40 minutes into an edit session has filesystem state worth saving too. So durability runs at two layers. Layer 1 — engine state (always on). The tables above. A completed task’s outputs, the frame history, suspension state. This layer alone gives you crash-resume of the workflow. Layer 2 — workspace durability snapshots (jj worktrees). While an agent works in a worktree, startDurability.js runs a filesystem watcher that commits the tree via jj on settle (Tier 2, source: "watch"), and agents that expose tool hooks get strict per-tool-call snapshots (Tier 1): the Claude Code plugin’s PostToolUse hook runs smithers snapshot-hook, which hits a local snapshot socket and captures the tree immediately after each mutation, labeled with the tool_use_id. In-process SDK agents get the same via an ambient tool-context wrapper (source: "wrap"). The snapshot service serializes captures per worktree, dedups by jj commit id, and assigns a monotonic seq per attempt. A capture that fails does not fail the agent — it is recorded as a gap in an NDJSON spool outside the worktree ($TMPDIR/smithers-durability/<runId>.gaps.ndjson), so you can always tell the difference between “nothing changed” and “we couldn’t look.” The two layers join at the attempt row: _smithers_attempts.jj_pointer records where the tree ended up, and _smithers_workspace_checkpoints records every intermediate step of how it got there. smithers snapshots <runId> lists them; smithers restore <runId> --node <id> [--seq N] puts a worktree back to any of them via jj restore --from <commit>.

​
The resume contract

Resume is legal from any of: running (stale), waiting-approval, waiting-event, waiting-timer, waiting-quota, paused, cancelled, finished, failed. The contract, precisely:
  1. Completed tasks never re-execute. Their output rows exist; on re-render the node is already satisfied and is skipped. Effectively-once, enforced by state, not by replay.
  2. In-flight attempts re-run. An attempt whose heartbeat is stale (the engine abandons in-progress attempts older than 15 minutes, STALE_ATTEMPT_MS) is marked abandoned and the task runs again as a fresh attempt. Task execution is therefore at-least-once — same as Temporal activities, LangGraph nodes, and every other system in the table. If your task has external side effects (posts a comment, sends a webhook), make it idempotent or gate it behind an <Approval>.
  3. Task ids must be stable across renders. Resume matches nodes by (node_id, iteration). id={review-$} resumes; id={Math.random()} orphans every completed row.
  4. The workflow source must not have changed. This is the workflow-hash constraint.

​
RESUME_METADATA_MISMATCH

At run start Smithers stores a hash of the workflow entry file and of the entire imported module graph in _smithers_runs.workflow_hash plus durability metadata in the run config. Resume recomputes both and compares, along with the workflow path and VCS root. Any drift throws RESUME_METADATA_MISMATCH rather than resuming a run whose completed rows were produced by different code — the same hazard Temporal’s non-determinism errors guard, caught by content hash instead of replay divergence. The hash is over file content, not git; no commit is required to trip it, and reverting the edit un-trips it. Your options when it fires, in order of preference:
  • Carry the edit forward on a fork: smithers fork workflow.tsx --run-id r1 --frame N branches a new run from a persisted frame under the new code.
  • Resume this run anyway: --accept-workflow-change waives workflow-hash mismatches only (never path/VCS-root drift). You are asserting the edit is compatible with the persisted state; Smithers cannot check that for you — it has no replay to check it with.
  • Revert the file and resume normally.
Unattended resumes (the supervisor, the gateway’s timer sweep) that hit a mismatch fail the run instead of throwing invisibly — an earlier design silently hot-looped forever on stale runs with drifted source (issues #494, #1361). Interactive --resume mismatches throw without failing the run, so you keep all three options.

​
Failure taxonomy

The run row still says running but its heartbeat stops. smithers supervise <run-id...> (or --all) polls for runs whose heartbeat is older than the stale threshold (default 30s), claims the run by stamping runtime_owner_id with supervisor:<id> so two supervisors can’t double-resume, and resumes it in a detached child. Completed work is skipped per the contract; the interrupted attempt re-runs.
Identical to a crash from the database’s perspective — the db is on disk, the heartbeat is stale. Run supervise under your init system (launchd/systemd) and reboots become a resume, not a loss. The essay’s claim that a suspended run “costs nothing while it waits” is literal: a waiting run is rows, no process.
Per-task, agent={[claude, codex]} fails over between harnesses. When every option is exhausted the run parks as waiting-quota with the provider’s reset time in error_json.resetAtMs; runsDueForQuotaResume wakes it automatically once the reset elapses (via supervisor or the gateway’s timer sweep). Quota parks with no known reset time wait for a human — guessing would burn the new quota re-failing.
<Approval> moves the run to waiting-approval and the process exits. The pending request is a durable row answerable from CLI (smithers approve / deny), web, or MCP at any hour; resolution triggers resume at exactly the gated node. Nothing replays, because nothing needs to — the state the run suspended with is the state it wakes with.
smithers pause <runId> sets pause_requested_at_ms; the driver stops scheduling new tasks, lets in-flight attempts drain, and parks the run as paused — resumable later, cleanly, with zero abandoned attempts. smithers cancel propagates abort to running agents and marks the run cancelled; even that is resumable, because cancellation deletes no state.

​
Guarantees and limits

Guaranteed: completed tasks are never re-executed; validated outputs, frames, events, approvals, and worktree checkpoints survive any process death once written; suspension costs no compute; every resume path is heartbeat-guarded against double-execution by two processes. Not guaranteed, honestly:
  • No exactly-once side effects. An attempt that posted a webhook and then died will post it again on retry. Temporal has the same property for activities; nobody in the table above escapes it. Idempotency keys are your job.
  • Coarser recovery than replay systems. Temporal resumes a workflow between two awaits; Smithers re-runs the interrupted task attempt. For an agent task that means re-paying that attempt’s tokens (Tier 1/2 checkpoints preserve the filesystem work, and <Task> fork can reuse a prior session’s context, but the interrupted agent conversation itself restarts). If your steps are minutes of pure compute with strict budgets, Temporal’s determinism contract buys you finer resume than we do — see Smithers vs Temporal.
  • No automatic code versioning. Temporal lets patched code coexist with old histories; Smithers makes you choose explicitly (fork, accept, or revert). Simpler, but a fleet of long-lived runs across frequent deploys is more manual here.
  • Tier 2 snapshots are settle-based, so the final keystrokes before a hard kill can miss the last checkpoint window; the attempt re-run covers correctness, and the gap spool records that a window was missed. Tier 1 requires an agent harness with tool hooks.
  • One database, one write path. Postgres gets you off the laptop, but there is no multi-region replicated log à la Restate or Temporal Cloud.

​
Operating it

Every command has an MCP twin (get_run, list_snapshots, restore_checkpoint, replay_run, …), so the agent operating a run gets the same durability surface you do.

​
See also