Run State

Every Smithers surface (CLI, Gateway, programmatic clients) answers “what is this run doing right now?” by calling computeRunState, never inferring status from ps, event absence, or partial table reads: it reads the same RunStateView, computed server-side from persisted state plus liveness signals.

RunState

type RunState =
  | "running"           // owner alive, work progressing
  | "waiting-approval"  // blocked on a human decision
  | "waiting-event"     // blocked on an external signal
  | "waiting-timer"     // blocked on a scheduled wakeup
  | "paused"            // gracefully paused; resume with `up --resume`
  | "recovering"        // supervisor replaying / resuming
  | "stale"             // owner heartbeat expired, not yet recovered
  | "orphaned"          // owner gone, no supervisor candidate
  | "failed"
  | "cancelled"
  | "succeeded"
  | "succeeded-with-failures" // finished, but tolerated child failures
  | "unknown"           // telemetry gap; never invent a state

When state can’t be determined (e.g. a missing heartbeat with ambiguous DB status), the runtime returns unknown. The legacy run-row status column maps to RunState like this:

`_smithers_runs.status`	`RunState`
`running` (fresh)	`running`
`running` (stale)	`stale` or `orphaned`
`waiting-approval`	`waiting-approval`
`waiting-event`	`waiting-event`
`waiting-timer`	`waiting-timer`
`paused`	`paused`
`finished`	`succeeded` or `succeeded-with-failures`
`continued`	`succeeded`
`failed`	`failed`
`cancelled`	`cancelled`
missing / unknown text	`unknown`

recovering is reserved for the supervisor takeover window. As of this writing, it’s defined but not yet emitted.

waiting-event is overloaded; don’t assume it always means “send a signal.” The same status covers two situations needing opposite recovery:

A genuine pause on <Signal> or <WaitForEvent>: recover by delivering the signal, bunx smthrs signal RUN_ID SIGNAL_NAME --data '{}'.
A run parked with no live event node (external-trigger, hot-reload, orphan-recovery wait, an aspect/alert human-request, or an approval decided while detached): a signal does nothing; resume the run (or retry-task / fork / answer the human request) after fixing the underlying cause.

Don’t guess: ask the runtime, which disambiguates on RunStateView.blocked.kind:

bunx smthrs why RUN_ID     # names the real blocker and prints the exact unblock command
bunx smthrs inspect RUN_ID # includes runState.blocked.kind

runState.blocked.kind === "event" means a node truly awaits an event. If the run row says waiting-event but no node does, runState.blocked.kind is approval-decided-resume-required (a detached approval continuation) or external-trigger (a generic parked external wait). why gives the most detailed unblock command and may report other blockers such as retries-exhausted or dependency-failed.

When a fail-closed composite such as <TryCatchFinally> terminates a run, its run-level SCHEDULER_ERROR includes the nearest completed output row’s summary and blockers fields. The leaf diagnostic is therefore visible in up, inspect, and autopsy output without a separate node/output lookup.

finished does not always mean clean success. Run-level lifecycle status is binary (finished or failed); a run fails only on an unhandled task failure. Two classes of failed children don’t count as run-level failures: tasks marked continueOnFail, and agent tasks failing with a transient error (rate limits, timeouts, aborts: SESSION_ERROR, TASK_TIMEOUT, TASK_HEARTBEAT_TIMEOUT, TASK_ABORTED, or failureRetryable). A fan-out where most agents hit provider rate limits can therefore persist status: "finished", but now derives RunState.state: "succeeded-with-failures" instead of claiming clean succeeded.Never trust top-level status alone; inspect per-node/agent outcomes:

bunx smthrs inspect RUN_ID   # top-level runState plus every node's state (look for failed nodes)
bunx smthrs node NODE_ID      # one node's failure/error, retries, and tool calls
bunx smthrs events --run RUN_ID  # the full event history, including per-task failures
bunx smthrs scores RUN_ID     # scorer results when you gate on judged output

No need to re-derive the count by eye: when a finished run tolerated at least one failed child, the masking surfaces as a first-class failedChildren signal in three places:

inspect adds failedChildren (a count) and failedChildKeys (the failed task state keys, nodeId::iteration) to its JSON next to runState, plus a node … call-to-action; both absent when nothing failed, so failedChildren > 0 is the degraded-run gate.
RunResult from a programmatic run carries the same failedChildren / failedChildKeys fields (omitted when zero).
The RunFinished event row records them too, so the CLI, Gateway, and DevTools can flag the degraded outcome from the event stream without re-reading every node row.

The run-row and RunResult.status stay finished. RunState reads the persisted RunFinished.failedChildren value and returns succeeded-with-failures when it is nonzero; historical or clean events still map to succeeded. This is deliberately a distinct derived outcome, not a third persisted RunStatus, so existing terminal pollers, Gateway clients, and status === "finished" callers continue to complete normally. State-aware consumers should treat both success states as terminal, while using succeeded-with-failures to request review or gate clean-success automation.bunx smthrs up keeps exit code 0 for this outcome. continueOnFail is an explicit instruction to finish the workflow; changing the exit code would silently break CI that already accepts finished. CI requiring a clean run should inspect runState.state or failedChildren.The same applies to loops that never converged: a <Loop>/<ReviewLoop> exiting via onMaxReached: "return-last" with its until predicate still false is recorded as exhausted. The finished run’s RunResult and RunFinished event carry exhaustedLoops (id, iteration, maxIterations), status reports the verdict degraded instead of done, and why names the loop and its unmet condition instead of “nothing is blocked”.To avoid rate-limit failures, bound fan-out concurrency with <Parallel maxConcurrency={N}> so you don’t burst past the provider’s limit.

ReasonBlocked / ReasonUnhealthy

ReasonBlocked and ReasonUnhealthy are optional reason payloads for waiting and unhealthy states; the type unions are wider than the variants currently derived from the DB rows.

type ReasonBlocked =
  | { kind: "approval"; nodeId: string; requestedAt: string }
  | { kind: "event";    nodeId: string; correlationKey: string }
  | { kind: "timer";    nodeId: string; wakeAt: string }
  | { kind: "approval-decided-resume-required"; nodeId: string }
  | { kind: "external-trigger" }
  | { kind: "provider"; nodeId: string; code: "rate-limit" | "auth" | "timeout" }
  | { kind: "tool";     nodeId: string; toolName: string; code: string }

type ReasonUnhealthy =
  | { kind: "engine-heartbeat-stale"; lastHeartbeatAt: string }
  | { kind: "timer-overdue"; wakeAt: string; overdueMs: number }
  | { kind: "ui-heartbeat-stale";     lastSeenAt: string }
  | { kind: "db-lock" }
  | { kind: "sandbox-unreachable" }
  | { kind: "supervisor-backoff"; attempt: number; nextAt: string }

Timestamps are ISO-8601 strings. Current computeRunState / deriveRunState emits approval, event, timer, approval-decided-resume-required, and external-trigger blocked reasons, plus engine-heartbeat-stale and timer-overdue unhealthy reasons; the rest are reserved by the public type for future run-state surfaces.

RunStateView

type RunStateWarning = {
  kind: "concurrency-ceiling-saturated";
  requestedDemand: number;
  effectiveCap: number;
  remediationCommand: string;
  observedAt: string;       // ISO-8601
};

type RunStateView = {
  runId: string;
  state: RunState;
  blocked?: ReasonBlocked;
  unhealthy?: ReasonUnhealthy;
  warnings?: RunStateWarning[];
  computedAt: string;        // ISO-8601
};

blocked is present for a waiting-* state when computeRunState can load matching pending approval/timer/event context, when a parked waiting-event run has no event-waiting node, or when that context is supplied to deriveRunState. A waiting-* state can be returned without blocked when the supporting row is unavailable. unhealthy is present for stale, orphaned, and overdue-timer results. recovering stays unemitted for now (see above), and terminal states (succeeded, succeeded-with-failures, failed, cancelled) carry neither. warnings retains durable operator diagnostics independently of lifecycle state. When automatic concurrency reaches its configured ceiling while demand remains queued, the latest warning reports the requested demand, effective cap, and exact --max-concurrency remediation command.

computeRunState

import { computeRunState } from "@smthrs/db/runState";

const view = await computeRunState(adapter, runId);
view.state;       // "running" | ...
view.blocked;     // present for waiting-* only when backing context is found
view.unhealthy;   // present for stale/orphaned heartbeat expiry or overdue timers
view.warnings;    // durable operator diagnostics, when present

computeRunState is pure over the DB plus the heartbeat / lease signals on the run row: no ps calls, no socket probes, no heuristics. deriveRunState is the underlying pure function, useful in tests or when you already have the rows in memory:

import { deriveRunState } from "@smthrs/db/runState";

const view = deriveRunState({
  run,
  pendingApproval,
  pendingTimer,
  pendingEvent,
  now: 1_700_000_000_000,
  staleThresholdMs: 30_000,
});

The default staleThresholdMs is 30_000, the same threshold the engine uses for isRunHeartbeatFresh.

Where it shows up

RunStateView is the wire format on every read surface:

bunx smthrs inspect RUN_ID: top-level runState field on the JSON output (and rendered in the human view).
Gateway RPC getRun: runState field on the response.
DevTools snapshot header: runState?: RunStateView field.
ps --json: compatible status: "finished" plus the distinct state: "succeeded-with-failures"; status, inspect, tail, why, and monitor views render the degraded outcome without keeping the run active.
Event stream: RunStatusChanged records persisted status transitions. RunStateChanged is a typed/reserved event variant, but the current runtime does not emit it; call getRun or computeRunState when you need the derived RunStateView.

A run id that doesn’t exist isn’t a RunState. It’s an error (RUN_NOT_FOUND); unknown is for ambiguity, not for “doesn’t exist.”

Overview

Core API

Runtime

Types & errors

Gateway RPC

RunState

ReasonBlocked / ReasonUnhealthy

RunStateView

computeRunState

Where it shows up

​RunState

​ReasonBlocked / ReasonUnhealthy

​RunStateView

​computeRunState

​Where it shows up

RunState

ReasonBlocked / ReasonUnhealthy

RunStateView

computeRunState

Where it shows up