Run State

Every Smithers surface (CLI, Gateway, programmatic clients) answers “what is this run doing right now?” by calling computeRunState, never inferring status from ps, event absence, or partial table reads: it reads the same RunStateView, computed server-side from persisted state plus liveness signals.

RunState

type RunState =
  | "running"           // owner alive, work progressing
  | "waiting-approval"  // blocked on a human decision
  | "waiting-event"     // blocked on an external signal
  | "waiting-timer"     // blocked on a scheduled wakeup
  | "paused"            // gracefully paused; resume with `up --resume`
  | "recovering"        // supervisor replaying / resuming
  | "stale"             // owner heartbeat expired, not yet recovered
  | "orphaned"          // owner gone, no supervisor candidate
  | "failed"
  | "cancelled"
  | "succeeded"
  | "unknown"           // telemetry gap; never invent a state

When state can’t be determined (e.g. a missing heartbeat with ambiguous DB status), the runtime returns unknown. The legacy run-row status column maps to RunState like this:

`_smithers_runs.status`	`RunState`
`running` (fresh)	`running`
`running` (stale)	`stale` or `orphaned`
`waiting-approval`	`waiting-approval`
`waiting-event`	`waiting-event`
`waiting-timer`	`waiting-timer`
`paused`	`paused`
`finished`	`succeeded`
`continued`	`succeeded`
`failed`	`failed`
`cancelled`	`cancelled`
missing / unknown text	`unknown`

recovering is reserved for the supervisor takeover window. As of this writing, it’s defined but not yet emitted.

waiting-event is overloaded; don’t assume it always means “send a signal.” The same status covers two situations needing opposite recovery:

A genuine pause on <Signal> or <WaitForEvent>: recover by delivering the signal, bunx smthrs signal RUN_ID SIGNAL_NAME --data '{}'.
A run parked with no live event node (external-trigger, hot-reload, orphan-recovery wait, an aspect/alert human-request, or an approval decided while detached): a signal does nothing; resume the run (or retry-task / fork / answer the human request) after fixing the underlying cause.

Don’t guess: ask the runtime, which disambiguates on RunStateView.blocked.kind:

bunx smthrs why RUN_ID     # names the real blocker and prints the exact unblock command
bunx smthrs inspect RUN_ID # includes runState.blocked.kind

runState.blocked.kind === "event" means a node truly awaits an event. If the run row says waiting-event but no node does, runState.blocked.kind is approval-decided-resume-required (a detached approval continuation) or external-trigger (a generic parked external wait). why gives the most detailed unblock command and may report other blockers such as retries-exhausted or dependency-failed.

succeeded/finished can mask failed child agents. Run-level status is binary (finished or failed); a run fails only on an unhandled task failure. Two classes of failed children don’t count as run-level failures: tasks marked continueOnFail, and agent tasks failing with a transient error (rate limits, timeouts, aborts: SESSION_ERROR, TASK_TIMEOUT, TASK_HEARTBEAT_TIMEOUT, TASK_ABORTED, or failureRetryable). A fan-out where most agents hit provider rate limits can still report finished → succeeded.Never trust top-level status alone; inspect per-node/agent outcomes:

bunx smthrs inspect RUN_ID   # top-level runState plus every node's state (look for failed nodes)
bunx smthrs node NODE_ID      # one node's failure/error, retries, and tool calls
bunx smthrs events --run RUN_ID  # the full event history, including per-task failures
bunx smthrs scores RUN_ID     # scorer results when you gate on judged output

No need to re-derive the count by eye: when a finished run tolerated at least one failed child, the masking surfaces as a first-class failedChildren signal in three places:

inspect adds failedChildren (a count) and failedChildKeys (the failed task state keys, nodeId::iteration) to its JSON next to runState, plus a node … call-to-action; both absent when nothing failed, so failedChildren > 0 is the degraded-run gate.
RunResult from a programmatic run carries the same failedChildren / failedChildKeys fields (omitted when zero).
The RunFinished event row records them too, so the CLI, Gateway, and DevTools can flag the degraded outcome from the event stream without re-reading every node row.

The run-row status stays finished (RunState stays succeeded), so existing finished === success callers and continueOnFail semantics are unchanged: the count is the signal, not a new terminal status.The same applies to loops that never converged: a <Loop>/<ReviewLoop> exiting via onMaxReached: "return-last" with its until predicate still false is recorded as exhausted. The finished run’s RunResult and RunFinished event carry exhaustedLoops (id, iteration, maxIterations), status reports the verdict degraded instead of done, and why names the loop and its unmet condition instead of “nothing is blocked”.To avoid rate-limit failures, bound fan-out concurrency with <Parallel maxConcurrency={N}> so you don’t burst past the provider’s limit.

ReasonBlocked / ReasonUnhealthy

ReasonBlocked and ReasonUnhealthy are optional reason payloads for waiting and unhealthy states; the type unions are wider than the variants currently derived from the DB rows.

type ReasonBlocked =
  | { kind: "approval"; nodeId: string; requestedAt: string }
  | { kind: "event";    nodeId: string; correlationKey: string }
  | { kind: "timer";    nodeId: string; wakeAt: string }
  | { kind: "approval-decided-resume-required"; nodeId: string }
  | { kind: "external-trigger" }
  | { kind: "provider"; nodeId: string; code: "rate-limit" | "auth" | "timeout" }
  | { kind: "tool";     nodeId: string; toolName: string; code: string }

type ReasonUnhealthy =
  | { kind: "engine-heartbeat-stale"; lastHeartbeatAt: string }
  | { kind: "timer-overdue"; wakeAt: string; overdueMs: number }
  | { kind: "ui-heartbeat-stale";     lastSeenAt: string }
  | { kind: "db-lock" }
  | { kind: "sandbox-unreachable" }
  | { kind: "supervisor-backoff"; attempt: number; nextAt: string }

Timestamps are ISO-8601 strings. Current computeRunState / deriveRunState emits approval, event, timer, approval-decided-resume-required, and external-trigger blocked reasons, plus engine-heartbeat-stale and timer-overdue unhealthy reasons; the rest are reserved by the public type for future run-state surfaces.

RunStateView

type RunStateView = {
  runId: string;
  state: RunState;
  blocked?: ReasonBlocked;
  unhealthy?: ReasonUnhealthy;
  computedAt: string;        // ISO-8601
};

blocked is present for a waiting-* state when computeRunState can load matching pending approval/timer/event context, when a parked waiting-event run has no event-waiting node, or when that context is supplied to deriveRunState. A waiting-* state can be returned without blocked when the supporting row is unavailable. unhealthy is present for stale, orphaned, and overdue-timer results. recovering stays unemitted for now (see above), and terminal states (succeeded, failed, cancelled) carry neither.

computeRunState

import { computeRunState } from "@smthrs/db/runState";

const view = await computeRunState(adapter, runId);
view.state;       // "running" | ...
view.blocked;     // present for waiting-* only when backing context is found
view.unhealthy;   // present for stale/orphaned heartbeat expiry or overdue timers

computeRunState is pure over the DB plus the heartbeat / lease signals on the run row: no ps calls, no socket probes, no heuristics. deriveRunState is the underlying pure function, useful in tests or when you already have the rows in memory:

import { deriveRunState } from "@smthrs/db/runState";

const view = deriveRunState({
  run,
  pendingApproval,
  pendingTimer,
  pendingEvent,
  now: 1_700_000_000_000,
  staleThresholdMs: 30_000,
});

The default staleThresholdMs is 30_000, the same threshold the engine uses for isRunHeartbeatFresh.

Where it shows up

RunStateView is the wire format on every read surface:

bunx smthrs inspect RUN_ID: top-level runState field on the JSON output (and rendered in the human view).
Gateway RPC getRun: runState field on the response.
DevTools snapshot header: runState?: RunStateView field.
Event stream: RunStatusChanged records persisted status transitions. RunStateChanged is a typed/reserved event variant, but the current runtime does not emit it; call getRun or computeRunState when you need the derived RunStateView.

A run id that doesn’t exist isn’t a RunState. It’s an error (RUN_NOT_FOUND); unknown is for ambiguity, not for “doesn’t exist.”

Overview

Core API

Runtime

Types & errors

Gateway RPC

RunState

ReasonBlocked / ReasonUnhealthy

RunStateView

computeRunState

Where it shows up

​RunState

​ReasonBlocked / ReasonUnhealthy

​RunStateView

​computeRunState

​Where it shows up

RunState

ReasonBlocked / ReasonUnhealthy

RunStateView

computeRunState

Where it shows up