> ## Documentation Index > Fetch the complete documentation index at: https://smithers.sh/llms.txt > Use this file to discover all available pages before exploring further. # > A heartbeat health check that watches one other run, classifies it into a closed condition set, and routes each condition to a handler that heals or escalates. ```ts theme={"theme":{"light":"github-light","dark":"github-dark"}} // Props import { Monitor } from "smithers-orchestrator"; type MonitorCondition = | "healthy" | "stalled" | "wedged-node" | "runaway-loop" | "awaiting-human" | "failing" | "unknown"; type MonitorProps = { id?: string; // default "monitor" watchRunId: string; // the run being watched, usually ctx.input.watchRunId agent: AgentLike | AgentLike[]; // samples and classifies healthOutput: OutputTarget; // must include `condition` and `runStatus` actionOutput?: OutputTarget; // defaults to healthOutput healAgent?: AgentLike | AgentLike[]; // defaults to `agent` intervalMs?: number; // delay BETWEEN beats, default 60000 maxChecks?: number; // default 120 stallBeats?: number; // beats without progress before `stalled` is expected, default 3 autoHeal?: MonitorCondition[]; // default ["stalled", "wedged-node"] handlers?: Partial>; guidance?: string; // repo-specific doctrine appended to the shipped prompt prompt?: string | ReactNode; // replaces the shipped prompt entirely skipIf?: boolean; children?: string | ReactNode; // alias for `prompt` }; ``` ```tsx theme={"theme":{"light":"github-light","dark":"github-dark"}} ``` ## Notes Each beat is a durable ``, one classifier ``, and a `` that routes the classified condition to a handler. Beat 0 skips the timer so the monitor looks immediately. The loop exits when the watched run reaches a terminal status, or after `maxChecks` beats, which is not a failure. `` is the router rather than `` because a monitor has exactly one subject and needs first-match, one-handler-wins semantics. `` classifies many items and fans every category out in ``, which is a different problem. The `healthOutput` schema must carry `condition` and `runStatus`; `targetNodeId`, `evidence`, and `summary` are read when present. It is `targetNodeId` rather than `nodeId` because `nodeId`, `runId`, and `iteration` are reserved output columns. Only `stalled` and `wedged-node` heal without a human by default, because resuming a run and retrying a node are both idempotent and reversible. Every other condition escalates through a durable human request. Add a condition to `autoHeal` to grant broader authority: adding `runaway-loop` swaps its handler from "escalate" to "cancel the run", which is destructive and therefore never a default. `handlers` replaces any element outright, and mapping one to `null` makes that condition a no-op. `healthy` contributes no rule to the table at all, so a healthy run routes to nothing. Observing and continuing to watch is the correct outcome, not a gap. The prompt shipped with the component tells the agent what healthy and unhealthy look like, what evidence to gather before acting, what it may do autonomously, and when it must escalate instead of guessing. It also binds the agent to reading run state through `smithers-orchestrator/gateway-client` or the public CLI, never the store. It is available as `monitorPrompt()` and as an MDX prompt component shipped beside the source, so a monitor file can import and extend it. Most monitors are declared as files rather than composed by hand. See [Monitor workflows](/guides/monitor-workflows) for the `.smithers/monitor/.tsx` layout, auto-discovery, the `--monitor` and `--no-monitor` flags, and the run lifecycle. ## Source The `` implementation and the files it imports, straight from the package source. This section is generated; edit the source, not this block. ```js Monitor.js theme={"theme":{"light":"github-light","dark":"github-dark"}} // @smithers-type-exports-begin /** @typedef {import("./MonitorCondition.ts").MonitorCondition} MonitorCondition */ /** @typedef {import("./MonitorProps.ts").MonitorProps} MonitorProps */ // @smithers-type-exports-end import React from "react"; import { Sequence } from "./Sequence.js"; import { Task } from "./Task.js"; import { Timer } from "./Timer.js"; import { Loop } from "./Ralph.js"; import { HumanTask } from "./HumanTask.js"; import { DecisionTable } from "./DecisionTable.js"; import { useOptionalSmithersContext } from "./useOptionalSmithersContext.js"; import { MONITOR_CONDITIONS, MONITOR_DEFAULT_AUTO_HEAL, MONITOR_TERMINAL_STATUSES, monitorPrompt, } from "./monitorPrompt.js"; /** * — a heartbeat health check that watches ONE other run and keeps it * healthy. * * Shape, per beat: `` paces the loop, one agent `` samples the * watched run and classifies it into exactly one {@link MonitorCondition}, and * a `` routes that condition to its handler. * * `` is the router rather than `` because the * two solve different problems: `` classifies MANY items and * fans every category out in ``, while a monitor has exactly one * subject (the watched run) and needs first-match, one-handler-wins semantics — * which is precisely ``. Routing here is * deterministic: the agent names a condition, the table picks the handler. * * The healing half is deliberately timid. Only `stalled` and `wedged-node` heal * without a human by default, because resuming a run and retrying a node are * the two repairs that are both idempotent and reversible. Everything else * escalates through a durable ``. Put a condition in `autoHeal` to * grant broader authority; pass `handlers` to replace any element outright, or * map one to `null` to make it a no-op. * * The monitor never reads the store: its prompt binds it to the Gateway client * and the public CLI surface. * * @param {MonitorProps} props */ export function Monitor(props) { if (props.skipIf) return null; const ctx = useOptionalSmithersContext(); const prefix = props.id ?? "monitor"; const intervalMs = props.intervalMs ?? 60_000; const maxChecks = props.maxChecks ?? 120; const stallBeats = props.stallBeats ?? 3; const autoHeal = props.autoHeal ?? MONITOR_DEFAULT_AUTO_HEAL; const healAgent = props.healAgent ?? props.agent; const actionOutput = props.actionOutput ?? props.healthOutput; const watchRunId = props.watchRunId; const checkId = `${prefix}-check`; // `latest` (not `outputMaybe`) is the correct reader for a loop's exit // condition and for routing: it resolves the newest iteration's row, so the // table routes on THIS beat's verdict rather than beat 0's. const health = ctx?.latest?.(props.healthOutput, checkId); const condition = typeof health?.condition === "string" ? health.condition : undefined; // A monitor must never outlive the run it watches. The loop's own exit is the // watched run reaching a terminal status; the CLI tears the monitor down as // well, so this is the graceful path, not the only one. const watchedRunFinished = MONITOR_TERMINAL_STATUSES.includes( /** @type {(typeof MONITOR_TERMINAL_STATUSES)[number]} */ (health?.runStatus), ); const beat = ctx?.iterations?.[`${prefix}-loop`] ?? ctx?.iteration ?? 0; const promptNode = props.prompt ?? props.children ?? monitorPrompt({ watchRunId, intervalMs, stallBeats, autoHeal, guidance: props.guidance }); const checkTask = React.createElement(Task, { id: checkId, output: props.healthOutput, agent: props.agent, // A sampling beat that fails must not kill the monitor: the next beat // re-samples, and a dead monitor is worse than a missed reading. continueOnFail: true, label: `Monitor beat ${beat + 1}: ${watchRunId}`, children: promptNode, }); /** * Build one handler ``. Handlers are named per condition so an operator * reading `bunx smithers-orchestrator inspect` on the monitor run can see which repair fired. * @param {MonitorCondition} name * @param {string} instruction */ const handlerTask = (name, instruction) => React.createElement(Task, { id: `${prefix}-${name}`, output: actionOutput, agent: healAgent, continueOnFail: true, needs: { health: checkId }, deps: { health: props.healthOutput }, label: `Monitor: ${name}`, children: (/** @type {Record} */ d) => [ `The heartbeat classified run ${watchRunId} as "${name}".`, "", `Health sample:\n${JSON.stringify(d.health ?? health ?? "(no sample)")}`, "", instruction, "", "Read run state only through `smithers-orchestrator/gateway-client` or the public CLI. Never open the store.", "Report exactly what you did and what changed. If the action did not change the symptom, say so plainly rather than trying something else.", ].join("\n"), }); /** * Escalation is a durable human request on the MONITOR run, so it survives a * restart and shows up in `bunx smithers-orchestrator human list` / the Gateway. * @param {MonitorCondition} name * @param {string} ask */ const escalate = (name, ask) => React.createElement(HumanTask, { id: `${prefix}-escalate-${name}`, output: actionOutput, label: `Monitor escalation: ${name}`, prompt: [ `Run ${watchRunId} looks "${name}" and the monitor is not allowed to fix it on its own.`, "", `Evidence: ${JSON.stringify(health?.evidence ?? health?.summary ?? "(none recorded)")}`, `Watched run status: ${health?.runStatus ?? "unknown"}`, health?.targetNodeId ? `Implicated node: ${health.targetNodeId}` : "", "", ask, ] .filter(Boolean) .join("\n"), }); /** @type {Record} */ const defaults = { // Nothing to do. Rendering nothing IS the handler. healthy: null, stalled: autoHeal.includes("stalled") ? handlerTask( "stalled", `Resume the run: \`bunx smithers-orchestrator up --resume --run-id ${watchRunId}\` (idempotent — it re-enters the same durable frame). Confirm with \`bunx smithers-orchestrator status ${watchRunId}\` that events resumed. Do nothing else.`, ) : escalate("stalled", "Should this run be resumed?"), "wedged-node": autoHeal.includes("wedged-node") ? handlerTask( "wedged-node", `Retry the single wedged node: \`bunx smithers-orchestrator retry-task ${watchRunId} --node \` using \`targetNodeId\` from the health sample (it adds an attempt and discards nothing). Retry it once, then report. Do not retry a second time — a node that wedges again needs a human.`, ) : escalate("wedged-node", "Should this node be retried, or is the failure real?"), // Cancelling is destructive, so it ships behind an explicit opt-in: it only // becomes the handler when the author puts "runaway-loop" in `autoHeal`. "runaway-loop": autoHeal.includes("runaway-loop") ? handlerTask( "runaway-loop", `Cancel the runaway run: \`bunx smithers-orchestrator cancel ${watchRunId}\`. Record the loop id, its iteration count, and the token burn that justified cancelling BEFORE you cancel.`, ) : escalate( "runaway-loop", "The loop is iterating without converging. Cancel the run, let it keep going, or change its exit condition?", ), // A parked run is healthy; surface it to the operator, never answer for them. "awaiting-human": handlerTask( "awaiting-human", "Do NOT resolve the approval or answer the human request. Report what is pending, who it is waiting on, and how long it has been parked, so the operator can act.", ), failing: handlerTask( "failing", "The run failed. Gather the failing node, its error, and the tail of the event log, and report them. Do not retry, rewind, or edit anything.", ), unknown: escalate( "unknown", "The monitor could not read consistent evidence and will not guess. What should it do next?", ), }; const resolved = /** @type {Record} */ ({ ...defaults, ...props.handlers, }); // First-match over the closed condition set. `healthy` (and any handler an // author mapped to `null`) contributes no rule, so it falls through to the // table's `default` — do nothing, keep watching. const rules = MONITOR_CONDITIONS.filter((name) => resolved[name] != null).map((name) => ({ when: condition === name, then: /** @type {React.ReactElement} */ (resolved[name]), label: name, })); const router = React.createElement(DecisionTable, { id: `${prefix}-route`, rules, strategy: "first-match", default: undefined, }); return React.createElement( Loop, { id: `${prefix}-loop`, until: watchedRunFinished, maxIterations: maxChecks, // Running out of beats is the monitor's job ending, not a failure. onMaxReached: "return-last", }, React.createElement( Sequence, null, React.createElement(Timer, { id: `${prefix}-beat`, duration: `${Math.max(0, Math.round(intervalMs))}ms`, // Sample immediately on the first beat; pace every beat after it. skipIf: beat === 0, }), checkTask, router, ), ); } ``` ```js Sequence.js theme={"theme":{"light":"github-light","dark":"github-dark"}} import React from "react"; /** @typedef {import("./SequenceProps.ts").SequenceProps} SequenceProps */ /** * @param {SequenceProps} props */ export function Sequence(props) { if (props.skipIf) return null; // Sequence carries only a display label; pass a sanitized bag (align with // the sanitizing structural components) so control props don't leak through. // `label` names the phase group in run views (graph, the Claude /workflows // mirror) and is preserved in the persisted frame XML. const next = { ...(props.label === undefined ? {} : { label: props.label }), ...(props.failurePolicy === undefined ? {} : { failurePolicy: props.failurePolicy }), }; return React.createElement("smithers:sequence", next, props.children); } ``` ```js Task.js theme={"theme":{"light":"github-light","dark":"github-dark"}} // @smithers-type-exports-begin /** * @template D * @typedef {import("./InferDeps.ts").InferDeps} InferDeps */ /** @typedef {import("./OutputTarget.ts").OutputTarget} OutputTarget */ // @smithers-type-exports-end import { applyCliToolAllowlist } from "./cliToolAllowlist.js"; import { createTaskComponent } from "./taskCore.js"; export { renderPromptToText } from "./taskCore.js"; /** * The Node/CLI-agent-aware `Task`. Every render-path behavior (deps * resolution, agent-chain assembly, MDX prompt rendering, static/compute/agent * branching) lives in `taskCore.js`; this file only supplies the CLI-agent * tool-allowlist enforcement step (`applyCliToolAllowlist`, which statically * imports `ClaudeCodeAgent`/`PiAgent`/`GeminiAgent`/`AntigravityAgent` — see * `cliToolAllowlist.js`). `Task.browser.js` builds the same component with a * no-op allowlist step instead, so it never pulls those Node-only * (`node:child_process`-backed) classes into a browser bundle. */ export const Task = createTaskComponent({ applyCliToolAllowlist }); ``` ```js Timer.js theme={"theme":{"light":"github-light","dark":"github-dark"}} import React from "react"; import { SmithersError } from "@smithers-orchestrator/errors/SmithersError"; /** @typedef {import("./TimerProps.ts").TimerProps} TimerProps */ /** * @param {TimerProps} props */ export function Timer(props) { if (props.skipIf) return null; const hasDuration = typeof props.duration === "string" && props.duration.trim().length > 0; const hasUntil = props.until !== undefined && props.until !== null && String(props.until).trim().length > 0; if ((hasDuration ? 1 : 0) + (hasUntil ? 1 : 0) !== 1) { throw new SmithersError("INVALID_INPUT", ` requires exactly one of "duration" or "until".`); } if (props.every !== undefined) { throw new SmithersError( "INVALID_INPUT", ` does not support "every" yet. Recurring timers ship in phase 2.`, ); } const untilIso = props.until instanceof Date ? props.until.toISOString() : typeof props.until === "string" ? props.until : undefined; const timerMeta = { timer: true, ...(hasDuration ? { duration: props.duration } : {}), ...(hasUntil ? { until: untilIso } : {}), ...props.meta, }; return React.createElement("smithers:timer", { id: props.id, key: props.key, duration: props.duration, until: untilIso, dependsOn: props.dependsOn, needs: props.needs, label: props.label ?? `timer:${props.id}`, meta: Object.keys(timerMeta).length > 0 ? timerMeta : undefined, __smithersTimerDuration: props.duration, __smithersTimerUntil: untilIso, }); } ``` ```js Ralph.js theme={"theme":{"light":"github-light","dark":"github-dark"}} // @smithers-type-exports-begin /** @typedef {import("./RalphProps.ts").RalphProps} RalphProps */ // @smithers-type-exports-end import React from "react"; /** @typedef {import("./LoopProps.ts").LoopProps} LoopProps */ /** * @param {LoopProps} props */ export function Loop(props) { if (props.skipIf) return null; // Sanitize to the loop's host props (align with other structural components); // key/skipIf are React/control props and children are passed separately. const next = { id: props.id, until: props.until, maxIterations: props.maxIterations, onMaxReached: props.onMaxReached, continueAsNewEvery: props.continueAsNewEvery, }; return React.createElement("smithers:ralph", next, props.children); } /** @deprecated Use `Loop` instead. */ export const Ralph = Loop; ``` ```js HumanTask.js theme={"theme":{"light":"github-light","dark":"github-dark"}} import React from "react"; import { renderPromptToText } from "./Task.js"; import { getTaskRuntime } from "@smithers-orchestrator/driver/task-runtime"; import { SmithersDb } from "@smithers-orchestrator/db/adapter"; import { buildHumanRequestId } from "@smithers-orchestrator/db/buildHumanRequestId"; import { SmithersError } from "@smithers-orchestrator/errors/SmithersError"; /** @typedef {import("./HumanTaskProps.ts").HumanTaskProps} HumanTaskProps */ /** * @param {unknown} value * @returns {value is import("zod").ZodObject} */ function isZodObject(value) { return Boolean(value && typeof value === "object" && "shape" in value); } /** * @param {HumanTaskProps} props * @returns {React.ReactElement | null} */ export function HumanTask(props) { if (props.skipIf) return null; const maxAttempts = props.maxAttempts ?? 10; const outputSchema = props.outputSchema ?? (isZodObject(props.output) ? props.output : undefined); const promptText = renderPromptToText(props.prompt); const humanMeta = { humanTask: true, maxAttempts, prompt: promptText, ...props.meta, }; /** * @returns {Promise} */ const computeHumanInput = async () => { const runtime = getTaskRuntime(); if (!runtime) { throw new SmithersError( "HUMAN_TASK_OUTSIDE_RUNTIME", "HumanTask can only be resolved while a Smithers task is executing.", ); } const adapter = new SmithersDb(runtime.db); const requestId = buildHumanRequestId(runtime.runId, props.id, runtime.iteration); const humanRequest = await adapter.getHumanRequest(requestId); const approval = await adapter.getApproval(runtime.runId, props.id, runtime.iteration); let rawInput = humanRequest?.responseJson ?? null; if ( rawInput == null && humanRequest?.status !== "cancelled" && humanRequest?.status !== "expired" && typeof approval?.note === "string" ) { rawInput = approval.note; await adapter.answerHumanRequest( requestId, rawInput, approval.decidedAtMs ?? Date.now(), approval.decidedBy ?? null, ); } if (rawInput == null) { if (humanRequest?.status === "cancelled") { throw new SmithersError("HUMAN_TASK_CANCELLED", `Human input for task "${props.id}" was cancelled.`); } throw new SmithersError("HUMAN_TASK_NO_INPUT", `No human input received for task "${props.id}".`); } let parsed; try { parsed = typeof rawInput === "string" ? JSON.parse(rawInput) : rawInput; } catch { throw new SmithersError("HUMAN_TASK_INVALID_JSON", `Human input for task "${props.id}" is not valid JSON.`); } // Validate against output schema if provided if (outputSchema) { const result = outputSchema.safeParse(parsed); if (!result.success) { throw new SmithersError( "HUMAN_TASK_VALIDATION_FAILED", `Human input for task "${props.id}" does not match the output schema: ${result.error.message}`, ); } return result.data; } return parsed; }; return React.createElement("smithers:task", { id: props.id, key: props.key, output: props.output, outputSchema, dependsOn: props.dependsOn, needs: props.needs, needsApproval: true, waitAsync: props.async === true, approvalMode: "decision", timeoutMs: props.timeoutMs, retries: maxAttempts - 1, retryPolicy: { backoff: "fixed", initialDelayMs: 0 }, continueOnFail: props.continueOnFail, label: props.label ?? `human:${props.id}`, meta: humanMeta, __smithersKind: "human", __smithersComputeFn: computeHumanInput, }); } ``` ```js DecisionTable.js theme={"theme":{"light":"github-light","dark":"github-dark"}} // @smithers-type-exports-begin /** @typedef {import("./DecisionRule.ts").DecisionRule} DecisionRule */ /** @typedef {import("./DecisionTableProps.ts").DecisionTableProps} DecisionTableProps */ // @smithers-type-exports-end import React from "react"; import { Branch } from "./Branch.js"; import { Parallel } from "./Parallel.js"; /** * Structured deterministic routing. Replaces deeply nested Branches with a * flat, declarative rule table. * * - `"first-match"` builds nested Branch elements so the first matching rule wins. * - `"all-match"` gathers all matching rules' `then` elements into a Parallel. * * Composes Branch and Parallel internally. * @param {DecisionTableProps} props */ export function DecisionTable(props) { if (props.skipIf) return null; const { rules, strategy = "first-match" } = props; if (strategy === "all-match") { const matching = rules.filter((r) => r.when).map((r) => r.then); if (matching.length === 0) { return props.default ?? null; } return React.createElement(Parallel, { id: props.id }, ...matching); } // "first-match": build nested Branches from the last rule backward. // The innermost else is the default fallback. let fallback = props.default ?? null; for (let i = rules.length - 1; i >= 0; i--) { const rule = rules[i]; fallback = React.createElement(Branch, { if: rule.when, then: rule.then, else: fallback, }); } return fallback; } ``` ```js useOptionalSmithersContext.js theme={"theme":{"light":"github-light","dark":"github-dark"}} import React from "react"; import { SmithersContext } from "@smithers-orchestrator/react-reconciler/context"; /** * Read the workflow context when React is rendering the component, but allow * direct structural expansion tests to call composite wrappers as plain * functions. In that direct-call path React throws the standard invalid hook * error; treating it as "no context yet" preserves the static element shape. */ export function useOptionalSmithersContext() { try { return React.useContext(SmithersContext); } catch (error) { const message = error instanceof Error ? error.message : String(error); if (/Invalid hook call|dispatcher\.useContext|useContext/i.test(message)) { return null; } throw error; } } ``` ```js monitorPrompt.js theme={"theme":{"light":"github-light","dark":"github-dark"}} /** @typedef {import("./MonitorCondition.ts").MonitorCondition} MonitorCondition */ /** * The monitoring doctrine `` ships with. * * Prompt text lives IN this package (a seeded workflow pack cannot import * `.smithers/prompts`), the same way `delegation/delegationPrompts.js` does. * `MonitorPrompt.mdx` renders these exact sections as an MDX prompt component * so a monitor file can import and extend it with JSX; keeping the strings here * means the `.mdx` and the component default can never drift apart. */ /** * The closed condition set, as a runtime value. Kept in the same order the * router evaluates it so the prompt and the switch always agree. * @type {readonly MonitorCondition[]} */ export const MONITOR_CONDITIONS = /** @type {const} */ ([ "healthy", "stalled", "wedged-node", "runaway-loop", "awaiting-human", "failing", "unknown", ]); /** Run statuses that mean the watched run is over and the monitor should stop. */ export const MONITOR_TERMINAL_STATUSES = /** @type {const} */ (["finished", "failed", "cancelled", "continued"]); /** * The conditions a monitor may repair on its own by default: the only two whose * repair is both idempotent (running it twice equals running it once) and * reversible (the run's durable state is unchanged if it was the wrong call). * @type {readonly MonitorCondition[]} */ export const MONITOR_DEFAULT_AUTO_HEAL = /** @type {const} */ (["stalled", "wedged-node"]); /** * How to read run state. This is an architectural rule, not a preference: the * store is private to the engine, and a monitor that opens it races the very * run it is watching. * @returns {string[]} */ export function monitorReadPathRules() { return [ "Read run state ONLY through the Gateway client (`smithers-orchestrator/gateway-client`) or the public CLI: `bunx smithers-orchestrator status`, `bunx smithers-orchestrator inspect RUN_ID --format json`, `bunx smithers-orchestrator events RUN_ID`, `bunx smithers-orchestrator node RUN_ID --node NODE_ID`, `bunx smithers-orchestrator why RUN_ID`, `bunx smithers-orchestrator ps`.", "NEVER open the store directly. Do not read `.smithers/*.db`, `.smithers/pg`, `smithers.db`, or any SQL over them, and do not parse the Gateway runtime state file. Those are private engine state; reading them races the run and is wrong even when it appears to work.", "If a read fails or returns nothing, that is evidence of `unknown` — not a licence to guess from stale memory.", ]; } /** * What healthy and unhealthy actually look like, stated concretely enough that * two different agents sampling the same run reach the same verdict. * @param {{ stallBeats?: number; intervalMs?: number }} [params] * @returns {string[]} */ export function monitorHealthSignals(params = {}) { const beats = params.stallBeats ?? 3; const intervalMs = params.intervalMs ?? 60_000; const stallMs = beats * intervalMs; return [ `HEALTHY looks like: the run's status is running (or a terminal ${MONITOR_TERMINAL_STATUSES.join("/")}); the newest event is more recent than ${stallMs}ms; at least one node changed state, produced output, or emitted events since the previous heartbeat; attempt counts are flat or rising by at most one per node; token burn is roughly linear against progress.`, "HEALTHY also covers a run that is legitimately slow: a single long agent task that is still streaming events is working, not stalled. Elapsed time alone is never unhealthy.", `STALLED looks like: status is running or paused, no new event for more than ${stallMs}ms, no pending approval and no open human request, and no node in a waiting-timer/waiting-event state that explains the silence.`, "WEDGED-NODE looks like: one node has burned repeated attempts whose error signatures are the same (or trivially different) each time, while the rest of the graph is idle behind it. Different errors each attempt is a node making progress through a problem, not a wedge.", "RUNAWAY-LOOP looks like: a Loop's iteration count is climbing while its exit condition is no closer, or token burn is accelerating with no matching node completions. Iterations alone are not runaway; iterations without convergence are.", "AWAITING-HUMAN looks like: a pending approval gate, an open human request, or a waiting-approval status. This is a healthy parked run, NOT a failure — it needs a person, not a repair.", "FAILING looks like: the run status is failed, or a node without continueOnFail has exhausted its retries and nothing downstream can proceed.", "UNKNOWN is the honest verdict when reads failed, the evidence contradicts itself, or nothing above fits. Prefer unknown over a confident wrong classification.", ]; } /** * The evidence contract: what must be in hand BEFORE a condition is named or an * action is taken. A monitor that acts on one reading is a monitor that flaps. * @returns {string[]} */ export function monitorEvidenceRules() { return [ "Gather, every heartbeat, before classifying: the run status; each node's state and attempt/retry count; the timestamp of the newest event (stall time); cumulative token usage; pending approvals and open human requests; and the error signatures in the most recent events.", "Quote the evidence you used in `evidence` — node ids, attempt numbers, timestamps, the actual error text. A classification with no citable evidence is an `unknown`.", "Never act on a single sample. A condition must hold across at least two consecutive heartbeats before you route it to a healing handler; say so in `evidence` when it does.", "Compare against the PREVIOUS heartbeat, not against your expectations. 'Nothing changed since last beat' is the observation that matters.", ]; } /** * What the monitor may do by itself, and where the line is. Biased hard toward * observing and reporting: the monitor's job is to keep the run alive, not to * take over from it. * @param {{ autoHeal?: readonly MonitorCondition[]; watchRunId?: string }} [params] * @returns {string[]} */ export function monitorAuthorityRules(params = {}) { const autoHeal = params.autoHeal ?? MONITOR_DEFAULT_AUTO_HEAL; const runId = params.watchRunId ?? "RUN_ID"; const allowed = autoHeal.length > 0 ? autoHeal.join(", ") : "(nothing — report only)"; return [ "Your default action is to OBSERVE AND REPORT. Doing nothing and watching another beat is a correct, complete answer, and it is the right one whenever you are unsure.", `You may act autonomously ONLY on these conditions: ${allowed}. Every other condition must be escalated to a human.`, `Every repair you are allowed to make is idempotent and reversible: resuming a stalled run (\`bunx smithers-orchestrator up WORKFLOW --resume --run-id ${runId}\`) re-enters the same durable frame, and retrying a wedged node (\`bunx smithers-orchestrator retry-task ${runId} --node NODE_ID\`) adds an attempt without discarding prior state. Running either twice is harmless.`, "You may NEVER take a destructive or irreversible action on your own. Do not cancel a run, do not rewind, revert, or time-travel it, do not resolve an approval or answer a human request on the human's behalf, and do not edit the repository. Those require a human decision, always.", "ESCALATE to a human instead of guessing when: the condition is not in your auto-heal set; the evidence is contradictory or unreadable; a repair you already applied did not change the symptom; the same condition recurs after two repairs; or the correct fix would be destructive.", "When you escalate, state what you saw, what you already tried, what you believe is wrong, and the single specific decision you need. Do not ask an open question.", ]; } /** * Assemble the full monitoring prompt. * * @param {{ * watchRunId?: string; * intervalMs?: number; * stallBeats?: number; * autoHeal?: readonly MonitorCondition[]; * guidance?: string; * }} [params] * @returns {string} */ export function monitorPrompt(params = {}) { const runId = params.watchRunId ?? "RUN_ID"; const sections = [ "# Monitor one Smithers run", `You are the heartbeat health check for run \`${runId}\`. You run alongside it as a sibling run, sample its health on an interval, classify what you see into exactly one condition, and let the workflow route that condition to a handler. You are not doing the run's work and you must not interfere with it.`, "## How to read the run", ...monitorReadPathRules().map((line) => `- ${line}`), "## Healthy vs unhealthy", ...monitorHealthSignals(params).map((line) => `- ${line}`), "## Evidence you must gather first", ...monitorEvidenceRules().map((line) => `- ${line}`), "## What you may do, and when you must ask", ...monitorAuthorityRules({ autoHeal: params.autoHeal, watchRunId: runId }).map((line) => `- ${line}`), "## Output", `Return exactly one condition from: ${MONITOR_CONDITIONS.join(" | ")}. Include \`runStatus\` (the watched run's status as reported by the CLI/Gateway), \`targetNodeId\` when a single node is implicated, \`evidence\` quoting what you actually read, and a one-line \`summary\`. When the watched run has reached ${MONITOR_TERMINAL_STATUSES.join("/")}, report it with condition \`healthy\` (or \`failing\` if it failed) and the monitor will stop.`, ]; if (params.guidance) sections.push("## Repo-specific guidance", params.guidance); return sections.join("\n\n"); } ``` ```ts MonitorCondition.ts theme={"theme":{"light":"github-light","dark":"github-dark"}} /** * The closed set of conditions a `` heartbeat may classify a watched * run into. Deliberately small: a monitor that can name a hundred states is a * monitor nobody can route deterministically. Every sample resolves to exactly * one of these, and each one maps to exactly one handler. * * - `healthy` — the run is progressing (or has finished cleanly). Do nothing. * - `stalled` — no event for longer than the stall budget, nothing waiting on a human. * - `wedged-node` — one node is burning attempts/retries without changing its error. * - `runaway-loop` — a loop or token burn is growing without approaching its exit condition. * - `awaiting-human` — the run is parked on an approval gate or human request. * - `failing` — the run (or a non-continueOnFail node) has failed terminally. * - `unknown` — evidence was unreadable or contradictory. Never guess from here. */ export type MonitorCondition = | "healthy" | "stalled" | "wedged-node" | "runaway-loop" | "awaiting-human" | "failing" | "unknown"; ``` ```ts MonitorProps.ts theme={"theme":{"light":"github-light","dark":"github-dark"}} import type React from "react"; import type { AgentLike } from "@smithers-orchestrator/agents/AgentLike"; import type { MonitorCondition } from "./MonitorCondition.ts"; import type { OutputTarget } from "./OutputTarget.ts"; export type MonitorProps = { /** ID prefix for generated task/component ids. Default `"monitor"`. */ id?: string; /** Run id this monitor watches. Usually `ctx.input.watchRunId`. */ watchRunId: string; /** Agent that samples the watched run and classifies its health. */ agent: AgentLike | AgentLike[]; /** * Output target for the heartbeat classification. The schema must carry at * least `condition` (a {@link MonitorCondition}) and `runStatus`; * `targetNodeId`, `evidence`, and `summary` are read when present. `nodeId`, * `runId`, and `iteration` are reserved output columns — hence `targetNodeId`. */ healthOutput: OutputTarget; /** Output target handler tasks write to. Defaults to `healthOutput`. */ actionOutput?: OutputTarget; /** Agent the healing/reporting handlers run on. Defaults to `agent`. */ healAgent?: AgentLike | AgentLike[]; /** Wall-clock delay between heartbeats, enforced by a durable ``. Default 60000. */ intervalMs?: number; /** Maximum heartbeats before the monitor stops watching. Default 120. */ maxChecks?: number; /** Heartbeats with no progress before `stalled` is the expected verdict. Default 3. */ stallBeats?: number; /** * Conditions the monitor may heal WITHOUT asking a human. Anything not listed * escalates instead. Default `["stalled", "wedged-node"]` — the two repairs * that are idempotent and reversible. */ autoHeal?: MonitorCondition[]; /** * Override the element rendered for a condition. `null` means "do nothing and * keep watching" for that condition; omit a key to keep the shipped default. */ handlers?: Partial>; /** Extra repo-specific doctrine appended to the shipped monitoring prompt. */ guidance?: string; /** Replace the shipped monitoring prompt entirely. Prefer `guidance` first. */ prompt?: string | React.ReactNode; skipIf?: boolean; /** Alias for `prompt`. */ children?: string | React.ReactNode; }; ```