Skip to main content

​
Herdr

Herdr is a terminal workspace manager. Smithers can mirror a run into a herdr workspace so you can watch the whole run at a glance, drop into any node, answer a parked gate in place, and take an agent over by hand, without giving up any of Smithers’ guarantees. Herdr is a mirrored presentation and steering plane. Smithers keeps owning everything that matters: it spawns the agents, isolates them, records the durable event log, and resumes after a crash. Herdr only renders that run and lets you steer it. The mirror is fed from Smithers’ event stream, so the herdr sidebar shows authoritative status pushed from Smithers rather than screen-scraped guesses. The integration is optional and degradable. If no herdr server is running, --herdr warns once on stderr and the run proceeds unchanged. If herdr dies mid-run, every call soft-fails and the run finishes normally. No herdr code is on the engine’s hot path.

​
Quickstart

Install and start a herdr server (see the herdr docs for platform details), then mirror a run:
--herdr targets herdr’s default session. Pass --herdr=SESSION to target a named session, or set HERDR_SESSION / HERDR_SOCKET_PATH (herdr’s own env) to pick the socket.

​
The adaptive layout

Smithers maps a run onto herdr’s own hierarchy (workspace, then tabs, then panes) so the workspace reads like a run and each tab reads like one thing you might need to look at.
  • One workspace per attached run, labelled WORKFLOW_ID [smithers:v1:ENCODED_RUN_ID]. The versioned suffix is Smithers’ ownership marker. The complete label is the find-or-create key, so a re-attach or resume reuses the same workspace without adopting an operator-created label that merely mentions the same run id.
  • The workspace’s first tab is the cockpit. By default it is a split: left = your harness shell (or a spawned harness), right = portable smithers supervisor (top --db <store> - long-lived fleet board; see The overview board and the workflow supervisor guide). Operator-owned workspaces can pre-split and dock top into the right pane without renaming the workspace.
  • Each mirrored node gets its own tab, labelled with the node id, holding one full-size pane running that node’s live output (tail RUN_ID --node NODE_ID --hud --linger) with a fixed bottom dual-control dock (s steer · h hijack · q close). Because only one tab renders at a time, every node pane gets the full terminal area instead of a shrinking sliver. The pane is named smithers:RUN_ID:NODE_ID.
Why tabs rather than splitting one tab into many panes: herdr splits divide a single shared area, so panes halve with each new node (a fourth node is already a narrow column, a sixth is an unreadable two-column sliver). Tabs each get the full area regardless of how many exist, so a tab per node stays legible where splits do not.

​
The adaptive cap

Auto-spawning a tab per node breaks down under fan-out: a swarm over dozens of issues would open dozens of tabs, most idle then gone. So the mirror uses an adaptive cap, tabCap (default 6 tabs including the overview), which leaves room for five mirrored node tabs. Past the cap an ordinary node stays unpaned and issues no herdr calls at all, so a large fan-out does not flood the tab bar or the socket. The overview tab always reflects every node (see the queue summary), and you can pull any unpaned node up on demand with herdr open.

​
Attention promotion (always bypasses the cap)

The cap only holds back ordinary nodes. Three kinds of node are attention-worthy and always get a tab, even past the cap:
  • Parked approval gates (NodeWaitingApproval / ApprovalRequested). A pure gate node carries no agent attempt row, so the ordinary agent-only pane filter would drop it; because a parked human gate is exactly what you need to see, it overrides both the filter and the cap and its pane runs approve --watch so you can decide in place.
  • Failed nodes (NodeFailed on the final attempt). A worker that fails deep in a swarm is promoted out of the unpaned pool into its own tab so the one thing that went wrong is visible without hunting for it.
  • Hijack panes (see Hijack into a pane).
Loop iterations reuse their node’s tab rather than opening a new one per iteration, so a validation loop that retries three times is one tab, not three. Focus is never stolen by any of this: a background gate or failure lights up the sidebar and fires a notification but never yanks your screen off what you are looking at. The one deliberate exception is a hijack pane, which does focus, because you asked to drive that agent right now. Everything in the mirror is soft-fail, replay-idempotent, cross-run-filtered, and seq-monotonic: a re-attach or resume adopts the tabs and panes already there instead of duplicating them, stale events from other runs are filtered out, and out-of-order pushes are dropped rather than flapping a pane’s status backward.

​
Status mapping

Smithers pushes each pane’s status from the run’s event stream. blocked always means “needs a human”, so approvals and failures both surface as blocked (with a notification), which is what makes the herdr sidebar a useful worklist. The gate question is pushed both as the pane’s status message and as its queryable custom_status, so herdr agent list shows what needs approving, not just that the pane is blocked. When such a gate clears, its pane resolves to idle with the custom status approved rather than the working/done an ordinary agent node reports. Because bunx smthrs up exits when it parks at a gate (exit code 3, awaiting a decision), you approve and resume in a new process. On resume Smithers re-adopts the parked gate pane and re-flags it, so the pane moves from blocked to approved when the resumed run clears the gate instead of staying stuck blocked in the mirror. When an agent reports a resumable session id, Smithers also forwards it to the pane so herdr can associate the pane with the agent’s session.

​
Outcome labels

When the run reaches a terminal state, Smithers renames its workspace with an outcome marker while keeping the run id, so the sidebar shows the result at a glance: The rename is derived from the clean label and never stacks markers on replay, and find-or-create/attach re-adopt a renamed workspace (the marker prefix is tolerated when matching), so a resumed run reuses its marked workspace rather than opening a fresh one. Panes still linger after the run ends (below), so a ✓ or ✗ workspace is a finished run you can still read; close it yourself, or sweep every terminal run’s workspace at once with herdr clean.

​
The supervision kit

The mirror is most useful with a few herdr settings turned on, so a parked gate or a failed worker rises to the top of the sidebar on its own instead of waiting to be found. Configure these in your herdr config (herdr --default-config prints the full template):
  • Sort the agent panel by attention. Set agent_panel_sort = "priority" in [ui]. This flips the agent list from grouped-by-workspace to a flat queue ordered blocked, done, working, idle, unknown. Since Smithers maps every parked gate and every failure to blocked, and herdr rolls blocked up over working/idle/unknown at the workspace level, the run that needs you is first in the list.
  • Filter to what is blocked with prefix+g. The session navigator opens on prefix+g and gives a searchable tree of every workspace, tab, and pane. Press / to fuzzy-find by text, or b / w / i / d to filter to blocked / working / idle / done panes. prefix+g then b is a one-keystroke “show me only what is waiting on a human” across every mirrored run.
  • Turn on notifications. Smithers fires a herdr notification when a gate parks, a node fails, a hijack is ready, and when a run ends. herdr only surfaces these if toasts are enabled: set [ui.toast] delivery = "herdr" for in-UI toasts (or system for desktop notifications). Then prefix+o (open_notification_target) jumps focus straight to the pane that raised the most recent toast, so an alert is one keystroke from the thing that raised it.
With those on, the day-to-day loop is: leave herdr on the priority agent panel, let runs mirror themselves, and act only when a row goes blocked, either by clicking it, pressing prefix+g b, or following a toast with prefix+o.

​
Answering a gate in-pane

A parked approval gate is promoted to its own tab whose pane runs an interactive watcher:
approve --watch polls the run for pending approval gates and human requests (ask / confirm / select / json) and answers each one by keystroke as it appears (y / n for a gate, a picker or prompt for a human request), committing through the exact same engine machinery as the single-shot approve / deny / human answer commands. After a decision it lingers and waits for the next gate, so one pane walks a multi-gate run end to end. It is a standalone command; you can run it in any terminal, and the herdr gate tab is simply where the mirror runs it for you. The single-shot forms are unchanged and still the right tool from a script or a one-off terminal:

​
The overview board

The cockpit right pane hosts smithers supervisor - a portable, long-lived workflow board (not a scrolling log). Herdr only places that process; the board itself is a standalone CLI surface. Full guide: workflow supervisor.
What you see at a glance:
  • Fleet - multi-run strip when the store has more than one run (j/k to focus).
  • Header - run id, status, elapsed, tallies (working / blocked / failed / done).
  • Attention - fails, gates, queued steers (empty when calm).
  • Board - per-node status (attention-first); large fan-outs trim to attention + queue summary.
  • Digest - deterministic progress (~30s), no LLM.
  • CTAs - exact approve / steer commands when something needs a human; dual-control keys live on the node tab dock (s steer · h hijack), not on the workflow supervisor outline.

​
The browser stays out of the way

Inside a herdr pane (HERDR_ENV=1), the commands that would normally launch a browser - monitor, ui, gui, and the post-run HTML summary - print their URL instead. The cockpit is already your live view, and a per-command --no-open is useless when an agent runs the command on your behalf.
Nothing is lost: the URL is printed and any report file is still written.

​
Packaging: workflow supervisor vs herdr-only

If you are writing guidance for herdr, document placement and keys around the host. Do not re-document the whole board - link to workflow supervisor.

​
Testing without LLMs

Herdr is one visibility plane. Agent-trace fixtures and core scenarios are shared across HUD, herdr, and other UIs - see Token-free visibility testing.

​
On-demand panes

The cap deliberately leaves most swarm workers unpaned. When you do want to look at a specific one (or re-open a finished node’s output), pull it up on demand:
herdr open places one full-size pane into the run’s mirror workspace. It is find-or-create and outcome-tolerant: an outcome-marked (✓/✗/◻) workspace is re-adopted, and a node that already has a pane is adopted rather than duplicated, so opening the same node twice is idempotent. This is the escape valve for “the mirror never paned that worker, but I need to see it now” and works whether the run is live or already finished.

​
Cleaning up

Finished runs leave their workspaces up on purpose so you can read them. When you are done, sweep them:
herdr clean closes every herdr workspace that mirrors a Smithers run whose run is terminal in the local store. Before closing anything, it requires the versioned ownership marker and an exact match with the workflow and run identity reconstructed from the DB row. Unknown, mismatched, and active-run workspaces are left open. It prints each workspace it closed.

​
Supervision profiles

Workflows differ in how much a human watches them, and the adaptive layout is built to match that spread. Four shapes cover the pack.

​
Bounded and steerable

“I want to watch this whole thing and jump in when it asks.” A short run (hello, research, review, plan) is one to three agent nodes. It fits under the cap comfortably: the overview tab plus a tab per node, and you can read the entire run. The interactive members of this shape (grill-me, a clarify phase) want a focused tab you type into, which is exactly a node tab; nothing steals focus away from it while you answer.

​
Pipeline

“I care about the current stage and the gate, not every token.” A sequential multi-stage run (implement, research-plan-implement, create-workflow, release) is a handful of distinct stages with zero to a few approval gates. Each stage gets a tab, and loop iterations (a validation loop retrying) reuse that stage’s tab instead of piling up. The approval gate is promoted to its own tab running approve --watch, so your attention lands on the gate and the current stage, which is where a pipeline’s decisions live, and the overview tab carries the banner for everything else.

​
Swarm and long-horizon

“I am an exception handler, not a babysitter.” A fan-out over issues, features, or tickets (fix-all-issues, merge-train-all-issues, mission, ralph) runs dozens of mostly-mechanical workers, most with no human gate at all. Here the cap earns its keep: the workers stay unpaned, the overview tab is the dashboard (its queue summary tallies working/blocked/failed/done across the whole swarm), and only attention-worthy nodes take a tab, a failing worker promoted out of the pool, an up-front plan-approval gate, a merge or final-report node. When you want a specific worker, herdr open RUN_ID worker-3 pulls it up; when a worker fails, it is already waiting in its own tab and the sidebar has gone blocked.

​
Co-pilot

“The agent is asking me questions; I want to answer fast.” Interactive workflows put the human upstream of the agents, answering clarify or grill questions rather than approving output. These want a single focused, low-latency tab. Because the mirror never steals focus, you drive the conversation from the node’s tab (reach it with prefix+g or a click), and a parked human request surfaces the same way a gate does: promoted, blocked, and answerable in place with approve --watch.

​
Attach to a run already in flight

bunx smthrs herdr attach mirrors an existing run into herdr and follows it live until the run ends or you press Ctrl-C:
Attach reconciles against existing herdr state: it finds-or-creates the run’s workspace by the deterministic label (tolerating an outcome marker), adopts the overview and node tabs already there, and creates tabs only for nodes that are still active or attention-worthy (long-finished nodes are not replayed). Detaching (run end or Ctrl-C) never closes the workspace. Attaching to an already-terminal run prints a final status line and creates nothing.

​
Check the server

bunx smthrs herdr status pings the server and reports its version, protocol, and whether this client is compatible. It needs no run and no herdr code path beyond the socket:
It exits non-zero with a clear message when no server is reachable.

​
Standalone tail

The panes are just smithers tail, which is a standalone, herdr-free command. Use it directly to follow a run, a single node, or the run-level board from any terminal:
It reads the run’s persisted and live event stream directly from the local store, so it works for detached runs and needs no gateway server. It follows a live run to completion by default and prints a one-line final status, then exits. Pass --linger to keep it open after the run reaches a terminal state until you press q, Enter, or Ctrl-C; this is what mirror panes use so they do not vanish at run end. See the CLI overview for the full flag list. On an interactive node tail (a TTY with --node, live or lingering), the same one-key controls apply - press s to steer this node’s agent, h to hijack it, q (or Ctrl-C) to close - shown as a subtle s steer · h hijack · q close hint (a lingering pane after run end drops the s, since a steer can no longer land once every node is terminal). Piped, --format jsonl, and non-TTY tails are unaffected: no raw mode, no keys.

​
Steer with one key

Steering a live run comes in two gestures, and they differ on one thing: whether the run keeps running.
  • A steer is a short instruction you drop into a running node. The run never stops - the steer is consumed on that node’s next agent step and the agent carries on.
  • A takeover hands you the agent’s live session to drive by hand. The run parks while you drive, and resumes when you hand control back.
In the mirror each is one key on an agent’s node tab. No run id, no node id, no env vars: the pane already knows its own run and node (it is a smithers tail --node) and which herdr session it lives in (herdr exports HERDR_SOCKET_PATH into every pane), so a single keystroke is the whole gesture. The live node tab shows a subtle s steer · h hijack · q close hint; a pane lingering after the run ends drops the s (a steer can no longer land once every node is terminal) and shows h hijack · q close. The overview tab spells the same options as text (in its node tab press s to steer · h to hijack), so you can act from wherever your attention landed. Prefer to type it? smithers steer is the one-word equivalent, with zero env ceremony:
With no run id it auto-picks the single active run (or prompts you among several); a bare smithers steer with no message prompts for one. For a takeover it auto-detects the run’s herdr mirror from the pane/env session and opens the hand-off as a hijack pane there; with no mirror reachable it hands the session off in your current terminal instead. Either way there is nothing to set up - this is the replacement for the old cd … && SMITHERS_HERDR=… SMITHERS_HERDR_HIJACK=1 … hijack --target … incantation.

​
The steering contract

Steer - the run never stops. A queued steer is consumed on the target node’s next agent generate() call - its first start, a retry attempt, or the next loop iteration - and injected as a user message before the structured-output schema wrap, so it never breaks a node that returns JSON. Consumption is recorded in the attempt’s persisted conversation, so a steer is replay-safe: resuming or replaying the run reproduces the identical turn without re-reading the inbox, and never double-injects. A steer that is never consumed - its node already finished, or the run ends first - expires deterministically when the run reaches a terminal state (finished / failed; a cancelled or hijacked run keeps its queued steers so a resume can still consume them), emitting SteerExpired; run-level expiry (rather than per-node) is what keeps a steer aimed at a <Loop> node’s next iteration from being thrown away early. The pane you are watching renders each step as a one-liner: ↪ steer queued: …, ✓ steer consumed by attempt N, ✗ steer expired - node finished first; press h to hijack. Hijack - run-wide today. A hijack is the hijack hand-off: Smithers parks the whole run (its status goes to cancelled) and hands you the target node’s live session; you resume when you are done. Because the hijack aborts the shared run signal, any other agent node that was mid-generate is cancelled and re-runs on resume. So steer --takeover (and the h key) gate on an honest warning: when the run has in-flight agent siblings it prints the exact count - ⚠ Takeover is run-wide: it aborts N in-flight siblings (…); they re-run on resume. - and requires a y confirmation (--yes to skip; on a non-TTY without --yes it refuses rather than silently aborting the wave). Answering n leaves the run completely untouched. A node-scoped takeover that parks only the one node and leaves its siblings running is on the roadmap; it needs a per-node interrupt the engine does not have yet, so today taking one worker over means taking the run over. herdr stays a pure mirror throughout: it never writes to the engine or the run’s store. Pressing s or h in a pane just spawns the same smithers steer the CLI would, and the pane only displays the SteerQueued / SteerConsumed / SteerExpired events Smithers emits.

​
Hijack into a pane

smithers steer --takeover (and the h key) is a thin, zero-ceremony front end over bunx smthrs hijack RUN_ID, the underlying hand-off. hijack by default launches the interactive agent CLI in your current terminal; the SMITHERS_HERDR_HIJACK env var opts it into a herdr pane inside the run’s mirror workspace instead. steer --takeover just sets that up for you after auto-detecting the mirror (and after the in-flight-sibling warning above), so you rarely reach for the raw form - but it stays available for scripts and for driving the pane-vs-terminal decision explicitly. The hijack pane is cap-exempt and does take focus, because you asked to drive that agent right now. Pane hosting is gated on SMITHERS_HERDR_HIJACK, and the session it targets comes from SMITHERS_HERDR (not from herdr’s own HERDR_SESSION). So to host the pane in the same named session your run mirrors into, set both variables:
SMITHERS_HERDR_HIJACK accepts 1, true, or an explicit session name. With 1 or true it reuses the session from SMITHERS_HERDR; if SMITHERS_HERDR is unset the pane carries no explicit session, so the client falls through to herdr’s own socket resolution (HERDR_SOCKET_PATH, then HERDR_SESSION, then the default session). That is why setting both variables, above, is the reliable way to host the pane in the same named session your run mirrors into. The hijack pane is an agent named smithers:RUN_ID:hijack:NODE_ID, marked blocked (“hijacked - attach to drive”). The command prints the exact agent name and attach command on stderr; attach and hijack with herdr’s own CLI:

​
Hijack afterlife

A hijack pane does not collapse to a bare shell when your session exits. Smithers wraps the launch spec so that when the agent CLI exits, the pane prints a handback summary (the exact resume command) and lingers for a keypress instead of vanishing, so a hijack you stepped away from is still readable when you come back. The inner exit code is preserved, and argv/cwd/env are passed through exactly; only the herdr-pane path is wrapped, the current-terminal hijack flow is byte-identical. Handback is manual, and the run does not keep running while you drive the agent by hand: on handoff Smithers parks the run (its status goes to cancelled), so the durable run stops advancing until you return control. Because the pane’s process is owned by herdr and its exit event carries no exit code, Smithers does not auto-resume (auto-resuming after an aborted or errored session could corrupt the run). When you are done, hand control back with the resume command the hijack prints:
If no herdr server is reachable or the pane launch fails, hijack falls back to the byte-identical current-terminal flow, so opting in never removes your ability to hijack.

​
Detached runs

bunx smthrs up --detach (-d) re-spawns the run in a background process and returns immediately. --herdr is honored there too: the parent hands the setting to the detached child through the SMITHERS_HERDR env var (1 or the session name), which the child inherits, so the mirror runs in the process that actually executes the run. You do not set SMITHERS_HERDR yourself; --herdr sets it for the handoff. The detach message leads its watch options with herdr attach RUN_ID so you can re-open the mirror after detaching.

​
Degradability

Herdr never affects a run. Every interaction is fire-and-forget with a per-call timeout (5s by default) and a consecutive-timeout circuit breaker that drops pushes fast once herdr stops answering. Concretely:
  • No herdr server reachable: --herdr logs one warning on stderr and the run proceeds with no mirror. For a detached run, that warning lands in the detach log file.
  • Herdr dies mid-run: calls soft-fail, the breaker opens, and the run reaches its terminal state normally.
  • Host shutdown is bounded: closing the surface abandons a hung herdr after a short deadline rather than blocking.

​
Version pinning

This client is built and tested against herdr 0.7.3 / wire protocol 16. bunx smthrs herdr status reports the server’s protocol and whether it matches, which is the quickest compatibility check. A protocol other than 16 is handled two different ways, on purpose:
  • Optional mirroring (up --herdr, auto-detected mirrors, the supervisor’s pane integration) degrades softly: one warning, then the run proceeds with no mirror. Nothing about execution or durability depends on herdr.
  • Explicit herdr commands (status, open, attach, clean) fail closed with HERDR_PROTOCOL_MISMATCH and exit 4, reporting both protocol numbers and confirming that no mutating call was made. Driving panes over a wire format the server may have redefined is not worth the risk of corrupting a workspace.
Herdr’s protocol advances faster than its user-visible version: releases after 0.7.3 speak higher protocol numbers and are not wire-compatible with this client, so herdr features are inert (safely) until the client is updated to match. Check herdr status client against the number above before filing a bug about herdr features doing nothing.

​
Remote runs over SSH

Herdr talks to Smithers over a local unix socket, so herdr runs where your agents run. To watch a run executing on a remote host, run both Smithers and herdr on that host and reach them over SSH (for example an SSH session into the host, then bunx smthrs herdr attach RUN_ID there). Do not try to forward or expose the socket itself; SSH to the host and run the client locally.

​
Security

The herdr control socket is unauthenticated local control. Anything that can reach it can start processes and drive panes. Keep it on a trusted local machine (the default socket lives under ~/.config/herdr/). Never expose it over the network or forward the raw socket. For a remote host, SSH in and run the client there rather than opening the socket up.

​
Mirror vs interactive moments

The mirror is the architecture: Smithers always owns execution, isolation, and durability, and herdr is a read-mostly presentation and steering plane fed from Smithers’ event stream. In-pane approvals (approve --watch) and hijack panes are the interactive escape hatches for the moments you need to decide or drive by hand. A per-task interactive mode, where herdr would own a task’s process end to end, is explicitly out of scope for this integration.