review is on attempt 3 of 4, or that a heartbeat timeout fired forty minutes ago. When the orchestrator process itself dies, a call-trace tool has nothing at all: the trace ends mid-span and the run’s true state lives nowhere.
Smithers takes the position that agent observability needs event-sourced run state, not just LLM call traces. The run’s history is the run: every lifecycle transition — node started, approval requested, heartbeat missed, snapshot captured, run forked — is appended to a durable log before anything else consumes it. Traces, metrics, structured logs, the Monitor UI, and the CLI are all derived views over that one record. Because the log is the source of truth rather than a telemetry side channel, it survives crashes, powers resume, and can answer “why is this stuck” after the process that got stuck is gone.
This is the same argument Honeycomb makes for wide events — capture the full context of each unit of work, derive dashboards later — applied to a domain where the unit of work is a multi-hour, human-in-the-loop run, not a request.
How the industry solves it
Two lineages converge here. Distributed tracing — Google’s Dapper paper, then Jaeger and Grafana Tempo — models work as span trees with propagated context. LLM observability platforms specialized that model for prompts and completions.
The common shape: instrument the calls, ship spans to a backend, analyze there. What none of these provide is the orchestration state machine itself — a span has no notion of “waiting for human approval since Tuesday” or “resumable from frame 12”. That state either lives in your framework’s private memory (and dies with it) or nowhere.
How Smithers does it
The durable event log
Every runtime event is a variant of theSmithersEvent discriminated union — RunStarted, FrameCommitted, NodeStarted, NodeRetrying, NodeWaitingApproval, ApprovalGranted, ToolCallFinished, TaskHeartbeatTimeout, SnapshotCaptured, TokenUsageReported, RunForked, and ~70 more. Events are persisted twice:
- Database: the
_smithers_eventstable —(run_id, seq)primary key,timestamp_ms,type,payload_json. Ordered, queryable, survives anything short of losing the DB. - NDJSON stream:
.smithers/executions/<runId>/logs/stream.ndjson, one JSON object per line,tail -f-able andjq-able.
type, runId, timestampMs; node-scoped events add nodeId and iteration; attempt-scoped ones add attempt. The full union and category table is in Event Types.
Subscribing in code
runWorkflow (and createSmithers run options) take onProgress and onError callbacks:
streamRunEventsResilient, useGatewayRunEvents) or the HTTP server’s SSE route GET /v1/runs/:runId/events?afterSeq=N. Because every event has a sequence number, a reconnecting client resumes from where it left off — the log, not the connection, is authoritative.
OTel traces, metrics, and logs
createSmithersObservabilityLayer (from smithers-orchestrator/observability) wires the Effect runtime’s logger, an OTLP exporter, and the metrics service in one layer:
smithers.run, smithers.task, smithers.agent, smithers.tool, with attributes normalized to smithers.run_id, smithers.node_id, smithers.iteration, smithers.attempt, smithers.agent, smithers.model, smithers.wait_reason, and friends — so every span joins back to the event log by run and node ID. Canonical agent-trace events and full provider session transcripts also export as OTel log records (verifiable end-to-end against Loki; see Agent Trace OTEL Verification).
The metric catalog is large and specific: 150+ definitions covering run/node/attempt durations, retries, approval wait time, token counters per direction (smithers.tokens.input_total, cache read/write, reasoning), scheduler queue depth, gateway connections, sandbox lifecycle, alert deliveries, and time-travel rollbacks. The HTTP server and serve mode expose them at GET /metrics in Prometheus exposition format (on by default with --serve).
A full local backend ships in the repo:
http://localhost:4318) points at exactly this collector.
Watching a run live
what and why are the event-sourcing payoff in miniature: they read the log, not the process, so they work on a run whose process died yesterday. The Monitor is the same idea as a web page — approvals inbox, live run rows, frames, and the raw event stream, served by the workspace gateway with zero UI code.
Alerts and provenance
Observability feeds two enforcement layers.alertPolicy on createSmithers() attaches ownership, severity, runbooks, and deterministic reactions to workflows; fired alerts become durable rows (_smithers_alerts) managed with smithers alerts — see Alerting. Provenance binding goes further: because every approval and artifact is an event with a digest, the engine can enforce that a task only runs against the exact upstream artifact that authorized it, parking the task when authority is stale.
Guarantees and limits
Guaranteed: every runtime-emitted event is persisted with a monotonic per-run sequence before consumers see it; the DB log and NDJSON stream survive process death; CLI and Gateway reads never require the run’s process to be alive; metrics and spans always carrysmithers.run_id/smithers.node_id for correlation.
Not guaranteed, honestly:
- No hosted dashboard. There is no smithers.sh cloud UI with retention, sharing, and org-wide search. You run the Monitor locally and bring your own Grafana/Tempo/Loki (or point OTLP at Datadog/Langfuse/Phoenix — the exporter is standard).
- No
gen_ai.*semconv yet. Spans usesmithers.*attributes, not the OTel GenAI conventions — defensible while those remain pre-stable, but tools that key ongen_ai.operation.namewon’t auto-recognize Smithers spans. - OTel export is best-effort telemetry, opt-in and off by default. The event log is the durable record; if the collector is down, spans drop, events don’t.
- Evals are adjacent, not fused. Scorer results are events (
ScorerFinished) and rows, but Smithers does not turn production traces into eval datasets the way Braintrust or LangSmith do;smithers evalruns suites explicitly. - A few union variants are reserved but not currently emitted (e.g.
RunStateChanged,OpenApiToolCalled) — documented as such in Event Types.
Operating it
Day to day, agents use the CLI above plus the MCP toolsget_run, get_run_events, get_node_detail, get_timeline, watch_run, and explain_run. Long-lived observers should go through the workspace gateway (smithers gateway status --format json, then SmithersGatewayClient) rather than polling files.
See also
- CLI Overview — the full command surface (absorbed the old monitoring & logs page)
- The Smithers Monitor — the zero-setup web UI
- Alerting — durable alert policies and rows
- Provenance binding — events as enforced authority
- Agent Trace OTEL Verification — reproducing the OTel log export locally
- Debugging — turning observations into fixes