> ## Documentation Index
> Fetch the complete documentation index at: https://smithers.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Every run is an append-only event log first — traces, metrics, logs, the Monitor, and the CLI are all views over the same durable record.

An agent run that fails at 3 a.m. leaves you two questions: *what happened* and *why is it stuck now*. Most LLM observability tools answer a third question — *what did the model say* — and only that one. A trace of prompts and completions tells you the agent burned 400k tokens; it does not tell you that the run is parked on an approval nobody granted, that node `review` is on attempt 3 of 4, or that a heartbeat timeout fired forty minutes ago. When the orchestrator process itself dies, a call-trace tool has nothing at all: the trace ends mid-span and the run's true state lives nowhere.

Smithers takes the position that agent observability needs **event-sourced run state**, not just LLM call traces. The run's history *is* the run: every lifecycle transition — node started, approval requested, heartbeat missed, snapshot captured, run forked — is appended to a durable log before anything else consumes it. Traces, metrics, structured logs, the Monitor UI, and the CLI are all derived views over that one record. Because the log is the source of truth rather than a telemetry side channel, it survives crashes, powers resume, and can answer "why is this stuck" after the process that got stuck is gone.

This is the same argument Honeycomb makes for [wide events](https://www.honeycomb.io/blog/observability-whats-in-a-name) — capture the full context of each unit of work, derive dashboards later — applied to a domain where the unit of work is a multi-hour, human-in-the-loop run, not a request.

## How the industry solves it

Two lineages converge here. Distributed tracing — Google's [Dapper](https://research.google/pubs/dapper-a-large-scale-distributed-systems-tracing-infrastructure/) paper, then [Jaeger](https://www.jaegertracing.io/) and [Grafana Tempo](https://grafana.com/oss/tempo/) — models work as span trees with propagated context. LLM observability platforms specialized that model for prompts and completions.

| System                                                                     | Mechanism                                                                           | Tradeoff                                                                                                   |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [LangSmith](https://docs.smith.langchain.com/)                             | Hosted trace tree per chain/agent invocation; datasets and evals attached to traces | Deep LangChain integration; hosted-first, call-centric                                                     |
| [Langfuse](https://langfuse.com/docs)                                      | Open-source traces/observations/scores model, self-hostable, OTel ingestion         | Rich UI; run *state* (paused, waiting-approval) is not a first-class concept                               |
| [Braintrust](https://www.braintrust.dev/docs)                              | Logs and experiments share one schema, so production traces become eval cases       | Eval-centric; orchestration lifecycle out of scope                                                         |
| [Arize Phoenix](https://arize.com/docs/phoenix)                            | Open-source, built on OpenInference/OTel spans                                      | Strong for retrieval/eval analysis of traces                                                               |
| [W\&B Weave](https://weave-docs.wandb.ai/)                                 | `@weave.op` decorators auto-trace call graphs                                       | Elegant capture; couples to the Python call stack — dies with the process                                  |
| [Helicone](https://docs.helicone.ai/)                                      | Proxy in front of the LLM API; logs every request                                   | Zero code change; sees only what crosses the proxy                                                         |
| [Datadog LLM Observability](https://docs.datadoghq.com/llm_observability/) | LLM spans joined with APM, infra, and cluster maps                                  | Full-stack correlation; proprietary, per-host pricing                                                      |
| [AgentOps](https://docs.agentops.ai/)                                      | Session replay + event stream for agent frameworks                                  | Agent-aware; hosted, replay is for viewing not resuming                                                    |
| [OTel GenAI semconv](https://opentelemetry.io/docs/specs/semconv/gen-ai/)  | Standard `gen_ai.*` span attributes for LLM/agent calls                             | Vendor-neutral; still pre-stable (Development status as of mid-2026, moved to a dedicated repo in v1.42.0) |

The common shape: instrument the *calls*, ship spans to a backend, analyze there. What none of these provide is the orchestration state machine itself — a span has no notion of "waiting for human approval since Tuesday" or "resumable from frame 12". That state either lives in your framework's private memory (and dies with it) or nowhere.

## How Smithers does it

### The durable event log

Every runtime event is a variant of the `SmithersEvent` discriminated union — `RunStarted`, `FrameCommitted`, `NodeStarted`, `NodeRetrying`, `NodeWaitingApproval`, `ApprovalGranted`, `ToolCallFinished`, `TaskHeartbeatTimeout`, `SnapshotCaptured`, `TokenUsageReported`, `RunForked`, and \~70 more. Events are persisted twice:

* **Database**: the `_smithers_events` table — `(run_id, seq)` primary key, `timestamp_ms`, `type`, `payload_json`. Ordered, queryable, survives anything short of losing the DB.
* **NDJSON stream**: `.smithers/executions/<runId>/logs/stream.ndjson`, one JSON object per line, `tail -f`-able and `jq`-able.

Every event carries `type`, `runId`, `timestampMs`; node-scoped events add `nodeId` and `iteration`; attempt-scoped ones add `attempt`. The full union and category table is in [Event Types](/reference/event-types).

### Subscribing in code

`runWorkflow` (and `createSmithers` run options) take `onProgress` and `onError` callbacks:

```tsx theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { runWorkflow } from "smithers-orchestrator";
import { Effect } from "effect";
import workflow from "./workflow";

await Effect.runPromise(runWorkflow(workflow, {
  input: { task: "fix the flaky auth test" },
  onProgress: (event) => {
    if (event.type === "NodeStarted")  console.log(`▶ ${event.nodeId} attempt ${event.attempt}`);
    if (event.type === "NodeWaitingApproval") notifySlack(event);
  },
  // Fires once per NodeFailed / RunFailed with a normalized report
  onError: (r) => Sentry.captureException(r.error, {
    extra: { runId: r.runId, nodeId: r.nodeId, phase: r.phase },
  }),
}));
```

Remote consumers get the same events over the Gateway's WebSocket (`streamRunEventsResilient`, `useGatewayRunEvents`) or the HTTP server's SSE route `GET /v1/runs/:runId/events?afterSeq=N`. Because every event has a sequence number, a reconnecting client resumes from where it left off — the log, not the connection, is authoritative.

### OTel traces, metrics, and logs

`createSmithersObservabilityLayer` (from `smithers-orchestrator/observability`) wires the Effect runtime's logger, an OTLP exporter, and the metrics service in one layer:

```ts theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { createSmithersObservabilityLayer } from "smithers-orchestrator/observability";

const layer = createSmithersObservabilityLayer({
  enabled: true,                          // or SMITHERS_OTEL_ENABLED=1
  endpoint: "http://localhost:4318",      // OTLP/HTTP; OTEL_EXPORTER_OTLP_ENDPOINT
  serviceName: "smithers",                // OTEL_SERVICE_NAME
  logFormat: "json",                      // logfmt (default) | json | pretty | string
  logLevel: "debug",                      // SMITHERS_LOG_LEVEL; default info
});
```

Spans use the names `smithers.run`, `smithers.task`, `smithers.agent`, `smithers.tool`, with attributes normalized to `smithers.run_id`, `smithers.node_id`, `smithers.iteration`, `smithers.attempt`, `smithers.agent`, `smithers.model`, `smithers.wait_reason`, and friends — so every span joins back to the event log by run and node ID. Canonical agent-trace events and full provider session transcripts also export as OTel *log records* (verifiable end-to-end against Loki; see [Agent Trace OTEL Verification](/guides/agent-trace-otel-verification)).

The metric catalog is large and specific: 150+ definitions covering run/node/attempt durations, retries, approval wait time, token counters per direction (`smithers.tokens.input_total`, cache read/write, reasoning), scheduler queue depth, gateway connections, sandbox lifecycle, alert deliveries, and time-travel rollbacks. The HTTP server and serve mode expose them at `GET /metrics` in Prometheus exposition format (on by default with `--serve`).

A full local backend ships in the repo:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bunx smithers-orchestrator observability   # docker compose -f observability/docker-compose.otel.yml
```

That starts an OTLP Collector (gRPC 4317, HTTP 4318), Prometheus (9090), Tempo for traces, Loki for logs, and Grafana with pre-built dashboards. The layer's default endpoint (`http://localhost:4318`) points at exactly this collector.

### Watching a run live

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bunx smithers-orchestrator status RUN_ID            # verdict, node counts, agent/model mix, gating nodes
bunx smithers-orchestrator inspect RUN_ID --watch   # full run state, live
bunx smithers-orchestrator events RUN_ID --watch    # lifecycle stream; --type tool-call --node analyze to filter, --raw for agent chunks
bunx smithers-orchestrator logs RUN_ID              # tail the event log
bunx smithers-orchestrator node NODE_ID --runId RUN_ID   # retries, tool calls, output for one node
bunx smithers-orchestrator what RUN_ID              # a cheap agent narrates the recorded facts
bunx smithers-orchestrator why RUN_ID               # names the real blocker and prints the exact unblock command
bunx smithers-orchestrator monitor                  # live web UI over every run in the workspace
```

`what` and `why` are the event-sourcing payoff in miniature: they read the log, not the process, so they work on a run whose process died yesterday. The [Monitor](/guides/monitor) is the same idea as a web page — approvals inbox, live run rows, frames, and the raw event stream, served by the workspace gateway with zero UI code.

### Alerts and provenance

Observability feeds two enforcement layers. `alertPolicy` on `createSmithers()` attaches ownership, severity, runbooks, and deterministic reactions to workflows; fired alerts become durable rows (`_smithers_alerts`) managed with `smithers alerts` — see [Alerting](/guides/alerting). [Provenance binding](/concepts/provenance) goes further: because every approval and artifact is an event with a digest, the engine can *enforce* that a task only runs against the exact upstream artifact that authorized it, parking the task when authority is stale.

## Guarantees and limits

**Guaranteed:** every runtime-emitted event is persisted with a monotonic per-run sequence before consumers see it; the DB log and NDJSON stream survive process death; CLI and Gateway reads never require the run's process to be alive; metrics and spans always carry `smithers.run_id`/`smithers.node_id` for correlation.

**Not guaranteed, honestly:**

* **No hosted dashboard.** There is no smithers.sh cloud UI with retention, sharing, and org-wide search. You run the Monitor locally and bring your own Grafana/Tempo/Loki (or point OTLP at Datadog/Langfuse/Phoenix — the exporter is standard).
* **No `gen_ai.*` semconv yet.** Spans use `smithers.*` attributes, not the OTel GenAI conventions — defensible while those remain pre-stable, but tools that key on `gen_ai.operation.name` won't auto-recognize Smithers spans.
* **OTel export is best-effort telemetry**, opt-in and off by default. The event log is the durable record; if the collector is down, spans drop, events don't.
* **Evals are adjacent, not fused.** Scorer results are events (`ScorerFinished`) and rows, but Smithers does not turn production traces into eval datasets the way Braintrust or LangSmith do; `smithers eval` runs suites explicitly.
* A few union variants are reserved but not currently emitted (e.g. `RunStateChanged`, `OpenApiToolCalled`) — documented as such in Event Types.

## Operating it

Day to day, agents use the CLI above plus the MCP tools `get_run`, `get_run_events`, `get_node_detail`, `get_timeline`, `watch_run`, and `explain_run`. Long-lived observers should go through the workspace gateway (`smithers gateway status --format json`, then `SmithersGatewayClient`) rather than polling files.

## See also

* [CLI Overview](/cli/overview) — the full command surface (absorbed the old [monitoring & logs](/guides/monitoring-logs) page)
* [The Smithers Monitor](/guides/monitor) — the zero-setup web UI
* [Alerting](/guides/alerting) — durable alert policies and rows
* [Provenance binding](/concepts/provenance) — events as enforced authority
* [Agent Trace OTEL Verification](/guides/agent-trace-otel-verification) — reproducing the OTel log export locally
* [Debugging](/guides/debugging) — turning observations into fixes