Skip to main content
Four observability surfaces:
  • Persisted lifecycle events
  • Structured logs
  • OpenTelemetry spans
  • Effect metrics exported over OTLP
The runtime instruments workflow runs, nodes, tools, cache, approvals, database access, hot reloads, HTTP requests, and JJ commands automatically.

​
Enable OpenTelemetry Export

import { Context, Effect, Layer, Schema } from "effect";
import { Model } from "@effect/sql";
import { createSmithersObservabilityLayer } from "smithers-orchestrator/observability";

const AppLive = Layer.mergeAll(
  AgentLive,
  createSmithersObservabilityLayer({
    enabled: true,
    endpoint: "http://localhost:4318",
    serviceName: "bugfix-worker",
    logFormat: "json",
  }),
);
Environment-based configuration: 
export SMITHERS_OTEL_ENABLED=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=bugfix-worker
export SMITHERS_LOG_FORMAT=json
export SMITHERS_LOG_LEVEL=info

​
Local Prometheus + Grafana Stack

docker compose -f observability/docker-compose.otel.yml up
ServiceEndpoint
OTLP collectorhttp://localhost:4318
Prometheushttp://localhost:9090
Grafanahttp://localhost:3000
Tempohttp://localhost:3200
The collector exports metrics on :8889; Prometheus scrapes it; Grafana ships with a pre-provisioned Smithers dashboard. The built-in HTTP server also exposes GET /metrics in Prometheus text exposition format.

​
Direct Prometheus Endpoint

curl http://localhost:7331/metrics
For custom servers: 
import {
  prometheusContentType,
  renderPrometheusMetrics,
} from "smithers-orchestrator/observability";

const body = renderPrometheusMetrics();
// return body with Content-Type: prometheusContentType

​
Built-in Metrics

import { smithersMetrics } from "smithers-orchestrator/observability";
CategoryMetrics
RunsrunsTotal, activeRuns, runsResumedTotal, runsFinishedTotal, runsFailedTotal, runsCancelledTotal, runsContinuedTotal
NodesnodesStarted, nodesFinished, nodesFailed, activeNodes, nodeRetriesTotal
DurationnodeDuration, attemptDuration, runDuration
ToolstoolCallsTotal, toolDuration, toolCallErrorsTotal, toolOutputTruncatedTotal
CachecacheHits, cacheMisses
DatabasedbQueryDuration, dbRetries
Database transactionsdbTransactionDuration, dbTransactionRetries, dbTransactionRollbacks
SchedulerschedulerQueueDepth, schedulerConcurrencyUtilization, schedulerWaitDuration
ApprovalsapprovalsRequested, approvalsGranted, approvalsDenied, approvalPending, approvalWaitDuration
Async / external waitexternalWaitAsyncPending
TimerstimersCreated, timersFired, timersCancelled, timersPending, timerDelayDuration
TokenstokensInputTotal, tokensOutputTotal, tokensCacheReadTotal, tokensCacheWriteTotal, tokensReasoningTotal, tokensInputPerCall, tokensOutputPerCall
HTTPhttpRequests, httpRequestDuration
Hot reloadhotReloads, hotReloadFailures, hotReloadDuration
SandboxsandboxCreatedTotal, sandboxCompletedTotal, sandboxActive, sandboxDurationMs
Sandbox transportsandboxTransportDurationMs, sandboxBundleSizeBytes, sandboxPatchCount
Prompt sizepromptSizeBytes
Response sizeresponseSizeBytes
Run ancestry depthrunsAncestryDepth
Continue-as-new staterunsCarriedStateBytes
Scorer eventsscorerEventsStarted, scorerEventsFinished, scorerEventsFailed
VCSvcsDuration
ProcessprocessUptimeSeconds, processMemoryRssBytes, processHeapUsedBytes
ErrorserrorsTotal, eventsEmittedTotal
Emitted by the engine automatically.

​
Observability as a Dependency

import { Context, Effect, Layer, Metric } from "effect";
import { SmithersObservability } from "smithers-orchestrator/observability";

const notificationsSent = Metric.counter("app.notifications.sent");

class Notifications extends Context.Tag("Notifications")<
  Notifications,
  {
    readonly send: (ticketId: string) => Effect.Effect<void>;
  }
>() {}

const NotificationsLive = Layer.effect(
  Notifications,
  Effect.gen(function* () {
    const obs = yield* SmithersObservability;

    return {
      send: (ticketId) =>
        obs.withSpan(
          "notifications:send",
          Effect.gen(function* () {
            yield* obs.annotate({
              ticketId,
              channel: "slack",
            });
            yield* Metric.increment(notificationsSent);
            yield* Effect.logInfo(`sending notification for ${ticketId}`);
          }),
          { component: "notifications" },
        ),
    };
  }),
);
Composition model:
  • Call createSmithersObservabilityLayer(...) once in the app layer
  • Depend on SmithersObservability in services that need Smithers-scoped spans
  • Use standard Effect Metric primitives for custom application metrics

​
Events and Logs

OTLP does not replace the durable event log. Every run still persists events to: 
.smithers/executions/<runId>/logs/stream.ndjson

tail -f .smithers/executions/<runId>/logs/stream.ndjson
sqlite3 smithers.db "SELECT seq, type, payload_json FROM _smithers_events WHERE run_id = '<id>' ORDER BY seq DESC LIMIT 20;"
See Events for the event model.

​
Server Instrumentation

HTTP server requests are instrumented:
  • smithers.http.requests counter
  • smithers.http.request_duration_ms histogram
  • Request, workflow-load, and body-read spans flow through the OTLP layer

​
Next Steps