Observability API

Observability ships as a set of Effect layers. One call to createSmithersObservabilityLayer installs the logger, the metrics registry, the tracing service, and (when enabled) an OTLP exporter. Metrics are always collected in-process and read back as a Prometheus exposition string; traces and logs are exported over OTLP only when enabled is set.

import {
  createSmithersObservabilityLayer,
  createSmithersRuntimeLayer,
  createSmithersOtelLayer,
  resolveSmithersObservabilityOptions,
  trackSmithersEvent,
  renderPrometheusMetrics,
  prometheusContentType,
  smithersMetrics,
  SmithersObservability,
} from "smithers-orchestrator";

Metric collection is global. renderPrometheusMetrics() and trackSmithersEvent read and write the process-wide Effect registry, so they work whether or not you have built a layer. Building a layer adds logging, tracing, and OTLP export on top.

SmithersObservability

An Effect Context.Tag for the resolved observability service. Provided by createSmithersObservabilityLayer; depend on it to read the resolved options or to open a span without importing withSmithersSpan directly.

import { Effect } from "effect";
import { SmithersObservability } from "smithers-orchestrator";

const program = Effect.gen(function* () {
  const obs = yield* SmithersObservability;
  yield* obs.withSpan("my-step", Effect.log("inside the span"));
});

SmithersObservabilityService

object

The shape behind the tag.

Show members

options

ResolvedSmithersObservabilityOptions

The fully resolved options the layer was built with.

annotate

(attributes) => Effect<void>

Attach attributes to the current trace span.

withSpan

(name, effect, attributes?) => Effect

Run effect inside a named child span.

Source SmithersObservability.js · SmithersObservabilityService.ts · Tests observability.test.js · See also Server integration

Layers

createSmithersObservabilityLayer

Builds the full observability layer: logger, MetricsService, TracingService, the SmithersObservability service, and BunContext. The OTLP exporter is folded in only when the resolved options have enabled: true.

function createSmithersObservabilityLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<MetricsService | TracingService | SmithersObservability | BunContext>;

options

SmithersObservabilityOptions

Partial options, resolved through resolveSmithersObservabilityOptions. See resolveSmithersObservabilityOptions for fields and defaults.

import { Effect, Layer } from "effect";
import { createSmithersObservabilityLayer } from "smithers-orchestrator";

const layer = createSmithersObservabilityLayer({
  serviceName: "my-control-plane",
  logFormat: "json",
});

await Effect.runPromise(myProgram.pipe(Effect.provide(layer)));

createSmithersRuntimeLayer

Alias of createSmithersObservabilityLayer, exported under the name used by the runtime. Same signature, same return type. Prefer it in runtime/server wiring so the intent reads clearly.

import { createSmithersRuntimeLayer } from "smithers-orchestrator";

const layer = createSmithersRuntimeLayer({ logLevel: "debug" });

createSmithersOtelLayer

The OTLP-export slice on its own. Returns Layer.empty unless enabled resolves to true, so it is safe to merge unconditionally. Use it when you already have a metrics/logging layer and only want to add OpenTelemetry traces.

function createSmithersOtelLayer(
  options?: SmithersObservabilityOptions,
): Layer.Layer<never>;

options

SmithersObservabilityOptions

Same options object. Only enabled, endpoint, headers, and serviceName affect this layer.

import { createSmithersOtelLayer } from "smithers-orchestrator";

// No-op unless SMITHERS_OTEL_ENABLED=1 (or enabled: true).
const otel = createSmithersOtelLayer({
  endpoint: "http://localhost:4318",
  serviceName: "smithers",
});

Source createSmithersObservabilityLayer.js · createSmithersOtelLayer.js · createSmithersRuntimeLayer.js · Tests observability.test.js · See also Agent trace OTel verification

resolveSmithersObservabilityOptions

Merge partial options with environment variables and defaults. The layer factories call this for you; call it directly to inspect what a deployment resolves to.

function resolveSmithersObservabilityOptions(
  options?: SmithersObservabilityOptions,
): ResolvedSmithersObservabilityOptions;

options

SmithersObservabilityOptions

Every field is optional. Unset fields fall back to env vars, then defaults.

Show SmithersObservabilityOptions

enabled

boolean

Enable OTLP export. Defaults from SMITHERS_OTEL_ENABLED (1/true), otherwise false.

endpoint

string

OTLP HTTP endpoint. Defaults from OTEL_EXPORTER_OTLP_ENDPOINT, otherwise http://localhost:4318.

headers

Record<string, string>

OTLP export headers. Defaults from OTEL_EXPORTER_OTLP_HEADERS, parsed as comma-separated key=value pairs with percent-encoded values decoded.

serviceName

string

Resource service name. Defaults from OTEL_SERVICE_NAME, otherwise smithers.

logFormat

"json" | "pretty" | "string" | "logfmt"

Logger format. Defaults from SMITHERS_LOG_FORMAT, otherwise logfmt.

logLevel

LogLevel | string

Minimum log level. Accepts an Effect LogLevel or a string (none/trace/debug/info/warning/error/fatal/all). Defaults from SMITHERS_LOG_LEVEL, otherwise Info.

installLogger

boolean

default:"true"

Replace the default Effect logger and set the minimum level. Set false to keep your own logger.

For authenticated SaaS exporters, set OTEL_EXPORTER_OTLP_HEADERS:

# Honeycomb
export OTEL_EXPORTER_OTLP_HEADERS='x-honeycomb-team=<key>'

# Grafana Cloud
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Basic <base64 token>'

ResolvedSmithersObservabilityOptions

object

Every field is present after resolution. headers is undefined when no headers are configured.

Show ResolvedSmithersObservabilityOptions

enabled

boolean

endpoint

string

headers

Record<string, string> | undefined

serviceName

string

logFormat

"json" | "pretty" | "string" | "logfmt"

logLevel

LogLevel

installLogger

boolean

import { resolveSmithersObservabilityOptions } from "smithers-orchestrator";

const resolved = resolveSmithersObservabilityOptions({ serviceName: "ci" });
// { enabled: false, endpoint: "http://localhost:4318", serviceName: "ci", ... }

Source resolveSmithersObservabilityOptions.js · ResolvedSmithersObservabilityOptions.ts · Tests observability-options.test.js

trackSmithersEvent

Map a SmithersEvent onto the metric registry. Returns an Effect<void> that increments the relevant counters, updates gauges, and records durations for that event type. Every event also bumps the smithers_events_emitted_total counter. The runtime calls this on each emitted event; call it yourself only when feeding events from outside the engine.

function trackSmithersEvent(event: SmithersEvent): Effect.Effect<void>;

event

SmithersEvent

required

A discriminated union keyed by type. See Event types.

import { Effect } from "effect";
import { trackSmithersEvent } from "smithers-orchestrator";

await Effect.runPromise(
  trackSmithersEvent({
    type: "NodeFinished",
    runId: "RUN_ID",
    nodeId: "NODE_ID",
    iteration: 0,
    attempt: 1,
    timestampMs: Date.now(),
  }),
);

Source trackEvent.js · SmithersEvent.ts · Tests effect-metrics-track.test.js · See also Event types

Prometheus

renderPrometheusMetrics

Render the entire metric registry as a Prometheus exposition string, with # HELP/# TYPE headers and every catalog metric present at zero even before it fires. Refreshes the process gauges (uptime, RSS, heap) on each call. This is what the server’s GET /metrics route returns.

function renderPrometheusMetrics(): string;

prometheusContentType

The exact Content-Type for a Prometheus response: text/plain; version=0.0.4; charset=utf-8. Pair it with the rendered body.

import { renderPrometheusMetrics, prometheusContentType } from "smithers-orchestrator";

// Inside any HTTP handler:
function handleMetrics() {
  return new Response(renderPrometheusMetrics(), {
    headers: { "Content-Type": prometheusContentType },
  });
}

The built-in server already exposes this at GET /metrics. See Server integration. Use these directly only when wiring metrics into your own HTTP layer.

Source renderPrometheusMetrics.js · prometheusContentType.js · Tests observability-prometheus.test.js · See also Server integration

Metrics catalog

smithersMetrics is the registry of every metric, an object keyed by the camelCase names below whose values are the underlying Effect Metrics. Each metric renders under its Prometheus name; histograms also emit _bucket, _sum, and _count series. The table groups the core metrics by area; the registry holds more (gateway, devtools, agent, alert, supervisor families).

import { smithersMetrics } from "smithers-orchestrator";

const everyMetric = Object.values(smithersMetrics); // all Effect Metrics

Runs and nodes

Metric	Kind	Meaning
`smithers_runs_total`	counter	Runs started
`smithers_runs_finished_total`	counter	Runs finished
`smithers_runs_failed_total`	counter	Runs failed
`smithers_runs_cancelled_total`	counter	Runs cancelled
`smithers_runs_resumed_total`	counter	Runs resumed
`smithers_runs_continued_total`	counter	Runs continued-as-new
`smithers_nodes_started`	counter	Nodes started
`smithers_nodes_finished`	counter	Nodes finished
`smithers_nodes_failed`	counter	Nodes failed
`smithers_node_retries_total`	counter	Node retries
`smithers_errors_total`	counter	Errors
`smithers_events_emitted_total`	counter	Events emitted
`smithers_runs_active`	gauge	Active runs
`smithers_nodes_active`	gauge	Active nodes
`smithers_scheduler_queue_depth`	gauge	Scheduler queue depth
`smithers_scheduler_concurrency_utilization`	gauge	Scheduler concurrency utilization
`smithers_run_duration_ms`	histogram	Run duration
`smithers_node_duration_ms`	histogram	Node duration
`smithers_attempt_duration_ms`	histogram	Per-attempt duration
`smithers_scheduler_wait_duration_ms`	histogram	Scheduler wait time
`smithers_snapshots_captured`	counter	Snapshots captured
`smithers_forks_created`	counter	Run forks created
`smithers_replays_started`	counter	Replays started
`smithers_snapshot_duration_ms`	histogram	Snapshot duration

Approvals and timers

Metric	Kind	Meaning
`smithers_approvals_requested`	counter	Approvals requested
`smithers_approvals_granted`	counter	Approvals granted
`smithers_approvals_denied`	counter	Approvals denied
`smithers_approval_pending`	gauge	Pending approvals
`smithers_external_wait_async_pending`	gauge	Pending external waits
`smithers_approval_wait_duration_ms`	histogram	Approval wait duration
`smithers_timers_created`	counter	Timers created
`smithers_timers_fired`	counter	Timers fired
`smithers_timers_cancelled`	counter	Timers cancelled
`smithers_timers_pending`	gauge	Pending timers
`smithers_timers_delay_ms`	histogram	Timer fire delay

Cache and tokens

Metric	Kind	Meaning
`smithers_cache_hits`	counter	Cache hits
`smithers_cache_misses`	counter	Cache misses
`smithers_tokens_input_total`	counter	Input tokens
`smithers_tokens_output_total`	counter	Output tokens
`smithers_tokens_cache_read_total`	counter	Cache read tokens
`smithers_tokens_cache_write_total`	counter	Cache write tokens
`smithers_tokens_reasoning_total`	counter	Reasoning tokens
`smithers_tokens_input_per_call`	histogram	Input tokens per call
`smithers_tokens_output_per_call`	histogram	Output tokens per call
`smithers_tokens_context_window_per_call`	histogram	Context window per call

Database

Metric	Kind	Meaning
`smithers_db_retries`	counter	DB retries
`smithers_db_transaction_retries`	counter	DB transaction retries
`smithers_db_transaction_rollbacks`	counter	DB transaction rollbacks
`smithers_db_query_ms`	histogram	DB query duration
`smithers_db_transaction_ms`	histogram	DB transaction duration

HTTP and hot reload

Metric	Kind	Meaning
`smithers_http_requests`	counter	HTTP requests
`smithers_http_request_duration_ms`	histogram	HTTP request duration
`smithers_hot_reloads`	counter	Hot reloads
`smithers_hot_reload_failures`	counter	Hot reload failures
`smithers_hot_reload_duration_ms`	histogram	Hot reload duration

Tools

Metric	Kind	Meaning
`smithers_tool_calls_total`	counter	Tool calls
`smithers_tool_calls_errors_total`	counter	Tool call errors
`smithers_tool_output_truncated_total`	counter	Tool outputs truncated
`smithers_tool_duration_ms`	histogram	Tool duration

Sandbox

Metric	Kind	Meaning
`smithers_sandbox_created_total`	counter	Sandboxes created
`smithers_sandbox_completed_total`	counter	Sandboxes completed
`smithers_sandbox_active`	gauge	Active sandboxes
`smithers_sandbox_duration_ms`	histogram	Sandbox duration
`smithers_sandbox_bundle_size_bytes`	histogram	Sandbox bundle size
`smithers_sandbox_transport_duration_ms`	histogram	Sandbox transport duration
`smithers_sandbox_patch_count`	histogram	Sandbox patch count

Memory

Metric	Kind	Meaning
`smithers_memory_fact_reads`	counter	Memory fact reads
`smithers_memory_fact_writes`	counter	Memory fact writes
`smithers_memory_recall_queries`	counter	Memory recall queries
`smithers_memory_message_saves`	counter	Memory messages saved
`smithers_memory_recall_duration_ms`	histogram	Memory recall duration

OpenAPI tools

Metric	Kind	Meaning
`smithers_openapi_tool_calls`	counter	OpenAPI tool calls
`smithers_openapi_tool_call_errors`	counter	OpenAPI tool call errors
`smithers_openapi_tool_duration_ms`	histogram	OpenAPI tool duration

VCS and process

Metric	Kind	Meaning
`smithers_vcs_duration_ms`	histogram	VCS operation duration
`smithers_process_uptime_seconds`	gauge	Process uptime
`smithers_process_memory_rss_bytes`	gauge	Process RSS memory
`smithers_process_heap_used_bytes`	gauge	Process heap used

Source smithersMetrics.js · metrics · Tests effect-metrics-definitions.test.js · See also Server integration, Event types

Overview

Core API

Runtime

Types & errors

Gateway RPC

Observability API

SmithersObservability

Layers

createSmithersObservabilityLayer

createSmithersRuntimeLayer

createSmithersOtelLayer

resolveSmithersObservabilityOptions

trackSmithersEvent

Prometheus

renderPrometheusMetrics

prometheusContentType

Metrics catalog

Runs and nodes

Approvals and timers

Cache and tokens

Database

HTTP and hot reload

Tools

Sandbox

Memory

OpenAPI tools

VCS and process

​SmithersObservability

​Layers

​createSmithersObservabilityLayer

​createSmithersRuntimeLayer

​createSmithersOtelLayer

​resolveSmithersObservabilityOptions

​trackSmithersEvent

​Prometheus

​renderPrometheusMetrics

​prometheusContentType

​Metrics catalog

​Runs and nodes

​Approvals and timers

​Cache and tokens

​Database

​HTTP and hot reload

​Tools

​Sandbox

​Memory

​OpenAPI tools

​VCS and process

SmithersObservability

Layers

createSmithersObservabilityLayer

createSmithersRuntimeLayer

createSmithersOtelLayer

resolveSmithersObservabilityOptions

trackSmithersEvent

Prometheus

renderPrometheusMetrics

prometheusContentType

Metrics catalog

Runs and nodes

Approvals and timers

Cache and tokens

Database

HTTP and hot reload

Tools

Sandbox

Memory

OpenAPI tools

VCS and process