Agent Trace OTEL Log Verification

Local stack

Start the local stack:

docker compose -f observability/docker-compose.otel.yml up -d
scripts/obs-wait-healthy.sh

Or use the all-in-one reset:

scripts/obs-reset.sh

Expected endpoints:

Grafana: http://localhost:3001
Prometheus: http://localhost:9090
Tempo: http://localhost:3200
Loki: http://localhost:3100
OTEL collector HTTP: http://localhost:4318

Validate the stack:

docker compose -f observability/docker-compose.otel.yml ps
curl -sf http://localhost:3100/ready
curl -sf http://localhost:3001/api/datasources | jq 'map({name,type,url})'
docker logs observability-otel-collector-1 | tail -n 80

Enable OTEL export

export SMITHERS_OTEL_ENABLED=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=smithers-dev
export SMITHERS_LOG_FORMAT=json

Reproduce with any agent run

With the OTEL env enabled, every agent task emits canonical trace events (AgentTraceEvent / AgentTraceSummary) plus persisted provider session transcript rows, and exports them to the collector. Launch any agent workflow with an explicit run id:

bun run apps/cli/src/index.js up .smithers/workflows/smoketest.tsx \
  --run-id agent-trace-demo \
  --annotations '{"custom.demo":true,"custom.ticket":"OBS-123"}'

Durable-local verification:

jq 'select(.type == "AgentTraceEvent" or .type == "AgentTraceSummary")' \
  .smithers/executions/agent-trace-demo/logs/stream.ndjson

find .smithers/executions/agent-trace-demo/logs/agent-trace -type f -maxdepth 1 -print

Then query Loki for AgentTraceEvent / AgentTraceSummary lines and the custom.demo / custom.ticket annotations to confirm the export path end to end.

The dedicated workflows/agent-trace-otel-demo.tsx demo workflow and the scripts/verify-observability.sh runner were removed in 0.30.0; the generic repro above replaces them.

Loki queries

Smithers OTEL attributes are exposed to Loki as sanitized structured metadata fields such as:

smithers_event_category
run_id
workflow_path
node_id
node_attempt
agent_family
agent_capture_mode
trace_completeness
event_kind
session_row_type

Smithers exports two related Loki log families:

agent-trace: normalized canonical execution events such as deltas, tool lifecycle, usage, and capture warnings/errors
agent-session: provider transcript/session rows observed live or backfilled from persisted session logs

Both families share the same run/workflow/node/attempt/agent correlation fields and redaction rules. artifact.created stays local-only, not exported to Loki. Use {service_name="smithers-dev"} as the stream selector, then filter on structured metadata in the LogQL pipeline; | json inspects the structured log body. All events for one run:

{service_name="smithers-dev"} | run_id="agent-trace-demo"

One node attempt:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | node_id="NODE_ID" | node_attempt="1"

Thinking deltas only:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | event_kind="assistant.thinking.delta"

Tool execution only:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | event_kind=~"tool\.execution\..*"

Capture errors only:

{service_name="smithers-dev"} | event_kind="capture.error"

Inspect the structured JSON body for one run:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | json

Canonical trace rows only:

{service_name="smithers-dev"} | smithers_event_category="agent-trace" | run_id="agent-trace-demo"

Provider session rows only:

{service_name="smithers-dev"} | smithers_event_category="agent-session" | run_id="agent-trace-demo"

Pi persisted session metadata:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | node_id="NODE_ID" | session_row_type="model_change"

Claude persisted session queue events:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | node_id="NODE_ID" | session_row_type="queue-operation"

Codex persisted session reasoning rows:

{service_name="smithers-dev"} | run_id="agent-trace-demo" | node_id="NODE_ID" | session_row_type="event_msg"

Redaction proof query:

{service_name="smithers-dev"} | run_id="agent-trace-demo" |= "REDACTED"

API query examples

Equivalent direct Loki API checks:

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-demo"' \
  --data-urlencode 'limit=200' | jq '.data.result[] | {stream, values: (.values | length)}'

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-demo" | event_kind="assistant.thinking.delta"' \
  --data-urlencode 'limit=20' | jq '.data.result[]?.values[]?[1]'

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-demo" | node_id="NODE_ID" | session_row_type="event_msg"' \
  --data-urlencode 'limit=20' | jq '.data.result[]?.values[]?[1]'

Tempo trace checks

Tempo search shows Smithers spans once a workflow has run:

curl -s http://localhost:3200/api/search | jq .
curl -s http://localhost:3200/api/search/tags | jq .
curl -s 'http://localhost:3200/api/search/tag/service.name/values' | jq .
curl -s 'http://localhost:3200/api/search/tag/runId/values' | jq .

Inspect one trace directly:

TRACE_ID=$(curl -s http://localhost:3200/api/search \
  | jq -r '.traces[] | select(.rootTraceName=="engine:run-workflow") | .traceID' \
  | head -n 1)

curl -s http://localhost:3200/api/traces/$TRACE_ID | jq .

Expected trace attributes include at least:

service.name = smithers-dev
runId = <RUN_ID>
workflowPath = <workflow path>

Verification checklist

stack starts in Docker
Loki is present and queryable
collector logs pipeline is active
Pi traces show text deltas, thinking deltas, tool execution lifecycle, final message, usage, and run/node/attempt correlation
Pi, Claude, and Codex session transcript rows are queryable in Loki
second agent family is exported with truthful final-only completeness classification
Gemini stream-json attempts preserve structured deltas truthfully
malformed or truncated structured streams emit capture.error and classify as capture-failed
artifact write failures emit capture.warning and degrade to partial-observed without losing durable DB truth
Tempo search shows Smithers spans and trace attributes including runId
Prometheus is still scraping the collector
secrets are redacted from canonical events, OTEL log bodies, and persisted trace artifacts

Automated coverage

apps/observability/tests/agentTrace.test.js covers the canonical contract:

capability profiles for every agent family
family + capture-mode detection
Pi / Claude / Codex / Gemini structured event normalization
redaction rules for API keys, bearer tokens, and secret-ish key=value pairs
canonical OTEL log record shaping with stable Loki query attributes
session-event OTEL log record shaping

The full provider-specific test matrix from PR #119 (truncated streams, artifact write failures, multi-family final-only classification) is partially covered; a follow-up will extend it.

Start

Articles

Learn

Capabilities

Build Workflows

Run and Operate

Workflow Pack

Components

Integrations

Agent Support

Examples

Contributing

Agent Trace OTEL Log Verification

Local stack

Enable OTEL export

Reproduce with any agent run

Loki queries

API query examples

Tempo trace checks

Verification checklist

Automated coverage

​Local stack

​Enable OTEL export

​Reproduce with any agent run

​Loki queries

​API query examples

​Tempo trace checks

​Verification checklist

​Automated coverage

Local stack

Enable OTEL export

Reproduce with any agent run

Loki queries

API query examples

Tempo trace checks

Verification checklist

Automated coverage