> ## Documentation Index
> Fetch the complete documentation index at: https://smithers.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Trace OTEL Log Verification

> Reproducible local verification for Smithers canonical trace and provider session transcript OTEL log export with Loki.

## Local stack

Start the local stack:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker compose -f observability/docker-compose.otel.yml up -d
scripts/obs-wait-healthy.sh
```

Or use the all-in-one reset:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scripts/obs-reset.sh
```

Expected endpoints:

* Grafana: `http://localhost:3001`
* Prometheus: `http://localhost:9090`
* Tempo: `http://localhost:3200`
* Loki: `http://localhost:3100`
* OTEL collector HTTP: `http://localhost:4318`

Validate the stack:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
docker compose -f observability/docker-compose.otel.yml ps
curl -sf http://localhost:3100/ready
curl -sf http://localhost:3001/api/datasources | jq 'map({name,type,url})'
docker logs observability-otel-collector-1 | tail -n 80
```

## Enable OTEL export

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
export SMITHERS_OTEL_ENABLED=1
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=smithers-dev
export SMITHERS_LOG_FORMAT=json
```

## Demo workflow

Use the built-in reproducible workflow at `workflows/agent-trace-otel-demo.tsx`.
It emits:

* one Pi-like high-fidelity attempt with canonical trace events plus persisted session transcript rows
* one Claude-like structured attempt with canonical trace events plus provider session transcript rows
* one Codex-like structured attempt with canonical trace events plus provider session transcript rows
* one Gemini-like structured attempt
* one SDK-style final-only attempt
* stable run annotations for Loki queries

Run the success case:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bun run apps/cli/src/index.js up workflows/agent-trace-otel-demo.tsx \
  --run-id agent-trace-otel-demo \
  --annotations '{"custom.demo":true,"custom.ticket":"OBS-123"}'
```

Run the malformed JSON failure case:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
bun run apps/cli/src/index.js up workflows/agent-trace-otel-demo.tsx \
  --run-id agent-trace-otel-demo-fail \
  --input '{"failureMode":"malformed-json"}' \
  --annotations '{"custom.demo":true,"custom.ticket":"OBS-ERR"}'
```

Optional durable-local verification:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
jq 'select(.type == "AgentTraceEvent" or .type == "AgentTraceSummary")' \
  .smithers/executions/agent-trace-otel-demo/logs/stream.ndjson

find .smithers/executions/agent-trace-otel-demo/logs/agent-trace -type f -maxdepth 1 -print
```

## End-to-end verification script

`scripts/verify-observability.sh` runs the full reset → start → demo workflow (success + failure) → Loki/Tempo query verification flow and writes a timestamped evidence bundle under `tmp/verification/<timestamp>`. Use this as the canonical reproduction path for reviewers:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
scripts/verify-observability.sh
```

The bundle contains every Loki query result, the Tempo trace export, Grafana datasource health, and the success/failure CLI run logs as JSON.

## Loki queries

Smithers OTEL attributes are exposed to Loki as sanitized structured metadata fields such as:

* `smithers_event_category`
* `run_id`
* `workflow_path`
* `node_id`
* `node_attempt`
* `agent_family`
* `agent_capture_mode`
* `trace_completeness`
* `event_kind`
* `session_row_type`

Smithers exports two related Loki log families:

* `agent-trace`: normalized canonical execution events such as deltas, tool lifecycle, usage, and capture warnings/errors
* `agent-session`: provider transcript/session rows observed live or backfilled from persisted session logs

Both families share the same run/workflow/node/attempt/agent correlation fields and the same redaction rules.
`artifact.created` remains local-only and is not exported to Loki.

Use `{service_name="smithers-dev"}` as the stream selector, then filter on structured metadata in the LogQL pipeline. Use `| json` to inspect the structured log body.

All events for one run:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo"
```

One node attempt:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | node_id="pi-rich-trace" | node_attempt="1"
```

Thinking deltas only:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | event_kind="assistant.thinking.delta"
```

Tool execution only:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | event_kind=~"tool\.execution\..*"
```

Capture errors only:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | event_kind="capture.error"
```

Inspect the structured JSON body for one run:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | json
```

Canonical trace rows only:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | smithers_event_category="agent-trace" | run_id="agent-trace-otel-demo"
```

Provider session rows only:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | smithers_event_category="agent-session" | run_id="agent-trace-otel-demo"
```

Pi persisted session metadata:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | node_id="pi-rich-trace" | session_row_type="model_change"
```

Claude persisted session queue events:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | node_id="claude-structured-trace" | session_row_type="queue-operation"
```

Codex persisted session reasoning rows:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | node_id="codex-structured-trace" | session_row_type="event_msg"
```

Redaction proof query:

```logql theme={"theme":{"light":"github-light","dark":"github-dark"}}
{service_name="smithers-dev"} | run_id="agent-trace-otel-demo" |= "REDACTED"
```

## API query examples

Equivalent direct Loki API checks:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-otel-demo"' \
  --data-urlencode 'limit=200' | jq '.data.result[] | {stream, values: (.values | length)}'

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | event_kind="assistant.thinking.delta"' \
  --data-urlencode 'limit=20' | jq '.data.result[]?.values[]?[1]'

curl -sG 'http://localhost:3100/loki/api/v1/query_range' \
  --data-urlencode 'query={service_name="smithers-dev"} | run_id="agent-trace-otel-demo" | node_id="codex-structured-trace" | session_row_type="event_msg"' \
  --data-urlencode 'limit=20' | jq '.data.result[]?.values[]?[1]'
```

## Tempo trace checks

Tempo search should show Smithers spans once a workflow has run:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -s http://localhost:3200/api/search | jq .
curl -s http://localhost:3200/api/search/tags | jq .
curl -s 'http://localhost:3200/api/search/tag/service.name/values' | jq .
curl -s 'http://localhost:3200/api/search/tag/runId/values' | jq .
```

Inspect one trace directly:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
TRACE_ID=$(curl -s http://localhost:3200/api/search \
  | jq -r '.traces[] | select(.rootTraceName=="engine:run-workflow") | .traceID' \
  | head -n 1)

curl -s http://localhost:3200/api/traces/$TRACE_ID | jq .
```

Expected trace attributes include at least:

* `service.name = smithers-dev`
* `runId = <RUN_ID>`
* `workflowPath = <workflow path>`

## Verification checklist

* stack starts successfully in Docker
* Loki is present and queryable
* collector logs pipeline is active
* Pi traces show text deltas, thinking deltas, tool execution lifecycle, final message, usage, and run/node/attempt correlation
* Pi session transcript rows are queryable in Loki
* Claude session transcript rows are queryable in Loki
* Codex session transcript rows are queryable in Loki
* second agent family is exported with truthful `final-only` completeness classification
* Gemini `stream-json` attempts preserve structured deltas truthfully
* malformed or truncated structured streams emit `capture.error` and classify as `capture-failed`
* artifact write failures emit `capture.warning` and degrade to `partial-observed` without losing durable DB truth
* Tempo search shows Smithers spans and trace attributes including `runId`
* Prometheus is still scraping the collector successfully
* secrets are redacted from canonical events, OTEL log bodies, and persisted trace artifacts

## Automated coverage

`apps/observability/tests/agentTrace.test.js` covers the canonical contract:

* capability profiles for every agent family
* family + capture-mode detection
* Pi / Claude / Codex / Gemini structured event normalization
* redaction rules for API keys, bearer tokens, and secret-ish key=value pairs
* canonical OTEL log record shaping with stable Loki query attributes
* session-event OTEL log record shaping

The full provider-specific test matrix from PR #119 (truncated streams, artifact write failures, multi-family final-only classification) is partially covered and will be extended in a follow-up.