Skip to main content
Eval suites protect an important workflow with repeatable cases, each getting a persisted run record, stable metadata, and a JSON report you can check in CI. You need a workflow file (e.g. workflow.tsx) and a working bunx smthrs init scaffold; each case runs that workflow with the given input and checks the result.
API reference: Scorers lists every scorer, its options, and links to source and tests.

​
1. Create cases

Create evals/smoke.jsonl:
Case files can also be a JSON array, or an object with a cases array:
Supported expected checks:
  • status: one of finished, continued, failed, cancelled, waiting-approval, waiting-event, or waiting-timer
  • output: exact JSON match against the workflow output
  • outputContains: recursive partial JSON match; array elements match by containment (at least one must match the partial object)
  • errorContains: substring match against thrown errors
For standard workflows, output assertions match the persisted output snapshot, keyed by output name.

​
Judge assertions

Use a judge assertion when correctness depends on meaning or tone rather than an exact JSON match:
judge.instructions must be a non-empty string; judge.threshold is optional (0 to 1, default 0.8). A case with both expected and judge passes only when every deterministic assertion passes and the judge score meets its threshold. Smithers automatically picks an authenticated local judge agent; use --judge-provider and --judge-model to select one explicitly:
If no usable judge agent is configured, judge assertions fail with setup guidance in their reason field; cases still run and the report still writes.

​
2. Dry-run the plan

Dry-run mode prints the planned case IDs, run IDs, and judge requirements without touching the database; pass --run-label <label> to keep dry-run and execution IDs consistent.

​
3. Execute the suite

By default the report is written to .smithers/evals/smoke.json; use --report path/to/report.json for a different location. The command exits 0 when all cases pass, 1 when any genuinely fail, 4 for invalid case files, and 5 when the only reds are INCONCLUSIVE: cases that died on a harness or environment fault (connection refused, TLS failure, network denied, missing binary, out of memory, rate limit) before the workflow’s behavior could be observed. Exit 5 means fix the harness, not the workflow; a loop that treats it like 1 will iterate on product code the suite never actually tested.
Warning: Bash-tool network access is off by default for every eval run. Loopback stays reachable (localhost, 127.0.0.1, *.localhost, unix sockets), so local compositions work, but a suite whose workflow reaches any real external service needs --allow-network or every case dies with TOOL_NETWORK_DISABLED and grades INCONCLUSIVE.
Tip: Pass a workflow ID (implement) instead of a file path for workflows discovered from .smithers/workflows to run the same suite on every checkout without hard-coded entry paths.

​
4. Use structured output in CI

The JSON payload includes the suite summary, per-case assertions, run IDs, inputs, outputs, errors, and report path.

​
Options that matter in production

  • --concurrency N: run cases concurrently; keep it low for stateful or expensive workflows.
  • --run-label LABEL: append a stable label to run IDs, useful for CI build IDs or benchmark names.
  • --max-concurrency N: pass a per-workflow task concurrency cap to each case.
  • --max-cases N: shard or sample a large suite.
  • --no-include-output: omit workflow outputs from the report when they’re too large or sensitive.
  • --allow-network: enable network access for bash tools in cases that need it.
  • --root PATH: set the sandbox root for tool execution.