smithers eval, producing a checked-in report), and GEPA prompt optimization (smithers optimize, which rewrites task prompts and only keeps the rewrite if the suite score actually improves).
How the industry does it
Two industry lessons Smithers takes seriously. First, from SWE-bench: executable checks beat opinions — when you can assert on status, schema, or content, do that, and reserve the judge for what only judgment can grade. Second, from the MT-Bench / LLM-as-a-judge paper (Zheng et al.): judges have position bias (favoring the first answer in pairwise comparison), verbosity bias, and self-preference (favoring outputs in their own style). Smithers’ judge assertions are therefore single-output graded against written instructions with a threshold — no pairwise comparison, so position bias doesn’t apply — but self-preference and verbosity bias remain real; using a different provider for the judge than for the workflow (
--judge-provider) is the standard mitigation.
Where another tool is the right choice: comparing base models, use lm-evaluation-harness or HELM. Safety evaluations, use Inspect. Prompt-matrix A/B across providers outside a workflow, Promptfoo is faster to set up. Smithers evals exist for one job those don’t do: regressing a durable multi-step workflow — retries, approvals, structured outputs and all — as the unit under test.
Scorers: grading every real run
A scorer grades one task’s output to a number in 0–1. Attach scorers to aTask via the scorers prop; they run after the task completes, never block or fail the workflow, and persist to the _smithers_scorers table.
sampling binding (all | ratio | none) is how you afford judge-based scorers in production: score 10% of runs instead of all of them. Built-ins cover the common axes — faithfulnessScorer (grounded in context), relevancyScorer, schemaAdherenceScorer, latencyScorer, llmJudge for custom rubrics — see the Scorers API reference for the full list, including the delegation-chain scorers.
Read scores back per run:
aggregateScores(adapter, { scorerId }) — count, mean, min/max, p50, stddev — straight from the persisted table. This is the same shape as Braintrust’s online scoring of production logs, minus the hosted dashboard: your scores live in your own database.
Eval suites: the regression contract
Cases are JSONL (or JSON), one per line, each naming an input and what must be true of the result:status, output (exact JSON), outputContains (recursive subset match), errorContains. The optional judge runs an LLM-as-judge pass; a case with both passes only when every deterministic assertion passes and the judge score meets its threshold — the executable-first, judge-second ordering.
.smithers/evals/smoke.json (per-case assertions, run IDs, outputs, summary) — check it into CI with --format json. Exit codes are the contract: 0 all pass, 1 genuine failures, 4 invalid case files, and 5 when the only reds are inconclusive — harness faults (network refused, missing binary, rate limit) that killed the case before the workflow’s behavior was observed. That distinction matters: a CI loop that treats exit 5 like 1 will “fix” product code the suite never actually tested. Network is off by default for eval runs (loopback stays open); pass --allow-network when a case genuinely needs the outside world.
Full walkthrough with all flags: Evals quickstart.
Optimization: GEPA over your suite
Once a suite scores a workflow, the score becomes a target.smithers optimize implements GEPA-style reflective prompt evolution (Agrawal et al., 2025): an optimizer model reads the baseline eval failures, proposes rewritten prompts for the workflow’s agent tasks, and Smithers verifies the rewrite by running the suite again.
--min-improvement. The artifact records baseline score, optimized score, improvement, and the per-nodeId prompt patches; it patches agent-backed <Task> prompts only — structure, schemas, retries, and approvals are untouched. Apply it to future evals with --optimization <artifact>. Optimizer backends span OpenAI-compatible, Anthropic, Gemini, Cerebras, and Moonshot APIs, plus a deterministic --provider heuristic for tests; see Workflow optimization.
This is a deliberately narrower bet than DSPy: no program restructuring, no few-shot demo bootstrapping (MIPROv2’s other half) — just prompt patches keyed to task node IDs, gated on a measured improvement over your own suite.
Guarantees and limits
- Scorers never block or fail a task — they run after completion, and scorer failures are logged, not thrown. Corollary: a scorer cannot gate a run; use a review/approval node for gating.
- Every scorer result and eval run is persisted locally — no hosted service, queryable via
smithers scores,aggregateScores, or SQL against_smithers_scorers. - Judge assertions inherit LLM-judge weaknesses: self-preference and verbosity bias (Zheng et al.) are not solved, only mitigated by cross-provider judging and low-variance rubric instructions. An unparseable judge reply scores 0, and a missing judge agent fails the assertion with setup guidance rather than crashing the suite.
- Eval cases run the real workflow with real agents — suites cost real tokens and real minutes. Use
--max-casesto shard and--concurrencyconservatively. optimizeguards against regressions on your suite, not in general: a small suite can be overfit by a prompt patch exactly the way a small test set overfits any optimizer. Grow the suite before trusting the artifact.- There is no hosted experiment-diff UI comparable to LangSmith or Braintrust; comparison happens through report files and score queries.
Operating it
See also
- Evals quickstart — case format, judge flags, CI wiring
- Scorers API — every scorer,
ScorersMap,aggregateScores - Workflow optimization — GEPA providers, artifacts, the Cerebras demo
- Structured output — schemas that make
outputContainsassertions possible