> ## Documentation Index > Fetch the complete documentation index at: https://smithers.sh/llms.txt > Use this file to discover all available pages before exploring further. # Evals and Optimization > Score task output with attached scorers, protect workflows with JSONL regression suites, and improve prompts with GEPA optimization. An agent workflow that "seems fine" is an unmeasured workflow. The failure mode is always the same: someone tweaks a prompt to fix one bad output, ships it, and silently breaks three cases nobody re-checked. Two weeks later a model upgrade shifts behavior again, and there is no record of what "good" looked like, so nobody notices. Vibes-based evaluation has no memory; every change is graded against whatever example the author happened to try last. The honest complication for coding agents specifically: there is often **no ground-truth label**. "Refactor this module" has thousands of acceptable outputs. What you *can* measure is cheaper and still decisive — did the run finish, did the output match its schema, does it contain the fields downstream tasks depend on, did it stay within a latency budget, and does an LLM judge with explicit instructions rate it above a threshold. None of these prove the output is optimal. Together, run after run, they catch regressions the moment they happen, which is what one-off manual review never does. Smithers ships this as three layers on the same substrate: **scorers** attached to tasks (graded on every real run, persisted next to the run), **eval suites** (JSONL cases replayed through the real workflow by `smithers eval`, producing a checked-in report), and **GEPA prompt optimization** (`smithers optimize`, which rewrites task prompts and only keeps the rewrite if the suite score actually improves). ## How the industry does it | System | Mechanism | Tradeoff | | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | [DSPy](https://dspy.ai) | Programs, not prompts; optimizers like [MIPROv2](https://arxiv.org/abs/2406.11695) tune instructions and few-shot demos against a metric | Excellent for compilable pipelines; requires restructuring your app as DSPy modules | | [GEPA](https://arxiv.org/abs/2507.19457) | Reflective prompt evolution: an LLM reads failure traces in natural language and proposes prompt mutations, kept via Pareto selection; beats GRPO by \~10% with far fewer rollouts | Needs a scored eval suite and a capable optimizer model; per-prompt, not weight-level | | [OpenAI Evals](https://github.com/openai/evals) | YAML/JSONL registry of eval templates (match, includes, model-graded) | Framework predates agents; graded completions, not multi-step runs | | [LangSmith](https://docs.smith.langchain.com/evaluation) | Datasets + evaluator functions (heuristic or LLM-as-judge) over traced runs, pairwise experiments in a UI | Hosted; deep when your stack is LangChain-traced | | [Braintrust](https://www.braintrust.dev) | `Eval()` harness with scorers (autoevals library), experiment diffing, online scoring of production logs | Strong product; commercial, hosted-first | | [Promptfoo](https://www.promptfoo.dev) | Declarative YAML test matrix across prompts × providers × assertions; red-teaming | Great for prompt-level A/B; not built around durable multi-step workflows | | [Inspect AI](https://inspect.aisi.org.uk) (UK AISI) | Python eval framework with solvers, tools, and sandboxed agentic tasks; the safety-eval standard | Research-grade rigor; heavier to adopt for product regression suites | | [HELM](https://crfm.stanford.edu/helm/) / [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | Standardized multi-metric benchmarks over model APIs | Grades models, not your workflow | | [SWE-bench](https://www.swebench.com) / [Terminal-Bench](https://www.tbench.ai) | Agentic benchmarks with executable ground truth: did the patch make the held-out tests pass, did the terminal task's checks succeed | The gold standard for coding agents — when executable checks exist; most product workflows have none | Two industry lessons Smithers takes seriously. First, from SWE-bench: **executable checks beat opinions** — when you can assert on status, schema, or content, do that, and reserve the judge for what only judgment can grade. Second, from the [MT-Bench / LLM-as-a-judge paper](https://arxiv.org/abs/2306.05685) (Zheng et al.): judges have **position bias** (favoring the first answer in pairwise comparison), **verbosity bias**, and **self-preference** (favoring outputs in their own style). Smithers' judge assertions are therefore single-output graded against written instructions with a threshold — no pairwise comparison, so position bias doesn't apply — but self-preference and verbosity bias remain real; using a different provider for the judge than for the workflow (`--judge-provider`) is the standard mitigation. Where another tool is the right choice: comparing base models, use lm-evaluation-harness or HELM. Safety evaluations, use Inspect. Prompt-matrix A/B across providers outside a workflow, Promptfoo is faster to set up. Smithers evals exist for one job those don't do: regressing a **durable multi-step workflow** — retries, approvals, structured outputs and all — as the unit under test. ## Scorers: grading every real run A scorer grades one task's output to a number in 0–1. Attach scorers to a `Task` via the `scorers` prop; they run after the task completes, never block or fail the workflow, and persist to the `_smithers_scorers` table. ```tsx theme={"theme":{"light":"github-light","dark":"github-dark"}} import { createScorer, schemaAdherenceScorer, latencyScorer, toxicityScorer, } from "smithers-orchestrator"; const mentionsRisk = createScorer({ id: "mentions-risk", name: "Mentions Risk", description: "Output names at least one concrete risk", score: async ({ output }) => ({ score: /risk/i.test(String(output)) ? 1 : 0, }), }); Analyze the report and call out risks. ``` The `sampling` binding (`all` | `ratio` | `none`) is how you afford judge-based scorers in production: score 10% of runs instead of all of them. Built-ins cover the common axes — `faithfulnessScorer` (grounded in context), `relevancyScorer`, `schemaAdherenceScorer`, `latencyScorer`, `llmJudge` for custom rubrics — see the [Scorers API reference](/reference/scorers) for the full list, including the delegation-chain scorers. Read scores back per run: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} bunx smithers-orchestrator scores RUN_ID ``` or aggregate across runs with `aggregateScores(adapter, { scorerId })` — count, mean, min/max, p50, stddev — straight from the persisted table. This is the same shape as Braintrust's online scoring of production logs, minus the hosted dashboard: your scores live in your own database. ## Eval suites: the regression contract Cases are JSONL (or JSON), one per line, each naming an input and what must be true of the result: ```jsonl theme={"theme":{"light":"github-light","dark":"github-dark"}} {"id":"simple-request","input":{"prompt":"Summarize the repository"},"expected":{"status":"finished"}} {"id":"structured","input":{"prompt":"Return the key risks"},"expected":{"status":"finished","outputContains":{"analysis":[{"riskLevel":"low"}]}}} {"id":"tone","input":{"prompt":"Summarize the launch plan"},"expected":{"status":"finished"},"judge":{"instructions":"Polite summary that mentions the deadline","threshold":0.7}} ``` Deterministic checks: `status`, `output` (exact JSON), `outputContains` (recursive subset match), `errorContains`. The optional `judge` runs an LLM-as-judge pass; a case with both passes only when every deterministic assertion passes *and* the judge score meets its threshold — the executable-first, judge-second ordering. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} bunx smithers-orchestrator eval workflow.tsx --cases evals/smoke.jsonl --suite smoke --force ``` Each case executes the real workflow as a persisted run and the report lands at `.smithers/evals/smoke.json` (per-case assertions, run IDs, outputs, summary) — check it into CI with `--format json`. Exit codes are the contract: `0` all pass, `1` genuine failures, `4` invalid case files, and `5` when the only reds are **inconclusive** — harness faults (network refused, missing binary, rate limit) that killed the case before the workflow's behavior was observed. That distinction matters: a CI loop that treats exit `5` like `1` will "fix" product code the suite never actually tested. Network is off by default for eval runs (loopback stays open); pass `--allow-network` when a case genuinely needs the outside world. Full walkthrough with all flags: [Evals quickstart](/guides/evals-quickstart). ## Optimization: GEPA over your suite Once a suite scores a workflow, the score becomes a target. `smithers optimize` implements GEPA-style reflective prompt evolution ([Agrawal et al., 2025](https://arxiv.org/abs/2507.19457)): an optimizer model reads the baseline eval failures, proposes rewritten prompts for the workflow's agent tasks, and Smithers verifies the rewrite by running the suite again. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} OPENAI_API_KEY=... bunx smithers-orchestrator optimize workflow.tsx \ --cases evals/smoke.jsonl \ --suite smoke-gepa \ --artifact .smithers/optimizations/smoke-gepa.json ``` The command runs the suite twice — baseline prompts, then GEPA-patched prompts — and writes the artifact **only if the optimized score improves** by at least `--min-improvement`. The artifact records baseline score, optimized score, improvement, and the per-`nodeId` prompt patches; it patches agent-backed `` prompts only — structure, schemas, retries, and approvals are untouched. Apply it to future evals with `--optimization `. Optimizer backends span OpenAI-compatible, Anthropic, Gemini, Cerebras, and Moonshot APIs, plus a deterministic `--provider heuristic` for tests; see [Workflow optimization](/guides/workflow-optimization). This is a deliberately narrower bet than DSPy: no program restructuring, no few-shot demo bootstrapping (MIPROv2's other half) — just prompt patches keyed to task node IDs, gated on a measured improvement over your own suite. ## Guarantees and limits * Scorers **never block or fail** a task — they run after completion, and scorer failures are logged, not thrown. Corollary: a scorer cannot gate a run; use a review/approval node for gating. * Every scorer result and eval run is **persisted locally** — no hosted service, queryable via `smithers scores`, `aggregateScores`, or SQL against `_smithers_scorers`. * Judge assertions inherit LLM-judge weaknesses: self-preference and verbosity bias ([Zheng et al.](https://arxiv.org/abs/2306.05685)) are not solved, only mitigated by cross-provider judging and low-variance rubric instructions. An unparseable judge reply scores 0, and a missing judge agent fails the assertion with setup guidance rather than crashing the suite. * Eval cases run the **real workflow with real agents** — suites cost real tokens and real minutes. Use `--max-cases` to shard and `--concurrency` conservatively. * `optimize` guards against regressions on **your suite**, not in general: a small suite can be overfit by a prompt patch exactly the way a small test set overfits any optimizer. Grow the suite before trusting the artifact. * There is no hosted experiment-diff UI comparable to LangSmith or Braintrust; comparison happens through report files and score queries. ## Operating it ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} bunx smithers-orchestrator eval workflow.tsx --cases evals/smoke.jsonl --suite smoke --dry-run # plan bunx smithers-orchestrator eval workflow.tsx --cases evals/smoke.jsonl --suite smoke --force # run + report bunx smithers-orchestrator scores RUN_ID # scorer results for one run bunx smithers-orchestrator optimize workflow.tsx --cases evals/smoke.jsonl --suite s --artifact a.json bunx smithers-orchestrator eval workflow.tsx --cases evals/smoke.jsonl --optimization a.json # verify a patch ``` ## See also * [Evals quickstart](/guides/evals-quickstart) — case format, judge flags, CI wiring * [Scorers API](/reference/scorers) — every scorer, `ScorersMap`, `aggregateScores` * [Workflow optimization](/guides/workflow-optimization) — GEPA providers, artifacts, the Cerebras demo * [Structured output](/guides/structured-output) — schemas that make `outputContains` assertions possible