Skip to main content
eval-author is an archived example, not installed by init. Copy it from examples/init-pack/ with its dependency closure, or ask create-workflow to build an equivalent. eval-author turns plain-English acceptance criteria into a runnable Smithers eval suite: cases, a .jsonl fixture under .smithers/evals/, and the exact bunx smithers-orchestrator eval command to run it (see Stages below). Use it when you have a goal or acceptance criteria and want a repeatable, regression-safe check for a workflow.

​
Stages

  1. derive: turn the criteria into a structured suite, with a kebab-case suiteName and a list of cases (id, input, expected, rubric).
  2. write: write the JSONL fixture to .smithers/evals/<suiteName>.jsonl and return its path, caseCount, and the runCommand (bunx smithers-orchestrator eval <workflow> --cases .smithers/evals/<suiteName>.jsonl --suite <suiteName>).

​
Inputs

The fixture’s assertions support status, output (exact match), and outputContains (partial match). See Recipes for the eval-suite format and how reports land in .smithers/evals/<suite>.json.