eval-author is an archived example, not installed by init. Copy it from examples/init-pack/ with its dependency closure, or ask create-workflow to build an equivalent.
eval-author turns plain-English acceptance criteria into a runnable Smithers eval
suite: cases, a .jsonl fixture under .smithers/evals/, and the exact
bunx smithers-orchestrator eval command to run it (see Stages below). Use it when
you have a goal or acceptance criteria and want a repeatable, regression-safe check
for a workflow.
Stages
- derive: turn the criteria into a structured suite, with a kebab-case
suiteNameand a list of cases (id,input,expected,rubric). - write: write the JSONL fixture to
.smithers/evals/<suiteName>.jsonland return itspath,caseCount, and therunCommand(bunx smithers-orchestrator eval <workflow> --cases .smithers/evals/<suiteName>.jsonl --suite <suiteName>).
Inputs
The fixture’s assertions support
status, output (exact match), and
outputContains (partial match). See Recipes
for the eval-suite format and how reports land in .smithers/evals/<suite>.json.