Skip to main content
eval-author is an archived example and is not installed by init. Copy it from examples/init-pack/ with its dependency closure, or ask create-workflow to build an equivalent. Once installed, run it with the command below. eval-author turns plain-English acceptance criteria into a runnable Smithers eval suite. It derives a set of cases (each an input plus an expected assertion and a review rubric), then writes a .jsonl fixture under .smithers/evals/ and reports the exact bunx smithers-orchestrator eval command to run it. Use it when you have a goal or acceptance criteria and want a repeatable, regression-safe check for a workflow.

​
Stages

  1. derive: turn the criteria into a structured suite, with a kebab-case suiteName and a list of cases (id, input, expected, rubric).
  2. write: write the JSONL fixture to .smithers/evals/<suiteName>.jsonl and return its path, caseCount, and the runCommand (bunx smithers-orchestrator eval <workflow> --cases .smithers/evals/<suiteName>.jsonl --suite <suiteName>).

​
Inputs

The fixture’s assertions support status, output (exact match), and outputContains (partial match). See Recipes for the eval-suite format and how reports land in .smithers/evals/<suite>.json.