Skip to main content
eval-author turns plain-English acceptance criteria into a runnable Smithers eval suite. It derives a set of cases (each an input plus an expected assertion and a review rubric), then writes a .jsonl fixture under .smithers/evals/ and reports the exact bunx smithers-orchestrator eval command to run it. Use it when you have a goal or acceptance criteria and want a repeatable, regression-safe check for a workflow. 
bunx smithers-orchestrator workflow run eval-author \
  --input '{"prompt":"Release notes must list every breaking change","workflow":".smithers/workflows/release.tsx"}'

​
Stages

  1. derive: turn the criteria into a structured suite, with a kebab-case suiteName and a list of cases (id, input, expected, rubric).
  2. write: write the JSONL fixture to .smithers/evals/<suiteName>.jsonl and return its path, caseCount, and the runCommand (bunx smithers-orchestrator eval <workflow> --cases .smithers/evals/<suiteName>.jsonl --suite <suiteName>).

​
Inputs

InputTypeDefault
promptstring"Describe the acceptance criteria / goal to turn into eval cases."
workflowstring or nullnull (leaves a <workflow> placeholder in the run command)
The fixture’s assertions support status, output (exact match), and outputContains (partial match). See Recipes for the eval-suite format and how reports land in .smithers/evals/<suite>.json.