Skip to main content
eval-suite-run is a hidden system workflow launched by the evals gateway extension’s client: it’s what “Run” does for a saved eval suite. Not meant to be authored by hand, it ships in the global seeded pack and every smithers gateway registers it, backed by suites saved through ext.evals.saveSuite.

​
What it does

Given { suiteId }, the workflow:
  1. plan reads the saved suite (name, target workflowKey, parsed case dataset) and seeds one queued row per case in the workspace DB upfront, so the live results table has something to render from the run’s start.
  2. cases is a <Parallel> fan-out with one <Task> per dataset case. Each case launches its OWN real, separately-addressable child workflow run (the suite’s target workflow, given that case’s input) via executeChildWorkflow with an explicit, collision-free run id, so a parent resume never re-launches a completed case. The task never throws: a failed or errored child run is graded, not fatal, so its attached scorer (eval-assertions) always fires and records a real score.
  3. verdict rolls every case’s pass/fail into { pass, paragraph }, the same verifier-output contract every delegated run’s settle path strict-parses.

​
Grading

A case’s expected value (from the authored dataset) is graded one of two ways:
  • Assertion spec: expected is absent, or an object whose keys are only status / output / outputContains / errorContains (the spec bunx smthrs eval has always supported).
  • Expected output: any other expected value is the literal/subset expected output: objects and arrays match by subset, everything else by deep equality, plus the implicit “the case run finished” assertion.
Target workflow requirement for output-value grading: a case’s actual comes from the target workflow’s designated output (RunResult.output, the same mechanism <Subflow> relies on for its child result), which requires either a schema key literally named output in the target workflow’s createSmithers({...}) call, or an explicit smithers(build, { output: outputs.<key> }). Without either, the workflow still runs and grades fine on status/errorContains, but expected-output and outputContains comparisons always see an empty actual.

​
Run it explicitly

Useful for debugging a suite outside the Evals canvas:
maxConcurrency (1-16, default 4) caps how many cases run at once. Like other system workflows, eval-suite-run is hidden from default listings but available via workflow list --system and explicit workflow run eval-suite-run.