Testing workflows

@smthrs/testing is the consumer-facing test surface for Smithers workflows; import it from the published facade:

import {
  auto,
  coverWorkflow,
  dryRun,
  expectFullCoverage,
  fakeAgent,
  renderPrompt,
  renderWorkflow,
  runTask,
  simMatchers,
  simulate,
  toHaveExecuted,
  toHaveExecutedInOrder,
  toHaveFinished,
} from "smthrs/testing";

Examples below use the createSmithers workflow API and bun:test; the testing helpers don’t replace workflow authoring components.

Cover a workflow in one call

coverWorkflow(workflowModule, options?) is the shortest conformance test for a complete workflow. It renders every frame, supplies validated schema-aware outputs for every agent, resolves approvals and human tasks, delivers waited events and signals, bounds loops, and runs each pass to completion. It uses an isolated temporary root and does not open the workflow’s configured database or call real agents, sandboxes, subflows, or compute functions.

import { expect, test } from "bun:test";
import WorkflowModule from "../simple-workflow.jsx";
import { coverWorkflow } from "smthrs/testing";

test("covers the workflow", async () => {
  const result = await coverWorkflow(WorkflowModule, { input: { topic: "Smithers" } });
  expect(result.coveredNodes).toContain("write");
});

By default, coverWorkflow calls expectFullCoverage(result) before returning. The assertion requires every pass to finish, every rendered task to execute, every expectedNodes entry to render, and every structured output to pass its schema. Set assert: false to inspect a failing result, or call expectFullCoverage later.

coverWorkflow checks workflow STRUCTURE, not behavior: agents, sandboxes, subflows, compute, and side effects are stubbed, so a red here is a graph or schema defect, never proof about your product. Keep the two loops separate. Behavior against real services belongs to bunx smthrs eval with real runs, and when an eval case dies on a harness or environment fault (connection refused, TLS, network denied) it grades INCONCLUSIVE and the command exits 5: repair the harness, do not iterate on the workflow. Do not respond to environment faults by building an ever-more-hermetic simulation of production inside your test harness; that trades observed behavior for reproducibility of a world that is not yours.

The result includes:

Field	Meaning
`passes`	One detailed result per input.
`executed`	Task IDs in execution order across all passes.
`taskOutputs`	Validated outputs grouped by task ID.
`outputs`	Validated outputs grouped by output-table name.
`finalOutputs`	Final output from each pass.
`definedNodes`, `coveredNodes`	Rendered and executed node IDs.
`unexecuted`, `unreached`	Rendered nodes that did not execute and expected nodes that never rendered.
`validations`, `approvals`, `errors`	Structured-output checks, gate decisions, and failures.

Use mocks for exact task IDs, task labels, or globs. Values accept the same raw output, callback, auto, or fakeAgent forms as simulate. Approvals default to approve and can be configured globally or per gate. signals and events provide payloads by node ID, label, event name, or "*".

const result = await coverWorkflow(WorkflowModule, {
  inputs: [{ mode: "fast" }, { mode: "safe" }],
  mocks: { classify: fakeAgent(classificationSchema, { route: "safe" }) },
  approvals: { publish: "deny" },
  signals: { ready: { accepted: true } },
  maxLoopIterations: 2,
  expectedNodes: ["classify", "publish"],
  allowUnreached: ["error-*"],
  assert: false,
});

Conditional branches that never render cannot be inferred from React at runtime. List them in expectedNodes when they must be covered across inputs; list intentional error-only or mutually exclusive paths in allowUnreached. Allowlist entries accept * globs. The harness replaces compute functions and known side-effect descriptors by default. Set executeCompute: true to run compute functions, or executeSideEffects: true to run marked side effects, sandboxes, and subflows. These opt-ins execute user code, so use them only when the test environment is already isolated. Use coverWorkflow for broad path and schema conformance. Use simulate directly when the test needs exact scheduling assertions, prompt inspection, compute behavior, or deliberately partial execution.

Unit-test a workflow

simulate(workflow, options?) renders and runs a workflow in memory: pass input via input, task responses via mocks. Mock keys are exact task IDs, task labels, or globs ("*" is the agent-task fallback); values can be a function, raw output, auto, or a FakeAgent. Unmocked agent tasks fail loudly, so a test can’t accidentally call a real agent. The returned Sim is lazy: await sim.run() executes it and populates the fields below.

/** @jsxImportSource smthrs */
import { describe, expect, test } from "bun:test";
import { z } from "zod";
import { createSmithers } from "smthrs";
import { auto, fakeAgent, simulate } from "smthrs/testing";

const schemas = {
  input: z.object({ name: z.string() }),
  greeting: z.object({ message: z.string() }),
  reply: z.object({ text: z.string() }),
};

function buildWorkflow() {
  const { Workflow, Task, smithers, outputs } = createSmithers(schemas);
  const workflowAgent = fakeAgent(schemas.greeting, {
    output: { message: "workflow fallback" },
  });

  return smithers((ctx) => (
    <Workflow name="greeting">
      <Task
        id="greet"
        output={outputs.greeting}
        agent={workflowAgent}
        noRetry
      >
        {`Greet ${ctx.input.name}`}
      </Task>
      <Task id="reply" output={outputs.reply} dependsOn={["greet"]}>
        {() => ({ text: `done ${ctx.input.name}` })}
      </Task>
    </Workflow>
  ));
}

describe("greeting workflow", () => {
  test("runs with a scripted agent", async () => {
    const scripted = fakeAgent.sequence(
      schemas.greeting,
      [{ message: "hello Ada" }],
    );

    const sim = simulate(buildWorkflow(), {
      input: { name: "Ada" },
      mocks: { greet: scripted },
    });

    await sim.run();

    expect(sim.status).toBe("finished");
    expect(sim.executed).toEqual(["greet", "reply"]);
    expect(sim.outputs.greeting).toEqual([{ message: "hello Ada" }]);
    expect(sim.outputs.reply).toEqual([{ text: "done Ada" }]);
    expect(scripted.calls).toHaveLength(1);
  });

  test("can generate a schema-shaped response automatically", async () => {
    const agent = fakeAgent(schemas.greeting, auto);

    await expect(agent.generate()).resolves.toEqual({
      output: { message: "string" },
    });
  });
});

The main Sim fields are:

Field	Meaning
`executed`	Task node IDs, in execution order.
`outputs`	Validated task outputs, as arrays keyed by output-table name (e.g. `greeting`, `reply`).
`status`	Run status: `pending`, `running`, `finished`, or `failed`.

sim.output is the latest workflow output; sim.task(id) returns a task’s status, outputs, and rendered prompts; unusedMocks lists unconsumed mock keys. A failed simulation exposes error and rejects sim.run(); output schemas are checked before values are recorded.

Scripted agents

fakeAgent(schema, script, options?) creates a FakeAgent; script is a bare schema value, { output, text, files }, auto, or a function returning one of those, and output values are validated against the schema. fakeAgent.sequence(schema, entries, options?) consumes a fixed list in order, for retries or repeated task runs. FakeAgent<T> describes the returned agent: generate() is Smithers-agent-compatible, while calls, lastPrompt(), and reset() inspect or clear its history. Optional settings: id, model, supportsNativeStructuredOutput. auto is a schema-derived sentinel that supplies an example-shaped response (requires an output schema); use an explicit value or function when the exact result matters. File responses are written below the rootDir passed to generate() or the test helper.

Render and run one task

renderWorkflow renders a frame or task descriptors without executing them: it resolves task output schemas, attaches the same subflow and sandbox compute functions as the engine, and returns a RenderedWorkflow with rendered tasks, ctx, runId, frameNo, and toXml().

import { renderPrompt, renderWorkflow, runTask } from "smthrs/testing";

const frame = await renderWorkflow(buildWorkflow(), {
  input: { name: "Ada" },
  runId: "render-test",
});
const task = frame.tasks.find(({ nodeId }) => nodeId === "greet");
if (!task) throw new Error("greet task was not rendered");

const prompt = renderPrompt(task.prompt);
const output = await runTask(task, { runId: frame.runId });

console.log({ prompt, output, xml: frame.toXml() });

renderPrompt(prompt) converts a rendered task prompt to the text form the production renderer uses. runTask(task, options?) executes one TaskDescriptor: static descriptors return their payload, compute descriptors call their compute function, and agent descriptors call their selected agent (RunTaskOptions forwards rootDir, attempt, and runId to the call); the result is validated when the descriptor has an output schema. dryRun(ast, options?) is the planning companion for the durability scenario API below: it canonicalizes and compiles the scenario, reports required capabilities and admissions, creates a replay-bundle skeleton, lists planned steps, sets executesAgents to false, and returns a run() function that executes the scenario.

Assertions with Bun matchers

The testing package augments bun:test’s matcher types; register the set once per file, then assert against a Sim:

import { expect, test } from "bun:test";
import { simMatchers, simulate } from "smthrs/testing";

expect.extend(simMatchers);

test("checks the simulated path", async () => {
  const sim = simulate(buildWorkflow(), {
    input: { name: "Ada" },
    mocks: { greet: { message: "hello Ada" } },
  });

  await sim.run();

  expect(sim).toHaveExecuted(["greet", "reply"]);
  expect(sim).toHaveExecutedInOrder(["greet", "reply"]);
  expect(sim).toHaveFinished();
});

toHaveExecuted(ids) requires every listed ID to appear; toHaveExecutedInOrder(ids) requires them as an ordered subsequence (unrelated tasks may fall between); toHaveFinished() checks for the exact "finished" status. The same functions are also exported directly as toHaveExecuted, toHaveExecutedInOrder, and toHaveFinished.

Advanced: durability scenario harness

For modeled scheduling and durability behavior, compose an immutable scenario AST from scenario, step, and fault (barrier and extension cover explicit synchronization and registered extensions). A step declares input, dependsOn, capabilities, and a run function, whose task runtime exposes mediated effect, virtual sleep, structured log, and explicitly opaque work; a fault names a phase and operation at which to inject a failure or ambiguity. runScenario(ast, options?) executes the AST and returns status, outputs, trace, replay identity, control log, capability report, ambiguity records, and determinism report. unitSimHarness, integrationHarness, and e2eHarness select the capability tier (deterministic virtual-time/seeded-interleaving simulation, a real database adapter, or a real process adapter); runScenario defaults to unitSimHarness(), and the real tiers, requiring verified executable production adapters, can report a capability failure or skip per policy.

Unit simulation doesn’t substitute for a database or child process: external effects remain at-least-once unless the application supplies idempotency, and effects outside the task-runtime mediation boundary are opaque. See packages/testing/README.md for the real-system boundary, replay, fault, and adapter details.

For LLM-judge assertions, see llmJudge and runScorersBatch in the Scorers reference; for dataset-driven regression suites, see the Eval Suites Quickstart.

Start

Articles

Learn

Capabilities

Build Workflows

Run and Operate

Workflow Pack

Components

Integrations

Agent Support

Examples

Contributing

Testing workflows

Cover a workflow in one call

Unit-test a workflow

Scripted agents

Render and run one task

Assertions with Bun matchers

Advanced: durability scenario harness

​Cover a workflow in one call

​Unit-test a workflow

​Scripted agents

​Render and run one task

​Assertions with Bun matchers

​Advanced: durability scenario harness

​Related evaluation tools

Cover a workflow in one call

Unit-test a workflow

Scripted agents

Render and run one task

Assertions with Bun matchers

Advanced: durability scenario harness

Related evaluation tools