Testing workflows

@smithers-orchestrator/testing is the consumer-facing test surface for Smithers workflows. Import it from the published facade:

import {
  auto,
  dryRun,
  fakeAgent,
  renderPrompt,
  renderWorkflow,
  runTask,
  simMatchers,
  simulate,
  toHaveExecuted,
  toHaveExecutedInOrder,
  toHaveFinished,
} from "smithers-orchestrator/testing";

The examples below use the normal createSmithers workflow API and bun:test; the testing helpers do not replace workflow authoring components.

Unit-test a workflow

simulate(workflow, options?) renders and runs a workflow in memory. Pass the workflow’s input through input, and provide task responses in mocks. Mock keys can be exact task IDs or globs; "*" is the fallback key for agent tasks. Each value can be a function, a raw output value, auto, or a FakeAgent. Agent tasks without a matching mock fail loudly, which keeps a test from accidentally calling a real agent. The returned Sim is lazy: call await sim.run() to execute it. It records the task IDs it ran, the values stored in each output table, and the final run status.

/** @jsxImportSource smithers-orchestrator */
import { describe, expect, test } from "bun:test";
import { z } from "zod";
import { createSmithers } from "smithers-orchestrator";
import { auto, fakeAgent, simulate } from "smithers-orchestrator/testing";

const schemas = {
  input: z.object({ name: z.string() }),
  greeting: z.object({ message: z.string() }),
  reply: z.object({ text: z.string() }),
};

function buildWorkflow() {
  const { Workflow, Task, smithers, outputs } = createSmithers(schemas);
  const workflowAgent = fakeAgent(schemas.greeting, {
    output: { message: "workflow fallback" },
  });

  return smithers((ctx) => (
    <Workflow name="greeting">
      <Task
        id="greet"
        output={outputs.greeting}
        agent={workflowAgent}
        noRetry
      >
        {`Greet ${ctx.input.name}`}
      </Task>
      <Task id="reply" output={outputs.reply} dependsOn={["greet"]}>
        {() => ({ text: `done ${ctx.input.name}` })}
      </Task>
    </Workflow>
  ));
}

describe("greeting workflow", () => {
  test("runs with a scripted agent", async () => {
    const scripted = fakeAgent.sequence(
      schemas.greeting,
      [{ message: "hello Ada" }],
    );

    const sim = simulate(buildWorkflow(), {
      input: { name: "Ada" },
      mocks: { greet: scripted },
    });

    await sim.run();

    expect(sim.status).toBe("finished");
    expect(sim.executed).toEqual(["greet", "reply"]);
    expect(sim.outputs.greeting).toEqual([{ message: "hello Ada" }]);
    expect(sim.outputs.reply).toEqual([{ text: "done Ada" }]);
    expect(scripted.calls).toHaveLength(1);
  });

  test("can generate a schema-shaped response automatically", async () => {
    const agent = fakeAgent(schemas.greeting, auto);

    await expect(agent.generate()).resolves.toEqual({
      output: { message: "string" },
    });
  });
});

The main Sim fields are:

Field	Meaning
`executed`	Task node IDs in the order they executed.
`outputs`	Arrays of validated task outputs keyed by output-table name, such as `greeting` or `reply`.
`status`	The current run status, including `pending`, `running`, `finished`, or `failed`.

sim.output is the latest workflow output. sim.task(id) returns that task’s status, outputs, and rendered prompts; unusedMocks identifies mock keys that were never consumed. A failed simulation exposes error and sim.run() rejects. Output schemas are checked before values are recorded.

Scripted agents

fakeAgent(schema, script, options?) creates a FakeAgent whose script can be a bare schema value, a { output, text, files } response, auto, or a function returning one of those forms. Responses with an output are validated against the supplied schema. fakeAgent.sequence(schema, entries, options?) consumes a fixed list in order, which is useful for retries or repeated task runs. The FakeAgent<T> type describes the returned agent. Its generate() method is compatible with Smithers agents, while calls, lastPrompt(), and reset() let a test inspect or clear its history. The optional agent settings are id, model, and supportsNativeStructuredOutput. auto is a schema-derived response sentinel. It requires an output schema and supplies an example-shaped output; use an explicit value or function when the exact result matters. File responses are written below the rootDir supplied to generate() or the test helper.

Render and run one task

Use renderWorkflow when a test needs a rendered frame or task descriptors without executing them. It resolves task output schemas and attaches the same subflow and sandbox compute functions as the engine. The returned RenderedWorkflow includes the rendered tasks, ctx, runId, frameNo, and toXml().

import { renderPrompt, renderWorkflow, runTask } from "smithers-orchestrator/testing";

const frame = await renderWorkflow(buildWorkflow(), {
  input: { name: "Ada" },
  runId: "render-test",
});
const task = frame.tasks.find(({ nodeId }) => nodeId === "greet");
if (!task) throw new Error("greet task was not rendered");

const prompt = renderPrompt(task.prompt);
const output = await runTask(task, { runId: frame.runId });

console.log({ prompt, output, xml: frame.toXml() });

renderPrompt(prompt) converts a rendered task prompt to the same text form used by the production task renderer. runTask(task, options?) executes only one TaskDescriptor: static descriptors return their payload, compute descriptors call their compute function, and agent descriptors call their selected agent. When the descriptor has an output schema, the result is validated. For agent descriptors, RunTaskOptions forwards rootDir, attempt, and runId to the agent call. dryRun(ast, options?) is the planning companion for the durability scenario API below. It canonicalizes and compiles the scenario, reports required capabilities and admissions, creates a replay-bundle skeleton, lists planned steps, and sets executesAgents to false. Its returned run() function is the point at which the scenario is actually executed.

Assertions with Bun matchers

The testing package augments bun:test’s matcher types. Register the exported matcher set once in the test file, then assert against a Sim:

import { expect, test } from "bun:test";
import { simMatchers, simulate } from "smithers-orchestrator/testing";

expect.extend(simMatchers);

test("checks the simulated path", async () => {
  const sim = simulate(buildWorkflow(), {
    input: { name: "Ada" },
    mocks: { greet: { message: "hello Ada" } },
  });

  await sim.run();

  expect(sim).toHaveExecuted(["greet", "reply"]);
  expect(sim).toHaveExecutedInOrder(["greet", "reply"]);
  expect(sim).toHaveFinished();
});

toHaveExecuted(ids) requires every listed ID to appear. toHaveExecutedInOrder(ids) requires them as an ordered subsequence, so unrelated tasks may occur between them. toHaveFinished() checks for the exact "finished" status. The same functions are also exported as toHaveExecuted, toHaveExecutedInOrder, and toHaveFinished for direct matcher use.

Advanced: durability scenario harness

For modeled scheduling and durability behavior, compose an immutable scenario AST with scenario, step, and fault; barrier and extension are available for explicit synchronization and registered extensions. A step can declare input, dependsOn, capabilities, and a run function. Its task runtime exposes mediated effect, virtual sleep, structured log, and explicitly opaque work. A fault names a phase and operation at which to inject a failure or ambiguity. runScenario(ast, options?) executes the AST and returns a result containing its status, outputs, trace, replay identity, control log, capability report, ambiguity records, and determinism report. unitSimHarness, integrationHarness, and e2eHarness select the capability tier: deterministic virtual-time and seeded-interleaving simulation; a real database adapter; or a real process adapter. runScenario defaults to unitSimHarness(); the real tiers require verified, executable production adapters and can report a capability failure or skip according to policy.

Unit simulation is not a database or child-process substitute. External effects remain at-least-once unless the application supplies idempotency, and effects outside the task-runtime mediation boundary are opaque. See packages/testing/README.md for the real-system boundary, replay, fault, and adapter details.

For LLM-judge assertions, see llmJudge and runScorersBatch in the Scorers reference. For dataset-driven regression suites, see the Eval Suites Quickstart.

Start

Articles

Learn

Build Workflows

Run and Operate

Workflow Pack

Components

Integrations

Agent Support

Examples

Contributing

Testing workflows

Unit-test a workflow

Scripted agents

Render and run one task

Assertions with Bun matchers

Advanced: durability scenario harness

​Unit-test a workflow

​Scripted agents

​Render and run one task

​Assertions with Bun matchers

​Advanced: durability scenario harness

​Related evaluation tools

Unit-test a workflow

Scripted agents

Render and run one task

Assertions with Bun matchers

Advanced: durability scenario harness

Related evaluation tools