> ## Documentation Index
> Fetch the complete documentation index at: https://smithers.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Parallelism and Isolation

> Fan work out across many agents in parallel worktree lanes, cap concurrency, and fold results back in without lanes stomping on each other.

Run two coding agents in the same checkout and they will corrupt each other. One runs `git checkout -b feature-a` while the other is mid-edit; one runs `pnpm install` while the other reads `node_modules`; both write to the same file and the last save wins. The failure isn't hypothetical — it's the default outcome of naive fan-out, and it's why most agent tools serialize everything and call it a day.

Serializing is the wrong fix. Agent work is embarrassingly parallel in shape — N tickets, N reviewers, N research angles — and agents are slow enough (minutes per task) that parallelism is the difference between an hour and a night. What you actually need is three separate mechanisms: **concurrency control** (how many lanes run at once), **filesystem isolation** (each lane gets its own working copy), and **fan-in** (a fold step that can see every lane's result). Smithers ships each as a first-class primitive: `<Parallel>`, `<Worktree>`/`<Sandbox>`, and `deps`-driven fold tasks.

Be honest about the costs before reaching for this. Anthropic's own [multi-agent research system write-up](https://www.anthropic.com/engineering/multi-agent-research-system) measured that multi-agent runs burn roughly **15x the tokens** of a single chat — parallelism multiplies spend linearly with lane count, plus the orchestrator's overhead. And the hard part is rarely the fan-out; it's the merge. Ten agents that each finish a feature in their own lane produce ten branches that touch overlapping files, and reconciling them is where runs actually die.

## How the industry solves it

Fan-out/fan-in is the oldest pattern in distributed computing — MapReduce is literally named after it — and every orchestration system has a version.

| System                                                                                                                 | Fan-out mechanism                                                        | Isolation                                                                   | Tradeoff                                                                                 |
| ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| [AWS Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html) | `Map` state; distributed map runs up to 10,000 parallel child executions | Separate Lambda/ECS invocations                                             | JSON state machines, no agents, per-transition billing                                   |
| [Argo Workflows](https://argo-workflows.readthedocs.io/)                                                               | DAG templates + `withItems`/`withParam` loops                            | One Kubernetes pod per step                                                 | Requires a cluster; container startup per step                                           |
| [Temporal](https://docs.temporal.io/child-workflows)                                                                   | Child workflows and parallel activities                                  | Process-level; no filesystem story                                          | Durable and battle-tested, but you write the isolation layer yourself                    |
| [Claude Code Dynamic Workflows](https://code.claude.com/docs/en/workflows)                                             | Model-generated script fans out to as many as 1,000 subagents            | Anthropic-managed, ephemeral                                                | Closed, Claude-only, cloud-gated — see [our comparison](/why/durable-open-orchestration) |
| [Conductor](https://conductor.build), [Crystal](https://github.com/stravu/crystal)                                     | Multiple Claude Code / Codex sessions side by side                       | **Git worktrees**, one per session                                          | Desktop apps; human merges by hand                                                       |
| [Cursor background agents](https://cursor.com/docs/background-agent), [Devin](https://devin.ai)                        | Background agent per task                                                | Remote VM / workspace clone per agent                                       | Vendor cloud; you don't own the machine                                                  |
| [E2B](https://e2b.dev), [Modal Sandboxes](https://modal.com/docs/guide/sandbox), [Daytona](https://daytona.io)         | Sandbox-per-agent APIs                                                   | [Firecracker](https://firecracker-microvm.github.io/) microVMs / containers | Strongest isolation, network hop and image management                                    |

Two industry conclusions worth stealing. First, **git worktrees have become the standard local isolation primitive for coding agents** — Conductor and Crystal are built entirely around them, because a worktree shares the object store (cheap, instant) while giving each agent a private index, HEAD, and working copy. Second, for untrusted or destructive work, filesystem isolation isn't enough and the industry reaches for microVMs (Firecracker underpins both AWS Lambda and E2B).

Smithers does both: worktrees for parallel lanes on your machine, provider-backed sandboxes when you need a real boundary.

## How Smithers does it

### `<Parallel>`: concurrency without chaos

`<Parallel>` runs children concurrently with two independent caps ([full reference](/components/parallel)):

* `maxConcurrency` caps **leaf tasks** in the group's innermost scope.
* `subtreeConcurrency` caps **direct children as whole subtrees** — the right cap when each child is a sequence (implement → review → land), because a nested `<Parallel>` escapes the outer leaf cap.

On top of every group cap sits a **run-wide cap**: unpinned it starts at 4 and grows with demand up to `SMITHERS_AUTO_MAX_CONCURRENCY_CEILING` (default 16); `--max-concurrency N` pins it. A task runs only if the leaf cap, subtree cap, and run cap all pass. Admission is deterministic in document order, so a resumed run re-admits the same lanes.

### `<Worktree>`: one working copy per lane

`<Worktree path="...">` roots its subtree in an isolated git or jj worktree; descendants inherit the absolute `worktreePath` as their `cwd` ([full reference](/components/worktree)). Worktree creation is serialized per VCS root because concurrent `git worktree add` / `jj workspace add` calls mutate the shared `.git` state (see `packages/engine/src/engine.js`).

```tsx theme={"theme":{"light":"github-light","dark":"github-dark"}}
/** @jsxImportSource smithers-orchestrator */
import { createSmithers, Parallel, Worktree, MergeQueue, ClaudeCodeAgent } from "smithers-orchestrator";
import { z } from "zod";

const { Workflow, Task, smithers, outputs } = createSmithers({
  impl: z.object({ ticket: z.string(), summary: z.string() }),
  land: z.object({ merged: z.boolean() }),
  fold: z.object({ report: z.string() }),
});

const agent = new ClaudeCodeAgent({ model: "claude-sonnet-5" }); // no cwd — Worktree supplies it
const tickets = ["auth", "billing", "search"];

export default smithers(() => (
  <Workflow name="fanout">
    <Parallel id="lanes" subtreeConcurrency={2}>
      {tickets.map((t) => (
        <Worktree key={t} id={`wt-${t}`} path={`/tmp/smithers/${t}`} baseBranch="main">
          <Task id={`impl-${t}`} agent={agent} output={outputs.impl}>
            Implement the {t} ticket. Run the tests inside this worktree.
          </Task>
          <MergeQueue>
            <Task id={`land-${t}`} agent={agent} output={outputs.land}>
              Merge the committed bookmark back into main. Resolve conflicts against
              whatever landed before you.
            </Task>
          </MergeQueue>
        </Worktree>
      ))}
    </Parallel>
    <Task
      id="fold"
      agent={agent}
      output={outputs.fold}
      needs={{ auth: "impl-auth", billing: "impl-billing", search: "impl-search" }}
      deps={{ auth: outputs.impl, billing: outputs.impl, search: outputs.impl }}
    >
      {(deps) =>
        `Write a combined report:\n` +
        [deps.auth, deps.billing, deps.search].map((d) => `- ${d.ticket}: ${d.summary}`).join("\n")
      }
    </Task>
  </Workflow>
));
```

Three verified traps, in order of how often they bite:

<Warning>
  **`needs` gates; it injects nothing.** A fold task with only `needs` waits for its lanes but its prompt receives none of their output — the agent then confidently synthesizes from nothing. Pair `needs` (id mapping) with `deps` (typed row resolution into the `children` callback), as above. This is the single most common fan-in bug in real Smithers workflows.
</Warning>

<Warning>
  **Files a lane produced live in the lane's worktree, not your checkout.** Typed output rows flow through `deps`/`ctx.outputs` regardless of where the task ran, but a *file* written by a lane exists only under that lane's worktree until the merge step lands it. A fold task that reads artifacts off disk must resolve the real path — `ctx.worktreePath("wt-auth")` or `ctx.resolveWorktreePath("/tmp/smithers/auth")` — never reconstruct it from `process.cwd()`. Better: keep structured results in typed output rows and let files travel through the merge.
</Warning>

<Warning>
  **Never pin `cwd` on an agent used inside a `<Worktree>`.** Resolution is `agent.cwd ?? worktreePath ?? repoRoot`, so an explicit `cwd` silently breaks isolation: the agent edits the repo root, the lane branch stays empty, and the land step merges nothing.
</Warning>

### Merging: the real bottleneck

Smithers does not auto-resolve conflicts — nobody credibly does. What it gives you is structure: `<MergeQueue>` serializes landing (default `maxConcurrency: 1`, priority `1000` so ready landing work jumps the queue), which converts an N-way conflict free-for-all into N sequential rebase-and-resolve steps, each performed by an agent that can see what landed before it. Cheap discipline that removes most conflicts: partition lanes by file ownership up front, and land early and often rather than letting ten lanes drift for hours. Note that jj-backed worktrees continuously snapshot the working copy onto the lane's bookmark, so merge steps should diff against the base branch (`git diff main...HEAD`), not trust `git status` dirty-state.

### `<Sandbox>`: when a worktree isn't enough

Worktrees isolate the *working copy*, not the machine — a lane can still `rm -rf ~`, read your env, or hit the network. For untrusted or destructive work, `<Sandbox>` runs a child workflow through a provider (`bubblewrap`, `docker`, or any object registered via `registerSandboxProvider`, including microVM backends like the `@smithers-orchestrator/daytona` and `microsandbox` packages), with `allowNetwork`, `egress` proxy/CA controls, resource limits, and diff review (`reviewDiffs` defaults to true) before changes touch your tree. See [`<Sandbox>`](/components/sandbox) and [sandbox providers](/components/sandbox-providers).

## Guarantees and limits

* **Guaranteed:** deterministic lane admission on resume; serialized worktree creation; worktree `cwd` inheritance; caps enforced at leaf, subtree, and run level; `DEPENDENCY_DEADLOCK` (not a hang) when a `deps` key names a task that never produces.
* **Not guaranteed:** conflict resolution (an agent or a human does it); dependency installation in fresh worktrees (lanes must `pnpm install` themselves — the isolation preamble tells agents this); cost control (N lanes ≈ N× tokens; cap concurrency and lane count deliberately).
* **Cleanup is yours to run:** finished runs leave worktrees behind until pruned.

## Operating it

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
smithers up fanout -d --max-concurrency 6   # pin the run-wide cap
smithers status <run>                        # which lanes are in flight / gating progress
smithers worktree list                       # every worktree Smithers created, and its owning run
smithers worktree prune --older-than 24h     # reclaim worktrees of finished runs
smithers worktree prune --force              # also drop uncommitted/unpushed lane work
smithers restore --node <id>                 # restore a lane's worktree to a durability checkpoint
```

## See also

* [`<Parallel>`](/components/parallel) · [`<Worktree>`](/components/worktree) · [`<MergeQueue>`](/components/merge-queue) · [`<Sandbox>`](/components/sandbox)
* [`<GatherAndSynthesize>`](/components/gather-and-synthesize) — packaged fan-out/fold for research-shaped work
* [Common footguns](/guides/common-footguns) — the output-pool and `outputMaybe` rules fold tasks depend on
* [Why: durable open orchestration](/why/durable-open-orchestration) — the Dynamic Workflows comparison in full