Skip to main content
Every database you have ever cursed at got that way for the same reason: someone mixed bookkeeping with business data. Audit timestamps crept into domain objects. Retry counters leaked into API responses. One day you open the schema and cannot tell what the system does from what the system needs to run. Smithers refuses to let that happen. It stores three kinds of data, and it keeps them apart on purpose:
  1. the run input payload
  2. task output rows
  3. internal workflow metadata
Why does the separation matter? Because the question “what did the task produce?” and the question “how many attempts did it take?” have different audiences, different lifecycles, and no business sharing a table. You will thank this design the first time you query your outputs without wading through orchestration columns.

​
Run Input

When you kick off a workflow, you hand it a payload. That payload is the entire context your workflow gets from the outside world: 
const result = await runWorkflow(workflow, {
  input: { description: "Auth tokens expire silently" },
});
Think of it as the function argument for the whole run. Smithers persists it once, and every task in the workflow reads it through ctx.input. If the run crashes and resumes, the same input is still there — unless you explicitly override it. So what should go in input? Three things:
  • user-supplied run context
  • durable across resume
  • available everywhere through ctx.input
Nothing more. If a value is produced during the run, it belongs in a task output, not in the input.

​
Task Outputs

Here is where your domain data lives. Most Smithers workflows define output schemas up front with createSmithers(...): 
const { Workflow, Task, smithers, outputs } = createSmithers({
  analysis: z.object({
    summary: z.string(),
    severity: z.enum(["low", "medium", "high"]),
  }),
});
Notice that the schema describes your data — a summary string and a severity level. That is it. No run IDs, no iteration counters, no attempt numbers. You define the shape of the answer; Smithers handles everything else. Behind the scenes, each schema key becomes a durable SQLite table. Smithers automatically:
  • creates the SQLite table
  • maps the schema key to a snake_case table name
  • adds runId, nodeId, and iteration bookkeeping columns
  • validates agent output before persisting it
Your prompt-facing schema stays clean: 
z.object({
  summary: z.string(),
  severity: z.enum(["low", "medium", "high"]),
})
“Wait,” you might be thinking, “if Smithers adds bookkeeping columns anyway, why can’t I just add them myself?” You can. But then your LLM prompt includes fields it should never fill, your validation conflates domain rules with runtime plumbing, and your query results mix what the task said with how it got there. You do not need to add fields like:
  • runId
  • nodeId
  • iteration
  • attempt
  • approval metadata
Smithers owns those. Let it.

​
Identity of an Output Row

Here is a subtlety that trips people up. Two different tasks can write to the same output schema. The same task can write to it ten times inside a loop. So how does Smithers know which row is which? The answer: output identity is not “table name only.” Each row is keyed by:
  • run id
  • task id (nodeId)
  • iteration when the task is inside a loop
That is why ctx.output(...), ctx.outputMaybe(...), and ctx.latest(...) all require both an output target and a nodeId. The table tells Smithers where to look. The node ID and iteration tell it which row you mean.

​
Custom Drizzle Tables

Sometimes you already have a table, or you need a schema that Smithers cannot auto-generate. In that case, <Task output={...}> can point at a custom Drizzle table. Fair warning: when you go this route, you take on responsibility that Smithers normally handles for you:
  • creating and migrating the table
  • including Smithers bookkeeping columns such as runId and nodeId
  • including iteration in looped tasks
  • optionally pairing the table with outputSchema for stricter validation
This is an escape hatch, not the default path. If createSmithers(...) can express your schema, use it.

​
Internal Smithers Metadata

Open your database and you will see tables prefixed with _smithers_. Do not be alarmed. These are Smithers’ own operational tables:
TablePurpose
_smithers_runsOne row per workflow run. Tracks status, heartbeat, VCS revision, and error.
_smithers_nodesCurrent state of each task node within a run (pending, running, finished, failed).
_smithers_attemptsEvery execution attempt for every node, including start/finish timestamps and error detail.
_smithers_framesThe rendered JSX tree at each commit boundary, stored as serialized XML.
_smithers_approvalsApproval requests and decisions for tasks gated by <Approval>.
_smithers_human_requestsHuman-in-the-loop requests (form fills, confirmations) and their responses.
_smithers_cacheCached task outputs keyed by workflow, node, schema signature, and agent signature.
_smithers_sandboxesSandbox session metadata for bubblewrap and container-based execution.
_smithers_tool_callsPer-call log of every tool invocation: input, output, latency, and status.
_smithers_eventsSequential event journal for a run. Source of truth for all observable events.
_smithers_ralphLoop (<Loop>) iteration counters and completion flags.
_smithers_cronCron schedule definitions, last-run and next-run timestamps.
_smithers_scorersScorer results for each task attempt: score, reason, and latency.
_smithers_vectorsRAG vector store: chunk text, embeddings (as BLOBs), and metadata.
_smithers_signalsInbound signals received by waiting runs.
This is the machinery that lets Smithers resume a crashed run, retry a failed task, or tell you exactly what happened at 3 a.m. It exists so your output tables never have to carry orchestration concerns.

​
Table Schema Ensurance and Auto-Migration

Smithers calls ensureSmithersTables() at startup, which runs CREATE TABLE IF NOT EXISTS for every internal table. You never need to run migrations by hand for _smithers_* tables. For your own output tables defined via createSmithers(...), Smithers also auto-migrates columns. When the Drizzle schema defines a column that is missing from the SQLite table on disk, Smithers issues an ALTER TABLE ... ADD COLUMN statement to add it. Columns that exist in the database but are absent from the schema are left in place — Smithers does not remove data. This forward-only migration means you can add fields to an output schema and existing runs will continue to work. Removing a field or changing a column type requires a manual migration or a fresh database.

​
Schema Signature Verification

Before persisting a cached task result, Smithers computes a schema signature for the output table. The signature is a SHA-256 hash of the table name and every column’s name, type, nullability, and primary key flag, all sorted alphabetically: 
sha256("tableName|colA:text:1:0|colB:integer:0:0|...")
This hash is stored as schema_sig in _smithers_cache. When a cached result is retrieved, Smithers checks that the current table’s signature still matches. If the schema changed since caching, the cached entry is ignored and the task runs fresh. You never get silently stale cache hits after a schema migration.

​
Transaction Model

SmithersDb uses a single-writer transaction model with a serial promise queue. Every write operation (including those outside an explicit transaction) acquires a turn in a transactionTail promise chain before proceeding. This serializes all writes even when multiple Effect fibers run concurrently. Explicit transactions use BEGIN IMMEDIATE so SQLite acquires a write lock immediately, preventing lock contention with concurrent readers: 
await adapter.withTransaction("my-write-group", effect);
Nested transactions from the same fiber are detected and rejected — SQLite does not support true savepoints through this interface. The transaction depth counter and owner-thread tracking ensure the same fiber can perform multiple writes within a single transaction without re-acquiring the queue turn.

​
Write Retry and Exponential Backoff

All write paths wrap the underlying operation with withSqliteWriteRetryEffect. When a write fails with SQLITE_BUSY, SQLITE_IOERR, “database is locked”, or “disk i/o error”, Smithers retries up to six times with exponential backoff:
  • Base delay: 50 ms
  • Maximum delay: 2,000 ms
  • Jitter: ±25% of the computed delay
  • Each retry increments the smithers.db.retries counter
After the maximum number of retries, the error propagates as a DB_WRITE_FAILED SmithersError. This makes Smithers resilient to transient WAL-mode lock contention without requiring any configuration.

​
Frame Codec

Render frames in _smithers_frames are stored in one of three encodings:
EncodingWhen usedDescription
fullFrame 0 and any keyframeComplete serialized XML of the render tree
deltaFrames between keyframesJSON patch (set, insert, remove ops) relative to the previous frame
keyframeEvery 50th frameSame as full; resets the delta chain
The keyframe interval is 50 frames (FRAME_KEYFRAME_INTERVAL = 50). Reading an arbitrary frame requires loading the nearest preceding keyframe and applying all deltas up to the target frame number. An in-memory LRU cache (up to 512 entries) stores reconstructed frame XML so repeated reads of hot frames are free. Delta encoding uses a structural diff algorithm that walks the XML JSON tree, emitting set, insert, and remove operations. It is node-ID-aware: when comparing adjacent objects in the tree, it uses the id prop of element nodes as a stable identity anchor, so reordered elements produce insert/remove pairs rather than spurious updates.

​
Signal Persistence

Signals are external messages sent to a running workflow. When a signal arrives, Smithers writes it to _smithers_signals with an automatically allocated sequence number. You never pick the seq yourself — Smithers computes MAX(seq) + 1 inside a BEGIN IMMEDIATE transaction so two concurrent signals never collide. Before inserting, the adapter checks whether an identical signal already exists (same runId, signalName, correlationId, payloadJson, receivedAtMs, and receivedBy). If a match is found, the existing seq is returned and no duplicate row is created. This deduplication prevents replay or retry from doubling signals.

​
Signal Query Filters

Querying signals supports four filters, all optional:
FilterColumnDescription
signalNamesignal_nameMatch a specific signal type
correlationIdcorrelation_idMatch a specific correlation key (supports null)
receivedAfterMsreceived_at_msOnly signals received at or after this timestamp
limit—Max rows to return (default 200)
Results are ordered by seq ASC, so you always see signals in arrival order.

​
Event Persistence

The _smithers_events table is the durable event journal for each run. Every SmithersEvent emitted during execution is persisted here with a sequential seq number that serves as the total ordering.

​
Auto-Sequence Allocation

Like signals, events get their seq via SELECT COALESCE(MAX(seq), -1) + 1 inside a BEGIN IMMEDIATE transaction. This guarantees gap-free, monotonically increasing sequence numbers per run.

​
Insert Deduplication

Before inserting, the adapter checks for an existing row matching the same runId, timestampMs, type, and payloadJson. If found, the existing seq is returned without creating a duplicate. This makes event insertion idempotent across retries.

​
Event Queue and Flush

For performance, events can be enqueued asynchronously via emitEventQueued. The event is emitted to listeners and tracked immediately, but database and log-file persistence happens in a background promise chain (persistTail). Call flush() to await all queued persistence — the engine does this at task boundaries and run completion to ensure nothing is lost.

​
Sequence Start Override

The EventBus constructor accepts a startSeq option, which sets the initial sequence counter. This is used on resume to continue from where the previous run left off, preventing sequence number collisions with already-persisted events.

​
Event History Queries

The adapter supports filtered history queries with these parameters:
FilterDescription
afterSeqReturn events with seq > afterSeq
limitMax rows
nodeIdFilter by $.nodeId inside the payload JSON
typesFilter to specific event type strings
sinceTimestampMsEvents at or after this timestamp
A separate countEventHistory method returns the count matching the same filters, useful for pagination.

​
Human Request Persistence

Human requests (form fills, confirmations, free-text prompts) are stored in _smithers_human_requests with lifecycle states: pending, answered, cancelled, expired.

​
Pending Inbox Query

listPendingHumanRequests returns all pending requests across all runs, joined with _smithers_runs and _smithers_nodes to include the workflowName, runStatus, and nodeLabel. Before returning, it automatically expires any requests whose timeoutAtMs has passed, transitioning them to expired status.

​
Answer Persistence

answerHumanRequest sets the response JSON, timestamp, and optional answeredBy field, transitioning the request from pending to answered. Only pending requests can be answered — the WHERE status = 'pending' clause prevents double-answering.

​
Cancellation

cancelHumanRequest transitions a pending request to cancelled. Like answering, it only operates on requests in pending status.

​
Cron Persistence

Cron schedules are stored in _smithers_cron and managed through the adapter:
OperationMethodDescription
Create/UpdateupsertCronInserts or updates a cron schedule by cronId
ListlistCrons(enabledOnly?)Returns all cron entries, optionally filtering to enabled = true
Track executionupdateCronRunTimeUpdates lastRunAtMs, nextRunAtMs, and optional errorJson
DeletedeleteCronRemoves a cron entry by ID
The enabled flag allows disabling a schedule without deleting it. The lastRunAtMs and nextRunAtMs columns let the scheduler know when to fire next without recomputing from the cron pattern on every poll. If a scheduled run fails, the error is stored in errorJson on the cron row for diagnostics.

​
Run Lifecycle Management

​
Stale Run Claims

The supervisor detects stale runs by querying _smithers_runs for rows with status = 'running' whose heartbeat_at_ms is older than the stale threshold (default 30 seconds). To safely resume a stale run without races, the supervisor uses a compare-and-swap pattern:
  1. Claim: claimRunForResume atomically sets runtime_owner_id and heartbeat_at_ms only if the current values match the expected stale state. The WHERE clause checks runtime_owner_id, heartbeat_at_ms, and the stale threshold in a single UPDATE, and returns whether the row was modified.
  2. Release: If the supervisor decides not to resume after claiming, releaseRunResumeClaim restores the original runtime_owner_id and heartbeat_at_ms, but only if the claim is still held (the current runtime_owner_id matches the claimer).
This two-phase claim prevents two supervisor instances from resuming the same stale run simultaneously.

​
Sandbox Tracking

Sandbox sessions (bubblewrap, Docker, or Codeplane) are tracked in _smithers_sandboxes. The adapter upserts sandbox rows keyed by (runId, sandboxId), recording runtime type, configuration, status, shipping and completion timestamps, and bundle paths.

​
Output Edge Cases

​
Payload-Only Tables

When an output table’s only non-bookkeeping column is payload, Smithers detects it and wraps the entire agent output into that single column instead of spreading fields across multiple columns. This is useful for unstructured or polymorphic outputs where a fixed column set does not make sense.

​
Boolean Column Coercion

Bun’s SQLite driver returns raw 0/1 integers for columns declared with { mode: "boolean" } in Drizzle. When loading output snapshots, Smithers detects these columns by inspecting the Drizzle table metadata and coerces the integer values to proper JavaScript booleans. Without this, strict equality checks like value === true would fail.

​
Schema Key Aliasing

When loading outputs via loadOutputs, each result set is stored under both the schema key (e.g., "analysis") and the actual SQLite table name (e.g., "analysis" or a custom name). This dual indexing lets downstream code reference outputs by either name, which matters when schema keys and table names diverge (e.g., with custom Drizzle tables).

​
Snapshot Persistence

Loading a complete workflow snapshot (loadInput + loadOutputs) reconstructs the full ctx state from SQLite. The input is loaded by filtering the input table for the current runId. Outputs are loaded by iterating every schema key, querying each table for rows matching the runId, applying boolean coercion, and indexing under both schema key and table name. This snapshot is what powers resume: when a crashed run restarts, the snapshot populates ctx so the JSX tree renders with all completed outputs already in place.

​
Transaction Internals

​
Read Gating

Reads, not just writes, also acquire a turn in the transactionTail promise queue. This prevents reads from seeing intermediate state during a multi-statement transaction. If the current fiber already owns the active transaction, reads proceed immediately without acquiring a new turn.

​
Commit Retry

The entire withTransaction call is wrapped in withSqliteWriteRetryEffect. If the COMMIT (or BEGIN IMMEDIATE) fails with SQLITE_BUSY or an I/O error, the retry mechanism rolls back and retries the full transaction from BEGIN, using the same exponential backoff as standalone writes.

​
Why the Separation Matters

Ask two questions about any completed task: Your workflow output answers: what did this task produce? Smithers metadata answers:
  • when did it run?
  • how many attempts did it take?
  • was it cached?
  • did it wait for approval?
  • which loop iteration produced it?
These are fundamentally different concerns. Mixing them is like storing a book’s page count in the same field as its ISBN — technically possible, obviously wrong. Keep them apart and both stay easy to reason about.

​
Schema Changes

Changing a Zod output schema is not just a prompt tweak. It is a persistence change. The table on disk has to match the schema in code. Typical examples:
  • adding a field
  • removing a field
  • changing a field type
  • tightening validation rules
In hot-reload mode, Smithers blocks these changes and requires a restart so output resolution stays deterministic. This is deliberate friction — it forces you to think about the migration before the data gets inconsistent. If you use custom Drizzle tables, you must manage those migrations yourself.

​
Direct Queries

Smithers does not hide SQLite from you. The database is right there. Open it, poke around, write queries. Use output tables when you care about business results. Use _smithers_* tables when you care about execution history. This is one of the advantages of keeping the layers separate: you can hand your output tables to an analyst who has never heard of Smithers, and the data makes sense on its own.

​
Mental Model

When in doubt, apply this rule of thumb:
  • ctx.input is run-scoped input
  • output tables hold validated task results
  • _smithers_* tables hold orchestration state
If a field only exists to help the runtime schedule or resume work, it belongs in Smithers metadata, not in your domain schema. If a field describes what the task actually produced, it belongs in an output table, not in _smithers_*. The line is clean. Keep it that way.

​
Next Steps

  • Execution Model — See how these tables participate in render, scheduling, and resume.
  • Structured Output — Validation and persistence details for task outputs.
  • Debugging — Query the internal tables directly when a run behaves unexpectedly.