Durability

Durable execution follows storage. A zero-config AgentHarness has no persistent storage, so every durability hook is a no-op. Give it a durable StorageAdapter and it is durable by default: a run cut off on one harness — host killed, network dropped mid-call — resumes on a new harness built over the same storage. Every step and model round that finished replays instantly and for free, and the run continues from the first one that did not. Add a durable SubprocessAdapter and long-running children survive a host restart too, with IPC streams resuming from the last acked frame.

Quick Example

import { AgentHarness } from '@noetic-tools/core';
import { createFileStorage, createLocalSubprocessAdapter } from '@noetic-tools/platform-node';

// Two distinct on-disk roots: subprocess manifests vs checkpoint snapshots.
const subprocessStorage = createFileStorage({
  root: `${process.env.HOME}/.noetic/subprocess`,
});
const checkpointStorage = createFileStorage({
  root: `${process.env.HOME}/.noetic/checkpoints`,
});

const harness = new AgentHarness({
  name: 'durable-agent',
  agentGraph: agent,
  params: {},
  environment: {
    subprocess: createLocalSubprocessAdapter({ storage: subprocessStorage }),
    // A storage adapter is enough: the checkpoint store defaults to one over it.
    storage: { adapter: checkpointStorage },
  },
});

Every run through this harness checkpoints as it goes and records each finished step. On restart, construct the same harness and resume:

// A run that died: restore it and run the same step on the same input.
const ctx = await harness.restore(executionId);
if (ctx) {
  await harness.run(agent, input, ctx); // finished steps replay; the rest runs live
}

execute() sessions resume on their own: re-deliver the cut-off turn's message with the same messageId and it picks up where it died (see Sessions). Any detachedSpawn lands a manifest entry, and reattachLiveChildren(harness) brings every still-running child back with its parent context rebuilt.

Set durability: false on the harness to turn every durability surface off — the adapter then backs context-layer persistence only. Pass your own createCheckpointStore({ storage }) to keep checkpoints on separate storage.

Storage is the only record. Nothing that decides how a run resumes lives only in a harness's memory, so a harness that just served a run and one built a moment ago behave the same. A failed run retried on the same context rebuilds from what storage holds — the stored baseline and ledger — exactly as a new harness restoring it would:

await harness.run(agent, input, ctx).catch(() => {}); // network drops mid-run
await harness.run(agent, input, ctx); // replays what finished, continues live

The Three Surfaces

Durable execution composes three primitives:

1. CheckpointStore + step ledger — saves and loads per-execution snapshots (where the run started, plus its live layer, channel, state, and usage data) and one ledger entry per finished step.
2. SubprocessAdapter.reattach / listLive — persists handle manifests for every long-lived child and rebinds them on parent restart.
3. Durable IPC (DurableOutboundQueue + protocol v2) — numbers every outbound IPC frame with a monotonic sequence, persists it, and resumes from the client's last ack on reconnect.

Each surface is independent. A host that needs durable checkpoints but in-process children gets the first without the other two. A host with long-lived OS subprocesses but no LLM state gets the second without the first. The CLI wires all three together; custom embedders mix and match.

Checkpoints

When Snapshots Fire

harness.checkpoint(ctx) runs automatically:

1. When the outermost run() on a context begins — so even a run whose first step dies can be restored.
2. After every step completes, anywhere in the tree. inParallel paths and spawn children checkpoint their root.
3. After runAppendPipeline() resolves.
4. When a run finishes — the snapshot then becomes the starting point for the next run on that context.

Captures for one execution are serialised, so an older snapshot never lands after a newer one. Any caller can also invoke harness.checkpoint(ctx) explicitly — it's an ordinary async method.

What's in a Snapshot

import type { CheckpointSnapshot } from '@noetic-tools/core';

interface CheckpointSnapshot {
  schemaVersion: 3;
  executionId: string;
  threadId?: string;
  resourceId?: string;
  frontier: Array<{ stepId: string; input: unknown; state?: unknown }>;
  layers: Record<string, unknown>;   // layerId → serialised state
  cwd: { current: string | null; previous?: string | null } | null;
  askUser: Array<{ id: string; input: unknown; createdAt: number }>;
  itemLog: { items: unknown[] };     // the log when the run in flight began
  occurrences: Record<string, number>; // dispatch counts at that same moment
  state?: unknown;                   // ctx.state, when it survives JSON encoding
  usage?: { tokens: TokenUsage; cost: number };
  channels?: ChannelSnapshot[];      // channels the execution used
  capturedAt: string;                // ISO-8601
}

type ChannelSnapshot =
  | { name: string; mode: 'value'; value?: unknown; hasValue: boolean }
  | { name: string; mode: 'queue'; capacity: number; queue: unknown[] }; // oldest first

itemLog and occurrences are the run's baseline: the item log and dispatch counts as they stood when the run in flight began (or the live values between runs). A resumed run's replayed steps put the rest of the history back from the step ledger, so the history is right on every context — including inParallel paths and spawned sub-agents, whose logs the snapshot never held. layers, channels, state, and usage are live, because replay redoes none of them: budgets like until.maxCost and state your steps wrote carry across a resume.

channels carries each value channel's current value and each queue channel's queued items. Restore refills them into the harness for any channel that has no live state yet. Parked recv/send calls and subscriptions are live promises, so they are not saved, and topic channels hold nothing to save. Snapshots written before channels were captured have no channels field and restore no channel state.

The snapshot is keyed by executionId and validated by CheckpointSnapshotSchema on load. Successive snapshots overwrite — no append-only log, no journaling. A failing save is logged and swallowed so a checkpoint never aborts an otherwise-successful step.

Restore

declare const harness: AgentHarness;
declare const executionId: string;

const restored = await harness.restore(executionId);
if (restored !== null) {
  // `restored.id === executionId`. It stands where the interrupted run
  // began: re-run the same step on the same input to resume it.
}

A snapshot is not required. If the snapshot write failed but the step ledger (below) has entries, restore returns a context with the original id and that ledger — but an empty item log, no restored layer or channel state, and the default cwd. Recorded steps replay and the rest run fresh. Returns null only when neither a snapshot nor a ledger entry is recorded. Throws NoeticConfigError with code: 'CHECKPOINT_SCHEMA_MISMATCH' when the persisted schemaVersion is unknown — callers discard via harness.clearCheckpoint(executionId) and start fresh. Schema v2 snapshots hold the item log at capture time rather than the run baseline, so restoring one would duplicate history; v1 snapshots key layer state by pre-rename ids (working-context, history-window, …). Both are rejected. There is no migration path.

Step-Level Resume

A restored context knows what happened but not how far it got: the frontier records which steps were in flight, never which finished. So alongside the snapshot, a harness with a checkpointStore keeps a step-completion ledger — one entry per successfully completed step, carrying that step's output.

A resumed run replays those outputs instead of re-running the steps. It is memoization, not fast-forward: a callModel step re-run to "catch up" would return something different from what the rest of the run already observed, so the recorded value is what flows downstream. Each entry also carries the items the step added to history, which replay puts back quietly, and a hash of the step's input. Set emitItemsOnReplay: true on the harness to also stream the items a step emitted again, for clients that reconnect to the resumed run. Only successes are recorded — a step that threw runs again — and a step whose id, kind, or input changed at a recorded position runs fresh, discarding its recorded subtree. That last rule is what keeps a restored chat from answering a new question with an old answer.

When a run finishes, its positional entries are dropped: the next run on the context starts past them. Entries live one key per step under execution:<id>:ledger:<seq>, so an append is O(1) regardless of run length. Nothing here is visible on a zero-config harness: without a checkpoint store the ledger is never constructed.

Usage on Replay

Replayed work keeps its usage figures but costs nothing now, so usage reports both:

billedCost sits next to cost on LLMResponse, StepMeta, Context, HarnessResponse, and SessionUsage. A context's billedCost counts this attempt's live calls — a restored context starts at 0 — so after a resume, ctx.cost - ctx.billedCost is what replay saved. A session's billedCost sums what storage knows its turns paid, plus the bills of cut-off calls once they are settled (see Paid Calls).

Model Rounds

An agent step — one callModel with tools — runs many rounds inside one step. Each finished round is recorded too: the model's output, the tool results, and the round's usage. A tool loop cut off on round five resumes with rounds one to four replayed: none of their tools run again, and only round five onward goes to the model. Rounds replay in order and stop at the first one that was not recorded. A tool that was running when the host died runs again.

A round also records while it streams: each output item the stream finishes is saved as it lands. A host killed mid-stream resumes from those items instead of asking the model for them again, and they keep their ids:

• Finished function calls run, and the model is asked again with their results, just as if the round had finished.
• A finished answer with no function calls ends the step. If the stream was cut between a message and the tool calls after it, that message becomes the answer.
• With nothing usable, such as reasoning alone, the round is asked again.

A round the loop rejects (an incomplete or failed response) does not keep its streamed items, and neither does the round that ends the loop once the step records, so a retry after a failed step asks the model again.

To know which streamed items are safe, watch {name}:model_round_recorded. It fires each time a round's record lands, with { round, itemIds, done }: a resume will not make any item in itemIds again. done is false while the round still streams. Don't use model_call_completed for this, since it fires before the round is recorded. A tool that was running when the host died, before its round recorded, still runs again.

Replay Identity

By default a step's ledger key is its execution path — its parent's path plus its id and a per-parent ordinal, like /plan#0/review-loop#0/body-llm#2. Replay identity is therefore positional: a resumed step gets the output recorded at the same position.

A step that fails gives its position back, so a retry — say, onError: 'retry' after a dropped connection — runs where a resume will look for it.

Each step runs on its own lane of the context — a view that shares state, history, and usage but carries the step's own path — so steps it dispatches hang under it even when other work runs concurrently on the same context. Two harness.run calls racing on one context get stable, separate paths:

• Different step ids under one parent always get distinct paths (/left#0, /right#0), whichever finishes first.
• The same step id run concurrently under one parent is numbered in dispatch order (/worker#0, /worker#1). That survives a resume only if the dispatch order does; when it depends on timing, give the step a replayKey.

That breaks when the work a position receives depends on timing. Concurrent workers pulling batches from a shared queue may pick different batches on resume, and positional replay would hand one batch's answer to another. Steps that return nothing and mutate state are worse off: replay skips them, so their state change is lost. Every step accepts two fields for this:

JSON workflow nodes take the same two fields; there replayKey is declarative — true or { "from": ["batchId"] } — and the hydrator builds the function. See JSON Runtime.

The usual shape is "replay the expensive leaves, re-run the cheap orchestration":

import type { ContextData } from '@noetic-tools/core';
import { inParallel, runCode } from '@noetic-tools/core';

const queue: string[] = ['batch-1', 'batch-2', 'batch-3'];

// The paid call: keyed by content, so it replays whichever worker runs it.
const gradeBatch = runCode<ContextData, string, string>({
  id: 'grade-batch',
  replayKey: (batch) => `grade:${batch}`,
  execute: async (batch) => `graded ${batch}`,
});

// The orchestration: cheap and timing-dependent, so it always re-runs.
const worker = (id: string) =>
  runCode<ContextData, string, string[]>({
    id,
    durable: false,
    execute: async (_input, ctx) => {
      const results: string[] = [];
      for (let batch = queue.shift(); batch; batch = queue.shift()) {
        results.push(String(await ctx.harness.run(gradeBatch, batch, ctx)));
      }
      return results;
    },
  });

const pool = inParallel<ContextData, string, string[]>({
  id: 'worker-pool',
  mode: 'all',
  durable: false,
  paths: () => [worker('worker-a'), worker('worker-b')],
  merge: (results) => results.flat(),
});

A durable: false step's channel sends run again on resume. The checkpoint already restored the queue items it sent before the crash, so each re-run send adds a duplicate. Send to a queue from a durable step when the item must land once.

schedule steps are never recorded, whatever durable says. A replayKey that throws costs that dispatch its durability, not its run: the step executes unrecorded and a warning is logged. If a different step id or kind shows up under a recorded key:, it runs fresh and only that entry is discarded.

Retention

Sharding bounds the cost of one append, not the total, so retention is bounded on two axes:

import { AgentHarness, DEFAULT_STEP_LEDGER_RETENTION } from '@noetic-tools/core';

const harness = new AgentHarness({
  name: 'durable-agent',
  params: {},
  environment: {
    storage: {
      adapter,
      stepLedgerRetention: {
        maxEntryBytes: 128 * 1024, // default: largest output recorded, UTF-8 bytes of its JSON
        maxEntries: 1e3,           // default: entries retained per execution
      },
    },
  },
});

Pass Infinity on either axis to disable that cap. A non-positive or NaN cap throws NoeticConfigError with code: 'STEP_LEDGER_RETENTION_INVALID' at construction — a silently-wrong cap would record nothing while the run still looked healthy.

Both degradations reduce to one thing: a step with no entry re-runs. That costs work, and re-does whatever effects the step has, but never replays a value that disagrees with the recorded run. Retention is not silent — the first drop of each kind logs a warning.

When to Clear a Ledger

// Discards the snapshot AND every ledger shard for this execution.
await harness.clearCheckpoint(executionId);

Call it in two situations:

• The workflow changed. Replay happens at the coarsest completed granularity — a composite step is an ordinary runCode step here, so a parent that finished replays wholesale without descending. An edit to one of its children is therefore invisible, and the old output wins. A host that edited the workflow must clear rather than resume onto the old ledger.
• The execution is abandoned. A finished run already drops its positional entries; checkpointStore.clear(executionId) alone strands the ledger's per-step keys, because nothing else enumerates them.

The Side-Effect Boundary

Replaying an output replays a step's value, not its effect. A runCode step that wrote a file, or an invokeTool step that opened a PR, returns its recorded output on replay while the effect is not redone — correct only if that effect was durable when it happened.

Core's ledger therefore covers control flow, callModel steps, and their rounds — a recorded round's tools never run again. It does not claim exactly-once execution for a tool interrupted mid-round: it has no durable pre-dispatch record of a tool call and no way to know whether a given tool is idempotent. (Paid model calls do get one — see Paid Calls.) Fence effects whose duplication matters at the tool/host boundary.

Sessions

harness.execute() sessions are durable too. With a storage adapter, each thread keeps its history, what its turns spent, and the turn in flight under session:<threadId> — and every turn reads them from storage rather than from the harness's memory:

• Any harness over the same storage continues a thread with its history, including harnesses taking turns on one thread. seedSessionHistory writes to storage too.
• A turn cut off mid-run resumes when its messages are delivered again with the same messageIds — the at-least-once retry most queues already do. Finished steps replay; the turn continues from where it died.
• Different messages after a cut-off turn abandon it and start fresh.
• Steers sent into a running turn are kept until it ends. A resumed turn applies the ones the cut-off run had not; ones it already applied come back with the replayed work and are not applied twice.

// First host: the turn dies mid-run.
await harness.execute(text, { threadId, messageId: event.id });

// Next host, same storage: the queue re-delivers the same event.
await harness.execute(text, { threadId, messageId: event.id }); // resumes
const response = await harness.getAgentResponse({ threadId });

A turn that finished is not replayed when re-delivered — it runs again as a new turn — so acknowledge messages once their turn completes.

Paid Calls

A model call cut off with its host — or by cancel(ctx), a steer, or a dropped connection — finished nothing, so the ledger has no entry for it. The provider billed it anyway: a cut-off stream is billed for every token generated, even after the client went away. So every paid call leaves a dispatch record in storage before it is sent — each callModel round and each native decide() call:

• Before send, the record holds the model, a hash of the request, a bound on the prompt's tokens, and the request's output cap.
• When the response headers arrive, the provider's generation id is saved to it — before the body is read, for decide() calls as for callModel rounds.
• When the bill arrives with the response, the cost is counted and the record is deleted. A response the harness rejects and asks again still counts, and a step that fails still charges its context with what its rounds spent.
• Otherwise the record stays, and the call is pending: held at its ceiling until its bill is known.

// Unsettled calls, all of them or one execution's, each with its ceiling.
const pending = await harness.pendingSpend(executionId);

// Price them through the provider's generation lookup and settle them into usage.
const settled = await harness.reconcileSpend({ executionId, patienceMs: 60_000 });

// A call that can never be priced: settle it at an amount you assert.
await harness.writeOffSpend(pending[0].id, { assumedCost: 0.05 });

Reconciliation runs on its own after restore() and the moment a call is cut off; call reconcileSpend() again whenever you like — at boot, for instance, to settle every execution's leftovers. The provider prices a call some seconds after it ends (7–32 s observed), so a lookup keeps asking, backing off from 1 s to 10 s, for up to patienceMs (default 60 s). Its timers never keep your process alive.

A settled bill is charged once: to the run in flight on that execution, else to the thread's stored session, else to the context restore() returned — so it lands in cost and billedCost like any other spend. {name}:spend_pending and {name}:spend_settled events report both ends.

Held spend stays bounded. Until it is settled, a pending call is held at its ceiling, an upper bound on its bill: the model's dearest published prices (long-context tiers, cache writes, and reasoning included) applied to the prompt bound and the output cap (params.maxTokens, else the model's published maximum, else the model's context). The prompt bound is the request's size in UTF-8 bytes, since no common tokenizer spends more than one token per byte of text, plus 1,024 tokens for the provider's own framing and 4,096 for each image, file, or audio part. A long PDF or audio file sent by URL can cost more than its allowance, and its call then more than its ceiling; so can server-tool fees such as web search, and a fallback provider that charges more than the model's published price. heldCost reports the hold beside cost and billedCost on Context, HarnessResponse, and SessionUsage, and until.maxCost counts it, so a resumed run cannot spend what its cut-off attempt may already have spent. A model with no published price, or no output cap anywhere, has no ceiling: spend_pending flags it unbounded and it adds nothing to heldCost, so until.maxCost cannot see it — hold the rest of your budget yourself when you see one. Set params.maxTokens on paid steps to keep every ceiling finite.

A damaged record is never dropped. A spend:<id> value that no longer reads as a record — torn by a crash mid-write, or corrupted, even one the store reads as nothing while its key still lists — still means a paid call was sent. pendingSpend() lists it with unreadable: true. Only its id, executionId, and threadId are real (the owner fields only when the damaged value still names them); its other fields hold placeholders (kind: 'model', model: 'unknown', requestHash: '', inputTokens: 0, sentAt the Unix epoch), so check unreadable before trusting them. spend_pending flags it unreadable and unbounded. Like an unbounded call it has no ceiling, adds nothing to heldCost, and is never priced, so hold the rest of your budget yourself or write it off:

for (const p of await harness.pendingSpend()) {
  if (p.unreadable) {
    await harness.writeOffSpend(p.id, { assumedCost: 0.5 });
  }
}

One that names no execution may be any execution's, so it is listed with each of them.

Some calls can only be bounded. A call cut off before any response byte has no generation id. It stays held until you write it off.

Evidence is never discarded. Records live under spend:<id>, apart from checkpoints, so clearCheckpoint and an abandoned turn leave them alone. A record leaves storage only when it is settled or written off.

Run one harness per storage when reconciling: a harness skips the calls it has in flight, but cannot see another harness's.

Implement getMany on a networked adapter

The step ledger stores one key per completed step, so restoring an execution means reading every key under execution:<id>:ledger:. StorageAdapter declares an optional batch read for exactly this:

getMany?<T>(keys: string[]): Promise<Map<string, T>>;

Implement it and a restore is one query. Omit it and the framework falls back to a parallel get per key — correct, but on a D1- or HTTP-backed store that is a round trip per step of the resumed run, arriving all at once on the recovery path. Missing keys must be absent from the returned map rather than mapped to null.

The built-in createInMemoryStorage() and createFileStorage() both implement it.

Restoring a decorated context

A snapshot recovers data, not the live objects a host attached when it built the original context — event broadcasters, message queues, abort registrations. Restore without them and the host gets a bare context back. Nothing throws; streaming and mid-turn injection just stop working, on precisely the runs that already crashed once.

restore() takes an optional second argument shaped like createContext, so a host hands back the wiring it supplied the first time:

declare const harness: AgentHarness;
declare const executionId: string;
declare const sessionState: { queue: unknown };

const restored = await harness.restore(executionId, {
  state: sessionState,
});

interface RestoreContextOptions {
  parent?: Context;
  rollUpUsage?: boolean;
  state?: unknown;            // wins over the snapshot's state when given
  contextLayers?: ContextLayer[];
}

Two rules:

• Snapshot-owned fields are not accepted. items, threadId, resourceId, and cwd always come from the persisted record — they are absent from RestoreContextOptions so you cannot pass a value that would be silently ignored. Caller options are applied first and the snapshot's values overwrite them, so the record always wins.
• Post-construction decoration is still yours. Fields you Object.assign onto a context, and abort registration, belong on the context restore() returns. Its id is already the original executionId, so anything keyed by execution id lands on the right key.

Limitations

A step whose side effect landed but whose ledger entry did not runs again on resume. The framework cannot make arbitrary runCode bodies idempotent — write bodies that are safe to re-execute where durability matters, or gate with an external idempotency key.

A model stream is not resumed mid-item. A round cut off while the model was generating resumes from the last item its stream finished; an item cut off part way is made again. The cut-off round's bill is not lost — see Paid Calls.

A durable: false step re-runs on resume, and so do its effects: its channel sends land again and its ctx.state writes apply again on top of the restored state.

Subprocess Adapter Durability

Every SubprocessAdapter exposes two durability hooks:

interface SubprocessAdapter {
  reattach(handleId: string): Promise<SubprocessHandle | null>;
  listLive(): Promise<ReadonlyArray<SubprocessHandle>>;
  // ... standard methods ...
}

When the adapter is constructed with a storage: StorageAdapter, every spawn() writes a manifest entry covering handleId, stepId, serializedInput, executionId (the child context), parentExecutionId (the context that spawned it), the transport identity (pid + pidStarttime for the local adapter, socketPath for IPC), and any caller-attached metadata, which rehydrated handles carry on handle.metadata. listLive() scans the manifest prefix; reattach(handleId) re-queries liveness and rebinds the handle.

Without storage, listLive() returns an empty set and reattach() returns null — every surface degrades gracefully to "fresh start".

Host Restart

The CLI helper reattachLiveChildren wires up the recovery step:

import { reattachLiveChildren } from '@noetic-tools/cli';

const { handles, contexts } = await reattachLiveChildren(harness);
for (const [handleId, ctx] of contexts) {
  // Each context has its pre-crash item log, layer state, and cwd.
  // Re-subscribe to the handle's IPC stream, replay pending ask-user
  // modals, continue from the restored frontier.
}

The helper calls harness.subprocess.listLive(), then harness.restore(executionId) for every handle that carries an executionId. With no durable storage configured the call is a cheap no-op. A host that decorates its contexts re-applies that wiring per restored context — see Restoring a decorated context.

Durable IPC

A server that composes DurableOutboundQueue wraps every outbound frame in a durable envelope keyed by a monotonic sequence number. The client tracks the highest seq it has durably consumed and sends durableResume { ackedThrough } on every reconnect. The server replays any frames the client has not acked, resumes live emission, and compacts the queue when durableAck { throughSeq } arrives.

import { createDurableOutboundQueue } from '@noetic-tools/platform-node';

const queue = await createDurableOutboundQueue({ storage, socketPath });

// On each outbound frame:
const { seq } = await queue.append(JSON.stringify(frame));
socket.write(encodeFrame({ type: 'durable', seq, frame }));

// On client durableAck:
await queue.ackUpTo(ack.throughSeq);

// On client durableResume:
for (const entry of await queue.frameRange(resume.ackedThrough + 1)) {
  socket.write(encodeFrame({
    type: 'durable',
    seq: entry.seq,
    frame: JSON.parse(entry.frame),
  }));
}

The queue is transport-agnostic. Any framed byte stream — unix socket, WebSocket, TCP — can use the same pattern. Frames are opaque strings to the queue; the server encodes them before append and the client decodes them after unwrap.

Protocol v2

Three new frame types extend the IPC wire protocol:

Protocol v2 is backward compatible. Peers that do not opt into durable delivery neither emit nor receive the new frames; v1 peers interoperate seamlessly.

Storage Layout

The CLI reserves three distinct on-disk roots:

Subprocess manifests and IPC queues can share a root because both are owned by the adapter. Keeping the checkpoint-snapshot root distinct means "discard all recovery data" is a single directory removal.

Run an Agent Out-of-Process

Switch a specific spawn to run in its own OS child by passing a local adapter as a per-call override:

import { createFileStorage, createLocalSubprocessAdapter } from '@noetic-tools/platform-node';

const localAdapter = createLocalSubprocessAdapter({
  storage: createFileStorage({
    root: `${process.env.HOME}/.noetic/subprocess`,
  }),
});

const handle = harness.detachedSpawn(
  researchAgent,
  'summarise the latest arXiv papers on RL',
  parentCtx,
  {
    subprocess: localAdapter,
    cwdInit: '/tmp/research-workspace',
  },
);

// The parent continues immediately. The child runs in a separate bun
// process. If the parent crashes, the manifest survives and the child
// keeps running; on restart, `reattachLiveChildren` rebinds it.
const result = await handle.await();

Per-step overrides work the same way: set subprocess: localAdapter on a runCode or spawn opts and every dispatch of that step uses the local adapter regardless of the harness default.

Survive a Host Crash

// First boot: configure durable storage.
const harness = new AgentHarness({
  name: 'crash-proof',
  agentGraph: agent,
  params: {},
  environment: {
    subprocess: createLocalSubprocessAdapter({
      storage: createFileStorage({ root: `${process.env.HOME}/.noetic/subprocess` }),
    }),
    storage: {
      checkpointStore: createCheckpointStore({
        storage: createFileStorage({ root: `${process.env.HOME}/.noetic/checkpoints` }),
      }),
    },
  },
});

// Launch a long-lived child.
harness.detachedSpawn(backgroundWorkerStep, jobSpec, parentCtx);

// ... process crashes ...

// Second boot: same harness construction, then:
import { reattachLiveChildren } from '@noetic-tools/cli';
const { handles, contexts } = await reattachLiveChildren(harness);
// `handles` contains the still-running background worker.
// `contexts` has its rebuilt parent context.

The pattern generalises to any host — not just @noetic-tools/cli. Any embedder that configures durable storage and calls listLive() + restore() on boot gets the same recovery.

Guarantees

With the full durable stack configured:

• Every completed step, finished model round, spawned child, and ask-user prompt survives a host crash — a new harness over the same storage replays them and sends the first unfinished model call exactly the history it had. Steps outside the ledger's retention bounds re-run.
• Restart rediscovers running children and rebuilds their parent contexts.
• IPC replay is exactly-once at the frame level when the client acks correctly.
• Schema version drift surfaces as a typed error rather than silent corruption.
• Every paid model call is recorded before it is sent. One cut off after its response headers arrived settles at the provider's exact cost; one cut off earlier is held at its ceiling. Neither is lost to a discard.

Without the full stack, the surfaces that depend on it are no-ops and the harness behaves as though durability were never requested.

Related Pages

• AgentHarness — the environment.subprocess and environment.storage.checkpointStore options.
• Spawn — subprocess override on spawn and detachedSpawn.
• Context — ctx.checkpoint() semantics and item log dedupe.