Durability

Durable execution is opt-in. A zero-config AgentHarness has no persistent storage, so every durability hook is a no-op and the harness behaves exactly like before. A harness wired with a durable StorageAdapter, a CheckpointStore, and a durable SubprocessAdapter gains full crash recovery — long-running children survive a host restart, parent contexts are rebuilt from snapshots, and IPC streams resume from the last acked frame with zero duplicates and zero losses.

Quick Example

import { AgentHarness, createCheckpointStore } from '@noetic-tools/core';
import { createFileStorage, createLocalSubprocessAdapter } from '@noetic-tools/platform-node';

// Two distinct on-disk roots: subprocess manifests vs checkpoint snapshots.
const subprocessStorage = createFileStorage({
  root: `${process.env.HOME}/.noetic/subprocess`,
});
const checkpointStorage = createFileStorage({
  root: `${process.env.HOME}/.noetic/checkpoints`,
});

const harness = new AgentHarness({
  name: 'durable-agent',
  agentGraph: agent,
  params: {},
  environment: {
    subprocess: createLocalSubprocessAdapter({ storage: subprocessStorage }),
    storage: {
      checkpointStore: createCheckpointStore({ storage: checkpointStorage }),
    },
  },
});

Any detachedSpawn through this harness lands a manifest entry. Any execute() turn lands a checkpoint. On restart, construct the same harness and call reattachLiveChildren(harness) — every still-running child comes back with its parent context rebuilt.

The Three Surfaces

Durable execution composes three primitives:

1. CheckpointStore — saves and loads per-execution snapshots covering the step frontier, context-layer state, cwd, pending ask-user queue, and item log.
2. SubprocessAdapter.reattach / listLive — persists handle manifests for every long-lived child and rebinds them on parent restart.
3. Durable IPC (DurableOutboundQueue + protocol v2) — numbers every outbound IPC frame with a monotonic sequence, persists it, and resumes from the client's last ack on reconnect.

Each surface is independent. A host that needs durable checkpoints but in-process children gets the first without the other two. A host with long-lived OS subprocesses but no LLM state gets the second without the first. The CLI wires all three together; custom embedders mix and match.

Checkpoints

When Snapshots Fire

harness.checkpoint(ctx) runs automatically at four boundaries:

1. After every execute() call that mutated the item log.
2. After detachedSpawn() settles (success or failure).
3. When an ask-user prompt is enqueued.
4. After runAppendPipeline() resolves.

Any caller can also invoke harness.checkpoint(ctx) explicitly — it's an ordinary async method.

What's in a Snapshot

import type { CheckpointSnapshot } from '@noetic-tools/core';

interface CheckpointSnapshot {
  schemaVersion: 2;
  executionId: string;
  threadId?: string;
  resourceId?: string;
  frontier: Array<{ stepId: string; input: unknown; state?: unknown }>;
  layers: Record<string, unknown>;   // layerId → serialised state
  cwd: { current: string | null; previous?: string | null } | null;
  askUser: Array<{ id: string; input: unknown; createdAt: number }>;
  itemLog: { items: unknown[] };
  capturedAt: string;                // ISO-8601
}

The snapshot is keyed by executionId and validated by CheckpointSnapshotSchema on load. Successive snapshots overwrite — no append-only log, no journaling. A failing save is logged and swallowed so a checkpoint never aborts an otherwise-successful step.

Restore

declare const harness: AgentHarness;
declare const executionId: string;

const restored = await harness.restore(executionId);
if (restored !== null) {
  // `restored.id === executionId`. The item log, layer state, and cwd
  // are rebuilt from the snapshot; the frontier tells the caller which
  // step to resume.
}

Returns null when no snapshot is recorded. Throws NoeticConfigError with code: 'CHECKPOINT_SCHEMA_MISMATCH' when the persisted schemaVersion is unknown — callers discard via harness.clearCheckpoint(executionId) and start fresh. Schema v1 snapshots (written before the layer-id rename) fall in this bucket: their layers record uses ids such as working-context, static-content, and history-window, while current layers use scratchpad, instructions, and history. They are rejected rather than restored with silently missing state. There is no migration path.

Step-Level Resume

A restored context knows what happened but not how far it got: the frontier records which steps were in flight, never which finished. So alongside the snapshot, a harness with a checkpointStore keeps a step-completion ledger — one entry per successfully completed step, carrying that step's output.

A resumed run replays those outputs instead of re-running the steps. It is memoization, not fast-forward: a callModel step re-run to "catch up" would return something different from what the rest of the run already observed, so the recorded value is what flows downstream. Only successes are recorded — a step that threw runs again — and a step whose identity changed at a recorded path runs fresh, discarding its recorded subtree.

Entries live one key per step under execution:<id>:ledger:<seq>, so an append is O(1) regardless of run length. Nothing here is visible on a zero-config harness: without a checkpointStore the ledger is never constructed.

Retention

Sharding bounds the cost of one append, not the total, so retention is bounded on two axes:

import { AgentHarness, DEFAULT_STEP_LEDGER_RETENTION } from '@noetic-tools/core';

const harness = new AgentHarness({
  name: 'durable-agent',
  params: {},
  environment: {
    storage: {
      checkpointStore,
      stepLedgerRetention: {
        maxEntryBytes: 128 * 1024, // default: largest output recorded, UTF-8 bytes of its JSON
        maxEntries: 1e3,           // default: entries retained per execution
      },
    },
  },
});

Pass Infinity on either axis to disable that cap. A non-positive or NaN cap throws NoeticConfigError with code: 'STEP_LEDGER_RETENTION_INVALID' at construction — a silently-wrong cap would record nothing while the run still looked healthy.

Both degradations reduce to one thing: a step with no entry re-runs. That costs work, and re-does whatever effects the step has, but never replays a value that disagrees with the recorded run. Retention is not silent — the first drop of each kind logs a warning.

When to Clear a Ledger

// Discards the snapshot AND every ledger shard for this execution.
await harness.clearCheckpoint(executionId);

Call it in two situations:

• The workflow changed. Replay happens at the coarsest completed granularity — a composite step is an ordinary runCode step here, so a parent that finished replays wholesale without descending. An edit to one of its children is therefore invisible, and the old output wins. A host that edited the workflow must clear rather than resume onto the old ledger.
• The execution is done with — finished, or abandoned. checkpointStore.clear(executionId) alone strands the ledger's per-step keys, because nothing else enumerates them.

The Side-Effect Boundary

Replaying an output replays a step's value, not its effect. A runCode step that wrote a file, or an invokeTool step that opened a PR, returns its recorded output on replay while the effect is not redone — correct only if that effect was durable when it happened.

Core's ledger therefore covers control flow and callModel steps. It does not claim exactly-once execution for tools: it has no durable pre-dispatch record and no way to know whether a given tool is idempotent. Fence effects at the tool/host boundary.

Implement getMany on a networked adapter

The step ledger stores one key per completed step, so restoring an execution means reading every key under execution:<id>:ledger:. StorageAdapter declares an optional batch read for exactly this:

getMany?<T>(keys: string[]): Promise<Map<string, T>>;

Implement it and a restore is one query. Omit it and the framework falls back to a parallel get per key — correct, but on a D1- or HTTP-backed store that is a round trip per step of the resumed run, arriving all at once on the recovery path. Missing keys must be absent from the returned map rather than mapped to null.

The built-in createInMemoryStorage() and createFileStorage() both implement it.

Restoring a decorated context

A snapshot recovers data, not the live objects a host attached when it built the original context — event broadcasters, message queues, abort registrations. Restore without them and the host gets a bare context back. Nothing throws; streaming and mid-turn injection just stop working, on precisely the runs that already crashed once.

restore() takes an optional second argument shaped like createContext, so a host hands back the wiring it supplied the first time:

declare const harness: AgentHarness;
declare const executionId: string;
declare const sessionState: { queue: unknown };

const restored = await harness.restore(executionId, {
  state: sessionState,
});

interface RestoreContextOptions {
  parent?: Context;
  state?: unknown;
  context?: ContextLayer[];
}

Two rules:

• Snapshot-owned fields are not accepted. items, threadId, resourceId, and cwd always come from the persisted record — they are absent from RestoreContextOptions so you cannot pass a value that would be silently ignored. Caller options are applied first and the snapshot's values overwrite them, so the record always wins.
• Post-construction decoration is still yours. Fields you Object.assign onto a context, and abort registration, belong on the context restore() returns. Its id is already the original executionId, so anything keyed by execution id lands on the right key.

Limitations

Durable execution can replay a step body whose prior completion's checkpoint failed to land. The framework cannot make arbitrary runCode bodies idempotent — write bodies that are safe to re-execute where durability matters, or gate with an external idempotency key.

LLM mid-stream is not resumed. If the host dies while the model is generating, on restart the turn is re-issued. The item log's response-id dedupe catches identical responses; different responses win as a new turn.

Subprocess Adapter Durability

Every SubprocessAdapter exposes two durability hooks:

interface SubprocessAdapter {
  reattach(handleId: string): Promise<SubprocessHandle | null>;
  listLive(): Promise<ReadonlyArray<SubprocessHandle>>;
  // ... standard methods ...
}

When the adapter is constructed with a storage: StorageAdapter, every spawn() writes a manifest entry covering handleId, stepId, serializedInput, executionId, the transport identity (pid + pidStarttime for the local adapter, socketPath for IPC), and any caller-attached metadata. listLive() scans the manifest prefix; reattach(handleId) re-queries liveness and rebinds the handle.

Without storage, listLive() returns an empty set and reattach() returns null — every surface degrades gracefully to "fresh start".

Host Restart

The CLI helper reattachLiveChildren wires up the recovery step:

import { reattachLiveChildren } from '@noetic-tools/cli';

const { handles, contexts } = await reattachLiveChildren(harness);
for (const [handleId, ctx] of contexts) {
  // Each context has its pre-crash item log, layer state, and cwd.
  // Re-subscribe to the handle's IPC stream, replay pending ask-user
  // modals, continue from the restored frontier.
}

The helper calls harness.subprocess.listLive(), then harness.restore(executionId) for every handle that carries an executionId. With no durable storage configured the call is a cheap no-op. A host that decorates its contexts re-applies that wiring per restored context — see Restoring a decorated context.

Durable IPC

A server that composes DurableOutboundQueue wraps every outbound frame in a durable envelope keyed by a monotonic sequence number. The client tracks the highest seq it has durably consumed and sends durableResume { ackedThrough } on every reconnect. The server replays any frames the client has not acked, resumes live emission, and compacts the queue when durableAck { throughSeq } arrives.

import { createDurableOutboundQueue } from '@noetic-tools/platform-node';

const queue = await createDurableOutboundQueue({ storage, socketPath });

// On each outbound frame:
const { seq } = await queue.append(JSON.stringify(frame));
socket.write(encodeFrame({ type: 'durable', seq, frame }));

// On client durableAck:
await queue.ackUpTo(ack.throughSeq);

// On client durableResume:
for (const entry of await queue.frameRange(resume.ackedThrough + 1)) {
  socket.write(encodeFrame({
    type: 'durable',
    seq: entry.seq,
    frame: JSON.parse(entry.frame),
  }));
}

The queue is transport-agnostic. Any framed byte stream — unix socket, WebSocket, TCP — can use the same pattern. Frames are opaque strings to the queue; the server encodes them before append and the client decodes them after unwrap.

Protocol v2

Three new frame types extend the IPC wire protocol:

Protocol v2 is backward compatible. Peers that do not opt into durable delivery neither emit nor receive the new frames; v1 peers interoperate seamlessly.

Storage Layout

The CLI reserves three distinct on-disk roots:

Subprocess manifests and IPC queues can share a root because both are owned by the adapter. Keeping the checkpoint-snapshot root distinct means "discard all recovery data" is a single directory removal.

Run an Agent Out-of-Process

Switch a specific spawn to run in its own OS child by passing a local adapter as a per-call override:

import { createFileStorage, createLocalSubprocessAdapter } from '@noetic-tools/platform-node';

const localAdapter = createLocalSubprocessAdapter({
  storage: createFileStorage({
    root: `${process.env.HOME}/.noetic/subprocess`,
  }),
});

const handle = harness.detachedSpawn(
  researchAgent,
  'summarise the latest arXiv papers on RL',
  parentCtx,
  {
    subprocess: localAdapter,
    cwdInit: '/tmp/research-workspace',
  },
);

// The parent continues immediately. The child runs in a separate bun
// process. If the parent crashes, the manifest survives and the child
// keeps running; on restart, `reattachLiveChildren` rebinds it.
const result = await handle.await();

Per-step overrides work the same way: set subprocess: localAdapter on a runCode or spawn opts and every dispatch of that step uses the local adapter regardless of the harness default.

Survive a Host Crash

// First boot: configure durable storage.
const harness = new AgentHarness({
  name: 'crash-proof',
  agentGraph: agent,
  params: {},
  environment: {
    subprocess: createLocalSubprocessAdapter({
      storage: createFileStorage({ root: `${process.env.HOME}/.noetic/subprocess` }),
    }),
    storage: {
      checkpointStore: createCheckpointStore({
        storage: createFileStorage({ root: `${process.env.HOME}/.noetic/checkpoints` }),
      }),
    },
  },
});

// Launch a long-lived child.
harness.detachedSpawn(backgroundWorkerStep, jobSpec, parentCtx);

// ... process crashes ...

// Second boot: same harness construction, then:
import { reattachLiveChildren } from '@noetic-tools/cli';
const { handles, contexts } = await reattachLiveChildren(harness);
// `handles` contains the still-running background worker.
// `contexts` has its rebuilt parent context.

The pattern generalises to any host — not just @noetic-tools/cli. Any embedder that configures durable storage and calls listLive() + restore() on boot gets the same recovery.

Guarantees

With the full durable stack configured:

• Every completed step, spawned child, and ask-user prompt survives a host crash — steps within the ledger's retention bounds replay their recorded output; anything outside them re-runs.
• Restart rediscovers running children and rebuilds their parent contexts.
• IPC replay is exactly-once at the frame level when the client acks correctly.
• Schema version drift surfaces as a typed error rather than silent corruption.

Without the full stack, the surfaces that depend on it are no-ops and the harness behaves as though durability were never requested.

Related Pages

• AgentHarness — the environment.subprocess and environment.storage.checkpointStore options.
• Spawn — subprocess override on spawn and detachedSpawn.
• Context — ctx.checkpoint() semantics and item log dedupe.