Writing Recoverable Tools

A process can be lost while a tool runs: a crash, a deploy, or a Cloudflare Durable Object evicted mid-step. Helix recovers that step without asking the model again. This page explains what recovery does with each call, and the two things a tool can give it: a replay policy and a dedupe key.

Save first

On runtime-js and the Cloudflare Durable Object, a model reply whose calls start a tool body is committed before any of those bodies starts: the reply, with every call's arguments, is saved in its own append checkpoint, then the step's step commit records the results. So if the run is lost while a tool runs, the store still holds exactly what the model asked for, and the next wake, resume or retry recovers the step from it.

A step whose reply starts no body (text only, __finish__, client-executed or approval-gated calls only) makes no saved commit: a client call is saved by its suspension commit.

What recovery does with each call

For every call of the saved reply that has no result yet:

The call	What recovery does
A pending client-tool or approval call	Nothing: the client's answer settles it, as before
A remote sub-agent call	Re-attaches to the child by the call's dispatch id, or starts it once under that id
A persistent-companion call (Durable Object)	Re-enters the child phase
A tool with replay: 'safe'	Runs it again with the same arguments and dispatch id, at most 3 bodies in all
A safe tool already run 3 times	Pairs it not executed, replay_limit_reached: "It will not be run again automatically"
Anything else	Pairs it not executed, outcome_unknown: "the tool may have run. Check whether it took effect before repeating it"

Two order rules move a call to outcome_unknown even when its tool is safe:

• Its group's order. A call behind an outcome_unknown or replay_limit_reached call of its concurrencyGroup, or behind one in the sequential chain, is outcome_unknown too. It was written to run after that call's effect, which is now in doubt.
• Local sub-agents. An in-process child died with its parent, so its call is outcome_unknown.

A call that returned but whose result was never saved (the crash came before the step commit) is outcome_unknown as well: recovery cannot see a result that was never written.

Then the run goes on from the step's results, and the model decides what to do with any call it is told may have run.

replay: 'safe'

import { defineTool } from '@helix-agents/core';
import { z } from 'zod';

const setPreference = defineTool({
  name: 'set_preference',
  description: "Set one of the user's preferences",
  inputSchema: z.object({ key: z.string(), value: z.string() }),
  // Writing the same value twice leaves the same result as writing it once.
  replay: 'safe',
  execute: async ({ key, value }, ctx) => {
    await preferences.put(ctx.sessionId, key, value);
    return { ok: true };
  },
});

replay is recovery policy only: it changes nothing while no run is lost. The default is unsafe: an unsafe call is never run again, and is paired outcome_unknown.

safe is a promise: two attempts of one call, even overlapping, leave the same result as one. Overlapping is real: a body that was still running when its run ended (a Durable Object whose stop deadline passed, a process cut off from its store) may finish after the recovery has started its own attempt. That late body's result and state writes are refused, but any external effect it made has happened. So mark a tool safe only when that is harmless:

• an upsert, a set, a delete by key;
• an external call made with an idempotency key (below);
• a read.

Not safe: sending a message, charging a card without an idempotency key, appending to a list, incrementing a counter.

Recovery checks the policy twice: the call's saved row records the policy its tool had when the model called it, and recovery reads the tool as it is now. Both must say safe. Recovery also re-validates the saved arguments against the tool's current inputSchema. So a deploy between the crash and the recovery can only make recovery more careful: a tool removed, a schema the stored arguments no longer pass, or a policy changed to unsafe each make the call outcome_unknown.

ctx.dispatchId: the dedupe key

Every call gets a dispatch id, minted by core and committed on the call's row before its tool starts. It is the same on every attempt of that call, on every entry, including a recovery in a later run. Key an external effect on it, scoped to the session:

const chargeCard = defineTool({
  name: 'charge_card',
  description: 'Charge the saved card',
  inputSchema: z.object({ amountCents: z.number().int().positive() }),
  // Safe because the payment provider deduplicates on the idempotency key.
  replay: 'safe',
  execute: async ({ amountCents }, ctx) => {
    const charge = await payments.charge({
      amountCents,
      idempotencyKey: `${ctx.sessionId}:${ctx.dispatchId}`,
    });
    return { chargeId: charge.id };
  },
});

Use ctx.dispatchId, not ctx.toolCallId: the tool call id is the provider's, and nothing vouches that it is unique or stable. ctx.dispatchId is set on runtime-js and the Cloudflare Durable Object. It is absent on the runtimes that do not save every call first yet, so a tool that needs it should fail loudly without it.

Which entries recover

Entry over a saved step	What happens
A Durable Object's wake, resume({ mode: 'continue' })	Recovery, as above
retry() with no message, after a run failed	Recovery: the retry rewinds past the failed run's outcome_unknown rows to the saved commit, then recovers
retry({ message }), resume({ mode: 'with_message' }), a new turn	The message redirects the turn: every call is paired outcome_unknown before it, nothing runs again
resume({ mode: 'from_checkpoint', checkpointId }) at a saved append	Recovery of that step, as above (see Checkpoints)
retry({ checkpointId, message }) at a saved append	Its calls are paired outcome_unknown before the message; without message it is refused (validation_error)
A stop	Finished calls keep their results; the rest are paired run_stopped

A run that fails after the save (a store error on the step commit, for example) never leaves a call unanswered: it pairs each saved call outcome_unknown before its terminal commit. A later retry() replaces those rows with what recovery decides.

Runtimes

Runtime	Saves first	ctx.dispatchId	Recovery
runtime-js	Every step that starts a body	Every call	Every rule above
Cloudflare Durable Object	Every step that starts a body	Every call	Every rule above, on its wake
Cloudflare Workflows, Temporal, DBOS	Only a step that delegates to a remote child	Not set (a remote child's dispatch id is on its binding)	Re-attach of remote children only

See also

• Defining Tools
• Checkpoints: a tool step writes two checkpoints
• Remote Agents: re-attaching to a remote child by its dispatch id