Interrupt and Resume
Helix Agents supports interrupting and resuming agent execution. This enables user-controlled pauses, crash recovery, time-travel debugging, and human-in-the-loop (HITL) workflows.
Overview
When an agent is running, you can:
- Interrupt — soft stop that saves state for later resumption.
- Resume — continue execution from where it stopped.
This is different from abort, which is a hard stop that fails the agent permanently.
v7 changes at a glance
Three things changed in v7 that touch this guide directly:
- Durable interrupt protocol. v6's
interrupt(handle)walked an in-memoryactiveHandlesmap and returned 503INTERRUPT_NOT_LOCALwhen the handle wasn't on the same process. A stop is now a durable record on the run (see Durable stop protocol); the running loop picks it up at its next boundary regardless of which process owns the run. The 503 response is gone. - Three new suspended statuses.
RunOutcome.kind(and thereforehandle.result().status) gains'suspended_client_tool','suspended_awaiting_children', and'suspended_step_partial'. Existingswitchstatements that only handle'completed' | 'failed' | 'interrupted'will fall through silently for HITL agents. - Resume is explicit for HITL. When a run suspends at a HITL boundary,
submitToolResultno longer wakes an in-memory waiter. The chat handler (or your code) must callexecutor.resume({ sessionId })after the submission lands.
If you are upgrading from v6, read the v6 to v7 migration guide end-to-end before deploying. The rest of this page describes the v7 model.
execute() vs resume() Semantics
Understanding the distinction between execute() and resume() is critical for correct agent lifecycle management.
execute() - Fresh Start
execute() always starts a fresh execution. Using the same sessionId continues the conversation within that session:
// New session with generated ID
const handle = await executor.execute(agent, 'Hello');
// New session with specific ID
const handle = await executor.execute(agent, 'Hello', { sessionId: 'my-session' });
// Continue conversation in existing session
const handle2 = await executor.execute(agent, 'Follow up', { sessionId: 'my-session' });When you call execute():
- New run begins - A run-start commit appends your input and creates the turn's run record in one atomic step. It is the only gate: a concurrent caller that loses it writes nothing.
- Stream continues - The session's stream is made live again for the new run. Earlier chunks are kept; the new run's chunks follow its
run_startedmarker. - Session state continues - Messages and custom state are preserved within the session
This means each execute() call starts a fresh run. If you call execute() with the same sessionId, the new run continues the conversation with access to previous messages.
resume() - Continue Execution
resume() continues from where the agent stopped:
// Continue from last checkpoint
const newHandle = await handle.resume();
// Continue with additional context
const newHandle = await handle.resume({
mode: 'with_message',
message: 'Please continue',
});When you call resume():
- New run begins - Every
resume()is a new run with its own run-start commit, which supersedes the suspended or interrupted run - Stream is preserved - Existing stream chunks remain intact
- State is loaded - Agent state is restored from the checkpoint
- Execution continues - The agent resumes from the checkpoint step
When to Use Each
| Scenario | Method | Reason |
|---|---|---|
| Starting a new conversation | execute() | Fresh start needed |
| User sends new message to completed agent | execute() | New run, new stream |
| Crash recovery | resume() | Continue where left off |
| User interrupted, wants to continue | resume() | Preserve progress |
| Time-travel to earlier state | resume({ mode: 'from_checkpoint' }) | Branch from history |
| Tool confirmation received | resume({ mode: 'with_confirmation' }) | Continue paused run |
| Retry after failure | retry() | Restore checkpoint, re-attempt |
Stream Behavior Difference
The key technical difference is stream handling:
// execute() starts a new turn on the session stream
// - Clients see: the new run's chunks after its run_started (no stream_resync)
// - Earlier turns' chunks stay in the stream
// resume() continues the turn
// - Clients see: existing chunks → new chunks from the checkpoint step
// - A from_checkpoint rewind also writes one stream_resyncThe chat handler reads the stream head before it calls execute() and streams only what follows it, so a new turn never replays the previous turn's chunks.
This distinction matters for frontend integration. If you're building a chat UI:
- New conversation: Use
execute()- the UI should clear and show fresh messages - Reconnecting after disconnect: Use
resume()- the UI should show existing messages and continue
Retrying Failed Agents
Failed agents cannot be resumed with resume(). Use the retry() method for failure recovery.
Why retry() is Separate from resume()
| Aspect | resume() | retry() |
|---|---|---|
| Purpose | Continue interrupted/paused execution | Recover from failure |
| Valid statuses | interrupted, paused | failed |
| Stream behavior | Preserves chunks | Writes a stream_resync |
| State handling | Continues from current | Restores from checkpoint |
retry() Method
const result = await handle.result();
if (result.status === 'failed') {
// Retry the failed run from its last committed step, with a message to
// continue with. (A run that failed on its first step can also be retried
// with no message: its own triggering message is re-sent.)
const retryHandle = await handle.retry({
message: 'Please continue',
});
// Or retry from a specific checkpoint
const retryHandle = await handle.retry({
message: 'Research quantum computing',
checkpointId: 'cpv1-...',
});
}Every retry() is a new run. Its run-start commit rewinds the log to the retry target (StartRun.truncateTo), appends the retried input and creates the run, in one atomic step, on every runtime. A concurrent retry() that loses the run start writes nothing, and so does a stale one: a retry() planned before another entry's run finished (or before the session was aborted) is refused in the commit instead of rewinding that run's work (AgentAlreadyRunningError, or RunStartRejectedError('entry_status_changed'); see Distributed Coordination).
Which checkpoint a retry restores. A failed run ends with a terminal state commit, like every run, so the session's latest checkpoint after a failure is that terminal commit. retry() never restores it. Core planRetry picks the target on every runtime:
- The failed run's last committed step. The run's latest
stepcommit, or a laterappendcommit of that run that carries no assistant message (for example afinishWithresult appended after its step). The retried run continues from there, withoptions.messageas its input (after amaxStepscut-off, no message is needed: see below). - Otherwise, the run's run-start commit. The run failed on its first step. The log is rewound to just before the run's own triggering user message, and the retried run appends that message again, followed by
options.messagewhen you pass one.
The restored customState, stepCount and forced-completion state come from that target. An explicit checkpointId is restored as given: the log is cut to its messageCount, and the retried input is options.message, or else the last user message after that checkpoint.
Message Requirement
Without message, retry() retries with the last user message after the restored target. That message exists only when the run failed on its first step: the rows were rewound to just before the run's own triggering message, so the trigger is re-sent. In every other case there is no user message after the target, and retry() without message throws validation_error (No user message found to retry with. Provide a message in options.):
- the run committed one or more steps before it failed (the target is its last step), unless it failed with
framework_max_steps_exhausted(below); - the session failed before its last entry's run-start commit (for example, the entry failed before it committed). Nothing of that entry reached the log, so the retry keeps the whole log;
- an explicit
checkpointIdhas no user message after it. This includes the failed run's own run-start checkpoint (kindinput): its rows already end at the run's triggering message, so nothing is left after it to re-send. Restore it withmessage. The default retry of a first-step failure rewinds to just before the trigger, so it re-sends the trigger itself.
Pass message when in doubt. On runtime-js, retry() with no message used to throw after a first-step failure; it now re-sends the trigger (CP-58). DBOS used to ignore RetryOptions; it now honours checkpointId and message like every other runtime (CP-30). If the session has no checkpoint at all, retry() restarts fresh from the triggering message.
Temporal: rewinding to a finishWith step's checkpoint deletes the call's result (RM-52)
Temporal commits a step that calls a finishWith tool before it appends that call's result, which lands in a separate append commit. The default retry target is that later commit, but a retry({ checkpointId }) or resume({ mode: 'from_checkpoint' }) to the step's own checkpoint truncates the transcript to it, which leaves the finishWith call's tool_use unpaired. The JS runtime commits the step with its result. On Temporal, do not rewind to the checkpoint of a finishWith step. See RM-52 in the conformance findings.
Retry after a maxSteps cut-off
A run that ran out of maxSteps fails with framework_max_steps_exhausted (see When maxSteps runs out), and its run record says completionReason: 'max_steps'. retry() treats it as unfinished work, not an error:
- it restores the run's last committed step (rule 1 above), keeping its log,
customStateand forced-completion state; - it restarts
stepCountat 0, so the retried run gets the fullmaxStepsbudget, even when the cut-off run was itself a resume that entered with a higher count; - it needs no message: the kept log ends in the last step's tool results, so the model carries on. A
messageyou pass is appended as new input.
This also holds when the cut-off run committed no step. For example, a client tool called on the last allowed step suspends the run; the resume then enters at the cap and is cut off before it calls the model. retry() keeps that resume's entry, including the submitted client-tool result, and continues from it with a full budget and no message.
An explicit checkpointId is still restored as given. This holds on JS, the Cloudflare DO and Cloudflare Workflows. Temporal and DBOS do not fail a cut-off run yet (DI-25), so there retry() refuses it as not failed (FU-CFD1C-DI29-DI25-TEMPORAL / -DBOS).
RetryOptions
| Option | Type | Default | Description |
|---|---|---|---|
checkpointId | string | (see above) | Which checkpoint to restore from. Default: the failed run's last step or its run-start commit. An unknown id throws state_checkpoint_not_found; another session's checkpoint throws state_checkpoint_foreign. Both are rejected before any state change. |
message | string | UserInputMessage[] | (extracted) | Message to retry with. Default: the last user message after the target (the failed run's trigger after a first-step failure). Not needed after a maxSteps cut-off. Required otherwise. |
abortSignal | AbortSignal | - | Abort signal for cancellation |
Concurrency Safety
One run at a time: the run-start commit
Every execute(), resume() and retry() call is an entry that starts a new run. Its run-start commit is the single gate, on every runtime. In one atomic step the store checks that the current run is still the one the entry saw, refuses while another live run owns the session, supersedes the previous run, applies the entry's rewind (if any) and creates the new run with its input rows. A caller that loses writes nothing: no input row, no stream change, no chunk.
A loser gets a typed error:
| Error | name | Meaning |
|---|---|---|
RunStartConflictError (extends AgentAlreadyRunningError) | AgentAlreadyRunningError | A live run owns the session (conflictCause: 'run_active'), or a newer run started since this call read the session ('expected_mismatch'). Wait, or attach. |
RunStartRejectedError | RunStartRejectedError | The commit was refused for another reason (rejectCause: idempotency_mismatch, entry_status_changed (the session changed under the entry: reload, then retry), entry_abandoned). Do not treat it as "already running". |
Both carry code: 'state_run_start_conflict'. A refusal before the commit (for example assertNoLiveOwner on JS) throws a plain AgentAlreadyRunningError, so check with instanceof AgentAlreadyRunningError:
import { AgentAlreadyRunningError } from '@helix-agents/core';
try {
await executor.execute(agent, message, { sessionId });
} catch (error) {
if (error instanceof AgentAlreadyRunningError) {
console.log(`Session ${error.sessionId} is already running`);
// Wait for the existing run to finish, then try again
}
}Who counts as a live owner depends on the runtime: on JS, a run whose lease (JS_RUN_LEASE_MS, 30 s, renewed every 10 s) has not lapsed; on Temporal and DBOS, a run whose workflow is still open; on Cloudflare Workflows, a run whose instance is live; on the Cloudflare DO, the execution in progress in the object. A run whose owner is gone (a crashed process) is superseded by the next entry. See Distributed Coordination.
Superseded runs
When an entry supersedes a run, that run's record turns superseded, and every later write the old executor attempts is fenced: commits, saveState, run status changes, stream chunks once the new run has claimed the stream, and tool executions all fail with RunSupersededError (state_run_superseded, never retried) and write nothing. The old handle's result() settles with status: 'superseded' on every runtime (on runtime-js with errorDetail.code: 'state_run_superseded'). onAgentSuperseded fires for it and onAgentFail never does. onAgentComplete does not either, unless the supersede landed after it had already fired and before the run's terminal commit: then the run fires onAgentComplete then onAgentSuperseded, and the last hook is authoritative (see Hooks).
const result = await handle.result();
if (result.status === 'superseded') {
// Another run took over this session. Its output is in the session, not here.
}A superseded run is not a failure, and the session is not failed because of it. On runtime-js an event loop blocked for longer than about 20 s (two thirds of the lease) can let another entry supersede the run; the stopped run reports superseded.
resume() against a live run is refused on every runtime (CP-83)
The resume CAS accepts active on purpose: a session awaiting client-tool submissions can read active (on Temporal, a session re-suspended on a partial client-tool batch does), and resuming it is correct. A session that reads active with no pending client-tool or approval calls is a live run, and resume() refuses it in every mode with AgentAlreadyRunningError, writing nothing: continue and with_message by that check, from_checkpoint by the rewind liveness gate. DBOS and the Cloudflare DO (409 while executing) always refused; Temporal, Cloudflare Workflows executor.resume() and JS from_checkpoint used to accept the live run and supersede it, and its answer was lost. Cloudflare Workflows handle.resume() had its own copy of the resume path without the check; it now runs executor.resume(), so the same guard applies. See CP-83 in the conformance findings.
Basic Usage
Interrupting an Agent
Use handle.interrupt() to pause execution:
const handle = await executor.execute(agent, 'Research quantum computing');
// Later, interrupt the agent. This resolves once the stop is recorded;
// it does not wait for the run to end.
await handle.interrupt('user_requested');
// The run ends at its next boundary
const result = await handle.result();
console.log(result.status); // 'interrupted'The agent will stop at the next safe point (between steps). Any in-progress step is not committed: it is rolled back to the last checkpoint. A run suspended on client tools is stopped at once.
The rollback truncates only rows past the latest checkpoint, and since every message row is written by a commit, the latest checkpoint counts every committed message: a turn's input is part of its run-start commit, and each step's rows are part of its step commit. An interrupt therefore never deletes the turn's input or an earlier step (RM-38, RM-41, RM-42). A stop that lands before a step's commit makes the commit refuse atomically (CommitStoppedError), and the step is discarded (on DBOS with a step_discarded chunk).
DBOS: an interrupt right after execute() keeps the turn's input (CP-79)
execute() on DBOS now returns its handle only after the workflow's run-start commit, which persists the input together with the run record. An interrupt through that handle, however soon, can no longer end the session interrupted with no user message. Earlier releases persisted the input several durable steps into the workflow, so an interrupt that cancelled the workflow first lost it. The handle's result() after such an interrupt is tracked as CP-57. See CP-79 in the conformance findings.
Durable stop protocol
A stop (interrupt or abort) is a durable record on the run it targets, not an in-memory signal. v6 looked the handle up in an in-memory activeHandles map and returned HTTP 503 INTERRUPT_NOT_LOCAL if the handle lived on a different process. Now every stop goes through core stopRun (WA MR2):
- The stop is recorded on the run (
requestStop). It names that run, so a stale stop never stops a newer run, and nothing clears it. - If an owner is executing the run, it is signalled and the call answers
stopping(HTTP 202). The run ends at its next boundary. - If nothing executes the run (it is suspended on client tools, or its process is gone), the stop's terminal is committed now: the answer is
interrupted/failed(HTTP 200). - With no live run, nothing is written:
not_stoppable(HTTP 409state_not_stoppable).
The call answers at once and never waits for the run to wind down: observe the end on the stream (run_interrupted) or through the status. So POST /chat/{id}/interrupt (or POST /interrupt on the executor route) succeeds from any process, with no sticky routing and no observation deadline (agent-server's interruptObservationDeadlineMs and its 504 are gone). An abort ends the run failed with framework_cancelled; an abort that arrives after an interrupt upgrades it. Handles' interrupt() / abort() resolve once the stop is durable. See the upgrade guide for the answers per runtime.
On the Cloudflare DO, a run that ignores the stop (a tool that ignores its abortSignal) is ended by the stop deadline (stopDeadlineMs, default 30 s). The other runtimes have no deadline yet.
Stopping from the chat UI (CP-12)
A chat client's Stop button reaches the server only through a stop route. Wire one, then point useHelixChat({ serverStop: { url } }) at it.
- The route. A Cloudflare DO host calls the chat handler factory's
stop({ sessionId, kind, reason }). Any other host callshandleChatStop(Express: throughexpressRequestToWebRequest). - Auth. A stop is a write: run the same auth and session-ownership check as the send route (
handleChatStop'sauthorizeis required). - Kinds. Accept only
interruptunless the app means to let a client destroy a turn (allowedKinds). - Only the explicit
stop()posts. Unmounts, page refreshes and resync re-anchors stay local, so a refresh still resumes the run. - Target. The stop targets the session's current live run.
See the useHelixChat and handleChatStop references.
Resuming an Agent
Use handle.resume() to continue execution:
// Check if we can resume
const { canResume, reason } = await handle.canResume();
if (canResume) {
// Resume execution
const newHandle = await handle.resume();
// Stream events from the resumed execution
for await (const chunk of (await newHandle.stream()) ?? []) {
console.log(chunk.type);
}
// Get final result
const result = await newHandle.result();
console.log(result.output);
}Resume Modes
The resume() method supports four modes:
continue (default)
Resume from where the agent stopped:
const newHandle = await handle.resume(); // Same as { mode: 'continue' }
const newHandle = await handle.resume({ mode: 'continue' });continue and with_message reject a session whose run is still executing, on every runtime: an active session with no pending client-tool or approval calls gets AgentAlreadyRunningError, and nothing is written. Before, Temporal and Cloudflare Workflows started a second run on it. An active session with pending calls is suspended and resumes normally. On Cloudflare Workflows handle.resume(options) runs executor.resume(), so it has the same guard and the same run-start commit before any cleanup or truncate (CP-83, CP-35), and it reactivates a stream the interrupt ended (CP-67). See the conformance findings.
with_message
Resume and add a new user message to the conversation:
const newHandle = await handle.resume({
mode: 'with_message',
message: 'Actually, focus on quantum entanglement specifically',
});The message field accepts a string or an array of UserInputMessage objects for multi-message input:
const newHandle = await handle.resume({
mode: 'with_message',
message: [
{
role: 'user',
content: 'Here is additional context from the system',
metadata: { source: 'system' },
},
{ role: 'user', content: 'Focus on quantum entanglement specifically' },
],
});This is useful when the user wants to redirect the agent's focus. The messages are the new run's input rows: they are written in its run-start commit, so they are never lost to an interrupt before the first step commits.
with_confirmation
Resume a paused tool call with confirmation data:
// Agent paused waiting for confirmation on a tool
const newHandle = await handle.resume({
mode: 'with_confirmation',
data: { approved: true, notes: 'Proceed with the action' },
});This mode is used when a tool requires human-in-the-loop approval. runtime-js and Cloudflare Workflows executor.resume() throw Resume mode "with_confirmation" is not yet implemented; submit the approval with submitToolResult and resume with continue instead (see Approval gates). The Cloudflare Durable Object's /resume body does not offer it at all (it accepts continue, with_message and from_checkpoint only; anything else is a 400 validation_error).
from_checkpoint
Resume from a specific historical checkpoint (time-travel):
// Get available checkpoints
const state = await handle.getState();
const checkpoints = await stateStore.listCheckpoints(handle.sessionId);
// Resume from an earlier checkpoint
const newHandle = await handle.resume({
mode: 'from_checkpoint',
checkpointId: checkpoints.checkpoints[0].id,
});from_checkpoint rewinds both what the model sees and the durable log, on every runtime:
- The message log is truncated to the checkpoint's
messageCount, and the checkpoint's state (customState,stepCount) is restored on this session. The next model call sees exactly the history up to the checkpoint, plus any new input. - The checkpoint must belong to this session. An unknown id throws
state_checkpoint_not_found; a checkpoint from another session throwsstate_checkpoint_foreign. Nothing changes in either case. - Pending client-tool or approval calls made after the checkpoint are discarded. Each gets a
tool_endchunk witherrorCode: 'aborted'on the stream, so your UI can settle it, and a latersubmitToolResultfor one returns{ status: 'unknown_tool_call' }. A client tool that was pending at the checkpoint and answered afterwards is pending again after the rewind if the store recorded it on the checkpoint; otherwise the next continuation closes it with a not-executed result, so the transcript stays valid. run_resumed.fromCheckpointIdand theonAgentResumedhook'sresumedFromCheckpointIdreport the target checkpoint.- A session suspended on a client tool can be rewound too. On DBOS, the suspended workflow is cancelled and a fresh one starts.
- Concurrent rewinds: at most one rewind and one run, on every runtime. The truncation rides the new run's run-start commit (
StartRun.truncateTo), so the rewind and the run it starts land together or not at all. That includes a rewind to the checkpoint the session is suspended at, where the pending call is kept and re-stamped to the new run inside the same commit.- Every other concurrent
resume()(a rewind, acontinueor awith_message) loses the run-start commit, getsAgentAlreadyRunningErrorand writes nothing (on DBOS, it may already have cancelled the suspended workflow, which the winner cancels too). - A failure before the commit changes nothing. If the entry fails after its commit (for example its stream claim fails), the new run and the session end
failedwith a typederrorDetail, and the stream is failed (RunEntryFailedError).
- Every other concurrent
- A session whose run is still executing (
activewith no pending client-tool or approval calls, for example blocked in a long model call) cannot be rewound:from_checkpointthrowsAgentAlreadyRunningErrorand changes nothing. Interrupt the run first, or wait for it to finish or suspend.
A completed or failed session is rewound too, on every runtime (time travel): the log is truncated to the checkpoint, the old output / error cleared, and a new run starts from there. (Before v0.50, Cloudflare Workflows and DBOS threw and Temporal returned the old terminal result; see the migration guide.) On the Cloudflare Durable Object host, the /resume body { agentType, options: { mode: 'from_checkpoint', checkpointId } } is honoured, as are continue and with_message (CP-34).
Resuming after HITL suspension (v7)
When a run reaches a client-tool boundary or an approval gate, the run loop suspends durably and handle.result() resolves with one of the suspended_* statuses. To continue:
const result = await handle.result();
switch (result.status) {
case 'suspended_client_tool':
// The client tool / approval gate is now pending submission.
// Submit results via executor.submitToolResult({ kind, ... })
// (or via chat.addToolOutput / chat.addToolApprovalResponse on the
// AI SDK side), then resume:
const resumed = await executor.resume({ sessionId: handle.sessionId });
await resumed.result();
break;
case 'suspended_awaiting_children':
// Parent is waiting for sub-agent(s) to finish their own HITL
// cycles. Once each child resumes and completes, the framework
// cascades the resume up to the parent automatically.
break;
case 'suspended_step_partial':
// Mid-step suspend — some tools in the step ran, at least one is
// pending submission. Submit + resume same as suspended_client_tool.
break;
case 'completed':
/* handle output */ break;
case 'failed':
/* handle error */ break;
case 'interrupted':
/* user-requested pause */ break;
}For chat-driven UIs, handleChatStream (the canonical chat handler in @helix-agents/ai-sdk) drives this lifecycle for you — submission + resume happens automatically on the server side, and AI SDK v6's stream-close-and-reopen lifecycle reattaches the client.
A partly answered batch is not resumable
When one step suspends on several client-tool or approval calls, resume() in continue or with_message mode waits for the whole batch. While any call of the batch still has no submitted result (and its deadline has not lapsed), every runtime refuses the resume with a retryable HelixError state_not_resumable and writes nothing. On runtime-js and the Cloudflare DO it is a ResumeRefusedError (sessionStatus: 'paused', awaitingToolCallIds) whose message names the awaited calls first, so they survive a typed HTTP body's 256-character scrub: Awaiting client tool call(s) <ids>: submit every result of the batch, then resume session '<id>'. The DO answers it 409 { error, code: 'state_not_resumable' }. Temporal, DBOS and Cloudflare Workflows throw a plain HelixError with the older wording (Cannot continue session '<id>': still awaiting results for client tool call(s) <ids>. …) and no awaitingToolCallIds. This is a behaviour change: DBOS used to attach to or continue the run, and a with_message resume could silently drop the message.
Use one of these instead:
- Submit every result of the batch. The run continues once the last one lands (on the Cloudflare DO and DBOS automatically; elsewhere, call
resume()then). - Stop the batch: call
interrupt(), which ends a paused run at once. Acontinueresume of a run whose stop is recorded but not yet resolved (for example, recorded by another process) is not refused either: it ends the run by that stop (interrupted, orfailedwithframework_cancelledfor an abort) with no model call.resume()reports that outcome on runtime-js, Cloudflare Workflows, Temporal and DBOS (on DBOS the parked workflow is cancelled first; on Cloudflare Workflows before any owed__submit__continuation is adopted, so none is created); the Cloudflare DO's/resumeanswers 409state_not_resumable("the suspended run was stopped") and settles the run (onRunSettledonce). Awith_messageresume over such a run settles it the same way and then runs as a new turn (Ruling 57), on every runtime; after an ABORT the settled session isfailed, so thewith_messageis refusedstate_not_resumable(retry()is the path). A new turn (execute()) over such a run settles it the same way and then runs (Ruling 57a), after an abort too, since afailedsession takes its next turn throughexecute(). This holds on every runtime, standard DBOS sessions included (DBOS persistent mode: FU-CP244-DBOS-PERSISTENT-MODE). Without a recorded stop,execute()over a session paused on client calls is still refused (AgentAlreadyRunningError). - Answer the rest with errors: submit an error result for each call you want to abandon, then resume.
A from_checkpoint resume is not refused. When the checkpoint keeps the unanswered calls, the new run takes them over and suspends again at once; when it discards them, it continues. See Client-executed tools for the submit side.
Crash Recovery
If your process crashes while an agent is running, the state is preserved in the state store. On restart, you can resume:
// After restart, reconnect to the session
const handle = await executor.getHandle(agent, savedSessionId);
if (handle) {
const { canResume, reason } = await handle.canResume();
if (canResume) {
// Resume from last checkpoint
const resumed = await handle.resume();
const result = await resumed.result();
}
}For this to work, use a persistent state store like Redis:
import { RedisStateStore, RedisStreamManager } from '@helix-agents/store-redis';
const stateStore = new RedisStateStore({ host: 'localhost' });
const streamManager = new RedisStreamManager({ host: 'localhost' });
const executor = new JSAgentExecutor(stateStore, streamManager, llmAdapter);What Happens During Resume
When you call resume() on a crashed or interrupted agent, the framework performs cleanup before continuing:
- Load checkpoint - Get the latest checkpoint (for
from_checkpoint, the target checkpoint — see from_checkpoint) - Run-start commit - One atomic commit truncates the messages past the checkpoint (for
from_checkpointalways; for acontinueon runtime-js and the Cloudflare DO, any rows past the latest checkpoint), appends awith_messageinput, supersedes the previous run and creates the new one. A caller that loses it changes nothing. - Make the stream live - The winner claims the session stream, reactivates it and writes
run_started - Clean up orphaned staging - Remove any uncommitted staged changes
- Clean up stream chunks (runtime-js, both Cloudflare runtimes and DBOS; Temporal does not clean stream chunks) - Remove the previous run's chunks after the checkpoint step
- Emit
stream_resync- Notify connected clients of the discontinuity: always after afrom_checkpointrewind (rollback) and aretry()(retry), on every runtime; after a plain resume only when step 5 removed chunks (crash_recovery) - Continue execution - Resume from the checkpoint step
graph LR
subgraph Before ["Before Crash"]
direction LR
S1["Step 1"] --> S2["Step 2"] --> S3["Step 3"] --> S4["Step 4 ✗"]
end
S3 -. "Checkpoint" .-> S3
subgraph After ["After Resume"]
direction LR
R1["Step 1"] --> R2["Step 2"] --> R3["Step 3"] --> R4["Step 4"] --> R5["Step 5"]
end
R3 -. "Resume from<br/>checkpoint" .-> R3A step's messages are written by its own step commit, together with its checkpoint, so a crash in step 4 leaves no rows of step 4 behind on any runtime. The truncation is a guard for rows past the latest checkpoint, which only a pre-upgrade session can still hold; it removes them so they are not duplicated when the step re-executes.
Message Count Protection
The framework only truncates messages when a checkpoint exists AND has messageCount > 0. This prevents data loss in edge cases like:
- First-step crash (no checkpoint yet)
- Old checkpoints from before
messageCountfield was added
A run that finished just before the crash
If a process dies after a run's terminal commit landed but before its run record was marked terminal, the session already reads completed / failed / interrupted while the run still reads as a live run (running or suspended_*). Recovery does not re-run it: once its owner is gone, the run is ended with the session's committed status and its stream ended or failed (core finishCommittedRun; the Cloudflare DO wake and CF Workflows' readers reach it through core's closed-owner reconciliation, planClosedOwnerReconciliation → reconcileClosedOwnerRun). At an entry, StartRun.finishCurrent ends it in the run-start commit, whether it reads running or suspended_*, even while the old owner is still alive and its lease is valid (it has nothing left to write but its tail; its own later status write is a no-op). So a stop on a paused run whose K5 (interrupted) landed just before the crash is recorded interrupted by the next turn, never superseded.
Crash recovery of a still-running run is different from a resume. A resume() is an entry: it starts a new run with its own run-start commit and supersedes the stranded one once its owner is gone. The Cloudflare DO's wake path instead calls runtime-js's internal executor.recover(agent, sessionId, { runId }), which writes no run-start commit and re-enters the loop under the same runId; it refuses with RunSupersededError unless that run is still the session's current running run, and it finishes the run instead when its terminal commit already landed.
Resumable Statuses
An agent can be resumed when its status is:
| Status | Resumable | Description |
|---|---|---|
running | Yes¹ | Crash recovery — process died but state says running. |
interrupted | Yes | User interrupted execution. |
paused | Yes | Waiting for tool confirmation (legacy v6 path). |
suspended_client_tool (v7) | Yes | One or more client-tool / approval calls are pending submission. |
suspended_awaiting_children (v7) | Yes | One or more sub-agents are awaiting submissions; parent waits for cascade. |
suspended_step_partial (v7) | Yes | A step finished some tools but at least one is suspended; mid-step retry. |
completed | No* | Agent finished successfully. |
failed | No* | Agent failed with error. |
¹ Only once the run's owner is gone (on JS, its lease has lapsed; on Temporal and DBOS, its workflow is closed). While the owner is live, resume() throws AgentAlreadyRunningError.
*resume() does not apply to terminal states — but a completed run is not a dead end. You can continue it as a new turn with execute({ sessionId }) (see Continuing a completed agent below), or time-travel into either terminal state with resume({ mode: 'from_checkpoint' }) on every runtime (see from_checkpoint).
The three new suspended_* statuses carry a result.suspended payload with the routing info (toolCallIds, children, stepId) the chat handler uses to drive resume. Most consumers do not read the payload directly — handleChatStream (from @helix-agents/ai-sdk) does it on their behalf — but it is part of the public type surface and exhaustive switches need to handle the variants.
Continuing a completed agent
resume() is for suspended runs (interrupted, paused, or a suspended_* HITL boundary). A completed run is continued differently: call execute({ sessionId }) against the completed session to start a new turn — it preserves the session's memory and produces a fresh result, with a fresh per-turn maxSteps budget.
// First turn runs to completion
const h1 = await executor.execute(agent, 'Draft a release note', { sessionId: 'rel-1' });
await h1.result(); // status: 'completed'
// Continue the SAME session as a new turn — memory is retained.
const h2 = await executor.execute(agent, 'Tighten the wording', { sessionId: 'rel-1' });
const r2 = await h2.result(); // fresh typed result, full prior history in contextA completed status can arrive before the previous run lets go (CP-84)
A run publishes completed in its terminal commit, then marks its run record terminal and ends its stream. Until the run record is terminal the finished run still owns the session, so an execute() for the next turn in that gap loses the run-start commit and is refused with AgentAlreadyRunningError. On runtime-js the refusal holds while the finished run's lease is valid; on Temporal, DBOS and Cloudflare Workflows while its workflow or instance is open. Through handleChatStream a refused send answers a retryable 409 { code: 'state_already_running' }. Retry after a short wait, or await result() on the previous handle, which resolves past the gap. See CP-84 in the conformance findings.
Fixed: the previous run's stream end no longer cuts the new turn (CP-85)
endStream, failStream and pauseStream are fenced on the stream's owner run. A finished run's late stream end is dropped once a newer run has claimed the stream, so it can no longer end the next turn's stream. Before, on JS and DBOS, the new turn failed with Cannot write to stream … in 'ended' state or streamed none of its output.
This now works for structured-output agents too. Previously a completed structured-output agent could not be continued on a real LLM provider: its transcript ended with a dangling __finish__ tool_use that had no matching tool_result, which Anthropic/OpenAI reject. The finish tool call is now paired with a synthetic { acknowledged: true } tool result, so the history is valid for continuation. See Finishing Agents — Continuing a completed structured-output agent.
The cross-runtime contract that execute(sessionId) is valid from completed is documented in the agent lifecycle table — see also the Agent Lifecycle Methods table in Core Concepts. The same continuation mechanism powers persistent companions: re-consulting a completed critic continues it on its preserved session. See the critic-loop recipe.
Stream Events
The framework emits stream chunks for interrupt/resume events:
run_interrupted
Emitted when an agent is interrupted:
{
type: 'run_interrupted',
runId: 'run-123',
checkpointId: 'cpv1-run-123-s5-t1234567890-abc123',
reason: 'user_requested'
}run_resumed
Emitted when an agent resumes:
{
type: 'run_resumed',
runId: 'run-123',
fromCheckpointId: 'cpv1-run-123-s5-t1234567890-abc123',
fromStepCount: 5,
mode: 'continue'
}run_paused
Emitted when an agent pauses for confirmation:
{
type: 'run_paused',
runId: 'run-123',
reason: 'tool_confirmation_required',
pendingToolName: 'delete_file',
pendingToolCallId: 'call-456'
}checkpoint_created
Defined in the stream schema, but not written by any runtime today. Read checkpoints with stateStore.getLatestCheckpoint(sessionId) / listCheckpoints(sessionId). The chunk shape is:
{
type: 'checkpoint_created',
runId: 'run-123',
checkpointId: 'cpv1-run-123-s5-t1234567890-abc123',
stepCount: 5
}Error Handling
AgentAlreadyRunningError
resume() throws this in every mode (continue, with_message and from_checkpoint), on every runtime, in two cases:
- The run is still executing: the session is
activewith no pending client-tool or approval calls, or its current run still has a live owner (see One run at a time). - A concurrent entry won: this call lost the run-start commit to it (
RunStartConflictError, whosenameisAgentAlreadyRunningError).
Nothing is written in either case. An active session that does have pending calls is suspended, not running, and resumes normally (once its whole batch is answered, see A partly answered batch is not resumable).
import { AgentAlreadyRunningError } from '@helix-agents/core';
try {
await handle.resume();
} catch (error) {
if (error instanceof AgentAlreadyRunningError) {
console.log(`Agent session ${error.sessionId} is already running`);
}
}AgentNotResumableError
Thrown when trying to resume an agent in a terminal state. It is a typed HelixError (code: 'state_not_resumable', never retryable; an optional hint is appended to the message):
import { AgentNotResumableError } from '@helix-agents/core';
try {
await handle.resume();
} catch (error) {
if (error instanceof AgentNotResumableError) {
console.log(`Cannot resume: ${error.currentStatus}`);
}
}Terminal-session refusals differ per runtime (DI-31)
A running session is refused with AgentAlreadyRunningError in every mode. A completed or failed session (one an abort stop ended included) is refused:
- on Cloudflare Workflows and DBOS, in
continueorwith_messagemode, withAgentNotResumableError(code: 'state_not_resumable', non-retryable; the hint says to send the message withexecute()as the next turn, or toretry()a failed session); - on Temporal, in
with_messagemode, withAgentNotResumableErrortoo; acontinuethere returns the previous result instead (CP-60); - on runtime-js and the Cloudflare DO, in
continueorwith_messagemode, with a typedResumeRefusedError(code: 'state_not_resumable',sessionStatus, non-retryable).
Match them on HelixError.isInstance(error) && error.code === 'state_not_resumable', which covers both classes. See DI-31 in the conformance findings.
Completion-support refusal
An outputSchema agent whose model has no completion strategy is refused at every entry, resume() and retry() included, before its run-start commit (on runtime-js before any store read or write): framework_completion_strategy_unavailable (for example an Anthropic model id the adapter's capability table does not know) or framework_completion_schema_unsupported (a completion schema that native structured output cannot carry). Both are non-retryable HelixErrors; the session is left exactly as it was, so the same resume() / retry() succeeds once the adapter declares the model (the capabilities override). On the Cloudflare DO the HTTP routes answer 400 { error, code }. On Temporal the refusal comes from the worker's startRunCommit activity, before the run-start commit, and the executor rethrows it typed. DBOS does not refuse yet.
A pending forced-completion phase survives resume(), retry() and branching unchanged, and its next forced call resolves the strategy again against the adapter it now runs on: the strategy is never stored, so a session started on one model or adapter version can be resumed on another.
Runtime Considerations
JavaScript Runtime
The JS runtime handles a stop through its run registry. When you call interrupt():
- The stop is recorded on the run, and the run's controller is aborted with it
- At its next boundary (
pollStop) the run ends: the in-flight step is not committed (its staged changes are discarded) - The session is committed
interrupted(the K5 terminal commit) - The run loop exits cleanly
A run another process executes (a live foreign lease) gets the record only and ends at its next boundary; a run with no live owner is ended by the stop call itself.
Temporal Runtime
Temporal records the stop and wakes the workflow with a signal:
interrupt()records the stop on the run, then signals the workflow (INTERRUPT_SIGNAL_NAME, carrying theStopRequest)- The workflow reads the run's stop record (
readStopActivity) at its next boundary and ends the run - State is persisted via Temporal's durability guarantees
- Resume creates a new workflow execution (
…__resume-<runId>, one per call) that continues from the checkpoint
A resume's run-start entry makes the session stream live again (claim, reactivate, run_started) before the resumed run writes, so the stream an interrupt ended accepts the resumed run's chunks. Earlier releases did not reactivate it (CP-76; see the Temporal runtime page).
Temporal: a failed resume() result can arrive before the failure is stored (CP-80)
The handle executor.resume() returns settles result() on the first error chunk, which the LLM activity writes before the workflow persists the failed status. When a resumed run (for example after a client-tool submission) ends on a terminal LLM error, result() can report failed while the stored session still reads active, and a retry() issued straight away throws Cannot retry: agent status is 'running'. Wait until the session reads failed before calling retry(). This was observed live with no injected delay, and the store caught up within about 1.5 s. A completed resume and the execute() / retry() handles are not affected. See CP-80 in the conformance findings.
Cloudflare Runtime
Cloudflare Workflows handle interrupts via instance coordination:
interrupt()records the stop on the run and wakes the open instance with an event- The workflow reads the run's stop record in a
step.doat its next boundary and ends the run - Resume creates a new workflow instance (
…__resume__<key>, a key minted by that call) with a new run id,${instanceId}:${agentType}-run-start:0. Instance ids are write-once, so everyexecute(),resume()andretry()runs in its own instance;getHandle()finds the live one from the session's current run record.
Interrupt Behavior During Sub-Agent Execution
When an agent spawns sub-agents (child workflows), interrupt signals propagate through the entire agent hierarchy. This enables responsive cancellation even during complex multi-agent operations.
How Interrupt Propagation Works
When you interrupt a parent agent that has running sub-agents:
- Parent receives the stop - The stop is recorded on the parent's run and its owner is signalled
- Children are stopped - The parent's stop reaches its running children (per runtime, below)
- Children stop gracefully - Each child ends at its next safe point
- Parent ends - The parent ends
interrupted(orfailedfor an abort)
Target Response Time: < 1 second from interrupt request to stopped execution, even with deeply nested sub-agents.
Per-Runtime Implementation
| Runtime | Mechanism | Latency |
|---|---|---|
| JS | The stop record + the run's aborted signal; in-process children are stopped with it | Immediate (< 100ms) |
| Temporal | The stop record (readStopActivity) + the INTERRUPT_SIGNAL_NAME wake-up | < 1 step boundary |
| Cloudflare DO | The stop record + the drive's aborted signal; children stopped through their routes¹ | < 1 step boundary |
| Cloudflare Workflows | The stop record (read in a step.do, and polled while a step runs) + an event wake-up² | Mid-step (~500 ms) |
| DBOS | The stop record (readStopStep) + DBOS.cancelWorkflow | < 1 step boundary |
² An in-flight model call, tool or hook gets an abortSignal that fires within stopPollIntervalMs (default 500 ms) of the stop, and a run parked on a sub-agent or a blocking companion wakes on the stop and propagates it to that child.
¹ The DO's /interrupt (and EndpointContext.stop) records the stop, signals the executing drive and answers at once (202); it never waits for the run. A run that ignores its abortSignal is ended by the stop deadline (stopDeadlineMs, default 30 s). A new /start, /resume or /retry while a run executes is refused 409 state_already_running by default; set concurrentEntry: 'interrupt_previous' to have the new entry stop the live run (after its own gates and beforeEntry pass) and wait for it to settle, bounded by the stop's deadline plus 5 s. This replaces calling executionState.interrupt() from a beforeStart hook. A chat send (createCloudflareChatHandler) is refused 409 while a run executes even under interrupt_previous (CP-13), until chat sends carry a request id: stop the run first, then send. See the DO drive pipeline upgrade guide.
JavaScript Runtime
The JS runtime uses AbortSignal for cooperative cancellation:
// Parent's AbortController is linked to child agents
// When parent.abort() is called, children receive abort signal immediately
// In custom tools, check the signal:
const myTool = defineTool({
execute: async (input, context) => {
for (const item of items) {
if (context.abortSignal.aborted) {
return { partial: true, processed: results };
}
// ... process item
}
},
});Temporal Runtime
handle.interrupt() records the stop on the run and signals its workflow (INTERRUPT_SIGNAL_NAME); a stop recorded from any process works the same way. The workflow reads the run's stop record before each step's LLM call and at a resume's entry, never while a step's LLM call, tools or sub-agent children are in flight, so they finish first. Child workflows are not signaled: a sub-agent child runs to its own end, and a non-blocking persistent companion keeps running. See Temporal → Interrupts and sub-agents.
Cloudflare Runtime
Cloudflare Workflows records the stop on the run and wakes the instance with an event:
- The stop is recorded on the run (
requestStop) - An interrupt event is sent to the open instance (
instance.sendEvent()) for an immediate wake-up - The workflow reads the run's stop record in a
step.dobefore each step and during a sub-agent wait - On a stop during a sub-agent wait, the stop is recorded on each pending child's run, and the parent ends by it
A paused run, or one whose instance already closed, is ended by the stop call itself.
Best Practices for Sub-Agent Interrupts
- Keep sub-agent operations bounded - Long-running sub-agents delay interrupt response
- Use interruptible tools - Check
context.abortSignal.abortedin loops - Configure poll intervals appropriately - Shorter intervals = faster response, more overhead
- Design for partial results - Sub-agents should return meaningful partial output when interrupted
Best Practices
1. Always Check canResume()
Before resuming, verify the agent can be resumed. canResume() is only a read: a concurrent caller can still win the run start, so handle AgentAlreadyRunningError as well (below).
const { canResume, reason } = await handle.canResume();
if (!canResume) {
console.log(`Cannot resume: ${reason}`);
return;
}2. Handle Resume Errors
Wrap resume calls in try-catch:
try {
const resumed = await handle.resume();
await resumed.result();
} catch (error) {
if (error instanceof AgentAlreadyRunningError) {
// Another process resumed first
} else if (error instanceof AgentNotResumableError) {
// Agent is in terminal state
}
}A completed or failed session is refused with AgentNotResumableError only on Cloudflare Workflows; runtime-js and the Cloudflare DO throw ResumeRefusedError (also state_not_resumable), and DBOS a plain Error (DI-31; see Error Handling), so match on the code and also handle an untyped error.
3. Use Persistent Storage for Production
For crash recovery to work, use Redis or another persistent store:
const stateStore = new RedisStateStore({
host: process.env.REDIS_HOST,
ttl: 86400 * 7, // 7 days retention
});4. Store Session IDs for Later Resumption
Save session IDs so you can reconnect after crashes:
const handle = await executor.execute(agent, input);
// Persist the session ID
await database.save({ sessionId: handle.sessionId, userId });
// Later, after restart
const savedSession = await database.get(userId);
const handle = await executor.getHandle(agent, savedSession.sessionId);Next Steps
- Client-executed tools — HITL primitive that triggers
suspended_client_tool - Approval gates — first-class
requireApprovalflag - v6 to v7 migration guide — full v7 changes
- Checkpoints - Understand the checkpoint system
- Distributed Coordination - Multi-process coordination
- Streaming - Handle interrupt/resume stream events