Helix Agents Framework Concepts
This document is the canonical reference for Helix Agents framework concepts. It describes current behavior (post-A.2 / A.3); for the v6 → v7 migration delta, see ../upgrade-guides/v6-to-v7-stateless-suspension.md.
For agents: Cross-runtime work (HITL, sub-agents, state store semantics) usually requires fetching this file in addition to the top-level
CLAUDE.md. The pointer table inCLAUDE.mdlists this file under "framework concepts deep-dive."
Session — The primary unit of agent conversation state. A session contains all messages, custom state, and checkpoints for a conversation. Sessions are identified by sessionId, which is the primary key for all state operations. Multiple runs can occur within a single session (e.g., after interrupts or when continuing a conversation).
Run — A single execution within a session. Every entry into running (execute(), every resume() mode, retry(), a sub-agent spawn, a companion continuation, a DBOS submit wake) creates a new run with a unique runId inside a run-start commit: the run record, its input rows and an input checkpoint land atomically, and the commit's rules, in order (validation, the abandoned-entry tombstone, idempotency, a CAS on the current run's id and then its status, the owner guard, the session-status gate, then supersede; see Entry gates (CP-96)), leave at most one live run per session and commit an entry only while the session and run it planned from are unchanged. Recovery of an in-flight run keeps its run. Runs track turn number, step count, status, timing, startSequence (the stream head at the run-start commit; the run's run_started is above it), startMessageCount / startUIMessageCount, and the owner fields (ownerToken, the JS leaseUntil). Run statuses are live (running, suspended_*) or terminal (completed, failed, interrupted, superseded), and terminal statuses are sticky. Use getCurrentRun(sessionId) to get the current run and listRuns(sessionId) to see run history. See ./session-model.md §Runs. As of v7, AgentResult.status may also be 'suspended_client_tool' | 'suspended_awaiting_children' | 'suspended_step_partial' for HITL agents that paused mid-run, and (breaking, pre-1.0) 'superseded' for a run another entry took over (see Superseded runs); exhaustive switch statements must handle these cases. The result.suspended field carries the routing info (toolCallIds, children, stepId) needed to drive resume.
Multi-turn continuation — executor.execute(agent, msg, { sessionId }) is portable for both fresh sessions and follow-up turns. Calling execute() again with the same sessionId after the prior run reached completed starts a new run/turn that sees the full prior conversation. A still-running session rejects with AgentAlreadyRunningError (single-writer mutex). The portable contract floor is completed: continuation from interrupted / failed / paused is not part of the cross-runtime guarantee — use resume() / retry() instead. runtime-js additionally accepts a broader prior-state set on execute(); that is a JS-specific superset, not a cross-runtime promise. Per-runtime mechanics (GL #74): runtime-js / runtime-dbos create a new run within the session; runtime-temporal reuses the sessionId-derived workflow id (Temporal's default reuse policy starts the new run once the prior closes); CF-DO inherits the JS executor's continuation path; CF Workflows writes only a write-once createSession for a new session before creating the instance, passes the body an explicit entry: 'new' | 'continue', and runs every turn in its own write-once instance (…__turn__<key>, keyed on a key minted by that call, never a counter, CP-88/CP-89); a clean paused session's execute() starts a new turn. On every runtime the new turn's run-start commit is the gate, and only its winner makes the ended stream live again (after its commit and stream claim, before its run_started); no executor touches the stream before that. The HTTP path uses this via POST /chat → the consumer's chatHandler (ai-sdk handle-chat-stream "continuing session" path → execute(sessionId)); agent-server's /start (startAgent) is fresh-start-only by design (remote-sub-agent protocol). On CF Workflows getHandle() resolves the live turn / resume / retry / rewind instance, and every instance id and event type is built by buildWorkflowInstanceId / buildWorkflowEventType (engine-valid: ≤ 100 chars of [A-Za-z0-9_-], CP-90).
Agent — Created with defineAgent(). Has a system prompt, tools, output schema (Zod), LLM config, and max steps. The outputSchema auto-injects a __finish__ tool for structured output.
Agent registry replace API. Runtimes that resolve agents BY NAME (Temporal, Cloudflare Workflows) provide a AgentRegistry.replace(config) method for swapping the registered reference — used most commonly in tests that need per-call hooks on the same agent type. JS and DBOS read the agent reference inline and don't need this API. See ../runtimes/temporal.md for the full API description.
Tool — Created with defineTool(). Has a name, description, Zod parameters schema, and execute function. Tools receive a ToolContext with getState(), updateState() (Immer), emit(), and abortSignal.
A tool-call id is unique within its session (tree client calls T0). A provider that repeats a tool-call id (twice in one response, or once per step: call_0 every step) has the repeat stored as <id>_<n>, the smallest free n. Core createCallIdRewriter(history) rewrites it at the step's stream boundary, before anything sees it: the argument-stream frames (wrapCallbacks), the assistant row and the results, hooks and child refs (rewriteResult) all carry the same id; a response whose ids are free comes back unchanged. One rewriter per model attempt, built from that attempt's committed history only, so a retried, replayed or memoized step gets the same ids. Wired at every LLM call site: core's step iterator (runtime-js, the CF Durable Object), CF Workflows' executeAgentStep, Temporal's runLLMStep, DBOS's callLLMStep. A model that re-emits an id already in the log therefore makes a NEW call; a recovery re-executes a committed call through its runtime's own re-entry (runtime-js / the DO: CP-106's child-phase re-entry; CF Workflows / Temporal / DBOS: their journals), never by calling the model again.
Tool Execution Order — When the LLM returns multiple tool calls in one response, they execute in two phases: (1) regular tools run in parallel, their state changes become visible (in-memory for JS runtime, committed to store for Temporal/Cloudflare); (2) finishWith tools run sequentially, seeing the updated state from phase 1. This ensures finishWith tools always see the complete state from all other tools in the batch. In the JS runtime (and the CF DO), sub-agents execute alongside regular tools in phase 1. In Temporal, sub-agents execute after phase 2 as child workflows. In Cloudflare Workflows, sub-agent, remote and companion calls run as child instances BEFORE phase 2, so a finishWith tool sees what they did, as on JS; its results are append commits after the step commit (CP-63). Sub-agents always execute even when a finishWith tool succeeds — completion is deferred until sub-agents finish. Companion tools have their own phase (existing behavior, unchanged). A __finish__ beside other calls is honoured in the same place as phase 2, after every other call of the step ran (spec 2026-11-03). v7 wraps this same phase logic inside the new runStepIterator (the step-iterator emits per-phase StepOutcomes), but the observable execution order is unchanged.
Sub-Agents (Ephemeral) — Created with createSubAgentTool(). Parent agents can delegate to child agents. The child runs to completion and returns the result as a tool result. Child sessions are isolated but stream events flow to the parent's stream. Each sub-agent gets its own sessionId. This is the default mode (mode: 'ephemeral' on SubSessionRef). Dispatch failures (failed createSession, child workflow start failure, DO instance creation error) surface to the LLM via tool_error chunk + synthetic tool result message + subagent.dispatch_failed ERROR log. Consistent across runtime-js, runtime-temporal, and runtime-cloudflare DO; parent execution continues so siblings in the same dispatch batch are unaffected.
Persistent Sub-Agents — Configured via persistentAgents on AgentConfig. Unlike ephemeral sub-agents, persistent children can receive follow-up messages and maintain state across multiple interactions. Two modes: blocking (parent waits for child to complete) and non-blocking (parent continues immediately, gets completion notification later). Persistent children are managed through auto-injected companion tools, not createSubAgentTool(). Each persistent child gets a stable session ID: {parentSessionId}-agent-{name}. Children can be named explicitly or auto-named ({agentType}-{counter}). State tracking uses SubSessionRef with mode: 'persistent' and name fields. D1StateStore requires V4 migration for the mode and name columns on __agents_sub_session_refs.
Companion Tools — Auto-injected into parent agents that have persistentAgents configured. Up to six tools prefixed with companion__: spawnAgent (create and start a child), sendMessage (interrupt/resume a running child with a new message; on Temporal and Cloudflare Workflows the interrupt ends the child's run without processing the message — CP-41), listChildren (list all persistent children), getChildStatus (check a specific child's status and output), terminateChild (kill a running child) — these five are always injected. The sixth tool, waitForResult (blocks until a child completes; a positive whole-number timeout in ms that passes first answers { name, status: 'timeout', currentStatus }, core companionWaitResultSchema(); on CF Workflows an untimed wait answers that after subAgentEventTimeout), is conditionally injected only when at least one persistent agent has mode: 'blocking' configured. Companion tools are handled separately from regular tools in the execution loop.
Remote Sub-Agents — Created with createRemoteSubAgentTool(). Parent agents can delegate to agents running on a separate HTTP service. Uses HttpRemoteAgentTransport for communication via HTTP + SSE. The remote service hosts agents using AgentServer from @helix-agents/agent-server. Remote sub-agents are first-class constructs across all runtimes (JS, Temporal, Cloudflare) with stream proxying, SubSessionRef tracking with remote metadata, deterministic session IDs, crash recovery via transport.getStatus(), and interrupt propagation via transport.interrupt(). A child whose status the parent cannot read on a resume or recovery settles failed with a typed, non-retryable framework_subagent_lost result when its producer does not know the session (cause: 'not_found'), refuses the read typed (refused:<code>), or stays unreadable past the child's own timeoutMs plus a minute (status_unreachable); any other failure is retried on the next reconcile (S-C). An entry that carries a new message settles its remote children first: a finished or lost child's result is committed before the message, a still-running child makes a resume / retry with a message refuse busy (state_already_running), and a new turn over an ended run stops it (its call keeps the heal's unpaired_on_continuation result). See the S-C upgrade notes. Each runtime routes remote sub-agent calls through a dedicated execution path separate from regular tool calls. A remote child that pauses (CP-157 L1) — on a client-executed or approval-gated call anywhere in its tree — PARKS its parent on runtime-js, the CF Durable Object and CF Workflows when the root can answer the call (core planRemotePause: the child's /status lists its tree's unanswered calls in pendingCalls, untruncated, and the transport implements submitToolResult). The parent suspends suspended_awaiting_children on a childKind: 'remote' wait, the child's ref turns paused_awaiting_client, and the calls are recorded in the ROOT's remotePendingCalls (never pendingClientToolCalls; in the suspension commit, or a fenced root write first for a nested parent). The root's snapshot renders them, its submit router forwards an answer to the producer (submitToolResult, then a keyed resume), and the resumed parent re-attaches the recorded child (no second /start) and pairs its real output; a child that pauses again on a NEW call re-parks. A remote producer that parks on its own remote child records the calls on itself and forwards an answer for one of them to that child: it carries its consumer's rootSessionId, which is never a local session. The spin bound: a re-sent answer the child already has (already_completed) while its /status still lists that call continues nothing (consumed, logged client_tool.remote_answer_not_taken), and a second re-park of the same ANSWERED call (the router stamps answeredAt before forwarding; a call nobody answered never counts, RemotePendingCallEntry.reparks) fails the delegating call typed with RemoteChildPausedError (the child interrupted, logged cp157.remote_park_no_progress), and the run goes on. An answer to a parked call on a root whose run has a recorded stop is refused run_stopped (never forwarded or queued). Every other pause keeps the typed failures: CP-45's RemoteChildPausedError (transport_stream_drop / remote_child_paused), or RemoteSubAgentClientToolUnsupportedError (failureReason: 'client-tool-unsupported') for a child's own client call — always on Temporal and DBOS (FU-CP157-TEMPORAL-DBOS-REMOTE-PARK). A remote call that fails interrupts its possibly live child first (stopRemoteChild), so a paused child is never left parked on its producer. A call answered only on the producer's own side does not wake the root until CP-157 MR 2's durable poll (CP-185). GitLab #107.
Remote settlement (Invariant S). A remote SubSessionRef leaves live (running / paused_awaiting_client) only in the same atomic store step that commits its call's real result: MessageCommit.settle (append and run-start commits) and SaveAndPromoteOptions.settle (step commits) carry SubSessionSettle { subSessionId, parentToolCallId, status, completedAt, expectGeneration?, stopOwed? } (a settle clears the ref's owed stop unless it carries one: CP-152's owed stop is absent only once the child answered the stop or ended), and every store applies core settleRefusal inside the commit: a ref no longer live (or at another outcomeGeneration) refuses the WHOLE commit with SubSessionRefConflictError (state_sub_session_ref_conflict, 409, non-retryable; on a run-start commit RunStartRejectedError('sub_session_ref_conflict'), re-planned once), a missing ref with SubSessionRefMissingError. Heal (not_executed) rows never settle (core remoteSettlementsFor). So a call gets one result: a duplicate tool_result is a provider 400 on every later call, and retry() keeps it (RM-98 / RM-99). What happens to a child is decided by the transport's answer alone (core remoteChildVerdict, the transitions table in the remote agents guide): a store error is never a verdict, a lost child is interrupted (CP-178), a child not found inside its deadline is still owed (CP-177), a gone tool is lost (tool_unavailable, RM-101), and an entry carrying input over a child unreadable inside its deadline is refused with the retryable RemoteChildStatusUnavailableError (framework_subagent_status_unavailable, 503, RM-100). Per runtime: runtime-js / the CF DO (entries and reconcileRemoteSubAgents) and CF Workflows (every commit that carries rows — the step commit, a cascade suspension's commit, appendCommitted, the stop pairing — derives its settles at one choke point via remoteSettlementsFor, with the read ref's generation; it has no entry settle since RM-69, because no committed remote call waits on a child there; and core recordSubSessionResult, which Temporal also uses) settle atomically; the in-run dispatch on runtime-js still writes the ref terminal before its step commit (the CP-146 repair pairs a call left unpaired by a crash in between); DBOS writes the ref before the workflow appends the result (FU-DBOS-REMOTE-SETTLE-ATOMIC) and Temporal / DBOS heal a still-running child's call at the next entry without settling its ref (FU-TEMPORAL-REMOTE-HEAL-SETTLE). The second door (D3): a remote ref whose call is in NO committed row (its parent step was lost: a step commits once, after its last result, so a run that died mid-dispatch committed no call) leaves live with no row, settled by an entry AFTER its run-start commit won (the old run can never commit that call): core lostStepRemoteRefs / lostStepRefSettlement (non-companion refs), the child interrupted best-effort and the ref settled failed (framework_subagent_lost, cause parent_step_lost) in a rows-free append commit of the entry's run, keeping the owed stop when the interrupt was not confirmed; an owed stop is retried, live or settled, until the child confirms it (CP-203, CP-217: core retryOwedRemoteStops on the OwedStopSchedule, default 30 s doubling to 1 h, abandoned after 24 h, remote_stop_owed_abandoned, record kept; every debt write a compare-and-set on the status and the debt read, UpdateSubSessionRef.expectStopOwed; paced by the CF DO's stop-cascade alarm and runtime-js's owed-stop timer, and run at every entry and stop cascade; see Remote stops owed). CF Workflows and runtime-js / the CF DO do this at every entry (runtime-js sweepLostStepRemoteChildren, after the run-start commit and before reconcileRemoteSubAgents, whose orphan branch now serves truncated companion calls only; CP-194): the stops run concurrently, each bounded by remoteChildStopTimeoutMs. The ref records the outcome for the PARENT, which never received the result: it reads failed / parent_step_lost even when the child's own session completed. Temporal and DBOS do not sweep yet (FU-D3-TEMPORAL-DBOS-LOST-STEP). A writer that re-states a ref status it read (a stop cascade's stopOwed stamp) passes UpdateSubSessionRef.expectStatus, a compare-and-set every store enforces (op 'stamp'), so a settle that landed since is never reverted (CP-195). Every in-run remote dispatch ref write is such a compare-and-set too (CP-202): the stream identity and progress writes of core executeRemoteSubAgentDispatch (Temporal), CF Workflows' executeRemoteSubAgentCall and runtime-js's remote executor, and runtime-js's terminal write before the step commit, each re-state the status the dispatch knows, so a superseded run's late write never reverts a ref an entry's sweep, a stop cascade or reconcile settled (a refused write is logged and left).
Persistent-companion continuation + the __finish__ heal (cross-runtime)
A persistent companion declares an outputSchema, so it always completes via the auto-injected __finish__ tool. Re-consulting a completed companion (companion__sendMessage / companion__spawnAgent re-using its name) no longer throws or deletes the session — it continues on the preserved child session (CAS completed → active, pair any trailing unpaired tool call if needed, append the consult, run a new turn with a fresh per-turn maxSteps budget). failed / terminated children still re-spawn fresh, and a first spawn of a name always gets a fresh child (on the CF DO every non-replay spawn asks the child DO to start fresh, so an earlier parent incarnation's child is wiped, CP-205). The continuation primitive necessarily diverges per runtime (different orchestration substrates), but the terminal-step pairing (C2/C3), the trailing continuation heal (C5), and the per-turn stepCount reset are uniform across all five. (For the heal mechanics themselves, see ./step-processing.md §The __finish__ history invariant; for the full companion guide, the Sub-Agents guide → Re-consulting a persistent companion.)
| Runtime | Continuation primitive (completed child) | Terminal-step pairing (C2/C3) | Trailing continuation heal (C5) | Per-turn stepCount → 0 |
|---|---|---|---|---|
JS (runtime-js) | In-process continuePersistentChild — reopen the preserved session and run a new turn in the same run loop. | ✅ | ✅ | ✅ |
| CF Durable Object | Child DO /start on the preserved session (drives the JS executor's existing-session continuation path). | ✅ | ✅ | ✅ |
| CF Workflows | Fresh write-once instance agent__<type>__<childSession>__continue__<parent-run suffix>-<stepCount>-<toolCallId> (C7); consult carried as the instance's newMessages, appended exactly once by the !isResumable continuation branch (no CAS in the companion step). | ✅ | ✅ (continuation-trailing-heal) | ✅ |
Temporal (runtime-temporal) | Activity-side store reopen (CAS + append consult inside executeCompanionToolCall) + wf.startChild with a __continue__<stepCount> workflow id. | ✅ | ✅ | ✅ |
DBOS (runtime-dbos) | startPersistentContinuation → deterministic ${childSessionId}-continue-${toolCallId} restart workflow (DBOS.startWorkflow dedupes on the id); consult carried as initialMessage, appended exactly once by the workflow body's checkpointed append. | ✅ | ✅ (also at root continuation) | ✅ |
All five runtimes converge on the same contract: terminal-step pairing (C2/C3) means a companion completes with an already-valid transcript in the first place, and the trailing continuation heal (C5) is the safety net for a run that stopped or was killed mid-tool. (Replay-idempotency on the durable runtimes — Temporal / DBOS — comes from the deterministic continuation ids plus, for DBOS, the metadata.dbosWorkflowId == ${childSessionId}-continue-${toolCall.id} marker short-circuit, which makes both spawn-continue and sendMessage-continue safe under workflow-body re-execution.)
Interrupted persistent companions stay resumable (CP-68). When a parent is interrupted while a persistent child is running, the child's session ends interrupted (a soft interrupt, not failed) and the parent's SubSessionRef reads 'interrupted', with no completedAt. companion__sendMessage then resumes the child on its preserved session and history. companion__waitForResult on an interrupted child returns { status: 'interrupted', message } with resume guidance instead of waiting until its timeout; it confirms the state over three consecutive polls, so a sendMessage resume running in the same step is not misread. companion__spawnAgent with the name of an interrupted child is refused (the error tells the model to use companion__sendMessage). companion__terminateChild still ends the child terminated. This holds on JS and CF Workflows (where a repeat waitForResult on a ref already reported interrupted also returns after three polls instead of parking). On CF Workflows the resumed child runs in a fresh write-once instance agent__<type>__<childSession>__resume__<stepCount>__<toolCallId> and writes to its own stream, not the parent's. On the CF Durable Object runtime (whose companions are sibling DOs, not the in-process run loop) only waitForResult changed: it returns interrupted after three polls of the child's live status instead of polling forever. Temporal and DBOS do not write 'interrupted' to the ref yet.
A companion call's frames on the parent's stream (RM-76). A persistent companion call (companion__spawnAgent, sendMessage, waitForResult, …) is announced and ended on the parent's stream like any tool call, so a live UI shows it running and then its result. On JS and the CF DO the step iterator writes both frames. On CF Workflows the companion calls get their tool_start from their own announce step, in call order, right before the companion loop (the only place they run), and their tool_end is built from the committed result row (core toolEndChunkForResultRow) and written by appendCommitted before the row's commit, so the commit's cursor covers it. Both frames are the parent run's own. One window, shared with regular tools (whose tool_end is written inside their own step.do, before the step commit): a crash between the frame and a failed append leaves a tool_end with no committed row; a reload corrects it, since the snapshot holds no row for the call. Temporal and DBOS still write neither frame (parked).
Stateless Suspension Model (v7) — At every HTTP request boundary, the runtime is free to die. The state store is the only durable thing across requests. When an agent reaches a HITL boundary (client-executed tool, approval-gated tool, sub-agent wait), the runtime writes SessionState.suspensionContext and emits the relevant stream chunk, then returns from the run. There is no in-memory waiter, no setTimeout promise, no DO hibernation guard. Resumption is driven by executor.resume() reading durable suspension context — never by signaling an in-memory waiter. This unblocks long pauses (~80% wall-time reduction on multi-minute HITL waits) and makes deadline semantics deterministic (deadlines measure durable clock time, not in-memory promise lifetime). v7.0 ships this model on all 4 runtimes: JS, Cloudflare DO, Temporal, and Cloudflare Workflows. The runtime-dbos runtime added by main uses its own DBOS-native DBOS.recv/DBOS.send primitives for HITL suspension (Postgres-backed workflow replay) and is not part of the unified suspensionContext model — see the runtime-dbos package docs for its specifics.
Client-Executed Tools (v7) — Created with defineTool({ execute: 'client' }). When the LLM calls such a tool, the runtime writes a pending entry to the session's pendingClientToolCalls map (durable state), emits a tool_start stream chunk, and returns from the run with RunOutcome.kind = 'suspended_client_tool' (which surfaces as AgentResult.status = 'suspended_client_tool'). The runtime does not block in-memory. Consumers call executor.submitToolResult({ kind: 'client-tool-result', toolCallId, result | error }) (or the 'approval-response' variant for approval-gated tools — both share the same SubmitToolResult union); the submission writes the result to durable state and answers the continuation it caused (core SubmitContinuation, W1 MR2b: awaiting_batch, caller_resumes, new_run, deferred, consumed), so the caller knows whether to executor.resume() (runtime-js, Temporal) or attach to a run the runtime started itself (CF Workflows' __submit__ instance, the CF DO's wake, the DBOS parked workflow's continuation run). The canonical cross-runtime signal for "awaiting client submission" is pendingClientToolCalls map presence; session-level SessionStatus remains 'active'. Mixing execute: 'client' with finishWith: true is rejected at defineTool time, as is mixing execute: 'client' with requireApproval. The framework maintains SessionState.clientToolCallOwnership on root sessions to route submissions to the owning sub-agent; SessionState.rootSessionId points each sub-agent at its root. Submissions always go against the ROOT sessionId. Per-runtime status:
- runtime-js: durable state writes +
executor.resume()on submission. No in-memory promise map. Process restart is safe — pending entries are recovered from the state store on the next request that touches the session. The runLoop reads its run's durable stop record (pollStop,readStop(sessionId, runId)) at every step boundary, so a stop recorded by another process is honoured at the next boundary (see Stop as one signal). - runtime-cloudflare (DO path): durable state writes via
DOStateStore. The hibernation guard is removed in v7; DOs are free to evict during HITL waits. A call's deadline fires on its own, with no request (S-A): the DO'sclient-tool-deadlinealarm subscriber computes the current run's earliest unsubmitteddeadlineAt(nextClientToolDeadlineAt), and at that time expires the lapsed callsclient_tool_timeout(expireOverdue) and continues the run through its normal wake. The expiry write is stop-guarded (saveState { fenceRunId, refuseIfStopped }, CP-173): a lapse after a stop is new work, so a stop recorded first wins (no answer, no continuation; the alarm settles the stop). A failed expiry goes through the drive failure table and the shared crash-loop budget (CP-174): a permanent failure ends the runfailedtyped, a transient one is retried 1 s later untilframework_auto_resume_exhausted(withcause), and a terminal that does not land is retried every 30 s (gave_up). The failed run keeps no unanswered pending call, so a retry or a new turn heals it (an ANSWERED sibling is drained on a retry, rule 6, and settled with its stored answer by a new turn's run-start commit, RM-81). A chat request that arrives first may still sweep a lapsed call itself (findExpiredPending, recorded asclient_tool_deadline_exceeded: FU-CLIENT-TOOL-TIMEOUT-TWO-CODES, open; !409 makes that sweep recordclient_tool_timeouttoo); whichever writes first wins. See Cloudflare Durable Objects (S-A). - runtime-temporal: Workflow exits cleanly on every HITL boundary (mirrors CFW Workflows). Suspension state is durable in the session store; the workflow returns
AgentWorkflowResult { status: 'suspended_*' }and Temporal releases the workflow.executor.resume(sessionId)starts a NEW workflow instance with workflow ID${prefix}__${agentType}__${sessionId}__resume-${runId}, keyed on the run id the executor minted for that call (never a counter, so a losing caller writes nothing); its entry activity commits the run start. The new workflow'smode='resume'branch calls theapplyResultsAndReloadactivity, which drains submitted client-tool results into messages, firesonMessage+afterToolhooks, synthesizes timeouts for expired deadlines, and drains completed sub-agent children viarecordSubSessionResult.submitToolResultis durable-only — no Temporal signal is sent (the workflow has already exited). Sub-agents are child workflows started viawf.startChild; on parent suspension, in-flight children are markedfailed:'parent_suspended'(mitigation #3) and re-spawned via the__resume-Nworkflow ID convention on parent's resume (γ-cascade, spec §5.2). Approval-gated tools share the same suspension primitive (tool_approval_requestchunk +'suspended_client_tool'status); approve / deny submissions are routed via the same durablesubmitToolResultflow. - runtime-cloudflare (Workflow path): Workflow returns early from
runAgentWorkflowon HITL boundaries withstatus: 'suspended_*'and durable suspension state viacommitSuspendedStepactivity. The submit that completes the batch starts a new workflow instance withmode: 'resume'(the deterministic__submit__continuation, which waits in its body for the suspended instance to close) that drains submissions viaapplyResultsAndReloadand continues;executor.resume({ mode: 'continue' })adopts it. A call's deadline fires on its own, with no request (CP-138): each client-tool suspension starts one__deadline__watcher instance that sleeps until the run's earliest unanswereddeadlineAt, expires the lapsed callsclient_tool_timeout(the submit's own version-pinned, run-fenced write, so the first writer wins) and, once the run's batch is complete while the run is still suspended (whoever answered it), runs the owed continuation (continueOwedSession, which adopts a submit's own); a transient failure to start it is retried by the watcher's step. The watcher never enters a run. A chat read (reconcileSession) runs the same expiry for a suspension behind a closed owner, so a watcher that was never created heals on the next read. SetWorkflowOptions.continuationPredecessorTimeoutMsto the executor's value so a deadline-driven continuation waits for its predecessor like a submit-driven one. Sub-agents cascade up — child suspensions propagate to parent'ssuspended_awaiting_children. γ-cascade re-spawn on parent resume (FU-A2-40, mirrors Temporal FU-A2-09): when the parent suspends mid-sub-agent dispatch,commitSuspendedStepmarks eachsuspendedAwaitingChildrenentry's child session asfailed:'parent_suspended'. On resume,applyResultsAndReloadsurfaces those children viachildrenToRespawn; the workflow body re-dispatches each viaworkflowBinding.create({ id: buildWorkflowInstanceId({ agentType, sessionId: childId, kind: 'respawn', key: attempt }) })(readable formagent__<type>__<id>__respawn__<attempt>), polls the child's durable state until terminal, and records the outcome viarecordSubSessionResult. A drain-clear step then resets the parent's suspension discriminators when fully resolved. Eliminates v6's billable wall-time during HITL waits (~80% reduction on multi-minute approvals). - runtime-dbos: uses its own DBOS-native suspension primitive (
DBOS.recv(toolCallId)over Postgres-backed workflow replay) rather than the unifiedsuspensionContextmodel. Both client-executed tools (dispatchClientTool) and approval-gated tools (dispatchApprovalGatedTool) share the samependingClientToolCallsmap +DBOS.recvwake. For approval-gates, approve (submitToolResult({ kind: 'approval-response', approved: true })) runs the tool with the original input; deny (approved: false) records a synthetic failed tool result ('Tool call was not approved by the user',helixToolFailed: true) and skipsexecute(). Function-formrequireApprovalis evaluated inside a@DBOS.step(ApprovalGateStep.evaluateApprovalGatePredicateStep) so its boolean is checkpointed in workflow history and the suspend-vs-run decision is replay-deterministic even for non-pure predicates; fail-closed semantics (throw → suspend) live inside the step so the checkpointed value is always a boolean (parity with runtime-temporal's activity-wrapped predicate evaluation; GL #111 Batch C / gap 5).onAgentSuspended/onAgentResumedfire for both client-tool and approval-gate suspend/resume cycles (the recv-driven auto-continue path wakes and resumes). Per-call hooks (agent.hooksfield onAgentConfig, plusExecuteOptions.hooks/ExecuteOptions.hookManager) fire on everyexecute()/resume()/retry()via the process-localHookManagerRegistryindexed byDBOS.workflowID(GL #111 Batch D / gap 4):executeImpl/resumeImpl/retryImplcompute the merged manager BEFOREDBOS.startWorkflow(constructor → agent.hooks → options.hooks order, mirroringruntime-js/src/js-agent-executor.ts:3065-3108buildHookManager) and register under the new workflowId;HookStep/ExecuteToolStepresolve the manager on every invocation and fall back to the constructor-bound static on cross-worker recovery (documented split-brain risk mirroring Temporal/CFreplaceAgentsemantics atruntime-cloudflare/src/registry.ts:182-193). All five GL #111 gaps (1–5) are closed.
A call's routing entry precedes its frame on runtime-js, the CF DO and Temporal (CP-253), so an answer can arrive before the suspension commits. The submit waits for that commit, up to submitOwnerReadyBoundMs (5 s), storing nothing, then answers the retryable state_submit_owner_not_ready (503). With no entry and no run that could commit one it answers unknown_tool_call. The wait keys on the session's run, so an id no call ever had, a reserved-name id on an executing child, or an entry a crash left behind answers the retryable error while that run executes and unknown_tool_call once it is not: an accepted cost, nothing is stored. DBOS and CF Workflows register before the frame and do not wait.
Known limitation — doubly-nested HITL (ephemeral sub-agent calling a client-executed tool) is not supported. Single-level HITL works on every runtime: a top-level agent's client-executed (or approval-gated) tool suspends the run with
'suspended_client_tool'and resumes viasubmitToolResult. The doubly-nested case — where an ephemeral child sub-agent itself calls aexecute: 'client'tool and is expected to suspend the parent with a routable sub-session id — is not a supported path. Observed behavior is that the parent run returns'completed'rather than suspending; the child's client-tool dispatch does not propagate a suspension up through the sub-agent boundary. Agents that need inner-agent human approval should either (a) hoist the client-executed tool to the top-level agent, or (b) use a persistent sub-agent (which has its own stable session id and can be driven through follow-up turns) rather than an ephemeral one. The cross-runtime matrix scenario for this case (cross-runtime-subagent-resume-matrix-suspended.integ.test.ts, S6) is pinned as a negative-invariant test (FU-MATRIX-DOUBLY-NESTED-HITL): it asserts the parent returns'completed'(not'suspended_client_tool') and that exactly onesubagent_start/subagent_endpair is emitted — so if a future change starts supporting nested HITL, the assertion fails and forces a conscious revisit. Tracked in GitLab #73 — currently documented-as-unsupported rather than scheduled for implementation.
Hook firing on the timeout path is consistent across all 4 runtimes. When a client-tool deadline elapses, every runtime appends a synthetic failed tool result (built by createToolResultMessage with success: false: content {"error": <humanized client_tool_timeout text>}, metadata.helixToolFailed: true), emits a tool_end chunk carrying that error, records usage with success: false, and fires onMessage + afterTool hooks with the timeout payload. CFW Workflows is the reference implementation; runtime-js (and CF DO via runtime-js) and runtime-temporal were brought to parity in commit 799aeea77.
Lifecycle hook firing order is canonical across all 5 stateless-suspension runtimes (DBOS achieved full parity in FU-DBOS-ONMESSAGE-ONSTATECHANGE). For the regular (and approval-gated approve) tool execution path, every runtime fires user-facing AgentHooks in the canonical sequence:
beforeTool → execute → onStateChange → onMessage → afterToolonStateChange reflects the immediate state mutation from execute; onMessage surfaces the result-as-message; afterTool is universal cleanup with the full result payload. Pre-2026-05-02 each of runtime-js, runtime-temporal, and CFW Workflows fired in a different order (runtime-js fired onMessage AFTER afterTool; runtime-temporal fired onMessage BEFORE onStateChange); sub-projects #2 + #3 unified the order so portable hook code can rely on cross-runtime sequencing. Per-runtime regression guards live in:
packages/runtime-js/src/__tests__/js-agent-executor-hooks.test.ts(regular path) andapprove-path-hooks.test.ts(approve drain path)packages/runtime-temporal/src/__tests__/v7-activities-hooks.test.ts(regular path) andv7-approve-path-hooks.test.ts(approve drain path)packages/runtime-cloudflare/src/__tests__/approve-path-hooks-do.test.ts(DO approve drain path)packages/e2e/src/__tests__/approval-gate-hook-parity.integ.test.ts(cross-backend)
Implementation note: all multi-tool runtimes (runtime-js, Cloudflare DO, CFW Workflows, runtime-dbos, runtime-temporal) execute a step's regular server tools in PARALLEL. Each tool seeds from the same committed / pre-step base, and the merge CONTRACT is identical everywhere — append ops compose (all survive regardless of arrival order), replace ops are last-write-wins. The MECHANISM by which writes are merged differs by runtime: runtime-js and Cloudflare DO (which inherits the JS executor) merge IN MEMORY between tools within a single runLoop step — no durable staging. The durable runtimes — runtime-temporal, CFW Workflows, and runtime-dbos — instead STAGE each tool's writes durably (stageChanges) and promote them atomically at the step boundary (saveStateAndPromoteStaging), because the orchestrator is a replay-deterministic sandbox that can't hold cross-activity in-memory state. The per-runtime mechanics for keeping hooks in canonical order under parallelism differ:
- runtime-js dispatches phase-1 tools via
Promise.all. To preserve LLM-input ordering ofstate.messageswhile maintaining the canonical hook order,runServerTool(inpackages/runtime-js/src/run-loop.ts) defersafterToolfiring back to the iterator's collection loop inpackages/core/src/orchestration/step-iterator.tsviaExecuteServerToolResult.deferredAfterTool. The iterator pushes the message → firesonMessage→ fires the deferredafterToolper result, in input order. Cloudflare DO inherits this path via the JS executor. - runtime-temporal dispatches one
executeToolActivityper regular server tool viaPromise.allfrom the workflow body; each activity STAGES its writes viastageChanges. Because the workflow sandbox can't run user hooks, a dedicatedfireDeferredToolHooksactivity firesonMessage/afterToolin index order after thePromise.allsettles; promotion reuses the existingcommitStep/commitSuspendedStep(saveStateAndPromoteStagingover the staged rows). CFW Workflows uses the same staged-then-promote shape with its own per-tool helpers. runtime-dbos runs each tool as an independent durable step and defersonMessage/afterToolto the workflow body (see the DBOS note below).
DBOS hook firing order — full parity (FU-DBOS-ONMESSAGE-ONSTATECHANGE closed). DBOS fires the canonical hook sequence (beforeTool → execute → onStateChange → onMessage → afterTool) with the same post-Promise.all timing as JS / Temporal / CF. The post-2026-05 implementation:
beforeTool+onStateChangefire insiderunExecuteTool(the@DBOS.stepbody), so they are checkpointed by the step boundary.onMessage+afterToolare deferred to the workflow body (shared.ts) so they land AFTER the phase-1Promise.allsettles AND after each tool-result message has been appended to the conversation log.ToolStepResult.durationMsis the deferred-afterTool signal — present on results fromrunExecuteTool(server-tool / approve-resume), undefined on client-tool / approval-deny results (theirafterToolis fired inline bydispatchClientTool/ not fired at all on deny — cross-runtime contract).
The previous "timing caveat" divergence note has been resolved by this work. Tracing pipelines that compare DBOS to JS / Temporal / CF observe identical sequencing now.
Hook firing parity table:
| Runtime | beforeTool | onStateChange | onMessage | afterTool | Promise.all timing |
|---|---|---|---|---|---|
| runtime-js | ✓ | ✓ | ✓ | ✓ deferred | After Promise.all |
| runtime-temporal | ✓ | ✓ | ✓ | ✓ deferred | After Promise.all |
CFW DO (via runtime-js) | ✓ | ✓ | ✓ | ✓ deferred | After Promise.all |
| CFW Workflows | ✓ | ✓ | ✓ | ✓ inline | After Promise.all |
| runtime-dbos | ✓ | ✓ | ✓ | ✓ deferred | After Promise.all |
onAgentSuspended / onAgentResumed:
| Runtime | client_tool reason | awaiting_children reason | Notes |
|---|---|---|---|
| runtime-js | ✓ | ✓ | Reference impl |
| runtime-temporal | ✓ | ✓ | Matches reference |
| CFW DO | ✓ | ✓ | Matches reference |
| CFW Workflows | ✓ | ✓ | Matches reference |
| runtime-dbos | ✓ | ✓ (post v7.0-final) | Earlier v7 versions skipped awaiting_children due to a guard bug; fixed per second-round review P1.1 |
CF Durable Object server hooks (WA MR1). The DO host's ServerHooks are not agent hooks: they fire around a drive, and the agent's lifecycle hooks above fire inside the run unchanged. Every DO entry, wake and submit continuation goes through one pipeline (runDrive), so one order holds on every path:
beforeEntry(request drives only:/start,/resume,/retry,/subagent/:type/start|resume) runs after the producer claim and the busy check, the type / identity gate andmaterializeAgent, so a refused entry never reaches it. It is admission only:{ abort: true, response }is answered verbatim and nothing is written. UnderconcurrentEntry: 'interrupt_previous'the executing run is stopped only afterbeforeEntryadmits the entry.- The agent's own hooks (
onAgentStart, the tool hooks,onAgentComplete/onAgentFail, …) fire inside the run, as on runtime-js. afterEntryfires once per drive, once the drive owns its run (after the run-start commit; for a wakerecover, after the in-memory claim). It is not awaited; a throw is logged.onRunSettledfires from the settle path, after the run's own stream is finalized and the execution registration is released (so a new entry is no longer blocked by it). It is delivered at least once per(runId, status)for every status at which nothing executes the run (suspended_*,completed,failed,interrupted,superseded), with a durable delivery mark in DO KV written after the hook returns or throws; a wake redelivers a missing mark. Hosts deduplicate ondeliveryId(${runId}:${status}). A paused run that is later stopped settles twice. With noonRunSettledconfigured, no mark is written.
/interrupt and /abort fire no host hook themselves. See the upgrade guide and the design spec ../superpowers/specs/2026-10-08-do-drive-pipeline-design.md §3.2, §3.6.
Resume runId semantics (cross-runtime)
When a session is resumed, the onAgentResumed hook fires with { runId, previousRunId, sessionId, resumedFromCheckpointId }. The relationship between runId and previousRunId differs by runtime:
runId !== previousRunId on every runtime:
- JS / Temporal / Cloudflare / DBOS fresh resume workflow: each
resume()call commits a new run (the guarded run-start) and runs the loop under the new runId. - DBOS recv-driven in-workflow continuation: the cross-runtime client-tool submit path finds the PENDING workflow blocked on
DBOS.recvand returns a handle to that SAME workflow. After the wake, the workflow commits a continuation run (startRunCommit({ kind: 'continuation' }), sameownerToken= DBOSworkflowID, runId${workflowID}:${stepID}) that supersedes the suspended run, and continues under it (store-enforced commit model, design spec §3.0). The former DBOS self-loop (runId === previousRunId) is gone.
previousRunId value contract (uniform across all four runtimes): regardless of whether the resume started a new workflow or continued in-workflow (DBOS recv-wake), previousRunId is always populated with the SUSPENDED run's runId. This holds on every runtime:
- JS —
js-agent-executor.tsresume()setspreviousRunId: previousRun?.runId(the suspended run, captured viagetCurrentRunat the top ofresume()before the new run is created). It must NOT readstate.runIdat the fire site —loadAgentState(sessionId, runId)overwrites that to the fresh resume runId. (resumeLoopalready used the correctpreviousRun?.runId; the publicresume()path was repaired in the greptile-P2 fix-forward.) - Temporal —
executor.resume()captures the suspended runId viagetCurrentRunBEFORE allocating the resumed runId, threads it throughAgentWorkflowInput.previousRunId→applyResultsAndReload→ theonAgentResumedfire site. (Before the greptile-P2 fix-forward it firedpreviousRunId: undefined— repaired so the contract holds.) - Cloudflare — the resume workflow sets
previousRunId: existingState.runId(the suspended run, fromloadAgentState→getCurrentRun). (Before the fix-forward it mistakenly passedexistingState.checkpointId— a checkpointId, not a runId — repaired.) - DBOS — the fresh resume workflow threads
previousRunIdthrough the workflow input; the recv-wake continuation firesonAgentResumedwithpreviousRunId= the suspended run andrunId= the continuation run.
Hook consumer contract: Use previousRunId (always populated with the suspended runId on resume) as the linkage signal for span stitching, audit linkage, and tracing parent identification. Prefer previousRunId over runId !== previousRunId for resume detection. tracing-langfuse does this correctly today: packages/tracing-langfuse/src/langfuse-hooks.ts:521-522 uses previousRunId only as metadata, and parent-span linkage uses tracingContext.lastActiveSpanId / rootSpanId, not runId equality.
The cross-runtime e2e parity test (packages/e2e/src/__tests__/lifecycle-hooks-parity.integ.test.ts Scenario 1) enforces both contracts: (1) an unconditional resumedCtx.previousRunId === suspendedCtx.runId assertion that holds on every runtime (the previousRunId value contract above), and (2) an unconditional runId !== previousRunId assertion (the DBOS runtime-conditional that FU-DBOS-RESUME-RUNID-CLIENT-TOOL's Option D introduced was removed once DBOS moved to continuation runs); the unconditional previousRunId assertion was added in the greptile-P2 fix-forward and is what surfaced (and is now backed by) the Temporal + CF previousRunId repairs.
Hook customState reconcile after preStep capture (commit 8654a2686) — Hooks that fire AFTER the iterator captures preStepCustomState (assistant onMessage, phase-1/phase-2 beforeTool/afterTool) call hookContext.updateState, which mutates nextState.customState in-place via Immer. They do NOT contribute to the staging-changes pipeline that commitStep promotes. After commitStep returns, the iterator reconciles by writing the freshest nextState.customState back to the store as a follow-up saveState IF it differs from the just-committed value. Hook-less steps still do exactly one durable write per step; steps that mutate via post-snapshot hooks do two (commit + reconcile). Best-effort: a follow-up save failure logs a warning but doesn't fail the step.
InMemoryStreamManager cursor rebase on cleanup (commit 65eaaf235) — InMemoryStreamManager tracks reader sequences as literal indices into the chunks array. cleanupToStep filters orphan-step chunks, shifting surviving chunks earlier. Active readers' currentSequence pointers are now rebased in cleanupToStep (via a Set<ReaderCursor> on the stream) so they observe chunks emitted just before cleanup (e.g. the run_interrupted boundary marker the runtime-js soft-interrupt path emits). Active createReader / createResumableReader consumers now correctly observe chunks that survived a cleanup — previously these were silently invisible to the reader.
D1 saveStateAndPromoteStaging atomicity (commit 7509872e3) — D1StateStore.saveStateAndPromoteStaging now correctly surfaces concurrent-CAS losses as StaleStateError (was sometimes D1StateError, breaking retry paths that gate on instanceof StaleStateError). UNIQUE-constraint violations on __agents_messages.(session_id, sequence) are caught and re-thrown as StaleStateError. The trailing DELETE on __agents_staging runs OUTSIDE the atomic batch — only after the version-pinned UPDATE confirms changes==1. Observable: rare network failures between successful main commit and staging DELETE leave a stale staging row that's harmless and self-healing (next stageChanges upserts the same key).
Driving the agent loop after submitToolResult: submitToolResult is a durable write, and its answer's continuation says who continues the loop (W1 MR2b): caller_resumes (runtime-js, Temporal) means the caller does; new_run / deferred mean the runtime already does (DBOS, CF Workflows, the CF DO), so a caller attaches instead. Consumers continue the loop in one of two ways:
- Use the framework's chat plumbing —
handleChatStream(server) anduseChat+useResumeClientTools(React) drive the resume internally. This is the recommended path for typical web app deployments. - Call
executor.resume(agent, sessionId)explicitly — when the answer iscaller_resumes(runtime-js, Temporal); on CF Workflows acontinueresume adopts the__submit__continuation the submit started (before its run-start commit, or its running run after it), and on DBOS and the CF DO the runtime already continues the turn, so attach instead. Returns a new handle observing the resumed run. Use this when calling the executor directly (custom server, integration tests). The pattern:await executor.execute(...)returns'suspended_*'→await executor.submitToolResult(...)writes durable result →const newHandle = await executor.resume(...)drives the loop forward →await newHandle.result()resolves with'completed'(or whatever terminal state the resumed run reaches).
This separation is by-design for v7 stateless purity — submission and resumption can happen in different processes, with no in-memory bridge between them.
Structured Logger events: client_tool.suspended, client_tool.submitted, client_tool.timeout, client_tool.aborted, client_tool.ownership_write_failed/ownership_clear_failed/ownership_retry, client_tool.validation_failed. Records CLIENT_TOOL_WAIT_MS_METRIC per call (now measured from durable suspension write to durable submission write); aggregate via __agents_usage (Postgres/D1 column source_type = 'client_tool_wait_ms', with per-tool breakdown via source_name). See ../guide/client-executed-tools.md for the full guide and ../upgrade-guides/v6-to-v7-stateless-suspension.md for v6→v7 migration steps including the operator runbook for force-failing stuck calls.
Approval-Gated Tools (v7) — First-class HITL primitive on defineTool: pass requireApproval: true (always require approval) or requireApproval: (input, ctx) => boolean (function form, evaluated per-call). When the gate matches, the runtime emits a tool_approval_request stream chunk with the parsed input and suspends with 'suspended_client_tool' (the same primitive carries both client-tool and approval flows; routing happens off the kind field of the submission, not the stream-chunk type). Resume by calling executor.submitToolResult({ kind: 'approval-response', toolCallId, approved, reason? }). On approved: true, the original execute() runs normally with the original input. On approved: false, the runtime appends a synthetic tool result with the canonical text 'Tool call was not approved by the user' and skips execute() entirely (there is no tool_error stream chunk; the persisted shape currently differs by runtime — failed on CF Workflows / DBOS, successful on JS / CF DO / Temporal — see CP-66 in the conformance findings). The function form fails-closed: an exception inside the evaluator is treated as requireApproval = true (matches the Mastra precedent — fail safe by requiring approval rather than silently bypassing). requireApproval is mutually exclusive with execute: 'client' and finishWith: true; both combinations are rejected at defineTool time. All 4 v7-stateless runtimes support approval flows on the v7 stateless model: the runtime suspends durably, and submitToolResult({ kind: 'approval-response', ... }) triggers a fresh run via executor.resume() (or via the framework's chat plumbing). CFW Workflows now uses the same v7 stateless model — workflow exits on approval-gate match; resume drains the approve/deny submission via applyResultsAndReload. runtime-dbos also supports approval-gate suspension (GL #75), but via its own DBOS-native DBOS.recv primitive rather than the suspensionContext model: dispatchApprovalGatedTool suspends on a requireApproval match, approve runs execute() with the original input, deny appends a synthetic failed tool result ('Tool call was not approved by the user') and skips execute(), and function-form requireApproval is evaluated inside a @DBOS.step so its boolean is checkpointed in workflow history (fail-closed semantics live inside the step; replay-deterministic suspend-vs-run decision even for non-pure predicates — GL #111 Batch C / gap 5). Per-call agent.hooks (and options.hooks / options.hookManager) fire on every workflow start via the process-local HookManagerRegistry indexed by DBOS.workflowID (GL #111 Batch D / gap 4 — cross-worker recovery falls back to the constructor-bound static, matching Temporal/CF replaceAgent split-brain semantics). All five GL #111 gaps (suspended.toolCallIds, exactly-once submit, client_tool_timeout tool_end, per-call hooks, checkpointed requireApproval) are closed.
Agent Server — The @helix-agents/agent-server package provides AgentServer for hosting agents over HTTP. Accepts any AgentExecutor implementation and exposes the following routes:
- Executor routes (always wired):
POST /start,POST /resume,GET /sse,GET /status,POST /interrupt,POST /abort,POST /submit-tool-result,GET /workspace. - Chat handler routes:
POST /chat,GET /chat/{sessionId}/stream,POST /chat/{sessionId}/submit-tool-result,POST /chat/{sessionId}/interrupt,POST /chat/{sessionId}/abort.POST /chatand the stream reconnect answer 501 without a configuredchatHandler; the stop routes and the submit route do not need one (the stop routes are also the operator's stop of any session by id). These layer on top of the executor and provide the canonical chat-style flow used byuseChat+useResumeClientTools.
Transport adapters: createHttpAdapter() (generic), createExpressAdapter() (Express). /interrupt and /abort go through the executor's stop(sessionId, …) (core stopRun; JSAgentExecutor has one, any other executor is stopped through its handle), so they work on any replica and on a paused session, and answer at once (202 / 200 / 409 / 404). Fail-closed auth: the constructor throws if neither authenticate hook nor explicit allowUnauthenticated: true is configured.
v7 removed the v6 INTERRUPT_NOT_LOCAL 503: a stop is a durable, run-scoped record (requestStop) honoured at the run's next boundary, whichever process holds the run. HTTP clients no longer need to retry against the "owning" server.
Runtime — Executes the agent loop. JSAgentExecutor runs in-process, TemporalAgentExecutor uses Temporal workflows for durability, DBOSAgentExecutor uses DBOS Transact (Postgres-backed workflow replay) for durability and supports the same client-executed-tools surface as Temporal and Cloudflare.
Workspace and HITL runtime support — Two orthogonal capabilities, with overlapping (but not identical) runtime support:
- Workspaces (
agent.workspace): runs on JS runtime and Cloudflare Durable Object runtime (via@helix-agents/agent-server). Temporal, CF Workflows, and DBOS do not support workspaces (Temporal and CF Workflows fail-fast at run-start; DBOS silently passesworkspaces: undefined). - HITL (client-executed tools,
requireApproval): runs on all 5 runtimes — JS, Cloudflare Durable Object, Cloudflare Workflows, Temporal, and DBOS. The first 4 use the v7 stateless suspension model (durable-state-only suspension viaSessionState.suspensionContext); DBOS uses its own DBOS-nativeDBOS.recv/DBOS.sendprimitives over Postgres-backed workflow replay (functionally equivalent but architecturally separate from the unifiedsuspensionContextmodel).
| Runtime | Workspaces | HITL | Notes |
|---|---|---|---|
JS (runtime-js) | Full | Full (v7 stateless) | All providers; recommended for dev + non-DO production. |
CF Durable Objects (via agent-server) | Full | Full (v7 stateless) | All providers; recommended for CF production. |
Cloudflare Workflows (runtime-cloudflare/src/workflow.ts) | Refused | Full (v7 stateless) | All providers; recommended for Cloudflare Workflows production. Workspaces are refused (typed framework_not_supported in execute() / resume() / retry(), before any write). |
Temporal (runtime-temporal) | Refused | Full (v7 stateless) | All providers; recommended for Temporal-backed production. Workspaces are refused (same guard). |
DBOS (runtime-dbos) | Refused | Full (DBOS-native) | Postgres-backed; recommended when consumers already use DBOS Transact. Workspaces are refused (same guard). |
Run records (listRuns) — full parity. Every entry writes exactly one run record, inside its run-start commit, on all runtimes: execute() (fresh and continued turns), every resume(), retry(), child sessions and companion continuations, plus DBOS's in-workflow continuation run on a submit wake. listRuns(sessionId).runs.length grows by one per entry, the turn numbers are contiguous (1, 2, 3, …), and the boundaries (startSequence, startMessageCount, startUIMessageCount) come from the input checkpoint the same commit wrote, so they are identical across runtimes (run.start-sequence, run.start-message-count, run.start-ui-index). This replaces the per-runtime createRun calls and fixes RM-43 to RM-47: boundaries recorded before the turn's input was appended (JS with_message, CF Workflows continued turns), a Temporal continuation reading them while its workflow ran, DBOS counts hard-coded to 0, Temporal retry() writing no run, and Temporal never writing a terminal run status. The findings records are closed by the conformance owner. See ./session-model.md §Run records.
Commits, runs and terminal order (cross-runtime)
The store-enforced commit model (design spec ../superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md; migration in the upgrade guide) holds on every runtime. The mechanics are in ./session-model.md §Commits and checkpoints and ./execution-flow.md.
| Concern | JS (runtime-js) | CF Durable Object | Temporal | DBOS | CF Workflows |
|---|---|---|---|---|---|
| Who commits the run start | the executor (commitRunStart) | DOAB, in one transactionSync with the claim and the entry state; every DO entry reaches it through one drive pipeline (runDrive, WA MR1) | the workflow's startRunCommit activity | the workflow's run-start step | the instance's ${agentType}-run-start step.do |
| Run id | minted per entry call (randomUUID) | minted by DOAB per request | minted by the executor (workflow input); children wf.uuid4() | minted by the executor (workflow input); in-workflow continuation <suspended run>:continuation (dbosContinuationRunId); companion ${childSessionId}-continue-${toolCallId} | ${instanceId}:${agentType}-run-start:0 (params.runId) |
ownerToken | random per entry | none | ${workflowId}:${workflowRunId} | workflowID | the exact instance id |
| Ownerless (a live run may be superseded when) | lease lapsed or absent (store clock) | the DO is not executing that run | its workflow is closed (describeWorkflow) | its workflow status is terminal | its instance is complete / errored / terminated, or unknown past cfwUnknownTerminateAfterMs |
| Step atomicity | one step commit per step | via JS | one step commit; a phase-2 finishWith result is a later append (RM-52) | one step commit with the step's tool results (RM-40), live sequence (RM-39) | every log mutation is a commit (RM-41) |
| Suspension order | frames → commit → suspended_* → pauseStream | via JS | frames → commit → suspended_* → suspension_marker → pauseStream | suspended_* → frames → commit → pauseStream | suspended_* → frames → commit → pauseStream |
| Terminal order (every exit) | hooks → K5 → run status → stream | via JS | hooks → K5 → run status → stream (commitTerminalState + finalizeTerminalRun) | hooks → K5 → run status → stream (finishTurn) | hooks → K5 → run status → stream (finishRun) |
Superseded between onAgentComplete and K5 | complete → superseded (last wins) | via JS | complete → superseded (last wins) | complete → superseded (last wins) | complete → superseded (last wins) |
resumeCount | never counted | never counted | the resume / retry winner (not a child re-spawn) | the resume / retry winner | the resume / retry winner (not a recovery) |
| Run-start fail-safe | n/a (the call is the entry) | n/a | entryOutcomeTimeoutMs (120 000 ms); a committed run gets its handle even if it never started (FU-TEMPORAL-ENTRY-FAILSAFE-SILENT-HANDLE) | runStartFailSafeMs (120 000 ms); same committed-run handle (FU-TEMPORAL-ENTRY-FAILSAFE-SILENT-HANDLE) | entryOutcomeFailSafeMs (120 000 ms); entryStartTimeoutMs replaces the wait. A run that committed but never started throws RunStartStalledError (never retryable; owner stopped → the run is settled through the closed-owner planner, settle reporting a settle that did not land or lost to a newer run), never a handle, except a run with an outcome of its own (K5 landed, committed suspension, already ended), which returns its handle. abandonRunStart is retried 3×; a failed abandon can end in RunStartOutcomeUnknownError |
| R49 start record | the code path (entry failures throw before the loop) | via JS | staged agent-start-hooks-<runId> (at least once on a crash) | agent-start-fired-<runId> | the recorded start-hook step.do (runEntered) |
| Superseded stop | superseded result (fence, a store-rejected lease renewal, post-loop check) | via JS; at entry the drive classifies it lost (a request answers 409, a wake does nothing); a run superseded later settles superseded (onRunSettled) | ApplicationFailure.nonRetryable → superseded | recorded as the step's outcome (never retried) → superseded | NonRetryableError in step.do → superseded |
| Retry | planRetry → run-start with truncateTo | via JS (/retry) | planRetry in the run-start activity | planRetry in the run-start step; honours checkpointId / message (CP-30) | planRetryEntry → planRetry |
| A throw inside the run loop | failed everywhere; onAgentFail once (CP-71) | via JS | failed | result() throws and the session stays non-terminal (CP-94) | not yet checked (no executor lane) |
| Retryable LLM error step | retried inside the step under the executor's llmStepRetry (default 3 retries, 1 s, exponential), each retry a new attempt of the same step; then fails with the last attempt's detail (MA-26) | via JS | ordinary call fails the run (FU-E5-LLM-ERROR-STEP-NOT-RETRIED); forced call → activity retry (an unclassified error too) | fails the run (FU-E5-LLM-ERROR-STEP-NOT-RETRIED) | step.do retries the LLM step under llmStepRetry, then fails with the provider's detail; an unclassified error is not retried (DI-35); before the model call (the boundary is the call itself) a retryable: false error fails at once, an unclassified one from the step's I/O keeps step.do's retry, and one from its deterministic work (tool set, prompt, systemPrompt function) or a hook fails at once |
Terminal errorDetail of an LLM error | classified (code, retryable) | via JS | classified (terminal.errorDetail across the activity boundary) | { message } only (DI-24) | classified |
| Hard abort | failed, framework_cancelled | via JS | no framework_cancelled code (FU-CFD1-DI03-TEMPORAL-DBOS) | no framework_cancelled code (FU-CFD1-DI03-TEMPORAL-DBOS) | terminates the owner instance, then K5 → run → stream failed, framework_cancelled (ruling D1); a K5 write that fails writes nothing and rejects with a retryable transport_error; result() / getHandle().result() report the persisted detail (DI-33) |
| Owner closed without a terminal commit | n/a (lease lapses; the next entry supersedes) | the wake (closed-owner reconciliation, then the drive: recover / continue through runDrive): a K5-landed run is finished and settled; a run whose suspension committed before its run status gets its suspended_* status, its stream paused, and settles suspended_*; an ended run's open stream is ended; otherwise it is re-driven under the crash-loop budget. A classified failure re-arms (transient, lastFailure recorded) or ends the run typed (refused / fatal; exhaustion is framework_auto_resume_exhausted with the last failure as cause) | recoverCommittedRun finishes a K5-committed run; otherwise the next entry supersedes it | settleDeadOwnerRun (reconnect / active handle, interrupt, abort) | every executor reader (result(), getHandle(), canResume(), abort()) and the parent's consult of a child reconcile it through the core planner (below): K5 landed → finished; suspension committed → reported suspended_*; nothing committed → failed (DI-34: terminated → framework_cancelled; errored → the engine's message; complete without K5 / gone → { message }); abort() executes only the finish and otherwise uses its own cancellation. A chat-only host reconciles too: handleChatStream / buildSnapshot call reconcileSession (W1 MR2b), which also starts an owed submit continuation |
| Closed-owner reconciliation (core planner) | its own finish-or-fail copy (FU-JS-RECONCILE-CLOSED-OWNER-ADOPT) | the DO wake (reconcileOnWake, WA MR1): the owner is closed when nothing is registered and no producer holds the claim; finish_committed / complete_suspension are executed, then settled (onRunSettled); repair_stream is executed (no hook); fail_uncommitted is re-driven (recover) under the crash-loop budget and executed only at a terminal rung (exhaustion, a permanent read) with that rung's detail | its own recoverCommittedRun (FU-TEMPORAL-RECONCILE-CLOSED-OWNER-ADOPT) | its own settleDeadOwnerRun (FU-DBOS-RECONCILE-CLOSED-OWNER-ADOPT) | planClosedOwnerReconciliation → reconcileClosedOwnerRun in every executor reader, the parent's consult (consultClosedChildRun), the owner's own R27 path (finishCommittedOutcome: finish / repair only) and the entry fail-safe once it stopped a stalled run's owner; an ended run's live stream is repaired by the next reader, and a stream that cannot be read or written never hides the outcome |
| Pre-commit entry failure (M1) | a session that has a run is untouched; a never-run session is recorded failed on the session only (failSessionWithoutRun: no stream write, no hook, R43) | via JS; also on /resume (I-D6: a request drive writes nothing before its run-start commit); a never-run session the wake finds is failed on the session only | untouched when the session has a run; a never-run session is saved failed and its stream failed (FU-TEMPORAL-NEVER-RUN-PRE-COMMIT-STREAM-WRITE) | records failed whenever no run is live (FU-DBOS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED) | untouched when the session has a run; a never-run session writes ONLY the session (R43, core failSessionWithoutRun); the failure write is retried by its step |
| Stop decision | core resolveStopDecision (step iterator), after the step's tools | via JS | shouldStopExecution with maxSteps only (DI-29, FU-CFD1C-DI29-DI25-TEMPORAL) | shouldStopExecution with maxSteps only (DI-29, FU-CFD1C-DI29-DI25-DBOS) | resolveStopDecision inside the LLM step's step.do (stopCause); acted on after the step's tools |
stopWhen fires | completed, completionReason: 'stop_when'; outputSchema → forced completion | via JS | ignored (DI-29) | ignored (DI-29) | as JS |
A throwing stopWhen | fails once (run-loop catch), after the step's tools ran | via JS | n/a (never called) | n/a (never called) | fails once (hook failure, never retried), before the step's tools run |
maxSteps cut-off (no outputSchema) | failed, framework_max_steps_exhausted, completionReason: 'max_steps' | via JS | completed, no output (DI-25) | completed, no output (DI-25) | as JS |
| Error stop reason at a stop cause | the step commits (step), then the run fails with the stop reason | via JS | completed (DI-26) | completed (DI-26) | no-tool step: its rows commit as an append and its staged state is discarded (a failing step is never a step commit), then it fails; tool step: committed, then fails |
retry() after a cut-off | continues from the last step, fresh budget, no message (C5) | via JS | refused: the run completed (DI-25) | refused: the run completed (DI-25) | as JS |
| String tool result | verbatim + helixContentEncoding: 'text' (RM-32) | via JS | verbatim + marker | verbatim + marker | verbatim + marker |
| Child instance ids | n/a | n/a | n/a | n/a | keyed on the parent run (respawnSuffix(runId); companions add the call id), C7 |
| Host entry pipeline | n/a (the caller calls the executor) | runDrive: claim and concurrentEntry → type / identity gate → materializeAgent → beforeEntry → buildExecutor → executor call + entry-outcome gate → settle path (every route, wake and submit continuation) | n/a | n/a | n/a |
| Host hooks | n/a | beforeEntry (admission, request drives), afterEntry (once per drive), onRunSettled (at least once per (runId, status), durable mark, redelivered on wake) | n/a | n/a | n/a |
| Consumer usage-store factory throws | n/a (the caller passes the store) | framework_usage_store_unavailable (retryable): a request answers 503, a wake re-arms; never a fallback to the internal store | n/a | n/a | n/a |
Client-tool submit continuation (W1 MR2b, CP-104) | caller_resumes | from the wake: new_run, deferred, consumed, else caller_resumes | caller_resumes (FU-TEMPORAL-SUBMIT-SELF-WAKE) | new_run { runId }: <suspended run>:continuation, the run the woken workflow continues in, or the current run once it committed (the OWNER session's run for a sub-agent-owned call; the handler pins against the root's run, so that attach re-anchors or hits the typed live-wait answer: FU-AISDK-ATTACH-PIN-OWNER-SESSION) | new_run: the submit creates or adopts the deterministic __submit__ instance (generations 1..3), which waits in its body for the suspended instance (continuationPredecessorTimeoutMs); resume({ mode: 'continue' }) adopts it, before or after its run-start commit; reconcileSession re-ensures it |
New message over pending client calls (abandonPending, D11) | ONE with_message entry (supportsAbandonOnResume) | via JS (/resume accepts abandonPending) | refused framework_not_supported (FU-TEMPORAL-ABANDON-SEND-STUCK) | refused framework_not_supported (FU-DBOS-ABANDON-SEND-MODEL-STEP) | ONE with_message entry; on a session already owed a continuation, the busy loser |
with_message over answered client calls (session paused on client calls alone; a crashed / interrupted session still puts the message first: FU-JS-WITH-MESSAGE-AFTER-CRASHED-DRAIN-ORDER; a children park: message first, model input paired, CP-237) | results (and a deferred finishWith's not-executed row) then the marked message, in the run-start commit (consume gate), or stashed on a pending entry when a tool must run first | via JS | message first, results after (FU-TEMPORAL-WITH-MESSAGE-DRAIN-ORDER) | message before the results (FU-DBOS-WITH-MESSAGE-DRAIN-ORDER) | drain, then the marked message as the entry's own append commit; after a submit owed a continuation, waits for it and is the next turn |
Companion sendMessage delivery (CP-105) | core CompanionDeliveryOutcome (exhaustive map of resumeChildAgent) | the child DO's /resume answer (child_busy, child_start_unconfirmed, child_entry_failed + errorDetail, child_exited) | delivered: true before the child's run starts; also answers child_start_failed outside the union (FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW) | waits for the run-start outcome; child_entry_failed carries errorDetail | the parent polls the child's exact run id in its body (companionEntryOutcomeTimeoutMs, 60 s); the ref moves only in the child's run-start commit (companionRef) |
Companion spawnAgent entry outcome and the held prior (CompanionSpawnResult, priorResult; spec 2026-10-15 D4, DO companion MR B) | every spawn on the core schema; a continue is opened only by its own run-start commit (CP-155), and a first spawn / respawn answers only after the child's run-start commit, its entry awaited and only its loop detached (CP-161); both answer every non-start typed (child_busy / child_terminated / child_entry_failed / child_start_unconfirmed, core companionStartOutcome), and a first spawn's child_busy writes no ref (R1), nor does a spawn that lost the parent ref's claim or reset (CP-161 fix 2: answered child_busy before any child IO); it holds the undelivered outcome before its commit and delivers it as priorResult; the step commit releases, the next iteration's held-prior pass settles the rest | every spawn on the core schema (the schema is load-bearing on the regular tool path); one entry table (childEntryOutcome) over a bounded /start (60 s); an unconfirmed start is re-probed, never respawned; the hold precedes beginChildConsult, the release is the next iteration's held-prior pass | the model sees core's spawn handoff, no priorResult (FU-COMPANION-SPAWN-HANDOFF-TYPED, FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY, parked) | bespoke handlers answer { name, status: 'spawned' }, no priorResult (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY, parked) | every spawn on the core schema (the entry poll's CompanionEntryOutcome on the child's exact run id through companionSpawnResultFromEntry: a continue, and since CP-161 a first spawn / respawn, whose non-start the body answers before any park and whose ref write is core's companionSpawnNonStartRefUpdate; the blocking park answers core's typed arm, companionSpawnResultFromTerminal); the continue dep holds, every record-step row carries priorResult, the release follows the row's appendCommitted, the per-turn notifier's held-prior pass covers a crash |
| Companion outcome delivery to the parent's next turn (CP-204) | the outlier (CP-207, open): the notifier reads a child only for a ref already terminal, and an in-process child's ref turns terminal only after its stream ended, so a turn that starts in between waits a turn (FU-JS-INPROC-COMPANION-REF-AFTER-STREAM-END) | the child ends its stream before it rings its parent, so the notifier's refresh runs first: every pass awaits the owed verified reads, and a run's first pass reads every outstanding consult itself (each waited at most 2 s, COMPANION_REFRESH_WAIT_MS); a slower child is delivered after its notice | the notifier reads the child at the pass (core completion-delivery) | the notifier reads the child at the pass (core completion-delivery) | checkCompletedChildren reads the child at the pass and syncs a stale ref |
handle.result(). On CF Workflows it waits, with no timeout, while the run's instance is live or operator-paused, and a failed status read is never cached (DI-18). On the other runtimes it awaits the run itself (runtime-js, the CF DO) or the workflow result (Temporal, DBOS). Once the instance has closed, that reader reconciles a live run (above): a run with no K5 and no committed suspension is settled failed (DI-34), a committed suspension is reported suspended_* with its suspended payload, and a failed result the instance did not return itself carries the session's persisted errorDetail (DI-33).
A failed stream's info. getStreamInfo(streamId) of a failed stream returns metadata.error and the structured metadata.errorDetail that failStream stored, on every built-in stream manager: memory, Redis, the binding-side DurableObjectStreamManager (its StreamServer /info), the in-process DOStreamManager, and the Worker-side DOStreamManagerClient (through the DOAB /snapshot's streamErrorDetail). The DO-side readers validate the stored detail with errorDetailSchema() and omit an invalid one. Contract case G5.fail-detail-on-info (RM-70).
Live statuses and the single live run. running and the three suspended_* statuses are live; every other status is terminal and sticky. A run-start commit supersedes every other live run (run.superseded-on-entry), so a session never has two live runs (run.single-live). A live run whose owner is gone (an expired or absent JS lease, a DO not executing it, a closed workflow or instance) is superseded by the next entry, so a session is never bricked by run_active (run.stale-running-supersedable). A pre-upgrade running row is supersedable the same way.
A resumed companion's parent ref (CP-103 A1, store contract). StartRun.resumedCompanionRef: { parentSessionId } is part of the store contract. A store applies it in the SAME atomic step as the run record, its rows and the input checkpoint: the parent's sub-session ref for the committing session becomes status: 'running', completionDelivered: false. The store checks the ref after the run-start rules. A refused start, or an idempotent replay (noop), applies nothing. A ref (or parent session) that does not exist fails the commit with RunStartRejectedError('resumed_companion_ref_missing'), non-retryable, and writes nothing. That cause is not a lost claim (isLostClaimError). It is a typed run-start rejection on every store, and on runtime-js a typed tool error (never child_busy). On Temporal the outcome matches the busy case (CP-105): the parent reports delivered: true, and the child workflow ends failed / state_run_start_conflict with details.cause: 'resumed_companion_ref_missing', having written nothing (FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW). On CF Workflows (W1 MR2b) the parent polls the child's exact run-start commit before it records the tool result, so the same refusal is reported { delivered: false, reason: 'child_entry_failed', errorDetail } (the resume instance closes failed / state_run_start_conflict with errorDetail.cause: 'resumed_companion_ref_missing'), and a lost start is child_busy. A ref that is terminated (the parent's companion__terminateChild landed before the resume's run start) fails it with RunStartRejectedError('resumed_companion_ref_terminated') and writes nothing, so a resume never flips a terminated ref back to running (WA MR2 Ruling 43; contract scenario commit.start-companion-ref-terminated; D1 pins status != 'terminated' in its claim guard). That cause IS a lost claim (the terminate won; classifyEntryFailure → lost): runtime-js answers the sendMessage { delivered: false, reason: 'child_terminated' }, and a CF Workflows resume instance refused this way did not start and notifies nobody (the parent's ref stays terminated); the parent's entry poll reports it { delivered: false, reason: 'child_terminated' } (WA MR2 §7.4). The field is not part of startDigest (rolling-deploy safety, like parent). Built-in stores: memory (applyRunStart), Redis (the commit script's single EVAL), Postgres (the run-start transaction) and D1 (the run-start batch, with the ref's existence pinned by the claim guard). DO-SQLite holds one session per Durable Object, so it refuses the field with UnsupportedOperationError and writes nothing. A new store or host must apply the field atomically or refuse the commit typed. It must never write the run without the ref. The contract scenario commit.start-companion-ref-atomic (core/src/testing/run-start-operations.ts) runs on every store; a store that cannot apply the field registers with resumedCompanionRef: 'refused'. Callers: runtime-js resumeChildAgent, Temporal initializeAgentState (companionResume) and CF Workflows startRunCommit (companionRef instance input, set on both the __resume__ and the __continue__ instance, refined to require parentSessionId). The namespaced CF DO keeps its own ref write after the child DO's /resume (different DOs, not atomic). Design: 2026-10-08-completion-hardening-fixes-design.md §2.8 (a).
Retry semantics (user ruling D1). Every terminal exit writes K5, and retry() never targets it: core planRetry picks the failed run's last step boundary (its latest step commit, or a later assistant-free append of that run), else the run's run-start commit (the rows rewind to just before the run's triggering user message, which the retried run re-appends, then options.message if given). An explicit checkpointId is restored as given. A first-step failure retries without a message (CP-58); a later-step failure needs options.message, except a maxSteps cut-off (completionReason: 'max_steps' on the run), which continues from its last step (or, with no step committed, its kept entry) with stepCount 0 (the full budget) and no message (C5). Every runtime runs it as a run-start commit with truncateTo. See ./session-model.md §Retry target.
Retry after a with_message resume (W1 MR2b, rules 4–6). A retry() after abandon-and-send, or after a drain-first with_message resume, works the same as any retry: when the run failed before committing a step no options.message is needed (rule 4); after committed steps it needs options.message, as above. Rule 1: the failed run's latest step commit. Rule 2: its entry (a run with its own trigger re-appends it; a hidden sub-agent completion notification or a memory injection is never the trigger). Rule 3: a maxSteps cut-off restarts with the full budget. Rule 4: a resumed run whose entry appended its own input row after its drain (marked COMMON_METADATA_KEYS.ENTRY_INPUT, found by marker, never by role) is rewound to just before that row and re-appends it and every row after it, restoring the state of the commit that contains it. Rule 5: while a message is still stashed on a pending entry (PendingClientToolCall.entryInput, runtime-js / the CF DO), the retry always drains first and re-stashes the input for the retried run (retryRunId); a given options.message replaces the failed run's own stashed rows. Rule 6: with answered pending calls trailing the kept log, the retry's input is stashed and drained after them; a stashed row that is not input (a hidden sub-agent completion notification) carries its own delivery marker (COMMON_METADATA_KEYS.STASH_DELIVERY), never the input marker, so its delivery is provable and a later retry never appends it twice. Rules 4–6 keep every tool_use adjacent to its tool_result and never lose or duplicate committed input. Every runtime that calls planRetry gets rule 4; rules 5–6 apply to the runtimes that pass retryRunId (runtime-js, the CF DO).
Stop decision and string results (MR 1c). resolveStopDecision checks terminal → stopWhen → maxSteps. Because a CF Workflows step that stops on an error stop reason with no tool calls commits its rows as an append (never a step commit), while JS commits that step and then fails, retry() of such a run (for example a content_filter stop on the cap step) targets a different checkpoint on the two runtimes: on JS the failed step itself, on CFW the step before it (or the run's run-start commit), and the failed step's staged state is discarded there. A string tool result is stored verbatim with the helixContentEncoding: 'text' marker by the shared builders, so the model sees hello, not "hello", on all five runtimes; readers decode with decodeToolResultContent.
A frame never decides a call's result (cross-runtime, CP-305)
Rows are the truth and the stream is a projection of them. Once a tool (or a child) has produced its result, a frame written after it (tool_end, state_patch, any chunk) is best-effort: a failed write is logged with its chunk type and call id, never its content, and the call's REAL result row is still recorded and the run goes on. A client that missed the frame re-anchors on its next snapshot read. A run fence on a write (RunSupersededError, CommitStoppedError, or a runtime's own, such as Temporal's superseded stop mapped to its ApplicationFailure) still throws: it decides the run.
The one helper is core's writeFrameBestEffort / bestEffortFrameWriter (store/best-effort-frames.ts); every runtime routes its post-result frames through it: runtime-js's regular tool path (the run loop's prepareServerToolCall), its tool-executor.ts paths (the approve drain, all-calls), its companion calls, its saved-step re-attach loop and remote dispatch, and its entry drains (deliverStash, settleLostStepRefs, the consumed entry calls) — the CF DO runs these; DBOS's tool step; CF Workflows' tool call path, its catch path included, so a failed frame never makes step.do re-run a tool body; Temporal's approve drain (its regular tool activity already wrote its frames with .catch). The re-attach loop's tool_start is covered too, though it is a pre-body frame: earlier calls' results are already known there, and a frame never loses them. Every other frame written BEFORE a body runs (tool_start, an approval request) is not covered yet (FU-PRE-BODY-FRAME-WRITES), and a DBOS sub-agent's writer setup and close are FU-DBOS-CHILD-WRITER-LIFECYCLE.
Superseded runs
A run is superseded when another entry took the session over (its run-start commit marked this run superseded), or when this run's own write hit the superseded-run fence, or (JS) when its lease was lost. The run stops with RunSupersededError (code: 'state_run_superseded', non-retryable everywhere), writes nothing more and does not touch the stream (its late chunks are dropped by the chunk fence).
- Result. Every runtime reports
AgentResult.status === 'superseded'witherrorDetail.code === 'state_run_superseded'(run.superseded-result-status); Temporal, DBOS and CFW map the non-retryable failure to it in their handle /getStatus, neverfailed. Breaking (pre-1.0): exhaustive switches overAgentResult.statusmust handle it. - Hooks. Neither
onAgentCompletenoronAgentFailfires, except in one window: when the supersede lands afteronAgentCompletefired and before the run's terminal (K5) commit,onAgentCompletehas already fired andonAgentSupersededfollows it (complete → superseded, user ruling "allow both, last wins"; the same shape as R40'scomplete → fail). The last terminal hook is authoritative. One exception on CF Workflows: an instance that closes between itsonAgentCompletestep and its K5 commit is settledfailedby the next executor reader with how it closed (terminated→framework_cancelled,errored→ the engine's message, gone →{ message }), and no further hook fires (a reader settle fires none), so thereonAgentCompleteis the last hook but not the outcome. Every runtime (JS, CF DO, Temporal, DBOS, CFW) behaves this way, proven by a contract test per runtime (FU-HOOKS-COMPLETE-THEN-SUPERSEDED-WINDOW). The optionalonAgentSuperseded(payload, context)hook fires instead, with{ error, finalState, stepCount, durationMs }(finalStateis the run's in-memory custom state; it is never persisted). - Tracing.
@helix-agents/tracing-langfusecloses the run span withlevel: 'WARNING'andstatusMessage: 'superseded', so no span is left open. - Consumers.
handleChatStreamdoes not treat it as a stream terminal: an attached client keeps the session stream open, because the superseding run streams on it (chat.superseded-keeps-session-stream). The CF DO drive classifies an entry's loss with coreclassifyEntryFailureand never rolls the session back tofailed: aRunSupersededError, aRunStartConflictErrorand aRunStartRejectedErrorwith a lost cause arelost(a request answers 409, a wake does nothing); a rejected run start answers 409state_run_start_conflictwith itscause. A late client-tool submit whose result append hits the fence answers 409{ code: 'state_run_superseded' }, never a 500. - Not the same as a step-commit conflict. A step commit that lost a version race (
StaleStateError; runtime-js / CF DO) fails the run withStepCommitConflictError; that run endsfailed, typed.
Stop as one signal (cross-runtime)
Every stop (interrupt or abort) on every runtime is one mechanism (WA MR2; design note ../superpowers/specs/2026-10-11-do-stop-as-one-signal-design.md, upgrade guide ../upgrade-guides/stop-request.md). The session-level interrupt flag is gone.
- Record, signal, answer at once. Core
stopRunrecords a run-scopedStopRequest(requestStop, keyed(sessionId, runId)), then asks the runtime'sStopOwnerProbewhether an owner is executing the run. If one is, it signals it and answers{ status: 'stopping', runId, kind }(HTTP 202). If nothing executes the run (it is suspended, or its owner is gone), corecommitStopTerminalcommits the stop's terminal now and the answer is{ status: 'interrupted' | 'failed', runId }(200). No live run is{ status: 'not_stoppable', currentStatus }(409state_not_stoppable), and nothing is written. A stop never waits for the run to wind down; the run's end is observed on its stream or status. - The terminal. An
interruptends the runinterrupted(resumable; every committed step kept). Anabortends itfailedwithframework_cancelled.commitStopTerminalwrites K5 (clearingpendingClientToolCalls), then the run status, then the stream (run_interrupted+ end, or fail), all fenced on the run; it never overwrites an outcome that already landed (already_ended). - A stop refuses new work, never the record of completed work (Ruling O-73). The result row of a call whose work finished (a tool that ran, a child outcome that arrived, a
finishWithor approved tool that ran, a drained client answer), with that tool's staged state writes, is committed run-fenced but not stop-guarded, so a stop never loses it (a superseded run still writes nothing); starting new work (a tool, a child, the next step) is gated on the run's stop record. The run still ends by its stop. With #181's Rule R this is one rule: a state write persists iff the row that records its work persists, so a stopped step's writes that no row records (itsafterLLMCallwrites, a call pairedrun_stoppedor held in a step commit the stop refused) are discarded on every runtime and for both stop kinds. - The stop pairs the run's calls that have no result (CP-118). What is left unpaired is work that never started, a wait the stop cut short, or a crash between the work and its record. The stop's terminal pairs, before its K5, every such trailing call with a typed
run_stoppednot-executed result (corepairUnfinishedCalls): abortedtool_endframes, then oneappendcommit under the stop's{ allowKind }, so the transcript stays valid (RM-29). A call is paired at most once: CF Workflows' sub-agent wave stops recording late outcomes once it sees the stop and awaits the records in flight before it pairs. On CF Workflows a stop the body reads in the middle of a step (before its sub-agents spawn, during a sub-agent or companion wait, before a suspension) commits that step as it stands under the stop's{ allowKind }, with its answered rows and therun_stoppedpairing of the rest in that samestepcommit (RM-69), so no checkpoint counts a call without its result; that commit drops the step's writes no row records (#181: itsafterLLMCallentry and the paired calls' keys, neutralised in staging) and flags its reply under an interrupt. Exception: pending client-tool calls (approvals included, an approved tool the stop kept from running too) stay unpaired until a resume drains them or a new turn settles them (RM-81), so a late submit is still answered{ status: 'unknown_tool_call', reason: 'run_stopped' }(Ruling 56). CF Workflows (commitTerminalState), runtime-js and the CF DO (the run loop's stop resolution andendEntryRunByStop) do it; Temporal does not yet (FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS); DBOS has no window (a step's rows land in one commit), except its approval continuation, which loses an approved tool's real result through a stop (FU-DBOS-APPROVED-RESULT-SURVIVES-STOP). A stop that corecommitStopTerminalcommits for a run nothing executes does not pair yet (FU-STOP-SETTLE-PAIRS-UNFINISHED-CALLS). Mechanics:./step-processing.md§Stop terminal pairing. - A failed run pairs the calls it still owes (RM-91). A suspended step's deferred calls (its
finishWithand, on CF Workflows, companion calls the suspension did not run, and its re-spawned children's calls) are owed work: while the run that owes them is live or resumable they sit trailing, unpaired and not pending, at the suspension commit and at the resuming run's own checkpoints. Aresumeruns them; aretry, a branch or a new turn heals them (C5). Every other unpaired call is still forbidden. A run that FAILS (no stop) before running them pairs themunpaired_on_continuationbefore its K5: runtime-js and the CF DO inonFailed(appendStopPairingwith that reason), CF Workflows in its catch-all (pairOwedCallsOnFailure). It is refused by a stop recorded meanwhile (the stop's K5 pairsrun_stopped) and is best-effort: a store that also refuses the append keeps the run's own error detail, and the next continuation heals. A run that crashes outside its loop (corefailCommittedRun) leaves them to the next entry's C5 heal. Temporal and DBOS do not pair at a failure yet (parked). - The stopped step is marked discarded (Ruling 66). A run stopped mid-step never commits that step (the stop refuses the step commit, a DBOS stop cancels the owner, the DO's deadline detaches the drive), but the step may already have streamed content. Every stop terminal marks the run's uncommitted steps
step_discardedbefore itsrun_interrupted/ failed stream, through one core rule,discardUncommittedSteps: each step the run streamed content for (its own stamped chunks; tool chunks whose call has a persisted row are committed) that has nostepcheckpoint of THIS run, exactly once (a step the run already marked or committed is skipped). It is keyed on the run's own commits, never on a step-count high-water: a turn's run-start checkpoint carries the previous turn'sstepCount. Callers:commitStopTerminal(the DO stranded / evicted stop, the stop deadline,settle_stopped, a run-start settle, a cancelled DBOS owner), runtime-js's in-run stop resolution (before itscleanupToStep), Temporal's refused step commit, DBOS's dead-owner settle. CF Workflows and DBOS's own loop write the same marker for the step they discard (on CF Workflows: a step whose commit a stop refused while it held only phase-1 results; a step that already holds sub-agent, remote, companion or phase-2 results is completed work and commits under the stop instead, RM-69). Temporal's terminal also writesrun_interruptedwhen a run ended by its refused commit has none. - An interrupt keeps the reply it cut off (#181). This is about a reply the stop CUT OFF: one stopped mid-model, or a step stopped before its reply was saved. On runtime-js and the CF DO a tool step's reply is saved before any of its tools starts (Save-First Tools, spec 2026-11-05 §6); a stop after that keeps the complete saved reply with no flag, and its step commits run-fenced with the results that came back (below). Otherwise the stopped step still never commits as a step, but an
interruptkeeps its reply as an assistant row flaggedmetadata.helixInterrupted = { stop: 'interrupt' }(coreisInterruptedMessage;interruptedPartialRows): the answer text and tool calls only (reasoning andpartsdropped), each call pairedrun_stopped, in oneappendcommit under the interrupt's own guard, before the K5 and after the step'sstep_discardedmarker. An empty reply keeps nothing; anabortkeeps nothing (D4), and an abort that upgraded the interrupt refuses the append. runtime-js, the CF DO and CF Workflows keep what the run produced up to when the server received the stop (more than the client rendered whenstop()'s request was still in flight: it does not wait for it), whatever the stopped call answered: the adapter's cut-stream result, or, when it answered no text of its own (a cancelled error, a provider error raised while aborted, a throw), the text deltas the step itself forwarded to the stream (nothing when none streamed). CF Workflows also keeps the whole reply of a step whose commit the interrupt refused; Temporal and DBOS never cut the call, so they keep the complete reply (FU-181-TEMPORAL-DBOS-STOP-ABORTS-CALL), and DBOS's cancelled-owner settle, with no row in hand, keeps the step's streamed text (interruptedPartialFromChunks).buildMessagesForLLMomits every flagged reply that no user message follows, with its paired results (withoutUnansweredInterruptedReplies; a message-less continue answers afresh in each of its steps, D1); once a user message follows it is replayed like any assistant turn. Arun_stoppedrow says the call's result was not recorded and the tool may have run (in a step the stop refused, a tool that ran is paired too, FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS, closed on runtime-js and the CF DO by Save-First Tools §6: after the saved reply a stop commits the step with the tool's real result and its staged writes; CF Workflows, Temporal and DBOS remain). The ai-sdk reload merge carries the flag (stockUIMessagemetadata, no new chunk). - The record is per run and never cleared. A stale stop never stops a newer run.
abortupgradesinterrupt(keeping the earlier deadline), never the reverse, and the first stop of a kind wins. - The store enforces it. A live run's step, append and terminal commits carry
refuseIfStopped, refused atomically (CommitStoppedError,state_commit_stopped) once a stop is recorded; the runtime then commits the stop's terminal instead (coreresolveTerminalWithStop/commitTerminalHonouringStop). So a stop recorded before the run's terminal commit always wins, even over an adapter that ignores the signal (CP-59). A stop's own K5 commits with{ allowKind }, so an abort that upgraded the interrupt meanwhile is never overwritten. A run-start that would supersede a live, stopped run is refusedexpected_run_stopped: a suspension continuation ends the stopped run by its stop and does not proceed; any other entry settles it and proceeds. - Closed owners. The closed-owner planner has a
settle_stoppedrung, afterfinish_committed(a run whose K5 landed keeps its outcome): a live run with a stop record is ended by the stop, never re-driven. - Handles (
interrupt()/abort()) stop their own run and resolve once the stop is durable (D3), also when the run already ended. On the CF DO the client handle sends itsrunIdto the stop route, which records withtarget: { kind: 'run' }; a named run that is no longer live is answered without any write (its terminal when it endedinterrupted/failed, elsenot_stoppable), so a stale handle never stops a newer run (F4-A P3). CF Workflows'abort()names the handle's own run the same way: a handle whose run is no longer live writes nothing, and only its own still-open instance is terminated (F4-B P3). A DBOSabort()over a session with no live run answersnot_stoppableand writes nothing, except for a persistent loop idling between turns that is still open: that loop is shut down and the session markedfailed(F4-B P2). - A session stop can name its run (K73, FU-DO-FRONTEND-GETHANDLE-BY-SESSION).
stopSession,SessionStopExecutor.stop(runtime-js), ai-sdkhandleChatStop,CloudflareChatHandler.stop, agent-server's/chat/{id}/interrupt|abortand the CF DO's own routes take an optionalrunId(corestopRunIdSchema: at most 512 characters). CoreanswerNamedRunStopis the one rule (lifted from the DO's routes): a live run of this session is stopped; one that already endedinterrupted/failedanswers that terminal with nothing written; a completed, superseded, unknown or FOREIGN run isnot_stoppable. So a host that holds run A never stops a newer run B.stopSession's handle path (an executor with no session stop: Temporal, DBOS, CF Workflows) signals the handle only while the named run is the session's current run, checked before and after the bind (FU-NAMED-STOP-OTHER-RUNTIMES wires their own session stop). - One default reason. A stop that names no reason records core
DEFAULT_INTERRUPT_REASON('User requested interrupt') orDEFAULT_ABORT_REASON('Aborted by user') (stopReasonOrDefault): runtime-js handles andstop(), agent-server's stop routes and the CF DO's routes (F4-A m3), the CF Workflows and DBOS handles (F4-B m3), and the Temporal handles (F4-C m3) use it. - Companions.
companion__terminateChildis anabortstop of the child's current run (stopPersistentChild). It answersterminated: trueonly once the child's stop is RECORDED: the callback resolves for a recorded stop (whose signal or terminal may still have failed: the record ends the child) or a child with nothing to stop, and throws otherwise; a throw is a tool failure (CompanionTerminateNotRecordedError) with nothing written (F4-A S1; CP-161 fix 4: the stop is recorded BEFORE the ref is writtenterminated, so there is no rollback and no generation bump, and a ref write that fails after the recorded stop leaves the child's own end at its generation). The CF Workflows and DBOS callbacks rethrow a failure only whenreadStopshows no record, and log one that came after the record. Where no record can be written, the stop itself must land: a CF Workflows child instance that never started and whose terminate failed while it is still open, or a DBOS idle loop whoseshutdownsend failed, throws (F4-B S1).companion__sendMessageto a running child is anappendonly; it never stops the child. - The cascade names its run (CF DO, F4-A D3). A stopped run's cascade reaches the live children that run started or reopened, and those an earlier run left running, never a child a later run started: a delayed or redelivered cascade covers what it would have covered at the run's end. Its done mark is written only when its reads succeeded (F4-A S2); otherwise it stays owed for the next wake.
- An owed continuation names its run (CF DO, F4-A D1). runtime-js
resume({ mode: 'continue', owedRunId })continues only while that run is current and suspended, else it is the silent loser (RunStartConflictError,lost): the DO's submit continuation and its CP-52 / unconsumed-submit wake pass the run they were owed for, so a stop that ends the run before the wake's entry is never followed by a heal and a model call. retry()after a stop of a paused run. The stop's K5 cleared the run's pending calls, so the retry's kept prefix can end with an unpairedtool_use; the retry pairs it with a not-executed result in its run-start commit (the C5 heal). runtime-js and the CF DO (F4-A P1), CF Workflows, DBOS and Temporal (F4-C P1) all do.
| Concern | JS (runtime-js) | CF Durable Object | CF Workflows | Temporal | DBOS |
|---|---|---|---|---|---|
| Owner "executing" | registered in this process, or running under a lease with leaseUntil + leaseTtl > now (Ruling 24) | a registered drive executing the run | the run's instance is open | the workflow execution its owner token names is open (describe(); a failed describe counts as open) | the run's workflow is live |
| Signal | aborts the run's controller with the StopRequest | aborts the drive's controller with the StopRequest | wakes the instance with an event | INTERRUPT_SIGNAL_NAME with the StopRequest; an abort also cancel()s | the control message, then cancelWorkflow; the stop then settles the run after 500 ms |
| Where the run reads the record | pollStop at every step boundary | via JS | a step.do at every step boundary, during a sub-agent wait, and after a suspension commit | readStopActivity at every main-loop iteration and at a resume's entry | readStopStep at its boundaries |
Stop deadline (deadlineAt) | none (null; FU-STOP-DEADLINE-NODE), unless the executor's stopDeadline option is set (the DO) | stopDeadlineMs (default 30 s) on every stop it records, handle stops included: a durable alarm ends the run and detaches its stuck drive | none | none | none |
| A stopped run whose owner closed | the next entry settles it; a stop settles it once its lease is more than one TTL past | the wake's settle_stopped | every reader's settle_stopped (settleClosedOwner) | the cancelled workflow's own K5; a handle reading its result; the next entry (FU-TEMPORAL-STOP-SETTLE-CLOSED-OWNER) | the stop call itself, after the cancel |
Handle abort() | resolves on the answer | the client posts /abort; typed errors | terminates the instance and settles the run, then resolves | resolves once recorded, signalled and cancelled; the workflow commits its own K5 (F4-C m4) | cancels the workflow, settles after 500 ms |
| Child cascade | in-process children; remote children when the parent was not executing, together, each bounded 10 s (CP-156); an ephemeral child ended here gets its parent ref terminal (CP-206); a run that FAILED with no stop stops only its ephemeral children, its persistent companions keep running (CP-201); a stop recorded while it failed (the stop won its K5) cascades as a stop | on the settle path, the stopped run's live children (and an earlier run's) through their own routes, never a later run's; a durable mark once its reads succeeded, re-run on wake; a run that FAILED with no stop record stops only its ephemeral children, its companions keep running (CP-201); one that failed under an interrupt record was stopped live and cascades as a stop, with the interrupt | the pending children of a sub-agent wait, each on its derived run id; live children when a stop ends a run nothing executes (CP-152: the executor's stop / abort, a reader's settle, the body's own stop for a child none of its dispatches or waits runs): remote ones bounded 10 s, an unanswered one owed on the ref (stopOwed, retried at later entries and cascades; its own pacer is CP-217 MR 2); local ones (CP-159) by stopping their current run, a suspended child's own children after it; a reader's settle of a run that FAILED with no stop (owner closed) leaves its persistent companions running (CP-199, companions are session-scoped); a body that FAILS runs the cascade in its own <agentType>-error-stop-children step after RM-91's pairing and before the K5: with no stop record only its ephemeral children (CP-201), always, even once a newer run took over during the level; with a stop record a stop's cascade (companions included, the stop's kind), each child owner-checked (a continue resume adopts a stopped run's suspended child); a stop recorded after that read, or over a planned completion, lands at the K5 and is cascaded as a stop right after it (<agentType>-stop-children-after-stop-<tag>) | no stop-record cascade (a companion terminate records on the child) | no stop-record cascade (a companion terminate records on the child) |
Remote stops owed (CP-217)
A parent's stop of a remote child is at-least-once (spec 2026-10-26-cp217-remote-stop-owed-design.md, user rules R1–R4). Every site that stops a remote child goes through core stopRemoteChild (bounded, never throws), naming the delegation: a remote / ephemeral child's dispatch ({ dispatchId }, which the child checks against its stored ChildBinding), or a companion's consult ({ parentToolCallId }, against its run's RunMetadata.parent). A mismatch writes nothing and answers state_stop_target_mismatch. A stop the child does not confirm is a debt on the parent's ref (SubSessionRef.stopOwed: { kind, at, target, reason, owedByRunId, attempts?, nextAttemptAt?, abandonedAt?, refusalLogged? }), written in the same ref write the site makes anyway, independent of the ref's status. Confirmed is core isRemoteChildStopConfirmed: answered, state_not_stoppable, state_stop_target_mismatch or no session. One retry, core retryOwedRemoteStops, serves every debt (live or settled), on core's OwedStopSchedule (default 30 s doubling to 1 h, abandoned after 24 h). Every debt write is a compare-and-set on the status and the debt it read (UpdateSubSessionRef.expectStopOwed, every store). Pacers run core owedRemoteStopsSource, a DueWorkSource that is neither run-fenced nor stop-guarded.
| Runtime | Sites that owe | Pacer |
|---|---|---|
| runtime-js | in-flight dispatch (parent stop: ref stays live; failed call), cascade, D3, CP-178, D4 | an in-process timer per session (unref'd), every entry; executor.wake replaces the trigger |
| CF DO | runtime-js's sites, plus its settle cascade (companions and remote children) | the stop-cascade alarm (companions through their route) |
| CF Workflows | cascade (live refs), D3 | entries and cascades, settled refs only (its step.sleep pacer: MR 2) |
| Temporal, DBOS | none (one stop, FU-TEMPORAL-/FU-DBOS-REMOTE-CHILD-STOP-OWED) | none |
The CF DO's cascade mark is written once every child is confirmed or durably owed (an amendment to MR C1 O1).
The post-commit child phase (runtime-js, CF DO)
A companion call (companion__*, core isChildPhaseCall) runs only after the commit that holds its tool_use (CP-106 MR A; spec ../superpowers/specs/2026-10-14-cp106-post-commit-child-phase-design.md, upgrade guide ../upgrade-guides/child-phase.md). The step announces every call in toolCalls order, commits (the step commit, or the suspension commit when the step also suspends), and then core runChildPhase runs its companion calls one at a time in call order. A blocking wait lets the next call start at onStarted. Results append as append commits in call order, each call's tool_end streamed right before its append (so the commit's checkpoint cursor covers it; a stop-refused append writes a stream_resync rollback). Each call:
- skips itself when its result is already committed (the replay check);
- writes its ref through the fenced
addSubSessionRefs(…, { fenceRunId, refuseIfStopped: true }), which stampsparentRunId(all five stores); - starts its child keyed on the committing run and its call id: ONE key per call, core
childStartRequestKey, for a start, a continue and a resume alike (D-2; a re-entry that routes differently still meets its own run), so a re-entered call never starts a second child run; - re-reads its run's stop record (the start fence): a stop stops the child by its consult and ends the phase with
CommitStoppedError(on the CF DO a child stop that does not land is owed to the run's stop cascade).
A step's finishWith runs after the phase, in the same iteration. A CF DO run recovered in the phase (a recover wake), and a resume() of a run that suspended with unfinished companion calls, re-enter the phase for them. A row fires onMessage and its deferred afterTool right after its own commit, in a step's phase and a re-entry alike (CP-274, core runChildPhaseWithHooks; what a hook can rely on across a crash: the hooks system). A stop the run executes pairs an unfinished call run_stopped (CP-118); otherwise the next entry's C5 heal pairs it unpaired_on_continuation. Remote and ephemeral sub-agent calls and CF Workflows join the phase in MR B; Temporal and DBOS still run companion calls inside the step (FU-TEMPORAL-CHILD-PHASE, FU-DBOS-CHILD-PHASE).
State size limits (cross-runtime)
A state or message write too large for the store is never lost silently (MA-31; design note ../superpowers/specs/2026-10-18-oversized-tool-state-typed-design.md, upgrade guide ../upgrade-guides/oversized-tool-state-typed.md).
- One error, at the store. A store declares
SessionStateStore.limits(StoreLimits:maxStateBytes,maxStagedWriteBytes,maxMessageBytes;Infinity= none) and throws coreStateSizeExceededError(state_size_exceeded, categorystate, non-retryable, HTTP 413) at the site that binds the bytes. DO-SQLite, D1 and Redis refuse before they bind; Postgres declares no limit and maps the SQLSTATE54000the database raises after the bind (the statement writes nothing). DO-SQLite and D1 declare onemaxStateBytes(default 1.8 MB) for every operation; Redis onlymaxStagedWriteBytes(1.8 MB, a Lua-time budget); Postgres and memory none. A driver "too big" error is mapped to the same error only by measured bytes (BoundSizeWitness.mapDriverError), never by its text. Recognise it by class or code, never byoperation. - Staging → a typed failed row; the run continues. A tool that succeeded but whose write the store refused is recorded (O-73) as a failed tool result with
metadata.helixSynthetic: { kind: 'state_write_rejected', … }(keys and byte counts, never values); its writes are discarded, nostate_patchis written and siblings are unaffected. A tool that threw is never that row. - A tool result too large to store → a typed failed row; the run continues. A tool or sub-agent result row over the store's
maxMessageBytes(runtime-js and the CF DO) or over CF Workflows' 1 MiB step-result cap is replaced by ONE core row,createResultTooLargeToolResult: a plain failed row (helixToolFailed: true, no other marker) saying the call ran but its result was too large to record (O-73), plus the typedStateSizeExceededError(operationstep) its hooks get. The measure is the store's own: runtime-js measures the row with coremessageRowBytes(fitResultRow), the same bytes the DO and D1 stores refuse a row by (BoundSizeWitness.assertMessage), beforetool_end, the run's messages andafterTool/afterSubAgent; the store's refusal stays the backstop. The texts differ only in the measure clause ("the recorded row would be N bytes, over the M-byte message limit" vs "the conservative measure passed N bytes, over the 1048576-byte step-result limit"). Every runtime with a finite message limit must measure at each of its result-row sites (not inherited from core): runtime-js does it in the run loop's server-tool and companion paths,runSubAgentCall,executeSingleRemoteSubAgentCall,executeSingleToolCall(the approve drain), the resume drain, thewith_messageentry rows, and where a child's outcome is injected (the step boundary,reconcileRemoteSubAgents), for success and failed rows alike (a failed row too large to record says "its error"; a thrown tool, abeforeToolhook error, a failed child); CF Workflows insteps.ts/workflow.ts(A11). - A row the runtime injects → the typed replacement, delivered once (A13). A hidden persistent-companion completion notice is fitted by core
fitCompletionNotice(runtime-js / the CF DO and CF Workflows fit each keyed notice their notifier builds;buildPendingCompletionNotifications(refs, results, { maxMessageBytes })does the same for its callers), which measures each notice row (messageRowBytes; the output is escaped twice, so the row is ~2× the output) and replaces one over the limit withbuildCompletionNoticeTooLargeMessage(Sub-agent '<name>' completed, but its result was too large to record: …/… failed, but its error …): still one hidden completion notification (isSubAgentCompletionNotification, the same metadata as the notice it replaces, its companion keyhelixCompanionNoticeincluded, so a recovery still finds it committed), so it is appended and the child marked delivered (MA-37: runtime-js / the CF DO used to swallow the refusal and retry it every iteration). CF Workflows does it INSIDEcheckCompletedChildren's step and then fits the whole step result under the 1 MiB cap (fitCompletionNoticesInStep, the largest notice replaced first; MA-38: an oversized notice used to fail the turn (the check step over the cap, or the inject step's D1 refusal) on every later turn). DBOS / Temporal pass their store's limit throughdeliverPendingChildCompletions(a no-op on stores without one). CF Workflows also fits a sub-agent result record (recordSubSessionMessage: the row againstmaxMessageBytes, the fold against the cap; the remote error record and its last-resort row, the respawn record and poll), a tool row againstmaxMessageBytes(the approve drain, a step's tool rows, and every row of completed workappendCommittedrecords withrecordsCompletedWork, a companion result included), and an acceptedsub-agent-completeevent (fitChildOutcomeInStep) (MA-31). The other injected rows are fixed short texts (heal and not-executed rows, stop pairing, synthetic finish acknowledgements); memory injection (MA-39) and the persistent-consult stranding (CP-155) are filed separately. - A commit → the run fails typed (runtime-js, the CF DO, CF Workflows). A step commit over the limit (the fold, the model's reply) fails the run
state_size_exceeded, and its failure terminal commits the stored state, so the session never strandsactive; these runtimes never retry a size refusal (the same bytes fail again). On Temporal and DBOS a commit refusal is unverified: there it can only come from Postgres's54000(their usual stores declare no finite limit), Temporal'scommitStephas no size conversion (its activity retry policy, 3 attempts, likely repeats it, and the run would end with DI-04's activity detail), and DBOS drops the code (DI-24) (FU-TEMPORAL-DBOS-COMMIT-SIZE-REFUSAL). - Every runtime must turn a staging refusal into the typed row at its own tool site (not inherited from core): runtime-js
stageToolWrites(the run loop and the approve drain, which the CF DO runs), CF Workflowssteps.ts, TemporalexecuteToolActivity, DBOSrunExecuteTool.
| Concern | JS / CF DO | CF Workflows | Temporal / DBOS |
|---|---|---|---|
| A refused tool write | the typed row; afterTool gets the StateSizeExceededError | the typed row, no step.do retry; afterTool gets the error | the typed row; afterTool gets new Error(<the row's text>) (the result crosses a JSON boundary) |
| A tool that threw | its own failed row; its writes are never staged | its own failed row; its partial write is staged, and discarded when refused as too large (FU-CFW-FAILED-TOOL-PARTIAL-WRITES-PARITY) | its own failed row |
| Pre-sizing the fold | persistViaSaveAndPromote measures the folded state before the step commit (finite maxStateBytes only) | the store's check | the store's check |
| An oversized tool result | the core typed failed row (the row over the store's maxMessageBytes, core messageRowBytes); the run continues | the same core row (over the 1 MiB step-result cap); the run continues | no message limit on their usual stores (Redis, Postgres); not converted on these runtimes |
| A commit refused | fails the run typed; failure K5 commits the stored state | a NonRetryableError step failure; the run fails typed over the stored state | unverified: no size conversion at commitStep (Temporal); DBOS persists { message } only (FU-TEMPORAL-DBOS-COMMIT-SIZE-REFUSAL, FU-DBOS-SIZE-ERROR-DETAIL-CODE) |
| The engine's own result cap | none | every step result vs 1 MiB (CFW_MAX_STEP_RESULT_BYTES): a tool / child result → a typed failed result, the run continues; an LLM step or any other result → the run fails typed | none (Temporal's deferred-hooks staging: FU-TEMPORAL-DEFERRED-HOOKS-STAGE-SIZE) |
| A submit over the store limit | 413 state_size_exceeded, nothing stored; maxResultBytes over the store limit refused at construction | the same | no construction check (their usual stores, Redis and Postgres, declare no finite maxStateBytes) |
LLM history loading, checkpoint rewind and branch (cross-runtime)
Five step loops assemble the model's input (core step-iterator for JS and CF DO, CF Workflows steps.ts, Temporal runLLMStep, DBOS shared.ts). They all load history the same way and resolve checkpoints the same way, through shared core primitives in packages/core/src/store/:
loadLLMHistory(store, sessionId)is the only way an LLM path reads history. It pages the whole log (no cap, no window) and checks the result againstgetMessageCount, a live row count on every store. A short read that survives the bounded re-reads fails the run witherrorDetail.code === 'state_history_incomplete'(non-retryable) and no model call (C9). A store that throws while paging gets the same code withretryable: true, and the runtime's retry policy applies (Temporal activity retry, DBOS / CF Workflows step retry); JS has no automatic retry, so there the run fails andretry()recovers.resolveCheckpoint(store, sessionId, checkpointId?)runs before any CAS, version bump, clone or workflow start. An unknown id throwsstate_checkpoint_not_found; another session's id throwsstate_checkpoint_foreign(C11, CP-38).assertRewindable(session)is an up-front liveness check on Temporal and CF Workflows: anactivesession is rewindable only when suspended on pending client-tool / approval calls; otherwise it throwsAgentAlreadyRunningErrorbefore any write. On every runtime the run-start commit's owner guard is the real gate: a rewind never truncates the log under a live, owned run.planRewindToCheckpoint({ sessionId, checkpoint, history, current })plans everyfrom_checkpointrewind (pure): the keep-count, the rewound state under the current session's identity, and the pending calls kept, restored and discarded. The rewind itself is part of the rewinding run's run-start commit (StartRun.truncateTo, with the pending re-stamp): the store truncates, writes atruncatecheckpoint attributed to the new run and creates the run in ONE transaction, so there is no separate claim, hold or half-rewound state (ruling R21; see execution-flow).rewindToCheckpointsurvives only as the in-run building block for custom loops; the built-in entries do not call it.loadBranchSource(store, branch)is the branch history for runtimes that do not usestore.cloneSession.checkpointIdwins overmessageIndex;messageIndexis a count of leading messages.
Entry input seeding: history and state (cross-runtime)
AgentInputObject.history and AgentInputObject.state SEED a new session only: one whose message log is empty (a new session, a branch target, a session row created ahead of its first run). On a continuation of a session that already has messages, both are ignored and the runtime logs <runtime>.history_dropped_on_continuation / <runtime>.state_dropped_on_continuation; the state warn names the dropped keys, never their values. The committed state stands; to change state on a continuation, send a statePatch with the turn (below), or write it from a tool or a hook (U23 / CP-223).
| Runtime | history | state |
|---|---|---|
| JS, CF Durable Object | seeds an empty log only | seeds an empty log only |
| CF Workflows | seeds a genesis turn only | seeds a genesis turn only |
| Temporal | seeds a fresh entry only | seeds a fresh entry only (temporal.state_dropped_on_continuation; CP-228) |
| DBOS | seeds an empty log only | never applied, even to a fresh session (CP-224) |
Child identity (cross-runtime)
A child's session id and its delegation's dispatch id are minted, never derived (spec 2026-10-25-child-dispatch-and-identity-design.md, MR 1). Every runtime upholds:
- Ids. A child-starting call's
dispatchId(d_<uuid>) is minted as the step's assistant row is built and persisted on the call; the child's session id ischild_<uuid>. Nothing parses either. Replay-safe minting per runtime: runtime-js and the CF DOcrypto.randomUUID()(coremintChildSessionId/mintDispatchId); CF Workflows inside the registerstep.do(journaled; a re-executed step reuses the ref already registered for the dispatch); Temporalwf.uuid4()in workflow code (children) and therunLLMStepactivity (dispatch ids, recorded with its result); DBOSDBOS.randomUUID()(checkpointed), one at a time before the concurrent fan-out. Ids built FROM a child id stay (child workflow ids,${childSessionId}-run-1). - The binding. A child session is created with its delegation's
ChildBinding, written once; the parent comes from it (MR 1 interim:parentSessionIdstays alongside, equal to it). Every store applies coredecideCreateBindingatcreateSession(same_delegationproceeds,takenisChildSessionTakenError) anddecideEntryBindinginside its run-start commit (a mismatch isRunStartRejectedError('child_binding_mismatch'), nothing written). - Only its parent enters a child. Every entry on a bound child carries the binding: the parent's spawn and re-spawn (CF Workflows' and Temporal's resume drain re-spawn the child with its stored binding through core
respawnBindingFor, only when it names this parent and call; every re-spawn site is listed inrespawn-binding-sites.test.ts), a remote parent's/start//resume, an app entry with an explicitbindingoption. The one exception is a runtime's own continuation of the child's existing work, which adopts the stored binding through coreadoptStoredBinding, pinned to the run it read (the run-start CAS keeps it pinned; it never starts a new turn): the CF DO's wake continue and stranded reset (runtime-jsowedRunId/strandedRunId), CF Workflows' owed__submit__continuation, DBOS'skind: 'continuation'entries. The callers are allowlisted byadopt-stored-binding-callers.test.ts. - One child per dispatch. An ephemeral
SubSessionRefrequiresdispatchId; a second ref for one dispatch under a parent is refused (assertNoDispatchClash). A remote child's/startkeys on the dispatch id, its re-attach/resumeondispatchResumeRequestKey. - The guard.
no-derived-child-ids.test.tsfails on any session id built from a parent id plus-sub-/__sub__/-remote-, and on any call ofgenerateSubSessionId(removed from core in this release). Companions keep their(parent, name)ids until MR 3 (allowlisted). - Gaps. Temporal and DBOS start their children before the step commit (no child phase yet), so their dispatch id is not durable before the start until FU-TEMPORAL-/FU-DBOS-CHILD-PHASE land; a recovery reads the minted ids back from the workflow's own history either way.
Per-turn entry state and start signals (cross-runtime, #177)
A turn's entry state (StartRun.entryState: its forced-completion phase, a new turn's cleared output, its customState with the entry's statePatch applied by core planEntryState, and its turn fields turnId / turnStart) is written inside the run-start commit, so the turn's input checkpoint records it. owesStartWork(state) (core) decides whether a run must run the turn's start work: the session has no turn yet, or its committed start (turnStart) names another turn. The start work runs once and lands in a start commit (run-fenced, not stop-guarded) before the first model call; it is written only when the agent has an onAgentStart or onMessage hook (HookManager.hasHook). A drain-first with_message entry stashes its patch with its message on a carrier pending call (PendingClientToolCall.entryStatePatches); every delivery plans through core planStashDelivery and commits the message with its patched state in one append commit (MessageCommit.entryState version CAS, consumePending).
| Runtime | statePatch | start commit | Start signal per run |
|---|---|---|---|
| JS, CF Durable Object | applied in the run-start commit | yes | onAgentStart (turn), onAgentResumed (continue / rewind), onAgentRetry (retry); owed start work: onAgentStart with startSignal: false |
| CF Workflows | applied in the run-start commit | yes | as JS |
| Temporal | refused framework_not_supported | no | unchanged: onAgentStart for every non-resume run (startSignal: true), no onAgentRetry |
| DBOS | refused framework_not_supported | no | unchanged, as Temporal |
Temporal and DBOS follow in #177 MR 2 (FU-177-MR2-TEMPORAL-DBOS-ENTRY-STATE).
Context rows, the context pass and the tool gate (Additive Sessions)
App guide: ../guide/changing-context.md. Spec: 2026-11-05-additive-sessions-context-pipeline-design.md.
- The row.
role: 'context',kind∈instruction|notice|tools|memory, asource, and on the pass's rows its keypass: { runId, step, attempt }. Atoolsrow must name its source (the schema refuses one without). Stored on all five stores like any message; skipped by UI messages, snapshots and the chat stream; placed in the model's input at every runtime'sbuildMessagesForLLMcall site (MR 3). The adapter renders rows from'app'/'hook'in a system form and every other source as a user turn. - The queue.
ctx.addContext(a tool: source forced to'tool'; a hook: defaults to'app') records items through corerecordContextItems: validated, size-checked (CONTEXT_ITEM_MAX_BYTESper item), insert-once per key, fenced on the live run.'hook'is reserved for the pass's ownstepContextrows. An app'sexecutor.addContextrecords through the same recorder, and itskeynames one request: the same items record nothing, other items (changed, more or fewer) are refusedstate_context_key_conflict, decided by the request digest stored with each item (StoredContextItem.batch). Tool and hook keys stay first-wins (a re-executed body repeats them). - The pass (
runContextPass, coreorchestration/context-pipeline.ts) runs before every model call — a step's first attempt and each in-step retry (a newattempt) — after the step's settle stage and before the call. It builds, in order: late answers' notices (source: 'late-answer'), queued items, the memory row (once per pass, deduped by memory id; an oversized recall is an empty row keeping its ids), and thestepContexthook's batch (from the cursor on the log's last hook row; fit toSTEP_CONTEXT_MAX_BYTESwith a left-out marker; skipped on a throw, a time-out or an invalid result). It commits them in ONEappend(no assistant row, so aretry()target extends past it) withrefuseIfStoppedandexpectMessageCount; on a count conflict it reloads and commits once more without calling the hook again. A log that already holds a hook row with this pass key skips the hook (§9.3). Each skip writes acontext_skippedchunk (dropped by the ai-sdk transformer) and logscore.context.skipped. - The gate (
gateStepCalls, coreorchestration/tool-gate.ts) runs after the pass and before any classifier: a call to a tool switched off (tool_inactive) or not declared (tool_unavailable) gets a typed not-executed result and no body, hook, approval request, client ask or child. The active set isfoldActiveToolsover the log:disablefrom any source;enableonly from'app'/'hook'(TOOL_ENABLING_SOURCES, an allowlist). A forced call's expected tool is never refused;__finish__can never be switched off. The approve drain (a resume, before the loop's first pass) folds the log AND the session's undeliveredtoolsqueue items (foldActiveToolsWithQueue, read only): an approved call to a tool switched off while its approval waited is pairedtool_inactiveand never runs (ruling 2026-11-09). - Hook firing order. The pass (and so
stepContext) runs beforebeforeLLMCall;beforeLLMCallsees the rows. - Rewind and branch (D13). A truncate removes context rows and nothing re-appends them: queue items re-open, the cursor reverts to the last surviving hook row, memory is recalled again, the fold follows the surviving log;
planRetrynever re-appends acontextrow. A branch copies rows (cursor and fold included), never the queue. - Per runtime. runtime-js and the CF DO run all of it. CF Workflows (MR 5), Temporal (MR 7) and DBOS (MR 8) do not yet: an agent tree that sets
stepContextis refused at every root entry, typedframework_not_supported, before anything is written (corerefuseUnsupportedAgentFeatures, the sharedUNSUPPORTED_AGENT_FEATUREStable), andctx.addContextrejects typed (unsupportedAddContext[Binder]). Their late notices and memory keep their pre-pass shapes (hiddenuserrows). A new runtime must callrunContextPassat its model-call site,gateStepCallsbefore its classifier (writing each refused row'stool_end), wirectx.addContextthroughtoolAddContext/hookAddContextBinder, and remove its phase from the table's rows.
The stop guard (runtime-js, CF DO)
AgentConfig.stopGuard (design 2026-11-07; guide Stop Guard) is app code called when a run would end on a natural text reply. Core owns all of it; a runtime only supplies the site.
- When. Core
isStopGuardOffered: stop reasonend_turn/stop_sequence, and not a forced call, a__finish__/finishWithstep, a truncated or error step, an aborted run or themaxStepscap step. InrunStepIteration's terminal branch, after the #181 abort check and before forced-completion entry. - One commit. Core
runStopGuardgives the decision, the guard's state writes (anImmerStateTrackerinarrayDeltaMode, applied to the step's custom state and streamed by the runtime as onestate_patch) and its rows (source: 'stop_guard', each stampedmetadata.helixStopGuard: { turnId, runId, step, verdict }, the feedback last behindSTOP_GUARD_FEEDBACK_LABEL). They join the replying step's own commit whatever it decides: acontinuecommits the step as an ordinary one and takes the next; astopfalls through to the terminal commit or the forced entry; afailcommits the step, then fails typed (framework_stop_guard_rejected). - Budget.
countStopGuardContinues: the distinct continue steps stamped with the currentturnId, a pure fold (no store field). An exhausted continue ends the run normally and drops its rows (keeping its writes). A guard that throws, times out or returns an invalid verdict commits nothing of itself, then stops the run (onError: 'stop') or fails it (framework_stop_guard_failed). - Persisted outcome. The run record's
completionReason:'stop_guard_exhausted'/'stop_guard_error'(every store; Postgres V32's CHECK), and itsstopGuardreport ({ continuesUsed, exhausted?, error? }: Postgres V33 JSONB, D1 35 / DO-SQLite 31 text, the Redis run hash, memory), written in the same terminalupdateRunStatus. A reconnect (corewithRunCompletionReason) and the DO's/statusread both back;finishCommittedRunrecovers them from a forced phase that carries the report. A completed session never carries anerrorDetail. - Rendering. A
stop_guardrow is shown to the model as a user turn (llm-vercel gives a system form only toapp/hookrows; core's lowering is always a user turn). Itstoolsrows may switch tools on (TOOL_ENABLING_SOURCES). - Hook firing order.
beforeLLMCall→ the model →afterLLMCall→ the guard → (continue) the step commit, then the next step's context pass; (stop) the terminal commit →onAgentComplete; (fail) the step commit →onAgentFail.onAgentCompletefires once, after the last decline. - Per runtime. runtime-js passes
StepIterationDeps.stopGuard(its turn id, the replying attempt's id, thestate_patchwrite, thecore.stop_guard.*log events) and writes the completion reason at the terminal run status; the CF DO inherits it. CF Workflows (MR 5), Temporal (MR 7) and DBOS (MR 8) refuse an agent tree that setsstopGuardat every root entry (theUNSUPPORTED_AGENT_FEATURESrow). A new runtime must callisStopGuardOffered/runStopGuardat its terminal-text site, commit the outcome with the replying step, write the completion reason and remove its phase from the table's row.
Completion strategies (cross-runtime)
An outputSchema agent's forced-completion call is enforced by one of two completion strategies, chosen per call from provider-neutral capabilities the LLM adapter reports (LLMAdapter.resolveCapabilities(config) → CapabilityResolution; spec 2026-10-04-completion-strategies-and-preserved-thinking-design.md):
forced_tool(forcedToolChoice): a namedtoolChoiceon the completion tool. On a model whose thinking is not history-bound it is byte for byte the legacy request (one tool, trimmed system prompt, reasoning disabled); on a history-bound one (reachable only through an override) it keeps the full tools and prompt.native_output(!forcedToolChoice && nativeStructuredOutput): the full tools and prompt,outputFormat(the completion tool's input schema) andminOutputTokens(16,000).planStepProcessingnormalises the JSON answer into a synthetic completion-tool call, so every downstream path (__finish__validation and synthetic result,finishWithphase 2, RM-29 pairing) is unchanged. A strict completion tool gets nooutputFormat(DI-80): the API refuses a strict tool and a format of the same schema together, so the strict tool is the constraint and the answer is a call to it, or the same JSON.strictstays on the tool, since thinking is bound to it.- Neither, or
unresolved→framework_completion_strategy_unavailable.
Core never reads a model id to decide: model knowledge lives in the adapter (@helix-agents/llm-vercel's table, with a user capabilities override). An adapter without resolveCapabilities is LEGACY_RESOLUTION, today's behaviour. What every runtime must wire (a new runtime adds these at its own sites; they are not inherited from core):
- Early refusal —
checkCompletionSupport(agent, adapter, options?)before the run-start commit of every entry, children included, so a refusal commits no run or message (runtime-js and the CF DO write nothing at all; CF Workflows and Temporal record a brand-new sessionfailedand leave an existing session untouched). CF Workflows writes only the session (user ruling R43: corefailSessionWithoutRun, no stream write); Temporal still fails the stream that entry made live (FU-TEMPORAL-NEVER-RUN-PRE-COMMIT-STREAM-WRITE). A runtime that runsllmConfigOverridebefore each LLM call (runtime-js, the CF DO) passes{ appliesLlmConfigOverride: true }: an agent with an override is then left to the per-call check, because the override may pick the forced call's model. - Per-call resolution (D7) —
resolveCompletionCapabilities(adapter, config)inside the unit that holds the adapter, with the config the call actually uses (afterllmConfigOverride), thenprepareForcedCompletionCall. The strategy is never persisted: retry, rewind and branch restore a pending phase and the next call resolves again. - Directive applied as built —
forcedDirectiveGenerateInput, trim the system prompt only whentrimSystemPrompt,assertForcedToolsUnchangedafter cache strategies, notext_deltaforwarding on anative_outputcall. - Response handling —
planStepProcessingwithforcedCompletion: { expectedToolName, strategy: 'native_output', callIdSalt }, the saltrunId:stepCount:attempt(replay-stable and unique), thenresolveForcedCallTransition. A provider error on a forced call fails the phase with the provider's detail; a runtime with a step retry rethrows a retryable one first. A native miss (no synthetic call: unparseable text,max_tokens) is planned withmetadata.hidden: true(R24), so it never reaches the UI but still replays to the model. - Prefix check —
computePromptFingerprint→checkPrefixStabilityat the LLM call site, andstampPromptFingerprinton the assistant message (below). - Adapter wrappers forward the capability methods (runtime-js's attempt-gate wrapper does), or the iterator reads
legacy.
| Runtime | Early refusal | Per-call strategy | Prefix check | Forced-call provider error |
|---|---|---|---|---|
| runtime-js | execute / resume / retry / handle.resume, child starts | step iterator, after llmConfigOverride | step iterator | fails the run (after adapter retries) |
| CF DO | /start, /resume, /retry, /submit-tool-result (typed 400, from materializeAgent); a wake that is refused ends the run typed (failCommittedRun) | step iterator (runs the JS executor) | step iterator | as runtime-js |
| CF Workflows | recorded step.do <agentType>-completion-support, before the run-start commit | inside the LLM step.do (steps.ts) | steps.ts | retryable → step.do retry (the provider's code is kept when the retries run out); else fails the run |
| Temporal | the initializeAgentState / startRunCommit activities, before the run-start commit (children at their own start) | inside the runLLMStep activity | runLLMStep, before step_start | retryable → activity retry, then the K5 commit fails the phase (failPendingForcedPhase); else fails the run |
| DBOS (MR 3) | none | LEGACY_RESOLUTION always | none | recoverable miss (R1 shim) |
On CF Workflows the decision is a recorded step result (the refusal is returned as data and the typed error rebuilt outside the step), so a replay of an in-flight instance reuses it: a deploy that changes how the agent's model resolves (a table change, a removed override) never refuses an instance admitted under the old resolution. An instance recorded before that step existed checks fresh on replay.
Test coverage. The native tests of runtime-js, the CF DO, CF Workflows and Temporal run against the provider-faithful MockLLMAdapter, which answers a native_output response with JSON only when the request carries outputFormat and rejects a tool-result continuation whose in-progress assistant turn lost its thinking blocks, so those tests also prove that each runtime sends the response format and replays the turn's parts (spec §6 item 4, "As built").
A run that fails for any other reason while its forced phase is still pending (a skills or memory failure on a forced call, a throwing generateStep) leaves different phase state: CF Workflows and Temporal fail the phase (failed, with the run's detail and an exhausted diagnostic chunk) in their K5 commit, while runtime-js and the CF DO leave it pending; retry() is unaffected (it restores from the step commit) (FU-COMPLETION-FAILED-RUN-PHASE-STATE).
Preserved thinking
Anthropic's Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1 and Mythos 5.1 run preserved thinking (historyBoundThinking): a thinking block's signature binds the top-level system prompt, the tool set and every earlier message, and a request whose prefix differs is rejected with HTTP 400 (enforced for accounts created on or after 2026-08-31; any request can opt in). Appending to history, changing parameters outside system / tools / messages (tool_choice, the output format, max_tokens), reordering tools and moving cache_control markers are allowed.
The append-only contract. Helix's message log is append-only, and the request prefix of every call must equal the earlier calls' prefix plus what was appended since. Helix guarantees, on runtime-js, the CF DO, CF Workflows and Temporal:
- Faithful replay. The adapter records each response's ordered
parts(AssistantMessage.parts: every reasoning block with its own signature, empty and redacted blocks included, text, tool-call references), every store persists them throughmessageSchema()with no migration, and the adapter replays them verbatim. (Capture and replay are adapter-side, so they also happen on DBOS.) Every part type carries the provider's own metadata (providerOptions, opaque to core, R14): llm-vercel records every key and replays only the key the request's provider reads back (see the llm-vercel guide). - Forced calls do not change the prefix. A forced call on a history-bound model keeps the full system prompt and tool set (
native_output, or a history-boundforced_tool); only the output format and tool choice differ, which the API allows. - No silent prompt changes. A skills provider that throws fails the step with the retryable
framework_skills_unavailableinstead of running without the catalog (every runtime except DBOS, which resolves the catalog in its workflow body and, until MR 3, logs the failure and runs without it);memoryManager.buildToolsthrowing fails the step (retryableframework_internal_error) instead of dropping the memory tools (JS / CF DO). An unknownpreloadSkillsname is still skipped with a warning, which is deterministic. - The prefix check. Each call fingerprints the request as sent (after cache strategies and
beforeLLMCall): a hash of every system message in order (the adapter hoists them all into the top-level system the API binds) and of the name-sorted tools (name, description, input schema,strict, provider options minus cache-control keys), plus the model id. A history-bound model's assistant message records it asmetadata.helixPrefix. Before each call,checkPrefixStabilityfinds the latest assistant message with signed reasoning and a fingerprint for the same model (another model's thinking is dropped by the API, so a model switch is not a conflict) and refuses a changed system prompt or tool set withframework_prompt_prefix_changed(non-retryable) before the API sees it, naming which changed. A refused forced call fails its forced phase. It also catches deploys that change prompt text, tool descriptions or schemas, skills or workspace config for live sessions. - The fingerprint stays server-side.
@helix-agents/ai-sdkstripshelixPrefixfrom UI message metadata.
What users must not do on these models:
- Change the rendered system prompt (a
systemPromptfunction whose output varies), the tools or their descriptions / schemas, the skills or the workspace config within a live session. It is refused; put changing context in an appended message, or start a new session. - Edit earlier messages in a
beforeLLMCallhook. The fingerprint does not cover messages, so this is not detected and the provider rejects the call. - Rewrite stored assistant messages or drop their
parts(a custom store transform, re-saving messages read withincludeThinking: false, which strips reasoning parts). A mid-history thinking-block change is a 400.
Rolling deploys. A reader from before parts existed strips the field and replays the merged legacy form, which an enforced account rejects until the deploy completes. DBOS does not run the prefix check yet, and its forced calls still use the legacy narrowed request, which changes the prefix (MR 3).
| Runtime | History for the model | Run-level C9 (state_history_incomplete → errorDetail) | from_checkpoint | Branch checkpointId / messageIndex | Foreign checkpoint rejected | Truthful terminal checkpoint |
|---|---|---|---|---|---|---|
JS (runtime-js) | loadLLMHistory at every entry point that seeds state.messages (execute on an existing session, resume, retry, child reloads, companion continuation from interrupted) | onFailed in the loop; failRetryRun after retry()'s own run-start (core failCommittedRun, fenced on the run); before a run-start commit persistPreCommitFailure (M1: a session that already has a run is untouched, the caller still gets the typed error) | resolve → planRewindToCheckpoint → ONE run-start commit with truncateTo (rewind + new run, atomic) to the target (no latest-checkpoint re-truncate, RM-25); the owner guard refuses a live-owned run | cloneSession after resolveCheckpoint on the source | retry, from_checkpoint, branch | ✅ (commitStep) |
| CF Durable Object | Via runtime-js | Via runtime-js | Via runtime-js; the DO host /resume drops checkpointId (CP-34) | Not supported (a DO reads only its own SQLite) | retry via runtime-js; from_checkpoint not reachable over DO /resume (CP-34); no branch | ✅ (saveState checkpoint now records the live message_count, RM-36) |
| CF Workflows | loadLLMHistory in executeAgentStep / loadAgentState; bookkeeping seams (failAgentStream, endAgentStream, handleInterrupt, forced-completion recorders) read leniently | ErrorDetail rides across step.do in the error message behind a marker, decoded in the outer catch | resolve → assertRewindable when the owner instance is not closed → the instance's run-start commit with truncateTo (any non-live status, completed / failed included); run_resumed.fromCheckpointId reports the target | execute() resolves synchronously, the workflow uses loadBranchSource (D1) | retry, from_checkpoint, branch | ✅ (text-only, structured-output, forced-completion and classified-failure tails, RM-33) |
| Temporal | Loaded inside runLLMStep (never in workflow code or activity payloads); the activity returns only the step's delta (newMessages, baseMessageCount) | Non-retryable ApplicationFailure with the ErrorDetail in details, extracted by the workflow catch | resume() dispatches on mode (CP-31): resolve and assertRewindable in the executor, then the startRunCommit activity's run-start commit with truncateTo; with_message is appended before the run starts; from_checkpoint rewinds a completed / failed session too (C7); continue / with_message on one return its old result | Executor resolves synchronously, activity uses loadBranchSource | retry, from_checkpoint, branch | ✅ |
| DBOS | getMessagesStep → loadLLMHistory (no 10k cap, RM-11); checkpoint messageCount from getMessageCount (RM-12) | finalizeFailure with the classified detail (also after a DBOS step-error revival) | resolve first; a live run is refused; a suspended session's PENDING workflow is cancelled (its owner closes) and a fresh workflow's run-start commit with truncateTo supersedes the suspended run; a completed / failed session is rewound the same way | cloneSession after resolveCheckpoint on the source | from_checkpoint, branch, retry (checkpointId / message, CP-30) | ✅ (text-only, structured-output, forced-completion and classified-failure tails, RM-33) |
Terminal checkpoint rule (user ruling D1, all runtimes): every terminal exit (completion, a framework-forced terminal tail, failure, history-read failure and stop) writes the K5 state commit before the terminal run status and the stream end, so the latest checkpoint of a failed session is its K5. retry() never targets it. Core planRetry (the failed run's checkpoints, stateless: listRunCheckpointMetas) restores the failed run's latest step commit (or a later append commit of that run with no assistant row); when the run failed on its first step it re-runs the run's entry from its run-start commit (the rows rewind to just before the run's triggering user message, which the retry re-appends, plus any results a resumed run drained before its first assistant row). An explicit checkpointId is restored as given. The same rule holds on every runtime: steps are atomic (an interrupted or failed step persists no rows), so what a retry keeps no longer depends on the runtime or store. Retry outcomes match the old "a failure writes no checkpoint" model (R28 / R40). Temporal and DBOS still treat a content_filter / refusal / error / unknown finish of an agent without outputSchema as completed, not failed (DI-26). A reply cut at the output token limit follows the truncation contract on every runtime (core truncationOutcome; Error handling).
Also part of this contract:
- Stores keep counts truthful after a truncate (RM-16 Postgres, RM-22 Redis, RM-30 D1):
truncateMessagesupdates the denormalized message count in the same transaction / Lua script / batch, so later checkpoints andlistSessions().messageCountmatch the live row count. The cross-store contract testcore/src/testing/message-count-operations.tsenforces it. - Retry and resume never write a content-less row (RM-28): DBOS starts retry/resume workflows with no input, and the body treats absent input as "append nothing".
- Hook
messageIndexis the absolute position in the persisted log on every runtime (RM-10 on Temporal, RM-11 on DBOS above 10k).
See ./execution-flow.md §LLM history loading and §Checkpoint rewind and branch, and the design spec ../superpowers/specs/2026-09-25-llm-history-completeness-design.md.
State Store — Persists session state (messages, custom state, checkpoints). Uses SessionStateStore interface with sessionId as the primary key. Implementations: InMemoryStateStore for dev, RedisStateStore for prod, PostgresStateStore for prod (works across all runtimes including Cloudflare Workers via Neon/Hyperdrive). All implementations guarantee atomic createSession() — concurrent calls for the same sessionId result in exactly one winner (others throw), and a concurrent reader sees either no session or the complete initial session, never a half-written one (on Redis the claim, hash, initial custom state and indexes are one Lua script — RM-50). This is the foundation for preventing duplicate execution across all runtimes.
Stream Manager — Handles real-time streaming of agent events. Implementations: InMemoryStreamManager (store-memory), RedisStreamManager (store-redis), DOStreamManager (runtime-cloudflare; binding-side manager that talks to the streaming Durable Object), and DurableObjectStreamManager + the StreamServer Durable Object (store-cloudflare; the DO-resident SQLite-backed implementation). The Cloudflare DO managers are the push transports that broadcast the truncated wire event for G4 (truncation surfacing).
LLM Adapter — Abstracts the LLM provider. VercelAIAdapter wraps the Vercel AI SDK. MockLLMAdapter for testing (with capabilities, a provider-faithful simulation). An adapter may also report model capabilities (resolveCapabilities, checkOutputFormat), which pick the completion strategy.
Checkpoint — Complete state snapshot saved after each step, and at the end of a turn that completes (text-only, structured-output and finishWith included) or ends in a framework-forced terminal tail (maxSteps, forced completion), so getLatestCheckpoint reflects the turn that just ended. Every other terminal exit (failure, stop) writes its K5 terminal commit too (user ruling D1); retry() never targets it and restores the failed run's last step commit or its run-start commit instead (see the terminal checkpoint rule below the parity table). Enables crash recovery, time-travel, and branching. Checkpoints are scoped to a session: retry / from_checkpoint / branch reject another session's checkpoint id with state_checkpoint_foreign.
Lock Manager — Distributed coordination interface. Prevents concurrent execution of the same agent across processes. Implementations: NoOpLockManager, InMemoryLockManager, RedisLockManager, PostgresLockManager, DurableObjectLockManager.
Logger — All SDK components accept an optional Logger interface (info, warn, error, debug? methods) defined in core/src/types/logger.ts. Defaults to noopLogger (silent). Use consoleLogger for development. Compatible with pino, winston, and other structured logging libraries. Configured via constructor options on executors, state stores, adapters, and tracing hooks. Zero bare console.* calls exist in production source files — all logging goes through Logger.
Error text — Error free text (an error's message, ErrorDetail.message / cause, SessionState.error, a stream's failure text) can carry credentials, so it never leaves the process raw. It stays raw in the session row and the server log. Everything sent to an HTTP / SSE / WebSocket client or a remote-agent parent goes through core sanitizeErrorText (presets sanitizeErrorMessage for run-level text, 256 characters; sanitizeToolErrorText for a tool's error, line breaks kept). Every stream manager sanitizes on write (sanitizeStreamChunk, and failStream's text), so no stream holds raw error text. Nothing is HTML-escaped. Machine fields are never changed. The model's own view of a failed tool, a failed plain sub-agent or a failed remote child is the one surface still raw (FU-RAW-ERROR-TEXT-MODEL-SINK). A persistent companion's failure is not: its answers (a park's failed errorDetail, every priorResult.errorDetail) and its hidden failure notice are built sanitized by core's builders on every runtime (DI-70, the client preset, paths redacted too), except Temporal's own blocking companion rows and DBOS's own spawn path (FU-TEMPORAL-DBOS-COMPANION-FAILURE-ROWS-SANITIZE). The child session's own error stays raw. See core reference: Error text on outward surfaces.
Tracing — @helix-agents/tracing-langfuse is the supported tracing adapter. As of v7, it seeds the Langfuse trace ID from sessionId (not runId) so that a single conversational session — which spans many runs once HITL boundaries are involved — appears as a single trace in the Langfuse UI. New onAgentResumed and onAgentSuspended hook handlers emit matching event spans inside the session-scoped trace, so you can visually see where the run paused and where it resumed. The legacy core/tracing/tracing-hooks.ts adapter is HITL-incompatible: it relies on an in-memory tracingStateMap that the stateless-suspension model cannot populate across process restarts, and v7 fail-fasts when requireApproval or client-executed tools are run with the legacy adapter. Use @helix-agents/tracing-langfuse (or implement the v7 hook interface in your own adapter) before upgrading.
Embedding Executor — Controls how vector embeddings are computed after a memory is saved. InlineEmbeddingExecutor (default) computes synchronously. BackgroundEmbeddingExecutor fires-and-forgets with maxConcurrency limit and shutdown() drain. Memories are saved with embeddingStatus: 'pending' and immediately FTS-searchable; the executor updates them to 'complete' once the embedding is computed. MemoryManager.processUnembeddedMemories() recovers orphaned pending memories (e.g., after embedding service failures). Configured via embeddingExecutor on MemoryConfig.
Skills (Progressive Disclosure)
Skills give an agent a library of specialized capabilities (workflows, runbooks, reference protocols) without paying for all of them on every request. They implement Anthropic's Agent Skills pattern — 3-level progressive disclosure — on Helix's existing append-only / cacheable substrate. Configured via AgentConfig.skills (see the design spec).
The 3-level model
- Level 1 — catalog (always resident). Every skill's
name+description(~tens of tokens each) is rendered into a## Skillssection appended to the system prompt. The catalog is deterministically sorted by name so it is byte-stable within a session — it lives in the cached prefix and never invalidates it. The fragment is produced bygenerateSkillsSystemPromptFragment(metadata[]); an empty skill set produces''(total no-op). The catalog carries a load-bearing guardrail ("A Skill is NOT a tool — callload_skill"), because models otherwise try to invoke skill names as if they were tools. - Level 2 — body (loaded on demand). The full skill body is loaded by the auto-injected
load_skilltool, which returns the body as the tool result (immediate — the model acts on it in the same continuation, no wasted round-trip). Because tool results only ever append to history, this is append-only and cache-safe.load_skillalso emits an informationalskill_loadedcustom stream event ({ name }, surfaced as adata-skill_loadedAI-SDK event) that consumers MAY render but are not required to handle. - Level 3 — resource files (read on demand). Bundled reference/script/asset files are read by the auto-injected
read_skill_filetool, which supports optionalstartLine/endLineranges and a path-traversal guard. The skills feature discloses content; it does not execute code — Level-3 "scripts" are readable, but running them is delegated to the agent's own shell/workspace tools. This keeps the feature decoupled from any execution environment.
SkillProvider and the two providers
Skills resolve to plain data behind a small async interface:
interface SkillProvider {
listSkills(): Promise<SkillMetadata[]>; // Level 1
getSkill(name: string): Promise<Skill | null>; // Level 2
readResource(name: string, path: string, range?: ReadResourceRange): Promise<string | null>; // Level 3
}Two providers ship in v1:
inCodeSkillProvider(in@helix-agents/core) — the "plain data" mode: skills are TypeScript data bundled with the agent. Dependency-free and Workers-safe (nonode:fs). Use this on Cloudflare Workers.fileSystemSkillProvider(in@helix-agents/skill-fs) — reads Anthropic-formatSKILL.mddirectories (<root>/<skill-name>/SKILL.mdwith YAML frontmatter + markdown body, optionalreferences//scripts//assets/). Node only (node:fs/promises+yaml); not usable on Cloudflare Workers.
A third delivery path, the build-time bake (@helix-agents/skill-cli), is not a runtime provider: it resolves remote skill packages at BUILD time and feeds the result back through inCodeSkillProvider (see below).
Remote skill packages (build-time bake)
@helix-agents/skill-cli lets you author skills as remote packages — a git repo, or a Claude plugin marketplace — yet still ship them through the Workers-safe in-code provider. Its helix-skills sync command resolves each manifest source, pins it at a version (tag/ref/sha), and bakes the selected skills into a generated SkillDefinition[] TypeScript module plus a committed lockfile (one sha256 integrity hash per manifest entry; --check fails CI on drift). You import { skills } from the generated module and pass it to defineAgent({ skills }), which routes through inCodeSkillProvider.
Because resolution happens entirely at BUILD time and the output is plain in-code data, the baked skills inherit the in-code provider's full cross-runtime support — including Cloudflare Workers — with zero runtime fetch and no node:fs. The CLI itself is Node-only and is never imported by the agent at runtime; only its generated output is. The SKILL.md parsing it shares with the filesystem provider lives in core as parseSkillFile (pure, Workers-safe), at its own entry @helix-agents/core/skill-file so that core's main entry does not load yaml. See the Skills guide → Loading remote skill packages and the @helix-agents/skill-cli reference.
Configuration
Set AgentConfig.skills to either a SkillProvider or an array of in-code SkillDefinitions (sugar for inCodeSkillProvider). When present, the framework appends the catalog to the system prompt and auto-injects load_skill + read_skill_file into the tool list (via shared buildEffectiveTools). The two tool names are reserved (RESERVED_TOOL_NAMES) — user tools cannot shadow them, and skill names are [a-z0-9-] (no underscores) so they can never collide with the tool names either. Sub-agents do NOT inherit a parent's skills — a sub-agent uses skills only if its own config declares them (matching how sub-agents already scope their own tools/state).
Minimal in-code example:
const agent = defineAgent({
name: 'assistant',
systemPrompt: 'You are a helpful assistant.',
llmConfig: { model },
skills: [
{
name: 'pdf-processing',
description:
'Extract text and tables from PDFs, fill forms, merge documents. Use when working with PDF files.',
body: '# PDF processing\n…full instructions…',
},
],
});Filesystem example (one line):
import { fileSystemSkillProvider } from '@helix-agents/skill-fs';
// …
skills: fileSystemSkillProvider({ roots: ['./skills'] }),Preloaded skills
AgentConfig.preloadSkills?: string[] injects the named skills' full bodies into the system prompt on every step — they are always in context, with no load_skill call needed. The bodies render as an ### Active Skills block inside the same deterministically-sorted, cache-stable fragment as the catalog (so the preloaded set is fixed per session and the prefix stays byte-stable). Preloaded skills also appear in the loadable <available_skills> catalog marked loaded="true" — a static marker (cache-safe, decided at config time) that tells the model not to reload them, while load_skill remains a recovery path (e.g. if history compaction later drops the system-prompt-injected body, load_skill can re-fetch it).
Each name must resolve in the agent's provider; unknown names warn-and-skip at resolution time (one bad name never crashes the agent or breaks the rest). Sub-agents do NOT inherit a parent's preloadSkills — same scoping as skills and tools. There is no token budget on preloaded bodies (future work, same as the catalog).
defineAgent({
/* … */ skills: [
/* … */
],
preloadSkills: ['deploy-runbook'],
});Cache-safety contract
Skill loading is purely additive by construction:
- The Level-1 catalog is a stable, sorted system-prompt fragment that is NEVER annotated with per-skill loaded-state (no "✓ loaded" marks) — annotating it would make the cached prefix volatile and bust the cache on every load. "Already loaded" handling lives entirely in the
load_skillresult, never in the catalog. - Level-2 bodies and Level-3 resources arrive append-only as tool results, covered by
anthropicCache's breakpoints (the system anchor + the latest turn's tool-result batch). The skills feature required no changes toanthropicCache/applyCacheStrategies/ the Vercel adapter.
The only inherited caveat (identical to memory injection): on the turn a body lands, the latest-turn breakpoint sits on/after it, so that one breakpoint won't cache-hit across that turn boundary; the system anchor still does.
Cross-runtime support matrix
All logic lives in shared core (buildEffectiveTools, buildMessagesForLLM, the skill-injection helpers), plus a one-line per-run catalog-resolution hook at each runtime's buildMessagesForLLM call site (the same place memory retrieval is resolved — where IO / non-determinism is allowed). The tools therefore work on all runtimes; the catalog string is threaded on JS / Temporal / Cloudflare / DBOS.
| Runtime | In-code provider | Filesystem provider |
|---|---|---|
| JS | ✅ | ✅ |
| Temporal | ✅ | ✅ (resolve catalog + tool reads in activities) |
| DBOS | ✅ | ⚠️ (catalog resolved in the workflow body — see note) |
| Cloudflare DO / Workflows | ✅ | ❌ (no node:fs; use in-code or a future workspace/D1 provider) |
fileSystemSkillProvider works wherever node:fs exists — i.e. NOT Cloudflare Workers; use the in-code provider there. On Temporal the provider's filesystem IO runs inside the per-step activity (where IO is allowed), never in workflow code. On DBOS, however, the catalog is resolved in @DBOS.workflow-body code (not yet wrapped in a @DBOS.step), so a fileSystemSkillProvider would run replay-sensitive node:fs IO in workflow code — prefer the in-code provider or build-time-baked skills on DBOS (the @helix-agents/skill-cli bake path makes baking ergonomic). In-code catalogs are deterministic data and need no special handling. (Tracked as FU-SKILL-DBOS-CATALOG-IN-WORKFLOW-BODY — wrap the DBOS resolution in a checkpointed step to mirror Temporal.)
Known limitations (v1)
- Re-loading a skill returns the body again.
ToolContextexposes no transcript access, so theload_skilltool cannot dedup on its own — v1 returns the body on every call (correct + cache-safe; rarely triggered because the model sees its own priorload_skillresults).collectLoadedSkillNames(messages)ships as the building block for programmatic dedup, but the dispatch-layer short-circuit is deferred. fileSystemSkillProvidertraversal guard is lexical. It rejects resolved paths that escape the skill dir but does NOT follow symlinks — safe for operator-provisioned skill directories.- fs staleness re-scan detects root-entry add/remove, not in-place edits. Editing an existing skill's files in place won't be picked up until restart or a touch of the root.
- DBOS resolves an async provider in the workflow body. Fine for the in-code provider (deterministic data); a provider that fetches at RUNTIME would move catalog resolution into a
@DBOS.step(). A runtime-fetching remote provider is still deferred (see the design spec future work). Note that build-time remote loading already shipped via@helix-agents/skill-cli, which sidesteps this entirely by baking remote packages down to in-code data (deterministic, no per-step IO).