Distributed Coordination

When running agents across multiple processes or servers, two callers can act on the same session at once. This guide covers how the framework decides which one runs, how it stops the other, and the tools you can use for your own coordination.

The Problem

In distributed deployments, multiple processes might try to:

• Resume the same agent simultaneously
• Write state at the same time
• Process the same step concurrently

Without coordination, this leads to:

• Split-brain - Two processes think they own the agent
• Lost updates - One process overwrites another's changes
• Corrupted state - Partial writes from concurrent modifications

How the framework coordinates runs

Every runtime coordinates through the state store. You do not need a lock manager to stop two executors from driving one session. Three mechanisms work together:

1. The run-start commit. Every entry (execute(), resume(), retry(), a sub-agent spawn) starts a new run with one atomic store write. It is the only gate: exactly one concurrent entry wins.
2. The superseded-run fence. Every write a run makes is checked against "this run is still the session's current live run". A run that lost its session writes nothing and stops as superseded.
3. Optimistic concurrency. State writes carry a version, and a stale version is refused with StaleStateError.

Breaking for custom stores

The run-start commit and the fence are enforced by the store. A custom SessionStateStore must implement them. See State Stores and the upgrade guide.

The run-start commit

A run starts with appendMessages(sessionId, rows, { streamSequence, runId, startRun }). The rows are the run's input (the new user message; it may be empty, for example on a resume). Inside one transaction, on the store's own clock, the store applies these rules in order:

1. Idempotency. A replay of the same runId with the same startDigest (a digest of the rows and startRun) writes nothing and succeeds. The same runId with a different digest throws RunStartRejectedError('idempotency_mismatch'). A runId that its caller gave up on (see Run-start fail-safes) throws RunStartRejectedError('entry_abandoned').
2. Compare-and-set. startRun.expectCurrentRunId must name the session's current run (null for none). On a mismatch, the commit throws RunStartConflictError('expected_mismatch') when the current run is live, or RunStartRejectedError('expected_mismatch_idle') when it is not. The run's status must also be the one the entry saw (startRun.expectCurrentRunStatus); otherwise the run it planned from has moved on, and the commit throws RunStartRejectedError('expected_run_status_changed').
3. Owner guard. A live current run can be superseded only by the same owner (ownerToken), or when it is ownerless (see below). Otherwise the commit throws RunStartConflictError('run_active').
4. Session gate. The session's status, read inside the same transaction, must be one of startRun.entryStatuses (the status the entry planned from). Otherwise the commit throws RunStartRejectedError('entry_status_changed').
5. Supersede. Every other live run of the session becomes superseded.
6. Optional rewind. With startRun.truncateTo, the store deletes the rows from that count on and writes a truncate checkpoint attributed to the new run. resume({ mode: 'from_checkpoint' }) and retry() rewind this way, so the rewind and the new run land together or not at all.
7. Create. The run record is created (status: 'running'), becomes the session's current run, and the input rows are appended with an input checkpoint.

A rejected commit writes nothing. A loser writes nothing anywhere: no input row, no resume count, no stream status change and no stream chunk. Only the winner claims the session's stream, makes it live and writes run_started.

Owner tokens and the ownerless predicate

An owner token identifies one execution. The owner guard lets a run continue past its own entries (for example a durable workflow that resumes after a suspension), and lets a new entry take over only when the old owner is gone.

Runtime	ownerToken	The current run is ownerless when
JS	random, one per entry	its run lease has lapsed, evaluated by the store on its own clock (see The JS run lease)
Cloudflare Durable Object	none (null)	the Durable Object is not executing that run. The check and the commit run in one synchronous transactionSync block (DOStateStore.runStartCommitSync)
Temporal	${workflowId}:${workflowRunId}	the owner workflow execution is closed. A suspended_* run whose workflow is still RUNNING is polled until it closes (bounded). A running workflow whose workers are all down blocks the entry
DBOS	the DBOS workflowID	the owner workflow's status is terminal
Cloudflare Workflows	the instance id	the instance status is complete, errored or terminated. An instance that reports unknown for longer than cfwUnknownTerminateAfterMs (default 10 minutes) is terminated by the next entry, which writes a run_terminated chunk for it. An operator-paused instance needs executor.terminateRun(sessionId)

On every runtime except JS the runtime proves the predicate and the store trusts the assertion (ownerless: { kind: 'asserted' }). A live run with no owner token (a row written before this release) is ownerless.

One run skips the owner guard: a current run whose K5 has landed. An entry that finds the session terminal with the current run's own state commit as its latest checkpoint passes startRun.finishCurrent, and when the store's own reads verify that K5 it ends the old run in the same run-start commit, even while its owner is alive (an unexpired JS lease, CP-84). The old owner has nothing left to write but its tail: its later write of the same terminal status is a no-op (core decideRunStatusTransition answers 'noop'), and its stream end is fenced out once the new run claims the stream (CP-85). An unverified finishCurrent still meets the owner guard.

Run ids

Durable runtimes mint the run id outside any retried body, so a retried entry replays the same runId and hits the idempotency rule instead of starting a second run. Temporal and DBOS mint it in the executor and pass it in the workflow input. Cloudflare Workflows uses ${instanceId}:${agentType}-run-start:0. JS and the Durable Object mint one per entry call.

What a losing caller sees

Error	name	Meaning	What to do
RunStartConflictError (run_active, expected_mismatch)	AgentAlreadyRunningError	another run is live and driving the session	treat it as "already running": attach to the stream, or wait
RunStartConflictError (expected_mismatch, from expected_run_status_changed)	AgentAlreadyRunningError	the run the entry saw has moved on since it planned	treat it as "already running"; the stale entry wrote nothing
RunStartRejectedError (expected_mismatch_idle)	RunStartRejectedError	the run you expected is no longer current, and no run is live	re-read the session; runtimes retry this once when their own read was stale
RunStartRejectedError (idempotency_mismatch, entry_abandoned, entry_status_changed)	RunStartRejectedError	the entry cannot commit	surface it; the chat handler answers 409 state_run_start_conflict

RunStartConflictError extends AgentAlreadyRunningError and keeps its name, so existing instanceof AgentAlreadyRunningError checks still match it. RunStartRejectedError deliberately does not: it is not "someone else is driving". Both carry code: 'state_run_start_conflict'. entry_status_changed is raised by every store when the session left the statuses the entry planned from (for example an /abort landed in between). expected_run_status_changed never reaches a caller as itself: asRunStartLoserError types it as the expected_mismatch conflict above, so "a newer run started" and "the run I saw finished or moved on" look the same. The full rule order and the per-runtime sources of the two fields are in Execution Flow §Entry gates.

Per runtime, the stale-entry race surfaces differently. On runtime-js and the CF DO a stale entry always gets the expected_mismatch conflict above. Temporal, DBOS and CF Workflows re-plan from a fresh read inside their entry, so a stale entry there gets one of three: the same expected_mismatch conflict when its commit lands after the winner started; RunStartRejectedError('entry_status_changed') from the plan-time gate when it plans after the winner finished; or, while the winner is still live, an owner-check refusal before its commit (Temporal: a plain AgentAlreadyRunningError with no conflictCause; DBOS and CF Workflows: RunStartConflictError('run_active')). Treat each as "another entry won" and reload (FU-CP96-DURABLE-LOSER-CLASS; details in the section linked above).

import { AgentAlreadyRunningError } from '@helix-agents/core';

try {
  const handle = await executor.resume(agent, sessionId, { mode: 'continue' });
  await handle.result();
} catch (error) {
  if (error instanceof AgentAlreadyRunningError) {
    // Another entry won: stream from the existing run instead.
    return;
  }
  if (error instanceof Error && error.name === 'RunStartRejectedError') {
    // The session changed under this entry: reload it before deciding again.
  }
  throw error;
}

The superseded-run fence

Once a newer run has started, every write of the old run is refused:

• Commits. Every commit that carries a runId (an append, a turn's start, a step, the terminal state commit, a truncate) checks that runId is the session's current live run.
• saveState. Runtimes pass { fenceRunId } (or { fenceIdle } for a write made when no run owns the session).
• Run status. updateRunStatus is sticky for terminal statuses: a completed, failed, interrupted or superseded run never changes status again.
• Stream. Once a newer run has claimed the session's stream, stream managers drop a chunk of that session stamped with another runId, and endStream / failStream / pauseStream take an owner fence. A sub-agent's chunks in its parent's stream are not fenced this way.
• Tools. Before every tool execution, sub-agent spawn, and client-tool or approval registration, each runtime checks that the run still owns the session (assertRunOwnsSession), so a superseded run never runs a tool.

A refused write throws RunSupersededError (code: 'state_run_superseded'), which is non-retryable on every runtime (Temporal ApplicationFailure.nonRetryable, excluded from DBOS step retries, a CF Workflows NonRetryableError). The run then ends with AgentResult.status === 'superseded', never failed. Neither onAgentComplete nor onAgentFail fires (one exception: a supersede that lands after onAgentComplete fired and before the run's terminal commit leaves onAgentComplete then onAgentSuperseded; the last terminal hook is authoritative); the optional onAgentSuperseded hook does. A late client-tool submit to a superseded run answers 409 { code: 'state_run_superseded' }.

const result = await handle.result();
if (result.status === 'superseded') {
  // Not a failure: a newer run owns the session. Follow that run instead.
}

The fence bounds, but does not remove, a superseded executor's last writes to the live stream before it stops: on JS, until its next lease renewal or fenced write; on the other runtimes, until its next fenced write, which can be a whole LLM step of streamed text.

The JS run lease

The JS runtime has no durable owner to probe, so a JS run holds a lease on the store:

• The run-start commit sets leaseUntil = storeNow + ttlMs, on the store's own clock (Postgres now(), Redis TIME, the process clock for memory, D1 and the Durable Object store).
• The executor renews it every ttl / 3 with renewRunLease(sessionId, runId, ttlMs), a compare-and-set that fails with RunSupersededError once the run is no longer the current running run.
• A rejected renewal stops the run at once as superseded. A transient store error is retried inside the remaining lease; if the lease would lapse first, the run stops the same way.
• Every chunk the run writes is gated on the lease locally, without I/O, so a run whose lease lapsed writes nothing more.
• The suspension or terminal status write releases the lease.
• In-process sub-agent and companion runs hold their own lease from their own run-start commit.

The TTL defaults to JS_RUN_LEASE_MS (30 000 ms, exported by @helix-agents/runtime-js). Configure it on the executor:

import { JSAgentExecutor, JS_RUN_LEASE_MS } from '@helix-agents/runtime-js';

const executor = new JSAgentExecutor(stateStore, streamManager, llmAdapter, {
  runLease: { ttlMs: 2 * JS_RUN_LEASE_MS },
});

A blocked event loop can lose the lease

A JS event loop blocked for longer than about ttl - ttl/3 (20 s by default), for example by a synchronous CPU-bound tool, can miss its renewals. A concurrent entry may then supersede the run. The old run stops as superseded (never failed) and its writes are fenced. Move CPU-heavy work off the event loop, or raise ttlMs.

The Cloudflare Durable Object runtime runs the JS executor without a lease: its run-start claim is decided inside the Durable Object.

Run-start fail-safes

On Temporal, DBOS and Cloudflare Workflows, execute() / resume() / retry() start a workflow and wait for its run-start outcome. If no worker takes the entry, or its start keeps failing, the call gives up after a bound:

Runtime	Option	Default
Temporal	entryOutcomeTimeoutMs	120 000 ms
DBOS	runStartFailSafeMs	120 000 ms
Cloudflare Workflows	entryOutcomeFailSafeMs (an explicit entryStartTimeoutMs overrides it)	120 000 ms

Past the bound, the call runs the store's abandonRunStart(sessionId, runId), which is atomic with the session's run-start commits:

• If the run already exists, nothing is written and the call returns its handle: the input was delivered. On Cloudflare Workflows the run must also have STARTED (its run_started in its own stream and its entry state written). A committed run gets the same bound again until its instance closes or is paused, measured from when the call first saw the commit (a transient owner read or abandon failure does not end it; so a real stall can take up to about twice the bound); a run that still has not started is a stall (below).
• Otherwise the store records a tombstone for runId. A run-start commit that arrives later with that runId (an activity or step still in flight) throws RunStartRejectedError('entry_abandoned') and writes nothing, so a retried call never delivers its input twice. The owner workflow or instance is then stopped best-effort.

The call then throws one of these core errors. None is persisted on the session by itself.

Error	code	Retryable
RunStartAbandonedError	transport_error	once the owner was stopped (ownerStopped), or always where a retry starts a fresh owner
RunStartOutcomeUnknownError	transport_error	no: the abandon itself failed, so the entry may still commit. Read the session (its current run, getStatus) before retrying
RunStartStalledError	transport_error	never: CF Workflows only, the run committed (its input is already in the log) but never started, so repeating the call would append the input twice. Recover with executor.retry(agent, sessionId). settle says whether the run was recorded failed with this error's detail ('failed') or left for a reader to settle before the retry() ('not_committed'). When the instance could not be stopped (the run may still go on), its start could not be read (startUnknown: nothing was stopped), or a newer run had taken the session over (settle: 'superseded'), read the session first

Optimistic Concurrency Control

State writes are also version-checked. This protects writes that are not run-scoped (admin tools, your own code) and catches concurrent writers inside one run.

How It Works

1. State includes a version field
2. On save, the store checks if version matches
3. If mismatched, save fails with StaleStateError and writes nothing

import { StaleStateError } from '@helix-agents/core';

// Process A loads state at version 5
const state = await stateStore.loadState(sessionId);
console.log(state.version); // 5

// Process B loads same state, modifies, saves -> version 6

// Process A tries to save its changes
try {
  await stateStore.saveState(sessionId, state); // Expected version 5, but store has 6
} catch (error) {
  if (error instanceof StaleStateError) {
    // Our changes are stale - another process modified state
    console.log('State was modified by another process');
  }
}

The D1 and Redis stores retry their guarded commits internally. When concurrent writers keep winning, they throw D1CommitContentionError (exported by @helix-agents/store-cloudflare) / RedisCommitContentionError (exported by @helix-agents/store-redis). Both are subclasses of StaleStateError, so catch them as StaleStateError.

StaleStateError inside a run

On runtime-js and the Cloudflare Durable Object runtime, a StaleStateError from a step commit fails the run with StepCommitConflictError (state_concurrency_conflict, non-retryable): the session, the result, the failed stream and onAgentFail carry the same detail, and a stock useChat client sees an ordinary error. This is different from being superseded by a newer run, which is RunSupersededError and ends the run superseded.

Error Types

Error	code	Thrown when
RunStartConflictError	state_run_start_conflict	a live run won the run-start commit (name is AgentAlreadyRunningError)
RunStartRejectedError	state_run_start_conflict	a run-start commit was rejected without a live winner
RunSupersededError	state_run_superseded	a write of a run that is no longer the session's current live run (non-retryable)
RunStartAbandonedError	transport_error	a run-start fail-safe abandoned the entry
RunStartOutcomeUnknownError	transport_error	a run-start fail-safe could not abandon the entry (non-retryable)
RunStartStalledError	transport_error	CF Workflows: a run-start fail-safe found the run committed but never started
StreamClaimStaleReadError	state_concurrency_conflict	a run that won its commit could not yet read itself as current while claiming its stream (retryable)
StaleStateError	none; classifyError gives state_concurrency_conflict	a state write's version did not match
StepCommitConflictError	state_concurrency_conflict	runtime-js / DO: a step commit hit StaleStateError; the run fails (non-retryable)
FencingTokenMismatchError	none; classifyError gives state_concurrency_conflict	a lock fencing token did not match (for your own lock-guarded code)

code is a property on the error itself. classifyError maps RunStartConflictError through its AgentAlreadyRunningError base to state_already_running. All of these are exported from @helix-agents/core. See Error Handling for the full code list.

Stream Events

run_started and run_terminated

The winner of a run-start commit writes run_started { runId } before any content of its run. On Cloudflare Workflows, an entry that terminated an unresponsive owner writes run_terminated { runId, reason } for the killed run first. Every chunk a run writes carries its runId.

Compare-And-Set Status

For atomic status transitions, use compareAndSetStatus. In v7 the return type is a discriminated union — on conflict the store returns the actual current status and version so callers can recover without an extra read:

// Only update status if currently 'interrupted'
const result = await stateStore.compareAndSetStatus(
  sessionId,
  ['interrupted'], // expected statuses (array)
  'active' // new status
);
//   result: { ok: true; newVersion: number }
//         | { ok: false; currentStatus: SessionStatus; currentVersion: number }

if (result.ok) {
  // Sync local state version to match store
  state.version = result.newVersion;
} else {
  // Status was changed by another process — no extra round-trip needed
  console.log(
    `Status transition failed: store has '${result.currentStatus}' @ v${result.currentVersion}`
  );
}

A session status CAS does not start a run. Deciding which execution runs is the run-start commit's job.

Version Increment on CAS

When compareAndSetStatus() succeeds, it atomically increments the state version along with the status change. This is critical for closing a subtle race condition in optimistic concurrency control.

The Race Condition Problem

Without version increment on CAS, a race condition exists:

// Timeline without version increment (BUG):
// 1. Process A loads state: version=5, status='interrupted'
// 2. Process B loads state: version=5, status='interrupted'
// 3. Process B: CAS succeeds, status='active' (version stays 5!)
// 4. Process A: CAS fails (status mismatch - correct)
// 5. Process B crashes without saving
// 6. Process A retries, loads state: version=5, status='active'
// 7. Process A does CAS: 'active' -> 'active' (succeeds, version stays 5)
// 8. Process A saves with version=5 -> SUCCEEDS (but shouldn't!)

The problem: If CAS doesn't increment version, a process that loaded state before another's CAS can still save successfully because versions match.

The Solution

CAS atomically increments version on success:

// Timeline with version increment (CORRECT):
// 1. Process A loads state: version=5, status='interrupted'
// 2. Process B loads state: version=5, status='interrupted'
// 3. Process B: CAS succeeds, status='active', version=6
// 4. Process A: CAS fails (status mismatch - correct)
// 5. Later, if Process A somehow tries to save with version=5:
//    -> StaleStateError (store has version=6)

Example

// Initial state: version = 5, status = 'interrupted'

// Process A: CAS succeeds
const a = await stateStore.compareAndSetStatus(sessionId, ['interrupted'], 'active');
// a = { ok: true, newVersion: 6 }

// Process B: CAS fails (status is now 'active')
const b = await stateStore.compareAndSetStatus(sessionId, ['interrupted'], 'active');
// b = { ok: false, currentStatus: 'active', currentVersion: 6 }

// After successful CAS, sync local state version to the store-returned value:
if (a.ok) {
  state.status = 'active';
  state.version = a.newVersion; // Now 6, matches store
}

// Later saves will use the correct version
await stateStore.saveState(sessionId, state); // Works because version=6 matches store

Implementation Across Stores

All state stores implement atomic CAS with version increment:

Store	Implementation
InMemoryStateStore	Direct increment: state.version = (state.version ?? 0) + 1
RedisStateStore	Lua script with HINCRBY key 'version' 1 for atomicity
PostgresStateStore	SQL UPDATE ... SET version = version + 1
D1StateStore	SQL UPDATE ... SET version = version + 1
DOStateStore	SQL transaction with version increment

The Lua script in Redis ensures the check-and-increment is atomic, preventing any race window.

Lock Managers

The runtimes do not take a lock manager: run ownership is decided by the run-start commit. Lock managers are available for coordinating your own work across processes (a scheduled job, a migration, a resource your tools share). A lock is a lease: it ensures only one holder at a time while it is held and renewed.

Available Implementations

Lock Manager	Use Case	Backend
NoOpLockManager	Single process	None
InMemoryLockManager	Development	Memory
RedisLockManager	Production	Redis
PostgresLockManager	Production	PostgreSQL
DurableObjectLockManager	Cloudflare	Durable Objects

import { NoOpLockManager } from '@helix-agents/core';
import { InMemoryLockManager } from '@helix-agents/store-memory';
import { RedisLockManager } from '@helix-agents/store-redis';
import { PostgresLockManager } from '@helix-agents/store-postgres';
import { DurableObjectLockManager } from '@helix-agents/store-cloudflare';

new NoOpLockManager(); // all lock operations succeed immediately
new InMemoryLockManager();
new RedisLockManager({ redis, keyPrefix: 'myapp:locks:' });
new PostgresLockManager(adapter, { tablePrefix: '__agents_' });
new DurableObjectLockManager({ lockBinding: env.LOCKS });

Acquire and Release

const resource = `report:${reportId}`;

const result = await lockManager.acquire(resource, {
  ttlMs: 30000, // 30 second lease
});

if (result.acquired) {
  try {
    await buildReport();
  } finally {
    await lockManager.release(result.lock);
  }
} else {
  console.log(`Lock held by ${result.heldBy} until ${result.expiresAt}`);
}

withLock Helper

Automatic acquisition, heartbeat, and release:

await lockManager.withLock(
  `report:${reportId}`,
  { ttlMs: 30000, heartbeatMs: 10000 },
  async (lock) => {
    // Lock is automatically refreshed every 10 seconds
    await buildReport();
    // Lock is released when function completes
  }
);

Fencing Tokens

Each acquired lock carries a monotonically increasing fencingToken. Pass it to the resource you protect and have that resource refuse a token lower than the last one it saw, so a holder whose lease expired cannot write after a newer holder. The state stores do not check lock fencing tokens.

Lock TTLs

Too short: the lock expires during long operations. Too long: a long wait if the holder crashes. A rule of thumb is 2-3x the expected operation time, with a heartbeat at a third of the TTL.

Runtime-Specific Notes

• JS. A run holds a store lease (see The JS run lease). Several processes can share one store safely; the run-start commit picks one winner per entry.
• Cloudflare Durable Object. The Durable Object serializes requests for its session, and its run-start claim runs in one transactionSync block.
• Temporal. A turn runs in the session's workflow (<prefix>__<agent>__<sessionId>); resume() and retry() start their own workflow (…__resume-<runId>, …__retry__<runId>). The run-start commit runs in an activity and is idempotent by run id, so an activity retry never starts a second run.
• DBOS. Each run gets its own workflow (…__run__<runId>, …__resume__<runId> for resume() and retry(); persistent agents keep a session-scoped workflow). The run-start commit is a durable step, idempotent by run id.
• Cloudflare Workflows. Every execute() / resume() / retry() runs in its own write-once instance. The instance's run-start commit decides the winner. See Cloudflare Workflows.

Best Practices

1. Treat AgentAlreadyRunningError as "someone else is driving". Attach to the session's stream or wait; do not retry the entry in a loop.
2. Treat status: 'superseded' as a normal outcome. It is not a failure. Follow the newer run.
3. Reload before acting on RunStartRejectedError. The session changed under the entry (for entry_status_changed, plan again from a fresh read).
4. Read the session before retrying RunStartOutcomeUnknownError. A blind retry could deliver the input twice. Never repeat a call that threw RunStartStalledError (its input is committed): use executor.retry(agent, sessionId).
5. Keep the JS event loop responsive, or raise runLease.ttlMs, so a run keeps its lease.

Next Steps

• Interrupt and Resume - Coordination for pause/resume
• Checkpoints - State snapshots in distributed systems
• State Stores - The store contract behind these guarantees
• Redis Store - Redis-based state and locking