Cross-Service Remote Agents
The Remote Agents guide covers the protocol: a parent agent delegates to a child that runs somewhere else, over createRemoteSubAgentTool and a RemoteAgentTransport. This guide covers what changes when that "somewhere else" is a real service boundary — a separate deploy, a separate Cloudflare Worker, a separate team's API:
- exposing a producer on Node (
AgentServer) or Cloudflare Durable Objects (createRemoteAgentWorkerHandler), - transports that cross the boundary (HTTP, service bindings, injected
fetch), - getting state (
getSnapshot) and usage (getUsage) across the boundary, - fidelity mounts — exposing the same agents at two different levels of detail,
- and the consumer reconstruction contract: how an outside observer rebuilds a live agent tree's state without corrupting it.
┌─────────────────────────────────┐ ┌──────────────────────────────────────┐
│ CONSUMER service │ │ PRODUCER service │
│ │ │ │
│ parent agent │ │ Node: AgentServer │
│ └─ createRemoteSubAgentTool ──┼─ ─ ─ ─▶│ (any AgentExecutor: │
│ │ │ │ JS / Temporal / DBOS / CF) │
│ └─ RemoteAgentTransport │ │ │
│ • HttpRemoteAgent… │ │ Cloudflare DO: │
│ • serviceBinding… │ │ createRemoteAgentWorkerHandler│
│ │ │ └─ DurableObjectAgentBase │
│ (runtime-js / temporal / dbos │ │ │
│ / cloudflare / BYO-loop via │ │ routes: start resume sse status │
│ executeRemoteSubAgentDispatch)│ │ snapshot usage interrupt abort│
└─────────────────────────────────┘ └──────────────────────────────────────┘Two working examples exercise everything on this page end to end:
- Remote Agents (Cloudflare DO → DO) — two workers over a service binding, raw + projected mounts, nested children, the child-snapshot proxy, a frontend that reconstructs the whole tree.
- Remote Agents (Temporal) — a Temporal consumer against a Node
AgentServerproducer, with snapshot/usage/re-validation.
Exposing a producer
Node — AgentServer
AgentServer hosts any AgentExecutor: JSAgentExecutor, TemporalAgentExecutor, DBOSAgentExecutor, and CloudflareAgentExecutor all implement that interface, so the producer's runtime is a deployment choice, not a protocol one.
Cross-service work added these to the Node server:
| Addition | Effect |
|---|---|
GET /snapshot?sessionId= | Checkpoint-pinned customState baseline (see Snapshots). |
GET /usage?sessionId= | Recursive usage rollup. |
AgentServerConfig.usageStore | Required for /usage. Without it, /usage responds 404 FEATURE_UNAVAILABLE and consumers degrade gracefully (skip the usage fold, log a warning). Wiring it does not unlock end.usage on this producer — see the note below. |
projection / usageProjection | Per-mount fidelity (see Fidelity mounts). |
authenticate operation kinds | Now includes 'snapshot' and 'usage'. |
Honest isExecuting on /status | Derived from this replica's live handles, not a status === 'active' echo. |
import { AgentServer, createHttpAdapter, createExpressAdapter } from '@helix-agents/agent-server';
import { JSAgentExecutor } from '@helix-agents/runtime-js';
import { VercelAIAdapter } from '@helix-agents/llm-vercel';
import {
InMemoryStateStore,
InMemoryStreamManager,
InMemoryUsageStore,
} from '@helix-agents/store-memory';
const stateStore = new InMemoryStateStore();
const streamManager = new InMemoryStreamManager();
const usageStore = new InMemoryUsageStore();
const agentServer = new AgentServer({
agents: { researcher: ResearcherAgent, summarizer: SummarizerAgent },
stateStore,
streamManager,
// Without this, GET /usage 404s (FEATURE_UNAVAILABLE) and the consumer's
// rollup would be missing the remote child's tokens entirely. It does NOT
// put `usage` on the SSE `end` frame for this (JS) executor — the JS run
// loop's `end` frame never carries `usage` today (see the note below);
// the store is still required so the `getUsage()` fallback round-trip
// works.
usageStore,
executor: new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter()),
// dev/test only; production deployments must wire `authenticate`.
allowUnauthenticated: true,
});
app.use('/', createExpressAdapter(createHttpAdapter(agentServer)));Wiring usageStore on the server is sufficient for the JS runtime: AgentServer forwards config.usageStore into the executor as ExecuteOptions.usageStore on every /start and /resume. (Temporal, DBOS, and Cloudflare Workflows executors bind their usage store at construction time instead.)
Multi-replica caveat on isExecuting
isExecuting is this replica's truth. A session executing on another replica of the same service also reports isExecuting: false here. Consumers only escalate after repeated attach failures, not on a single false — but if you run replicas, put sticky routing or a shared liveness check in front of the producer. See Producer liveness & zombies.
Cloudflare Durable Objects — createRemoteAgentWorkerHandler
DO agents already speak the protocol internally (/subagent/:agentType/* on DurableObjectAgentBase). createRemoteAgentWorkerHandler (from @helix-agents/runtime-cloudflare) lifts that onto a Worker fetch handler so a different worker — over a service binding or the public internet — can consume the DO service as a remote-agent producer.
import { consoleLogger } from '@helix-agents/core';
import { createRemoteAgentWorkerHandler } from '@helix-agents/runtime-cloudflare';
const mount = createRemoteAgentWorkerHandler<Env>({
// The DO namespace hosting your agents (the same binding the DOs use as
// `subAgentNamespace`).
namespace: (env) => env.AGENTS,
// agentType allowlist. Default: no allowlist — every agentType is forwarded to
// the DO, which validates it against the session's stored agent_type.
agents: ['planner', 'thread-researcher'],
// Route prefix. Default: '/agents'.
basePath: '/internal/agents',
// Gates ALL EIGHT routes. Return `true` to proceed, `false` for a 401
// envelope, or a `Response` to short-circuit with it.
auth: (request, env) => request.headers.get('Authorization') === `Bearer ${env.API_KEY}`,
// Set `true` to acknowledge + silence the no-auth warning instead.
allowUnauthenticated: false,
// Optional. A pure state mapper; its PRESENCE makes this a projected mount.
// projection: externalStateProjection,
// Optional. Maps the rollup served on /usage and end.usage.
// usageProjection: externalUsageProjection,
// The no-auth warning goes through this. Default: noopLogger (silent).
logger: consoleLogger,
});
export default {
fetch: (request: Request, env: Env) => mount(request, env),
};Routes. {basePath}/:agentType/(start|resume|sse|status|snapshot|usage|interrupt|abort) — eight actions. sessionId is read from the body for POSTs (start/resume/interrupt/abort) and from the query string for GETs (sse/status/snapshot/usage). The DO instance is resolved with namespace.idFromName(sessionId).
Headers are forwarded verbatim to the DO, so products can map bearer tokens to tenants producer-side.
Allowlist descendants
agents must include descendant agentTypes if consumers will fetch nested baselines. A consumer calling getSnapshot(grandchildSessionId) addresses the route {basePath}/thread-researcher/snapshot — the handler's own agents allowlist rejects any agentType not listed with a 404 NOT_FOUND (a separate check from the DO's own SQL validation of :agentType against the session's stored agent_type), so agents: ['planner'] alone would 404 the grandchild. The example's internal mount allowlists both: agents: ['planner', 'thread-researcher'].
Unknown session → 404 with a RemoteAgentErrorResponse envelope ({code: 'NOT_FOUND', error}), never null. This is the reason the bridge handlers are new protocol-shaped handlers rather than the DO's native /snapshot and /usage routes: the native ones answer from in-memory execution state, can serve 200 null on a cold DO, and are not checkpoint-pinned.
Non-goals on this handler (stated plainly, so you don't go looking):
- No workspace wiring validation. DO producers wire workspaces in DO config; misconfiguration surfaces at runtime as
WorkspaceFailedError, not at construction. - No
/workspaceintrospection bridging. If you need workspace introspection across the boundary, build it as a product route.
Cloudflare Workflows
Supported, with caveats. A feasibility spike (recorded in full at docs/dev/remote-agents-cfw-producer-spike.md) verified that a plain Worker can host AgentServer + createHttpAdapter over a CloudflareAgentExecutor and serve the whole protocol: @helix-agents/agent-server has zero Node-only imports, CloudflareAgentExecutor implements AgentExecutor, and /status, /snapshot, /usage were verified live under workerd (they never touch the executor at all). The four recorded caveats:
- Interrupt latency, not correctness.
/interruptwrites a durable interrupt flag unconditionally; the Workflow polls it at every step boundary. A slow in-flight step (a long LLM call) can push observation past the default 5000 msinterruptObservationDeadlineMsand surface as HTTP 504 — the flag is still durable and still honored on the next step. - SSE connection cap applies (see Connection limits).
- A live run needs a real
AGENT_WORKFLOWbinding. The spike verified host-and-read; a live Workflow run over this topology is not covered by the repo's Miniflare setup. /startcan race the Workflow's own first step.CloudflareAgentExecutor.execute()creates the Workflow instance before its owncreateSession()pre-persist, so under adverse timing the outer call can observe "already exists" and throwAgentAlreadyRunningErroron a nominally-fresh session. It is a typed, attach-safe failure — treat it exactly like the protocol'sALREADY_RUNNINGrule below (attach, don't fail).
Service bindings & transports
serviceBindingTransport — zero-hop DO → DO
Cloudflare service bindings give you a worker-to-worker call with no network hop and no public exposure. serviceBindingTransport (from @helix-agents/runtime-cloudflare) is a thin wrapper: it returns an HttpRemoteAgentTransport whose injected fetch is the binding's, pointed at a synthetic internal base URL (service bindings ignore the authority; only the path reaches the peer).
import { serviceBindingTransport } from '@helix-agents/runtime-cloudflare';
import { createRemoteSubAgentTool } from '@helix-agents/core';
const transport = serviceBindingTransport({
binding: env.PRODUCER, // the service binding
basePath: '/internal/agents', // must match the producer mount's basePath
agentType: 'planner', // REQUIRED against a worker-handler mount — see below
// headers: { 'x-tenant': tenantId }, // static object or async factory
});
const researchTool = createRemoteSubAgentTool('research', {
description: 'Delegate a research report to the remote planner service.',
inputSchema: z.object({ topic: z.string() }),
outputSchema: ResearchOutputSchema,
transport,
remoteAgentType: 'planner',
timeoutMs: 120_000,
streamRetries: 3,
maxFoldStateBytes: 262_144, // default; see the fold section
resumeDeadProducer: false, // default; see producer liveness
});agentType is required against a createRemoteAgentWorkerHandler mount
ServiceBindingTransportConfig.agentType is optional in the type because a Node AgentServer behind a binding takes the agentType in the /start body. But a worker-handler mount's routes are {basePath}/:agentType/<action> — omit agentType and every request lands on {basePath}/<action>, which has no :agentType segment, misses the route regex, and 404s. Set it.
HttpTransportConfig.fetch — inject any fetch
HttpRemoteAgentTransport takes { url, headers?, maxRetries?, retryBaseDelayMs?, fetch? } (note: url, not baseUrl). The fetch option is used for every request the transport issues — POSTs, /status, /snapshot, /usage, and the SSE stream — which is what makes Cloudflare service bindings, Miniflare's dispatchFetch in tests, and Node-side proxies all work through the same class. It defaults to globalThis.fetch, resolved lazily.
At-least-once POSTs and the ALREADY_RUNNING → attach rule
HttpRemoteAgentTransport retries POST /start on network errors. That means a lost response can make the retry 409 against its own first attempt. The protocol makes this safe rather than pretending it doesn't happen:
/startagainst a live session returns a typed error envelope{code: 'ALREADY_RUNNING', streamId}(surfaced asRemoteAgentAlreadyRunningError), and- every consumer treats it as "attach" — proceed to stream from the existing session, not fail.
The same rule absorbs the check-then-act race where a consumer's own getStatus wakes a child DO, and the Cloudflare Workflows /start race noted above.
Custom transports
RemoteAgentTransport has eight methods: the original six plus getSnapshot(sessionId) and getUsage(sessionId). Both shipped transports (HttpRemoteAgentTransport, DOStubTransport) implement them and Zod-validate responses.
Consumers degrade gracefully when a producer can't serve a read: RemoteAgentNotFoundError and RemoteAgentFeatureUnavailableError (e.g. a /usage with no usageStore wired) both mean "skip that fold, log a warning" — they never fail the run. The same path also covers a producer that is genuinely unresponsive: each read is deadline-bounded, and a timeout degrades identically.
Snapshots: the state baseline
const snapshot = await transport.getSnapshot(sessionId);
// RemoteSnapshotResponse
// {
// sessionId: string;
// streamId: string; // the sequence space this snapshot is aligned to
// state: Record<string, unknown>;// THIS session's own customState (possibly projected)
// streamSequence: number; // checkpoint-pinned; -1 = NO POSITION
// oldestRetainedSequence: number;// retention floor; 0 = full log retained
// stepCount: number;
// checkpointId?: string;
// }The checkpoint-pinned contract
Patches reach the wire per tool, at stage time, but writes become visible to loadState() only at the step-boundary promote. So no live two-read ordering is sound: "state first, sequence second" pairs step N−1's state with a mid-step-N sequence and the in-flight step's patches are skipped forever; the inverse ordering double-applies them. Normatively, therefore:
- Non-terminal sessions: the producer serves the latest checkpoint row —
stateandstreamSequencecome from one atomically-written record. The state is exact at a step boundary, and the consumer re-receives the in-flight step's patches from the durable log. - Terminal sessions (
completed/failed): the final state withstreamSequence: -1.
fromSequence is exclusive everywhere (sequence > fromSequence). Pass snapshot.streamSequence directly — no +1, no -1.
streamSequence: -1 means NO POSITION — not "a low number"
This is the single easiest way to corrupt a reconstructed document. -1 is served when:
- the session is terminal (
completed/failed) — the producer serves the livecustomStaterow; - there is no checkpoint yet (fresh session, before its first promote);
- the checkpoint's recorded sequence is the degraded
0sentinel, or the pointer-verify loop was exhausted.
In every case the producer is handing you a state that pins a moment, not a position. There is no S to attach at, so you cannot know which patches on the wire are already baked into it. Comparing sequences against -1 admits every patch (they all exceed -1) onto a state that may already contain them — and array appends are emitted as non-idempotent add /<key>/- ops, so entries get duplicated.
Rule: against an unpinned baseline, suppress every patch for that document (including the document's own local patches — see the contract below) and re-snapshot at a boundary (that session's
subagent_end, or end-of-stream). Stale beats corrupt.
Baselines are required in all cases
Initial state is seeded server-side (RemoteStartRequest.state, stateSchema defaults) and is never on the wire. Patch streams alone are never sufficient — not for the root session, not for children. Nested baselines are obtained per-descendant: subagent_start chunks carry subSessionId; call getSnapshot(childSessionId).
Obtaining a baseline is uniform at any depth. Aligning it is not — see the reconstruction contract.
Post-terminal snapshots pin a moment, not an immutable value
onAgentComplete mutates customState after status flips, and companion continuation can reopen a completed session (completed → active). "Terminal" is per-turn. And the three recovery reads — getStatus → getSnapshot → getUsage — are not atomic, so a snapshot may reflect a later turn than the output you just harvested. Fold it as-is; if you need exactness, gate on checkpointId.
Truncation
oldestRetainedSequence is the retention floor (0 = full log retained). Streams get trimmed — Redis maxChunks, DO stream resets. If you ask /sse for a fromSequence below the floor, the producer emits a structured, non-recoverable error frame with code: 'TRUNCATED' (or a stream_resync chunk) rather than silently skipping. Refetch the snapshot and resume from its sequence. Never assume a silent gap.
One name, three snapshots
Three different things in this codebase are called "snapshot". If you are integrating services, you want (1).
| # | Surface | What it is |
|---|---|---|
| 1 | transport getSnapshot() (this page) | Protocol-layer, customState-only, checkpoint-pinned, crosses services. The one the reconstruction contract is built on. |
| 2 | ai-sdk chat-handler getSnapshot / FrontendSnapshot | Frontend hydration: messages + root state for useChat. A local, same-service surface. |
| 3 | DO-native /snapshot | A chat-shaped legacy payload for same-worker frontends. Not protocol-shaped, not checkpoint-pinned. Migrating it to checkpoint-pinning is tracked as a follow-up. |
Usage across the boundary
const { sessionId, rollup } = await transport.getUsage(childSessionId);getUsage returns a RemoteUsageResponse — { sessionId, rollup } — where rollup is the producer's getRollup(sessionId, { includeSubAgents: true }). The recursion flag is forced by the protocol handlers, so the rollup is recursive by construction: the remote child's own grandchildren are folded on the producer before anything crosses the wire.
- On a running session: a partial rollup, as-of-now.
- Once terminal: at least as of the
endevent. - Without a
usageStore:404 FEATURE_UNAVAILABLE. Consumers skip the usage fold with a warning.
Only the Cloudflare DO producer attaches usage to the end frame
end.usage is not producer-uniform. The DO producer (createRemoteAgentWorkerHandler / DurableObjectAgentBase.endStreamWithTerminalInfo) synchronously calls getRollup(sessionId, { includeSubAgents: true }) before ending the stream, so its end frame carries usage whenever a usageStore is configured and the terminal persist committed.
The other three producers — Node/JS AgentServer (JSAgentExecutor), Temporal, and DBOS — never attach usage to the end frame today, regardless of whether a usageStore is wired:
computeTerminalUsage()inruntime-js's run loop (run-loop.ts) is an unimplemented stub that always returns{}— the JS runtime does not thread agetRollup-capableUsageStorehandle through to the stream-finalization call site.- The Temporal and DBOS producers'
endStream()call sites don't pass aterminalpayload at all, so neitherstatenorusagerides theirendframe. - The Cloudflare Workflows producer (
CloudflareAgentExecutor, hosted the same way as the Node server) is in the same boat as Temporal/DBOS — itsendStream()call sites omitterminaltoo.
This is correctness-neutral: the consumer reconstruction contract never treats end.usage as authoritative — it is a fast-path optimization. Whenever end.usage is absent, consumers transparently fall back to the getUsage() recovery round-trip (one extra request), which works identically on every producer as long as a usageStore is configured. Wiring usageStore is still required on every producer for /usage and getUsage() to work — it just doesn't (yet) unlock end.usage outside the DO producer.
How it reaches the parent's rollup. The consumer's executor records a SubAgentUsageEntry pointer with the child's rollup embedded as remoteRollup (plus remoteRollupOrigin: 'end' | 'recovery'). At read time, aggregation:
- dedupes by
subSessionId; - resolves locally first (if the child lives in the same usage store), else selects exactly one embedded
remoteRollupby a total order —originpriority (recovery>end, because recovery rollups are supersets), then timestamp, then store insertion ordinal.
Duplicate entries are expected under durable retries and the dual fast/recovery recording paths. They are deduped at read, never summed.
Orphan spend is counted — and that is intended
After a JS/DO parent crash, the parent truncates to its checkpoint and re-rolls the LLM with new toolCallIds — so two pointer entries with different child sessionIds can exist for one logical call. Aggregation counts both children. The orphaned child really did burn tokens on the producer. Spend is spend.
DO usage caveat
The DO usage-store factory negative-caches failures: during a consumer-store outage it can route entries to an internal fallback store. end.usage / getUsage can therefore undercount after an outage window.
See the Usage Tracking guide for the rollup shape itself.
Fidelity mounts & projections (v1)
Mount the same agents twice with different projection / usageProjection / auth, and you get two fidelities: a raw internal view and a projected external view. The producer example does exactly this — /internal/agents/* (raw, service-binding-only) and /agents/* (projected, bearer-auth).
projection is a pure mapper, (state) => state. It is applied producer-side to point-in-time full states only: getSnapshot().state and end.state.
export function externalStateProjection(state: Record<string, unknown>): Record<string, unknown> {
const parsed = PlannerStateSchema.safeParse(state);
if (!parsed.success) return { phase: 'unknown', threadCount: 0, findingCount: 0 };
return {
phase: parsed.data.phase,
threadCount: parsed.data.threads.length,
findingCount: parsed.data.findings.length,
};
}On a projected mount, every state_patch frame is suppressed
All of them — root, descendant, and developer-authored CustomStateStreamer frames (those are state_patch chunks too). Passing them through would leak the raw state and defeat the whole point of the split. Projected-mount consumers get state via snapshot + end.state. Live incremental projected state streaming is explicitly v2: it needs a second durable projected chunk log to stay resumable.
All other frames (text, tool, step, sub-agent lifecycle) pass through untouched, with their original sequence numbers — so fromSequence resumption is unaffected.
usageProjection: (rollup: UsageRollup) => Record<string, unknown> applies to /usage and end.usage — e.g. mapping tokens to billing units. Absent = the raw rollup. Note that usageProjection alone does not suppress state_patch frames; only projection does.
Executors must consume RAW mounts
Executor-to-executor integrations (i.e. anything using createRemoteSubAgentTool) must point at a raw mount. A transport aimed at a projected mount folds the lossy projected view into the parent as the child's customState.
Mount isolation is auth-only
idFromName is mount-agnostic: raw and projected mounts reach the same Durable Objects. Fidelity separation exists only at the worker layer. Raw mounts must be auth-gated whenever they are not private by construction.
Auth posture
The two servers differ deliberately:
| Without an auth hook and without the opt-in | |
|---|---|
AgentServer (Node) | Fail-closed — the constructor throws. Provide authenticate, or pass allowUnauthenticated: true to acknowledge the risk (which then logs a loud warning through the configured logger). |
createRemoteAgentWorkerHandler (CF) | Warn-and-serve — it logs once, through the injected Logger (default noopLogger), and proceeds. Worker handlers commonly sit behind private service bindings, where a hard throw would be a false alarm. allowUnauthenticated: true acknowledges and silences it. |
The worker handler's auth hook gates all eight routes — including GET /sse and GET /snapshot (the highest-exfiltration routes) and the spend-inducing /start and /resume. Return true to proceed, false for a 401 envelope, or a Response to short-circuit.
Stated plainly:
- Service bindings are private by construction. No auth hook there is a legitimate posture (opt in explicitly with
allowUnauthenticated: true). - Public mounts MUST set
auth. - There is no built-in rate limiting and no SSE connection bounding. Put those in front of the mount yourself.
See the Security guide for the broader posture.
The consumer reconstruction contract
This is the heart of the feature: how anything outside the producer rebuilds a session's customState from a snapshot plus a patch stream, without corrupting it. It applies to independent late subscribers, to parent frontends, and to any parent-stream state consumer.
Use the shipped helper
RemotePatchFilter / createRemotePatchFilter (exported from @helix-agents/ai-sdk) implements rules 3–4 below — agentId filtering, provenance dedup, unpinned-baseline suppression, and unalignable-descendant suppression — and signals via onUnalignableBaseline. It is the normative implementation of rules 3–4 — do not hand-roll those. Rules 1–2 (getting a baseline, and attaching the stream at that baseline's own S) remain your obligation as the consumer: the filter cannot detect a cursor-less or misaligned attach from the wire, since a data-state-patch part carries no stream sequence. Get rules 1–2 wrong and the filter's rules 3–4 do not save you — see the "Consumer obligation" note in remote-patch-filter.ts.
1. Get a baseline
getSnapshot(sessionId) → a baseline at streamSequence S. A baseline is required in all cases — initial state is never on the wire.
2. Attach at the baseline's own S — mandatory, not optional
Stream /sse?fromSequence=S (exclusive). This is what guarantees no delivered patch precedes the baseline, which is precisely what makes it safe to apply the document's own local patches.
A cursor-less attach (fromSequence omitted) replays the run from its startSequence and double-applies the document's own pre-snapshot patches. It is only ever valid when the baseline is unpinned — because then every patch is suppressed anyway (rule 4).
Corollary: one attach cursor aligns one document. If you maintain several documents from a single stream whose baselines pin different positions in the same sequence space, you cannot pin them all with one cursor — seed those from a boundary snapshot instead, or track them in origin mode.
3. Filter by agentId, dedup by provenance
Apply state_patch chunks filtered by agentId = the session whose document you maintain. A parent frontend must never apply a forwarded child patch to the parent's document.
Forwarded child chunks carry remoteSource: { childSessionId, childSequence, streamId }. Dedup on (remoteSource.childSessionId, remoteSource.childSequence) — re-forward windows exist (durable retries re-stream a bounded suffix) and this makes them harmless.
For an alignable child patch: apply it only when remoteSource.childSequence > childSnapshot.streamSequence and the streamId matches.
4. Alignment is exact for the DIRECT child only
This is the rule most likely to bite you.
Every hop's dispatch re-stamps remoteSource into that hop's own sequence space (the parent-stream dedup key must be this hop's), while chunk.agentId stays the emitting agent and is never rewritten. On any parent stream:
remoteSource.childSessionId == the DIRECT child's sessionId (always)
agentId == the EMITTING agent's sessionIdSo for a grandchild (depth ≥ 2), the patch carries agentId = grandchildSessionId but remoteSource.childSequence in the direct child's space — while getSnapshot(grandchildSessionId) is pinned to the grandchild's space. Two unrelated axes. And it fails unsafely: the direct child only checkpoints after its tools return, so its pinned sequence predates every grandchild chunk, and the comparison admits patches the grandchild's snapshot already contains.
Discriminator (normative): a patch is alignable iff
agentId === remoteSource.childSessionId.
When they differ, do not align. Instead:
- seed the descendant's document from
getSnapshot(descendantSessionId), - suppress its parent-stream patches, and
- re-snapshot at boundaries — that session's
subagent_end, or end-of-stream.
The shipped filter reports this as onUnalignableBaseline(sessionId, 'nested_descendant') (latched, and permanently so — no re-snapshot can make a foreign-stamped patch alignable). It is deliberately conservative: a local in-process sub-agent of the direct remote child also carries a foreign stamp and is also suppressed, even though it would in fact have been alignable. Suppressing goes stale; admitting corrupts.
The same suppression applies to an unpinned baseline (onUnalignableBaseline(sessionId, 'unpinned_baseline'), recoverable — a later pinned snapshot re-arms it), for the reasons in Snapshots.
5. Re-snapshot triggers (mandatory)
Discard the reconstructed document and re-getSnapshot() on any of:
- a
stream_resyncchunk, - a
step_discardedchunk (patches from a step that never committed are removed from the log), - an
errorframe withcode: 'TRUNCATED'(or aStreamTruncatedError), - a
streamIdchange.
A session's sequence space is scoped to a per-session streamId that is stable for the session's lifetime, with a counter monotonic across runs, resumes, and resets. If streamId changes, your cursor is meaningless.
6. In-step weakening
Within a step, cross-sibling patch order on the wire is nondeterministic (parallel tools), and same-key parallel replaces may transiently diverge from the committed value. Coherence is guaranteed at step boundaries — not inside a step.
7. CustomStateStreamer producers MUST use a distinct agentId
Not a session id. Its document is not customState; mixing the two documents under one agentId corrupts reconstruction on the consumer side.
Late-joining parent frontends — the child-snapshot proxy
A frontend of the consumer has a problem: it cannot reach the producer (the service binding is private), and the parent's snapshot contains no child state.
The pattern: the consumer service — which does hold the transport — exposes a proxy route.
import { serviceBindingTransport } from '@helix-agents/runtime-cloudflare';
import { RemoteAgentNotFoundError, RemoteAgentFeatureUnavailableError } from '@helix-agents/core';
// consumer worker: GET /api/child-snapshot?sessionId=<childSessionId>&agentType=<agentType>
async function childProtocolRoute(env: Env, url: URL, route: 'snapshot' | 'usage') {
const childSessionId = url.searchParams.get('sessionId');
if (!childSessionId) return Response.json({ error: 'Missing sessionId' }, { status: 400 });
const transport = serviceBindingTransport({
binding: env.PRODUCER,
basePath: '/internal/agents', // the RAW mount
// A nested GRANDCHILD needs its own agentType — the producer validates the
// path :agentType against the session's stored agentType.
agentType: url.searchParams.get('agentType') ?? 'planner',
});
try {
return Response.json(
route === 'snapshot'
? await transport.getSnapshot(childSessionId)
: await transport.getUsage(childSessionId)
);
} catch (err) {
if (err instanceof RemoteAgentNotFoundError)
return Response.json({ error: err.message }, { status: 404 });
if (err instanceof RemoteAgentFeatureUnavailableError)
return Response.json({ error: err.message }, { status: 501 });
return Response.json({ error: String(err) }, { status: 502 });
}
}On the wire to the browser, ai-sdk data-state-patch parts carry agentId and remoteSource (StatePatchPartData), so the frontend can demux. Frontends MUST drop parts whose agentId ≠ the document they maintain — that is rule 3, and RemotePatchFilter.shouldApply() is its implementation.
Explicit limitation
In-flight remote-child state is not reconstructable from the parent surface alone in v1. A late-joining parent frontend sees child state from exactly three places:
- live patches that arrive after it joins,
- the proxy above, or
- parent-folded summaries —
afterSubAgent→childCustomState→ parentupdateState(), which makes them visible in the parent's snapshot.
Parent-log replay-from-0 is NOT a valid child-state oracle. The parent's log retains child patches from steps the child later discarded.
Folding child results into the parent
The child's final customState reaches the parent hook as AfterSubAgentPayload.childCustomState. It is populated from:
end.state— only whenend.status === 'completed', orgetSnapshot()on recovery paths, for terminal statuses (including interrupted-with-output).
Terminal-only by design. Mid-flight folding is an explicit non-goal: it would drag mid-flight sequence reconciliation into the parent. Parents that want live child state consume the stream.
The end handling rule: only status: 'completed' folds output/state/usage directly. paused and interrupted route through status-based recovery. (This discriminator is why a paused child no longer surfaces to the parent LLM as "completed, output null".)
hooks: {
afterSubAgent: (payload, context) => {
const child = payload.childCustomState;
if (!child) return;
context.updateState((draft: WriterState) => {
// Idempotent SET, not an append.
draft.research = { phase: child.phase, findingCount: child.findings.length };
draft.researchSessionId = payload.subSessionId;
});
},
}Fold guidance: idempotent set/mark, not append
On at-least-once runtimes, afterSubAgent can re-fire. state.results.push(...)duplicates in crash-retry windows. On JS/DO the exactly-once fold guarantee is only per run-attempt — a parent crash + resume re-rolls the step — so the same guidance applies there too. Prefer idempotent set/mark folds.
Output re-validation is default-on: end.output is safeParsed against the tool's outputSchema, and a failure surfaces as a structured tool error rather than flowing garbage into the parent's LLM context.
maxFoldStateBytes (default 262_144) caps the serialized child state routed through the fold. Over-cap: the state fold is skipped with a warning; output and usage are unaffected.
The compact-customState-summary pattern
Store-derived state views are deliberately out of the SDK — the protocol's resumable state view is customState, full stop. If your producer keeps rich derived state in an external store (a D1 research tree, say), the sanctioned pattern is to also maintain a compact summary in customState:
{ phase: 'researching', threadCount: 4, findingCount: 17 }That summary streams as patches, snapshots cheaply, folds into parents, and survives projection. Serve the rich view out-of-band from your product API.
Producer liveness & zombies
Node producers have no self-heal. Unlike the DO wake ladder, if the process dies mid-run, the session stays running forever. This is why /status.isExecuting became honest (this replica's live handles).
The shared dispatch escalates after N attach attempts against status === 'running' && isExecuting === false:
- with
RemoteSubAgentConfig.resumeDeadProducer: true, it callstransport.resume({ mode: 'continue' })— opt-in, because a blind resume is wrong for a multi-replica producer whereisExecuting: falsemay just mean "executing on another replica"; - otherwise it fails fast with a distinct
'producer-dead'failure reason, instead of burning the whole retry budget.
Run a cleanup sweep on Node producers (expiredSessionCleanup from @helix-agents/agent-server) so zombie sessions eventually become failed.
Consumer-side orphan reconciliation (JS/DO). On parent resume after a crash, the executor sweeps SubSessionRefs with status: 'running' + remote metadata:
- if the producer reports terminal, the result is reconciled into the parent (and the tool result injected, if the tool call survived truncation);
- if the producer is non-terminal and the parent's tool call did NOT survive truncation, the child is an orphan — reattach is impossible (a new LLM roll produces new toolCallIds) — so the executor issues a best-effort
transport.interrupt()to bound producer-side spend and marks the reffailed; - if the tool call did survive, the ref is left
running: the LLM re-calls the tool and the deterministic child sessionId reconnects to the existing session.
The orphan's tokens still count — see Usage.
Interrupt propagation is best-effort per hop.
Connection limits & fan-out
A Workers request context allows roughly 6 simultaneous open connections, and each in-flight remote child holds a long-lived SSE fetch. A parent that fans out many parallel remote children over bindings or stubs can stall — the connections are also competing with the parent's own LLM call and any other subrequests.
Bound your parallel remote fan-out. Sequence threads, or batch to ~4–5 concurrent remote children, leaving headroom.
Limitations
These are v1 constraints, stated so you don't discover them in production.
- Client-executed tools and approval-gated tools inside remote children are not supported. Both suspend via
pendingClientToolCalls, and both hit the consumer's fail-fast guard (which names both cases:RemoteSubAgentClientToolUnsupportedError, failure reason'client-tool-unsupported'). Cross-service suspension is a separate project. Theend.status: 'paused'discriminator is what makes the guard's recovery deterministic instead of a silent success-null. - Persistent / companion remote children are not expressible.
RemoteSubAgentConfighas nomode; remote refs are ephemeral. Extending the companion protocol across services is future work. - No distributed trace continuity. Remote children start unlinked traces. Thread trace ids yourself via
RemoteStartRequest.metadata— every consumer forwards the parent'suserId/tags/metadatato the child — plus producer-side hooks. - No packaged late-subscriber client. The reconstruction contract on this page (and
RemotePatchFilter) is the documented pattern; a fully packaged independent-subscriber client is future work. - No protocol version on the wire. There is no version field and no negotiation: every producer and consumer ships in this repo and deploys in lockstep. A breaking wire change is a coordinated redeploy, not a runtime handshake.
- No automatic tree snapshots. Per-descendant
getSnapshot(childSessionId)is the uniform way to obtain a baseline; recovery is not uniform (rule 4 above). - Live incremental projected state streaming is v2. Projected mounts serve state via snapshot +
end.state.
Next steps
- Remote Agents — the base protocol,
createRemoteSubAgentTool,HttpRemoteAgentTransport. - Cloudflare DO Runtime —
createAgentServer,DurableObjectAgentBase, and the DO producer surface. @helix-agents/agent-serverreference — every route, config field, and auth hook.@helix-agents/runtime-cloudflarereference —createRemoteAgentWorkerHandler,serviceBindingTransport.@helix-agents/corereference — the wire-protocol types and schemas, theRemoteAgentTransportinterface, and the completion-fold helpers (processRemoteSubAgentCompletion,validateRemoteOutput).- Usage Tracking — the rollup shape and the aggregation contract.
- Example: Remote Agents (Cloudflare DO → DO) — everything on this page, running.
- Example: Remote Agents (Temporal) — the Node
AgentServerproducer path.