| CP-01 | S1 | handleChatStream path 5 attaches to a live run without reading params.messages: a new user message is dropped with HTTP 200. A unit test pins this as intended. | ai-sdk/src/handler/handle-chat-stream.ts:284-354 (pre-fix, origin/main 5bd30e20da); ai-sdk/src/__tests__/handler/handle-chat-stream.test.ts:419; conformance: fixed by !352 (WE-2, db8d0763ba), the same defect as CP-13's reject half: path 5 now defers any request whose trailing message is a new user message (ai-sdk/src/handler/handle-chat-stream.ts pathFiveAllowed, the hasTrailingUserMessage exclusion), so a send during a live run takes the fresh-turn path and is refused with a typed 409 state_already_running instead of being attached and dropped. chat.send-during-live-run-refused and chat.send-during-live-run-refusal-names-session pass on cf-do and js-chat-host, and chat.send-during-live-run-awaiting-children on js-chat-host (owner batch 29 window on db8d0763ba, 3 of 3; green in CI since) | source, live-repro | verified | fixed (5 cells) |
| CP-02 | S1 | AlreadyRunning fallback in handleFreshTurnPath drops the message with HTTP 200. | ai-sdk/src/handler/handle-chat-stream.ts:963-1008 (pre-fix, origin/main 5bd30e20da); ai-sdk/src/handler/handle-chat-stream.ts:1222-1227 and :2194-2203 (fix evidence, !288, as on origin/main 137b07d57a) (handleFreshTurnPath: an AlreadyRunning refusal of a request carrying new user content returns alreadyRunningSendResponse, never an attach to the running run; alreadyRunningSendResponse: 409 { error, code: 'state_already_running', retryable: true } with X-Session-Id). WA MR1 (!340) routes the same refusal through typedEntryErrorResponse (handle-chat-stream.ts:1518-1525 (origin/main 487090d3fb), handleFreshTurnPath's call; the busy branch at :2874-2877 (origin/main 487090d3fb)) to alreadyRunningResponse (:2793-2805 (origin/main 487090d3fb): the same 409, its message scrubbed); conformance: lifecycle.send-in-terminal-tail (host, owner batch 15, f083dafb67 product code) passes on cf-do. Turn 1 completes with the DO's host onComplete held, so the session reads completed while the DO still executes; a chat send of turn 2 in that window is refused with a typed, retryable 409 state_already_running (.send-not-dropped), and turn 1 then completes (.turn1-completes). The fix predates the cell, so the red half is a minimal mutation of its mechanism, not an MR revert: with handle-chat-stream.ts:1226 (origin/main 137b07d57a) returning the pre-fix attach fallback (streamLiveResponse on the session's stream from its live head, as at :958-1007 (origin/main 5bd30e20da); working tree only, rebuilt, then reverted), the cf-do cell failed exactly .send-not-dropped: the send answered HTTP 200 and the session never held TURN-2. js-chat-host runs the same handler but has no host hook to hold the tail open (a declared gap), so it is not in provenBy; conformance: provenBy moved (WA MR1 Task 20, owner) from lifecycle.send-in-terminal-tail to lifecycle.send-during-entry-admission. WA MR1 removed the terminal tail: the DO releases its execution registration right after the run's own stream end, with nothing awaited in between (runtime-cloudflare/src/do/drive/settle.ts:327-329, after runtime-js's synchronous K5 → run status → endStream on DO-local SQL; do-stream-manager.ts:514), and delivers onRunSettled after the release. So a send can no longer land between completed and the release, and the old cell's held onComplete has no production counterpart. The new cell drives the same handleFreshTurnPath fallback through a real window: send A is held in the host's beforeEntry admission (the drive holds the DO's producer claim across it, run-drive.ts runDrive), and send B, which read the session as completed, is refused at the claim as already running, then answered with a typed 409 (.send-not-dropped). Run (WA MR1 Task 20 preliminary, dbf1d82406 + the owner commits): the cf-do host cell passes. Red half, batch 15's mutation (handleFreshTurnPath's AlreadyRunning branch answering streamLiveResponse on the session's stream; working tree only, rebuilt, then reverted): only .send-not-dropped fails (B answered 200; the session holds only turn 1), and the cell passes again after the revert. Final verification on WA's final head | source | verified | fixed (1 cells) |
| CP-03 | S1 | Generic execute() failure → HTTP 200 SSE containing only data-resume-rejected; no client code reads it. | ai-sdk/src/handler/handle-chat-stream.ts:1026-1030, 2153-2164 (pre-fix, origin/main 5bd30e20da); related: ledger split with DI-39: this record owns the generic send fallback; DI-39 owns the completion-support refusal on the chat send, resume and submit paths; ai-sdk/src/handler/handle-chat-stream.ts (fix evidence, !403, 3419d624ed) (a failed chat request is a typed HTTP error, never a 200 that no client reads); conformance: fixed. chat.send-unclassified-failure-host (owner batch 40) passes on cf-do 3 of 3 at owner batch 7be28c1611 (main 22cb4e513a + the owner's batches), 2026-10-18, and fails .refused-typed and .never-silent-200 3 of 3 with !403's product diff reverse-applied (owner c2, working tree only, reverted). js-chat-host is a declared gap (no DO answer to fault); conformance: re-anchored with !409 (WA S-B (F), orchestrator ruling 2026-10-19): the cell's injection, a malformed 2xx /start answer, is now typed by the DO client itself (its readEntryAnswer in !409: a non-retryable transport_error, since the DO answered success), so .refused-typed expects 500 transport_error typed where it expected 503 framework_internal_error; .never-silent-200 and .nothing-written are unchanged, and the cell still fails on this record's defect (a 200 of only data-resume-rejected). The route's fallback for a codeless throw (503 framework_internal_error) gets its own witness in a sibling scenario (owner batch, pending); conformance: fixed (!409 merged, main cd2d6e3169; its head fc0fbe4c0b, MR pipeline 2964774349 green with the re-anchored cell): chat.send-unclassified-failure-host passes on cf-do 3 of 3 and chat.send-codeless-entry-failure-host (the route's fallback for an entry error with no code, over the js-chat-host control fault-entry-throw) passes on js-chat-host 3 of 3 (owner slot s64, 2026-10-19, owner batch 4ab4ffd345 = main cd2d6e3169 + the owner batch; the codeless cell also 3 of 3 in owner slot s62 on main f0b3da371e with the re-anchor); cf-do is a declared gap for the codeless cell (its DO client types every entry failure) | source | verified | fixed (2 cells) |
| CP-04 | S2 | streamHandle null / throwing after retries → 200 rejected-only, although the run was started and persisted. | ai-sdk/src/handler/handle-chat-stream.ts:1933-1966 (pre-fix, origin/main 5bd30e20da); ai-sdk/src/handler/handle-chat-stream.ts (fix evidence, !403, 3419d624ed) (a stream that fails to open after the run's commit is answered typed, never a silent 200); conformance: fixed. chat.stream-open-fails-after-commit-host (owner batch 40) passes on cf-do 3 of 3 at owner batch 7be28c1611 (main 22cb4e513a + the owner's batches), 2026-10-18, and fails .answer-typed and .never-silent-200 3 of 3 with !403's product diff reverse-applied (owner c2, working tree only, reverted). js-chat-host is a declared gap | source | verified | fixed (1 cells) |
| CP-05 | S2 | Abandonment path polls 5 s for paused to clear, then falls into the AlreadyRunning drop. | ai-sdk/src/handler/handle-chat-stream.ts:446-480 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-06 | S1 | No idempotency: UIMessage.id is dropped on input; a retried POST after completion re-executes and duplicates the turn. | ai-sdk/src/handler/handle-chat-stream.ts:1602-1719 (pre-fix, origin/main 5bd30e20da); core/src/store/request-key.ts:35-45 (fix evidence, WE-2 D-2 MR 1: parseRequestKey, 1-128 chars of [A-Za-z0-9._:-], a malformed key a typed validation_error) and :161-172 (planKeyedSend: a send is one request whichever route each attempt took; the same key with other input a conflict); ai-sdk/src/handler/handle-chat-stream.ts:1625 (the chat handler's key check: the session's current run holding the key with the same input is streamed or replayed, other input a 409 state_request_key_conflict); ai-sdk/src/react/use-helix-chat.ts:518 (useHelixChat mints a key per send and keeps it on retry()); ai-sdk/src/cloudflare/chat-handler-factory.ts:83-99 (the CF chat handler factory's handleChat takes the incoming request, REQUIRED, so handleChatStream reads the key from it on every CF chat route; requestKey an override for non-HTTP callers); conformance: red on main 17904c3a8d, 3 of 3 (owner window, host load 29 to 37; the D-2 cells with the harness's request-key, resend-request and client-retry controls): a repeat of a live run's keyed send is refused 409 state_already_running (chat.send-retried-attaches, and chat.send-retried-suspended-attaches on js-chat-host); a repeat of a finished turn, or of one a later turn followed, runs it again (chat.send-retried-after-terminal, chat.send-retried-after-later-turn: the user row twice); the same key with another message runs a second turn (chat.send-key-conflict); a malformed key is accepted and runs (chat.send-key-malformed); the product client sends no X-Helix-Request-Key at all (client.send-carries-request-key, client.retry-reuses-key). The keyless control (chat.send-keyless-unchanged) is green; conformance: fixed on WE-2's D-2 MR 1 (head e73c6d5c9d plus the cells and this record): every cell below passes, judged by WE-2's lock window with this commit applied; chat.send-keyless-unchanged stays green (a keyless repeat still runs again); conformance: client.retry-reuses-key's .send-settles is judged on the TURN-2 POST's own answer since !381 (owner): the client tier now mounts the example's S5 page, which passes resync to useHelixChat, and a send refused 409 state_already_running re-anchors at once and follows the live run (ai-sdk/src/react/use-helix-chat.ts:541-566 (origin/main af9dfde232)), so with turn 1 held the client never goes quiet before the release. Re-checked in the owner's !381 window: both client cells pass, and a mutation dropping the refused send's key from retry()'s re-send turns .retry-same-key red on cf-do and js-chat-host, nothing else. Its .turn1-settles also waits, on cf-do, for the host to report the run no longer executing before the retry: a cf-do send between the reply's end and the DO drive's tail is refused 409 (its own finding), which made .retry-resent a race (!386's pipeline, job 17251605874: [409, 409]) | source, live-repro | verified | fixed (15 cells) |
| CP-07 | S1 | HelixChatTransport (createHelixChatTransport, the agent-server transport) sends no fromSequence on /sse, turns a network error in sessionExists into a /start, sends only the last user message's text, re-sends the previous user text on a tool-output submit, and cannot send approvals. (Its unchecked /start / /resume answers and its /sse read that ignored response.ok are fixed by WE MR C1: see the fix evidence.) | ai-sdk/src/transport/helix-chat-transport.ts:96-135, 211-268, 322-349 (pre-fix, origin/main 5bd30e20da); ai-sdk/src/transport/helix-chat-transport.ts:144-160 (pre-fix, origin/main b6b3eeb141) (postJSON returns the Response) with :335-349 and :358 (pre-fix, origin/main b6b3eeb141) (every /start and /resume call site discards it: a send's /resume and /start, and the continuation /resume after tool outputs) and :162-173 (pre-fix, origin/main b6b3eeb141) (openSSE checks only response.body, never response.ok). So a refused entry (409 state_already_running, a request-key conflict, an already-completed key, a 400) still opened /sse and rendered whatever run that stream carried as this send's reply, the CP-13 class of silent wrong reply; related: found by WE-2 while writing WE MR C1 (2026-10-16); the reject half's red-first unit test is ai-sdk/src/__tests__/transport/helix-chat-transport-refused-send.test.ts:103 (409 state_already_running, 409 state_request_key_conflict, 409 state_request_already_completed and 400 validation_error each went on to GET /sse before the fix) with its accepted-send control at :125; ai-sdk/src/transport/helix-chat-transport.ts:176-187 with :379-396 and :402 (fix evidence, WE MR C1: postEntry; a non-2xx /start or /resume rejects sendMessages with its body as the message, which parseHelixChatError reads, and nothing is opened after it, at all three entry sites) and :189-207 (fix evidence, WE MR C1 Task 10: openSSE rejects a non-2xx /sse with its typed body), proven by ai-sdk/src/__tests__/transport/helix-chat-transport-refused-send.test.ts:103 and ai-sdk/src/__tests__/transport/helix-chat-transport-wire-v2.test.ts:56 (a 503 /sse answer rejects with the typed error parseHelixChatError reads). No conformance cell: the client tier mounts only DefaultChatTransport against the chat routes; a page over createHelixChatTransport against agent-server is planned after the C1 cells | source | verified | open |
| CP-08 | S2 | ai@6 Chat.sendMessage never rejects (third-party design); the SDK layers no delivery contract on top. | ai@6.0.91 dist/index.mjs:12781-12794, 12498-12512 | source | verified | open |
| CP-09 | S2 | useResumeClientTools: addToolOutput is fire-and-forget; a tool without a handler is silently left input-available. | ai-sdk/src/react/use-resume-client-tools.ts:460-495 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-10 | S1 | Client-tool calls owned by an in-process sub-agent are not continued: on current main the answer is stored on the CHILD's pending entry (the submit routes to the call's owner), but the continuation resumes only the ROOT, whose resume has no step for an awaited local child (its batch gate and drain read the root's pending map only, no heal runs for a paused root, nothing reads suspendedAwaitingChildren), so the root's run loop calls the model with the sub-agent's tool_use unpaired (RM-29 class: a provider rejects it) while the child stays paused with its answer undrained. Earlier symptom (pre-Part B): the call was invisible to handleChatStream and rejected as unknown-tool-call with a 200. | live: WE-2's probe (2026-10-19, a plain runtime-js executor on main 4980e8a8fb's source, also re-run as a temporary vitest file), two facts beyond the client-tool case: (1) the APPROVAL variant, a local child's requireApproval call, answered approved on the root (submitToolResult approval-response: accepted, caller_resumes), is silently never executed after resume(root): the child's entry keeps submittedResult approved unconsumed and the tool never runs; (2) a root WITHOUT an outputSchema then ends completed, with history [user, assistant(sub-agent call), assistant(next step), tool] and the sub-agent call unpaired in a completed session (RM-29), while a root WITH an outputSchema ends failed. Recorded first as CP-189 (withdrawn, a duplicate); runtime-js/src/run-loop.ts:2652 (origin/main 4980e8a8fb) (suspendedAwaitingChildren is written on the park; core run/reconcile-closed-owner-run.ts:183, origin/main 4980e8a8fb, is its only reader) with runtime-js/src/__tests__/submit-tool-result-routing.integ.test.ts:149-157 (origin/main 4980e8a8fb) (the routing test resumes the CHILD by hand before the parent); related: fix W1's draft !421 (0a51ab0170, stacked on !420 and !417; orchestrator, 2026-10-19); CP-186's .run-completes cells wait on it; conformance: fixed by !421 (W1): W1's gate gCombo (2026-10-19) on the tree !421's head e3bcc5fbfa plus the owner's cells 9c455b3dfe (tree a211f5dceb): the 12 executor cells (subagent.resume-drives-suspended-child-approval, subagent.resume-drives-suspended-child-approval-os, subagent.resume-drives-suspended-child-client and subagent.resume-drives-suspended-child-client-os) on js-memory / js-redis / js-postgres pass 3 of 3 and fail at !420's runtime-js product 1 of 1 on exactly .child-ran-call, .child-completed and .sub-call-paired (preconditions passing); the ledgered host cells submit.subagent-client-tool-host (.child-answered) and submit.subagent-client-tool-paired-host (.root-input-paired, .root-sees-child-result, .child-leaves-paused) on js-chat-host are unexpected-pass 3 of 3 at !421 and expected-fail at !420's product; the results files read by the owner; contrast: the same executor cells pass at main 4980e8a8fb on temporal-memory / temporal-redis / temporal-postgres, dbos-postgres and cfw-workflows-d1, 4 shapes each (owner slots s74 / s75): the defect is runtime-js only (Temporal re-spawns the parked child on resume; DBOS's submit wakes the waiting root itself; CF Workflows re-spawns it); they stay as guards; ai-sdk/src/handler/extract-resume-intent.ts:446; ai-sdk/src/handler/handle-chat-stream.ts:238, 371, 446-449 (pre-fix, origin/main 5bd30e20da); runtime-js/src/run-loop.ts:1802-1902 (pre-fix, origin/main 5bd30e20da); conformance: submit.subagent-client-tool-host (owner batch 40; a parent's in-process sub-agent suspends on a client-tool call, the client answers through the chat route) fails .child-answered on js-chat-host (host), 3 of 3 at owner batch 7be28c1611 (main 22cb4e513a, which includes !403's Part B), 2026-10-18: the submit is visible, accepted and completes and the parent completes, but the child session stays paused with no tool rows, and no transcript holds the answer: it went nowhere. cf-do is a declared gap (DO sub-agents are remote). Part B (!403) did not close this lane; back with W1; runtime-js/src/js-agent-executor.ts:5776 (origin/main 22cb4e513a) (submitToolResult's startContinuation answers caller_resumes for the ROOT session) with ai-sdk/src/handler/handle-chat-stream.ts (origin/main 22cb4e513a) (handleChatStream resumes the root on caller_resumes); core client-tool route-submit.ts:463 (origin/main 22cb4e513a) (the submit routes by the root's clientToolCallOwnership to the owning child and stores submittedResult on its pending entry); runtime-js/src/run-loop.ts:2599 (origin/main 22cb4e513a) (the only reference to suspendedAwaitingChildren in runtime-js is its write: the root's resume never resumes an awaited local child); reconcileRemoteSubAgents filters to remote refs, and the iteration-top injection takes only terminal children (W1's root cause, peer-reported, 2026-10-18); related: a root resume() also succeeds while the child's call is still UNANSWERED (the gate ignores children), the root's ownership entry is never cleared, and a re-sent answer ends in a refused resume of the completed root (W1). Contrast: CF Workflows' root resume re-spawns awaited children (runtime-cloudflare/src/workflow.ts:1675-1735 (pre-fix, origin/main 826d9665af), childrenToRespawn); on the CF DO each child is its own DO and decides its own continuation. Fix direction (W1, after RM-82): the root's resume drives the cascade; the batch gate covers the whole tree and awaited local children are resumed first through their own entries; conformance: submit.subagent-client-tool-paired-host (owner batch 43j) fails .root-input-paired, .root-sees-child-result and .child-leaves-paused on js-chat-host 3 of 3 at owner batch 1f937ca39c (main 22cb4e513a), 2026-10-18: the root's later model call holds the sub-agent call unpaired and no result from the child, and the child stays paused | source | partly-inferred | fixed (14 cells) |
| CP-11 | S1 | DO discards approval decisions: the DO forwards an approval-response submit to the low-level JsClientToolResolver.submit, which drops approved/reason (result and error both undefined), so the decision never reaches the approval gate. (The JS runtime's own public JSAgentExecutor.submitToolResult persists { approved, reason } correctly — the resolver primitive is reached only by the DO.) Fixed on the CF DO by fb7c1e12f5 (WA MR1): the resolver maps an approval-response to { approved, reason } and the DO's submit route delegates to the executor's submitToolResult. | runtime-cloudflare/src/do/durable-object-agent-base.ts:2988-2994, 3040-3046 (pre-fix, origin/main 5bd30e20da); runtime-js/src/client-tool-resolver.ts:654-655 (pre-fix, origin/main 5bd30e20da) (submit(): approval-response → result/error undefined); runtime-js/src/js-agent-executor.ts:1266-1272 (pre-fix, origin/main 5bd30e20da) (consumer: isApprovalResponseResult(undefined) is false, so the gate never sees a decision); contrast: runtime-js/src/js-agent-executor.ts:3985-3991 (pre-fix, origin/main 5bd30e20da) (public submitToolResult persists { approved, reason }); related: fixed in code by fb7c1e12f5 (WA MR1, !340), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-js/src/client-tool-resolver.ts:636-647 (origin/main fb9150cc16) (submit() records an approval-response as its { approved, reason } payload; its comment names the CP-11 trap) and the DO's submit route delegates to the executor's submitToolResult (WE-2, verified); unit witness runtime-js/src/__tests__/submit-local-owner.test.ts (WA MR1 Task 6). Open until its cf-do cell (an approve and a deny through the DO, each reaching the gate); related: fixed in code by fb7c1e12f5 (WA MR1, !340), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/client-tool-resolver.ts:659-665 (origin/main 56d564d073) (submit() maps an approval-response to its { approved, reason } payload, with the CP-11 comment at :654-657 (origin/main 56d564d073)), runtime-cloudflare/src/do/durable-object-agent-base.ts:4348 (origin/main 56d564d073) (the DO's submit route delegates to the executor's submitToolResult); conformance: approval.do-approve-and-deny-reach-the-gate (cf-do host): an approve and a deny POSTed to the DO's own /submit-tool-result route are each answered 200 accepted and each reach the gate: the approved call's row is the tool's real output, the denied call's row is the gate's canonical denial, and the next model call carries both, 3 of 3 identical on cf-do|host (WA slot slot-cp11 v2, lock A, 2026-11-04, tree 782872563a). With fb7c1e12f5's rule restored (the router's local store routed back to JsClientToolResolver.submit, whose approval drop is restored), .approved-tool-ran, .denied-tool-not-run and .model-told-both fail. The deny's reason is sent but not asserted: the gate answers a denial with its canonical text, so no product output carries it. | source | verified | fixed (1 cells) |
| CP-12 | S1 | The client stack still cannot stop a run: useHelixChat / useChat().stop() and the ai-sdk transports only abort the local stream while the server run continues to completion, the generic handleChatStream (non-Cloudflare) and the Express adapter expose no stop entry, and the opennext example wires no stop route. (The server half for the CF Durable Object is fixed: the CF chat handler factory's stop(), !355.) | ai-sdk/src/{handler,cloudflare,adapters,transport,react,client} (grep: no stop / interrupt wiring); core/src/types/runtime.ts:604-630 (pre-fix, origin/main 5bd30e20da); agent-server/src/http/handler.ts:124-155 (pre-fix, origin/main 5bd30e20da); related: narrowed after !355 (WA MR2): the CF chat handler factory now exposes stop({ sessionId, kind?, reason? }) (ai-sdk/src/cloudflare/chat-handler-factory.ts:146-154, implementation at :246), posting only to the DO's /interrupt or /abort and passing its typed answer through (202 stopping, 200 interrupted / failed, 409 state_not_stoppable, 404 state_session_not_found, 503 transport_error); the DO routes are covered by do-stop's cells. Still open: nothing on the client calls a server stop (ai-sdk/src/client/resync.ts and react/resync-hooks.ts stop only the local stream), handleChatStream and the Express adapter have no stop entry, and the opennext example wires no stop route; conformance: client.stop-reaches-server (client tier, the real useHelixChat; owner commit d31ec0ee6c, carried by !379) on cf-do and js-chat-host, before the fix (main 124149eacb plus the cell, no serverStop), 3 of 3 each (host load 8 to 10 on 14 cores): turn 1 streams one delta and is held, the client calls the hook's own stop(), then the model's hold is released. The client ends chat:ready:abort, but the server session reads completed, not interrupted (.server-run-stopped), the full held reply is persisted (.no-full-reply), and a reloaded client shows the whole reply (.reload-sees-stopped); ai-sdk/src/react/use-helix-chat.ts:131 and :486-511 (fix evidence, W1's CP-12 MR !379: useHelixChat's serverStop option; the hook's returned stop() posts the server stop once through requestServerStop, fire-and-track, then stops the local stream, and re-anchors when the server answers stopped), with ai-sdk/src/client/server-stop.ts:22 and :48 (ServerStopError, requestServerStop, also for a non-React driver); ai-sdk/src/handler/handle-chat-stop.ts:76-122 (fix evidence, !379: handleChatStop, the generic chat stop route, authorize required, allowedKinds default ['interrupt']) over core/src/run/stop-session.ts:65-83 (fix evidence, !379: stopSession, the one session-level stop path, which agent-server's stopAgent also uses, agent-server/src/agent-server.ts:946-951), with ai-sdk/src/adapters/express.ts:287-302 (fix evidence, !379: expressRequestToWebRequest for the Express stop route) and examples/opennext-cloudflare-do/src/app/api/chat/[sessionId]/stop/route.ts:28-46 plus examples/opennext-cloudflare-do/src/lib/use-snapshot-chat.ts:93 (fix evidence, !379: the example's stop route and its serverStop wiring); conformance: fixed by W1's !379: client.stop-reaches-server passes every id on cf-do and js-chat-host, client tier, at !379's head d31ec0ee6c (the client harness gives useHelixChat serverStop { url: <chat path>/stop }; js-chat-host serves that route with handleChatStop and cf-do with the CF factory's stop()), 3 of 3 each (W1's lock slot, 2026-10-16, host load 15.9 / 9.8 / 7.8). The Express adapter and the opennext route are covered by !379's own tests, not by a conformance cell | source, live-repro | verified | fixed (2 cells) |
| CP-13 | S1 | A chat send (a new user message) while a run is live is attached by handleChatStream (path 5) to that run's stream instead of being refused or started: the request gets 200 with the running turn's text and the message is silently dropped. So "last message wins" supersede (formerly beforeStart, now concurrentEntry: 'interrupt_previous') is unreachable through the chat route, and the consumer's research and writer cancel routes, which rely on it, never stop the run. | ai-sdk/src/handler/handle-chat-stream.ts:298-345 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/types.ts:207-210 (pre-fix, origin/main 137b07d57a); live: writer cancel route's handleChat('cancel') attached as passive subscriber, message dropped, run continued seq 2303→4452 over 4+ min; runtime-cloudflare/src/do/types.ts:308-316 (as before WA MR2, origin/main cae966c0a7) (still open on WA MR1 (!340): the beforeStart + executionState.interrupt() example is deleted, and its replacement is concurrentEntry: 'interrupt_previous', example at :269), but handleChatStream path 5 still attaches a send to a live run before /start is ever called (ai-sdk/src/handler/handle-chat-stream.ts:409-452 (origin/main 6ab56e113a), streamLiveResponse at :442 (origin/main 6ab56e113a)), so the supersede is still unreachable through the chat handler; conformance: narrowed to the reject half, the common case (WE-2's probe on main 6ab56e113a: a run mid-step with its stream active, then a send with a new user message, answered 200 with turn 1's text and executed 0). chat.send-during-live-run-refused (owner batch 29, host tier, cf-do and js-chat-host): turn 1 streams one text delta and is held, and a send with a new message is made. On main 6ab56e113a plus owner batches 25 to 28 it fails .send-refused-typed on both hosts, 3 of 3: the send is not answered within 5 s, attached to turn 1's live stream, and after the release turn 1 completes with the history holding only turn 1, the new message dropped. The control chat.reconnect-during-live-run-attaches (a reconnect GET with no new message during the same held turn) passes on both hosts. The record's interrupt_previous half, a host's concurrentEntry policy reached through the chat route, moves to WE-2's D-2 slot and has no cell yet; the 'refused 409 since !288' reading holds only in the run-start window and the CP-84 shape. Not covered: the X-Session-Id header on the 409, and a session paused awaiting children (a held child) with no pending client tools; conformance: the awaiting-children case. chat.send-during-live-run-awaiting-children (owner batch 29, host tier, js-chat-host): the parent's sub-agent child suspends on a client-executed tool, so the parent's run is suspended_awaiting_children and its session paused with no pending call of its own; then a send with a new message. On main 6ab56e113a plus owner batches 25 to 28 it fails .send-refused-typed 4 of 4: the send attaches to the paused stream and ends at its pause (stream:data-run-paused); nothing is written. On cf-do it is a declared gap: the DO runs every sub-agent through its remote rewrite, where a client-executed tool is unsupported by design, so no parent awaits a suspended child there; conformance: fixed by !352 (WE-2, db8d0763ba). Path 5 no longer attaches a send (ai-sdk/src/handler/handle-chat-stream.ts:436-441: a trailing user message defers to the fresh-turn path), so the runtime's refusal reaches the client as the typed 409 state_already_running. The DO client marks every chat entry with an internal header (runtime-cloudflare/src/do/do-clients.ts:1098-1102), and the drive refuses a marked entry over an executing run, or a marked /start over a run suspended on its children or a partial step (runtime-cloudflare/src/do/drive/run-drive.ts:394-416 (pre-fix, origin/main 487090d3fb)). On main db8d0763ba plus batch 29, 3 of 3: chat.send-during-live-run-refused and chat.send-during-live-run-refusal-names-session pass on cf-do and js-chat-host, and chat.send-during-live-run-awaiting-children passes on js-chat-host (cf-do is the declared subagent-client-tool gap); the control chat.reconnect-during-live-run-attaches stays green on both hosts; related: D-2 design note (D-2: an idempotency key for retried requests, §5: a keyed send is not marked, so the DO's concurrentEntry policy applies to it and interrupt_previous becomes reachable through the chat route); CP-13 backlog row 'interrupt half with D-2'; FU-S5-PATH5-FOREIGN-TAB-SEND in docs/dev/follow-ups.md. The interrupt_previous half was a missing capability, not a silent failure: until D-2's key landed (MR 1.5b), a chat send over a live run on an interrupt_previous DO is refused with the same typed 409 (do-clients.ts:1098-1102), never attached, so nothing is dropped silently; runtime-cloudflare/src/do/do-clients.ts:1137-1148 (fix evidence, WE-2's D-2 MR 1.5b: the interrupt half. The DO client marks only a KEYLESS chat entry (REJECT_CONCURRENT_ENTRY_HEADER), which the drive refuses over a live run; a keyed entry carries its request key instead, so the DO's own concurrentEntry policy applies to it, and interrupt_previous is reachable through the chat route); conformance: the interrupt half, proven on D-2 MR 1.5b at c055488ae2 (WE-2's lock slot, 2026-10-16, host load 11.4), cf-do, 3 of 3 each: chat.keyed-send-interrupts-live-run (host tier, an interrupt_previous DO set through the concurrent-entry-policy control): a keyed send over a held turn stops that turn (interrupted), its own turn completes, and its response carries only its own run's reply; client.keyed-send-interrupts-live-run (client tier, the real useHelixChat) renders turn 2's reply with none of turn 1's tail; the control chat.keyless-send-interrupt-host-refused (a keyless send on the same DO) is still the typed 409 state_already_running. js-chat-host has no concurrentEntry policy: declared gaps. A missing capability until then, so no red ledger | source, live-repro | verified | fixed (8 cells) |
| CP-14 | S1 | DO /abort cancels nothing: the execution-state abort controller's signal is never passed to the executor; /abort awaits the whole run, then marks it failed. | runtime-cloudflare/src/do/durable-object-agent-base.ts:2792-2819, 1768-1810 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/execution-state.ts:134-152 (as before WA MR2, origin/main cae966c0a7); runtime-cloudflare/src/do/durable-object-agent-base.ts:2792-2819 (pre-fix, origin/main 5bd30e20da) (handleAbort awaits this._executionState.abort(body.reason) first; the CAS active to failed and failStream run only after it returns) and runtime-cloudflare/src/do/execution-state.ts:134-149 (as before WA MR2, origin/main cae966c0a7) (abort() aborts the DO-owned _abortController, created at runtime-cloudflare/src/do/durable-object-agent-base.ts:1773 (pre-fix, origin/main 5bd30e20da) and never passed to executor.execute at runtime-cloudflare/src/do/durable-object-agent-base.ts:1796-1805 (pre-fix, origin/main 5bd30e20da), then awaits this._execution; it never calls this._handle.abort()); the JS executor owns its own controller (runtime-js/src/js-agent-executor.ts:2307 (pre-fix, origin/main 5bd30e20da)) and only handle.abort reaches it (runtime-js/src/execution/handle-factory.ts:225-226 (pre-fix, origin/main 5bd30e20da)); contrast: runtime-cloudflare/src/do/execution-state.ts:107-112 (as before WA MR2, origin/main cae966c0a7) (interrupt() goes through this._handle.interrupt, which is why a DO interrupt does stop the run); live: real-provider smoke after !279 (main f27a0c570e, claude-haiku-4-5 through llm-vercel, wrangler dev), the cf-do tool-loop scenario with maxSteps unset: a POST /do/:id/abort sent after 12 tool rounds was never answered; the cell ended with fetch failed after about 313 s when the worker was torn down (the smoke's post() has no timeout), with 346 model calls and about $6.17 spent in the cell window; the session stayed running and isExecuting throughout; live: same smoke: while the abort was pending the DO answered GET /do/:id/status in 3 ms (the smoke's wrangler.log), so requests were still dispatched: the hang is the handler awaiting the run, not a blocked input gate (there is no /abort log line because wrangler logs a request only when it completes); live: same smoke: on the next worker start over the same .wrangler state, the first log lines after Ready are model calls at message counts 1015, 1017, 1019 and on with no inbound request: wake recovery revived the aborted loop, which made 78 more model calls (up to message count 1169); runtime-cloudflare/src/do/durable-object-agent-base.ts:999-1010 (pre-fix, origin/main 5bd30e20da) (onStart runs recoverStrandedExecution wake; the execution-watchdog alarm also re-drives it) and runtime-cloudflare/src/do/durable-object-agent-base.ts:1061-1086 (pre-fix, origin/main 5bd30e20da) (it proceeds when the status is active and nothing is executing; the only stop signal it reads is checkInterruptFlag) and runtime-cloudflare/src/do/wake-recovery.ts:184-191 (as before WA MR2, origin/main cae966c0a7) (an interrupt flag is the only rung that blocks a resume): handleAbort writes no flag or other durable marker and its status write never ran, so the aborted run is revived as an ordinary stranded run; contrast: runtime-cloudflare/src/do/durable-object-agent-base.ts:2764 (pre-fix, origin/main 5bd30e20da) (handleInterrupt persists setInterruptFlag before it awaits anything, so an interrupted run that is evicted is demoted on wake, not resumed); inferred: fix scope: /abort must reach the executor (this._handle.abort) AND persist its outcome (the CAS active to failed) or a durable abort marker before it waits on the run; otherwise an abort that lands during a tool that ignores its signal, followed by an eviction, is still auto-resumed by wake recovery; related: DI-28 (no default step cap on the DO, so an aborted uncapped run is unbounded in time and spend; the same smoke spent about $17 in all across this run and its revival); CP-20 (DOAgentExecutor handle.abort posts the same /abort and would hang on it); MA-09 (the wake path fires no ServerHooks, so the revived run is invisible to host hooks); severity S1 (unchanged; now observed live): the stop control stops nothing, the request never answers, and the run outlives a worker restart; conformance: owner batch 21 (cf-do host), observed on main fb7c1e12f5 and on main + W1 MR2b a09ba78b9d, the same verdict. stop.running-host-stop-returns: while the run is inside a tool that ignores its abort, POST /do/:id/abort gives no answer within 2 s (.abort-answers-at-once); once released, the run ends failed with framework_cancelled, which passes (WA MR1 passes the drive's signal into the executor, so the abort's first half is fixed). stop.abort-evicted-not-revived: the same abort, then evict and a request wake: no answer within 2 s (.abort-answers-at-once), and the wake re-drives the run, which calls the model again and does not end failed framework_cancelled (.no-model-call-after-abort, .settles-cancelled); runtime-cloudflare/src/do/drive/stop.ts:12-18 (fix evidence, WA MR2 (!355), design note 2026-10-11 §3.2, user ruling D-1: every DO stop path, /interrupt, /abort, EndpointContext.stop and interrupt_previous, goes through handleStop at :112-183, core stopRun: a running run is recorded and signalled and answered 202 stopping at once; a stranded or suspended run is recorded and ended by the stop's K5 now and answered 200 with its terminal; no live run is 409 state_not_stoppable); runtime-cloudflare/src/do/drive/reconcile-wake.ts:15 (fix evidence, WA MR2 (!355) §3.5: on wake a live run with a durable stop record is settled from that record, the settle_stopped rung, commitStopTerminal, and is never re-driven, so an evicted aborted run is not revived) with runtime-cloudflare/src/do/drive/stop.ts:459 (endRunAtStopDeadline, the stop-deadline alarm subscriber registered at durable-object-agent-base.ts:1749 (pre-fix, origin/main 487090d3fb)); runtime-cloudflare/src/__tests__/integration/do-stop.integ.test.ts:2276 (proof of the no-re-drive half, real workerd: abort-evicted-not-revived, /abort while a model call is held answers 202 at once, the DO is evicted before the call returns, and a wake by request long before the stop deadline ends the run failed framework_cancelled with no model call after the abort and no second run) and :561 (stop-on-stranded-run: a running run nothing executes answers 200 with its terminal, committed then); conformance: fixed. stop.running-host-stop-returns .abort-answers-at-once and stop.abort-evicted-not-revived (outcome-only; the integ test above proves no re-drive) pass on cf-do, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12) | source, live-repro | verified | fixed (2 cells) |
| CP-15 | S2 | CF Workflows handles call instance.abort(), which the real WorkflowInstance does not have (it has terminate); likely throws in production after persisting failed. | runtime-cloudflare/src/executor.ts:1883-1885, 1994-2014 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/bindings.ts:107 (pre-fix, origin/main bd26dc9b57); runtime-cloudflare/src/executor.ts:1742-1840 (as before WA MR2, origin/main cae966c0a7) (fix evidence, !288, re-pointed for !334: every handle's abort() goes through abortSession, from the active handles at :4447-4449 and :4546-4548 and the reconnect handle at :4763-4767. abortSession terminates the owner instance through the owner probe, runtime-cloudflare/src/run-entry.ts:221-231 (binding.get(id), then terminate()), and then settles the session failed with a typed framework_cancelled detail. No instance.abort() call remains) and runtime-cloudflare/src/bindings.ts:95-110 (fix evidence, CF-D1 MR 1b (!287): the WorkflowInstance binding type no longer declares abort(), so a call to it no longer type-checks), proven by runtime-cloudflare/src/__tests__/commit-checkpoint-truth.wf-noiso.test.ts:1314 (an own-run abort terminates the owner, settles the run failed and fails the stream). That test drives an in-test engine in place of the binding, so it does not exercise the real WorkflowInstance; conformance: fixed (owner batch 18b, on !334's head fedd857efb, merged as 137b07d57a with the same code; the fix is !288 / !287, above). control.abort-terminates-owner holds the run's model call and hard-aborts it through its handle (abort): abort() resolves, the run's owner instance reads terminated in the binding's own status (observe-workflow-instances), and the session ends failed with framework_cancelled. Green 4 of 4 on cfw-workflows-d1. Red with one mutation (named here, applied to the working tree only and reverted): abortSession's terminateBestEffort throws in place of the owner probe's terminate(), as the pre-fix instance.abort() did (runtime-cloudflare/src/executor.ts:1900 (pre-fix, origin/main 487090d3fb)). The cell then fails .owner-terminated: the owner instance is not terminated (it read complete). No other lane has a workflow instance (a declared gap); related: since CP-16 an abort's owner can close two ways, both the product's: abortRun records the stop and signals the body, which aborts its in-flight model call and ends itself with its abort terminal (the instance closes complete), often before the executor's terminateBestEffort lands (runtime-cloudflare/src/executor.ts:2006-2008 (origin/main 459fa678ca)); otherwise the terminate wins (terminated, terminatedBy product). !444's CI (job 17291098628, 2026-10-19) read complete, so .owner-terminated now accepts either ending (never open, never reaped), and .persisted-cancelled (failed with framework_cancelled) still excludes a normal completion (FU-FLAKE-CFW-LANE-ABORT-OP-TERMINATE-RACE); related: the mutation above (terminateBestEffort throws) no longer discriminates: with CP-16 the body ends itself, so its owner closes complete either way, and an owner left running is unreachable. CP-15's defect is told apart by .abort-call-succeeds instead, re-proven by a mutation restoring the pre-fix rule (the handle's abort throws after the session is persisted failed, as the missing instance.abort() did): see the conformance line below; conformance: re-proven (owner slots s57 / s57b, 2026-10-19, on main 459fa678ca plus this record's change): control.abort-terminates-owner (both owner endings accepted) green 5 of 5 on cfw-workflows-d1, and main's previous cell 5 of 5 in the same build (the race did not show in 5; CI's ejects, jobs 17286942229 and 17291098628, are its evidence). Red 2 of 2 with the pre-fix rule restored (abortRun throws after the stop is recorded, as the missing instance.abort() did; working tree only, reverted), on .abort-call-succeeds alone: the abort answered { ok: false, error: instance.abort is not a function } | source | partly-inferred | fixed (1 cells) |
| CP-16 | S1 | Temporal, DBOS, CF Workflows never cancel an in-flight LLM call (no abortSignal passed); interrupt takes effect at step boundaries. Fixed on CF Workflows by a83323af26 (the LLM call gets the run's own-stop watch signal); still open on Temporal and DBOS. | runtime-temporal/src/activities.ts:2147-2163 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/workflows/shared.ts:1820-1834 (pre-fix, origin/main 5bd30e20da) (the only dispatcher.callLLM caller passes no signal; runtime-dbos/src/steps/call-llm.ts:44, 122 (pre-fix, origin/main 5bd30e20da) declare and forward it, so the plumbing exists but is never fed); runtime-cloudflare/src/steps.ts:1233, 581 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/steps.ts:2005-2025 (origin/main 2424cd0180) (CF Workflows today: executeAgentStep calls llmAdapter.generateStep with no abortSignal) and :727 (pre-fix, origin/main 487090d3fb) (the hook context's abortSignal is a fresh AbortController that is never aborted); conformance: stop.mid-llm-call on cfw-workflows-d1 (the plan 0c MR-2 lane flip: 5 full lane runs over origin/main d2e95ad181 product code, the same failed ids in every run) — .provider-call-aborted: the held model call is not aborted within 2000 ms of stop(); the same cell's .outcome-interrupted / .persisted-interrupted (the lost stop) are CP-97's; conformance: the dbos-postgres cell's red (stop.mid-llm-call .provider-call-aborted: DBOS does not cancel the in-flight call either) is unledgered since the CP-57 MR (owner, orchestrator ruling 2026-10-16): the cell is a declared gap there (it needs 'ordered-stop-settle', which the lane does not claim: its post-stop handle outcome is a race, FU-CONF-CP57-DBOS-STOP-SETTLE-ORDER); it is ledgered again from observed runs once the lane claims it; conformance: on CF Workflows, fixed by W1's CP-16 + CP-23 fix (branch reliability-cf-w1-cp16-cp23-cfw-stop at 2f0bf8dc89): stop.mid-llm-call passes .provider-call-aborted on cfw-workflows-d1 (slot W3, 2026-10-18); before it, it failed (its former cfw ledger row). The Temporal rows stay open (Temporal parked); related: fixed in code on CF Workflows by a83323af26 (W1, CP-16 + CP-23), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/steps.ts:2759-2771 (origin/main 56d564d073) (executeAgentStep starts the run's own-stop watch and passes its signal as the LLM call's abortSignal), runtime-cloudflare/src/own-stop.ts:44 (origin/main 56d564d073) (watchOwnStop polls the run's stop record and aborts the signal); still open on Temporal: runtime-temporal/src/activities.ts:3251 (origin/main 56d564d073) (fenceLlmAttempt aborts only on heartbeat loss or supersession) and :3696 (origin/main 56d564d073) (that fence's signal is the call's only abortSignal, so a recorded stop is not heard); still open on DBOS: runtime-dbos/src/workflows/shared.ts:2895-2909 (origin/main 56d564d073) (dispatcher.callLLM is called with no signal); conformance: usage.aborted-call-usage-reported (CP-242's cell: a call that streamed one delta, paused, then the run is stopped) never reaches its usage question on Temporal and DBOS, because the paused call never observes the stop's abort: no abort signal reaches the LLM call (the pins above), so it is structural, not a timing window. Owner slot s182 (2026-10-27, 3 of 3 identical, on temporal-memory, temporal-redis, temporal-postgres and dbos-postgres): .stop-reached-call fails (waitForAbort 10 s), so .run-interrupted, .hook-fired and .hook-saw-usage fail as its dependents; conformance: MA-43's transcript.stopped-turn-partial-persisted on temporal-memory, temporal-redis and temporal-postgres (owner slot s198, 2026-10-28, WE-2's #181 at 5565eee652, archived at orch-archive/batch-2026-10-29-1-11774a6828; landed in batch 3's squash f85872de1e; 3 of 3 identical, and again in owner slot s204 on the batch-3 tree) fails only .cut-at-stop: the stop does not cancel the held model call, so the reply it keeps (flagged, ruling D2) is the call's whole answer, not the text streamed before the stop | source | verified | open |
| CP-17 | S2 | DBOS interrupt calls cancelWorkflow immediately (graceful path unreachable), emits no run_interrupted, does not end the stream (hard path; the getHandle() flag path ends it cleanly with no interrupt marker, which reads as a normal finish), leaves paused sessions paused with pending calls cleared. | runtime-dbos/src/lifecycle/interrupt.ts:67-125 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/handles/reconnect-handle.ts:135-141 (pre-fix, origin/main 5bd30e20da) (getHandle().interrupt() → interruptImpl(cfg, sessionId, null, reason): the flag path, no cancel; runtime-dbos/src/lifecycle/interrupt.ts:125-133 (pre-fix, origin/main 5bd30e20da) is that flag-only branch) and runtime-dbos/src/workflows/shared.ts:1540-1552 (pre-fix, origin/main 5bd30e20da) (the loop-top flag check → finalizeRun 'interrupted' → dispatcher.endStream({ streamId }): the stream is ENDED cleanly, with no run_interrupted, no output and no terminal); runtime-dbos/src/steps/stream-lifecycle.ts:246 (pre-fix, origin/main 5bd30e20da) (endStream(streamId, finalOutput), no terminal); agent-server/src/agent-server.ts:1137 (pre-fix, origin/main 5bd30e20da) (AgentServer keeps the executor's ActiveHandle) and :918 (pre-fix, origin/main 5bd30e20da) (interruptAgent calls handle.interrupt() on the same replica), so a same-replica POST /interrupt on DBOS takes the hard cancelWorkflow path, runtime-dbos/src/lifecycle/interrupt.ts:66-124 (pre-fix, origin/main 5bd30e20da), while a cross-replica one takes the flag path; live: probe (batch-8 verification, pre-fix, origin/main 0ffde4e65b; RUNTIME_DBOS_INTEG=1, private Postgres DB on :5433, Redis db 13 on :6390): a stock ai client (AbstractChat + DefaultChatTransport) over handleChatStream + DBOSAgentExecutor; the chat stop applied via executor.getHandle().interrupt() during a 2 s tool. The session and handle result were 'interrupted'; the stream ended with no endStatus; the client received start, tool-input-available, tool-output-available, finish (no data-run-interrupted) and settled 'ready'. JS control (setInterruptFlag): the client received … data-run-interrupted, finish, and the persisted assistant message kept a data-run-interrupted part; live: hard path, cross-service (same probe run as CP-87, pre-fix, origin/main 0ffde4e65b): AgentServer.interruptAgent on the replica running a DBOS child → ActiveHandle hard cancel; the child session was 'interrupted' but its stream stayed 'active' with no end frame, and the runtime-js remote parent waited its full timeoutMs and failed "Remote sub-agent 'worker' timed out after 15000ms"; related: CP-87 (on the flag path the ended stream carries no endStatus, so AgentServer reports the interrupted child to a remote parent as completed; a separate cause: the fix is endStream's terminal argument, not the run_interrupted frame. A CP-17 fix on the flag path should pass terminal { status: 'interrupted' } in the same change); severity S2 (re-checked in batch 8): a stock ai client sees 'finish' and 'ready' on every runtime, and data-run-interrupted is an opt-in Helix part with no in-tree consumer; session, snapshot and result() stay 'interrupted' and resume works. The wrong-success consequence for remote parents is CP-87 (S1), not this record | source, probe, live-repro | verified | open |
| CP-18 | S2 | Stop on a paused session: DO returns 400; agent-server waits then 504s. DO /interrupt returns 400 "No agent to interrupt" whenever the in-memory session id is unset (after eviction). | runtime-cloudflare/src/do/durable-object-agent-base.ts:2737-2789 (pre-fix, origin/main 5bd30e20da); agent-server/src/agent-server.ts:886-964 (pre-fix, origin/main 5bd30e20da); conformance: stop.paused-host-stop (owner batch 19; host tier: the host's stop route on a session paused on an unanswered client call must record the stop or refuse it with a typed code): cf-do fails .stop-recorded-or-typed: the DO's /interrupt answers 400 {"error":"Agent already terminated","status":"paused"} (no code) and the session stays paused, identically on main f12eef982d and on WA MR1's head 9d7bfc8e27 (MR1 does not change this shape). js-chat-host is a declared gap: the agent-server interrupt route waits on the paused session until the harness's interrupt-observation deadline (10 s), which the harness reports as its own failure: this record's agent-server half, not yet a verdict; related: fix owner (orchestrator, 2026-10-11): WA MR2, the stop MR (CP-14, CP-18, CP-20, CP-21, CP-37, CP-59, CP-19), with stop.paused-host-stop|cf-do|host as its red-on-main proof cell. The agent-server half (a paused session's stop waits, then 504s) is B4-Node's (parked); the js-chat-host cell is a declared gap until a harness follow-up surfaces the deadline as the product's 504 answer; runtime-cloudflare/src/do/drive/stop.ts:12-18 (fix evidence, the DO half, WA MR2 (!355), design note 2026-10-11 §3.2, user ruling D-1: every DO stop path, /interrupt, /abort, EndpointContext.stop and interrupt_previous, goes through handleStop at :112-183, core stopRun: a running run is recorded and signalled and answered 202 stopping at once; a stranded or suspended run is recorded and ended by the stop's K5 now and answered 200 with its terminal; no live run is 409 state_not_stoppable); agent-server/src/agent-server.ts:929-979 (fix evidence, the agent-server half, WA MR2 (!355) Task 19: interruptAgent / abortAgent call the hosted executor's stop, core stopRun, and return its answer at once, never waiting for the run, so a paused session's stop no longer waits and 504s); conformance: fixed. stop.paused-host-stop passes on cf-do and on js-chat-host, whose stop route is AgentServer.interruptAgent, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12). The js-chat-host declared gap was lifted in WA MR2 G3 | source | verified | fixed (2 cells) |
| CP-19 | S1 | A durable interrupt flag set just before a run ends is not cleared by DO /start / /retry, JS execute, or CF Workflows execute → it interrupts the next run. On DBOS the standard workflow reads the flag without ever clearing it and execute does not clear it either, so the session's next execute is interrupted (resume does clear it, runtime-dbos/src/lifecycle/resume.ts:541 (pre-fix, origin/main 5bd30e20da)). | runtime-cloudflare/src/do/durable-object-agent-base.ts:2764 (pre-fix, origin/main 5bd30e20da) (/interrupt sets the durable flag); contrast: runtime-cloudflare/src/do/durable-object-agent-base.ts:2086 (pre-fix, origin/main 5bd30e20da) (/resume DOES clear the flag — /start and /retry do not); runtime-js/src/run-loop.ts:4318-4349 (pre-fix, origin/main 5bd30e20da) (pollDurableInterruptFlag: the flag is cleared only when a running loop observes it); runtime-cloudflare/src/workflow.ts:1831-1836 (pre-fix, origin/main a118f8e17e); runtime-dbos/src/workflows/shared.ts:1540-1551 (pre-fix, origin/main 5bd30e20da) (standard loop: checkInterruptFlag then finalize interrupted, no clear); runtime-dbos/src/lifecycle/interrupt.ts:67 (pre-fix, origin/main 5bd30e20da) (setInterruptFlag); contrast: runtime-dbos/src/lifecycle/resume.ts:541 (pre-fix, origin/main 5bd30e20da) (resume clears it — lifecycle/execute.ts never does); store-postgres/src/state/postgres-state-store.ts:2256-2267 (pre-fix, origin/main 5bd30e20da) (checkInterruptFlag is a plain SELECT, non-consuming); conformance: stop.mid-tool-ignores-abort on dbos-postgres — .continuation-completes: the new message's run ends interrupted; probe: the interrupt_flags row for the session was still present after the continuation ran; conformance: transcript.heal on dbos-postgres — .continuation-completes-on-interrupted: a new message on the stopped session ends interrupted without ever calling the model; conformance: stop.stale-does-not-stop-next-run|cf-do|host (owner batch 21) passes on main + W1 MR2b a09ba78b9d: EndpointContext.stop('interrupt') on a session paused on an unanswered call, then a new turn, which completes. The DO's remaining shape after R39 (a paused partial batch keeps the flag) is not reached through the chat route there, because the send abandons the pending calls first, completing the batch, and the run-start then clears the flag. On bare main fb7c1e12f5 the same cell is refused 409 state_already_running, which is CP-111's cause, fixed by W1 MR2b. The cell stays as an unledgered guard; this record's DBOS and JS halves are unchanged; runtime-dbos/src/steps/stop-request.ts:14-23 (fix evidence, the DBOS half, WA MR2 (!355): the interrupt flag is replaced by a run-scoped durable stop record, read by runId and never cleared, so a stop of an earlier run never reaches the next one) with runtime-dbos/src/workflows/shared.ts:2604-2611 (the loop's stop check reads only this run's record); conformance: fixed. transcript.heal passes on dbos-postgres at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos): the new message's run on the stopped session completes. stop.mid-tool-ignores-abort on dbos-postgres now fails only CP-57's .outcome-interrupted, so it no longer names this record; stop.stale-does-not-stop-next-run|cf-do|host passes at the same head | source, probe | partly-inferred | fixed (2 cells) |
| CP-20 | S2 | Client-side handle failures swallowed: DOExecutionHandleImpl.interrupt/abort only warn on non-2xx; DOAgentExecutor handle abort ignores the response. | runtime-cloudflare/src/do/do-clients.ts:1357-1387 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/do-executor.ts:747-753 (pre-fix, origin/main 5bd30e20da); related: CP-14 (the DO /abort that this handle posts to never answers while the run continues, now observed live; DOAgentExecutor handle.abort would hang on it); runtime-cloudflare/src/do/do-clients.ts:1506-1535 (origin/main 6ab56e113a): DOExecutionHandleImpl.abort and .interrupt still only log a warning on a non-2xx answer and resolve; runtime-cloudflare/src/do/do-executor.ts: deleted by WE MR B (!342, cae966c0a7), so the DOAgentExecutor half is gone; narrowed to the DOFrontendExecutor handle; conformance: handle.do-frontend-op-failure-typed (owner batch 29, host tier, cf-do; the new do-frontend-handle-op control calls getHandle() and then the operation in the Worker, over the chat handler's namespace, so fault-do-submit-answer can answer the handle's request): while a run is held, the DO's /interrupt answer and then its /abort answer are faulted once with a typed 503. On main 6ab56e113a plus owner batches 25 to 28 it fails .interrupt-rejects-typed and .abort-rejects-typed 4 of 4: both operations resolve. Every other id passes: each fault was served, the run is still live with one model call, an unfaulted interrupt() resolves and the run ends interrupted; conformance: fixed by !355 (WA MR2): the DO frontend handle's abort() and interrupt() go through its stop(), which POSTs the run-scoped stop with postStop (runtime-cloudflare/src/do/do-clients.ts:1557-1577) and throws the typed error a refused or failed stop answers (runtime-cloudflare/src/do/stop-client.ts:36-90), never a warning. handle.do-frontend-op-failure-typed passes on cf-do on main e96f88bc98 plus batch 29, 3 of 3 (.interrupt-rejects-typed, .abort-rejects-typed); it was red 4 of 4 before (the ledger above, removed) | source | verified | fixed (1 cells) |
| CP-21 | S2 | JS ExecuteOptions.abortSignal is ignored by execute / resume / retry (fresh controllers). | runtime-js/src/js-agent-executor.ts:2151, 3158, 3710 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-22 | S2 | Temporal abort has no non-cancellable scope; persistTerminalState can be cancelled, leaving status active. | runtime-temporal/src/workflow.ts:1707-1766, 319-331 (pre-fix, origin/main a118f8e17e); runtime-temporal/src/executor.ts:805-816 (pre-fix, origin/main a118f8e17e); inferred: (verification kept partly-inferred) the structural claim that Temporal abort has no non-cancellable scope is read from source, and the observed cells (discipline.abort-result-detail on temporal-memory / -redis / -postgres, test-server and docker; worker log persist_terminal_state_failed with persistError "Workflow cancelled") show the persist cancelled and the session never reaching a terminal status ("did not settle within 10000ms"), but no cell reads the persisted status value back as active. | source | partly-inferred | open |
| CP-23 | S1 | Parent interrupt does not reach blocking companions / children on Temporal, DBOS, CF Workflows (persistent waits), or the DO park / waitForResult loops. Fixed on CF Workflows by a83323af26 and 079b079413 (W1) and on the CF DO by 28a3a157e0 (its park and waitForResult loops check the parent's stop); still open on Temporal and DBOS. | runtime-temporal/src/workflow.ts:338-341 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/workflow.ts:3466-3528, 3597-3652 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/workflow.ts:3602-3683 (origin/main 62d3661b45) (CF Workflows today: the blocking companion__spawnAgent wait, the step named <agentType>-wait-persistent-<childName>-<callId>, waits only for the child's sub-agent-complete event, a timeout or a dead serving run; it never races the parent's interrupt event, which the regular sub-agent wait does at :2942-2971 (origin/main 62d3661b45), never reads the parent's interrupt flag and never interrupts the child) and :3754-3810 (origin/main 62d3661b45) (the companion__waitForResult wait, the same); runtime-cloudflare/src/do/persistent-companion-tools.ts:843-942, 1017-1073 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/persistent-companion-tools.ts:1008 (pre-fix, origin/main 5bd30e20da) pollChildToTerminal while(true) /status poll never checks parent abort nor interrupts child (live: stop during companion__spawnAgent took 31,367ms to unwind; companion__waitForResult up to 39s); conformance: history.companion-continue on temporal-memory/temporal-redis/temporal-postgres — after the parent's stop() returns ok while its blocking companion's call is held, .child-ref-interrupted observes the parent's SubSessionRef still 'running' and .child-session-interrupted observes the child session still 'active' (the interrupt never reached the child); .continue-completes, .same-child-continued, .child-made-post-interrupt-call and .post-interrupt-child-call-sees-turn-a fail only as preconditions of .child-session-interrupted; conformance: harness: !275 (https://gitlab.com/helix-ai/open-source/agents/-/merge_requests/275) reaps each Temporal cell's whole workflow tree at cell end; the pre-fix symptom was the companion-continue child orphaned by CP-23 writing to the torn-down parent stream — 20–65 unhandled 'Cannot write to stream … in failed/ended state' rejections per temporal conformance run (runtime-temporal activities.ts emit via onError), reduced to 0 with the reap (verified on test-server and Docker, 96/96 cells); conformance: history.companion-continue on cfw-workflows-d1 (the plan 0c MR-2 lane flip: 5 full lane runs over origin/main d2e95ad181 product code, the same failed ids in every run) — after the parent's stop() returns { ok: true } while its blocking companion's call is held, .child-ref-interrupted observes the parent's SubSessionRef still 'running' and .child-session-interrupted observes the child session persisted 'completed' (the interrupt never reached the child, which ran to completion); the four continuation ids fail only as preconditions of .child-session-interrupted. CP-68 (an interrupted companion left resumable) is not reached: the child was never interrupted; runtime-cloudflare/src/do/persistent-companion-tools.ts:1314-1323 (origin/main fb7c1e12f5; the DO's pollChildToTerminal still takes no parent signal) and :1149 (origin/main fb7c1e12f5; the companion__waitForResult park loop, the same); related: the DO half was re-found by WA MR2's survey (defect 4) and is fixed in its scope (design note §3.8); the Temporal and CF Workflows halves stay with their buckets; conformance: narrowed on cfw-workflows-d1 by CP-159's fix (!452, W1: CF Workflows' stopChildrenOfRun stops local children; MR pipeline 2965117354, job 17299104753, head 9735505441): history.companion-continue now fails ONLY .child-ref-interrupted (after the parent's stop() returns ok the child session reads interrupted and the continuation completes on the same child, but the parent's SubSessionRef for it still reads 'running'); observed in that one CI run; conformance: the CF Workflows half is fixed by CP-159's fix (!452, W1, head 8c0e6a8248: the stop cascade reaches the blocking companion child, waits, bounded, for its own terminal and writes the parent's ref from it): history.companion-continue passes on cfw-workflows-d1 3 of 3 on that head and fails only .child-ref-interrupted 1 of 1 before the ref write (9804ba40ab; W1's gate gQ159c, 2026-10-19 18:55-18:57); the cfw ledger row is dropped, and CP-23 stays open for Temporal; related: correction (W1, 2026-10-18; verified by the owner at origin/main 88b6e983ae): the DO-half evidence above (pollChildToTerminal) is stale. pollChildToTerminal is gone; the DO's blocking park is now parkOnChild, at runtime-cloudflare/src/do/persistent-companion-tools.ts:2383 (origin/main 88b6e983ae), which checks the parent run's abort signal, :1296 and :2163 (origin/main 88b6e983ae), before every read, posts the stop to the child for this consult and answers parent_stopped; the companion__waitForResult wait does the same. It landed in 28a3a157e0 (DO companion lifecycle MR C1, child stop truth; tagged CP-40, the DO half of CP-23). The ledger holds no DO row (the cell is executor tier; the DO is host tier): its DO witness is a cf-do host variant of history.companion-continue (owner to-do); the CF Workflows half is W1's fix (reliability-cf-w1-cp16-cp23-cfw-stop); related: fixed in code by a83323af26 (W1, CP-16 + CP-23 CF Workflows) with 079b079413 (W1, !452) on CF Workflows and by 28a3a157e0 (DO companion lifecycle MR C1) on the CF DO, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/workflow.ts:3247-3250 (origin/main 56d564d073) (the run's own stop is read and its interrupt event awaited), runtime-cloudflare/src/workflow.ts:3527 (origin/main 56d564d073), :3612 (origin/main 56d564d073) and :3762 (origin/main 56d564d073) (the blocking companion spawn, continue and waitForResult waits each race it), runtime-cloudflare/src/workflow.ts:7373-7426 (origin/main 56d564d073) (the wait helper's own-stop arm), runtime-cloudflare/src/do/persistent-companion-tools.ts:2561-2584 (origin/main 56d564d073) (the DO's parkOnChild checks the parent signal before every read, posts the stop to the child and answers parent_stopped), runtime-cloudflare/src/do/persistent-companion-tools.ts:2339-2340 (origin/main 56d564d073) (the DO's waitForResult loop does the same); still open on Temporal: runtime-temporal/src/workflow-dispatch.ts:805-822 (origin/main 56d564d073) (a blocking companion spawn awaits the child workflow's result with no race against the parent's stop); runtime-dbos/src/lifecycle/interrupt.ts:4-17 (origin/main a203fec499) (the stop flow: the stop is recorded on the run, then the control message is sent to and cancelWorkflow is called on ONLY the run's own workflow, controlOpsWake :120-149; no step stops the run's children) and runtime-dbos/src/steps/execute-companion-tool.ts:358 (origin/main a203fec499) (a child stop exists only as the companion terminate, stopPersistentChild); conformance: history.companion-continue on dbos-postgres (WE-2 slot w-cp69b-1031-022828, 3 of 3, identical sets, at CP-69's fix head b03061b924): after the parent's stop() returns { ok: true } while its blocking companion's turn B is held, .child-ref-interrupted reads the parent's ref 'running' (turn A's completedAt, outcomeGeneration 1) and .child-session-interrupted reads the child session 'completed', not 'interrupted'; the 4 continuation ids fail as their preconditions. The same 6 ids as the Temporal row. Before CP-69's fix (!601) the cell failed earlier, at turn A's settle (CP-69), which masked this half; conformance: re-confirmed at !601 de0c3029fd (its Greptile fixes; WE-2 slot w-601g-1031-050650, dbos-postgres, 3 of 3, identical to the w-cp69b sets) | source, live-repro | verified | open |
| CP-24 | S1 | Disputed: source shows the DO interrupt aborting the in-flight model call (js-agent-executor.ts:3296-3312 (as before WA MR2, origin/main cae966c0a7) → run-loop.ts:1767 (pre-fix, origin/main 7a390aaf8d) → step-iterator.ts:1103 → llm-vercel/src/vercel-adapter.ts:382 (origin/main a67fc68801)), but the consumer measured 37–62 s to settle a stop pressed mid-step (possibly pre-dating its switch to /interrupt, or a long tool call, or /interrupt awaiting cleanup + onComplete). Settled by stop.* cells. | consumer docs/operations/audio-stream-attach-latency.md | consumer-documented, live-repro | verified | withdrawn — Resolved by live repro 2026-09-23: DO in-flight provider abort fires ~1ms after /interrupt (loop unwinds ~+104ms, /interrupt responds ~+220ms incl. onComplete). The 37–62s consumer measurement is CP-23 (companion wait ignores abort). |
| CP-25 | S2 | The CF DO's chat stream silently ends mid-run when the browser's connection is lost and the client never reconnects. Narrowed by WE MR C1 and C2b: an EOF the reader did not cause is now a drop, never a clean end (C1), and an eviction now hands the client a typed evicted frame before its EOF rather than a clean writer.close() (C2b, SSEEviction); what stays open is the client's re-attach after a drop or an eviction (stock useChat never reconnects; the useHelixChat / useReanchorOnError re-anchors are what this record's client-tier cell must prove). The client's re-attach after a drop or an eviction is now implemented in useHelixChat and useReanchorOnError by 8022df47e7 and 5cf0bf0ae9 (one re-attach controller); stock useChat still never reconnects, by design. | core/src/stream/sse-connection-manager.ts:1-12, 172, 202-208; runtime-cloudflare/src/do/do-clients.ts:451-474, 484-487 (pre-fix, origin/main 5bd30e20da); live: 0 evict_slow_consumer/evict_stale events across ~45 connections in 5 runs; path real in code, not hit live; runtime-cloudflare/src/do/do-clients.ts:599-606 and :793-795 (fix evidence, WE MR C1, stream wire v2: an EOF the reader did not cause is a drop, and the reader throws a retryable StreamDropError instead of ending clean); conformance: narrowed by WE MR C1 (the owner's C1 window 2026-10-17, 3 runs each, host load at or below 14): stream.cut-sse-never-clean-end (cf-do host tier: the DO's /sse connections cut mid-reply, SSEConnectionManager.closeAll(), with no terminal frame) fails .no-clean-truncated-end 3 of 3 at main af9dfde232 (the reply ends clean with its text cut short) and passes 3 of 3 at C1's head c3792719af (an EOF is a drop, never a clean end). Still open, for WE MR C2: SSEConnectionManager still evicts with a clean close, and the client does not reconnect after the drop; core/src/stream/sse-connection-manager.ts:66-80 (origin/main 6a2a66817c) (fix evidence, WE MR C2b, !416: an eviction aborts the writer with an SSEEviction, and createSSEStream enqueues its evicted frame and closes, so the evicted client reads the frame, then EOF; no clean close any more); related: the title's 'evicts with a clean writer.close()' was stale after !416 (WD's retro review, 2026-10-19); narrowed by the owner, verified at main 6a2a66817c; related: client half fixed in code by 8022df47e7 (useReanchorOnError, the send-drop re-attach on a plain useChat) and 5cf0bf0ae9 (!444, one re-attach controller), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/client/reattach.ts:1-48 (origin/main 56d564d073) (the controller re-anchors a dropped send on the snapshot when the server holds the turn, and re-anchors or retries a dropped resume under a budget), ai-sdk/src/react/use-helix-chat.ts:580 (origin/main 56d564d073) (useHelixChat drives it), ai-sdk/src/react/use-reanchor-on-error.ts:103 (origin/main 56d564d073) (useReanchorOnError drives the same controller); server half: runtime-cloudflare/src/do/do-clients.ts:950 (origin/main 56d564d073) (a drop or an eviction past the re-open budget ends the DO reader typed, never clean); stock useChat with no hook still never reconnects, by design | source | verified | open |
| CP-26 | S1 | An SSE client of a Durable Object (the agent DO's /sse, and the standalone StreamDurableObject's) that stops reading (a backgrounded tab, a stalled mobile link) makes the DO buffer that client's frames without bound, until the isolate's 128 MB limit kills it and every other connection and the running agent with it. Both build their stream as new TransformStream(undefined, undefined, { highWaterMark: SSE_READABLE_HIGH_WATER_MARK }), and in workerd a TransformStream whose readable has ANY high-water mark above 0 applies no backpressure: every write resolves at once and the writer's desiredSize stays 1. SSEConnectionManager's evictions all depend on backpressure, so none can fire: its pending-write count never grows, the heartbeat's desiredSize <= 0 check never holds, and lastSuccessfulWrite keeps advancing, so the stale check never fires either. (Whether the mark counts chunks or bytes is moot in workerd; a byte-length strategy behaves the same.) | runtime-cloudflare/src/do/durable-object-agent-base.ts:3857-3859 (origin/main 880cdb8c14) (the agent DO's /sse: new TransformStream<Uint8Array>(undefined, undefined, { highWaterMark: SSE_READABLE_HIGH_WATER_MARK })) with store-cloudflare/src/stream-durable-object.ts:553-555 (origin/main 880cdb8c14) (StreamDurableObject's SSE, the same shape) and core/src/stream/sse-connection-manager.ts:27 (origin/main 880cdb8c14) (SSE_READABLE_HIGH_WATER_MARK = 256 * 1024, documented as bytes but a chunk count: no size function); core/src/stream/sse-connection-manager.ts:201-226 and :260-271 (origin/main 880cdb8c14) (the slow-consumer eviction counts only writes not yet resolved; a resolved write advances lastSuccessfulWrite) with :342-360 (origin/main 880cdb8c14) (the heartbeat evicts on desiredSize <= 0 or on a stale lastSuccessfulWrite); live: workerd probe (W1, branch reliability-cf-w1-cp26-sse-byte-hwm, peer-reported 2026-10-17; W1's probe file run by the owner 3 of 3 the same day in runtime-cloudflare's .cf pool, @cloudflare/vitest-pool-workers 0.9.14, miniflare 5.20260925.0-alpha, workerd 1.20260925.1, compatibilityDate 2024-01-01, nodejs_compat; each row writes 8 KiB frames, never reads, and waits two macrotasks): a TransformStream with no arguments, readable highWaterMark 0, or a transformer with default strategies resolves 0 of 64 writes, desiredSize -63 (backpressure). With readable highWaterMark 2, a ByteLengthQueuingStrategy of 256 KiB, or either with an identity transformer, it resolves 64 of 64, desiredSize 1; the DOs' own shape (readable highWaterMark 262144) resolves 2048 of 2048 (16 MiB held, unread), desiredSize 1. Contrast: a plain ReadableStream with the 256 KiB byte strategy reports desiredSize -262144 after 64 frames. Not measured: isolate memory, and whether workerd's HTTP layer reads a Response body whose client stopped reading (the DO cannot tell either case apart); related: the earlier statement of this finding (a chunk-count high-water mark makes eviction almost never fire, because workerd keeps writes pending until read) is replaced by the mechanism above: workerd keeps writes pending only on an identity stream with no readable high-water mark (the no-arguments row), and the DOs set one. Fix owner W1 (orchestrator-approved, branch reliability-cf-w1-cp26-sse-byte-hwm): in both DOs, a no-buffer stream whose writes resolve when the client reads them, and a per-connection unread-byte budget (256 KiB) in SSEConnectionManager that depends on no stream strategy; severity S1 (re-rated from S3 by the owner, orchestrator concurring, 2026-10-17): one unread client grows a DO without bound until the isolate is killed, taking every other connection and the running agent down with it; core/src/stream/sse-connection-manager.ts, runtime-cloudflare/src/do/durable-object-agent-base.ts and store-cloudflare/src/stream-durable-object.ts (fix evidence, !397, 968e106dcf) (both DOs' SSE streams apply backpressure, and SSEConnectionManager evicts a connection by its unread bytes); conformance: fixed for the agent DO's /sse. stream.unread-client-evicted-host (owner batch 40; the chat route's upstream body from the DO held unread) passes on cf-do 3 of 3 at owner batch 7be28c1611 (main 22cb4e513a + the owner's batches), 2026-10-18, and fails .evicted 3 of 3 with !397's product diff reverse-applied (owner c2, working tree only, reverted). The StreamDurableObject site is unit-evidenced only (!397's .cf test): no conformance lane can hold an unread StreamDurableObject reader (FU-CONF-STREAM-DO-UNREAD-READER-HOLDER), and the cell observes the released slot only indirectly (the held body ends before the turn's last sequence) | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-27 | S2 | No byte on attach: the first byte is the 15 s heartbeat; clients cannot distinguish "connected, waiting" from "connecting". | ai-sdk/src/response/sse-builder.ts:66-94 (pre-fix, origin/main a1c6655f56) | source, live-repro | verified | open |
| CP-28 | S2 | DOStreamManagerClient / DOFrontendExecutor turn failures into null / [] / EOF / sequence 0; getHandle returns null on any failure; a new session's JSON-null snapshot is treated as a schema failure and the preflight silently falls back to startFromSequence: 0. | runtime-cloudflare/src/do/do-clients.ts:418-421, 614-777, 871-895, 954-989, 1025-1048 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/durable-object-agent-base.ts:3684-3699 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/do-clients.ts:470-518, 769-796 (origin/main 6ab56e113a): DOStreamManagerClient.fetchFromDO still answers null for any non-2xx, schema failure or throw, and getStreamInfo reads that null as no stream; conformance: chat.reconnect-stream-info-read-fails (owner batch 29, host tier, cf-do; fault-do-submit-answer gained a search match, so only the stream manager's query-less /snapshot read is faulted): during a held run a reconnect's liveness read fails once. It passes 4 of 4 on main 6ab56e113a plus owner batches 25 to 28: the handler's next liveness read finds the stream live, so one failed read is not seen at a reconnect. A green guard; the null reads stay open by source; runtime-cloudflare/src/do/do-clients.ts:141-207 (fix evidence, WE MR C1: getFromDO, the DO clients' one read rule: null on a 404 only; a failed request or a non-2xx answer is a typed DOPeerError, a body that fails its schema a transport_error); conformance: narrowed by WE MR C1 (the owner's C1 window 2026-10-17, 3 runs each, host load at or below 14): stream.info-read-failure-rejects-host (cf-do host tier: the DO's /snapshot read faulted under DOStreamManagerClient.getStreamInfo) fails .read-rejects 3 of 3 at main af9dfde232 (the failure read as absence) and passes 3 of 3 at C1's head c3792719af (it rejects typed). Still open: getHandle returns null on any failure, and a new session's JSON-null snapshot is treated as a schema failure (C1 does not change either); conformance: re-armed for GL #178 (owner, !552 66b2f948b1): the stream manager's and the state store's head reads now all send /snapshot?messages=none, so stream.info-read-failure-rejects-host arms search '?messages=none' (the only DO request of its read) and chat.reconnect-stream-info-read-fails adds skip: 2 (handleChatStream awaits loadState, then getCurrentRun, then getStreamInfo). With getStreamInfo mutated back to answer null on a failed read (owner slot s167b, 2026-10-26, tree 62a1095838 plus 66b2f948b1), stream.info-read-failure-rejects-host fails exactly .read-rejects: it is the read-fault half's exact witness. chat.reconnect-stream-info-read-fails passes under that mutation with skip: 2 and with skip: 0: since RM-73 the reconnect's path 2b reads getStreamInfo again (handle-chat-stream.ts:641 at 62a1095838), so one failed read no longer reaches the misread there and the cell does not discriminate the reader (FU-CONF-RECONNECT-STREAM-READ-DISCRIMINATE) | source, live-repro | verified | open |
| CP-29 | S2 | Resume options must be passed identically to prepareHelixChatRequest and prepareHelixReconnectRequest or the stream silently dies; values are frozen at transport construction, so later POSTs carry a stale X-Resume-From-Sequence. | ai-sdk/src/client/prepare-helix-chat-request.ts; consumer lib/helix-stream-transport.ts:21-34 | source, consumer-documented | verified | open |
| CP-51 | S3 | handleChatStream GET with no messages on an active session whose current run is not running falls through to replayTerminalStream. | ai-sdk/src/handler/handle-chat-stream.ts:486-499, 901-918 (pre-fix, origin/main 5bd30e20da) | peer-reported | unverified | open |
| CP-30 | S1 | DBOS retry ignores all options (_opts): checkpointId and message dropped; no truncation; startMessageCount hard-coded 0. | runtime-dbos/src/lifecycle/retry.ts:52, 128, 135-136 (pre-fix, origin/main 5bd30e20da); conformance: lifecycle.retry-message on dbos-postgres — .retry-message-reaches-model: the model never sees the retry message; conformance: lifecycle.foreign-checkpoint on dbos-postgres — .retry-foreign-rejected-typed: retry() with session B's checkpointId starts a run on session A instead of throwing (the checkpointId is never examined; the scenario scripts no model reply for a foreign attempt, so that run ends failed on the exhausted script), and .retry-foreign-no-state-change: session A's version moves from 2 to 4; the branch and resume legs of the same cell pass; conformance: fixed by !288 — retryImpl honours RetryOptions: lifecycle.retry-message (.retry-message-reaches-model) and lifecycle.foreign-checkpoint (.retry-foreign-rejected-typed, .retry-foreign-no-state-change) pass on dbos-postgres; conformance: stream.retry-resync on dbos-postgres — .retry-completes, .one-resync-per-retry, .resync-after-run-started-before-content, .resync-carries-truncate-checkpoint and .loser-injects-no-resync fail on origin/main 5bd30e20da and pass with !288; runtime-dbos/src/lifecycle/retry.ts:80-98 (fix evidence, !288: retryImpl reads RetryOptions; resolveCheckpoint rejects a foreign or unknown checkpointId typed before any write, the message is parsed into the retry's rows, and core planRetry picks the target, so the rewind rides the retry's run-start commit) | source | verified | fixed (3 cells) |
| CP-31 | S1 | No Temporal handle forwards resume options: executor.resume() now honours mode (the LLM-history completeness fix), but the reconnect handle calls it without its options and the execute() / factory handles throw, so the documented handle.resume({ mode }) is honoured by no Temporal handle. | runtime-temporal/src/executor.ts:1174-1178 (pre-fix, origin/main a118f8e17e) (resume(agent, sessionId, options)); :1220 (pre-fix, origin/main a118f8e17e) (from_checkpoint resolves the target and rewinds after the claim) and :1517-1520 (pre-fix, origin/main a118f8e17e) (with_message appends the parsed message before the workflow starts) — the LLM-history completeness fix fixed the executor half; :2034 (fromSequence is still read from the same options, readFromSequence in resume()); runtime-temporal/src/executor.ts:1078-1084 (pre-fix, origin/main a118f8e17e) (the getHandle() reconnect handle's resume(_options) calls this.resume(agent, sessionId) without its options, dropping mode, message, checkpointId and fromSequence); executor.ts:1705-1711 (createActiveHandle's resume() throws) and runtime-temporal/src/handle.ts:221-225 (pre-fix, origin/main 5bd30e20da) (the execute() handle's and the factory handle's resume() throw) — so the documented handle.resume({ mode: 'with_message' | 'from_checkpoint' }) (runtime-temporal/README.md, Resume Interrupted or Suspended Execution) is honoured by no Temporal handle; not yet reachable by conformance (the with_message half, through executor.resume): lifecycle.resume-with-message on Temporal never gets a genuinely interrupted session (CP-56 makes the first run end completed), so resume() takes the completed-session short-circuit (CP-60) before any mode handling. lifecycle.from-checkpoint on temporal-* is behind the same CP-56 setup precondition. The from_checkpoint half is proven by lifecycle.from-checkpoint-after-suspend on temporal-memory/temporal-redis/temporal-postgres (crashed in the Temporal lane cascade pre-fix; passes post-fix) and by the history.count-after-truncate rewind leg; conformance: history.count-after-truncate on temporal-memory/temporal-redis/temporal-postgres — the rewind route's own outer settle (history.count-outcome-settles) times out after resume mode from_checkpoint on a session suspended on an unanswered client-tool call; .rewind-completes and .rewind-checkpoint-count-matches-rows fail as preconditioned cascades of that same settle timeout; conformance: history.count-after-truncate on temporal-memory/temporal-redis/temporal-postgres — post-fix (with !279) the rewind leg settles and completes; the cells pass once the rewind-route floor is corrected, so the fragment above is removed. Pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) they failed .count-outcome-settles, .rewind-completes, .rewind-final-checkpointed and .rewind-checkpoint-count-matches-rows; conformance: lifecycle.from-checkpoint-after-suspend on temporal-memory/temporal-redis/temporal-postgres — proves the from_checkpoint half post-fix: the persisted history and the model input are rewound to the target checkpoint | source, probe | verified | open |
| CP-32 | S2 | DBOS resume from_checkpoint silently becomes continue. | runtime-dbos/src/lifecycle/resume.ts:447 (pre-fix, origin/main 0ffde4e65b) (the only mode branch — from_checkpoint falls through to the continue path); runtime-dbos/src/lifecycle/resume.ts:512-531 (pre-fix, origin/main 0ffde4e65b) (the workflow restarts from the current state; resumedFromCheckpointId is the session's latest state.checkpointId — opts.checkpointId is never read); conformance: lifecycle.from-checkpoint-after-suspend on dbos-postgres — proves the fix: pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) all eight of its ids failed (masked by RM-33's missing turn-1 checkpoint, so unledgered); post-fix (with !279) the cell passes: the persisted history and the model input are rewound to the target checkpoint; not yet reachable (conformance cell) — lifecycle.from-checkpoint on dbos-postgres is still behind CP-57 (its .setup-interrupted precondition fails, so the rewind leg never runs); runtime-dbos/src/__tests__/integration/history-b2.integ.test.ts covers the DBOS from_checkpoint rewind directly | source | verified | fixed (1 cells) |
| CP-33 | S2 | Temporal and CF Workflows branch ignore checkpointId / messageIndex and copy all messages. | runtime-temporal/src/activities.ts:6756-6769 (pre-fix, origin/main 0ffde4e65b); runtime-cloudflare/src/steps.ts:676-704 (pre-fix, origin/main 0ffde4e65b); conformance: history.branch-from-earlier on temporal-memory/temporal-redis/temporal-postgres — .branch-excludes-later-turns: a branch from turn 1's checkpoint still carries turn 2's messages (TURN-TWO/REPLY-TWO leak into the branch); .branch-includes-checkpoint-history passes (the branch DOES include the checkpoint prefix — it copies everything, not too little); conformance: history.branch-at-message-index on temporal-memory/temporal-redis/temporal-postgres — .branch-excludes-after-index: same over-copy when branching by messageIndex instead of checkpointId; .branch-includes-prefix passes for the same reason; conformance: history.branch-from-earlier and history.branch-at-message-index on temporal-memory/temporal-redis/temporal-postgres — prove the Temporal fix: pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) .branch-excludes-later-turns / .branch-excludes-after-index failed in all six cells; post-fix (with !279) all six pass; conformance: CFW / DO half proven by runtime-cloudflare tests (runtime-cloudflare/src/__tests__/integration/steps.integ.test.ts:1563 'branch with checkpointId copies only the checkpoint history', :1594 foreign-checkpoint rejection; history-b2.wf-noiso.test.ts on real workerd); the cfw-workflows-d1 cells (next note) now prove the CF Workflows half on the conformance lane; conformance: history.branch-from-earlier and history.branch-at-message-index on cfw-workflows-d1 — prove the CF Workflows half across turns: .branch-excludes-later-turns / .branch-excludes-after-index and the prefix controls .branch-includes-checkpoint-history / .branch-includes-prefix pass in all 5 full lane runs of the plan 0c MR-2 lane flip (origin/main d2e95ad181 product code) (two turns, then a branch from turn 1's checkpoint or message index). Re-open if a cfw cell fails | source, probe | verified | fixed (8 cells) |
| CP-34 | S2 | DO /resume accepts checkpointId / modifyState / appendMessages in its schema and returns 200 while ignoring them. | runtime-cloudflare/src/do/durable-object-agent-base.ts:351-356, 2194-2197 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/drive/request-schemas.ts:68-82 (pre-fix, origin/main 487090d3fb) (fix evidence, WA MR1 (!340), fb7c1e12f5: DoResumeOptionsSchema is a .strict() union of continue / with_message / from_checkpoint, so modifyState, appendMessages and a retry mode are refused) with :228-234 (pre-fix, origin/main 487090d3fb) (the /resume parse answers validation_error) and runtime-cloudflare/src/do/drive/run-drive.ts:570 (pre-fix, origin/main 487090d3fb) (the options reach executor.resume verbatim); conformance: fixed. resume.do-native-options-strict (owner batch 28, host tier, cf-do; do-request): after two turns, POST /resume bodies carrying modifyState, appendMessages and mode 'retry' are each answered 4xx validation_error with nothing written and no model call, and a from_checkpoint resume to turn 1's last step checkpoint answers 2xx and completes with turn 1's messages followed by the rewound reply, turn 2's message gone; green on main 8dc3a5c967 plus owner batches 24 to 28, in the owner lock window of 2026-10-13 with every lane run in full. Red with the pre-fix rule restored (the three variants' .strict() dropped, so unknown keys are accepted and ignored; working tree only, reverted): .modify-state-refused and .append-messages-refused fail, the bodies reaching the resume itself (answered state_not_resumable for the completed session) | source | unverified | fixed (1 cells) |
| CP-35 | S1 | CF Workflows resume() (and the handle.resume() path) truncates messages and cleans up the stream before a status CAS that passes no expectedVersion (TOCTOU). A losing concurrent resume can delete the winner's messages, and two concurrent resumes of a paused / interrupted / suspended session both pass the CAS and each start a workflow instance. The LLM-history completeness fix (!279) closes this on executor.resume(): every mode now claims with a version-pinned CAS before any staging cleanup, truncate or stream cleanup. handle.resume() is unchanged. Separately, retry() calls cleanupToStep without startSequence. (A resume against a live run is CP-83.) Since fixed: handle.resume() is executor.resume() (CF-D1 MR 1a, 5bd30e20da), and every rewind truncation rides the run-start commit (CF-D1 MR 1b, !287). | runtime-cloudflare/src/executor.ts:676 (pre-fix, origin/main 0ffde4e65b) (cleanupOrphanedStaging), :694-696 (pre-fix, origin/main 0ffde4e65b) (truncateMessages to latestCheckpoint.messageCount) and :705-712 (pre-fix, origin/main 0ffde4e65b) (cleanupToStep) all run BEFORE :756-760 (pre-fix, origin/main 0ffde4e65b) (compareAndSetStatus(["paused","interrupted","active"] → "active") with no expectedVersion); the comment at :748-755 (pre-fix, origin/main 0ffde4e65b) claims a version pin that is never passed; runtime-cloudflare/src/executor.ts:1708-1773 (pre-fix, origin/main a118f8e17e) (private resumeWorkflow — the handle.resume() path — unchanged by the fix: cleanupOrphanedStaging :1723, truncateMessages :1740-1741 and cleanupToStep :1749-1755 run before the CAS at :1762-1766, which passes no expectedVersion); runtime-cloudflare/src/executor.ts:1374 (pre-fix, origin/main a118f8e17e) (retry() calls cleanupToStep(streamId, checkpoint.stepCount) with no startSequence; its CAS at :1286 runs before the truncation, so retry() has no TOCTOU); runtime-cloudflare/src/executor.ts:1001-1026 (origin/main a118f8e17e) (fix evidence, !279: executor.resume() in continue / with_message claims with compareAndSetStatus pinned to the version it read at :983, BEFORE the staging cleanup, truncate and cleanupToStep at :1028-1071, which roll the claim back if they throw), proven by runtime-cloudflare/src/__tests__/executor.test.ts:2031 (origin/main a118f8e17e) ("lost claim: a concurrent version bump between entry and the claim is rejected, changing nothing else") and :1602 (origin/main a118f8e17e) ("a continue whose claim lands after a concurrent from_checkpoint loses (pinned claim), having truncated and cleaned up nothing"); from_checkpoint claims pinned at runtime-cloudflare/src/executor.ts:846-851 (origin/main a118f8e17e) before rewindToCheckpoint at :891-895 (origin/main a118f8e17e), proven by runtime-cloudflare/src/__tests__/executor.test.ts:1325 (origin/main a118f8e17e) ("two concurrent from_checkpoint resumes: exactly one wins"). handle.resume() (runtime-cloudflare/src/executor.ts:1708 (pre-fix, origin/main a118f8e17e)) has neither guard, and no conformance cell covers the fix (the cfw-workflows-d1 lane runs, but no cell there races two resumes), so the record stays open; live: [pre-fix, origin/main 3560d6c72a] probe (batch-5 verification, unit level: CloudflareAgentExecutor + a real InMemoryStateStore + a mock workflow binding): two concurrent resume() calls on a paused session both fulfilled and created __resume__1 and __resume__2. The same probe's live-run case is CP-83 evidence. With !279, executor.resume() rejects the second concurrent caller; handle.resume() is unchanged; related: the original title's "cleanupToStep without startSequence" now applies only to retry() (runtime-cloudflare/src/executor.ts:1374 (pre-fix, origin/main a118f8e17e)); resume() passes resumingRun?.startSequence at :1064-1071 and :1749-1755 (pre-fix, origin/main a118f8e17e). runtime-temporal/src/executor.ts:1286-1297 (pre-fix, origin/main a118f8e17e) carries the same version-pin comment, and there the LLM-history completeness fix made it true: the CAS pins expectedVersion (:1350-1359 (pre-fix, origin/main a118f8e17e)) and runs before any truncation; CP-83 (resume() against a live run, including this runtime's handle.resume()); related: CP-100 (since !288, bd26dc9b57, the resume claim and its rollback described in the !279 fix evidence above no longer exist: the entry plans and claims inside the workflow's run-start commit, and an entry that fails before that commit records an existing session failed instead of restoring its previous status); live: peer-reported; runtime-cloudflare/src/executor.ts:2239 (origin/main 5bd30e20da) and :2374 (origin/main 5bd30e20da) (fix evidence, CF-D1 MR 1a: the handle.resume() path no longer has its own claim; it runs executor.resume(), which claims pinned BEFORE any staging cleanup, truncation or stream cleanup, closing the remainder this record named in the private resumeWorkflow, now deleted), proven by runtime-cloudflare/src/__tests__/executor.test.ts:1164 (origin/main 5bd30e20da) ("claims before it truncates (CP-35): a lost claim leaves the log untouched", red on main: the old CAS accepted the session another writer had made active and truncateMessages ran); runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:790 (fix evidence, CF-D1 MR 1b (!287): the retry() remainder is closed on the snapshot base, whose run-start commit truncates only the retried run's own output under a version-pinned claim; this real-engine test pins it: after turn 1 completes and turn 2 fails, retry() leaves turn 1's stream chunks leading the stream); conformance: the concurrent-resume half holds. lifecycle.concurrent-resume (owner batch 28, executor tier, every lane): a run is interrupted mid-tool (a gated tool held, the stop, the release, so no terminal-step stop is involved), then two resume({ mode: 'continue' }) calls are started at once with one model response scripted. Green 3 of 3 on cfw-workflows-d1 (and on js-memory) on main 8dc3a5c967 plus batches 24 to 27: exactly one completes, the other is refused typed, one model call is made, and the persisted history ends with exactly one copy of the winner's answer. A handle's resume() is executor.resume() since CP-67, so racing two executor resumes covers the handle path. No red mutation was run: the pre-fix truncation and stream cleanup before the claim no longer exist, so there is no single rule to restore; the entry plans and claims in its run-start commit; related: fixed in code by CF-D1 MR 1a (5bd30e20da deleted the private resumeWorkflow) and MR 1b (!287), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-cloudflare/src/executor.ts holds no truncateMessages, cleanupToStep or compareAndSetStatus call; the handle paths call this.resume (:5274-5277 and :5515-5518 (origin/main fb9150cc16)); retry() (:4014-4085 (origin/main fb9150cc16)) only reads before the instance exists, and the truncation runs inside the run-start commit (runtime-cloudflare/src/run-entry.ts:1101-1109 (origin/main fb9150cc16)). Open until its cfw-workflows-d1 cell; related: fixed in code by 5bd30e20da (CF-D1 MR 1a) with CF-D1 MR 1b (!287), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/executor.ts:5367 and :5605 (origin/main 56d564d073) (every handle resume runs executor.resume(), and executor.ts holds no truncateMessages, cleanupToStep or compareAndSetStatus call), runtime-cloudflare/src/run-entry.ts:1109 (origin/main 56d564d073) (a rewind's truncation rides the guarded run-start commit as truncateTo), runtime-cloudflare/src/steps.ts:1909-1914 (origin/main 56d564d073) (the only entry stream cleanup runs after the run start won, scoped to the previous run's startSequence); conformance: lifecycle.retry-after-turn-stream (W1, slot w1-cp35-slot-20261104-003331, tree c0f813131e = c6813b7286 on main b9cbd26272): turn 1 runs three steps (its reply at step 3), turn 2 commits one step and then fails, and retry({ message }) targets that step commit (step count 1); turn 1's stored chunks, found by their run's runId stamp from a baseline read once turn 1 settled, must survive the retry unchanged and come before the retried run's run_started. Passes 3 of 3 on cfw-workflows-d1 (js-memory, the contrast lane, 3 of 3). Red with the record's own pre-fix retry() rule restored (the retry entry calls cleanupToStep(streamId, retryTarget.stepCount) with no startSequence, as runtime-cloudflare/src/executor.ts:1374 (pre-fix, origin/main a118f8e17e)): exactly .turn1-chunks-survive fails, 9 of turn 1's 15 chunks deleted | source, probe, peer-reported | verified | fixed (2 cells) |
| CP-36 | S3 | CF Workflows retry flattens a multimodal retry message with JSON.stringify. | runtime-cloudflare/src/executor.ts:1382-1385 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:790 (fix evidence, CF-D1 MR 1b (!287): on the snapshot base retry() plans through core planRetry, which replays the failed run's own trigger row verbatim, so no JSON.stringify path remains; this real-engine test pins it: a text + image turn that fails non-retryably is retried with its content unchanged, exactly once); conformance: fixed. lifecycle.retry-multimodal-message (owner batch 28, executor tier, every lane): the first turn's model call fails, then retry() with a message of a text part and an image file part completes; the retried model call's input and the persisted history each hold exactly one user message carrying both content parts, never their JSON as text. Green 3 of 3 on cfw-workflows-d1 (and on js-memory) on main 8dc3a5c967 plus batches 24 to 27. Red with the pre-!288 rule restored on CF Workflows (the retry message flattened with JSON.stringify before parseAgentInput, working tree only, reverted): .parts-reach-model and .parts-persisted fail, the model and the history holding the parts' JSON as text | source | verified | fixed (1 cells) |
| CP-37 | S2 | JS retry ignores abortSignal and usageStore (the DO passes one for nothing). | runtime-js/src/js-agent-executor.ts:3709-3779 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-38 | S2 | JS / Temporal / CF retry accept a checkpointId belonging to another session. | runtime-js/src/js-agent-executor.ts:2711-2720 (pre-fix, origin/main 0ffde4e65b); runtime-temporal/src/executor.ts:1613-1622 (pre-fix, origin/main 0ffde4e65b); runtime-cloudflare/src/executor.ts:905-914 (pre-fix, origin/main 0ffde4e65b); runtime-cloudflare/src/executor.ts:714-746 (pre-fix, origin/main 0ffde4e65b) (CFW from_checkpoint writes restoredState.sessionId from the checkpoint — a foreign checkpoint targets another session's row); conformance: lifecycle.foreign-checkpoint on js-memory/js-postgres/js-redis — .retry-foreign-rejected-typed and .retry-foreign-no-state-change: retry() silently accepts session B's checkpoint and mutates session A's state; .branch-foreign-rejected-typed: branch() silently succeeds using session B's checkpoint (no rejection at all); .resume-foreign-rejected-typed: resume() DOES reject (no state change), but throws an untyped Error, not the typed state_checkpoint_foreign code; conformance: lifecycle.foreign-checkpoint on temporal-memory, pre-fix (observed by the conformance review of !279: origin/main 7581ad127e product code with this suite, where the cell no longer crashes in the lane cascade) — .retry-foreign-rejected-typed, .resume-foreign-rejected-typed and .branch-foreign-rejected-typed failed with the matching .*-foreign-no-state-change ids, plus an .outcome-settles timeout (CP-31); the earlier unfixed runs never got this far because the Temporal lane crashed before the cell (backend poisoned by the upstream history.companion-continue crash). Post-fix the temporal-* provenBy cells pass; conformance: lifecycle.foreign-checkpoint on js-memory/js-redis/js-postgres/temporal-memory/temporal-redis/temporal-postgres — proves the fix: pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) the js-* cells failed the four ledgered ids and the temporal-* cells crashed in the Temporal lane cascade; post-fix (with !279) all six pass. On dbos-postgres the branch and resume legs now pass; its retry legs fail on CP-30 (DBOS retry() ignores checkpointId), not on this finding; conformance: CFW / DO half proven by runtime-cloudflare tests (runtime-cloudflare/src/__tests__/executor-retry-genesis.test.ts:606 retry and :660 branch foreign-checkpoint rejection; runtime-cloudflare/src/__tests__/history-b2.wf-noiso.test.ts:649 and :731 on real workerd); CFW cross-turn proof is limited to multi-step single instances until CP-67 is fixed; re-open if a Phase B cfw / cf-do cell fails | source, probe | verified | fixed (6 cells) |
| CP-39 | S1 | DO: non-blocking companions never notify the parent (persistentAgents: undefined in the DO rewrite disables the notifier). | runtime-cloudflare/src/do/durable-object-agent-base.ts:1557 (pre-fix, origin/main 5bd30e20da); runtime-js/src/run-loop.ts:481, 888 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/durable-object-agent-base.ts:2303 (origin/main 6ab56e113a) (still present: the companion rewrite sets persistentAgents: undefined) against runtime-js/src/run-loop.ts:598 (origin/main 6ab56e113a) (hasPersistentAgents gates the notifier); and nothing writes a child's terminal status back to the parent's sub-session ref: a child DO has no callback to its parent, and refs are updated only by the parent's own polls (runtime-cloudflare/src/do/persistent-companion-tools.ts:1219 (origin/main 6ab56e113a)); runtime-cloudflare/src/do/persistent-companion-tools.ts:946-947 (origin/main 6ab56e113a) (companion__sendMessage to a child whose live status is completed answers { delivered: false, reason: 'child_exited' }), while a same-name companion__spawnAgent reads the parent's stale running ref and answers 'Agent … is already running. Use sendMessage to communicate with it.': a finished non-blocking companion can be reached by neither, so the documented re-consult (docs/guide/sub-agents.md, the critic loop) fails on the DO; conformance: companion.nonblocking-reconsult-host (owner batch 29, host tier): turn 1 spawns a non-blocking companion and completes, the child finishes, and turn 2 re-consults it by a same-name spawn. On cf-do (main 6ab56e113a plus owner batches 25 to 28) it fails .parent-ref-terminal (the parent's ref still reads running after the child's session reads completed) and .reconsult-continues (the spawn answers 'already running' and the child makes no second model call), 3 of 3; on js-chat-host, where runtime-js's notifier runs, it passes 3 of 3; runtime-cloudflare/src/do/durable-object-agent-base.ts:786 (pre-fix, origin/main 2e5afe376b) (fix evidence, WA's MR A: the DO hands its executor companionNotifier: consultOutcomeNotifier(this.stateStore), defined at runtime-cloudflare/src/do/companion-settle-notice.ts:628 (pre-fix, origin/main 2e5afe376b)) with runtime-js/src/run-loop.ts:671-672 (pre-fix, origin/main 2e5afe376b) (hasPersistentAgents is also true with a companionNotifier, so the notifier runs on the DO though its rewrite leaves persistentAgents undefined) and :1166 (pre-fix, origin/main 2e5afe376b) (the notifier reads each ref's outcome through it); conformance: fixed by WA's MR A (!389; the owner's MR A window 2026-10-17, 3 runs, at 9fd0cc5daf plus the owner's cell commit): companion.nonblocking-reconsult-host passes every id 3 of 3 on cf-do (red on main: .parent-ref-terminal and .reconsult-continues; !389's pipeline 2958578834 reported it an unexpected pass) | source | verified | fixed (1 cells) |
| CP-40 | S1 | DO: a companion paused on a client tool / approval. (b) The blocking spawn's park and an untimed waitForResult never end, because a committed suspension reads as live: the DO /status reports only a terminal committed outcome, so every consult read succeeds and the failure window never trips. (a) The call is unreachable: the child's best-effort ownership write to the root assumes a shared store, which a DO child does not have, so the write is skipped with a warning. (The interrupted half and the parent-abort half are fixed; see the evidence.) | runtime-js/src/run-loop.ts:1842-1901 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/persistent-companion-tools.ts:1063-1111, 867-872 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/persistent-companion-tools.ts:1008-1075 (pre-fix, origin/main 5bd30e20da) (poll loop); runtime-cloudflare/src/do/persistent-companion-tools.ts:1093-1112 (pre-fix, origin/main 5bd30e20da) (mapRemoteStatusToRefStatus: interrupted maps to interrupted, paused falls through to the caller's running fallback); runtime-cloudflare/src/do/persistent-companion-tools.ts:106-132 (pre-fix, origin/main 5bd30e20da) (failure window; recordSuccess at 1032); core/src/tools/companion/helpers.ts:9 (TERMINAL_STATUSES excludes interrupted — intentionally: interrupted is resumable; the defect is the unbounded wait, not the set); runtime-cloudflare/src/do/persistent-companion-tools.ts:1314-1323 (origin/main fb7c1e12f5; still true after WA MR1: pollChildToTerminal's endless while-true loop takes no signal and never reads the parent's stop) and :717 (origin/main fb7c1e12f5; the blocking spawn's waitForChildTerminal awaits it); runtime-cloudflare/src/do/persistent-companion-tools.ts:1149 (origin/main fb7c1e12f5; companion__waitForResult's park loop, the same: no parent-stop check); related: re-found by WA MR2's survey as its defect 4, a stop never reaching a blocking companion; filed here rather than as a new record because this record's title already names the missing parent-abort check (owner, 2026-10-11). WA MR2's fix direction (design note §3.8): the DO's poll and park loops take the parent run's signal and check it on every poll, then cascade the stop to the child; runtime-cloudflare/src/do/persistent-companion-tools.ts:1326-1392 (origin/main 6ab56e113a) (still present: pollChildToTerminal is a while (true) loop with no deadline and no abort check; a paused child maps to running, :1411-1430 (origin/main 6ab56e113a)); conformance: companion.blocking-wait-bounded-host (owner batch 29, host tier, cf-do; static pair conf-bw-host-*): the parent spawns a BLOCKING companion whose first step calls a client-executed tool, so the child's session reads paused. On main 6ab56e113a plus owner batches 25 to 28 it fails 3 of 3: .parent-not-hung (the parent's start answer is still open 20 s after the child paused), .stop-succeeds (a stop of the parent does not answer within its 20 s bound) and .parent-stop-ends-wait (the answer stays open after the stop); the cell leaves the parent run executing, so the lane rebuilds its backend after it. js-chat-host is a declared gap (no live execution flag); conformance: on main e96f88bc98 (!355, WA MR2: the DO stop deadline) plus batch 29, .stop-succeeds and .parent-stop-ends-wait pass 3 of 3: the stop answers and ends the parent's wait. .parent-not-hung still fails 3 of 3: unstopped, the parent's start answer stays open 20 s after its child paused, which is this record's defect; related: re-pinned at origin/main 0889c0ee05 (owner, 2026-10-20, from WA's local CP-40 investigation note, §1, the cites below verified by the owner). pollChildToTerminal and mapRemoteStatusToRefStatus, which the earlier lines cite, no longer exist: MR A / B replaced them with parkOnChild and the verified read. The interrupted half and the parent-abort half are fixed (C1 / D7; WA MR2 / C1; the conformance line on .parent-stop-ends-wait above); runtime-cloudflare/src/do/child-consult-types.ts:174 and :185-188 (origin/main 0889c0ee05) (the DO /status emits only committedOutcome 'terminal', from consultStatusFields), so runtime-cloudflare/src/do/persistent-companion-tools.ts:725 (origin/main 0889c0ee05) (readChildWord) returns live for a child whose run is suspended_client_tool; runtime-cloudflare/src/do/persistent-companion-tools.ts:2396 and :2441 (origin/main 0889c0ee05) (L1, the blocking spawn's parkOnChild: a while (true) loop, each pass a successful live read and a sleep, ended only by the parent's stop or an all-failed window) and :2151 (origin/main 0889c0ee05) (L2, companion__waitForResult's while (true) loop, bounded only by a caller timeout); runtime-cloudflare/src/do/companion-settle-notice.ts:30 (origin/main 0889c0ee05) (RINGING_STATUSES excludes a suspension, so the child's settle never rings the parent); runtime-js/src/run-loop.ts:2768-2779 (origin/main 0889c0ee05) ((a) the child's best-effort root ownership write fences on input.rootRunId, else getCurrentRun(rootSessionId) on the CHILD's store, which on a DO child holds only the child's session, so it warns 'no live root run to fence the ownership write on' and returns; rootRunId is never threaded on the CF DO, per WA §1 (a)); related: the fix is MR C's C2 (WA, after CP-161; orchestrator ruling 2026-10-20, decision D2 (c)): the CF DO refuses at spawn a companion whose tools, transitively, include a client or approval tool (not_spawned / child_not_supported). Witness: companion.client-tool-companion-refused-host (cf-do); this record's ledger moves there when C2 lands, since companion.blocking-wait-bounded-host cannot flip as written under the refusal. Residual, not closed by C2: the other_consult park (L6, FU-DO-COMPANION-PARK-OTHER-CONSULT-LOOPS, MR C's C3), so CP-40 is never closed as every unbounded park fixed; conformance: companion.client-tool-companion-refused-host (cf-do host; owner slot s89, 2026-10-20, the owner next batch a41d4dc482 = main 0889c0ee05 + owner records, no product change) fails all three ids 3 of 3, each on the defect, not a harness error: .spawn-refused (the parent's history holds no result row for the blocking spawn call, which was not refused but parked), .no-child (the parent's ref for the companion reads running) and .parent-completes (60 s after the spawn the parent is still running, parked on the paused companion; read at the bound, not a timeout). Ledgered below; when MR C's C2 lands the cell turns green and this record's ledger moves to it (orchestrator ruling 8); related: Fixer: WA (orchestrator, 2026-10-24), after DI-75; the fix direction is MR C2 (refuse at spawn, ruling D2 (c)); runtime-cloudflare/src/do/persistent-companion-tools.ts:1424-1436 (fix: CP-40 C2 T3, WA; as on branch reliability-cf-wa-cp40-c2 3b864439b0) (refuseChild: a companion whose tree can call a client or approval tool is refused at the parent's spawn, before anything is written, with core companionNotSupportedDetail, packages/core/src/tools/companion/companion-not-supported.ts); runtime-cloudflare/src/do/drive/materialize.ts:140-159 (fix: CP-40 C2 T2, WA; as on branch reliability-cf-wa-cp40-c2 3b864439b0) (step 2c, the child's own backstop: a consult entry whose resolved tree offends is refused framework_companion_not_supported before the rewrite); conformance: companion.client-tool-companion-refused-host passes all three ids on cf-do (host), 3 of 3, at WA's C2 T4 head 24b615b21f plus the owner's C2 cells (owner slot s146, 2026-10-25): the blocking spawn is refused not_spawned / child_not_supported and the parent completes. companion.client-tool-slip-refused-host passes 3 of 3 there, and its red mutation (step 2c disabled) fails exactly .spawn-refused and .no-child-run 3 of 3. Both halves of this record are closed on the DO by the refusal; its ledger is removed; conformance: companion.blocking-wait-bounded-host is withdrawn with C2 (orchestrator rulings Q4 and Q7, 2026-10-24): its premise, a DO companion pausing on a client tool, is unreachable once the DO refuses such a companion (at the parent's spawn, and the child's materialize backstop), and no product seam skips both checks. Its replacement for the backstop is companion.client-tool-slip-refused-host (cf-do, host): a clientToolSlip factory fault gives only the child DO's resolution a client tool, and the child's start is refused typed (framework_companion_not_supported) and answered not_spawned / child_not_supported; runtime-cloudflare/src/do/persistent-companion-tools.ts:785-849 (fix: CP-40 C3, WA; as on branch reliability-cf-wa-cp40-c3-other-consult 109b656d82) (readChildWord names the consult's own run; when the child's current run is another run, the consult run's own end answers); runtime-cloudflare/src/do/persistent-companion-tools.ts:852 (fix: CP-40 C3, WA; as on branch reliability-cf-wa-cp40-c3-other-consult 109b656d82) (consultRunWord: a superseded consult run reads interrupted, reason superseded_by_entry); runtime-cloudflare/src/do/persistent-companion-tools.ts:3062 (fix: CP-40 C3, WA; as on branch reliability-cf-wa-cp40-c3-other-consult 109b656d82) (fenceUnconfirmed: an unconfirmed consult over a committed consult-less run settles through the parent fence); runtime-cloudflare/src/do/child-consult-types.ts:244 (fix: CP-40 C3, WA; as on branch reliability-cf-wa-cp40-c3-other-consult 109b656d82) (consultRunFields, the run-scoped answer of /status); core/src/run/committed-run-outcome.ts:75 (fix: CP-40 C3, WA; as on branch reliability-cf-wa-cp40-c3-other-consult 109b656d82) (readRunOutcome: one run of a session, current or not); conformance: companion.direct-entry-stops-consult-host and companion.direct-entry-after-completed-consult-host pass on cf-do (host) at 109b656d82 on origin/main aa4f781a6c (WA slot-c3b, x1; x3 at e30ca2fe12 in slot-c3). They prove a direct entry after the consult's run ended: stopped, then the typed interrupted word; completed, then that run's real output. With b2's read removed at 109b656d82 (M-b2, proven in the rebuilt dist) both fail .parent-answered by its bounded timeout, setup held; at the red commit 4f62ffc83d on origin/main fcb82e788e they fail the same way; related: unit-tested only, no cell: the superseded_by_entry branch, b3's parent fence, and a current run serving another consult (do-companion-verify). FU-CONF-CP40-C3-OWED-CELLS names the owed cells; related: the stranded reset no longer parks a confirmed consult (WA trace at 109b656d82, reading only, after the owner's review): at a reset the current run is live-suspended, so its continuation inherits the parent by default, or it ended with its own K5, so its end is the consult's real word. No writer of active over an ended current run was found on the DO: run-start commits active in its own transaction, and every state save is version-pinned, so a state read before a K5 cannot land after it. That is a failed search, not a proof. The design note's b1 (parent: inherit on the continuation) is dropped; runtime-cloudflare/src/do/persistent-companion-tools.ts:2748 (fix: CP-40 C3 part 2, WA; as on branch reliability-cf-wa-cp40-c3-wait-fence fe704ebd04) (the untimed wait's loop decides an unconfirmed consult itself) and :3138 (as on branch reliability-cf-wa-cp40-c3-wait-fence fe704ebd04) (decideUnconfirmedRead) and :2979 (as on branch reliability-cf-wa-cp40-c3-wait-fence fe704ebd04) (the park reads its ref's current consult the same way when recordStarted answered stale). A fenced consult answers failed with reason outcome_unknown, since the child's fence reads only its current run; conformance: fixed. companion.unconfirmed-wait-settles-host (cf-do, host; a faulted /start, a direct entry held live, the wait reading it, then the entry committing during the wait) passes every id 3 of 3 at a81bda6850 and 3 of 3 at fe704ebd04 (WA, read by the owner from its result files). Its red is the fix turned off (drv-c4b, proven in the rebuilt dist; the scenario is not at the pre-fix head): m-wire and m-entry hold .spawn-unconfirmed, .direct-entry-live, .wait-reading and .direct-entry-committed and fail .wait-answered by its bounded timeout. js-chat-host is a declared gap. The direct-entry cells now wait for the child's settle (9ecd780df5; a teardown flake seen once at a81bda6850, then 12 of 12 clean); related: unit-tested only, no cell: the park's stale path, a run that really ran with its answer lost (FU-CONF-CP40-C3-OWED-CELLS), and a child with a session and no run, which is bounded, answered unreachable and writes nothing (reachability unproven; the FU says what would make it a finding). The park's failed answer carries the unknown-outcome text with no typed reason (orchestrator ruling) | source, live-repro, peer-reported | verified | fixed (5 cells) |
| CP-41 | S1 | Temporal / CF Workflows: sendMessage to a running companion sets parent_message, which those runtimes treat as an interrupt: the running child turn ends interrupted instead of taking the message in, and the call returns delivered: true. DBOS delivered the message to the child's inbox first but cut the in-flight turn short the same way (interrupted). On Temporal a later waitForResult on that child can hang (CP-73). On CF Workflows, the CP-68 fixes narrowed the consequence: the interrupted child is reported 'interrupted' (resumable), not terminal, and a follow-up sendMessage resumes it with the stranded message in its history. Fixed on CF Workflows and Temporal by e96f88bc98 (WA MR2): delivery to a child with a live run is an append only, with no flag and no stop, and the child loads it at its next step; on DBOS the same change leaves the inbox delivery as the only effect, and the child takes the message as its next turn. | core/src/orchestration/companion-tool-dispatch.ts:710-840 (pre-fix, origin/main 5bd30e20da) (handleSendMessage appends the message, sets the 'parent_message' interrupt flag on a child with a live loop at :817-819, and returns delivered: true); runtime-temporal/src/workflow.ts:1014-1022 (pre-fix, origin/main a118f8e17e) (the per-step durable interrupt check calls finishInterrupted for ANY flag reason) and :937-946 (pre-fix, origin/main a118f8e17e) (on resume, a set flag of any reason makes the deferred-finishWith path end the run interrupted); runtime-cloudflare/src/workflow.ts:1831-1840 (pre-fix, origin/main a118f8e17e) (the per-iteration check hands ANY flag to handleInterruptFlag, :1671-1725, which ends the run interrupted and notifies the parent 'terminated'); a flag seen inside the step instead surfaces as state.aborted (runtime-cloudflare/src/steps.ts:530-531 (pre-fix, origin/main 5bd30e20da), :951-959 (pre-fix, origin/main a118f8e17e)) and is handled at runtime-cloudflare/src/workflow.ts:1920-1980 (pre-fix, origin/main a118f8e17e) (discard staged, failAgentStream, finalizeRun 'failed', result 'cancelled') — neither branches on the reason; runtime-cloudflare/src/steps.ts:530-531 (pre-fix, origin/main 5bd30e20da) (loadAgentState turns ANY interrupt flag into aborted: true / abortReason with no reason branch; 'parent_message' appears in runtime-cloudflare/src/steps.ts only in comments, :5343 and :5561 (pre-fix, origin/main a118f8e17e)); runtime-temporal/src/activities.ts:6061 (pre-fix, origin/main 5bd30e20da) (consumeInterruptFlag returns { interrupted, reason } and clears the flag, with no reason branch); runtime-temporal/src/workflow.ts:1014-1022 (pre-fix, origin/main a118f8e17e) (the step-boundary check ends the run via finishInterrupted for any reason, parent_message included); contrast: runtime-js/src/run-loop.ts:675-720 (pre-fix, origin/main 5bd30e20da) (on reason 'parent_message' the loop reloads messages, clears the flag and continues); runtime-dbos/src/steps/execute-companion-tool.ts:1156-1163 (pre-fix, origin/main 5bd30e20da) (not a correct contrast, and not claimed in this title: DBOS sends the message to the child's inbox FIRST, then sets parent_message for a running child); runtime-dbos/src/workflows/shared.ts:1539-1551 (pre-fix, origin/main 5bd30e20da) (the running turn exits 'interrupted' on any flag), runtime-dbos/src/workflows/persistent-workflow.ts:549-553 (pre-fix, origin/main 5bd30e20da), :617-621 (pre-fix, origin/main 5bd30e20da) (the recv loop clears the flag and runs the inbox message as the next turn). So DBOS does deliver the message, but on every parent_message it cuts the in-flight turn short as 'interrupted' — a narrower partial defect of the same kind, unlike JS, which continues the turn. CP-42 covers DBOS after a continuation, a different situation; related: CF Workflows consequence narrowed by the CP-68 fixes in !279: CFW record steps map an interrupted child's 'terminated' completion event to a resumable 'interrupted' ref and tool error, runtime-cloudflare/src/workflow.ts:3654-3687 (origin/main a118f8e17e) via runtime-cloudflare/src/companion-ref-status.ts resolvePersistentChildRefStatus; the sendMessage resume uses a fresh instance id; core handleSendMessage decides resume from the child session's own status, so a self-interrupted CFW child with a stale 'running' ref is resumed and no flag is stranded, core/src/orchestration/companion-tool-dispatch.ts:790-803 (origin/main 5bd30e20da); core handleWaitForResult returns 'interrupted' for an interrupted companion, debounced by ORPHAN_CONFIRMATIONS, :1007 and :1092-1103 (origin/main 5bd30e20da). Tests: runtime-cloudflare/src/__tests__/companion-ref-status.test.ts:57, runtime-cloudflare/src/__tests__/cfw-persistent-companion.test.ts:533 (origin/main 5bd30e20da), runtime-cloudflare/src/__tests__/cfw-continue.wf-noiso.test.ts:476 (origin/main 5bd30e20da) (real workerd), core/src/orchestration/__tests__/companion-tool-dispatch.test.ts:1931 (origin/main 5bd30e20da), :1951 (origin/main 5bd30e20da), :1963 (origin/main 5bd30e20da); inferred: the CP-68 fixes did not remove a CF Workflows waitForResult hang, and none is claimed fixed here. CFW does not route waitForResult through core: its own handleWaitForResult (runtime-cloudflare/src/steps.ts:5747-5860 (pre-fix, origin/main a118f8e17e)) returns a blocking descriptor for any ref that is not completed / failed / terminated, and the workflow waits for the child's completion event bounded by subAgentEventTimeout (runtime-cloudflare/src/workflow.ts:3597-3652 (pre-fix, origin/main a118f8e17e)). Before and after the fix, the parent_message interrupt notifies the parent (workflow.ts:1703, :1941 (origin/main a118f8e17e)), so the wait's outcome depends on that event being delivered to the later wait; that was not checked. The fix changed the recorded outcome from terminal 'terminated' to resumable 'interrupted'; related: likely fix direction (from a code review, peer-reported): on reason 'parent_message', clear the flag, reload messages and continue the loop, as JS does. That remains open on Temporal and CF Workflows; related: CP-73 (the Temporal waitForResult consequence); CP-68 (fixed: interrupted companions stay resumable on JS and CF Workflows); related: fixed in code by e96f88bc98 (WA MR2: stop as one signal; its message does not name CP-41), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): core/src/orchestration/companion-tool-dispatch.ts:1968-1974 (origin/main fb9150cc16) (the doorbell is not a stop: delivery to a child with a live run is the append and nothing else, no flag and no stop record); parent_message and setInterruptFlag no longer occur anywhere in packages/*/src; Temporal reloads the history at every step (runtime-temporal/src/activities.ts:3298 (origin/main fb9150cc16)). Open until its cell; related: fixed in code by e96f88bc98 (WA MR2, stop as one signal), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/companion-tool-dispatch.ts:1977-1991 (origin/main 56d564d073) (sendMessage to a child with a live run is an append commit on the child and nothing else: no interrupt flag, no stop record), runtime-cloudflare/src/steps.ts:10342-10363 (origin/main 56d564d073) (CF Workflows routes sendMessage through that core dispatcher), runtime-dbos/src/steps/execute-companion-tool.ts:22-24 (origin/main 56d564d073) (DBOS delivers to the running child's inbox and nothing else); parent_message and setInterruptFlag no longer occur in any package's product source; conformance: companion.send-mid-run-appends (W1, executor tier, every lane; js-* the contrast): turn 1 spawns a non-blocking companion held mid-run at a tool gate; with its run live, turn 2 sends MID-RUN; the gate releases. The child takes the message in (its next step on js-*, temporal-* and cfw-workflows-d1; its next turn on dbos-postgres, the inbox) and its turn completes, the run live at the send ends completed, no child run is interrupted, and the send answers delivered: true. At W1's head ed0099c4ab (on the Redis lane TTL fix, slot w1-cp41-slot-20261104-030511, 2026-11-04) every id passes, 3 of 3 on all eight lanes. Red with the pre-fix rule restored: a parent_message stop on the live child in core's dispatch (m-cp41-core) fails .child-takes-message, .child-not-interrupted and .delivery-matches on cfw-workflows-d1, temporal-* and js-* (DBOS has its own sendMessage, unchanged); the same stop after DBOS's inbox delivery (m-cp41-dbos) fails .child-not-interrupted on dbos-postgres only | source, peer-reported | verified | fixed (5 cells) |
| CP-42 | S1 | DBOS: after a continuation, sendMessage targets the dead initial workflow while interrupting the live continuation child; returns delivered: true. | runtime-dbos/src/steps/execute-companion-tool.ts:923, 1105, 1161-1163 (pre-fix, origin/main 5bd30e20da) | source | partly-inferred | open |
| CP-43 | S1 | JS / DO: sendMessage to an interrupted companion resumes it with no history (state built from loadState, which has no messages). | runtime-js/src/run-loop.ts:4604 (pre-fix, origin/main 0ffde4e65b) (resumeChildAgent), :4650 (pre-fix, origin/main 0ffde4e65b) (the child state is loaded with stateStore.loadState — a SessionState, which carries no messages: core/src/types/session.ts:88), runtime-js/src/run-loop.ts:4675-4678 (pre-fix, origin/main 0ffde4e65b) (it is cast straight to AgentState with no getAllMessages hydration) and :4685-4691 (pre-fix, origin/main 0ffde4e65b) (runSubAgentLoop runs the child on it); call site :5217 (pre-fix, origin/main 0ffde4e65b); contrast: runtime-js/src/run-loop.ts:5144-5147 (origin/main a118f8e17e) (the COMPLETED-child continuation path reads the preserved history before running the child; the LLM-history completeness fix replaced its getAllMessages read, run-loop.ts:4937 (pre-fix, origin/main 0ffde4e65b), with loadLLMHistory); inferred: messages is undefined (not []) on the loaded state, so the resumed loop may throw on a state.messages access (e.g. core/src/orchestration/step-iterator.ts:652 (origin/main a118f8e17e)) rather than run with an empty history; the resume catch (run-loop.ts:4698-4719 (pre-fix, origin/main 0ffde4e65b)) would then mark the ref failed. Either way the child never sees its history — the exact failure shape is not traced end to end; related: CP-68 — on JS this path is currently unreachable: a parent-interrupted companion's session and parent ref are both recorded 'failed' (CP-68), so handleSendMessage throws "not active (status: failed)" (core/src/orchestration/companion-tool-dispatch.ts:706-727 (pre-fix, origin/main 0ffde4e65b)) and never returns resumeMetadata; reaching resumeChildAgent (companion-tool-dispatch.ts:770, :791-799 (pre-fix, origin/main 0ffde4e65b)) also needs a ref status of 'interrupted', which no runtime-js updateSubSessionRef call writes — so a CP-68 fix must record 'interrupted' for CP-43 to become observable; conformance: history.companion-continue on js-memory/js-redis/js-postgres — proves the fix: pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) .post-interrupt-child-call-sees-turn-a failed (with CP-68's ids, which made the resume path unreachable); post-fix (with !279) all three cells pass; not yet reachable (conformance cell) — history.companion-continue on temporal-* is behind CP-23 (the parent's stop never reaches the child) and on dbos-postgres behind CP-69 (turn A never settles), so neither reaches the post-interrupt continuation; conformance: CFW / DO half proven by runtime-cloudflare tests (runtime-cloudflare/src/__tests__/cfw-continue.wf-noiso.test.ts:605, real workerd: an interrupted blocking companion resumed by sendMessage sees its history; the DO base runs runtime-js's JSAgentExecutor, covered by runtime-js/src/__tests__/run-loop-persistent.test.ts:1364 (origin/main 5bd30e20da)); re-open if a Phase B cfw / cf-do cell fails | source | partly-inferred | fixed (3 cells) |
| CP-44 | S2 | JS: a companion suspended on a client tool is recorded completed with no output. | runtime-js/src/run-loop.ts:4669-4688, 5018-5031 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-45 | S2 | Remote child interrupted / paused with any (stale) output is treated as success. Fixed by 459fa678ca (!442) on every runtime: a stopped or paused remote child is never the call's success. | core/src/orchestration/remote-sub-agent-dispatch.ts:682-693; runtime-js/src/execution/remote-subagent-executor.ts:668-681, 787-797; runtime-dbos/src/steps/execute-remote-subagent.ts:558-566 (pre-fix, origin/main 5bd30e20da); conformance: unreachable on the suite's producers (owner slot s65, 2026-10-19, batch 6186ec9f34 = main cd2d6e3169 + the owner batch): a cfw cell seeded a remote child session on its producer (the CF DO running runtime-js) with a completed turn, then a turn paused on its approval; the paused child read output null, because the producer clears a session's output when a new turn starts (and its stop terminal clears it too), so neither an interrupted nor a paused child ever carries an earlier turn's output there, and a pre-fix consumer behaves like the fixed one. The cell (subagent.remote-child-stale-output-not-success) was withdrawn; a discriminating witness needs a scripted producer that answers paused / interrupted with a stale output (orchestrator's call); core/src/orchestration/remote-sub-agent-dispatch.ts (fix evidence, !442, 459fa678ca) (a stopped remote child is never the call's success, whatever stale output it carries; a paused child past the retries fails typed, RemoteChildPausedError), with its unit tests: core remote-sub-agent-dispatch.test.ts and remote-child-paused-error.test.ts, runtime-cloudflare cfw-cp45-paused-remote-child.test.ts and steps.test.ts, runtime-js remote-subagent-executor.test.ts, runtime-dbos execute-remote-subagent.test.ts; related: fixed in code by 459fa678ca (!442) on every runtime, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/remote-sub-agent-dispatch.ts:582-588 (origin/main 56d564d073) (an interrupted child is a typed failure whatever output it carries) and :883-891 (origin/main 56d564d073) (a paused child after a drop is never a success, its output never read), runtime-js/src/execution/remote-subagent-executor.ts:476-485 (origin/main 56d564d073) (runtime-js throws a typed RemoteAgentFailedError for an interrupted child), runtime-cloudflare/src/steps.ts:6556-6566 (origin/main 56d564d073) (CF Workflows the same) and :6596 (origin/main 56d564d073) (a paused child past the retries is RemoteChildPausedError), runtime-dbos/src/steps/execute-remote-subagent.ts:461-468 (origin/main 56d564d073) (DBOS fails an interrupted child whatever its output) and :738 (origin/main 56d564d073) (a paused child's drop is RemoteChildPausedError) | source | verified | open |
| CP-46 | S1 | Temporal: a startChild failure for a companion spawn or a sendMessage resume is only logged, and the tool reports success. Spawn: the parent's ref stays running with no child behind it. Resume: the ref is left at its prior status (since !325 the resume's running write is in the child's run-start commit), but the model is told delivered: true for a message that is dropped. Continue: no longer silent (!288): the reopen is rolled back, the call reports delivered: false, and the ref is restored to completed. | runtime-temporal/src/workflow-dispatch.ts:778-784, 809-815, 910-916 (pre-fix, origin/main 5bd30e20da); runtime-temporal/src/workflow-dispatch.ts:754-766 (spawn: the tool result is built from the activity's success before wf.startChild), :777-778 (wf.startChild) and :880-886 (a start failure is only logged as persistent_agent.start_failed; the message keeps the activity's success result, the spawn echo). The ref was written running before the start by core/src/orchestration/companion-tool-dispatch.ts:579-593 (pre-fix, origin/main 487090d3fb) (first spawn, addSubSessionRefs) or :569-575 (pre-fix, origin/main 487090d3fb) (a re-spawn after a failed or terminated child, which also deleted that child's session); core's compensation to failed (:649-671 (pre-fix, origin/main 487090d3fb)) runs only when spawnPersistentChild throws, and on Temporal that callback is a no-op echo (runtime-temporal/src/activities.ts:8907-8910 (!325)), so nothing restores it: the ref stays running with no child workflow; runtime-temporal/src/workflow-dispatch.ts:890-928 (!325) (resume, a sendMessage to an interrupted child: wf.startChild of a fresh detached child with the deferred message as its input and companionResume: true) and :929-935 (!325) (a start failure is only logged as persistent_agent.resume_failed; the result stays core's delivered: true, core/src/orchestration/companion-tool-dispatch.ts:938-946 (pre-fix, origin/main 487090d3fb) (!325)). Since !325 (CP-103 A1) handleSendMessage writes no ref for a deferred resume (:919 (pre-fix, origin/main 487090d3fb) and :921-936 (pre-fix, origin/main 487090d3fb) (!325)); the child's run-start commit writes it running through StartRun.resumedCompanionRef (runtime-temporal/src/activities.ts:9736-9739 (pre-fix, origin/main 487090d3fb) (!325)), so a child workflow that never starts leaves the ref at its prior status. Before !325 handleSendMessage wrote it running before the start (core/src/orchestration/companion-tool-dispatch.ts:870-876 (pre-fix, origin/main 0625ffeac6)), so a start failure left it running. On Temporal that prior status is usually a stale running already (CP-73: the ref of a child that ended interrupted is never set interrupted), which this call no longer writes. An interrupted child that still has a live run is not deferred: its ref is written running at :928-936 (pre-fix, origin/main 487090d3fb) (!325), backed by that run; runtime-temporal/src/workflow-dispatch.ts:1076-1086 (!325) (continue, fixed by !288, bd26dc9b57: a start failure rolls the reopen back through abandonCompanionContinuation, which restores the ref to completed with its prior delivery flag, runtime-temporal/src/activities.ts:8640-8644 (!325), and reports { delivered: false, reason: 'child_start_failed' }, also logged as persistent_agent.continue_failed). Residual edge: a rollback that itself fails is only logged as persistent_agent.continue_rollback_failed (runtime-temporal/src/workflow-dispatch.ts:692-699 (!325)); the call still reports not delivered, but the ref stays running until a later consult rolls the reopen back (runtime-temporal/src/activities.ts:8564-8571 (!325), only once the continuation is abandoned); core/src/orchestration/companion-tool-dispatch.ts:916-919 (pre-fix, origin/main 487090d3fb) and :938-946 (pre-fix, origin/main 487090d3fb) (!325) (since !288 an interrupted child's message is not appended to the child's log; it rides resumeMetadata.message into the child workflow's run-start commit), so a resume whose wf.startChild throws drops the message, which before !288 stayed in the child's log for a later run; severity S1 (re-rated from S2 with CP-103, by the owner's ruling that severity follows the consequence): on the resume path the LLM is told delivered: true for a message that is now dropped, the same wrong model behaviour and lost message as CP-103. The S2 was rated at 5bd30e20da, when the message was appended to the child's log before the start and so survived it; related: fix planned: the next Temporal bucket (orchestrator), for the spawn and resume paths: a spawn whose wf.startChild throws reports the call failed and restores the parent's ref (as core's spawn compensation and !288's continuation rollback do); a sendMessage resume whose wf.startChild throws reports the message not delivered (its ref needs no restore since !325); related: CP-103 (a started child whose entry refuses or loses the child leaves the ref 'running' the same way; the ledger split: CP-46 owns a child workflow that never starts, CP-103 one that starts and then refuses) and CP-72 (the second of two same-step sendMessage resumes collides on the child workflow id, one cause of a start failure here) | source | verified | open |
| CP-47 | S1 | CF Workflows: children notify agent__{type}__{sessionId}, but a resumed parent runs as …__resume__N, so completions go to a finished instance and the parent times out. | runtime-cloudflare/src/workflow.ts:2686, 3460, 3784, 3953 (pre-fix, origin/main a118f8e17e) (parentWorkflowId: agent__AGENTTYPE__SESSIONID at each child-dispatch site); runtime-cloudflare/src/executor.ts:1122-1125 (pre-fix, origin/main a118f8e17e); runtime-cloudflare/src/workflow.ts:3122-3125 (pre-fix, origin/main 487090d3fb) (fix evidence, CF-D1 MR 1a (!286): every child-dispatch site passes the parent's own instance id, parentWorkflowId: event.instanceId, at :3125 (pre-fix, origin/main 487090d3fb), :3965 (pre-fix, origin/main 487090d3fb), :4323 (pre-fix, origin/main 487090d3fb) and :4566 (pre-fix, origin/main 487090d3fb), so a child notifies the instance that is actually running, whether it is a per-turn, __resume__ or __retry__ instance; since !288 a re-spawned child likewise notifies the live parent through its owner token, :1652-1655 (pre-fix, origin/main e96f88bc98)), proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:1266 (a sub-agent spawned by a turn-2 parent, which runs under a per-turn instance id, delivers its completion). No runtime test spawns a child from a __resume__ instance; the conformance line below does; conformance: lifecycle.resume-after-tool-stop (executor) passes on cfw-workflows-d1 at HEAD (owner batch 15, 93cb5c49bf product code; every red and green half re-run on f083dafb67 after the !316 rebase, same verdicts). The run resumed by executor.resume() runs as a __resume__ instance, delegates to a sub-agent, and completes with the child's output in its tool result (.resumed-subagent-delivered). The fix (!286) predates the cell, so the red half is a minimal mutation of its mechanism, not an MR revert. With the per-step spawn site's parentWorkflowId (runtime-cloudflare/src/workflow.ts:3125 (pre-fix, origin/main 487090d3fb)) set back to the base buildWorkflowInstanceId({ agentType, sessionId }) id (working tree only, rebuilt, then reverted), the child notified that base id, the resumed parent waited on it, .resume-settles did not settle within 30000ms, and every later id failed on that precondition. Only the per-step dispatch site (:3125 (pre-fix, origin/main 487090d3fb)) is exercised; the sites at :3965 (pre-fix, origin/main 487090d3fb), :4323 (pre-fix, origin/main 487090d3fb) and :4566 (pre-fix, origin/main 487090d3fb) are not | source | partly-inferred | fixed (1 cells) |
| CP-48 | S2 | DBOS: a missing child definition throws inside dispatch and fails the whole parent instead of returning a tool error. | runtime-dbos/src/workflows/standard-workflow.ts:473-481 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/workflows/shared.ts:2251 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-49 | S2 | DBOS: a submit whose owning workflow cannot be resolved is acknowledged already_completed without delivery. | runtime-dbos/src/dbos-agent-executor.ts:534-536 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/lifecycle/resolve-owner-workflow.ts:38-57 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-50 | S2 | Temporal terminatePersistentChild is a no-op; DBOS terminate is flag-only (child finalizes as completed after idle timeout). | runtime-temporal/src/activities.ts (~6427); runtime-dbos/src/steps/execute-companion-tool.ts:263-273 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/workflows/persistent-workflow.ts:676-700 (pre-fix, origin/main 5bd30e20da) | source | verified | open |
| CP-52 | S1 | Client-tool submit landing during a paused run's teardown is dropped: ensureExecutionContinues returns early on isExecuting=true ('the live run will observe the state change') but the run already decided to suspend, so the server drops the submit's wake-up: the session is stranded 'paused' with the result submitted, never resumed (the caller-side half, the auto-send's /resume 409 answered with data-resume-rejected, is CP-53). Deterministic on a fresh session's first client-tool round (3/3 on two articles); delaying the POST 1.5s → 3/3 complete. | runtime-cloudflare/src/do/durable-object-agent-base.ts:644-653 (pre-fix, origin/main 5bd30e20da); ai-sdk/src/handler/handle-chat-stream.ts ~748-800; core/src/orchestration/step-iterator.ts:1754 (pre-fix, origin/main a118f8e17e) then :1774 (pre-fix, origin/main a118f8e17e) (onAgentSuspended is awaited after persistSuspension, so the session reads paused for the whole hook — a slow hook widens the window); runtime-cloudflare/src/do/durable-object-agent-base.ts:1061-1073 (pre-fix, origin/main 5bd30e20da) (recoverStrandedExecution only recovers 'active', so the stranded 'paused' session is never re-driven); live: live-repro (batch-6, origin/main e37d676945): executor tier, cf-do-d1 in the e2e workerd pool — lifecycle-hooks-parity scenario 2 with a 1 s onAgentSuspended delay fails 4/4 (expected 'suspended_client_tool' to be 'completed'), session paused with tc-1 submitted, still paused 5 s later; 3/3 pass without the delay. CI: jobs 16758514869, 16759542448, 16760323616, 16764206430; main pipelines 2887177079 / 2886530232 (peer-reported); conformance: submit.race-a on cf-do (host tier) — with the DO held inside onComplete after the client-tool pause (.window-open: a caller inside the hook AND isExecuting true), the auto-send's /resume 409s ('Agent already running') and the caller gets only data-resume-rejected (.submit-accepted, .continuation-reaches-caller), and the submitted answer never resumes the run: .run-completes does not settle within 15000ms, .not-stranded-paused sees status paused with the answered call still pending, .model-sees-result-once never gets a second model call; contrast: submit.no-race on cf-do (the same round answered 1500ms after the pause, no hold; also 3/3 at 0ms) — the run completes server-side (.run-completes, .model-sees-result-once and .not-stranded-paused pass): only the held window strands it; contrast: submit.no-race passes on js-chat-host (host tier); submit.race-a is a declared gap there (no host hook to hold, no execution flag); related: CP-53 (the no-hold control submit.no-race fails the same two caller-side ids against an IDLE DO — a different mechanism, ledgered there); conformance: race A's caller-side ids (.submit-accepted, .continuation-reaches-caller) are CP-52's while the run is held; after CP-52 is fixed they may fail under CP-53 until CP-53 is fixed — re-attribute from the observed run, never pre-assign (the same rule for the -client variant); conformance: submit.race-a-client on cf-do (client tier, the real useHelixChat / DefaultChatTransport auto-send) — with DI-20 fixed (!268) the Chat now answers the tool: .submitted and .window-open pass, the auto-send's continuation is only data-resume-rejected (.submit-accepted, .continuation-reaches-caller), and the run is stranded paused with the answer submitted (.run-completes does not settle within 15000ms, .not-stranded-paused, .model-sees-result-once) — race A's host shape, identical 5/5. Before !268 these ids were masked by DI-20 (the Chat errored on finish-step before the tool part). A CP-52 fix MR has the same caller-side caveat here as for the host cell; conformance: for CP-52's fix MR — the caller-side 409 → data-resume-rejected in race A comes from the same /resume fallback CP-53 describes, so a CP-52 fix alone will likely not turn submit.race-a's .submit-accepted / .continuation-reaches-caller green. The fix MR cannot name submit.race-a|cf-do|host in provenBy until those two ids are re-attributed from its observed run (likely to CP-53); moving them is part of that MR, not a sign the fix failed; conformance: race A's hold bracket releases on the auto-send's response headers (.submit-returned). A fix that withholds that response until the DO is idle would deadlock against the hold and show as a .submit-returned timeout — a harness bracket to change (release on the request's arrival at the host), not a regression (FU-CONF-RACE-A-RELEASE-ON-HEADERS); conformance: submit.during-suspend-hook on cf-do (host) — .run-completes, .result-consumed-once and .not-stranded-paused fail on origin/main 5bd30e20da and pass with !288; the submit.race-a{,-client} caller-side ids moved to CP-53; runtime-cloudflare/src/do/durable-object-agent-base.ts:1629-1635 (pre-fix, origin/main e96f88bc98) (fix evidence, !288, re-pointed for WA MR1 (!340), which moves the wake into the drive pipeline: requestWake records a wake that lands while an execution is live as _pendingWake instead of dropping it) and :1508-1515 (pre-fix, origin/main e96f88bc98) with :1525-1547 (pre-fix, origin/main e96f88bc98) (the settle path's afterRelease, its step 6, runtime-cloudflare/src/do/drive/settle.ts:394 (pre-fix, origin/main e96f88bc98), hands it to continuePendingWake once that execution settles, which re-reads the session and continues an unconsumed submit through ensureExecutionContinues, a wake drive) | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-53 | S1 | Resume-path fallback reads state once: when the submit auto-resumes an idle DO (producer claim held), the explicit /resume 409s and the single loadState still shows pendingClientToolCalls → rejectedOnlyResponse, while the resumed run continues with zero SSE listeners and emits the next client-tool call nobody receives. Polling the gate (100ms up to 5s) cleared in ~105ms and fixed it (3/3). | ai-sdk/src/handler/handle-chat-stream.ts:888-892 (pre-fix, origin/main 137b07d57a) (the callExecutorResume call whose 409 starts the fallback); ai-sdk/src/handler/handle-chat-stream.ts:893-984 (pre-fix, origin/main 137b07d57a) (its catch: the single loadState at :915-920, stillPending at :921-923, the rejection at :980-984); conformance: submit.no-race on cf-do (host tier; no hold, answered 1500ms after the pause) — the auto-send's /resume 409s ('Agent already running') and the caller's continuation is only data-resume-rejected (.submit-accepted, .continuation-reaches-caller), while the woken run completes server-side (.run-completes passes); conformance: submit.race-a on cf-do (host tier) fails the same two caller-side ids under the held window, where the DO is still executing — that is CP-52's mechanism and is ledgered there, not here; contrast: submit.no-race passes on js-chat-host (host tier); related: CP-52; related: CP-104 (the same stillPending read on CF Workflows, where nothing resumes the run, so the stored answer is stranded); conformance: race A's caller-side ids (.submit-accepted, .continuation-reaches-caller) are CP-52's while the run is held; after CP-52 is fixed they may fail under CP-53 until CP-53 is fixed — re-attribute from the observed run, never pre-assign (the same rule for the -client variants); conformance: submit.no-race-client on cf-do (client tier, the real useHelixChat / DefaultChatTransport auto-send, no hold) — with DI-20 fixed (!268) the Chat answers and auto-sends (.submitted passes), the continuation is only data-resume-rejected (.submit-accepted, .continuation-reaches-caller), and the woken run completes server-side (.run-completes, .model-sees-result-once, .not-stranded-paused pass) — the host control's shape, identical 5/5. Before !268 these ids were masked by DI-20; contrast: submit.no-race-client passes on js-chat-host (client tier); conformance: submit.race-a and submit.race-a-client on cf-do — .submit-accepted and .continuation-reaches-caller: the auto-send /resume answers 409 "Agent already running" and the single-read fallback returns data-resume-rejected (the server-side CP-52 ids pass); conformance: submit.race-a and submit.race-a-client are retired (WA MR1 Task 20, owner): their window, the DO still executing the paused run inside a held host onComplete, no longer exists. WA MR1 delivers the settle hook (onRunSettled) after the execution release (runtime-cloudflare do/drive/settle.ts), and nothing between the paused frame and the release awaits. Their only real cells were cf-do host and client, both expected-fail here on the same two caller-side ids. Every other lane was a declared gap (the executor tier on js-*, temporal-*, dbos-postgres and cfw-workflows-d1; js-chat-host host and client; batch-17 full matrix, a07e5aa1fb). So no runtime lost coverage; the record's remaining cells were submit.no-race / submit.no-race-client on cf-do (fixed, next line); conformance: fixed by WA MR1 (owner, Task 20; orchestrator ruling): both ledgered cells (submit.no-race cf-do host, submit.no-race-client cf-do client, the only lanes this record was ever ledgered on) now pass, an unexpected pass first seen in the Task 20 preliminary run on dbf1d82406 + the owner commits. Mechanism (from source, WA MR1's final shape): the DO's /submit-tool-result, once the whole batch is answered, awaits continueAfterSubmit within concurrentEntryTimeoutMs (runtime-cloudflare/src/do/durable-object-agent-base.ts:2979-2984 (pre-fix, origin/main e96f88bc98), continueAfterSubmit at :1635-1645 (pre-fix, origin/main e96f88bc98): requestWake runs the continuation as a wake drive, runDrive → wakeDrive → enter, and waits for its entry outcome); past the bound it answers a typed submitContinuationTimeoutError (:2994-3003 (pre-fix, origin/main e96f88bc98)) and the alarm still continues the run. So the submit is answered only after the continuation's entry has started and drained the submitted result, and the handler's single post-submit read, ai-sdk/src/handler/handle-chat-stream.ts:938-946 (pre-fix, origin/main 16de0b648c), unchanged in shape, sees the call consumed and reattaches to the continuation. The red is this record's own ledgered failure on main. The handler's read still depends on that runtime timing: FU-CHAT-RESUME-FALLBACK-SINGLE-READ-TIMING (owner WC; related CP-104, CF Workflows, open); related: an e2e sighting of this record's mechanism on main: !338 gating pipeline 2934478005, test:cloudflare job 17086002986, e2e/src/__tests__/refresh-c1-cf-do.cf.test.ts › "ai 6.0.49: control hitl-continuation (no refresh)": the live render lacked the continuation step's text the final snapshot held. WA MR1 reproduced it by force in a test-only file (not merged) holding the afterTool hook for 3 s on the answered ask_colour call inside runtime-js drainOrphanClientToolCalls (runtime-js/src/js-agent-executor.ts:1761 and :2338-2339 (origin/main f12eef982d)), after the run-start commit and the result append and before the pending entry is cleared: red 3 of 3 on main f12eef982d (rendered ≠ final snapshot, step 2 missing), green 3 of 3 on WA MR1 a22ccc7893, the hold firing each time; an unforced ×10 alternating A/B was green on both (a rare race). WA MR1 (!340) also makes the e2e C-1 settle wait fail on a data-resume-rejected part with the server's reason (e2e/src/helpers/refresh-char-exact-scenario.ts, b3e59cba13), closing the test gap that let such a rejection pass silently. FU-FLAKE-CF-DO-REFRESH-C1-HITL-CONTROL-LIVE-RENDER (owner batch 19) is re-titled as this sighting | source, live-repro | verified | fixed (2 cells) |
| CP-54 | S1 | Abandonment path auto-resumes the OLD run on the DO: abandonPendingTools submits client_tool_abandoned, the submit poke resumes the old run which continues its plan and suspends on a new client tool; execute(new message) then fails 'session is already running with unresolved client-tool calls' — new user messages dropped (1 of 4 persisted live). | ai-sdk/src/handler/handle-chat-stream.ts:446-480 (pre-fix, origin/main 5bd30e20da); runtime-cloudflare/src/do/durable-object-agent-base.ts handleSubmitToolResult → ensureExecutionContinues; ai-sdk/src/handler/handle-chat-stream.ts:662-675 (fix evidence, W1 MR2b (!344), 62f880c3bf: when the executor declares supportsAbandonOnResume, which DOFrontendExecutor and JSAgentExecutor do, a new message over unanswered client calls is ONE resume({ mode: 'with_message', abandonPending }) entry that abandons the calls inside the new run's run-start commit, so no per-call submit wakes the old run); a suspension that also awaits children still takes the per-call path, FU-WITH-MESSAGE-MIXED-CHILDREN-ORDER (docs/dev/follow-ups.md), not reached by any cell; conformance: fixed for a run paused on client calls alone. chat.send-over-pending-old-plan (owner batch 29, host tier, cf-do and js-chat-host): the first turn suspends on an unanswered client call, then a send carries a new message; the send's run completes, the next model call holds the abandoned call's result and then the message, and the message is persisted exactly once, before the reply. Green 3 of 3 on both hosts on main 6ab56e113a plus owner batches 25 to 28. Red with the pre-!344 rule restored (the handler's per-call abandon submits always taken; working tree only, reverted): .send-completes, .next-call-carries-message and .message-persisted-once fail on both hosts, the send refused as already running while the woken old run continued without the message, the record's shape | live-repro, source | partly-inferred | fixed (2 cells) |
| CP-55 | S1 | Client hydrate re-executes already-applied client tools: after reload the snapshot shows the session 'paused' with input-available parts; useResumeClientTools (per-mount optimistic-done set, empty after reload) re-runs the handlers and auto-sends submit-message POSTs, although the DO already recorded those results (submitted before reload, stranded by CP-52). The snapshot does not expose submitted-but-unconsumed results, so the client cannot distinguish 'pending, never executed' from 'executed, submitted, run stranded'. Live: insertAfter applied twice. | ai-sdk/src/react/use-resume-client-tools.ts (hydrate dispatch, per-mount done set); runtime-cloudflare/src/do/durable-object-agent-base.ts /snapshot (pendingClientToolCalls payload); runtime-cloudflare/src/do/durable-object-agent-base.ts orphan recovery, in the runtime-js executor the DO runs (runtime-js/src/js-agent-executor.ts:1685-1697 (pre-fix, origin/main 5bd30e20da) labels a submission the prior process already captured inherited_success); ai-sdk/src/handler/handle-chat-stream.ts:663-674 (pre-fix, origin/main 5bd30e20da) (unknown-tool-call rejection on the second POST); ai-sdk/src/handler/snapshot.ts:219-223 (origin/main 6ab56e113a) (markPendingApprovals visits a client call's input-available part and skips a pending entry with submittedAt; nothing else re-renders it, so a stored, not yet consumed answer still renders as input-available) with ai-sdk/src/react/use-resume-client-tools.ts:336 (origin/main 6ab56e113a) (the client's dedupe is per mount, empty after a reload); related: Ruling O-69 (orchestrator, 2026-10-13): a submitted-but-unconsumed client call renders in the snapshot as output-available from submittedResult with providerExecuted false (the answer is durable, R77); rendering it as pending is what makes a reload dispatch the handler again. The CP-52 / CP-53 durable wake now continues a stranded submit, so the window is narrower than when this record was filed, but it remains; conformance: snapshot.submitted-unconsumed-host (owner batch 29, host tier, cf-do): the run suspends on one client call with its onAgentSuspended hook held, the answer is POSTed and stored (submittedAt set, no tool result yet), and the snapshot is read before the release. On main 6ab56e113a plus owner batches 25 to 28 it fails .submitted-renders-output 3 of 3: the call's part is dynamic-tool with state input-available and no output. js-chat-host is a declared gap (no live execution flag to hold the window); related: the client half is not covered yet: a client-tier cf-do cell that answers a client call, reloads while the answer is stored but unconsumed, and asserts the handler is not dispatched again and nothing is re-submitted. It needs a per-call handler-dispatch count in ClientObservation, and a hold on the submit's wake-drive entry, since beforeEntry covers request drives only; ai-sdk/src/handler/snapshot.ts:181-190 (fix evidence, W1 !408, 4fbeda19e6) (buildSnapshot renders pending calls for any LIVE run, active or paused, so the window between the suspension commit and the suspended status is covered) with :265-282 (markPendingCalls: a stored, not yet consumed answer renders as history, approval-responded for an approval answer, output-error for a submitted error, output-available from submittedResult with providerExecuted false; never input-available); conformance: fixed. snapshot.submitted-unconsumed-host|cf-do|host passes .submitted-renders-output at W1 !408 head 4fbeda19e6 (W1's conformance-cf run, 2026-10-17: the cell's ledgered failure was an unexpected pass, the other 231 cells as ledgered). The client half (no re-dispatch, no re-submit after a reload) has no conformance cell; it is covered by the real-client unit test ai-sdk/src/__tests__/handler/submitted-answer-reload.test.ts (W1 !408) | live-repro, source | verified | fixed (1 cells) |
| CP-56 | S1 | Temporal drops a pending interrupt when the in-flight step is also the terminal step: the durable interrupt flag is only sampled at the TOP of the main loop, before starting the next step (runtime-temporal/src/workflow.ts:1014-1018 (pre-fix, origin/main a118f8e17e)); a step that completes the run via shouldStop returns straight from the completion branch without ever re-checking the flag (runtime-temporal/src/workflow.ts:1695-1704 (pre-fix, origin/main a118f8e17e)). Combined with CP-16 (no abortSignal reaches the in-flight LLM call), a stop pressed while the last step is running is silently discarded and the run persists 'completed', not 'interrupted'. Reproduced by conformance stop.mid-llm-call on temporal-memory/temporal-redis/temporal-postgres: outcome() and observe().persisted both report 'completed' after release. | runtime-temporal/src/workflow.ts:1014-1018 (pre-fix, origin/main a118f8e17e); runtime-temporal/src/workflow.ts:1695-1704 (pre-fix, origin/main a118f8e17e); conformance: stop.mid-llm-call on temporal-memory/temporal-redis/temporal-postgres — stop.outcome-interrupted and stop.persisted-interrupted both observe status completed; inferred: the lost interrupt flag is never cleared on Temporal (only consumeInterruptFlag clears it, runtime-temporal/src/activities.ts:6077 (pre-fix, origin/main 5bd30e20da)) -> likely interrupts the next run on the session (CP-19 pattern); conformance: lifecycle.resume-with-message on temporal-memory/temporal-redis/temporal-postgres — lifecycle.resume-setup-interrupted observes status completed instead of interrupted (same held-terminal-step pattern via stop()); conformance: lifecycle.from-checkpoint and lifecycle.resume-after-turn on temporal-memory/temporal-redis/temporal-postgres — .setup-interrupted observes stop()={ok:true} but outcome completed (same held-terminal-step pattern); every other failing id in those cells fails only as a precondition of .setup-interrupted; conformance: stop.continued-turn and stop.resumed-with-message-turn on temporal-* — the held step is the terminal step, so the stop is discarded and the turn ends completed; related: CP-97 (the same lost stop on CF Workflows: its loop also reads the interrupt flag only at the top of an iteration; a different fix site); runtime-temporal/src/activities.ts:7055-7061 (fix evidence, WA MR2 (!355): the terminal (K5) commit goes through core commitTerminalHonouringStop under refuseIfStopped, so a stop recorded while the terminal step was in flight, read at :7070, or landing between that read and the commit, which the store refuses atomically, commits the stop's terminal instead of completed) with runtime-temporal/src/workflow.ts:411-414 (stopAtBoundary: the run's durable stop record, else a signalled stop, read at every step boundary); conformance: fixed. lifecycle.resume-with-message, lifecycle.resume-continue, lifecycle.from-checkpoint, lifecycle.resume-after-turn, stop.continued-turn and stop.resumed-with-message-turn pass on temporal-memory, temporal-redis and temporal-postgres at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos): each held terminal step's stop now ends the run interrupted. stop.mid-llm-call on temporal-* now fails only CP-16's .provider-call-aborted, so it no longer names this record | source, probe | verified | fixed (18 cells) |
| CP-57 | S1 | DBOS ActiveHandle.result() hardcodes status 'failed' whenever dbosHandle.getResult() throws, without checking the session's persisted terminal status. interruptImpl cancels the workflow via cancelWorkflow whenever it holds a workflowId (making getResult() reject); the session row then reads 'interrupted' only because a forced CAS fallback (active → interrupted, after a fixed delay) sets it — not via any graceful interrupt path (there is none, see CP-17). So the persisted row says 'interrupted' but the caller-facing AgentResult from the same result() call reports status: 'failed'. Reproduced by conformance stop.mid-llm-call on dbos-postgres: outcome() reports 'failed' while observe().persisted reports 'interrupted'. | runtime-dbos/src/handles/active-handle.ts:162-177 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/lifecycle/interrupt.ts:65-124 (pre-fix, origin/main 5bd30e20da) (inside if (workflowId): 83-92 cancelWorkflow, 110-124 forced CAS fallback active → interrupted); conformance: stop.mid-llm-call on dbos-postgres — stop.outcome-interrupted observes status failed while stop.persisted-interrupted passes; related: CP-17 (DBOS interrupt calls cancelWorkflow immediately); conformance: lifecycle.resume-with-message on dbos-postgres — lifecycle.resume-setup-interrupted observes status failed instead of interrupted; conformance: stop.mid-tool-ignores-abort on dbos-postgres — .outcome-interrupted observes status failed ("Workflow ... has been cancelled") while .persisted-interrupted passes; conformance: lifecycle.from-checkpoint and lifecycle.resume-after-turn on dbos-postgres — .setup-interrupted observes stop()={ok:true} but outcome failed ("Workflow ... has been cancelled"); every other failing id in those cells fails only as a precondition of .setup-interrupted; conformance: stop.continued-turn and stop.resumed-with-message-turn on dbos-postgres — the stop reads failed ("Workflow … has been cancelled"), not interrupted; conformance: stop.mid-llm-call-abort-ignored (the CP-59 cell) fails .outcome-interrupted on dbos-postgres at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos) with this record's shape: stop() reports ok, and result() reports failed with 'Workflow … has been cancelled'; conformance: lifecycle.concurrent-resume's prelude (a mid-tool stop) fails .setup-interrupted on dbos-postgres (main 8dc3a5c967 plus owner batches 24 to 28, in the owner lock window of 2026-10-13 with every lane run in full) with this record's shape: stop() reports ok and result() reports failed with 'Workflow … has been cancelled'; the cell's other ids fail on that precondition; runtime-dbos/src/lifecycle/interrupt.ts:126-142 and :48-52, :259-264 (origin/main b6b3eeb141) (interruptImpl's waker sends the control message, then cancelWorkflow; settleCancelledOwner sleeps DEFAULT_STATUS_UPDATE_DELAY_MS, 500 ms, before committing the stop's interrupted terminal) with runtime-dbos/src/handles/active-handle.ts:228-251 (origin/main b6b3eeb141) (the handle's rejected-getResult() path probes the owner and calls settleDeadOwnerRun with no fallback: the committed outcome when the terminal landed, else status failed) (traced by W1, 2026-10-16); conformance: withdrawn from the ledger as a race (owner, orchestrator ruling 2026-10-16). companion.spawn-discarded-on-stop on dbos-postgres reported UNEXPECTED PASS (.parent-interrupted: result() reported interrupted) on !379's pipeline 2957355618 (job 17249007858), whose diff touches no runtime-dbos code and nothing on the executor stop path, while subagent.spawn-discarded-on-stop stayed red on .parent-interrupted in the same job. Every one of this record's 13 DBOS cells judges the handle's outcome after a stop through the same path, so each is a declared gap on dbos-postgres until the handle awaits its own stop's terminal; conformance: it reproduces on an in-flight interrupt too: in WE-2's w-rec5 (2026-11-08, RM-119's stream.step-discarded-names-cut-calls-in-flight-interrupt on dbos-postgres) the run's handle outcome was failed, a workflow cancelled error, while the session reads interrupted; not ledgered there, since that cell judges the stream, never the handle outcome | source, probe | verified | open |
| CP-58 | S2 | runtime-js retry() without a message on a session that failed at its first LLM call throws 'No user message found to retry with.': the checkpoint written on failure has messageCount >= 1, so the candidate slice is empty, contrary to the code's stated intent to preserve the original triggering message. | runtime-js/src/js-agent-executor.ts retry() (message recovery via checkpoint.messageCount truncation, 'preserve the original triggering message' fallback); runtime-js/src/run-loop.ts persistTerminalState (invoked from onFailed) mints the failure checkpoint whose messageCount already includes the triggering message; conformance: canary drop-retry-message (lifecycle.retry-message on js-memory) — retry({}) throws instead of completing, and the strip-retry-marker-from-model-input canary on the same scenario shows the fault alone (without this defect in the way) only fails lifecycle.retry-message-reaches-model; live: probe (batch-6 verification, origin/main e37d676945, runtime-js + InMemoryStateStore): also reproduces on a continued turn — turn 1 completes, turn 2's first LLM call fails, and retry() without a message throws because the failure checkpoint's messageCount (3) already includes TURN-2; on a store whose saveState mints no checkpoint (D1-like) the same retry() completes and the model receives TURN-2, so on the two stores probed the defect needs a store whose failure path mints a checkpoint (memory does; the D1-like store does not). Postgres is inferred to behave like memory from its checkpoint-minting saveState, not probed; Redis is not established either way, because its saveState pointer names a checkpoint that was never written (RM-38); core/src/store/retry-plan.ts:198 (origin/main 5bb1eaaf2b) (fix evidence, !288: planRetry): retry() with no message after a FIRST-step failure re-runs the run's trigger (the run-start commit's rows rewind to before the triggering user message, which the retried run re-appends); the drop-retry-message canary now fails only lifecycle.retry-message-reaches-model. Flip to fixed once a JS cell proves retry({}) completes; conformance: lifecycle.retry-no-message (executor) passes on every executor lane at HEAD (js-memory, js-redis, js-postgres, temporal-*, dbos-postgres, cfw-workflows-d1): after a first-call failure, retry() with no message completes (.completes), and the retried model call carries the original user message exactly once (.retried-call-sees-trigger-once). The fix (core/src/store/retry-plan.ts:766, !288, bd26dc9b57: a run with its own trigger is truncated to just before it, and the retried input re-appends it) predates the cell, so the red half is a minimal mutation of its mechanism, not an MR revert: with the trigger dropped from the retried input (working tree only, rebuilt, then reverted), the js-memory, js-redis, js-postgres and cfw cells failed .retried-call-sees-trigger-once. Re-run on f083dafb67 after the !316 rebase, the temporal-* cells fail it too (Temporal now plans its retry through the same core planner). provenBy lists only js-memory, the one lane whose pre-fix defect was probed (the live line above). js-redis and js-postgres also go red under the mutation, which shows those cells exercise the fixed planner, not that those stores had the defect (Postgres inferred, Redis not established), so they are not in provenBy. CF Workflows runs the same core planner and passes as the contrast. The CF DO embeds runtime-js but has no executor lane; related: runtime-temporal/src/executor.ts:1661-1687 (pre-fix, origin/main e37d676945) (Temporal retry() had the same pre-fix slice at the latest checkpoint and the same throw; whether it had the defect is not established, and its cells stayed green under the mutation before !316 and go red under it after; this record is scoped to runtime-js) | source, probe | verified | fixed (1 cells) |
| CP-59 | S2 | core step-iterator commits 'completed' on the terminal (no-tool) branch without re-checking the abort signal: when the adapter returns normally after abort, a stop that reported ok is silently lost and the run ends completed (runtime-js / Cloudflare DO). JS/DO counterpart of CP-56. | core/src/orchestration/step-iterator.ts terminal-without-tool-execution branch (structured_output / __finish__ / end-of-turn text) — commits and returns completed without checking deps.abortSignal.aborted; the abort re-check exists only on the tool-execution/commit branch; conformance: canary ignore-abort-signal (stop.mid-llm-call on js-memory) — probe: stop() reports {"ok":true}, outcome() reports completed, observe().persisted reports completed; related: CP-56 (same class on Temporal); severity S2 (not S1): the committed-completed outcome needs the adapter to return a NORMAL (non-error) result after the abort. The shipped adapter, @helix-agents/llm-vercel, forwards abortSignal (llm-vercel/src/vercel-adapter.ts:382 (origin/main a67fc68801)) and on an aborted call returns an error StepResult {type: error, shouldStop: true, stopReason: error} (:541-547 (pre-fix, origin/main 5bd30e20da), cancellation classified framework_cancelled at :692-707 (origin/main a67fc68801)); step-processor.ts:559-577 turns that into statusUpdate failed + isTerminal, step-iterator.ts:1395-1408 returns kind failed before the completed commit (step-iterator.ts:1410+) is reached, and runtime-js run-loop.ts:1269-1302 (as before WA MR2, origin/main cae966c0a7) remaps failed + aborted + interrupted to interrupted. So the stop is honoured with llm-vercel; the defect bites only with an adapter that ignores the signal or when the provider completion races the abort; core/src/orchestration/step-iterator.ts:559-573 (fix evidence in code, WA MR2 (!355), spec 2026-10-11 §3.6: completedUnlessAborted turns a completed step outcome into the aborted one once the run's signal is aborted, applied by runStepIteration at :597-606 and by runDeferredFinishWith, so a stop the adapter, a tool or the step commit ignored never ends the run completed; the runtime's stop resolution decides interrupted or failed); conformance: the canary ignore-abort-signal (stop.mid-llm-call on js-memory) now fails only its own fault id, .provider-call-aborted, at WA MR2's pre-run head b43b8e1a31: .outcome-interrupted and .persisted-interrupted pass, so its mustFail no longer names this record. The record stays open until a cell proves the fix on every lane: a stop.mid-llm-call sibling whose held call ignores its abort and then returns a text terminal (plan 0d); core/src/run/resolve-terminal-with-stop.ts:128 (fix evidence, WA MR2 (!355): commitTerminalHonouringStop, a run's terminal commit under refuseIfStopped that commits the recorded stop's terminal instead, so an adapter that returns normally after the abort no longer commits completed over a stop that reported ok) reached by runtime-js/src/run-loop.ts:4461 (pre-fix, origin/main 487090d3fb), which the CF DO runs, and runtime-temporal/src/activities.ts:7076; conformance: fixed. stop.mid-llm-call-abort-ignored (owner, WA MR2 flip commit: the held model call ignores its abort and then returns a text terminal) fails .outcome-interrupted and .persisted-interrupted on main 8dc3a5c967 plus owner batches 24 to 28 on every lane: js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres, cfw-workflows-d1, cf-do and js-chat-host, each completing after a stop that reported ok. At WA MR2 head 7d4aef1be8 it passes on all of them. On dbos-postgres it fails for CP-57's reason (result() reports failed after the cancel) and is ledgered there | source, probe | verified | fixed (9 cells) |
| CP-60 | S1 | Temporal resume() on a completed (or failed) session silently returns the previous result for every resume mode except from_checkpoint — with_message drops the message, continue never reaches the model, and with_confirmation is equally ignored: resume() short-circuits to attachToCompletedWorkflow whenever the persisted status is completed/failed and no rewind target was resolved, so no workflow starts and the model is never called — while JS, DBOS, CF Workflows and CF DO all throw for these modes (JS/CF: 'Cannot resume: agent already completed'). (from_checkpoint with a resolved target is exempt since !279 and rewinds the session.) | runtime-temporal/src/executor.ts:1258-1276 (pre-fix, origin/main a118f8e17e) (terminal short-circuit: status completed/failed and rewindTarget === null (:1267 (pre-fix, origin/main a118f8e17e)) → attachToCompletedWorkflow; rewindTarget is set only for mode from_checkpoint with a resolvable checkpoint, :1219-1222, so continue / with_message / with_confirmation always short-circuit, and with_message's append at :1517-1519 runs only after it; failed branch: source-only); runtime-temporal/src/executor.ts:1230-1243 (pre-fix, origin/main 0ffde4e65b) (the short-circuit ran on status alone; the only option read before it was the stream cursor fromSequence at :1210, and options.mode was read nowhere in resume(), so from_checkpoint was ignored too (CP-31, whose executor half !279 fixed)); runtime-temporal/src/executor.ts:1849-1886 (pre-fix, origin/main a118f8e17e) (attachToCompletedWorkflow resolves the prior status/output from persisted state); runtime-js/src/js-agent-executor.ts:2817-2822 (pre-fix, origin/main 5bd30e20da) (throws 'Cannot resume: agent already completed' / 'agent has failed'); runtime-dbos/src/lifecycle/resume.ts:141-390 (pre-fix, origin/main 5bd30e20da) (CAS ['interrupted','paused'] → active; a completed session throws 'not in a resumable state'); runtime-cloudflare/src/executor.ts:724-729 (pre-fix, origin/main a118f8e17e) and runtime-cloudflare/src/do/do-executor.ts:504-509 (pre-fix, origin/main 12ed5c2518; DOWorkflowExecutor, removed with the code in WE MR B !342, user ruling 2026-10-11) (CF Workflows / CF DO throw 'Cannot resume: Agent already completed' / 'Agent has failed'); conformance: lifecycle.resume-with-message (mode with_message) on temporal-memory/temporal-redis/temporal-postgres — the first run ends completed (CP-56), resume() then reports completed with no new model call (lifecycle.resume-message-reaches-model: "no new model call for the session (had 1 before, 1 after)"; lifecycle.resume-completes passes); runtime-temporal/src/executor.ts:2065-2070 (fix evidence for the with_message half only, WA MR2 (!355), R2c a666352049: a with_message resume over a completed or failed session throws core AgentNotResumableError, code state_not_resumable, before the short-circuit, so the message is no longer silently dropped; notResumableError at :276-288) with :2071-2084 (continue and with_confirmation still short-circuit to attachToCompletedWorkflow and return the previous result). Ruling 64 (WA, 2026-10-12): CP-60 stays open for continue and with_confirmation; making them refuse typed is the CP-60 fix proper, parked with Temporal under the CF-first direction. At WA MR2's head f1c07c892f the with-message-refused-typed ids of lifecycle.resume-completed-refused and lifecycle.resume-failed-refused pass on temporal-*, and continue / with_confirmation still fail; conformance: lifecycle.resume-completed-refused and lifecycle.resume-failed-refused (owner batch 23; DI-31's cells) judge the claim directly, on a session that completed or failed by itself: on temporal-memory, temporal-redis and temporal-postgres (main ad8f96ff32) continue, with_confirmation and with_message each return the previous result, terminal 'completed' or 'failed', instead of throwing state_not_resumable. The failed branch, until now source-only, is observed here; conformance: lifecycle.resume-continue (mode continue) on the same three backends — lifecycle.continue-reaches-model fails the same way ("no new model call (1 before, 1 after)"), confirming the short-circuit is not specific to with_message; runtime-temporal/src/executor.ts:2332-2346 (pre-fix, origin/main 638aaed36c) (the short-circuit: on a completed or failed session, continue and with_confirmation recover the committed run, then return attachToCompletedWorkflow at :2338, defined at :2624); runtime-temporal/src/executor.ts:2321-2328 (as on branch reliability-cf-wa-cp60 6ff080c5ab) (fix: CP-60, WA: every resume mode but from_checkpoint over a completed or failed session finishes a committed run, then is refused typed, notResumableError, state_not_resumable, as with_message already was; attachToCompletedWorkflow is deleted, so getHandle(...).resume() of a finished session is refused too); conformance: fixed. lifecycle.resume-completed-refused and lifecycle.resume-failed-refused pass on temporal-memory, temporal-redis and temporal-postgres at the fix and fail exactly .continue-refused-typed and .with-confirmation-refused-typed with the base's executor.ts, in WA's slots drv-tc2 (part B, run at 88228f34da, the same product as 6ff080c5ab) and drv-tc2b; the mutation m60 (an untyped refusal) also fails .with-message-refused-typed. drv-tc2's one CHECK FAIL was runtime-temporal's old unit 'should attach to a completed agent (graceful, no throw) on resume' at 88228f34da, updated by 6ff080c5ab; drv-tc2b ran the full runtime-temporal suite green (987 of 987) | source, probe | verified | fixed (6 cells) |
| CP-61 | S2 | agent-server's chat routes never pass the HTTP request to the chat handler, so Last-Event-ID / X-Resume-From-Sequence / X-Existing-Message-Id never reach handleChatStream and its request-size gate never runs: every Node reconnect replays from the run start under a new message id. | agent-server/src/http/handler.ts:955-961 (pre-fix, origin/main 5bd30e20da) (POST /chat passes only sessionId/messages/userId/metadata); agent-server/src/http/handler.ts:966-990 (pre-fix, origin/main 5bd30e20da) (GET /chat/{id}/stream passes messages: [] only); agent-server/src/types.ts:31-36 (the chatHandler type has no request); ai-sdk/src/handler/handle-chat-stream.ts:551-582 (pre-fix, origin/main 5bd30e20da) (resolveResumeHints reads headers only from request); ai-sdk/src/handler/handle-chat-stream.ts:198-217 (pre-fix, origin/main 5bd30e20da) (size gate needs request); related: CP-29 (client-side resume values frozen) | source | verified | open |
| CP-62 | S1 | Stop while a server tool is running: Temporal and DBOS never fire the tool's abortSignal (each tool gets a fresh, never-aborted AbortController, and interrupt only sets a flag / cancels the workflow at the next step boundary), so the run cannot end until the tool returns on its own. A cooperative tool waiting on its signal hangs the run indefinitely (outcome never settles). | runtime-temporal/src/activities.ts:1289, 1364 (pre-fix, origin/main 5bd30e20da) (executeToolActivity: new AbortController() handed to tool.execute as abortSignal, never aborted; the activity cancellationSignal is not bridged); runtime-temporal/src/workflow.ts:1078-1099 (pre-fix, origin/main a118f8e17e) (regular tools awaited via Promise.all of executeToolActivity; no cancellation scope tied to interrupt); runtime-temporal/src/workflow.ts:341-344 (pre-fix, origin/main a118f8e17e) (the interrupt signal only sets inProcessInterrupt), 1014-1018 (sampled at the top of the next loop iteration); runtime-temporal/src/executor.ts:818-822 (pre-fix, origin/main a118f8e17e) (interrupt() only signals the workflow); runtime-dbos/src/workflows/shared.ts:2200-2208 (pre-fix, origin/main 5bd30e20da) (dispatcher.executeTool called with no signal); runtime-dbos/src/steps/execute-tool.ts:261, 278 (pre-fix, origin/main 5bd30e20da) (args.signal ?? new AbortController().signal handed to the tool); runtime-dbos/src/lifecycle/interrupt.ts:83-92 (pre-fix, origin/main 5bd30e20da) (interrupt cancels the workflow; a running step is not preempted), 94-100 (every @DBOS.step() after cancellation throws); runtime-dbos/src/workflows/shared.ts:1927 (pre-fix, origin/main 5bd30e20da) (assistant message with the tool calls appended BEFORE the tools run), 2239-2240 (tools awaited), 2411 and 2458 (promoteStaging / tool-result appendMessages — steps that throw once the workflow is cancelled); conformance: stop.mid-tool-honors-abort on temporal-memory/temporal-redis/temporal-postgres/dbos-postgres — .outcome-settles does not settle within 30000ms after stop() returned ok (dependents fail their precondition); the lane drains only because the scenario releases the tool in its finally; conformance: stop.mid-tool-ignores-abort on dbos-postgres — .transcript-paired: persisted history unpaired [slow, note] (the note call had completed); conformance: stop.mid-tool-honors-abort's stop lands after slow is running AND the co-emitted note has run (.tool-reached). Before that (FU-CONF-CP62-DBOS-SETTLE-TIMING, CI job 16756806141) a DBOS note step not yet started at the cancel threw 'cancelled' through its three retries, failing the run in ~3s with a different id set; re-checked 5/5 identical on dbos-postgres and the three temporal backends (this fragment unchanged); contrast: stop.mid-tool-honors-abort and stop.mid-tool-ignores-abort both pass on js-memory/js-redis/js-postgres (runtime-js aborts the running tool); conformance: stop.mid-tool-ignores-abort passes on temporal-memory/temporal-redis/temporal-postgres (once the tool returns, the flag is honored at the next step boundary); conformance: stop.mid-tool-keeps-finished (RM-120's cell, a completed call beside a held one) fails .abort-reaches-tool 3 of 3 on temporal-memory, temporal-redis, temporal-postgres and dbos-postgres (WA's slot-c1 at d1b1b09c7a, read by the owner): the held sf_slow's abort signal never fires within 10000ms of the stop. Its other ids there are RM-120's; related: CP-16 (the same missing abort for an in-flight LLM call), CP-17 (DBOS interrupt cancels the workflow immediately), CP-57 (DBOS outcome misreported failed), CP-19 (DBOS interrupt flag outlives the stopped run); conformance: fixed by !288 (DBOS step atomicity, runtime-dbos/src/workflows/shared.ts:1815-1820) — on origin/main 5bd30e20da, once the tool did return, the cancelled DBOS workflow never committed the step's tool results, so persisted history kept the assistant tool_use calls with no tool_result; with !288 the assistant tool_use row and its tool results commit in one step commit, so stop.mid-tool-ignores-abort|dbos-postgres .transcript-paired passes (6/6 under load); the stop.mid-tool-honors-abort fragment stays | source, probe | verified | open |
| CP-63 | S3 | CF Workflows runs a step's finishWith calls (phase 2) BEFORE dispatching the sub-agents co-emitted in the same step, while core (JS / DO) runs sub-agents in phase 1 and finishWith strictly after, so on CF Workflows a finishWith tool observes state from before its sibling sub-agents ran. Completion is still deferred until the sub-agents return, so the transcript stays paired; only the ordering (and what the finishWith tool can see) diverges. | runtime-cloudflare/src/workflow.ts:2517-2583 (pre-fix, origin/main a118f8e17e) (phase 2: finishWith tools executed); runtime-cloudflare/src/workflow.ts:2586 (pre-fix, origin/main a118f8e17e) (sub-agent dispatch only AFTER phase 2); contrast: core/src/orchestration/step-iterator.ts:1223-1224 (pre-fix, origin/main a118f8e17e) (phase 1 = regular server tools + sub-agents in parallel; phase 2 = finishWith, deferred to step 9); inferred: the user-visible effect needs a co-emitted sub-agent whose completion changes state the finishWith tool reads; not yet observed; related: RM-29 (terminal-step ordering and pairing); runtime-cloudflare/src/workflow.ts:4047 (origin/main 5bb1eaaf2b) (fix evidence, CF-D1 MR 1b (!287): Phase 2 runs after the step's sub-agent, remote and companion calls, as in core; its results are append commits, the first success wins and later calls get typed not-executed results, staged writes fold in with a fenced merge; a suspension before it defers the calls to the resumed run's run-once path), proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:1285 (a parent step with a sub-agent call and a finishWith call: the child's tool runs before finishWith, the parent completes with the finishWith output and each call is paired once; red before: ['finish', 'child']); conformance: transcript.finish-with-after-subagent (executor, owner batch 15, f083dafb67 product code) passes on every executor lane, first run (js-memory, js-redis, js-postgres, temporal-* on test-server, dbos-postgres, cfw-workflows-d1). One step co-emits a sub-agent call and finish_gated, a finishWith tool held at the tool gate; while it is held, the child's sub-session exists and reads completed (.finish-with-sees-child-done), and once released the run completes with the finishWith output (.completes). The fix (CF-D1 MR 1b, !287) predates the cell, so the red half is a minimal mutation of its mechanism, not an MR revert: with the Phase 2 finishWith block (runtime-cloudflare/src/workflow.ts:4619-4680 (pre-fix, origin/main 65061fbeb5)) moved back in front of the step's sub-agent dispatch (:3016 (pre-fix, origin/main e96f88bc98)), the pre-fix order (working tree only, rebuilt, then reverted), the cfw cell failed exactly .finish-with-sees-child-done. The other lanes run core's own step order (sub-agents in phase 1, finishWith after) and pass as the contrast | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-64 | S2 | DBOS: submitToolResult wakes the waiting workflow itself (DBOS.send → recv), so a caller's documented follow-up resume() races it: once the workflow has cleared its pending entry, resume() throws AgentAlreadyRunningError (or 'not in a resumable state' once it completes), and the executor offers no way to attach to the continued run (the execute handle's result is cached as suspended; getHandle's result is a one-shot status snapshot). The continuation's outcome is unobservable to the caller. | runtime-dbos/src/dbos-agent-executor.ts:700 (pre-fix, origin/main 5bd30e20da) (submitToolResult → DBOS.send(wfId, payload, submit.toolCallId, idempotencyKey) into the workflow still blocked on DBOS.recv — the submit itself wakes the run); runtime-dbos/src/lifecycle/resume.ts:209-217 (pre-fix, origin/main 5bd30e20da) (the accepted-race comment: once clearPendingClientToolCallStep ran, resume() sees no pending calls and the caller 'should retry or fall back to awaiting the original handle.result()'), :366 (pre-fix, origin/main 5bd30e20da) (throw new AgentAlreadyRunningError(sessionId, 'active') when the session is active with no pending calls), :387-391 (pre-fix, origin/main 5bd30e20da) ('not in a resumable state' once the woken run completed); runtime-dbos/src/handles/active-handle.ts:126-133 (pre-fix, origin/main 5bd30e20da) (result() memoises _resultPromise — the execute handle that reported suspended_client_tool keeps returning it, so the documented fallback of awaiting it never sees the continuation); runtime-dbos/src/handles/reconnect-handle.ts:77-117 (pre-fix, origin/main 5bd30e20da) (getHandle's ReconnectHandle.result() is a one-shot loadState snapshot: an 'active' session reads as status 'failed' — no way to await the continued run); conformance: smoke.client-tool-completes on dbos-postgres — the responder submits, waits for the woken run to clear the pending entry (clientToolSubmitWakesRun: the legal order a late caller hits), then resume()s: .completes fails with outcome threw 'Agent session … is already running (status: active)'; .llm-input-has-result and .llm-input-paired fail only as its preconditions (failPreconditions). Without the wait the same cell failed intermittently (1 of 6 full DBOS matrix runs) when resume() lost the race.; inferred: while CP-64 is open the DBOS continuation's correctness is unobserved — the smoke's model-input ids fail only as preconditions of .completes; inferred: the harness waits only for the pending entries to clear, so resume() may land while the woken run is still active OR after it completed — a fix must make resume() attach (or return the continued outcome) in BOTH cases, otherwise soak verdicts become mixed; related: CP-53 (the DO host-level race B), CP-49; conformance: submit.partial-batch on dbos-postgres — .completes-after-c2: the follow-up resume() after the last submit throws "Agent session … is already running (status: active)"; conformance: owner batch 37's deferred-call cells on dbos-postgres (main 39793853de plus owner batches 35 to 37, 3 of 3): companion.spawn-deferred-behind-client-tool, companion.spawn-and-finish-deferred-behind-client-tool and stream.deferred-finish-calls-share-attempt fail .turn-completes the same way (the responder's follow-up resume() after the woken run cleared its pending entry throws 'Agent session … is already running (status: active)'); every other id of each fails only as its precondition. Nothing in them is specific to the calls deferred beside the client call: every other lane completes the turn; conformance: history.resume-tail-owed-only (RM-91's guard) on dbos-postgres, 3 of 3 at the owner batch 660dbc5f1c (owner s70) plus 1 of 1 at main 4980e8a8fb with the cells (owner s69): .turn-completes fails the same way (the responder's follow-up resume() after the woken run throws 'Agent session … is already running (status: active)', state_already_running); .resume-checkpointed and .every-checkpoint-owed-only fail only as its preconditions; conformance: RM-113's finish.deferred-by-client-tool and finish.deferred-by-approval on dbos-postgres (re-homed here by the orchestrator's ruling, 2026-11-03), owner slot s270 (tree 96b2f8aa32 = RM-113's records on WA's finish-siblings head 298fc57bc4), 3 of 3 identical, and both full DBOS lanes of owner slot s271 (tree 1471bd8503): .turn-completes fails the same way (the responder's follow-up resume() after the woken run cleared its pending entry throws 'Agent session … is already running (status: active)', state_already_running); .ack-last, .one-model-call, .output-is-finish, .persisted-paired, .sibling-effect, .sibling-ran-once and .suspending-call-result fail only as its preconditions. As CP-64's other deferred-call rows, nothing in the failure is specific to the deferred __finish__ | source, probe | verified | open |
| CP-65 | S2 | Temporal resume() on a session that is still suspended after the drain (nothing submitted, or only part of a parallel client batch submitted) never settles: the workflow re-suspends via the hasMoreSuspended early return without emitting a new suspension_marker, while the resume handle only observes chunks after the stream’s latest sequence, so handle.result() waits forever. | runtime-temporal/src/workflow.ts:751-759 (pre-fix, origin/main a118f8e17e) (resume-mode path — input.mode === 'resume' guards this block from :367 — after applyResultsAndReload's drain, the drainResult.hasMoreSuspended branch re-suspends by deriving the suspension status/payload straight from durable state via deriveSuspensionFromState and returns, without ever calling activities.emitSuspensionMarker); runtime-temporal/src/workflow.ts:1297-1302 (pre-fix, origin/main a118f8e17e) (contrast: the normal suspend branch calls activities.emitSuspensionMarker(streamId, suspensionKind, ...) BEFORE the durable commit — the only place in the workflow a suspension_marker stream chunk is written; the hasMoreSuspended early return at :751-759 skips this call entirely); runtime-temporal/src/executor.ts:1224-1252 (pre-fix, origin/main a118f8e17e) (C-3: when the caller omits fromSequence, resume() probes the stream and defaults fromSequence to its CURRENT latestSequence rather than 0 — 'pins the new handle to observe ONLY chunks emitted by the resumed workflow' — so a resumed run that emits no new chunk leaves the handle nothing to observe); runtime-temporal/src/executor.ts:1710-1715 (pre-fix, origin/main a118f8e17e) (resume() returns createTemporalAgentHandleV7({ ..., fromSequence, ... }) — the stream-observed v7 handle, not the native workflowHandle.result() that createActiveHandle uses for execute()); runtime-temporal/src/handle.ts:115-146 (pre-fix, origin/main 5bd30e20da) (observeAndProject: for await (const chunk of reader) over the reader created from fromSequence; the returned AgentResult resolves only when a terminal output/error/run_interrupted/suspension_marker chunk is observed on the stream, or the reader drains and fallbackStatusQuery runs); runtime-temporal/src/handle.ts:185-188 (pre-fix, origin/main 5bd30e20da) (result() returns the cached projection or awaits observeAndProject()’s promise — the only resolution path for this handle; with the stream still open and no chunk emitted after fromSequence, the for-await loop at :126 never yields and result() never resolves); runtime-temporal/src/workflow.ts:751-758 (pre-fix, origin/main a118f8e17e) (the same hasMoreSuspended early return is reached after a partial submit drains the submitted call of a parallel client batch, not only when nothing was submitted); live: real-provider smoke after !279 (main f27a0c570e, claude-haiku-4-5 through llm-vercel, temporal-postgres): the model emitted 2 parallel client calls; after submitting only the first, resume() returned a handle whose result() did not settle within the 60 s bound (TIMEOUT(60000ms) on the partial resume) while the session held correctly (one call still pending, no model call during the partial resume); after the second submit, a fresh resume completed; inferred: the hung resume handle reads the stream from the latestSequence it pinned at resume (runtime-temporal/src/executor.ts:1224-1252 (pre-fix, origin/main a118f8e17e)), so a later resume's terminal chunks land after it; it may resolve with the next run's outcome rather than its own. Not verified; live: peer-reported, observed while writing Temporal from_checkpoint integration tests (resume with no newly submitted client-tool results hangs handle.result()); related: CP-92 (the DBOS analogue for a partial batch, by a different mechanism: resume() attaches to the workflow still blocked on the remaining call with the suspension poll disabled); conformance: submit.partial-batch-resume on temporal-* — nothing submitted: .nothing-submitted-settles (a continue before any submit settles, refused typed: .nothing-submitted-refused-typed); part of the batch submitted: .with-message-settles, .continue-settles and .from-checkpoint-settles. They settle with !288; on origin/main 5bd30e20da the resume handle hangs; runtime-temporal/src/executor.ts:1692-1694 and :913-927 (as before WA MR2, origin/main cae966c0a7) (fix evidence, !288: resume() refuses continue / with_message on a partial batch, nothing or part submitted, with a typed state_not_resumable before any write or workflow start) and runtime-temporal/src/workflow.ts:877-889 (a from_checkpoint resume that re-suspends records suspendRun and pauses the stream, so its handle observes the suspension) | source, live-repro, peer-reported | verified | fixed (3 cells) |
| CP-66 | S2 | A denied approval-gated tool call is persisted in two different shapes depending on the runtime: failure-shaped ({"error":...}, helixToolFailed true) on CF Workflows and DBOS, but a success-shaped string (helixToolFailed false) on JS / CF DO and Temporal — so the same user decision renders differently in the UI and reads differently to the model. Which shape is correct is a contract decision. | runtime-cloudflare/src/steps.ts:4627-4632 (pre-fix, origin/main a118f8e17e) (deny path builds the tool_result via createToolResultMessage({ success: false, error: 'Tool call was not approved by the user' }); core createToolResultMessage stamps metadata.helixToolFailed = !success and content JSON.stringify({error}) — failure-shaped); runtime-dbos/src/workflows/shared.ts:981-982 (pre-fix, origin/main 5bd30e20da) (dispatchApprovalGatedTool deny branch returns { toolCallId, ok: false, error: denyMsg }; dispatched as a phase-1 call at :2187, and the workflow body's phase-1 append at :2422-2429 threads success: res.ok into createToolResultMessage — failure-shaped, same as CF Workflows); runtime-js/src/js-agent-executor.ts:158 (pre-fix, origin/main 5bd30e20da) (APPROVAL_DENIED_TOOL_RESULT = 'Tool call was not approved by the user'), :1277 (pre-fix, origin/main 5bd30e20da) (orphan-drain deny branch sets result = APPROVAL_DENIED_TOOL_RESULT while outcome.ok is true), :1470-1479 (pre-fix, origin/main 5bd30e20da) (createToolResultMessage({ result, success: outcome.ok, ... }) — success: true, content is a JSON-stringified plain string, no error wrapper — helixToolFailed false); runtime-temporal/src/activities.ts:3608-3618 (pre-fix, origin/main 5bd30e20da) (deny path — since !257 / DI-17 built via createToolResultMessage({ success: true, result: 'Tool call was not approved by the user' }): content JSON.stringify of the plain string and metadata.helixToolFailed explicitly false, the same success-shaped string as JS; the comment there states the parity with runtime-js is deliberate); runtime-cloudflare/src/do/durable-object-agent-base.ts:79 (origin/main 5bd30e20da) (durable-object-agent-base.ts imports JSAgentExecutor from @helix-agents/runtime-js — CF DO embeds the JS runtime's deny path, so it inherits the success-shaped string, not the CF Workflows failure shape); related: CP-11 (the DO discards approval decisions), MA-20 (DBOS approval gate fails open); inferred: no scenario yet; once the contract picks a shape, an executor-tier scenario (approval-gated tool, deny via the responder/submit-approval) asserting the persisted result shape witnesses it on every backend | source, peer-reported | verified | open |
| CP-67 | S1 | CF Workflows: handle.resume() after an interrupt never reactivates the root stream the interrupt ENDED, so the resumed run cannot write to it ('Stream is ended, cannot write'). The active and reconnect handles' resume() both go through the private resumeWorkflow, which calls only initStream, and initStream does not reactivate. (executor.resume() had the same gap until !279, which made it reactivate an ended or failed stream in every mode before creating its instance.) The resumed instance hits the rejection inside its first LLM step. The step reports it as a recoverable error with no tool calls, so the workflow takes the 'stopped' branch: it marks the session failed with the misleading error "Agent stopped without completing (no structured output)", and its fail-stream step then fails again on the same ended stream (retried under the default step.do policy). The outer catch re-marks the session failed and its fail-stream fails a third time, and neither finalize-run step runs. The resumed turn is lost. The loss is silent on the stream, which never shows the resumed turn or its failure; a handle.resume() caller learns of it only when the handle's result() reports the run failed. | runtime-cloudflare/src/workflow.ts:1696-1699 (pre-fix, origin/main a118f8e17e) (end-stream-interrupted: an interrupted root run ENDS its stream); runtime-cloudflare/src/executor.ts:719-729 (pre-fix, origin/main a118f8e17e) (resume() rejects completed / failed sessions in every mode except from_checkpoint, so interrupted → resume is the supported follow-up path); resumeWorkflow (private, :1708 (pre-fix, origin/main a118f8e17e); the whole of handle.resume() for the active handle :2051-2053 (pre-fix, origin/main a118f8e17e) and the reconnect handle :2185-2187 (pre-fix, origin/main a118f8e17e)) :1762-1766 (pre-fix, origin/main a118f8e17e) (CAS from paused / interrupted / active) and :1816 (pre-fix, origin/main a118f8e17e) (initStream only); the reactivateStream calls in runtime-cloudflare/src outside tests are failRewoundResume :338-339 (pre-fix, origin/main a118f8e17e), executor.resume() :1117-1119 (pre-fix, origin/main a118f8e17e) and retry() :1415-1416 (pre-fix, origin/main a118f8e17e), none of them on the resumeWorkflow path; store-cloudflare/src/stream-durable-object.ts:364-367 (pre-fix, origin/main a118f8e17e) (handleWrite rejects every write to a non-active stream with HTTP 400 "Stream is ${status}, cannot write"); ended → active only via handleReactivate (:885-896 (pre-fix, origin/main a118f8e17e)) or handleReset (:1128-1139 (pre-fix, origin/main a118f8e17e), which also deletes all chunks — runtime-cloudflare never calls resetStream); handleInit :420-429 (pre-fix, origin/main a118f8e17e) only inserts the init_claimed marker; store-cloudflare/src/durable-stream.ts:557-563 (pre-fix, origin/main a118f8e17e) (the writer turns the 400 into a thrown StreamConnectionError), :510-524 (pre-fix, origin/main a118f8e17e) (initStream = POST /init — does not reactivate); runtime-cloudflare/src/executor.ts:788-803 (pre-fix, origin/main 0ffde4e65b) (executor.resume() created the __resume__<n> workflow instance) then :821-840 (pre-fix, origin/main 0ffde4e65b) (initStream only, then run_resumed written via withStreamWriter to the still-ended stream, so resume() rejected after the instance was already created); runtime-cloudflare/src/executor.ts:1108-1119 (origin/main a118f8e17e) (fixed by !279, f27a0c570e: executor.resume() calls initStream and then reactivates an ended or failed stream, in every mode, before it creates the instance at :1138-1142 and writes run_resumed at :1208-1232; the handle.resume() path above has no equivalent); runtime-cloudflare/src/steps.ts:1067-1072 (pre-fix, origin/main a118f8e17e) (executeAgentStep's writer on state.streamId) and :1752-1763 (pre-fix, origin/main a118f8e17e) (its catch turns ANY throw inside the step body, try at :1085, into { error, recoverable: true, toolCallCount: 0 } — the workflow's llmStepRetry is not exercised); runtime-cloudflare/src/workflow.ts:2110-2167 (pre-fix, origin/main a118f8e17e) (not-continuing, zero-tool-call branch: mark-stopped records "Agent stopped without completing (no structured output)", then fail-stream-stopped at :2128 → steps.ts:3867-3887 failAgentStream writes an error chunk to the ended stream before failStream, so the step throws after its default retries and finalize-run-stopped (workflow.ts:2151 (pre-fix, origin/main a118f8e17e)) never runs); runtime-cloudflare/src/workflow.ts:4153-4210 (pre-fix, origin/main a118f8e17e) (outer catch: error-mark-failed, then error-fail-stream throws again on the same write; the enclosing try swallows it and skips finalize-run-catch :4199); contrast: runtime-js/src/js-agent-executor.ts:3198-3208 (pre-fix, origin/main a118f8e17e) (runtime-js resume() resumes a paused stream and reactivates an ended / failed one before writing); related: CP-76 (the Temporal twin: resume() calls initStream, runtime-temporal/src/executor.ts:1702 (pre-fix, origin/main a118f8e17e), and reactivates only for a completed / failed from_checkpoint rewind, :1491-1498 (pre-fix, origin/main a118f8e17e), while its interrupted terminal path ends the stream, runtime-temporal/src/activities.ts:5312-5315 (pre-fix, origin/main a118f8e17e); reproduced outside the suite (peer-reported). A conformance stop there can end the run completed instead, CP-56, and resume() then short-circuits, CP-60); related: runtime-dbos/src/lifecycle/resume.ts:647 (pre-fix, origin/main a118f8e17e) (DBOS resume() calls only initStream, which does not reactivate); whether the DBOS stream is ended at that point depends on its interrupt path (runtime-dbos/src/workflows/shared.ts:1540-1551 (pre-fix, origin/main a118f8e17e) ends it on a graceful flag check; interruptImpl cancels the workflow — CP-17 / CP-57); not assessed; live: peer-reported (real workerd via vitest-pool-workers, before the executor.resume() fix; pre-fix, origin/main 0ffde4e65b): observed the resumed run failing on the ended stream for executor.resume() with continue and with_message. The handle.resume() path that still has the gap is source-only here; inferred: which write fails first inside the step depends on the LLM adapter (llm-vercel converts a callback throw into a type 'error' step, llm-vercel/src/vercel-adapter.ts:516-547 (pre-fix, origin/main a118f8e17e); MockLLMAdapter awaits onTextDelta (since the DI-23 fix) and lets a rejected write escape generateStep, core/src/llm/mock-adapter.ts:470-476); any throw that escapes to the steps.ts:1752 (pre-fix, origin/main a118f8e17e) catch takes the path above. The stall length is Cloudflare's default step.do retry policy (not defined in this repo); runtime-cloudflare/src/executor.ts:2239 (origin/main 5bd30e20da) and :2374 (origin/main 5bd30e20da) (fix evidence, CF-D1 MR 1a: the active and reconnect handles' resume(options) call executor.resume(agent, sessionId, options); the private resumeWorkflow (no reactivation, truncate before an unguarded CAS) is deleted; resume() first waits, bounded, for the interrupted instance's terminal tail, :882), proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:582 ("T8 (CP-67): interrupt, then getHandle().resume(): the resumed turn reaches the stream and completes", red on the pre-fix executor: the resumed run ended 'failed') and :529 (origin/main 5bd30e20da), and runtime-cloudflare/src/__tests__/executor.test.ts:1144 (origin/main 5bd30e20da) (unit: delegation, stream reactivated, run_resumed written). What the plan 0c CF Workflows lane reaches is in the conformance note below; conformance: (history, superseded by the owner batch 15 line below) not proven on cfw-workflows-d1 by the plan 0c MR-2 lane flip (5 full lane runs, origin/main d2e95ad181 product code). No cell resumes an interrupted root run there. The executor driver resumes through executor.resume() and never through a handle's resume(). The cells that stop a root run and then resume it (lifecycle.resume-with-message, lifecycle.resume-continue, lifecycle.resume-after-turn and stop.resumed-with-message-turn) fail at their .setup-interrupted id in every run: the stop is lost (CP-97), the run completes, and the resume is refused or never reached. stop.mid-tool-honors-abort and stop.mid-tool-ignores-abort do interrupt a root run, and pass, but they continue it with a new execute() turn, not a resume; conformance: lifecycle.resume-after-tool-stop (executor) passes on cfw-workflows-d1 at HEAD (owner batch 15, 93cb5c49bf product code; every red and green half re-run on f083dafb67 after the !316 rebase, same verdicts). A root run stopped mid-tool ends interrupted, which ENDS its stream, and executor.resume() continues it: .resumed-completes and .resumed-stream-live (the resumed run's text reaches the stream) pass. The fix predates the cell, so the red half is a minimal mutation of its mechanism, not an MR revert. With runtime-cloudflare/src/run-entry.ts:1276 (pre-fix, origin/main 487090d3fb) (claimCfwEntryStream) making a __resume__ entry's stream live through initStream only, never reactivateStream (:831), the pre-fix resumeWorkflow shape (working tree only, rebuilt, then reverted), the resumed entry's run_started write hit the ended stream on every step retry. .resume-settles did not settle within 30000ms, and every later id (.resumed-completes, .resume-keeps-committed, .resumed-stream-live, .resumed-run-resumed-written, .resumed-subagent-delivered) failed on that precondition. This proves the executor.resume() path. handle.resume() delegates to it (runtime-cloudflare/src/executor.ts:5015 (origin/main 487090d3fb), :5119 (origin/main 487090d3fb) and :5360 (origin/main 487090d3fb)), and per the owner ruling that one-line delegation is not proven separately (a handle-resume leg waits for H3) | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-68 | S2 | runtime-js and CF Workflows: an interrupted persistent companion is not left resumable. On runtime-js, a companion whose parent is interrupted via handle.interrupt() is recorded 'failed', not 'interrupted', in both its own session and the parent's SubSessionRef: the in-process child loop is not given the parent's interrupt state, so the interrupt reaches it only as a hard abort through the parent's AbortController. On CF Workflows, the parent's record steps map an interrupted child's 'terminated' completion event to a terminal ref ('failed' for a blocking spawn or continue, 'terminated' for waitForResult); a non-blocking interrupted child keeps a 'running' ref, so sendMessage sets a parent_message flag no loop will read and reports delivered:true; and the sendMessage resume branch re-creates the child's write-once base instance id, so the resumed child can never start. Either way a later companion__sendMessage cannot resume the companion (a tool error "not active (status: failed)", or a stranded message), and spawnAgent under the same name deletes its session and starts fresh, losing the companion's memory. | runtime-js/src/js-agent-executor.ts:1887-1896 (pre-fix, origin/main 0ffde4e65b) (handle.interrupt sets interruptState.interrupted AND aborts the run's AbortController, :1926 passes that controller's signal as the run's abortSignal); runtime-js/src/run-loop.ts:4796-4804 (pre-fix, origin/main 0ffde4e65b) (spawn) and :5069-5076 (pre-fix, origin/main 0ffde4e65b) (continuation): the persistent child gets its own AbortController, aborted when the parent abortSignal aborts — the only link from a parent interrupt to the child; runtime-js/src/run-loop.ts:1375-1461 (pre-fix, origin/main 0ffde4e65b) (runChildLoop calls runLoop with abortSignal: signal, :1435, but no interruptState — unlike the root run); runtime-js/src/run-loop.ts:1080-1087 (pre-fix, origin/main 0ffde4e65b) (an aborted signal with no interruptState.interrupted is a HARD abort, abortInfo.aborted false) → onFailed :3328-3350 (pre-fix, origin/main 0ffde4e65b) (not a soft interrupt, so state.status = 'failed', persisted by persistTerminalState :3365-3367 (pre-fix, origin/main 0ffde4e65b)); the run's RunOutcome is kind 'failed'; runtime-js/src/run-loop.ts:4515-4519 (pre-fix, origin/main 0ffde4e65b) (runSubAgentLoop throws on a 'failed' outcome) → the spawn wrapper's catch writes the parent ref 'failed' (:4838-4850 (pre-fix, origin/main 0ffde4e65b)); the continuation wrapper does the same (:5107-5120 (pre-fix, origin/main 0ffde4e65b)). Even a soft interrupt reaching the child (finalizeDurableInterrupt :4209-4252 (pre-fix, origin/main 0ffde4e65b), session 'interrupted') returns kind 'failed', so the ref is still written 'failed'; core/src/orchestration/companion-tool-dispatch.ts:706-727 (pre-fix, origin/main 0ffde4e65b) (handleSendMessage: a ref in TERMINAL_STATUSES — which includes 'failed', core/src/tools/companion/helpers.ts:9-13 — can only be continued when 'completed'; otherwise it throws "Child agent '<name>' is not active (status: failed)"); companion-tool-dispatch.ts:502-512 (pre-fix, origin/main 0ffde4e65b) (spawnAgent over a failed ref deletes the child session and re-spawns fresh); contrast: runtime-js/src/run-loop.ts:3372-3376 (origin/main 5bd30e20da) (a ROOT run given interruptState records a soft interrupt as 'interrupted' and stays resumable) — the child path simply never receives interruptState; related: CP-43 (the interrupted-companion resume path, resumeChildAgent, is unreachable on JS because the ref is never left non-terminal — see there); distinct from CP-23 (parent interrupt not reaching children on Temporal / CF Workflows / the DO park loops) and CP-50 (terminate); not yet reachable by this chain — a DURABLE-flag interrupt of the parent (stateStore.setInterruptFlag, polled at runtime-js/src/run-loop.ts:673 (origin/main 5bd30e20da) → :724-732 (origin/main 5bd30e20da)) finalizes the parent via finalizeDurableInterrupt (:4363-4406 (origin/main 5bd30e20da)), which sets interruptState without aborting the run AbortController (:728-731 (origin/main 5bd30e20da)) and never touches childAbortControllers (only terminateChild aborts one, :5363-5366 (origin/main 5bd30e20da)), so the in-process child keeps running and is not marked failed. That path is not claimed here and needs its own look (an orphaned running child after a parent interrupt); contrast: CF DO is unaffected — its persistent companions are sibling DOs driven by runtime-cloudflare/src/do/persistent-companion-tools.ts, not the in-process runChildLoop or the CF Workflows record steps; inferred: DBOS / Temporal were not assessed for this mechanism (on Temporal the parent interrupt does not reach the child at all — CP-23); severity S2: the misreport is loud rather than silent — the LLM gets a tool error, not a false delivered:true — and re-spawning is a workaround, but it discards the companion session; runtime-cloudflare/src/workflow.ts:3390-3418 (pre-fix, origin/main 0ffde4e65b) (CFW blocking spawn, record-persistent step: the ref is written status childResult.status === 'complete' ? 'completed' : 'failed' with completedAt, so an interrupted child's 'terminated' event records a terminal 'failed' ref and a failed tool result); runtime-cloudflare/src/workflow.ts:3491-3528 (pre-fix, origin/main 0ffde4e65b) (CFW record-waitfor step: a 'terminated' event records a terminal 'terminated' ref); runtime-cloudflare/src/workflow.ts:3672-3699 (pre-fix, origin/main 0ffde4e65b) (CFW record-continue step for a blocking continuation: the same 'complete' ? 'completed' : 'failed' mapping); runtime-cloudflare/src/workflow.ts:3718-3737 (pre-fix, origin/main 0ffde4e65b) (CFW sendMessage resume branch: workflowBinding.create with the child's base instance id agent__<type>__<childSession>, which is write-once, so the resumed child never starts); core/src/orchestration/companion-tool-dispatch.ts:765-801 (pre-fix, origin/main 0ffde4e65b) (handleSendMessage: a non-blocking interrupted child whose ref is still 'running' passes the completed/failed guard at :765, gets a parent_message interrupt flag no live loop will read at :782-784, and the call returns delivered:true at :801); conformance: history.companion-continue on js-memory/js-redis/js-postgres — proves the runtime-js fix: pre-fix (the !279 suite on the unfixed origin/main 59d47f1e0f product code) .child-ref-interrupted and .child-session-interrupted observed 'failed', and .continue-completes, .same-child-continued and .child-made-post-interrupt-call failed with them; post-fix (with !279) all three cells pass; not yet reachable (conformance cell) — CFW half proven by runtime-cloudflare tests (runtime-cloudflare/src/__tests__/cfw-continue.wf-noiso.test.ts:605 on real workerd: a blocking companion interrupted mid-run stays resumable and sendMessage resumes it with its history; runtime-cloudflare/src/__tests__/cfw-persistent-companion.test.ts:755 (as before WA MR2, origin/main cae966c0a7): an interrupted child with a stale 'running' ref is resumed, no flag stranded; runtime-cloudflare/src/__tests__/companion-ref-status.test.ts:57: a 'terminated' event from an interrupted child records 'interrupted'); no CFW conformance cell proves it: the cfw-workflows-d1 lane runs history.companion-continue, but there the parent's stop never reaches the companion (CP-23's cfw fragment), so the child is never interrupted and this record's path is not reached | source, peer-reported | verified | fixed (3 cells) |
| CP-69 | S1 | DBOS: a BLOCKING persistent-companion spawn returns only when the child's persistent workflow idles out (persistentIdleTimeoutMs, 24h by default), not when the child's first turn completes. The parent's step sits in a durable poll of the child's SESSION status, which stays 'active' across turns. JS, Temporal, CF Workflows and CF DO return the blocking spawn after the child's turn. Nothing errors: the parent turn simply stalls until the idle timeout plus a 30s margin. | runtime-dbos/src/steps/execute-companion-tool.ts:561 (pre-fix, origin/main 5bd30e20da) (idleTimeoutMs = persistentIdleTimeoutMs ?? 86_400_000 — 24h default), :619 (pre-fix, origin/main 5bd30e20da) (the spawned child runs as a persistent workflow via ops.startPersistentWorkflow); runtime-dbos/src/steps/execute-companion-tool.ts:712-731 (pre-fix, origin/main 5bd30e20da) (blocking branch: pollChildSessionTerminal(childSessionId, deps, idleTimeoutMs + BLOCKING_SPAWN_MAX_WAIT_MARGIN_MS); the in-code SEMANTICS comment concedes it blocks until the child's idle timeout, NOT its first turn), :764-766 (pre-fix, origin/main 5bd30e20da) (250ms poll, 30s margin), :782-806 (pre-fix, origin/main 5bd30e20da) (pollChildSessionTerminal returns only on a completed / failed / interrupted SESSION status, else 'running' once the bound elapses); runtime-dbos/src/workflows/persistent-workflow.ts:480-499 (pre-fix, origin/main 5bd30e20da) (each turn runs with skipSessionStatusUpdate: true — the session stays 'active' between turns; also :598, :664), runtime-dbos/src/workflows/shared.ts:1081 (pre-fix, origin/main 5bd30e20da) (finalizeSessionId undefined in that mode), persistent-workflow.ts:514-546 (pre-fix, origin/main 5bd30e20da) (only an idle recv timeout or shutdown exits the loop) and persistent-workflow.ts:680-697 (pre-fix, origin/main 5bd30e20da) (finalizeLoop is the only session-status write, 'completed'); contrast: runtime-js/src/run-loop.ts:4970-5064 (pre-fix, origin/main 5bd30e20da) (spawnPersistentChild runs the child loop in-process) and :5067-5087 (pre-fix, origin/main 5bd30e20da) (the blocking waitForChildTerminal awaits that runPromise, which resolves when the child run ends); contrast: runtime-temporal/src/workflow-dispatch.ts:680-722 (pre-fix, origin/main 5bd30e20da) (blocking spawn: wf.startChild of the standard agentWorkflowFn, then awaits childHandle.result() — one child run); contrast: runtime-cloudflare/src/workflow.ts:3466-3500 (pre-fix, origin/main a118f8e17e) (blocking spawn waits for the child workflow sub-agent-complete event, which the child emits via notifyParentCompletion on completing its run, e.g. :2016); contrast: runtime-cloudflare/src/do/persistent-companion-tools.ts:445-451, 1008 (pre-fix, origin/main 5bd30e20da) (the DO blocking park polls the child DO /status until terminal); inferred: the child DO runs runtime-js, whose session reaches completed at the end of the turn; related: docs/dev/follow-ups.md FU-DBOS-BLOCKING-SPAWN-SEMANTICS (the tracked follow-up; the user-facing caveat is in docs/guide/sub-agents.md and docs/runtimes/dbos.md, which recommend non-blocking spawn + companion__waitForResult on DBOS); conformance: history.companion-continue on dbos-postgres — history.companion-outcome-settles: turn A's blocking companion spawn does not return within the 30s settle bound; .turn-a-completes fails only as a precondition of that settle, and the other eight ids (stop() is never called, so .stop-succeeds included) fail only as preconditions of .turn-a-completes; conformance: history.companion-continue-completed on dbos-postgres (owner batch 15, 93cb5c49bf product code; re-confirmed on f083dafb67 after the !316 rebase; the same turn A without a later stop): .turn-a-settles does not settle within 30000ms, and every later id fails as its precondition. The other lanes pass it; severity S1 (re-graded from S2, orchestrator 2026-10-30, Temporal / DBOS S1s unparked): the ledgered cells show the observable ruled on, an ordinary blocking companion spawn / continue holding the parent's turn until the child's idle-out bound (24 h by default), not until the child's turn ends: a liveness failure. The earlier S2 rested on the stall being bounded and documented, with a workaround (non-blocking spawn + waitForResult, or a short persistentIdleTimeoutMs), and conceded a de facto hang at the default; inferred: the follow-up's second consequence (a failed child turn reported 'completed' after idling out) is only partly mitigated by runtime-dbos/src/steps/run-lifecycle.ts:238-240, 260-320 (pre-fix, origin/main 5bd30e20da) (shouldPreservePriorRunStatus preserves a failed / interrupted status on the clean-exit finalize only when the most recent turn failed or was interrupted; an earlier failed turn followed by a completed turn can still be reported completed) — not claimed here; conformance: companion.respawn-after-failure-fresh on dbos-postgres (owner slot s215b, 2026-10-31, the owner stack 417d92b1ab on main 5e938d3776, 3 of 3, identical sets): after the non-blocking companion's model call fails non-recoverably, its last run record reads failed while its SESSION stays active (every run: 'session active, last run record failed (1 runs)'), so the cell's seeded precondition .child-failed never holds and .turn2-settles, .respawn-runs, .fresh-history and .fresh-state fail as its dependents: this record's per-turn session status, on a companion turn that FAILED. At CP-69's fix head (b03061b924 with the owner cells, WE-2 slot w-rsx-1031-031939, 3 of 3) the cell passes every id; conformance: fixed by !601 (CP-69's MR: a DBOS persistent turn writes its terminal to the session). history.companion-continue-completed passes every id (WE-2 slot w-cp69b-1031-022828, dbos-postgres, 3 of 3, identical sets, at CP-69's fix head b03061b924 (restacked as !601 72f71bcf0d on main a203fec499)); history.companion-continue's CP-69 ids (.companion-outcome-settles, .turn-a-completes, .child-call-reached, .stop-succeeds) pass there too, and its 6 later ids still fail for CP-23's mechanism (ledgered there), so that cell is not in this record's provenBy; conformance: re-confirmed at !601 de0c3029fd (its Greptile fixes; WE-2 slot w-601g-1031-050650, dbos-postgres, 3 of 3, identical to the w-cp69b sets); conformance: companion.respawn-after-failure-fresh (its row above, ledgered at main) passes every id at !601's own base (WE-2 slot w-rsx-1031-031939, dbos-postgres, 3 of 3, CP-69's fix head b03061b924 with the owner cells): the failed companion's session now reads failed, so the cell's seeded precondition holds and the re-spawn runs fresh; its row is dropped and it joins this record's provenBy | source, peer-reported | verified | fixed (2 cells) |
| CP-70 | S2 | DO wake silently strands an already-'paused' session when executor.resume() rejects before claiming the run. ensureExecutionContinuesImpl surfaces a resume failure only when the session reads 'active' afterwards. It re-arms only when this same call reset the session from 'active' to 'paused'. A session that was ALREADY paused matches neither, so the error is logged at warn and dropped. That includes the normal idle-DO wake after a client-tool submit or timeout. The session stays 'paused' forever, with no errorDetail, no failed stream chunk and no onAgentFail, and the client that submitted waits indefinitely. | runtime-cloudflare/src/do/durable-object-agent-base.ts:789-835 (pre-fix, origin/main 5bd30e20da) (the resume() .catch: AgentAlreadyRunningError is benign; otherwise it fails the session ONLY if stateAfter.status === 'active' (:808-815 (pre-fix, origin/main 5bd30e20da)), and re-arms ONLY if didResetFromActive && status === 'paused' (:827-834 (pre-fix, origin/main 5bd30e20da)); an already-paused session falls through with just a warn log); runtime-cloudflare/src/do/durable-object-agent-base.ts:674 (pre-fix, origin/main 5bd30e20da) (resumableStatuses includes 'paused', so an idle DO whose session is paused is exactly the wake-after-submit path); live: observed during the LLM-history completeness fix, via a DO integ fixture whose seeded assistant row was invalid (content: []) — DOStateStore.getMessages skips the row while getMessageCount counts it, the new history loader (loadLLMHistory) rejects resume() with a typed state_history_incomplete before claiming, and the session stayed 'paused' with no failure recorded (do-hibernation-wake.integ.test.ts before its fixture was fixed); inferred: any other pre-claim resume() rejection (a transient store error, an agent-validation failure) takes the same path; not observed; related: MA-01 (agent-factory throw on wake, also only warn-logged, but that path is retried by the watchdog); CP-52 / CP-53 (other ways a client-tool submit to the DO is lost); RM-13 (corrupt rows skipped while counted); runtime-cloudflare/src/do/drive/run-drive.ts:1716-1736 (origin/main f7a78e05d1) (fix evidence, WA MR1 (!340), origin/main f7a78e05d1: a wake drive's failure is classified by core classifyEntryFailure and applied by wakeActionFor; a fatal one is terminal) with runtime-cloudflare/src/do/drive/failure-policy.ts:20-23 (origin/main f7a78e05d1) (lost: noop, refused: terminal, transient: rearm, fatal: terminal); conformance: fixed. wake.paused-submit-fails-loudly (owner batch 38, host tier, cf-do; new control fault-do-store, messageCountSkew: the DO store's getMessageCount reads one row more than the log holds, the record's own live mechanism): the session pauses on a client call, the fault is armed, the DO is evicted, and the answer is POSTed as the auto-send. The continuation's history load fails non-retryable state_history_incomplete before claiming. At main f7a78e05d1 plus !386 and owner batches 37 and 38, 3 of 3, every id passes: the session ends failed with that detail and a failed stream, never left paused. Red mutation (fatal -> noop): .error-detail-typed fails; the watchdog's owed-submit wakes still end the session (a bare exhaustion), so the detail id is what catches it | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-71 | S1 | runtime-js strands a session 'active' forever when anything inside the step loop THROWS instead of returning a failed outcome. runLoop and runWithNewRunLoop have try/finally but no catch, so the throw reaches the handle factory's safety net, which its comments once called unreachable (they now say it is reachable and cite this finding). That net reports { status: 'failed' } to the caller but writes nothing. The session stays 'active', the run 'running', and the stream 'active' with no terminal chunk, and onAgentFail never fires. Every later execute() throws AgentAlreadyRunningError, retry() refuses ('status is running') and resume() refuses ('already running'), so the API cannot recover the session. Triggers include a systemPrompt function that throws, a schema-invalid __finish__ from an adapter that reports it as a tool call, and an adapter StepResult that breaks the contract. | runtime-js/src/run-loop.ts:949 (pre-fix, origin/main 5bd30e20da) (await runStepIteration with no catch), :568 (pre-fix, origin/main 5bd30e20da) and :1234 (pre-fix, origin/main 5bd30e20da) (the runLoop body try has only a finally); runtime-js/src/js-agent-executor.ts:4581-4750 (pre-fix, origin/main 5bd30e20da) (runWithNewRunLoop: statelessRunLoop at :4710 inside try / finally, no catch), :2320-2328 (pre-fix, origin/main 5bd30e20da) (comment, rewritten to cite CP-71: runWithNewRunLoop DOES reject when something inside the loop throws, and the handle factory safety net persists nothing; same note at :3432 and :3786 (pre-fix, origin/main 5bd30e20da)); the pre-fix comment said the runLoop "never rejects under normal operation" (js-agent-executor.ts:1940-1943 (pre-fix, origin/main 59d47f1e0f)); runtime-js/src/execution/handle-factory.ts:181-200 (pre-fix, origin/main bd26dc9b57) (the .catch safety net: it caches { status: failed, error, errorDetail } and persists nothing — no session status, no run status, no stream end, no hook; its comment at :181-187 now says the branch IS reachable and persists nothing, citing CP-71, where it used to say the branch is "unreachable in normal operation" (handle-factory.ts:182-183 (pre-fix, origin/main 59d47f1e0f))); core/src/orchestration/step-iterator.ts:665-667 (pre-fix, origin/main a118f8e17e) (buildMessagesForLLM evaluates a systemPrompt function, unguarded), :893-897 (pre-fix, origin/main a118f8e17e) (planStepProcessing, unguarded; the in-code note: ordinary __finish__ completions still parse and throw); core/src/orchestration/step-processor.ts:262-267 (origin/main da5f85ce42) (coerceSubAgentCallList calls .map on an absent subAgentCalls); contrast: runtime-js/src/run-loop.ts:984-1016 (pre-fix, origin/main 5bd30e20da) (a StaleStateError surfaced as a failed OUTCOME is persisted: status failed, run failed, stream finalized); core/src/orchestration/step-iterator.ts:851 (pre-fix, origin/main a118f8e17e) (an adapter throw is caught and becomes a failed outcome); live: probe (runtime-js + store-memory dist, this session): an agent whose systemPrompt function throws. The result is { status: 'failed', error: 'systemPrompt boom' }. One second later the session is 'active', the run 'running' and the stream 'active', and an onAgentFail hook ran 0 times. A second execute() throws AgentAlreadyRunningError, retry() throws "agent status is 'running'", resume() throws 'agent is already running'; live: probe (this session) — an adapter returning tool_calls without subAgentCalls strands the session identically (TypeError "reading 'map'"); the independent verifier reproduced the same with a __finish__ tool call carrying schema-invalid args (ZodError) and located the throw with an inspector; runtime-cloudflare/src/do/durable-object-agent-base.ts:1826 (pre-fix, origin/main 5bd30e20da) (the DO /start awaits handle.result(), which RESOLVES to status 'failed', so the .catch arm at :1907 that fails the session and stream never runs), :1855 (pre-fix, origin/main 5bd30e20da) (the stream is ended only on 'completed') — the DO /start path does not rescue the session; inferred: the DO wake path treats 'active + not executing' as a stranded run and auto-resumes it (runtime-cloudflare/src/do/durable-object-agent-base.ts:668-674 (pre-fix, origin/main 5bd30e20da)); a deterministic throw would recur on each resume until the watchdog cap fails the stream (MA-01, auto_resume_exhausted), with the real error lost. Not verified; Temporal, DBOS and CF Workflows not checked; inferred: llm-vercel is not exposed to the __finish__ trigger — it returns __finish__ as structured_output (llm-vercel/src/vercel-adapter.ts:449-465 (pre-fix, origin/main 5bd30e20da)), which does not parse there; severity S1: the caller is told 'failed' while the durable session is permanently wedged 'active', with no failure hook, no terminal stream chunk and no API path to recover it; a user systemPrompt function is an ordinary trigger; runtime-js/src/run-loop.ts:1472 (origin/main 5bb1eaaf2b) (fix evidence, CF-D1 MR 1b (!287): runLoop gains a catch; a throw before the loop's terminal handling ends through onFailed, whose terminal commit is fenced on the run, then finalizeStreamForFailure, so the session and run are failed with the resolved errorDetail, the stream fails and onAgentFail fires once; a throw after terminal handling started still surfaces; the handle-factory safety net comments now say it sees only throws outside the loop), proven by runtime-js/src/__tests__/terminal-errors.test.ts:90 (a throwing systemPrompt: session, run and stream failed, onAgentFail once, the next execute() completes; red before: the session stays active) and runtime-cloudflare/src/__tests__/terminal-errors-do.cf.test.ts:69 (the same on the CF DO's real DOStateStore under workerd; red without the catch).; conformance: lifecycle.loop-throw-fails-run (a systemPrompt that throws on every step) proves the fix on js-memory, js-redis and js-postgres (CF-D1 MR 1b, !287): the outcome, the persisted session and the stored stream are failed with the prompt's error, onAgentFail fires once, and a follow-up execute() is accepted. On origin/main bd26dc9b57 the same cells fail .persisted-terminal, .session-failed, .stream-failed, .on-agent-fail-once and .follow-up-accepted. The cell also passes on temporal-memory, temporal-redis and temporal-postgres, on main as well (Temporal never stranded the session); dbos-postgres fails it at its own seam (CP-94) | source, probe, peer-reported | verified | fixed (3 cells) |
| CP-72 | S1 | Temporal: two companion__sendMessage calls to the same INTERRUPTED persistent child in one parent step compute the same child workflow id. The id is …__persistent__<childSessionId>__resume__<stepCount>, with no tool-call id in it. Companion calls run concurrently, so both activities can read the child's session status as 'interrupted' (wasInterrupted is keyed on the child session, with the parent's ref only as a fallback, and a call updates only the parent's ref, so the session stays 'interrupted' until the resumed child workflow claims it), append their message, report delivered: true, and return resumeMetadata. The child is started with WORKFLOW_ID_REUSE_POLICY_ALLOW_DUPLICATE, so the second wf.startChild fails only while the first child workflow is still open. When it does fail, the error is only logged. If the first child run had already loaded its history, the second message stays in the child's log unprocessed, although the parent was told it was delivered. | runtime-temporal/src/workflow-dispatch.ts:787-816 (pre-fix, origin/main 5bd30e20da) (sendMessage-to-interrupted branch: childWorkflowId built at :790 from parentWorkflowId, childSessionId and stepCount only; wf.startChild at :792; the catch at :809-815 logs persistent_agent.resume_failed and continues); runtime-temporal/src/workflow-dispatch.ts:616-630 (pre-fix, origin/main 5bd30e20da) (dispatchCompanionTools runs every companion call of the step concurrently via Promise.all); core/src/orchestration/companion-tool-dispatch.ts:790-839 (pre-fix, origin/main 5bd30e20da) (handleSendMessage: wasInterrupted from the child's own session status loaded at :754, falling back to the ref only when the session is unreadable, :800-802; the message is appended at :812, only the parent's ref is set to 'running' at :821-827, and delivered: true with resumeMetadata is returned at :829-836) — nothing here moves the child session off 'interrupted', so two calls both read it until the first resumed child claims the session; core/src/orchestration/companion-tool-dispatch.ts:770 (pre-fix, origin/main 0ffde4e65b) (before the CP-68 change in f27a0c570e: wasInterrupted came from the parent's ref alone, which the first call sets to 'running', so the two calls had to interleave between that read and that write); contrast: runtime-cloudflare/src/workflow.ts:3753 (pre-fix, origin/main a118f8e17e) (CF Workflows suffixes its companion continuation instance id with the companion call id — the fix direction: append the toolCallId); inferred: since CP-68 the second call no longer needs to interleave with the first activity: it collides whenever it reads the child session before the first resumed child has claimed it (only after that claim does it read a live child and take the parent_message flag path at core/src/orchestration/companion-tool-dispatch.ts:817 (pre-fix, origin/main 5bd30e20da) instead, CP-41). It also needs the first child still running when the second start is attempted: under ALLOW_DUPLICATE (runtime-temporal/src/workflow-dispatch.ts:806-807 (pre-fix, origin/main 5bd30e20da)) a closed first child lets the second start succeed, and that run reloads the whole log, so nothing is lost. The same stepCount-only suffix is used for the continuation id (runtime-temporal/src/workflow-dispatch.ts:833 (pre-fix, origin/main 5bd30e20da)); a second continuation there is expected to fail its completed-to-active CAS first, so it is not claimed. Not observed; related: CP-41 (sendMessage to a running companion on Temporal is treated as an interrupt); CP-46 (Temporal startChild failures for companion spawn / resume / continue are only logged); core/src/orchestration/companion-tool-dispatch.ts:916-919, :938-947 (pre-fix, origin/main 487090d3fb) (since !288, bd26dc9b57, the consequence is worse than this record's pinned evidence shows: an interrupted child's message is no longer appended to its log), and each call's message rides its own resumeMetadata into its own wf.startChild (runtime-temporal/src/workflow-dispatch.ts:893-928; the id is still stepCount-only at :892, and the calls still run concurrently, :723). So the second start's failure (only logged, :929-935) drops the second message entirely, with the parent told delivered: true. inferred: (unverified) when the first child has already closed, the second start succeeds but carries the first activity's childSeenRunId, so its entry presumably loses its run-start to the first child's newer run and ends without the message (CP-103's lost run-start); severity S1 (re-rated from S3 with CP-46 and CP-103, by the owner's ruling that severity follows the consequence, not the frequency): a message silently lost while the LLM is told it was delivered. The trigger stays narrow (two sendMessage calls to the same interrupted child in one step); related: fix planned: the next Temporal bucket (orchestrator): a per-call child workflow id for the sendMessage resume (append the toolCallId, as CF Workflows does), with CP-46's start-failure handling; severity, superseded (S3, as rated at 5bd30e20da): needs two sendMessage calls to the same interrupted child in one step, the second reading the child session before the first resumed child claims it, with the first child still running at the second start; the loss is one message, which stays in the durable log | source, peer-reported | partly-inferred | open |
| CP-73 | S2 | Temporal never records an interrupted persistent companion as interrupted on its parent. When the child workflow ends 'interrupted', the parent-side SubSessionRef keeps a stale 'running': the completion bookkeeping maps only 'completed' / 'failed', and core's ref sync deliberately leaves an interrupted child's ref at 'running'. Core's handleWaitForResult returns only on a terminal ref. Its orphan detection runs only where the runtime supplies a liveness probe, and only runtime-js does. So an untimed companion__waitForResult on that child on Temporal never returns 'interrupted'. It keeps polling until the companion activity itself times out, and the parent's turn stalls, then gets a generic failure. | runtime-temporal/src/activities.ts:5938-5979 (pre-fix, origin/main 5bd30e20da) (markPersistentChildCompleted: maps only a 'completed' / 'failed' child session, returns without writing for any other status, :5951-5953); the only other ref writes are the continuation reset to 'running' (:6413-6417 (pre-fix, origin/main 5bd30e20da)) and the unused helper runtime-temporal/src/state-helpers.ts:179-192 (pre-fix, origin/main 5bd30e20da) — nothing in runtime-temporal writes 'interrupted' to a SubSessionRef; core/src/orchestration/companion-tool-dispatch.ts:871-927 (pre-fix, origin/main 5bd30e20da) (resolveAndSyncSubSessionRefs maps only completed / failed; the comment at :891-894 keeps an interrupted child's ref at 'running'); core/src/tools/companion/helpers.ts:9-13 (TERMINAL_STATUSES = completed, failed, terminated); core/src/orchestration/companion-tool-dispatch.ts:986-1156 (pre-fix, origin/main 5bd30e20da) (handleWaitForResult: without a timeout the loop exits on a terminal ref, an interrupted ref (below), isCancelled(), or orphan detection; orphan detection at :1114-1117 requires deps.isChildExecutionLive, which only runtime-js supplies, runtime-js/src/run-loop.ts:5375 (pre-fix, origin/main 5bd30e20da)). On Temporal (ref stuck at running, no liveness probe) none of these fires; runtime-temporal/src/activities.ts:6432-6439 (pre-fix, origin/main 5bd30e20da) (isCancelled reads the activity cancellation signal); runtime-temporal/src/workflow.ts:301-312 (pre-fix, origin/main a118f8e17e) (the activity proxy: startToCloseTimeout 5 minutes, maximumAttempts 3); runtime-temporal/src/workflow-dispatch.ts:636-651 (pre-fix, origin/main 5bd30e20da) (a failed companion activity becomes a generic failed tool result); inferred: the untimed wait is therefore bounded only by the activity timeout and retries (roughly three 5-minute attempts), after which the tool reports a generic activity failure, not 'interrupted'. Whether the timed-out attempts' polling loops are cancelled on the worker (no heartbeat is sent) was not checked. Not observed; contrast: runtime-cloudflare/src/workflow.ts:1701-1710 (pre-fix, origin/main 5bd30e20da) (a CF Workflows child ending on an interrupt notifies its parent with status 'terminated'), and :3597-3712 (pre-fix, origin/main a118f8e17e) (waitForResult waits for that event, bounded by subAgentEventTimeout, and records it); runtime-js records the interrupted child's ref 'interrupted' (CP-68, fixed) and core's waitForResult then returns status 'interrupted' after three consecutive observations; related: CP-41 (a Temporal sendMessage to a running companion is treated as an interrupt, so the child ends interrupted; its 'waitForResult can hang' is this finding's consequence); CP-23 (a parent interrupt does not reach Temporal children, a different path); CP-68 (JS records the interrupted child 'failed'); related: likely fix (from a code review, peer-reported): Temporal writes the ref 'interrupted' when the child workflow ends interrupted, and waitForResult treats an interrupted ref as a result; severity S2: needs an interrupted companion plus an untimed waitForResult; the parent turn stalls for the activity timeout and then gets a misleading failure instead of 'interrupted'; inferred: the fix has two halves — Temporal must write an interrupted child's ref, AND core must treat an interrupted ref as settled for an untimed waitForResult. The core half is fixed (below); the Temporal half is open; DBOS not checked; core/src/orchestration/companion-tool-dispatch.ts:1074-1107 (origin/main 5bd30e20da) and :1184-1196 (origin/main 5bd30e20da) (isAwaitingResume) — the latent core half, fixed in the LLM-history completeness fix: core's waitForResult did not treat 'interrupted' as settled (TERMINAL_STATUSES at core/src/tools/companion/helpers.ts:9-13 excludes it). It now returns { status: 'interrupted', message } for an interrupted ref, debounced over three consecutive observations, with or without a liveness probe; unit tests core/src/orchestration/__tests__/companion-tool-dispatch.test.ts:1931 (origin/main 5bd30e20da), :1951 (origin/main 5bd30e20da), :1963 (origin/main 5bd30e20da). It becomes reachable on Temporal once Temporal marks the child's ref 'interrupted'; conformance: precondition for flipping CP-73: the MR that fixes the Temporal half must add a conformance scenario that calls companion__waitForResult on an interrupted child (no current scenario does, so no cell can prove this finding yet); core/src/orchestration/companion-tool-dispatch.ts:1240-1276 (as before WA MR2, origin/main cae966c0a7) (fix evidence for the untimed-wait symptom only: !325, its CP-103 W1 change): without a liveness probe, isAwaitingResume now treats a 'running' ref over an 'interrupted' child session with no live child run as awaiting a resume, so an untimed companion__waitForResult on an interrupted Temporal companion returns 'interrupted' after the ORPHAN_CONFIRMATIONS debounce (:1104 (pre-fix, origin/main 487090d3fb)) instead of polling until the activity times out. The ref itself still reads 'running' on Temporal, so companion__getChildStatus and companion__listChildren still report 'running': this record stays open | source, peer-reported | partly-inferred | open |
| CP-74 | S1 | DBOS persistent agent: a user message sent just as the persistent workflow exits on its idle timeout is silently lost. The workflow drains its inbox, then finalizeLoop marks the session 'completed', but the workflow stays DBOS-PENDING until its finally cleanup step returns. An execute() in that window routes on the DBOS status alone, so it takes the SEND path (DBOS.send to 'inbox') into a workflow that never calls recv again. execute() returns a handle with no error, the message stays unconsumed in dbos.notifications, the session stays 'completed', and no new workflow starts. | runtime-dbos/src/workflows/persistent-workflow.ts:533-537 (pre-fix, origin/main 5bd30e20da) (idle timeout: one non-blocking inbox drain, then finalizeLoop) and :680-696 (pre-fix, origin/main 5bd30e20da) (finalizeLoop writes session status completed via finalizeRun); runtime-dbos/src/workflows/persistent-workflow.ts:301-365 (pre-fix, origin/main 5bd30e20da) (the finally block still runs HookStep.cleanupPerCallHookManagerStep after finalizeLoop, so the workflow is PENDING after the session reads completed); runtime-dbos/src/persistent/routing-decision.ts:34 (pre-fix, origin/main 5bd30e20da) (ALIVE_STATUSES = PENDING, ENQUEUED) and :53 (pre-fix, origin/main 5bd30e20da) (send when alive) — routing reads only the DBOS workflow status, never the session status, and nothing verifies the send is consumed; live: peer-reported, reproduced locally with no product change (a 1 s idle TTL; execute, wait for 'completed', execute again immediately): 5 of 6 runs lost the second message, the workflow was PENDING at 'completed' with finalizeRunStep as its last step, and the inbox row stayed in dbos.notifications; live: CI pipeline 2885443694 job 16754864348 — multi-message-execute-dbos-persistent.integ.test.ts "idle-timeout exit then new message starts a fresh persistent workflow" failed at :442 (expected 2 user messages, got 1); it passed on the automatic retry (job 16755337462); related: CP-69 (DBOS blocking persistent spawn waits for the idle timeout); other runtimes' persistent routing not checked; related: FU-DBOS-PERSISTENT-SEND-INTO-EXITING-WORKFLOW (pending in MR !259 / !266, not yet on main); runtime-dbos/src/workflows/persistent-workflow.ts:534-567 (origin/main 638aaed36c) (fix, present at main's root 1c5caeaa23: after the idle drain finds nothing, finalizeLoop records the exit and the loop reads its inbox ONCE more, R43 (b), :557); runtime-dbos/src/lifecycle/execute.ts:488-498 and :80-145 (origin/main 638aaed36c) (fix, cb8175c1d5, !617, CP-69 batch 4: a loop that recorded its exit is never sent to, a new loop is started; awaitInboxTaken answers not_taken once the exit marker shows with no run, so a new loop takes the message); conformance: fixed. persistent.message-after-idle-exit-consumed (executor, WA's !797 at 210c58b4fd; needs 'persistent-root-loop', a declared gap on the 7 other executor lanes) passes 3 of 3 on dbos-postgres in WA's slot-cp74b (40 checks, 0 failed). There is no red phase, since the fix predates the cell; the two-sided mutation m74.py (the executor sending to an exited loop with awaitInboxTaken reading a recorded exit as taken, and the loop skipping its post-finalize read, the losing order held by two observed events) fails exactly .turn2-input-persisted and .turn2-answered | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-75 | S3 | companion__sendMessage to a persistent child whose session does not exist yet is not treated as 'not started'. Core's handleSendMessage loads the child session, and a null result passes every guard (the C-4 pending-client-tool check and the I1 terminal check both read it through optional chaining), so it goes on to append the message. The child's ref already reads 'running' in that window: a re-spawn deletes the child session and sets the ref 'running' before the child is recreated, and on Temporal and CF Workflows the child session is created only later, by the child workflow. Every store's append now throws for a missing session (memory, Postgres and Redis 'Session not found', D1 'State not found for sessionId'), so the model gets a tool error for a child that is starting normally, and has to retry the send. Since RM-50's fix (MR !269) Redis no longer writes a status-less hash, sets a stale parent_message flag or reports delivered: true here. The DBOS and CF DO runtimes use their own sendMessage handlers and are not affected. Fixed in core (runtime-js, Temporal, CF Workflows) before 1c5caeaa23: the append runs only under a live child run, else the typed child_not_running. | core/src/orchestration/companion-tool-dispatch.ts:754 (pre-fix, origin/main 5bd30e20da) (childSessionState = loadState(child)), :766-778 (pre-fix, origin/main 5bd30e20da) (C-4 guard, optional-chained, so a null state passes), :786 (pre-fix, origin/main 5bd30e20da) (I1 terminal guard, same), :812 (pre-fix, origin/main 5bd30e20da) (appendMessages to the child — throws for a missing session, so :818 setInterruptFlag and :839 (pre-fix, origin/main 5bd30e20da) delivered: true are not reached); the dispatcher's catch at :454-455 (pre-fix, origin/main e96f88bc98) turns the throw into a failed tool result; core/src/orchestration/companion-tool-dispatch.ts:513-535 (pre-fix, origin/main 5bd30e20da) (re-spawn of a failed / terminated child: deleteSession at :521, then the ref is set 'running' at :529-535, before spawnPersistentChild at :572 recreates the child) and :539-552 (pre-fix, origin/main 5bd30e20da) (first spawn adds a 'running' ref before the child exists); runtime-temporal/src/activities.ts:6285-6288 (pre-fix, origin/main 5bd30e20da) (the spawn activity's spawnPersistentChild is a no-op), runtime-temporal/src/workflow-dispatch.ts:680 (pre-fix, origin/main 5bd30e20da) (the workflow starts the child workflow after the companion activity returns), runtime-temporal/src/activities.ts:7246 (pre-fix, origin/main 5bd30e20da) (the child session is created only in the child workflow's initializeAgentState) — so the window outlasts the spawn call; runtime-temporal/src/workflow-dispatch.ts:630 (pre-fix, origin/main 5bd30e20da) (a step's companion calls run concurrently via Promise.all, so a spawn and a sendMessage in the same step race directly); runtime-cloudflare/src/steps.ts:5363-5366 (pre-fix, origin/main a118f8e17e) (CF Workflows: spawnPersistentChild is likewise a no-op; the workflow body starts the child afterwards); store-memory/src/in-memory-state.ts:336-341 (pre-fix, origin/main 5bd30e20da) and store-postgres/src/state/postgres-state-store.ts:772-785 (pre-fix, origin/main 5bd30e20da) (appendMessages throws "Session not found"); store-cloudflare/src/d1-state.ts:256 (pre-fix, origin/main 5bd30e20da) with store-cloudflare/src/errors.ts:142 (pre-fix, origin/main 5bd30e20da) (StateNotFoundError, "State not found for sessionId: …"); store-redis/src/redis-state.ts:1165-1185 (pre-fix, origin/main 5bd30e20da) (appendMessages is one Lua script guarded on HEXISTS sessionId and throws "Session not found" — the RM-50 fix, MR !269); live: probe (batch-6 verification, store-redis dist on origin/main e37d676945 against Redis :6390): appendMessages to a never-created session threw 'Session not found'; afterwards sessionExists returned false, no key for the id existed, and loadState returned null. The pre-!269 probe (appendMessages did not throw, left a status-less hash that loadState rejected as InvalidStoredStateError, and a stale parent_message flag) no longer reproduces; inferred: runtime-js has the same window — the child session is created inside the detached child IIFE after its first await (runtime-js/src/run-loop.ts:5008-5016 (pre-fix, origin/main 5bd30e20da)) — but it is short; not observed; contrast: runtime-dbos/src/steps/execute-companion-tool.ts:556 (pre-fix, origin/main b3386d0a09) / :576 (pre-fix, origin/main 5bd30e20da) also adds the ref before createSession, but DBOS runs companion calls sequentially in the workflow body (runtime-dbos/src/workflows/shared.ts:2612-2627 (pre-fix, origin/main 5bd30e20da)) and routes sendMessage through its own handler (execute-companion-tool.ts:967 (pre-fix, origin/main 5bd30e20da)), so no DBOS sendMessage can run inside that window; the CF DO runtime also has its own sendMessage (runtime-cloudflare/src/do/persistent-companion-tools.ts), which reads the child DO's /status instead — neither is this defect; related: RM-50 (fixed by MR !269): its atomic appendMessages removed the Redis delivered:true / status-less-hash / stale-flag branch this record used to carry, which is why the title and severity were narrowed; runtime-js/src/run-loop.ts:4753-4761 (pre-fix, origin/main 5bd30e20da) (a failed append of the child's initial message is only warn-logged); severity S3 (narrowed from S2 in batch 6): on every store the model now sees a transient tool error for a child that is starting normally and can retry the send; nothing is corrupted or reported as delivered; related: peer-reported from a caller audit; the source and the Redis probe were re-checked in batch 6; related: fixed in code, before the repository's earliest reachable commit 1c5caeaa23 (2026-10-10), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): core/src/orchestration/companion-tool-dispatch.ts:1992-2016 (origin/main fb9150cc16) (the append runs only under a live child run; otherwise the answer is the typed child_not_running), which CF Workflows reaches through core's dispatcher (runtime-cloudflare/src/steps.ts:9728 (origin/main fb9150cc16)); on the CF DO the send's /resume reaches a resume entry that refuses a missing session typed before any write (runtime-js/src/js-agent-executor.ts:5068-5070 (origin/main fb9150cc16)), answered child_entry_failed. Open until its cell; related: fixed in code by a commit before 1c5caeaa23 (the repository's earliest reachable commit, 2026-10-10; no fix sha is reachable here), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/companion-tool-dispatch.ts:1905 (origin/main 56d564d073) (handleSendMessage reads the child's current run), core/src/orchestration/companion-tool-dispatch.ts:2001-2026 (origin/main 56d564d073) (the message is appended only under a live child run; a child with no live run that is not interrupted, a never-created session included, answers the typed { delivered: false, reason: 'child_not_running' }, never a store throw) | source, probe, peer-reported | partly-inferred | open |
| CP-76 | S1 | Temporal: resume() after a stop never reactivates the root stream that the stop ENDED, so the resumed run cannot write to it. execute() and retry() both reactivate an ended or failed stream before starting their workflow; resume() reactivates only when it rewinds a completed or failed session (from_checkpoint), so a resume of an interrupted session (any mode) only calls initStream, which does not. The resumed workflow's first stream write is rejected ("Cannot write to stream … in 'ended' state"), and — as peer-reported — the run fails with the executor's synthetic "Stream closed without terminal observation". The resumed turn does not complete, and the stream never shows it. (The separate with_message message drop on executor.resume(), the executor half of CP-31, is fixed on main.) This is the Temporal twin of CP-67 (CF Workflows), with its own fix site. | runtime-temporal/src/executor.ts:1547-1548 (pre-fix, origin/main a118f8e17e) (resume() reads getStreamInfo only for latestSequence), :1663-1676 (pre-fix, origin/main a118f8e17e) (starts the __resume-N workflow; !288 names it __resume-<runId>), :1702 (pre-fix, origin/main a118f8e17e) (then initStream only); the one reactivateStream call in resume() is :1491-1498 (pre-fix, origin/main a118f8e17e), inside the from_checkpoint rewind branch and only when the session was completed or failed, so an interrupted session never reaches it; contrast: runtime-temporal/src/executor.ts:445-454 (pre-fix, origin/main a118f8e17e) (execute() reactivates an ended / failed stream BEFORE startWorkflow, with a comment naming this exact throw) and :2054-2060 (pre-fix, origin/main a118f8e17e) (retry() does the same); runtime-temporal/src/activities.ts:5312-5315 (pre-fix, origin/main 5bd30e20da) (an interrupted root run ends its stream: endStream(streamId, undefined)); store-memory/src/in-memory-stream.ts:236-240 (pre-fix, origin/main 5bd30e20da) and store-redis/src/redis-stream.ts:653 (pre-fix, origin/main 5bd30e20da) (writers throw "Cannot write to stream ${streamId} in '${status}' state" for a non-active stream); runtime-temporal/src/executor.ts:1754-1759 (pre-fix, origin/main 5bd30e20da) (the handle's fallback when the observed stream closes with the session still active or paused: status 'failed', error 'Stream closed without terminal observation'); live: peer-reported, reproduced on origin/main (pre-fix, origin/main 0ffde4e65b) and on the LLM-history completeness branch that became !279 (f27a0c570e): packages/e2e/src/__tests__/cross-runtime-resume-matrix.integ.test.ts, on main since f27a0c570e, defines TEMPORAL_ENDED_STREAM_SKIP at :162-163 (citing CP-76) and skips the completion legs 'B2 C3/C4 (completion): resume(with_message) after a stop completes…' (:909-914) and 'B2 C7 (interrupted route, completion): resume(from_checkpoint) after a stop completes…' (:994-997) on every Temporal combo (isTemporal, :759); their '(input)' twins (:863, :959) still run on Temporal, so the resumed model input is right and only the completion is lost. Not re-run here (no Temporal server in this session); inferred: which the caller sees first depends on timing — the handle observes from the ended stream, so result() may resolve with the synthetic failure while the workflow is still retrying its LLM activity against the ended stream; the session status left behind was not established; related: CP-31 (its executor half fixed by !279; pre-fix, origin/main 0ffde4e65b, Temporal resume() ignored options.mode and the message and always sent newMessages: [], runtime-temporal/src/executor.ts:1342-1348 (pre-fix, origin/main 0ffde4e65b); after that fix with_message appends at runtime-temporal/src/executor.ts:1517-1519 (origin/main a118f8e17e) and from_checkpoint rewinds at :1434 (origin/main a118f8e17e), before the same newMessages: [] input at :1588-1594 (origin/main a118f8e17e)); CP-67 (the CF Workflows twin; its own related note flagged this gap as latent from source); CP-56 (a stop on the terminal step can end the run completed instead, and then resume() short-circuits, CP-60) — this gap is reached when the stop lands mid-run and leaves the session interrupted; severity S1: a lost user action — resume after an interrupt is the supported follow-up, and every such resume on Temporal fails to complete its turn (HITL paused resumes are not affected: they do not end the stream) | source, live-repro, peer-reported | partly-inferred | open |
| CP-77 | S2 | Cloudflare DO: in the remote-agents-cloudflare-do example, the producer's planner DO intermittently fails to continue after its two parallel sub-agent children finish. The planner calls its sub-agent tool twice in one step; the DO rewrites that tool onto DOStubTransport, so each child runs in its own DO. Both children make their final __finish__ LLM call, but the planner's next LLM call (the one carrying both children's results) is never made within the e2e test's 90 s window, so the consumer's snapshot never gains state.research and the test times out. Root cause unknown. | runtime-cloudflare/src/do/durable-object-agent-base.ts:1441-1475 (pre-fix, origin/main 5bd30e20da) (rewriteSubAgentTools: with subAgentNamespace set, every local sub-agent tool becomes createRemoteSubAgentTool over a DOStubTransport, one child DO per call, timeoutMs default 10 min); runtime-cloudflare/src/do/do-stub-transport.ts:81 (DOStubTransport, the parent-to-child-DO transport those calls use); live: examples/remote-agents-cloudflare-do/producer/src/agents/planner.ts:41-47 (threadTool = createSubAgentTool(ThreadResearcherAgent)); examples/remote-agents-cloudflare-do/producer/src/my-agent-server.ts:22 (subAgentNamespace set); the recorded planner response 878ab27f8ce90470 carries two subagent__thread-researcher calls in one step; live: examples/remote-agents-cloudflare-do/tests-e2e/cross-service.spec.ts:40-51 (polls /api/snapshot/<id> for state.research, timeout 90 s); playwright.config.ts sets retries: 0; live: replay fixtures (examples/remote-agents-cloudflare-do/test-utils/openai-fixtures/): d503428640967220 is a thread researcher's final call (its response is __finish__), and 6676bfdb88d8563e is the planner's call whose input ends with both thread outputs as tool results. Peer-reported failing runs' replay logs stop after the 9th LLM call (d503428640967220) and never request 6676bfdb88d8563e; live: peer-reported CI sightings: jobs 16755700166 (MR !262), 16756092320 (MR !268), 16756390704 (MR !259); not re-checked here (no GitLab access in this session); inferred: candidate causes, none confirmed: (a) a missed child-completion or stream-end wake in core/src/orchestration/remote-sub-agent-dispatch.ts when two DOStubTransport children end close together; (b) a CP-70-like DO wake strand of the planner DO; (c) interference from a concurrent /api/child-snapshot read; inferred: the rewritten tool times out after 10 minutes by default, so the 90 s test window cannot tell a permanent hang from a delay longer than 90 s; related: CP-70 (DO wake silently strands a paused session); FU-REMOTE-06 (fold read timeouts bound the parent, not the request); severity S2 (provisional): a parent that never continues after its children complete is a silent stall with no error; S2 rather than S1 until a repro shows it is unbounded and not example-specific | source, peer-reported | unverified | open |
| CP-78 | S2 | Remote sub-agents: a 409 ALREADY_COMPLETED from /start reaches the parent as a plain Error, and the parent FAILS a delegation that succeeded. HttpRemoteAgentTransport.start() retries its POST on network errors and 5xx responses. If the first POST started the child but its response was lost, and the child finished before the retry (1 s, then 2 s, then 4 s later by default), the retry gets ALREADY_COMPLETED. Both transports map only ALREADY_RUNNING, NOT_FOUND and FEATURE_UNAVAILABLE to typed errors, and every remote dispatch site attaches only on RemoteAgentAlreadyRunningError and rethrows anything else. The sub-agent call therefore fails with "Remote agent start failed (409): …" (failureReason 'transport-error') although the child completed, and its output never reaches the parent. Fixed on every runtime by b6b3eeb141 and 9a27ac7da3 (D-2 MR 1 and MR 2): a /start carries a request key and is never blindly retried, a lost answer is re-sent with the same key, and the agent-server producer answers a keyed repeat from the run that holds it. | core/src/transport/http-transport.ts:129-143 (start() → fetchWithRetry, then toProtocolError on !ok), :330-369 (fetchWithRetry: a thrown fetch error is retried at :360-364, a 5xx at :353-357, a 4xx is returned at :348-351; maxRetries 3 and retryBaseDelayMs 1000 by default, :126-127), :317-329 (pre-fix, origin/main 137b07d57a) (toProtocolError maps only ALREADY_RUNNING / NOT_FOUND / FEATURE_UNAVAILABLE; ALREADY_COMPLETED falls through to a generic Error); runtime-cloudflare/src/do/do-stub-transport.ts:297-307 (pre-fix, origin/main 137b07d57a) (the same three-code mapping; DOStubTransport.start at :110 does not retry, so its gap is latent); agent-server/src/agent-server.ts:386-400 (startAgent throws ALREADY_COMPLETED for a session that is 'completed' or 'failed' — so a child that FAILED quickly is reported as a transport-error with the 409 body instead of remote-failed with the child's own error); agent-server/src/http/handler.ts:862-863 (pre-fix, origin/main 5bd30e20da) (mapped to HTTP 409); core/src/orchestration/remote-sub-agent-dispatch.ts:482-493 (start catch: attach only on RemoteAgentAlreadyRunningError, else rethrow) and :804-811 (the outer catch returns kind failed, failureReason transport-error, with the error message); runtime-js/src/execution/remote-subagent-executor.ts:373-384, runtime-dbos/src/steps/execute-remote-subagent.ts:438-449 (pre-fix, origin/main 5bd30e20da), runtime-cloudflare/src/steps.ts:3254-3266 (pre-fix, origin/main a118f8e17e) (the same attach-or-rethrow start catch in each runtime-specific dispatch); live: probe (this session, core dist): HttpRemoteAgentTransport.start with a fetch that throws 'fetch failed' on the first call and returns 409 {code:'ALREADY_COMPLETED'} on the retry made 2 calls and threw a plain Error (not RemoteAgentAlreadyRunningError): 'Remote agent start failed (409): {"error":"Session s1 already completed","code":"ALREADY_COMPLETED"}'; inferred: the end-to-end effect (the parent's sub-agent tool result reports a failure for a child that completed) follows from the dispatch code above; not reproduced against a live producer. The pre-start getStatus check (remote-sub-agent-dispatch.ts:402-434) handles a child that is already completed on re-entry, but not one that completes during start()'s own retries; contrast: a Cloudflare DO producer does not answer ALREADY_COMPLETED at all (the only ALREADY_* code runtime-cloudflare/src/do/durable-object-agent-base.ts emits is ALREADY_RUNNING, :1590, :1676, :1687 (origin/main 5bd30e20da)); what a retried /start does to a completed DO session was not assessed; related: CP-07 (HelixChatTransport ignores /start responses including ALREADY_COMPLETED — a different component); docs/guide/cross-service-remote-agents.md "At-least-once POSTs" (documents the ALREADY_RUNNING → attach rule, which does not cover this case); severity S2: needs a lost /start response plus a child that finishes within the retry backoff, but the result is a wrong failure reported to the model for a completed delegation, and the parent may repeat the work; related: WA MR1 (!340) moves the three-code mapping into core remoteProtocolError (core/src/transport/protocol-error.ts:47-81), shared by both transports: an ALREADY_COMPLETED envelope that carries an errorCode becomes a typed HelixError (:67-75), one without it a generic Error (:80). Neither is a RemoteAgentAlreadyRunningError, so dispatch still rethrows and the parent still fails a delegation that succeeded; the defect stands. Also since WA MR1, a CF DO producer does answer ALREADY_COMPLETED, for a /subagent resume of a completed session (runtime-cloudflare/src/do/drive/wire-answer.ts:35 and :119), so the pinned contrast above holds only for its sha; agent-server/src/agent-server.ts:380-395 (fix evidence, D-2 MR 1, origin/main 9a27ac7da3) (startAgent: a keyed /start is looked up with getRunByRequestKey before the completed / failed refusal, so a repeat of a delegation whose run already finished is answered from its key: attached to that run, read to its end frame) with :519-526 (origin/main 9a27ac7da3) (the same lookup on /resume); the consumer half is D-2 MR 2 (!404): core remote-sub-agent-dispatch.ts:241-269 (fix evidence, origin/main 9a27ac7da3) (startRemoteChild sends the key and re-sends a lost answer with it); related: docs/guide/remote-agents.md (origin/main 9a27ac7da3), the ALREADY_COMPLETED (CP-78) warning: a keyed repeat of a finished run is answered attached; 409 ALREADY_COMPLETED with state_request_already_completed means a later run took the child over, a typed non-retryable failure. The record stays open until a conformance cell witnesses it (an agent-server producer that loses the first /start answer); related: fixed in code by b6b3eeb141 (D-2 MR 1, the producer) and 9a27ac7da3 (D-2 MR 2, !404, the consumer), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): agent-server/src/agent-server.ts:385-397 (origin/main 56d564d073) (a keyed /start whose key a run holds is answered from that run, attached or read to its end, before the completed / failed refusal at :399-400 (origin/main 56d564d073)), agent-server/src/agent-server.ts:522-530 (origin/main 56d564d073) (the same lookup on /resume), core/src/transport/http-transport.ts:330-351 (origin/main 56d564d073) (a /start is posted once with its request key header, and a lost answer throws RemoteEntryOutcomeUnknownError, never a blind retry), core/src/orchestration/remote-sub-agent-dispatch.ts:267-280 (origin/main 56d564d073) (startRemoteChild re-sends a lost answer with the same key); every runtime calls startRemoteChild: runtime-js/src/execution/remote-subagent-executor.ts:542 (origin/main 56d564d073), runtime-cloudflare/src/steps.ts:6240 (origin/main 56d564d073), runtime-dbos/src/steps/execute-remote-subagent.ts:509 (origin/main 56d564d073), and Temporal through core/src/orchestration/remote-sub-agent-dispatch.ts:617 (origin/main 56d564d073) | source, probe, peer-reported | partly-inferred | open |
| CP-79 | S1 | DBOS standard mode: an interrupt() that lands before the workflow's in-body appendMessages step silently drops that turn's user input. DBOS execute() passes the input only as a workflow argument, and the workflow appends it several durable steps in. interrupt() on the handle execute() returned calls DBOS.cancelWorkflow, so the next pre-append step throws and the append never runs. The interrupt path then CASes the session active → interrupted. The stored session ends interrupted with no user message and no error / errorDetail (the same handle's result() reports failed, CP-57, which does not name the lost input), and nothing re-supplies the input: resume() starts the workflow with input: undefined, so the LLM answers with only the system prompt (plus prior turns) and the run reads completed. Fresh sessions and continued turns are both affected. Only a handle that holds the workflow id cancels: a getHandle() / ReconnectHandle interrupt only sets the flag, so the append runs and the input survives. JS appends the input inside execute() before it returns the handle. Temporal and CF Workflows interrupt through a flag or signal that never stops the input append. | runtime-dbos/src/lifecycle/execute.ts:434-440 (pre-fix, origin/main 5bd30e20da) (createSession) / :464-468 (continuation CAS to active) and :689-703 (pre-fix, origin/main 5bd30e20da) (DBOS.startWorkflow receives input; execute() persists no user message before it returns the handle); runtime-dbos/src/workflows/standard-workflow.ts:490 (pre-fix, origin/main 5bd30e20da) (beginRun step) and :528 (pre-fix, origin/main 5bd30e20da) (runAgentLoopOneTurn); runtime-dbos/src/workflows/shared.ts:1092 (pre-fix, origin/main 5bd30e20da) (cleanupOrphanedStaging), :1099 (pre-fix, origin/main 5bd30e20da) (loadState), :1123 (pre-fix, origin/main 5bd30e20da) (initStream), :1194 (pre-fix, origin/main 5bd30e20da) (getMessageCount) and only then :1217-1219 (pre-fix, origin/main 5bd30e20da) (appendMessages of messagesToAppend, the history prefix + user input, built at :1216 (pre-fix, origin/main 5bd30e20da)) — five-plus durable steps before the input is durable; runtime-dbos/src/lifecycle/interrupt.ts:67 (pre-fix, origin/main 5bd30e20da) (setInterruptFlag), :86 (pre-fix, origin/main 5bd30e20da) (DBOS.cancelWorkflow: every later @DBOS.step throws, so the append is never reached), :115-116 (pre-fix, origin/main 5bd30e20da) (after 500 ms, CAS active → interrupted from outside the workflow). The graceful in-loop flag check at runtime-dbos/src/workflows/shared.ts:1541 (pre-fix, origin/main 5bd30e20da) sits AFTER the append and is never reached; runtime-dbos/src/handles/active-handle.ts:587-591 (pre-fix, origin/main 5bd30e20da) (the execute() handle passes its workflowId to interruptImpl, so it takes the cancel path); runtime-dbos/src/lifecycle/get-handle.ts:88 and runtime-dbos/src/handles/reconnect-handle.ts:135-140 (pre-fix, origin/main 5bd30e20da) (a getHandle() handle passes null, the flag-only path at interrupt.ts:125-134, so the append runs and the input survives); agent-server/src/agent-server.ts:906-918 (pre-fix, origin/main 5bd30e20da) (interruptAgent writes the flag, then calls the same-replica activeHandles handle, so a UI stop hits this only when it lands on the replica that started the run); runtime-dbos/src/lifecycle/resume.ts:619-622 (pre-fix, origin/main 5bd30e20da) (only with_message appends anything) and :733-750 (pre-fix, origin/main 5bd30e20da) (the resumed workflow gets input: undefined, isResume: true, so the dropped input is never re-appended); live: live-repro (this verification, RUNTIME_DBOS_INTEG=1, private Postgres DB on :5433, runtime-dbos integ context from origin/main, no product change): (a) with no hold, execute() then handle.interrupt() straight away: 10 of 10 runs ended interrupted with 0 messages stored; (b) with a hold on getMessageCount, a fresh session ended interrupted with an empty transcript and 0 LLM calls. resume() then completed and the LLM input was only [system]. The transcript became a lone assistant message with no user message; (c) with a hold, turn 2 of a completed session ended interrupted with only turn 1 [user TURN-1, assistant] stored, so the turn-2 input was gone. An independent review re-run (same setup, private DB) reproduced it: fresh unheld 5/5, continued turns unheld 3/3, and resume() answering [system] only and reporting completed; contrast: runtime-js/src/js-agent-executor.ts:4352-4382 (pre-fix, origin/main 5bd30e20da) (execute() awaits saveAgentState + appendMessages before the handle exists, so no interrupt can precede the append; a first turn keeps its input); contrast: runtime-temporal/src/executor.ts:818-821 (pre-fix, origin/main a118f8e17e) (interrupt is a workflow signal, not a cancel) and runtime-temporal/src/workflow.ts:341-344 (pre-fix, origin/main a118f8e17e) (the handler only sets a flag), so the initializeAgentState activity still appends the input (runtime-temporal/src/activities.ts:7449-7451 (pre-fix, origin/main 5bd30e20da)) before the loop observes the interrupt; contrast: runtime-cloudflare/src/executor.ts:1681-1701 (pre-fix, origin/main a118f8e17e) (CF Workflows interrupt = durable flag + best-effort event, never a terminate), so the workflow init step still appends the input; the CF DO embeds the JS executor (append before the handle); related: CP-17 (DBOS interrupt calls cancelWorkflow immediately, so the graceful path is unreachable — the root mechanism; a graceful interrupt would let the append run before the flag check at runtime-dbos/src/workflows/shared.ts:1541 (pre-fix, origin/main 5bd30e20da), so a CP-17 fix likely fixes this too, but the input loss is a distinct S1 observable, as with CP-57); CP-57 (the same cancel makes ActiveHandle.result() report failed while the stored session reads interrupted); RM-42 (runtime-js deletes a continued turn's input on an interrupt during the first step — a different mechanism, the orphan-cleanup truncation); CP-19 (stale durable interrupt flag); RM-43 / RM-41 (weaker: CF Workflows appends the turn input outside the turn-start boundaries); live: first peer-reported. The debug report the brief cites (debug-interrupt-resume-dbos-report.md) could not be found on disk, so this verification re-derived the claim from source and reproduced it independently (the live-repro item above); severity S1: the user's own message is silently lost and the stored session carries no error, which a stop through the execute() handle right after send reliably triggers (10/10 unheld; through agent-server only when the stop lands on the executing replica). A later resume() cannot recover it, and it produces an answer to no question that reads completed; runtime-dbos/src/lifecycle/execute.ts:334-368 (origin/main 638aaed36c) (fix, present at main's root 1c5caeaa23 with S5's guarded run-start entry: execute() waits for its ENTRY's outcome, awaitExecutorEntry, so the handle exists only after the run-start commit that carries the turn's input); runtime-dbos/src/workflows/shared.ts:1586-1617 (origin/main 638aaed36c) (fix: startRunCommit writes the entry's user messages in the run-start commit itself, so no later step has to append them) and runtime-dbos/src/lifecycle/interrupt.ts:216-289 (origin/main 638aaed36c) (the stop is core stopRun's run-scoped request, never a cancel before the entry committed); conformance: fixed. stop.input-survives-immediate-stop (executor, WA's !792 at aa0e03eaae) passes 3 of 3 on all 8 executor lanes in WA's drv-cp79 (46 checks, 0 failed): a stop sent straight after execute() keeps the turn's input, fresh and continued. There is no red phase, since the fix predates the cell; the mutation mlate.py (DBOS appending the input in a later step, with the turn's own stop held until DBOS records the workflow CANCELLED) fails exactly .fresh-input-persisted and .continued-input-persisted on dbos-postgres. The other lanes are guards | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-80 | S2 | Temporal: the handle resume() returns settles result() on a terminal LLM error before the session is persisted as failed. The v7 resume handle projects stream chunks, not the workflow result, and treats the first error chunk as terminal. runLLMStep's onError callback writes that chunk inside the LLM activity. The workflow runs commitStep and then persistTerminalState(failed) only after the activity returns. So result() reports failed while the stored session still reads 'active'. A caller that acts on the result straight away gets a stale read. The documented recovery, retry(), then throws "Cannot retry: agent status is 'running'" because its ['failed'] CAS fails. The window is inherent: it was observed with no injected delay. The completed path is not affected, because the output chunk is written by persistTerminalState after its saveState. execute() and retry() handles are not affected either, because they await the workflow result. | runtime-temporal/src/executor.ts:1710-1716 (pre-fix, origin/main a118f8e17e) (resume() returns createTemporalAgentHandleV7, the stream-projecting handle), contrasted with :1158-1172 (as before WA MR2, origin/main cae966c0a7) (execute() and retry() use createActiveHandle, which awaits workflowHandle.result(), so the workflow has already persisted); runtime-temporal/src/handle.ts:126-135 (pre-fix, origin/main 5bd30e20da) (result() resolves on the first chunk projectChunk accepts) and :266-277 (pre-fix, origin/main 5bd30e20da) (any 'error' chunk projects to status 'failed'); runtime-temporal/src/activities.ts:3050-3062 (pre-fix, origin/main 5bd30e20da) (runLLMStep's onError writes an 'error' chunk, recoverable: false, from inside the LLM activity); runtime-temporal/src/workflow.ts:1347 (pre-fix, origin/main a118f8e17e) (commitStep runs after the activity returns) and :1565-1600 (pre-fix, origin/main a118f8e17e) (only then does the terminal-failed branch call persistTerminalState(failed) with skipErrorEmit: true, because the chunk was already written); runtime-temporal/src/activities.ts:5224 (pre-fix, origin/main 5bd30e20da) (persistTerminalState saveState is the write that makes the session read failed); runtime-temporal/src/executor.ts:1949-1960 (pre-fix, origin/main a118f8e17e) (retry() CASes ['failed'] → 'active' and throws while the session still reads active); contrast: runtime-temporal/src/activities.ts:5224 (pre-fix, origin/main 5bd30e20da) then :5261-5280 (pre-fix, origin/main 5bd30e20da) (on completion persistTerminalState saves status completed BEFORE it writes the output chunk, so a completed resume() result is never ahead of the store; the peer claim that runPhase2FinishWith emits the output chunk early is refuted: activities.ts:5269 (pre-fix, origin/main 5bd30e20da) is the only output-chunk writer in runtime-temporal). The order still holds: persistTerminalState commits the terminal state (commitTerminalState) and only then runs finalizeTerminalRun, runtime-temporal/src/activities.ts:6927-6931, whose output write at :7361-7371 is still the only output-chunk writer; live: live-repro (this verification, HELIX_TEMPORAL_BACKEND=test-server, runtime-temporal integ harness with InMemoryStateStore, origin/main 3560d6c72a, no product change): client-tool suspend → submitToolResult → resume(), the resumed LLM step returns a non-recoverable error. result() returned 'failed' while loadState() read straight after returned status 'active' (2 of 2 runs, with no delay injected and with an 800 ms hold on the terminal saveState). The same session read 'failed' 1.5 s later. The completed variant (resumed step calls __finish__) read 'completed' at resolve time in both runs. An immediate executor.retry() after the failed result threw "Cannot retry: agent status is 'running'"; inferred: an adapter that returns an error step with shouldStop: false (a recoverable error; the core MockLLMAdapter can, llm-vercel always returns shouldStop: true) also writes the error chunk, so the resume handle would report 'failed' for a run that continues. Not reproduced; related: CP-57 (DBOS result() reports failed while the store reads interrupted — the same result/store disagreement class, different mechanism); CP-65 (the same Temporal resume handle never settles when nothing new is drained); DI-24 (Temporal kept the classified error detail only on the stream chunk, not in the persisted row, until Completion strategies MR 2 fixed that half); the batch-5 peer candidate #3 prompted the check, but its mechanism is refuted (see contrast) and this failed-path variant was found and reproduced independently, so peer-reported is not claimed; severity S2: the terminal status is right and converges within the persist latency, but a caller that reads the session or calls the documented retry() as soon as result() says failed gets a stale 'active' read or a thrown retry, with no signal to wait | source, live-repro | verified | open |
| CP-81 | S1 | Duplicate of CP-52. CF DO: a client-tool submit that arrives while the suspending run is still inside onAgentSuspended is dropped. persistSuspension has already written the session paused, so /status and the harness report the suspension while the hook is still running. /submit-tool-result stores the result and calls ensureExecutionContinues, which returns early because isExecuting is still true. The run then exits suspended, the scenario's explicit resume() right after the submit did not rescue it in the repro, and nothing wakes the session later: the watchdog recovers only active sessions. The session stays paused with the result stored and never consumed. | core/src/orchestration/step-iterator.ts:1754 (pre-fix, origin/main a118f8e17e) (persistSuspension) then :1774 (pre-fix, origin/main a118f8e17e) (onAgentSuspended is awaited after it, so the session reads paused for the whole hook); runtime-cloudflare/src/do/durable-object-agent-base.ts:3107-3109 (pre-fix, origin/main 5bd30e20da) (handleSubmitToolResult → ensureExecutionContinues) and :644-653 (pre-fix, origin/main 5bd30e20da) (returns early on _executionState.isExecuting: 'the live run will observe the state change', but the run has already decided to suspend); runtime-cloudflare/src/do/durable-object-agent-base.ts:1061-1073 (pre-fix, origin/main 5bd30e20da) (recoverStrandedExecution returns unless the session is 'active', so a stranded 'paused' session with a submitted result is never recovered); live: live-repro (batch-6 verification, origin/main e37d676945, no product change): the lifecycle-hooks-parity scenario 'tracingContext set by onAgentSuspended persists through resume → completion' copied into a temp cf-do-d1 test (e2e workerd pool, Miniflare) with a 1 s delay added to onAgentSuspended failed 4 of 4 runs with the CI signature (expected 'suspended_client_tool' to be 'completed'); the session read paused with tc-1 submittedAt set and the result stored, and still read paused with tc-1 pending 5 s later. Without the delay it passed 3 of 3; contrast: runtime-js (probe, batch-6 verification): submit + resume({ mode: 'continue' }) issued while a 1 s onAgentSuspended hook was still running were accepted and the run completed with the real result, because JS continuation is caller-driven and the resume CAS takes the already-paused session. Temporal (caller-driven resume whose CAS accepts paused, interrupted and active) and DBOS (DBOS.send is queued durably to the waiting call's recv topic) have no such window, from source; related: CP-52 (the same drop, same mechanism and fix site: ensureExecutionContinues returns early on isExecuting while the run is tearing down a suspension; CP-52 observed it through the ai-sdk host on a fresh session's first client-tool round); peer-reported (2026-09-27) from CI jobs 16758514869, 16759542448, 16760323616, 16764206430 and main pipelines 2887177079 / 2886530232 (lifecycle-hooks-parity.cf.test.ts, cf-do-d1); severity S1: lost user action (the submitted client-tool result is never consumed and the session is stranded paused) | source, live-repro, peer-reported | verified | withdrawn — Duplicate of CP-52: the same submit-during-suspension-teardown drop, with the same mechanism (ensureExecutionContinues returns early on isExecuting after the run has decided to suspend) and fix site. The onAgentSuspended hook only widens CP-52's teardown window. This record's executor-tier repro, the watchdog gap and the CI signature were moved to CP-52's evidence; track and fix the defect there. |
| CP-82 | S1 | A partial client-tool batch continues the paused run, and the unanswered calls get a fabricated 'runtime restarted' failure. When one step emits two client-tool calls (c1, c2) and only c1 is submitted, the CF Durable Object's /submit-tool-result wakes the run itself (ensureExecutionContinues → executor.resume({ mode: 'continue' })), and runtime-js's resume() does the same when a caller resumes after a partial submit. resume() has no 'every pending call is answered' gate. Its orphan drain, written for calls stranded by a crashed process, treats every pending entry with no submission as lost and synthesizes runtime_restarted for c2. The model then gets c1's real answer plus an error telling it the runtime restarted, and the run completes. The client's real c2 answer, when it arrives, is refused as already_completed, which the ai-sdk handler treats as success, so the user's action is dropped with no error anywhere. The ai-sdk handler resumes after every tool-result POST without checking for other pending calls, so js-chat-host is exposed the same way. Temporal and DBOS keep waiting for c2 and use its real answer. | runtime-cloudflare/src/do/durable-object-agent-base.ts:3107-3109 (pre-fix, origin/main 5bd30e20da) (handleSubmitToolResult wakes the DO after any locally-routed accepted submit) and :662-790 (pre-fix, origin/main 5bd30e20da) (ensureExecutionContinuesImpl: only a resumable-status check, then executor.resume(agent, sessionId, { mode: "continue" }) at :779; nothing checks whether other pending calls are still unanswered); runtime-js/src/js-agent-executor.ts:3304 (pre-fix, origin/main 5bd30e20da) (resume() drains orphan client-tool calls before the loop) and runtime-js/src/client-tool-resolver.ts:898-903 (pre-fix, origin/main 5bd30e20da) (drainOrphans' last branch: an entry with neither submittedResult nor submittedError yields outcome { ok: false, error: 'runtime_restarted' }, whatever its age or deadline); ai-sdk/src/handler/handle-chat-stream.ts:659-719 (pre-fix, origin/main 5bd30e20da) (the tool-result path submits each intent in the POST; 'accepted' and 'already_completed' are both success at :712-713) then :748 (pre-fix, origin/main 5bd30e20da) (callExecutorResume mode continue, with no check for other pending calls); live: live-repro (batch-6 verification, origin/main e37d676945, no product change) on the CF DO: a temp cf-do-d1 test in the e2e workerd pool (Miniflare, no Docker). One step emits c1 and c2 (client tools); submitToolResult(c1) returned accepted; 4 s later the session read completed with no pending calls after 2 LLM calls, and the second model input was [system, user, assistant(c1, c2), tool c1 {"a":"ONE"}, tool c2 {"error":"The 'ask' could not be completed because the runtime restarted while it was pending. …"}]. A late submitToolResult(c2) returned already_completed; live: probe (batch-6 verification, unit level, runtime-js JSAgentExecutor + InMemoryStateStore): the same batch, submitToolResult(c1), then resume({ mode: 'continue' }): result completed, pending cleared, persisted [user, assistant(c1, c2), tool c1, tool c2 runtime-restarted error, assistant final]; a late submitToolResult(c2) returned already_completed; contrast: live-repro (batch-6 verification) on Temporal (HELIX_TEMPORAL_BACKEND=test-server, runtime-temporal integ harness): after submit(c1) + resume(), the session stayed waiting with c2 pending and 1 LLM call; submit(c2) + resume() then completed with both real answers in the transcript (runtime-temporal/src/activities.ts:3718 (pre-fix, origin/main 5bd30e20da), an unsubmitted entry stays pending; the resumed handle itself never settled while c2 was pending, the CP-65 mechanism). On DBOS (RUNTIME_DBOS_INTEG=1, private Postgres DB on :5860, Redis db 14 on :6826) the workflow waits on a per-call DBOS.recv (runtime-dbos/src/client-tool-resolver.ts:74 (pre-fix, origin/main 5bd30e20da)): after submit(c1) it stayed active with c2 pending, and submit(c2) completed the run with both real answers; inferred: (unverified) even a full batch from the stock client may race on the DO. The Helix chat transport sends one /submit-tool-result request per tool output, concurrently (ai-sdk/src/transport/helix-chat-transport.ts:228-236 (pre-fix, origin/main 5bd30e20da), Promise.all), each accepted submit wakes the DO (runtime-cloudflare/src/do/durable-object-agent-base.ts:3107-3109 (pre-fix, origin/main 5bd30e20da)), and the wake runs executor.resume detached (:777-790 (pre-fix, origin/main 5bd30e20da)), so c1's wake-and-orphan-drain can run before c2's submit lands. A stock useChat only auto-sends once every tool call of the message has a result, so a one-by-one client (or a direct /submit-tool-result caller) hits it deterministically; whether the concurrent full-batch path loses the race in practice has not been reproduced; related: CP-52 (a submit landing in the DO suspension teardown is dropped: the opposite failure, the run never wakes); CP-65 (the Temporal resume handle does not settle while a call is still pending); peer-reported (2026-09-27, with a reviewer probe on its branch); the DO and JS behaviour were reproduced independently on main here; the fix is planned in the snapshot-truth MR, cell submit.partial-batch; severity S1: lost user action. The client's real answer to c2 is discarded, the model is told the runtime restarted, and the run completes as a success; the late submit is acknowledged as already_completed, so nothing surfaces the loss; conformance: submit.partial-batch-host (cf-do, js-chat-host) and submit.partial-batch on js-* — .stays-paused, .c2-still-pending, .nothing-synthesized and .completes-after-c2 fail on origin/main 5bd30e20da and pass with !288; runtime-js/src/js-agent-executor.ts:3266-3274 (origin/main fdc56520a5) (fix evidence, !288: continue / with_message on a paused session continue only once every awaited call has its result; refused typed before any write) and runtime-cloudflare/src/do/durable-object-agent-base.ts:2976-2980 (pre-fix, origin/main 487090d3fb) (fix evidence, !288, re-pointed for WA MR1 (!340), which moves the gate: a submit wakes the DO only once the whole batch is answered, shouldWakeAfterSubmit, R42) and runtime-js/src/js-agent-executor.ts:1135-1166 (origin/main 47dd092fb9) (assertClientToolBatchComplete, called at :5209 (origin/main e3f03e8196), which the DO's /resume now reaches through the runtime-js resume it drives: a partial batch is refused typed, state_not_resumable, before any write) | source, live-repro, peer-reported | verified | fixed (5 cells) |
| CP-83 | S2 | resume() against a live run was accepted, and it took the session away from that run. Until !279 (pre-fix, origin/main 0ffde4e65b) this held on Temporal (every mode), CF Workflows executor.resume() (continue / with_message) and runtime-js (from_checkpoint), and until !286 it also held on CF Workflows handle.resume(). Temporal (pre-!279): the resume CAS accepted active as a source state, and nothing distinguishes an active session that is awaiting client-tool submissions from one whose run is still executing. Accepting active is deliberate and must stay: in the v7 stateless model a session awaiting a client tool can read active (the SF-D2 comment at runtime-temporal/src/executor.ts:2087-2092), and a Temporal session re-suspended on a partial client-tool batch reads active while the remaining calls are pending, so a later resume() on it is the only way to finish the batch (CP-82's Temporal contrast, CP-65). The defect is that an active session with no pending client-tool calls, i.e. a live run, is accepted too: resume() starts a __resume-N workflow (__resume-<runId> since !288) beside the live one. In the probe the live execute() run ended failed, the resumed run completed, and the live run's answer never reached the transcript. runtime-js (pre-!279): from_checkpoint skipped the running-session pre-check the other modes have, so the same happened there: the live run is superseded and fails, the resumed run fails on the ended stream, and the session ends failed. CF Workflows: executor.resume() was the same before !279. handle.resume() went through the private resumeWorkflow, whose CAS stayed unguarded after !279 and accepted the live run; it also truncated the live turn's uncheckpointed messages first (CP-35) and started a second instance beside the running one. !286 deleted resumeWorkflow, and every handle's resume() now calls executor.resume(), which refuses a live run. The reference behaviour is DBOS's guard, which accepts active only when client-tool calls are pending and otherwise throws AgentAlreadyRunningError; the CF DO also refuses (409). The LLM-history completeness fix (!279) adds that guard to Temporal, CF Workflows executor.resume() and runtime-js, and !286 routes CF Workflows handle.resume() through executor.resume(), so no runtime is known to still accept a live run. lifecycle.resume-while-running proves the guard on every lane that had the defect (see the conformance line). | runtime-temporal/src/executor.ts:1253-1264 (pre-fix, origin/main 0ffde4e65b) (the CAS accepts 'active' on purpose, because a session awaiting a client-tool submission can read 'active') and :1265-1275 (pre-fix, origin/main 0ffde4e65b) (compareAndSetStatus(['paused', 'interrupted', 'active'] → 'active'): no check for pending client-tool calls or a live workflow, so an 'active' session with none pending, i.e. a running one, is accepted too) and :1364 (pre-fix, origin/main 0ffde4e65b) (a new __resume-N workflow is started (the pre-!288 id scheme; !288 names it __resume-<runId>); options.mode is never read, CP-31, so every mode behaves the same); contrast: reference behaviour, runtime-dbos/src/lifecycle/resume.ts:194-366 (pre-fix, origin/main 5bd30e20da) (the DBOS guard: when the CAS finds 'active', it loads the session and accepts only if pendingClientToolCalls is non-empty; 'Active session with no pending client tools — genuinely running' throws AgentAlreadyRunningError at :366). A fix that drops 'active' from the Temporal CAS wholesale would strand the partial-batch continuation (CP-82 contrast, CP-65); the discriminator is pending client-tool calls (or a live workflow), not the status alone; runtime-js/src/js-agent-executor.ts:2098-2126 (pre-fix, origin/main 0ffde4e65b) (from_checkpoint loads the checkpoint with no status check; the running/completed/failed pre-checks at :2128-2142 apply only to the other modes) and :2304-2325 (pre-fix, origin/main 0ffde4e65b) (the from_checkpoint save sets state.version = the version just loaded, so it is no guard against a live run: a version pin only arbitrates between concurrent resumers, since a resumer reads the live run's current version); runtime-cloudflare/src/executor.ts:649-654 (pre-fix, origin/main 0ffde4e65b) (the resume() pre-checks refuse only a completed or failed session) and :756-760 (pre-fix, origin/main 0ffde4e65b) (the continue / with_message CAS accepts 'active' with no pending-call check); :716-743 (pre-fix, origin/main 0ffde4e65b) (from_checkpoint saves the restored state pinned to the version it has just loaded, which is no guard against a live run, as on runtime-js; inferred, not probed); :1380-1384 (pre-fix, origin/main 0ffde4e65b) (handle.resume() → resumeWorkflow: the same unguarded CAS); live: live-repro (batch-6 verification, origin/main e37d676945, no product change; HELIX_TEMPORAL_BACKEND=test-server, runtime-temporal integ harness, InMemoryStateStore): an agent whose LLM call is held, so the run is live and no client-tool call is pending; while the execute() run is active, resume({ mode: 'continue' }) and resume({ mode: 'with_message' }) both returned a handle without throwing. After the hold was released the original execute() result was failed, the resumed handle completed, 2 LLM calls were made, and the persisted transcript was [user hi, assistant b]: the live run's answer is not in it; live: probe (batch-6 verification, unit level: runtime-js JSAgentExecutor + InMemoryStateStore): with the execute() run's LLM call held, resume({ mode: 'from_checkpoint', checkpointId: <the latest checkpoint> }) returned a handle. The live run failed with "Executor superseded for session … - another executor has taken over", the resumed run failed with "Cannot write to stream … in 'ended' state", and the session ended failed with only [user hi] persisted; live: [pre-fix, origin/main 3560d6c72a] probe (batch-5 verification, moved here from CP-35; unit level: CloudflareAgentExecutor + a real InMemoryStateStore + a mock workflow binding): session active with [user TURN-1, assistant] under a messageCount-2 checkpoint and turn 2 input user TURN-2 appended (first step in flight); resume({mode:"continue"}) succeeded, deleted TURN-2 ([TURN-1, answer-1, TURN-2] → [TURN-1, answer-1]) and created agent__a__s1__resume__1 beside the live agent__a__s1. With !279, executor.resume() rejects this case with AgentAlreadyRunningError before any write; contrast: runtime-js continue / with_message / with_confirmation on a running session throw (untyped on main, DI-31; typed with !279); DBOS (RUNTIME_DBOS_INTEG=1 probe, batch-6 verification) threw AgentAlreadyRunningError for continue, with_message and from_checkpoint on a running session with no pending client tools (runtime-dbos/src/lifecycle/resume.ts:366 (pre-fix, origin/main 5bd30e20da)); the CF DO /resume returns 409 while isExecuting (runtime-cloudflare/src/do/durable-object-agent-base.ts:2050-2051 (pre-fix, origin/main 5bd30e20da)); related: CP-35 (CF Workflows resume() truncates before an unpinned CAS, so two concurrent resumes of a resumable session both win: the concurrency half of the same code, closed on executor.resume() by !279); CP-31 (Temporal resume ignores options.mode); CP-60 (Temporal resume on a completed session returns the old result); CP-65 and CP-82 (the Temporal partial-batch path that legitimately resumes an active session); DI-31 (the refusals that do happen on runtime-js are untyped); found while checking a peer report of the JS untyped error (batch-6 item 13), not itself peer-reported; related: MR !279 addresses this record on every affected runtime, with red-first tests (fix-evidence lines below): its from_checkpoint liveness gate (a typed AgentAlreadyRunningError on runtime-js, CF Workflows and Temporal) and, in the same MR, Temporal and CF Workflows resume() in continue / with_message (and with_confirmation on Temporal) reject an active session with no pending client-tool calls (a live run) before any write, as runtime-js and DBOS already did, while an active session with pending calls stays resumable. The record stayed open until the lifecycle.resume-while-running conformance cell proved the fix (the conformance line below); runtime-temporal/src/executor.ts:1307-1322 (origin/main a118f8e17e) (fix evidence, !279: from_checkpoint calls assertRewindable at :1308; every other mode throws AgentAlreadyRunningError(sessionId, 'active') at :1321 for an active session with no pending calls, before the pinned CAS at :1350-1359), proven by runtime-temporal/src/__tests__/executor-resume-modes.test.ts:861 (origin/main 5bd30e20da) ("continue on an ACTIVE session with no pending calls (a live run) rejects AgentAlreadyRunningError and starts nothing"), :161 (origin/main 5bd30e20da) ("from_checkpoint on an EXECUTING session (active, no pending calls) rejects AgentAlreadyRunningError and changes nothing") and :871 (origin/main 5bd30e20da) (an active session WITH pending calls, the SF-D2 partial batch, is still accepted); runtime-cloudflare/src/executor.ts:983-989 (origin/main a118f8e17e) (fix evidence, !279: executor.resume() continue / with_message reject an active session with no pending calls with AgentAlreadyRunningError before any write) and :844-845 (origin/main a118f8e17e) (from_checkpoint: assertRewindable before the pinned claim), proven by runtime-cloudflare/src/__tests__/executor.test.ts:1980 (origin/main a118f8e17e) ("continue on an ACTIVE session with no pending calls (a live run) rejects AgentAlreadyRunningError and writes nothing"), :1265 (origin/main a118f8e17e) (a rewind of an EXECUTING session is rejected, changing nothing), :1999 (origin/main a118f8e17e) (an active session WITH pending calls is still accepted) and runtime-cloudflare/src/__tests__/integration/executor.integ.test.ts:498 (origin/main a118f8e17e) (the rewind rejection against real D1). Not fixed by !279: handle.resume() (resumeWorkflow, runtime-cloudflare/src/executor.ts:1708-1773 (pre-fix, origin/main a118f8e17e)) had no live-run check, and its CAS still accepted 'active' (:1762-1766 (pre-fix, origin/main a118f8e17e)); runtime-cloudflare/src/executor.ts:5014-5015 (origin/main 487090d3fb), :5118-5119 (origin/main 487090d3fb) and :5359-5360 (origin/main 487090d3fb) (fix evidence, CF-D1 MR 1a (!286), the handle.resume() half: the active handles' and the reconnect handle's resume(options) call executor.resume(agent, sessionId, options), and resumeWorkflow is deleted. executor.resume() refuses a run whose owner instance is live with AgentAlreadyRunningError before any write, through refuseIfOwnerLive at :915-925 (origin/main 487090d3fb) (called at :3572 (origin/main 487090d3fb); since !288), and gates a from_checkpoint rewind of a live run with assertRewindable at :3581 (origin/main 487090d3fb)), proven by runtime-cloudflare/src/__tests__/executor.test.ts:2098 (origin/main 487090d3fb) ("continue on an ACTIVE session with no pending calls (a live run) rejects AgentAlreadyRunningError and writes nothing"), :1972 (origin/main 487090d3fb) (a rewind of an executing session is rejected, changing nothing) and :3686 (origin/main 487090d3fb) (a reconnect handle's resume() delegates to executor.resume() with its options). Unit tests only; no test drives handle.resume() itself against a live run; runtime-js/src/js-agent-executor.ts:2639-2640 (origin/main 5bd30e20da) (fix evidence, !279: from_checkpoint calls assertRewindable before its pinned claim at :2664-2678) and :2823-2839 (origin/main 5bd30e20da) (the running-session pre-check of continue / with_message / with_confirmation now throws AgentAlreadyRunningError, DI-31), proven by runtime-js/src/__tests__/js-agent-executor-history.test.ts:273 (origin/main 5bd30e20da) ("from_checkpoint while a run is EXECUTING (LLM call held, version unchanged) rejects AgentAlreadyRunningError and leaves the log untouched") and :331 (origin/main 5bd30e20da) (continue and with_message on a running root session reject AgentAlreadyRunningError and write nothing); core/src/store/checkpoint-ops.ts:73-83 (assertRewindable: an active session is rewindable only with pending calls none of which is rewindHeld), proven by core/src/store/__tests__/checkpoint-ops.test.ts:1095 (origin/main 5bd30e20da); runtime-temporal/src/executor.ts:1323-1331 (origin/main a118f8e17e) and :1663-1666 (origin/main a118f8e17e), runtime-cloudflare/src/executor.ts:990-997 (origin/main a118f8e17e) and :1122-1125 (origin/main a118f8e17e), runtime-js/src/js-agent-executor.ts:2823-2840 (origin/main 5bd30e20da) (fix evidence, !279: the rewind-versus-continue fencing. A from_checkpoint rewind that keeps pending calls marks them rewindHeld, so assertRewindable reads the session as running until the rewind's own run takes them over, and a continue that takes a crashed hold over starts under an id derived from the hold's key), so a rewind racing a continue yields exactly one run. Proven by runtime-temporal/src/__tests__/executor-resume-modes.test.ts:404 (origin/main 5bd30e20da), runtime-cloudflare/src/__tests__/executor.test.ts:1601 (origin/main a118f8e17e) and runtime-js/src/__tests__/js-agent-executor-history.test.ts:874 (origin/main 5bd30e20da), :905 (origin/main 5bd30e20da), :936 (origin/main 5bd30e20da), :963 (origin/main 5bd30e20da). All of this is proven by unit and integration tests only; no conformance cell covers it yet; severity S2: a stray or duplicated resume() (a reconnecting client, a retrying caller) kills the user's in-flight turn: its caller gets failed and its answer is lost. It needs a resume against a running session, and the turn can be resent; conformance: lifecycle.resume-while-running (executor) passes on every executor lane at HEAD (owner batch 15, 93cb5c49bf product code; every red and green half re-run on f083dafb67 after the !316 rebase, same verdicts). Turn 1 completes; turn 2's model call is held, so its run is live with nothing pending; resume() in continue, with_message and from_checkpoint (turn 1's checkpoint) each throws AgentAlreadyRunningError (.continue-refused-typed, .with-message-refused-typed, .from-checkpoint-refused-typed); the persisted status, version and messages are unchanged (.no-writes); and the live run completes with its answer last (.live-run-completes). The fixes (!279, !286, and the owner guards !288 added) predate the cell, so each red half is a minimal mutation of the guards, not an MR revert. Each runtime now refuses a live run twice, at the executor call and again at the run-start, and both are guards the fixes added, so per the owner ruling each mutation removes both (working tree only, rebuilt, then reverted). CF Workflows: refuseIfOwnerLive (runtime-cloudflare/src/executor.ts:3572 (origin/main 487090d3fb)) and the entry's assertCfwOwnerless (runtime-cloudflare/src/run-entry.ts:1041-1043 (pre-fix, origin/main 487090d3fb)). The continue resume was accepted, took the session over and failed, the live run never completed, and every id from .continue-refused-typed on failed (with_message then threw a plain 'Cannot resume: Agent has failed'). Removing refuseIfOwnerLive alone left every id green: the entry refused the resume typed. Temporal: assertNoLiveOwner (runtime-temporal/src/executor.ts:2093) and the refusal in assertTemporalOwnerless (runtime-temporal/src/run-entry.ts:154-156). Every mode was accepted, and every id from .continue-refused-typed on failed on temporal-memory, temporal-redis and temporal-postgres (HELIX_TEMPORAL_BACKEND=test-server); removing assertNoLiveOwner alone reddened only the from_checkpoint attempt. runtime-js: the from_checkpoint live-owner refusal (runtime-js/src/js-agent-executor.ts:5165 (origin/main e3f03e8196)) and core's run-start owner guard (core/src/store/run-start.ts:281-282, which every store applies). The from_checkpoint resume was accepted, and .from-checkpoint-refused-typed, .no-writes and .live-run-completes failed on js-memory, js-redis and js-postgres. The pre-fix runtime-js defect sat in the executor (from_checkpoint skipped the running-session pre-check), so it was store-independent, and all three js lanes are in provenBy. DBOS refused a live run before the fix too (contrast) and passes | source, probe, live-repro | verified | fixed (7 cells) |
| CP-84 | S3 | A new turn started right after the previous one reads a terminal status (completed, or interrupted after a stop) is refused as AgentAlreadyRunningError on Temporal, because completed is published before the finished run releases the session. Temporal's persistTerminalState saves the session as completed and then ends the stream, and the workflow returns only after that. The next execute() starts a workflow under the same deterministic id ${prefix}__${agent}__${sessionId}, collides with the still-RUNNING one, and the collision is reported as "session was claimed by a concurrent request". Nothing else is running on the caller's behalf, and the same call succeeds once the old workflow closes. CF Workflows cannot reach this window: a second execute() on a session is blocked there regardless of timing, because every turn reuses the write-once instance id agent__${name}__${sessionId} (GitLab #109, documented in the Cloudflare Workflows runtime page), so the refusal is not recoverable by waiting. runtime-js and DBOS accept the new turn in this window on main, and then hit CP-85. | runtime-temporal/src/activities.ts:5224 (pre-fix, origin/main 5bd30e20da) (persistTerminalState: saveState with status completed) and :5280 (pre-fix, origin/main 5bd30e20da) (endStream after it, inside the same activity); the workflow returns only after this activity resolves; runtime-temporal/src/executor.ts:396 (pre-fix, origin/main a118f8e17e) (execute(): the workflow id is workflowIdPrefix__agentName__sessionId, the same for every turn) and :456-471 (pre-fix, origin/main a118f8e17e) (startWorkflow; "Workflow execution already started" is translated to AgentAlreadyRunningError "Cannot execute: session was claimed by a concurrent request."); contrast: out of scope on CF Workflows, runtime-cloudflare/src/executor.ts:614-617 (pre-fix, origin/main a118f8e17e) (execute() always creates instance id agent__agentName__sessionId, and Workflows instance ids are write-once, so a second turn on the same session is refused whether or not it lands in this window; GitLab #109, docs/runtimes/cloudflare-workflows.md:400-401). Its completion does publish completed before finalizeRun (runtime-cloudflare/src/workflow.ts:1997-2012 (pre-fix, origin/main a118f8e17e)) and execute() refuses while the run is running (executor.ts:521-526 (pre-fix, origin/main a118f8e17e)), so this window would reappear there once #109 gives each turn its own instance id; live: live-repro (batch-7 verification, origin/main 0ffde4e65b, no product change; e2e harness temporal-memory, HELIX_TEMPORAL_BACKEND=test-server): a text-only agent, turn 1 run to completion with the stream manager's endStream held. While held, the session read completed (the run record still read running) and execute() for turn 2 threw AgentAlreadyRunningError: "... (status: Cannot execute: session was claimed by a concurrent request.)". After the hold was released and the old workflow closed, the same execute() completed. A hold on updateRunStatus('completed') is never reached on main: Temporal writes no terminal run status there; contrast: live-repro (same verification): runtime-js (js-memory, js-postgres, js-redis) and DBOS (RUNTIME_DBOS_INTEG=1, private Postgres DB on :5433) accept execute() in the same window, because their claim is a session-status CAS (runtime-js/src/js-agent-executor.ts:2088-2101 (pre-fix, origin/main 5bd30e20da); runtime-dbos/src/lifecycle/execute.ts:464 (pre-fix, origin/main 5bd30e20da) for the CAS, and :673 (pre-fix, origin/main 5bd30e20da), whose workflowIds.standardRun id carries the runId); the new turn then collides with the old run's endStream, which is CP-85; inferred: (source only, not probed) the CF Durable Object answers /start with 409 ALREADY_RUNNING while isExecuting (runtime-cloudflare/src/do/durable-object-agent-base.ts:1667-1678 (pre-fix, origin/main 5bd30e20da)), and the flag clears only when the execution promise settles, after endStream (runtime-cloudflare/src/do/execution-state.ts:96 is the isExecuting getter; reset() at :273-284 clears the execution at :280, called from the execution promise finally block). A status read taken after the runtime-js loop persists completed and before the end frame therefore sees completed and still gets 409; a client driven by the end frame should not hit it; related: CP-85 (the same window on runtime-js and DBOS, where the new turn is accepted and the old run's endStream then ends its stream); CP-74 (the DBOS persistent-agent idle-exit window, where the same kind of message is silently lost); peer-reported (2026-09-27). On the reporter's branch (on top of reliability-b2-core-llm-history, not on main) Temporal also writes a terminal run status and gains a second refusal window before it (an owner check), and runtime-js refuses in its status-write window (a lease): the same cause, completed published before the owner releases the session. Those extra windows fold into this record as further evidence when that work lands on main; they need no new ids; severity S3: loud and recoverable. The caller gets a typed error and a retry after the owner closes succeeds, but the message ("claimed by a concurrent request") describes a conflict that does not exist, and a client that sends the next message as soon as it sees completed can be refused; runtime-cloudflare/src/do/drive/settle.ts:326-329 (fix evidence for the DO half only, WA MR1 (!340): the settle path finalizes the run's own stream, then releases the execution registration before any hook runs, so a drive in its tail is not executing and a new turn is not refused 409 there; the inferred DO evidence above no longer holds). The Temporal half, this record's title, is unchanged: the record stays open for it; live: the interrupt variant, in batch !599's pipeline (lane 3/7, job 17627569168, 2026-10-31): transcript.stopped-turn-partial-persisted on temporal-memory, after a mid-reply interrupt whose run record and session already read interrupted, saw the next turn's execute() refused state_already_running ('session was claimed by a concurrent request') with no model call: the stopped run's workflow was still open. The cell anchors on the stopped run's RECORD, which goes terminal before the workflow closes (FU-CONF-TEMPORAL-OWNER-CLOSED-OBSERVE: the lane cannot yet observe the close); related: CP-321 (batch 22): the same terminal window on runtime-js and the CF Durable Object; runtime-js now refuses there through the CP-96 run-status check (a different gate from the lease window this record noted), and the DO through its stage-1 drive gate | source, live-repro, peer-reported | verified | open |
| CP-85 | S2 | A finished run's endStream is not fenced against a newer run, so a turn started in the gap between the old run writing completed and ending its stream has its stream ended under it. runtime-js persists completed first, then writes the run status and runs the onAgentComplete hooks, and only then calls endStream unconditionally. A new execute() in that gap claims the session (the CAS accepts completed), finds the stream still active so it does not reactivate it, and starts. The old run's endStream then ends the shared stream. On runtime-js the new turn fails with "Cannot write to stream … in 'ended' state" (memory, Postgres and Redis). On DBOS the new turn reports completed, but every stream write it made after the end was dropped: its streaming client sees the previous run's end and none of the new turn's output. Temporal refuses the new turn in this window instead (CP-84), and from source so does the CF DO, which is why they don't show it. CF Workflows cannot reach the window at all: a second execute() on a session is blocked there by the write-once instance id (GitLab #109). Fixed on runtime-js, the CF DO and DBOS: every terminal endStream is now run-fenced (a no-op once another run owns the stream), and a new turn's run-start commit claims the stream for its own run. | runtime-js/src/run-loop.ts:3243 (pre-fix, origin/main 5bd30e20da) (onCompleted: persistTerminalState writes status completed first), :3251-3264 (pre-fix, origin/main 5bd30e20da) (updateRunStatus completed), :3266 (pre-fix, origin/main 5bd30e20da) (onAgentComplete hooks), then :1154-1159 (pre-fix, origin/main 5bd30e20da) (finalizeStreamForCompletion) and :4228 (pre-fix, origin/main 5bd30e20da) (endStream(state.streamId, ...)): no check that this run is still the session's current run; runtime-js/src/js-agent-executor.ts:2088-2101 (pre-fix, origin/main 5bd30e20da) (execute(): compareAndSetStatus accepts a completed session) and :2250-2258 (pre-fix, origin/main 5bd30e20da) (the stream is reactivated only when getStreamInfo already reads ended or failed; an active stream is left as is); store-memory/src/in-memory-stream.ts:236-240 (pre-fix, origin/main 5bd30e20da) and store-redis/src/redis-stream.ts:653 (pre-fix, origin/main 5bd30e20da) (a write to an ended stream throws "Cannot write to stream … in 'ended' state"); runtime-dbos/src/workflows/shared.ts:2919 (pre-fix, origin/main 5bd30e20da) (finalizeRun) then :3026 (pre-fix, origin/main 5bd30e20da) (endStream, unconditional); runtime-dbos/src/lifecycle/execute.ts:617-637 (pre-fix, origin/main 5bd30e20da) (a continuation reactivates the stream only when it already reads ended or failed); runtime-dbos/src/steps/call-llm.ts:98-112 (pre-fix, origin/main 5bd30e20da) and :151 (pre-fix, origin/main 5bd30e20da) (a failed stream write is logged at debug and dropped); live: live-repro (batch-7 verification, origin/main 0ffde4e65b, no product change; e2e harness): a text-only agent, turn 1's endStream (or its updateRunStatus('completed')) held after the session reads completed; execute() for turn 2; once turn 2's run is current, release. On js-memory, js-postgres (private DB on :5433) and js-redis (db 13 on :6390), with either hold, turn 2 resolved failed: "Cannot write to stream <sessionId> in 'ended' state"; live: live-repro (same verification, RUNTIME_DBOS_INTEG=1, dbos-postgres): with turn 1's endStream held, turn 2 resolved completed, but the stream ended holding only turn 1's chunks [step_start, text_delta ONE]. The control run without the hold held [step_start, ONE, step_start, TWO]; contrast: live-repro (same verification): temporal-memory refuses the new turn in this window (CP-84), so its stream is never shared across the gap; the CF DO (runtime-cloudflare/src/do/durable-object-agent-base.ts:1667-1678 (pre-fix, origin/main 5bd30e20da), 409 while isExecuting) is fenced the same way from source. CF Workflows is not fenced but unreachable: runtime-cloudflare/src/executor.ts:614-617 (pre-fix, origin/main a118f8e17e) reuses the write-once instance id agent__agentName__sessionId for every turn, so a second execute() on the session is refused regardless of timing (GitLab #109); related: CP-84 (the refusal the fenced runtimes give in the same window; fixing it by accepting the turn early would expose them to this record, so its fix must wait for the owner to close or fence the stream); CP-83 (a live-run resume on runtime-js ends in the same ended-stream failure); peer-reported (2026-09-27); severity S2: a lost turn. On runtime-js the user's next message fails, and a streaming client first sees the previous run's end; on DBOS the turn is reported completed while its streaming client receives none of it. It needs a new turn to land inside the old run's terminal tail, which includes the onAgentComplete hooks (a slow tracing flush widens it) and is reachable from another server instance that reads the session as completed; related: fixed in code by the S5 snapshot MR (its commit predates 1c5caeaa23, the earliest commit reachable from origin/main, so no fix sha can be cited), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/store/stream-manager.ts:381-391 (origin/main 56d564d073) (endStream takes a run fence: once another run owns the stream, the end is a no-op), store-memory/src/in-memory-stream.ts:507-508 (origin/main 56d564d073) (the memory manager honours it), runtime-js/src/run-loop.ts:6855-6859 (origin/main 56d564d073) (the completion end is fenced to the run) and :6940-6943 (origin/main 56d564d073) (the interrupt end the same), runtime-dbos/src/steps/stream-lifecycle.ts:361-366 (origin/main 56d564d073) (the DBOS terminal end is fenced to its run), core/src/run/commit-stop-terminal.ts:268 (origin/main 56d564d073) (core's stop terminal fences its end and fail), core/src/run/fail-session-without-run.ts:103-108 (origin/main 56d564d073) (the pre-commit failure path the same) | source, live-repro, peer-reported | verified | open |
| CP-86 | S2 | On Temporal, a resume() that finds client-tool calls still unanswered leaves the session active with no workflow executing. The resume CAS moves the paused session to active before the workflow starts. The workflow drains the submitted results, sees calls still pending, and returns through the hasMoreSuspended early exit without writing the session status back to paused. So after a partial batch (c1 submitted, c2 not) plus resume(), the session status says the agent is running, and execute() is refused as "session is already running" rather than with the pending-client-tool message the same call gets on a paused session. (The early return writes no run status either, so the resumed run record also stays running; a fix that routes the re-suspend through the suspend-commit path, which writes the suspended run status, would fix that too. RM-47 covers the missing terminal run status.) Submitting c2 and resuming again still completes the run, so nothing is lost, but the session misreports a live run that does not exist until then. | runtime-temporal/src/executor.ts:1350-1366 (pre-fix, origin/main a118f8e17e) (resume(): a version-pinned compareAndSetStatus(['paused', 'interrupted', 'active'] → 'active'; a from_checkpoint resume also admits completed and failed) before any workflow runs); runtime-temporal/src/workflow.ts:751-759 (pre-fix, origin/main a118f8e17e) (resume mode: when applyResultsAndReload reports hasMoreSuspended, the workflow returns a suspended result derived from state; no activity writes the session status back to paused) and runtime-temporal/src/activities.ts:4359 (pre-fix, origin/main 5bd30e20da) and :4474 (pre-fix, origin/main 5bd30e20da) (by contrast, the first suspension's commitSuspendedStep forces status 'paused'); runtime-temporal/src/executor.ts:329-336 (pre-fix, origin/main a118f8e17e) (execute(): an AgentState status of running, which is what an active session maps to, throws 'Cannot execute: session is already running') vs :439-451 (as before WA MR2, origin/main cae966c0a7) (the paused-with-pending guard, whose message points the caller at submitToolResult() / interrupt()); live: live-repro (batch-7 verification, origin/main 0ffde4e65b, no product change; e2e harness temporal-memory, HELIX_TEMPORAL_BACKEND=test-server): one step emits client calls c1 and c2. After execute() the session read paused with [c1, c2] pending. submitToolResult(c1) was accepted, then resume(): its handle did not settle within 5 s (CP-65); 1.5 s later the session read active with [c2] pending (the current run also read running), while the backend's own execution status showed both workflows (the original and __resume-1, the pre-!288 id scheme; !288 names it __resume-<runId>) closed and none open. execute() then threw AgentAlreadyRunningError "Cannot execute: session is already running. Wait for the current execution to complete, or use resume() for crash recovery.", and the session stayed active; contrast: live-repro (same verification) on js-memory: the same steps complete the run (the CP-82 defect, c2 is fabricated), so the session never sits in this state; batch-6 verification (CP-82) observed that on Temporal a later submit(c2) + resume() completes with both real answers, so the state is recoverable through the documented path; related: RM-47 (Temporal wrote no terminal run status, fixed in code by 2026-10-10; commitSuspendedStep does write the suspended statuses, runtime-temporal/src/activities.ts:4713-4738 (pre-fix, origin/main 5bd30e20da), but this record's early return bypasses it, and runtime-temporal/src/executor.ts:1573-1586 (pre-fix, origin/main a118f8e17e) has recorded the resumed run as running, so the run record also stays running here); CP-65 (the same early return emits no suspension_marker, so the resume handle never settles: the observation side of this path, a different write; a fix that only emits the marker leaves the status wrong); CP-83 (Temporal's resume CAS accepts active precisely because this path leaves a pending session active; MR !279's discriminator, pending client-tool calls, keeps it resumable, and restoring paused here would make that special case unnecessary); CP-82 (the partial-batch scenario); DI-16 (runs left running with no owner); peer-reported by the reporting session as a failing submit.partial-batch .stays-paused id on Temporal (2026-09-27, on its base and its branch); severity S2: a wrong status report with no executor behind it. Session status reads, reconnecting clients and operators see a live run that does not exist, and a new turn is refused as already running with a message that hides the real cause (an unanswered client tool). No data is lost and the documented submit-then-resume path still completes the run, which keeps it below S1; conformance: submit.partial-batch and submit.partial-batch-resume on temporal-* — .stays-paused, .execute-refused-pending (execute() after the partial resume is refused with the pending-client-tool message, not as a running session) and the resume settle ids fail on origin/main 5bd30e20da and pass with !288 (RM-47 is a separate id, not asserted here); runtime-temporal/src/executor.ts:1692-1694 (as before WA MR2, origin/main cae966c0a7) (fix evidence, !288: continue / with_message on a partial batch are refused before the entry, so no resume moves the session to active) and runtime-temporal/src/activities.ts:5193-5223 (a from_checkpoint takeover of a still-partial batch writes the session back to paused, fenced on the new run, and the workflow re-suspends the run) | source, live-repro, peer-reported | verified | fixed (6 cells) |
| CP-87 | S1 | A remote child interrupted on a Temporal, DBOS or CF Workflows producer reaches its cross-service parent as completed with null output. AgentServer builds the end frame's status as endStatus ?? 'completed', and endStatus is recorded only when the producer passes endStream's terminal argument, which only runtime-js and the CF DO do. The other three end an interrupted run's stream without it, so the frame says completed. Every consumer folds a completed end straight to success with output = end.output ?? null, skipping the D5 guard that re-checks status for interrupted ends. With a null-accepting remote outputSchema the parent's tool result is a success with null and the SubSessionRef reads completed. With an object schema the call fails as "output failed schema validation … received null", which blames the child's output instead of reporting the interrupt. Fixed on every runtime by 6ef75cb6c9: endStream's status is required and no reader defaults to completed. | agent-server/src/agent-server.ts:612-648 (pre-fix, origin/main 5bd30e20da) (synthesizeEndEvent; :638 status = info.status === 'paused' ? 'paused' : (info.metadata.endStatus ?? 'completed')); core/src/store/stream-manager.ts:248-256 (pre-fix, origin/main 5bd30e20da) (endStream(streamId, finalOutput?, terminal?): endStatus is persisted only from terminal.status); store-memory/src/in-memory-stream.ts:396 (pre-fix, origin/main 5bd30e20da); runtime-temporal/src/activities.ts:5312-5316 (pre-fix, origin/main 5bd30e20da) (interrupted: endStream(streamId, undefined), no terminal) and :5159-5175 (pre-fix, origin/main 5bd30e20da) (the already-terminal retry heal, added by !279: endStream(streamId, completed ? output : undefined), also no terminal, so an interrupted session healed there also ends completed); runtime-dbos/src/workflows/shared.ts:1540-1552 (pre-fix, origin/main 5bd30e20da) (interrupt flag → finalizeRun 'interrupted' → dispatcher.endStream({ streamId })) and runtime-dbos/src/steps/stream-lifecycle.ts:246 (pre-fix, origin/main 5bd30e20da) (endStream(streamId, finalOutput), no terminal); runtime-cloudflare/src/workflow.ts:1695-1700 (pre-fix, origin/main a118f8e17e) and :3035-3039 (pre-fix, origin/main a118f8e17e) (root interrupted: endStream(streamId), no terminal); :671 (pre-fix, origin/main a118f8e17e) (isRootAgent = !parentSessionId; AgentServer passes no parentSessionId, agent-server/src/agent-server.ts:402-429, and runtime-cloudflare/src/executor.ts:592-607 (pre-fix, origin/main a118f8e17e) builds workflowInput without one, so a remote child is always root); core/src/orchestration/remote-sub-agent-dispatch.ts:493-510, runtime-js/src/execution/remote-subagent-executor.ts:532-555, runtime-dbos/src/steps/execute-remote-subagent.ts:496-503 (pre-fix, origin/main 5bd30e20da), runtime-cloudflare/src/steps.ts:3375-3384 (pre-fix, origin/main a118f8e17e) (end.status === 'completed' → output = event.output ?? null, no status re-check); core/src/orchestration/remote-sub-agent-completion.ts:149-158 (completed end folds directly) vs :159-200 (the D5 getStatus guard, reached only for non-completed ends); core/src/orchestration/validate-remote-output.ts:42-58 (the only remaining check); contrast: runtime-js/src/run-loop.ts:4228 (pre-fix, origin/main 5bd30e20da), :4291 (pre-fix, origin/main 5bd30e20da) and runtime-cloudflare/src/do/durable-object-agent-base.ts:4293-4297 (pre-fix, origin/main 5bd30e20da) pass terminal, so a JS or DO producer sends end.status interrupted; runtime-js/src/run-loop.ts:4270-4279 (pre-fix, origin/main 5bd30e20da) already documents the failure mode (withholding terminal makes synthesizeEndEvent fall back to completed, and a remote parent then folds the child as a successful one with null output); live: probe (batch-8 verification, origin/main 0ffde4e65b, before !279, which changed none of the paths this probe exercised (it added a second Temporal end without terminal, the retry heal cited above); no product change; e2e createRemoteAgentTestContext with the producer executor injected, as in remote-agent-matrix.integ.test.ts; a runtime-js parent over HttpRemoteAgentTransport; the child's first model call held, then interrupted via agentServer.interruptAgent or producer getHandle().interrupt()). Temporal (HELIX_TEMPORAL_BACKEND=test-server), both interrupt paths: child session interrupted, output null; child stream ended with no endStatus; the parent's transport received {type:'end', status:'completed'}. With remote outputSchema z.unknown() the parent's tool result was content "null", helixToolFailed false, SubSessionRef completed; with z.object({answer}) it was the error "Remote sub-agent 'worker' output failed schema validation: … expected object, received null". DBOS (RUNTIME_DBOS_INTEG=1, private Postgres DB on :5433, Redis db 13 on :6390), getHandle().interrupt() (flag path): identical. Control: a JS producer sent {type:'end', status:'interrupted'} and the parent failed the call ("last observed status: 'interrupted' (D5 …)"); inferred: CF Workflows end to end (only the producer leg was run, in a unit harness, runtime-cloudflare/src/__tests__/workflow.test.ts runAgentWorkflow with the interrupt flag set and a real InMemoryStreamManager): result interrupted, endStream called as ['stream_b8'] with no terminal, stream ended with endStatus undefined. The AgentServer and consumer legs are not run on CFW (no CFW producer harness; remote-agent-matrix's cfwProducer is a stub) but are the same code as the Temporal and DBOS legs; contrast: docs/guide/cross-service-remote-agents.md:411-424 documents that Temporal, DBOS and CF Workflows omit terminal and calls it correctness-neutral, discussing only state and usage, not status; related: CP-45 (a remote child the consumer KNOWS is interrupted or paused, with stale output, is folded as success on the getStatus recovery path; a different cause and fix site: here the recovery path is never reached); CP-17 (a DBOS same-replica interrupt via ActiveHandle hard-cancels and leaves the stream active, so a remote parent instead waits out timeoutMs and fails 'timed out'); DI-19 (a failed DBOS ephemeral child reported as success, the same shape for local children); severity S1: with a null-accepting remote outputSchema the parent model is told a stopped delegation succeeded with null output (wrong model behaviour, live on Temporal and DBOS). With an object schema the defect is an S2-grade misattributed failure; runtime-dbos/src/lifecycle/interrupt.ts:497 and :526 (origin/main 6856de1efd) (settleDeadOwnerRun sets the dead owner's run to its fallback status, then always ends the stream with no status) with :545-551 (origin/main 6856de1efd) (settleCancelledRun(..., 'failed') is that fallback's only source, a public helper with no in-repo production caller) and core/src/stream/endpoint.ts:86 (origin/main 6856de1efd) (an ended stream with no endStatus reads as completed): the class also covers a FAILED run reported completed, with no errorDetail; related: the other stream-end sites that omit the status (WE-2, 2026-10-18, source-read at origin/main 6856de1efd; the DBOS site verified by the owner): core commit-stop-terminal.ts:332, run-markers.ts:320, fail-committed-run.ts:414, reconcile-closed-owner-run.ts:673; CF Workflows steps.ts:5986 (via workflow.ts:1026 / :1053); Temporal activities.ts:7420 / :7430 and executor.ts:1504; DBOS run-entry.ts:558, stream-lifecycle.ts:356, workflows/shared.ts:2235. Through commit-stop-terminal, a CF DO child stopped while suspended serves end{completed} from its /sse. Fix (approved, after !429): endStream's status becomes required, readers stop defaulting to completed, and the DBOS case moves to failStream with the framework_cancelled detail (its own red: a DBOS integ through settleCancelledRun(..., 'failed')); related: two more sites in the class (WE-2, 2026-10-18, each with a red-first unit test on reliability-we-cp87-end-status a78c209986; verified by the owner at origin/main f3c20316d2): (1) runtime-js run-loop.ts:5776-5784 builds the completion end's terminal only when terminalPersisted (:4374, the K5 committed and onAgentComplete did not throw, :4269-4275), so a completed run whose hook threw ends its stream with NO status, though its interrupted sibling lets status ride unconditionally (:5851-5863); (2) the CF writers and readers default to completed: runtime-cloudflare do-stream-manager.ts:550 (endStream: terminal?.status ?? completed), durable-object-agent-base.ts:5073 (sendInDOTerminalFrame: getEndStatus() ?? completed, the late-join / SSE-race end frame), and store-cloudflare stream-durable-object.ts:142 (EndStatusSchema = z.enum([completed, interrupted]).catch(completed)), which the /end route parses (:820), getEndStatus() reads (:1723-1725) and the ended stream's metadata endStatus reads (:1324), so a missing status is written and read as completed; related: fixed in code by 6ef75cb6c9 (a stream end always records the run's status), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): core/src/store/stream-manager.ts:386-391 (origin/main fb9150cc16) (endStream's terminal argument is required); agent-server/src/agent-server.ts:748-753 (origin/main fb9150cc16) (an ended stream with no endStatus is logged and read as a retryable drop, never completed); the producers pass it (runtime-temporal/src/activities.ts:7481 (origin/main fb9150cc16); runtime-dbos/src/steps/stream-lifecycle.ts:361-366 (origin/main fb9150cc16); CF Workflows runtime-cloudflare/src/steps.ts:7271-7276 (origin/main fb9150cc16)). Open until its cell; related: fixed in code by 6ef75cb6c9 (CP-87: a stream end always records the run's status), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/store/stream-manager.ts:386-391 (origin/main 56d564d073) (endStream's terminal argument is required), agent-server/src/agent-server.ts:750-753 (origin/main 56d564d073) (an ended stream with no endStatus is logged and answered as a stream drop, never completed), core/src/stream/endpoint.ts:103-108 (origin/main 56d564d073) (the same rule for the shared endpoint), store-cloudflare/src/stream-durable-object.ts:162 (origin/main 56d564d073) (the end status schema no longer defaults to completed), runtime-cloudflare/src/do/do-stream-manager.ts:812 (origin/main 56d564d073) (the DO writes the run's own terminal status, no default), runtime-dbos/src/lifecycle/interrupt.ts:477 and :536 (origin/main 56d564d073) (the dead-owner settle fails the stream for a failed fallback instead of ending it) | source, probe, live-repro | verified | open |
| CP-88 | S3 | CF Workflows cannot run a second execute() turn on a session: the workflow instance id is fixed per session (agent__<name>__<sessionId>) and instance ids are write-once, so the second turn's create() throws the engine's duplicate-instance error ((instance.already_exists) Workflow instance with id … already exists under Miniflare). Nothing catches or types it, so the caller gets an untyped error that classifies as framework_internal_error (not retryable) instead of a typed refusal, and getHandle() looks up only that base id, so it never returns an active handle for a __resume__ / __retry__ / __rewind__ instance. The limitation is documented (GitLab #109); the error the caller gets is not. Every other runtime continues a completed session into a new turn. | runtime-cloudflare/src/executor.ts:614-617 (pre-fix, origin/main a118f8e17e) (execute(): create({ id: agent__NAME__SESSIONID, params }) with no per-turn suffix and no try/catch; the only try/catch in execute() wraps createSession, :631-663); runtime-cloudflare/src/executor.ts:610 (pre-fix, origin/main a118f8e17e) and :630 (pre-fix, origin/main a118f8e17e) (sessionExists is read only to skip createSession; there is no continuation branch in the executor) and :521-527 (pre-fix, origin/main a118f8e17e), :544-555 (pre-fix, origin/main a118f8e17e), :565-571 (pre-fix, origin/main a118f8e17e) (the pre-create guards refuse only a running run, a paused session with pending client tools, and a foreign branch checkpoint, so a completed or interrupted prior turn reaches create()); runtime-cloudflare/src/executor.ts:610-617 (pre-fix, origin/main a118f8e17e) (everything before create() only reads: getCurrentRun, loadState, resolveCheckpoint, parseAgentInput, sessionExists; the writes come after it, initStream at :621, createSession at :632, and the body appends the user message, runtime-cloudflare/src/workflow.ts:794-796), so the failure leaves the session and stream untouched; runtime-cloudflare/src/workflow.ts:729-857 (pre-fix, origin/main a118f8e17e) (the body HAS a continuation branch for a completed, failed or interrupted session, which a root execute() can never reach because create() fails first) and :3753 (pre-fix, origin/main a118f8e17e) (companion re-consults avoid the collision with a __continue__ suffix; root execute() has no equivalent); runtime-cloudflare/src/executor.ts:685 (pre-fix, origin/main a118f8e17e) (getHandle: get(agent__NAME__SESSIONID) only) and :693-696 (pre-fix, origin/main a118f8e17e) (a missing or non-running base instance returns a reconnect handle); the live instance after resume() / retry() is __resume__N, __retry__N or __rewind__K (:1124-1125 (pre-fix, origin/main a118f8e17e), :1444 (pre-fix, origin/main a118f8e17e), :1784 (pre-fix, origin/main a118f8e17e)); runtime-cloudflare/src/executor.ts:614-617 (pre-fix, origin/main a118f8e17e) reaches the engine duplicate check: Miniflare 5.20260925.0-alpha (the runtime-cloudflare and e2e devDependency) dist/src/workers/workflows/binding.worker.js:4921-4922 (create() throws duplicateInstanceError when the id exists), :1447-1452 (message and code instance.already_exists) and :1384-1385, :1395-1396 (a plain Error subclass WorkflowError, message prefixed with the code); core/src/errors/classify-error.ts:90-94 (pre-fix, origin/main a118f8e17e) maps an unrecognised Error to framework_internal_error, retryable false; contrast: docs/runtimes/index.md:46, docs/runtimes/cloudflare-workflows.md:401 and docs/guide/concepts.md:498 document that a second execute() on a completed session is blocked on CF Workflows (GitLab #109) and that every other runtime continues it; none says what the caller receives; related: docs/dev/follow-ups.md:1549-1552 and :1576-1579 (pre-fix, origin/main 2b6878644f) (the multi-turn continuation fix covered JS, DBOS, Temporal and the CF DO; CF Workflows needs a per-turn write-once instance id, getCurrentRun handle reconstruction and stream reactivation, GitLab #109); CP-84 and CP-85 (their CF Workflows halves are out of scope because of this collision); CP-67 (the stream-reactivation half on the resume path); CP-47 (the same root for the handle lookup: code that addresses the base instance id while the live instance is a __resume__ / __retry__ one); plan 0c (the CF Workflows executor lane) requires this fix before the lane flips, because every second-turn scenario would fail at the collision; severity S3: a loud failure of a documented limitation; the defect in the record is the untyped, non-retryable-classified error a caller cannot tell apart from an internal fault, and the handle lookup that cannot find a resumed instance; runtime-cloudflare/src/executor.ts:1122-1300 (origin/main fdc56520a5) (fix evidence, CF-D1 MR 1a (5bd30e20da), reworked onto the run-start commit by !288 (bd26dc9b57): execute() continues a completed / failed / interrupted / clean-paused session, the JS / CF DO set (:92 (origin/main da5f85ce42); a paused session still awaiting a client tool or a sub-agent is refused, :1159-1187 (origin/main da5f85ce42)); every turn runs in its own write-once instance, buildWorkflowInstanceId({kind: 'turn', key}) at :1271-1276, with a key minted by the call (entryInstanceKey, :183-185 (origin/main da5f85ce42)), so a second turn never collides with the first; the run id embeds the instance id (runIdFor, :367-369 (origin/main da5f85ce42)), so the turn's run record names its owner instance; the single-winner gate is the instance's run-start commit (runtime-cloudflare/src/steps.ts:818 (pre-fix, origin/main d2e95ad181)), and a call that loses it writes nothing and is refused with AgentAlreadyRunningError (awaitEntryRunStart, runtime-cloudflare/src/executor.ts:437 (origin/main fdc56520a5)); a continuation first waits, bounded, for the previous instance to close, :1130-1135 (origin/main fdc56520a5) and :1372-1391 (origin/main fdc56520a5)) and :1305-1330 (origin/main fdc56520a5) (getHandle() → resolveLiveInstance(), :1342-1359 (origin/main fdc56520a5): the live instance is read from the current run record's owner token, so getHandle() finds any turn / resume / retry instance). MR 1a's first shape (sequential turn keys, 1 then <resumeCount after the claim's bump>+1, behind a version-pinned CAS claim, an engine duplicate mapped to AgentAlreadyRunningError, and a getHandle() keyed on resumeCount) is at runtime-cloudflare/src/executor.ts:685-823, :882 and :935 (origin/main 5bd30e20da). Proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:345 ("T2 (CP-88): a second execute() continues the session (history kept, stepCount reset, 2 runs)", red on main: (instance.already_exists)), :366 (continuation of failed / interrupted), :384 (a clean paused session), :560 ("T7 (CP-88): getHandle() during turn 2 returns an active handle on the live instance"), :849 (3+ turns mixed with an interrupt + resume) and runtime-cloudflare/src/__tests__/executor.test.ts:3395, :3545 and :3605 (unit: one instance per execute(), the settle-wait, getHandle() from the run record). The plan 0c CF Workflows lane proves the second-turn half only (see the conformance note); runtime-cloudflare/src/steps.ts:998-1001 (pre-fix, origin/main e96f88bc98) (fix evidence, CF-D1 MR 1a (5bd30e20da), reworked by !288 (bd26dc9b57): a turn entry's run-start commit resets the PERSISTED step count, not only the loop-local counter, so maxSteps is per-turn on CFW as on every runtime, also for companion __continue__ instances (a child's turn is a turn entry, runtime-cloudflare/src/workflow.ts:1329-1333 (pre-fix, origin/main e96f88bc98)); a continue / with_message resume of an interrupted turn 2+ keeps the whole log and truncates nothing, runtime-cloudflare/src/steps.ts:1007-1020 (origin/main fdc56520a5)). MR 1a's first shape (a reset-step-count step, and a turn-start checkpoint written after the new user message so a resume / retry of an interrupted turn 2+ did not truncate to the prior turn's terminal checkpoint and delete that message; !288 removed the turn-start checkpoint) is at runtime-cloudflare/src/workflow.ts:843 and :888 (origin/main 5bd30e20da); conformance: the second-turn half is proven on cfw-workflows-d1 by history.prior-turns (a second execute() continues the session with its history), history.branch-from-earlier, history.branch-at-message-index, lifecycle.retry-after-turn and the transcript.* second turns (transcript.finish-with-siblings, transcript.finish-cosiblings, transcript.double-finish, transcript.finish-with-single), all passing in all 5 full lane runs of the plan 0c MR-2 lane flip (origin/main d2e95ad181 product code). The getHandle() half is not exercised: the executor driver re-attaches through getHandle() only on a backend whose submit continues the run itself, and cfw-workflows-d1 continues through resume(); conformance: fixed. The getHandle() half: lifecycle.get-handle-live-turn (owner batch 28, executor tier, every lane): turn 1 completes, turn 2's model call is held, and getHandle() (the reconnect op) is called while turn 2 is live. On cfw-workflows-d1 the handle stays unsettled while turn 2 is held and, after the release, settles completed with turn 2's answer as the session's last message; green 3 of 3 on main 8dc3a5c967 plus batches 24 to 27. Red with the pre-fix rule restored (the live instance resolved from the session's first run, one instance per session, working tree only, reverted): .handle-not-stale fails, the handle settling completed at once with turn 1's result. (On js-memory the same cell fails differently: runtime-js's reconnect handle is a snapshot whose result() throws for a live run; that is not this record) | source | verified | fixed (3 cells) |
| CP-89 | S1 | CF Workflows execute() on a new session races its own workflow body. The executor creates the workflow instance BEFORE it writes the session, and create() returns before the body starts, so the two run concurrently. If the session row lands before the body's check-existing step, the new session reads active (mapped to running) and the body takes its resume branch, which never runs initializeAgentState: onAgentStart and the user message's onMessage never fire, the branch and history inputs are dropped, the state-schema defaults are never applied, and the run records a start message count of 0, all with execute() returning normally. If the body's init wins instead, createSession fails and the caller gets an AgentAlreadyRunningError blaming a concurrent request that never existed, while the workflow keeps running (the executor's instance.abort() does not exist, CP-15) against a stream the executor just marked failed. | runtime-cloudflare/src/executor.ts:614 (pre-fix, origin/main a118f8e17e) (workflowBinding.create), then :621 (pre-fix, origin/main a118f8e17e) (initStream), then :632 (pre-fix, origin/main a118f8e17e) (createSession, only when the session did not exist, :630 (pre-fix, origin/main a118f8e17e)): the instance exists before the session row does; runtime-cloudflare/src/executor.ts:614 (pre-fix, origin/main a118f8e17e) reaches the engine create: Miniflare 5.20260925.0-alpha dist/src/workers/workflows/binding.worker.js:4939 (initPromise = stub.init(...)) and :4967 (returns { id } under waitUntil(initPromise)), so the body runs concurrently with the executor writes that follow create(); runtime-cloudflare/src/workflow.ts:700-702 (pre-fix, origin/main a118f8e17e) (check-existing step: loadAgentState) and :220 (pre-fix, origin/main a118f8e17e) (status: sessionStatusToAgentStatus); core/src/types/session.ts:874 (pre-fix, origin/main a118f8e17e) (a created session is active; store-cloudflare/src/d1-state.ts:2228-2237 (pre-fix, origin/main a118f8e17e) creates it through that helper) and core/src/types/session.ts:956-957 (pre-fix, origin/main a118f8e17e) (active maps to running); runtime-cloudflare/src/workflow.ts:707-708 (pre-fix, origin/main a118f8e17e) (isResumable: running or paused), :710-728 (pre-fix, origin/main a118f8e17e) (the ONLY initializeAgentState call is on the no-existing-state branch, :713) and :840-867 (pre-fix, origin/main a118f8e17e) (the resume branch: createResumeRun, no reset, then the new messages appended with no hook); runtime-cloudflare/src/steps.ts:728-935 (pre-fix, origin/main a118f8e17e) (initializeAgentState is the only place that fires onAgentStart, :870-880; fires onMessage per user message, :894-903; loads the branch source, :756-785; seeds history, :797-812; applies stateSchema.parse defaults, :819-826; records the run start message count after the append, :925-930); on the resume branch the session keeps the raw initialState createSession stored (store-cloudflare/src/d1-state.ts:2231 (pre-fix, origin/main a118f8e17e)) and createResumeRun reads startMessageCount before the append (runtime-cloudflare/src/steps.ts:6019 (pre-fix, origin/main a118f8e17e), runtime-cloudflare/src/workflow.ts:862-866 (pre-fix, origin/main a118f8e17e)), so it records 0; runtime-cloudflare/src/workflow.ts:859-861 (pre-fix, origin/main a118f8e17e) (the body's own comment names the race: the pre-persist race where createSession ran before this workflow started) versus runtime-cloudflare/src/executor.ts:623-629 (pre-fix, origin/main a118f8e17e) (the executor's comment claims the body's initializeAgentState will find the existing session and carry its version, which is true only when check-existing ran first); runtime-cloudflare/src/executor.ts:640-662 (pre-fix, origin/main a118f8e17e) (when createSession loses with 'Session already exists', store-cloudflare/src/d1-state.ts:2294-2299 (pre-fix, origin/main a118f8e17e): runtime-cloudflare/src/executor.ts:642-646 (pre-fix, origin/main a118f8e17e) calls instance.abort() inside a swallowing catch, :648-652 (pre-fix, origin/main a118f8e17e) fails the stream, :656-660 (pre-fix, origin/main a118f8e17e) throws AgentAlreadyRunningError 'session was claimed by a concurrent request'); the abort is a no-op because a Workflows instance has no abort() (CP-15); inferred: which ordering a fresh session takes is not measured. Because create() returns before the engine initialises and the executor needs only initStream and one insert before the body reads, the resume-branch ordering (the silent one) is reachable on every fresh execute(); how often each ordering wins is not known (the O5 note below saw the loud ordering in about 2 of 3 in-pool runs under vitest-pool-workers, a different engine build). Plan 0c T0 measures it over 20 back-to-back execute() calls; related: the O5 workerd parity harness design (docs/superpowers/specs/2026-05-30-o5-workerd-parity-harness-design.md:136) observed AgentAlreadyRunningError in about 2 of 3 in-pool runs, which is the loud ordering above (the body init wins); CP-15 (instance.abort() does not exist); RM-43 (the resume branch run-record ordering); CP-38 (the branch checkpoint check execute() now does before create()); plan 0c requires this fix before the CF Workflows lane flips, because a nondeterministic masker cannot meet the 5x determinism rule; severity S1: in the silent ordering a new conversation starts without its history input, branch source, start hook or message hooks and without schema defaults, while execute() reports success; the model is given the wrong context and hook-based integrations (tracing, audit) miss the turn. The harm is confined to agents that use those features: the user message itself still lands (the resume branch appends it), so a plain agent with no hooks, history or branch input and no schema defaults sees a correct first turn; runtime-cloudflare/src/executor.ts:1236-1254 (origin/main fdc56520a5) (fix evidence, CF-D1 MR 1a (5bd30e20da), reworked onto the run-start commit by !288 (bd26dc9b57): execute() creates a NEW session's row write-once BEFORE workflowBinding.create, and a concurrent creator is not a conflict; nothing before create() is a claim, so a create() that throws rolls nothing back and a crash between the two strands nothing: the next execute() runs the session, :1267-1270 (origin/main fdc56520a5) and :1278-1279 (origin/main fdc56520a5); the single-winner gate is the instance's run-start commit, runtime-cloudflare/src/steps.ts:818; the dead instance.abort() call is gone) and runtime-cloudflare/src/workflow.ts:1344-1366 (pre-fix, origin/main e96f88bc98) (the body takes a turn entry for any session with no run, no messages and no steps, so a session row written before the body's check-existing read no longer sends a fresh execute() down the resume branch; the turn entry seeds the history input and applies the state-schema defaults inside the run-start commit, runtime-cloudflare/src/steps.ts:949-997 (pre-fix, origin/main 487090d3fb), then fires onAgentStart and the user messages' onMessage once, runtime-cloudflare/src/workflow.ts:1506-1522 (pre-fix, origin/main e96f88bc98)). MR 1a's first shape (a version-pinned claim before create(), a failed create() that deleted the new session or restored the prior status, and a stranded new-session claim reclaimed under the write-once turn__1 id) is at runtime-cloudflare/src/executor.ts:685-823, :698 and :913 (origin/main 5bd30e20da), with the body's entry switch at runtime-cloudflare/src/workflow.ts:719-721 (origin/main 5bd30e20da). Proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:499 ("T4 (CP-89): 20 fresh executes each run initializeAgentState (hooks once, history seeded, schema defaults)", red on main: the loud ordering refused fresh executes with 'session was claimed by a concurrent request'), :277 (deterministic since the fix), :683 (a session row left by a crashed execute() runs on the next execute()) and runtime-cloudflare/src/__tests__/executor.test.ts:3395 and :3523 (unit: one instance per execute(); a crash between the session create and create() strands nothing, and the next execute() runs). The plan 0c MR-1b T0 spike re-measured the fixed path (20 back-to-back fresh execute() calls: no refusal, and one user-message onMessage per session), but the spike is not a lane cell; what the lane proves is in the conformance note below; conformance: the loud ordering is proven gone on cfw-workflows-d1 in all 5 full lane runs of the plan 0c MR-2 lane flip (origin/main d2e95ad181 product code): no fresh execute() was refused. session.create-observed-whole (.runs-complete) completes 48 fresh execute() calls per run, and smoke.hello-completes (.completes) one more. The branch input of the silent ordering is proven by the fresh branched execute() of smoke.branch-completes (.history-has-turn1), history.branch-from-earlier (.branch-includes-checkpoint-history) and history.branch-at-message-index (.branch-includes-prefix): each branch session carries its source history. No executor-tier cell asserts the rest of the silent ordering: onAgentStart firing once (the hooks channel records no onAgentStart), the user message's onMessage on a fresh turn (history.prior-turns' hook checks require only an assistant onMessage and matching indices), the history input (no scenario passes one), the state-schema defaults (no conformance agent declares a stateSchema) and the run's start message count; conformance: fixed. The silent half: lifecycle.fresh-entry-initializes (owner batch 28, executor tier, every lane): a fresh execute() with a history input (an earlier user and assistant exchange) on conf-fresh-entry, whose state schema defaults one key. On cfw-workflows-d1 the run completes, onAgentStart fires once (the hooks channel now records it, relayed by the worker bridge), the user message's onMessage fires once, the model receives the history then the turn's message, the persisted custom state holds the default and the run record's startMessageCount is 3, the user message's index plus one; green 3 of 3 on main 8dc3a5c967 plus batches 24 to 27, and green on js-memory. Red with the pre-fix rule restored (the body's check-existing treats any existing session row as a turn to continue, so a fresh session takes the resume branch; working tree only, reverted): .start-hook-once, .user-message-hook-once, .history-reaches-model and .start-message-count fail; .schema-default-persisted still passes there, because the session row already holds the parsed initial state. session.create-observed-whole and smoke.branch-completes re-confirmed passing on cfw-workflows-d1 at main | source | partly-inferred | fixed (3 cells) |
| CP-90 | S2 | CF Workflows builds workflow instance ids by interpolating the caller's session id and agent names, with no validation or encoding, but the engine accepts an instance id only if it is at most 100 characters and matches ^[a-zA-Z0-9_][a-zA-Z0-9-_]*$. A session id that every other runtime accepts (a dot, colon, slash, @ or space, or simply a long id) makes execute() throw an untyped "Workflow instance has invalid id", and because sub-agent and companion ids grow with nesting (agent__<type>__<session>__sub__<callId>, …-agent-<name>__continue__<n>__<callId>), ordinary names can push a child id past 100 characters, failing the spawn inside the parent's workflow step. No doc states the constraint, and core's companion-name limit (128) claims to keep derived ids "well under platform limits". | runtime-cloudflare/src/executor.ts:614-615 (pre-fix, origin/main a118f8e17e) (execute: agent__NAME__SESSIONID), :1122-1125 (pre-fix, origin/main a118f8e17e) and :1186-1187 (pre-fix, origin/main a118f8e17e) (__rewind__ / __resume__), :1444 (pre-fix, origin/main a118f8e17e) (__retry__) and :1784 (pre-fix, origin/main a118f8e17e) (resumeWorkflow __resume__): every id interpolates the raw session id and agent name; no validation or encoding of either anywhere in runtime-cloudflare/src outside tests; runtime-cloudflare/src/workflow.ts:1024-1025 (pre-fix, origin/main a118f8e17e) (__respawn-N), :2674-2675 (pre-fix, origin/main a118f8e17e) with :2594 and :2665 (pre-fix, origin/main a118f8e17e) (ephemeral sub-agent: agent__TYPE__SESSION__sub__CALLID), :3445-3446 (pre-fix, origin/main a118f8e17e) (persistent child, whose session id is PARENT-agent-NAME, core/src/types/session.ts:930-931 (pre-fix, origin/main a118f8e17e)), runtime-cloudflare/src/workflow.ts:3753 and :3769 (pre-fix, origin/main a118f8e17e) (__continue__STEP__CALLID) and :3939 and :3944 (pre-fix, origin/main a118f8e17e) (__resume__STEP__CALLID): body-side ids grow with each nesting level; runtime-cloudflare/src/executor.ts:614-615 (pre-fix, origin/main a118f8e17e) reaches the engine id check: Miniflare 5.20260925.0-alpha (the runtime-cloudflare and e2e devDependency) dist/src/workers/workflows/binding.worker.js:1457 (MAX_WORKFLOW_INSTANCE_ID_LENGTH = 100, ALLOWED_STRING_ID_PATTERN ^[a-zA-Z0-9_][a-zA-Z0-9-_]*$), :1463-1464 (isValidWorkflowInstanceId) and :4918-4920 (create() throws WorkflowError Workflow instance has invalid id; the same check in createBatch, :4985-4987); runtime-cloudflare/src/executor.ts:614-617 (pre-fix, origin/main a118f8e17e) (execute() does not wrap create(), so the raw WorkflowError reaches the caller; resume() and resumeWorkflow roll back their claim and rethrow it unchanged, :1143-1144, :1195-1197, :1797-1812); core/src/errors/classify-error.ts:90-94 (pre-fix, origin/main a118f8e17e) (an unrecognised Error classifies as framework_internal_error); core/src/tools/companion/helpers.ts:44-55 (pre-fix, origin/main a118f8e17e) (MAX_CHILD_NAME_LENGTH = 128 with an intentionally unrestricted charset; the comment says 128 keeps the derived ids well under platform limits, false on CF Workflows, whose whole instance id is capped at 100); contrast: core/src/types/client-tool-submit.ts:108 and :12 (the submit envelope accepts session ids up to 256 characters of any charset); runtime-temporal/src/executor.ts:396 (pre-fix, origin/main a118f8e17e) and runtime-dbos/src/workflow-ids.ts:21-58 interpolate raw ids too but impose no charset limit in repo; docs/workspaces/cloudflare-filestore.md:394 (the workspace provider hex-encodes session ids for a constrained Cloudflare API, the precedent the runtime does not follow); no page under docs/runtimes or docs/guide states a CF Workflows session-id constraint; inferred: the constraint is read from the Miniflare engine, which is built from Cloudflare's shared Workflows validator source; production Workflows is expected to enforce the same rule but no installed package documents it for production. Not run: a nested child id past 100 characters failing inside the parent's step.do is inferred from source; related: plan 0c-cfw (the CF Workflows executor lane) changes the conformance cell id to a short base36 id on every lane so harness session ids stay inside this constraint, and adds a dedicated scenario that mints an invalid id on purpose; CP-15 (untyped engine-facing failures on the same executor); related: CP-98 (CF Workflows step names interpolate the same unbounded names with no length bound, and the engine caps them at 256 characters; this fix does not cover them); severity S2: a loud but untyped failure for inputs every other runtime accepts, with no documented constraint; for nested sub-agents and companions the failure surfaces mid-run inside the parent workflow rather than at the caller; runtime-cloudflare/src/instance-id.ts:84 and :108 (origin/main 5bb1eaaf2b) (fix evidence, CF-D1 MR 1a: buildWorkflowInstanceId / buildWorkflowEventType keep the readable id when the engine accepts it and, for instance ids, no part contains '__' or a leading / trailing '_'; otherwise h_<first 32 hex of SHA-256 of the JSON-encoded parts>_<end of the id>, <= 100 chars, deterministic and injective) used at every instance-id site in executor.ts and workflow.ts and at every waitForEvent / sendEvent type (Cloudflare docs, workflows/build/events-and-parameters: event types are capped like instance ids and '.' fails with workflow.invalid_event_type; Miniflare does not check them), proven by runtime-cloudflare/src/__tests__/instance-id.test.ts:16, :165 and :188 (unit, incl. injectivity) and runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:418 (origin/main 5bd30e20da) ("T6 (CP-90): ... runs two turns to completion" for a dotted/@/colon id and a 150-char id, red on main: 'Workflow instance has invalid id') and :637 (origin/main 5bd30e20da) ("T6 (CP-90): a nested sub-agent whose logical instance id exceeds 100 chars completes"); conformance: every cfw-workflows-d1 cell runs session ids that contain '.' (the scenario id) and per-cell agent names of about 70-80 characters, so every root instance id is past 100 characters or carries an invalid character and goes through the encoded form. smoke.hello-completes, history.prior-turns (two turns) and history.subagent-sees-input (an ephemeral sub-agent child) complete in all 5 full lane runs of the plan 0c MR-2 lane flip (origin/main d2e95ad181 product code). The companion ids are exercised only by history.companion-continue: turn A's child completes via __finish__, so turn B's companion__sendMessage takes the __continue__ path (runtime-cloudflare/src/workflow.ts, the sendMessage-to-completed-child branch, buildWorkflowInstanceId kind 'continue') and .child-call-reached passes in all 5 runs. That cell carries a CP-23 ledger fragment, so it cannot be in provenBy. The __resume__ id (a sendMessage to an INTERRUPTED child, the companionResult.resume branch, kind 'resume') is never reached: the parent's stop never interrupts the child (CP-23); conformance: owner batch 15 (93cb5c49bf product code; every red and green half re-run on f083dafb67 after the !316 rebase, same verdicts) proves the encoder on the child sites. history.companion-continue-completed (new; turn A's companion completes via __finish__, turn B's companion__sendMessage re-consults it on a __continue__ instance, with no stop, so no CP-23 fragment touches its cfw cell) passes on every lane but dbos-postgres (CP-69's blocking spawn) and on cfw-workflows-d1. The fix predates both cells, so each red half is a minimal mutation of the encoder's use at one site, not an MR revert (working tree only, rebuilt, then reverted). With the __continue__ id built from the raw agent__TYPE__SESSION__continue__KEY template instead of buildWorkflowInstanceId (runtime-cloudflare/src/workflow.ts:4281-4286 (pre-fix, origin/main 487090d3fb)), the cfw cell failed .turn-b-settles, .turn-b-completes, .child-call-reached, .child-settles and .child-answered: the continuation instance was never created, so the companion never got turn B. With the ephemeral sub-agent's spawn id built the same raw way (workflow.ts:3090-3095 (pre-fix, origin/main 487090d3fb)), history.subagent-sees-input's cfw cell failed history.subagent-outcome-settles, .parent-completes, .child-called and .child-sees-parent-input. The root id is not in provenBy: with buildWorkflowInstanceId returning the raw logical id everywhere (instance-id.ts:96-102), execute() itself throws 'Workflow instance has invalid id', which the executor driver reports as a cell crash, not a verdict (smoke.hello-completes, history.subagent-sees-input and history.companion-continue-completed all crashed). Per the owner ruling the record flips on the shared encoder; the __resume__ id (a sendMessage to an interrupted child) is reached only once CP-23 is fixed on CF Workflows | source | partly-inferred | fixed (2 cells) |
| CP-91 | S1 | CF Workflows never marks a session completed when a turn ends on plain text (any agent without an outputSchema): the text branch returns isComplete without setting a status, and the completion path ends the stream and finalizes only the RUN row. The workflow instance is complete and the run row completed, but the session stays active, so onAgentComplete never fires and every handle's result() reports { status: 'failed', error: 'Unknown error' } for a run that produced its answer. The stranded session then refuses resume() as already running and retry() as not failed. It affects root turns, resume and retry runs, and sub-agent children; the CF DO, runtime-js, Temporal and DBOS persist completed for the same turn. | runtime-cloudflare/src/steps.ts:1631-1647 (pre-fix, origin/main a118f8e17e) (executeAgentStep text branch, no outputSchema, non-error stop: saveAgentState + writeTerminalCheckpoint, then return isComplete: true; state.status is never set); core/src/orchestration/step-processor.ts:234-247 (origin/main 4c3cc736b9) (case 'text' plans statusUpdate: null, so the statusUpdate commit at runtime-cloudflare/src/steps.ts:1473-1481 (pre-fix, origin/main a118f8e17e) never runs for a text step); runtime-cloudflare/src/workflow.ts:1973-2030 (pre-fix, origin/main a118f8e17e) (isComplete path: endAgentStream :1995-2005, finalizeRun status completed :2008-2013, returns { status: 'completed', output } :2026-2030; no session status write); runtime-cloudflare/src/steps.ts:5973-5987 (pre-fix, origin/main a118f8e17e) (finalizeRun writes only stateStore.updateRunStatus(runId, ...), the run row, never the session status); runtime-cloudflare/src/steps.ts:3754-3758 (pre-fix, origin/main a118f8e17e) (endAgentStream sets status completed only when finishWithOutput is defined) and :3761 (pre-fix, origin/main a118f8e17e) (onAgentComplete fires only if state.status === 'completed', so it never fires for a text completion); runtime-cloudflare/src/executor.ts:455-498 (pre-fix, origin/main a118f8e17e) (mapWorkflowStatusToAgentResult requires status complete AND a completed session for 'completed', :474 (pre-fix, origin/main a118f8e17e); otherwise failed with status.error?.message ?? state?.error ?? 'Unknown error', :484 (pre-fix, origin/main a118f8e17e); the workflow's own returned status completed is never read); used by createActiveHandle.result() :1964-1991 (pre-fix, origin/main a118f8e17e) and createResumedHandle.result() :1852-1881 (pre-fix, origin/main a118f8e17e); the reconnect handle from getHandle() does the same at :2135-2151 (pre-fix, origin/main a118f8e17e) with errorDetail undefined; runtime-cloudflare/src/executor.ts:984-989 (pre-fix, origin/main a118f8e17e) (resume continue / with_message: an active session with no pending calls throws AgentAlreadyRunningError, so the stranded session cannot be resumed), :1286-1296 (pre-fix, origin/main a118f8e17e) (retry()'s CAS from failed fails: Cannot retry: agent status is active) and :2028-2048 (pre-fix, origin/main a118f8e17e) (canResume() reports canResume: true for the same session); runtime-cloudflare/src/workflow.ts:707-708 (pre-fix, origin/main a118f8e17e) and :729 (pre-fix, origin/main a118f8e17e) (an active session maps to running, core/src/types/session.ts:956-957 (pre-fix, origin/main a118f8e17e), and is treated as resumable, so a later turn on a stranded session takes the resume branch rather than the completed-continuation branch); runtime-cloudflare/src/steps.ts:5885-5890 (pre-fix, origin/main a118f8e17e) (a non-blocking persistent child is delivered only once its persisted status is completed or failed); runtime-cloudflare/src/__tests__/history-b2.wf-noiso.test.ts:58-68 (pre-fix, origin/main a118f8e17e) (the test header already records that a plain text completion never persists the session status and works around it with an outputSchema agent); :842-868 (pre-fix, origin/main a118f8e17e) (the RM-33 text-only case asserts only the workflow return and the checkpoint, not the session status); inferred: (unreachable) a text-completing non-blocking companion would therefore never be delivered to its parent, and a re-consult of one would take the resume branch; core refuses such a companion at definition (see the core/src/types/agent.ts line below), so neither case can occur; contrast: core/src/orchestration/step-iterator.ts:1207-1219 (pre-fix, origin/main a118f8e17e) and runtime-js/src/run-loop.ts:3228 (pre-fix, origin/main a118f8e17e) (runtime-js, and the CF DO, which runs its executor, persist completed for a plain text terminal turn); runtime-temporal/src/workflow.ts:1689-1704 (pre-fix, origin/main a118f8e17e) and runtime-dbos/src/workflows/shared.ts:2904-2926 (pre-fix, origin/main a118f8e17e) (Temporal and DBOS persist completed); runtime-cloudflare/src/steps.ts:1792 (pre-fix, origin/main a118f8e17e) and :3755 (pre-fix, origin/main a118f8e17e) (the CF Workflows forced-completion and finishWith paths do set completed); live: peer-reported real-provider smoke after !279 (the program's live-smoke report of 2026-09-30; main f27a0c570e, claude-haiku-4-5 through llm-vercel, wrangler dev with D1): a first turn reported failed (Unknown error) with the session active and roles U A; a second scenario reported failed (Unknown error) with a correct answer; resume and retry runs reported failed (Unknown error) with the session active; a resume after an interrupt threw Agent session ... is already running (status: active). The D1 instance status was complete with output status completed, and the run row was completed. A __finish__ turn did persist completed; related: DI-18 (the same failed / 'Unknown error' surface from a different mechanism: the result() poll treats a non-terminal instance status as done and memoizes it; this record gives it deterministically for every text completion, with no poll race); CP-88 (the second execute() collision; once it is fixed, a stranded active session would route the next turn into the resume branch); plan 0c-cfw (the CF Workflows executor lane) requires this fix before the lane flips (user decision, 2026-09-30), because nearly every cell would fail at its outcome; severity S1: every non-outputSchema turn on CF Workflows (root, resume, retry, sub-agent) reports failure to its caller although the answer was produced, the session is stranded active so it can be neither resumed nor retried, and onAgentComplete never fires; runtime-cloudflare/src/steps.ts:1639-1650 (origin/main 5bd30e20da) (fix evidence, CF-D1 MR 1a: executeAgentStep's text branch, no outputSchema and a non-error stop, sets state.status = 'completed' with output undefined in the SAME saveAgentState as the step count; a second save would carry a stale version), proven by runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts:261 ("T1 (CP-91, workflow body): a plain-text turn persists completed and fires onAgentComplete once", red on main: the session stays 'active'), :277 ("T1 (CP-91): ... result() is completed", through the real CloudflareAgentExecutor) and :529 (origin/main 5bd30e20da) (3+ turns mixed with an interrupt + resume) on the real Workflows engine + D1 under vitest-pool-workers; runtime-cloudflare/src/__tests__/history-b2.wf-noiso.test.ts RM-33 text-only case now also asserts the persisted status. The plan 0c CF Workflows lane proves the result() and persisted-status half only (see the conformance note); conformance: the result and persisted-status half is proven on cfw-workflows-d1 in all 5 full lane runs of the plan 0c MR-2 lane flip (origin/main d2e95ad181 product code): smoke.hello-completes (.completes: result() completed; .persisted: the session completed) for a root text turn, and the text turns that end lifecycle.retry-message (.retry-completes, a retry) and smoke.client-tool-completes (.completes, a resume after a client tool). Not asserted by any executor-tier scenario: that onAgentComplete fired once on a completed run, and a text-completing sub-agent child's persisted status; conformance: lifecycle.text-turn-completes (executor) passes on cfw-workflows-d1 and on every Node lane: a plain-text turn's outcome is completed (.completed), the session is persisted completed (.persisted-completed), and onAgentComplete fired exactly once, read after the hook quiet period (.on-agent-complete-once). The fix (CF-D1 MR 1a, 5bd30e20da, moved into finishRun's terminal hooks and the K5 commit by !288, bd26dc9b57) predates the cell, so its red half is a minimal mutation of the mechanism, not an MR revert: with finishRun firing its complete hooks only for a finishWith completion (runtime-cloudflare/src/workflow.ts:837 (pre-fix, origin/main e96f88bc98), working tree only, rebuilt, then reverted), the cfw cell failed .on-agent-complete-once (0 firings), as CP-91's plain-text end did; conformance: the persisted-status half has its own red half too (owner batch 15 review): with finishRun skipping its K5 terminal state commit (runtime-cloudflare/src/workflow.ts:873 (pre-fix, origin/main e96f88bc98)) for the per-step completion path (tag completed, :2618 (pre-fix, origin/main e96f88bc98); working tree only, rebuilt, then reverted), the session stays active as CP-91 left it, and the cfw lifecycle.text-turn-completes cell failed .persisted-completed (its persisted-terminal poll, .settles, never saw a terminal status), as smoke.hello-completes failed .persisted and .persisted-terminal. .completed, the handle result(), stayed completed under that mutation: result() now reads the run's own terminal status, so the pre-fix result() failure is not re-shown red; core/src/types/agent.ts:504-509 and core/src/types/tool.ts:841-846 (core refuses a persistent companion with no outputSchema when the agent is defined, since 672680871a, and a sub-agent with none when its tool is built by createSubAgentTool, since c360f713e3; a remote sub-agent's text completion comes back to the consumer as a structured tool error, never through CF Workflows' child delivery; so a text-completing CHILD, the sub-agent half of the pending reason and the inferred non-blocking companion line, cannot be built, and this record is fixed without it) | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-92 | S2 | DBOS: after a caller submits only part of a parallel client-tool batch, resume() attaches to the workflow that is still blocked in DBOS.recv for the remaining call and returns a handle with skipPendingClientToolPolling, assuming the submit just woke the run toward completion. That handle's result() therefore never reports suspended_client_tool for the remaining call: it does not settle until that call is submitted or its client-tool timeout fires (5 minutes by default; approval gates have no deadline), and then reports the run's later outcome. The held state is correct (the remaining call stays pending), but the caller cannot learn from the handle that the run is waiting on it. | runtime-dbos/src/lifecycle/resume.ts:236-238 (pre-fix, origin/main 5bd30e20da) and :274-352 (pre-fix, origin/main 5bd30e20da) (an active session that still has a pending client call, plus a PENDING workflow: resume() attaches to the workflow blocked on DBOS.recv and returns a ResumedHandle with skipPendingClientToolPolling: true, :342-352; the comment assumes the submit just woke the workflow toward completion and that the pending entry is merely stale); runtime-dbos/src/handles/active-handle.ts:153-162 (pre-fix, origin/main 5bd30e20da) (with skipPendingClientToolPolling, result() awaits dbosHandle.getResult() only; the pending-client-tool poll that would return suspended_client_tool is skipped); runtime-dbos/src/client-tool-resolver.ts:61-78 (pre-fix, origin/main 5bd30e20da) and runtime-dbos/src/workflows/shared.ts:445 (pre-fix, origin/main 5bd30e20da) (the workflow stays in DBOS.recv for the remaining call until it is submitted or tool.clientToolTimeoutMs ?? DEFAULT_CLIENT_TOOL_TIMEOUT_MS elapses, 5 minutes per core/src/constants.ts:117, after which the call resolves as a client-tool timeout and the run continues on an error result); approval gates default to no deadline (runtime-dbos/src/workflows/shared.ts:846 (pre-fix, origin/main 5bd30e20da), Number.MAX_SAFE_INTEGER); live: real-provider smoke after !279 (main f27a0c570e, claude-haiku-4-5 through llm-vercel, dbos-postgres): the model emitted two parallel client calls; after submitting only the first, resume() returned a handle whose result() did not settle within the 60 s bound (TIMEOUT(60000ms) on the partial resume) while the session held correctly (one call still pending, no model call during the partial resume); the second submit then woke the run, which completed before the follow-up resume() (that refusal, not in a resumable state (current: completed), is CP-64); related: CP-64 (the complementary branch: once no call is pending, resume() throws instead of attaching, and the executor offers no way to attach to the continued run); CP-65 (the Temporal analogue: the resume handle never settles when the session is still suspended after the drain, including after a partial batch submit); CP-82 (runtime-js and the CF DO do not hold a partial batch at all: they continue the run with a synthesized failure for the unanswered call); severity S2: a caller that resumes after a partial submit waits up to the client-tool timeout (unbounded for approval gates) on a handle that never tells it the run is waiting on the remaining call; the held state itself is correct and nothing is lost; conformance: submit.partial-batch-resume on dbos-postgres — .with-message-settles and .continue-settles hang on origin/main 5bd30e20da and settle with !288; runtime-dbos/src/lifecycle/resume.ts:150-156 (as before WA MR2, origin/main cae966c0a7) (fix evidence, !288: continue / with_message on a partial batch are refused typed before any attach) and :568-574 (the attached handle races the pending-client-tool poll, so it settles suspended_client_tool instead of waiting for the workflow result) | source, live-repro | verified | fixed (1 cells) |
| CP-93 | S3 | DBOS keeps a session active, not paused, while its run waits inline on client tools: JS and Temporal read paused for the same suspension, so a status-based caller sees the runtimes diverge. | runtime-dbos/src/steps/run-lifecycle.ts:86-93 (!288: finalizeRun maps the suspended_* run statuses to no session transition and defers the 'paused' write to a commitSuspendedStep that runtime-dbos never calls, so no DBOS path writes 'paused' for an inline client-tool wait; the same on origin/main 5bd30e20da); runtime-dbos/src/lifecycle/resume.ts:219 (!288: the #105 design, an 'active' session suspended INLINE on a client tool whose workflow is PENDING on DBOS.recv; resume() keys on 'active' plus pending client-tool calls); conformance: submit.partial-batch on dbos-postgres — .stays-paused: the baseline read (both calls pending, version quiescent) is already active, before any submit, and stays active through the window; .c2-still-pending and .nothing-synthesized pass (the held state is correct); conformance: submit.partial-batch on dbos-postgres — .execute-refused-pending: a new execute() over the held batch is refused as a live run ("Agent session '…' is already running (status: active)", runtime-dbos/src/lifecycle/run-start-event.ts:333-343, !288: assertNoOpenOwner refuses while the run's owner workflow is still open, which it is while DBOS waits inline), not with the pending-client-tool message JS and Temporal give, because the session reads active (the same divergence); contrast: submit.partial-batch .stays-paused and .execute-refused-pending pass on js-* and temporal-*; related: CP-82 (the partial-batch contract), CP-64 (the same cell's .completes-after-c2 on DBOS); severity S3: a status misreport with no data loss. The held state is correct (both calls stay pending and nothing is synthesized) and submit-then-resume still completes the run, but a caller or UI that keys on the session status shows a DBOS run awaiting client input as running, unlike JS and Temporal for the same suspension | source, probe | verified | open |
| CP-94 | S2 | DBOS evaluates a function systemPrompt in execute(), before any run starts (registerAgent → serializeAgent calls it with empty state), so a prompt that throws makes execute() itself throw after the session row is created. No run starts or fails: the session never reaches a terminal status, no failed stream is written, onAgentFail never fires, and every later execute() throws the same error. The persistent-mode entry and retry() resolve the prompt at the same seam. | runtime-dbos/src/lifecycle/execute.ts:261 (pre-fix, origin/main bd26dc9b57) (createSession writes the session row) then :292 (pre-fix, origin/main bd26dc9b57) (registerAgent(agent), before any run-start commit, workflow or stream); runtime-dbos/src/serialize.ts:69-72 (pre-fix, origin/main bd26dc9b57) (serializeAgent resolves a function systemPrompt by calling it with empty state; the comment calls the result a fallback only, but a throw escapes execute()); runtime-dbos/src/lifecycle/execute.ts:437 (pre-fix, origin/main bd26dc9b57) (the persistent-mode entry registers the agent at the same seam) and runtime-dbos/src/lifecycle/retry.ts:101 (pre-fix, origin/main bd26dc9b57) (retry() likewise); conformance: lifecycle.loop-throw-fails-run on dbos-postgres — .first-failed, .persisted-terminal, .session-failed, .stream-failed, .on-agent-fail-once and .follow-up-accepted: the first outcome is threw with the prompt's own message (execute() / result() rejects instead of resolving a failed result); after the 10 s persisted-terminal bound the session reads status active, version 0, no messages and no checkpoints, and the store holds no run row for it; the follow-up execute() is also threw with the same error. Stable 5/5 on !287 and failing the same way on origin/main bd26dc9b57; contrast: lifecycle.loop-throw-fails-run passes on js-memory, js-redis and js-postgres (CP-71, fixed by CF-D1 MR 1b, !287) and on temporal-memory, temporal-redis and temporal-postgres; related: MA-23 (the same serializeAgent seam: DBOS resolves a function systemPrompt once with {} and sends that string to every LLM call; a per-step fix for MA-23 would likely move this record's throw from execute() into the run, where it should fail the run); related: CP-71 (runtime-js: a throw inside the loop stranded the session; there the run had started, here no run starts); severity S2: the caller does get the prompt's error, from execute() itself, so the failure is not silent to it; but the durable session is left active with no run, no terminal stream and no onAgentFail, so every other reader sees a live-looking session that never ends. Not shown unrecoverable: with no live run, a later entry after the prompt is fixed is not blocked (inferred, not exercised); conformance: lifecycle.loop-throw-unclassified-detail and lifecycle.loop-throw-cause-kept on dbos-postgres (owner batch 15, f083dafb67 product code) fail the same way: execute() throws the prompt's error (outcome threw) instead of a failed run, so .failed fails and .persisted-terminal and the verdict id fail as its preconditions; conformance: lifecycle.loop-throw-not-retried fails .fails-with-prompt-error on dbos-postgres (main 8dc3a5c967 plus owner batches 24 to 28, in the owner lock window of 2026-10-13 with every lane run in full): execute() itself throws 'conf systemPrompt exploded' (outcome threw), this record's shape | source, probe | verified | open |
| CP-95 | S1 | Remote sub-agents: re-sending a remote entry (/start or /resume) is still unsafe until the remote transports carry D-2 request keys. The double run this record was filed for is closed by WE MR C1: an entry is sent exactly once, and a child that was started is attached to, never started again. What remains: (a) after an entry whose outcome is unknown (a thrown fetch, an untyped 5xx), the dispatch reads the child status once and attaches to a child that answers, never re-sending; when the read finds no child the call fails typed (transport_remote_entry_outcome_unknown), and a /start still in flight can then land and leave an orphan child; (b) a start that succeeded but whose ref stream-identity write was lost re-POSTs /start on re-entry (the identity is recorded after the start, best-effort); (c) a re-sent /resume is not yet safe. All three close only when the transports send request keys (D-2 MR 2), so a re-sent entry attaches; DOAB honours keys since !366. Narrowed (owner, 2026-10-24): (a) and (b) are closed by D-2 MR 2 (9a27ac7da3, a lost /start is re-sent with the same request key); only (c) remains, on CF Workflows, whose resumeDeadProducer path sends its /resume with no request key. | core/src/transport/http-transport.ts:129-143 (start() → fetchWithRetry), :348-351 (a 4xx is returned to the caller), :353-357 (a 5xx is retried after a backoff), :360-364 (a thrown fetch error is retried); maxRetries 3 and retryBaseDelayMs 1000 by default, :121-122. Every retry re-sends the identical /start body; runtime-cloudflare/src/do/worker-handler.ts:198 (createRemoteAgentWorkerHandler forwards the protocol /start to the DO route /subagent/:agentType/start); runtime-cloudflare/src/do/durable-object-agent-base.ts:3729-3773 (origin/main 62d3661b45) (the /subagent start bridge rebuilds the request as a native /start with the same message and state, input: { message, state }, then calls handleStart) and :3775-3800 (origin/main 62d3661b45) (the protocol already has a separate route for a further turn: /resume with a message continues with appendMessages); runtime-cloudflare/src/do/durable-object-agent-base.ts:2206-2218 (origin/main 62d3661b45) (/start answers ALREADY_RUNNING while this instance is executing) and :2221-2233 (origin/main 62d3661b45) (the persisted-state check answers ALREADY_RUNNING only for status 'active'). A 'completed' session, or a 'failed', 'paused' or 'interrupted' one, passes both checks. It gets a fresh runId (:2181 (origin/main 62d3661b45)) and reaches executor.execute() with the same sessionId (:2345 (origin/main 62d3661b45)); runtime-cloudflare/src/do/durable-object-agent-base.ts:2345 (origin/main 62d3661b45) (the execute() it calls is the runtime-js executor's: runtime-js/src/js-agent-executor.ts:4641-4651 (origin/main fdc56520a5) takes the implicit-continuation path for an existing session; :4711 (origin/main fdc56520a5) lets the entry's input state replace the loaded customState; :4754 (origin/main fdc56520a5) appends the input's message to the existing history as a new user turn). For the DO's own chat /start this is the intended multi-turn entry; behind the remote protocol a transport retry hits the same path; core/src/orchestration/remote-sub-agent-dispatch.ts:467-493 (the shared dispatch calls transport.start once, and only a RemoteAgentAlreadyRunningError counts as attach), :402-434 (the pre-stream getStatus check, which returns a child that has already completed without calling /start, runs before start() and so does not cover start()'s own retries) and :515-521 (it then streams the child from fromSequence undefined, i.e. from 0, for a fresh ref); live: probe (this session, origin/main 4d9b433551, no product change; a throwaway e2e test on the cross-service harness under Miniflare, deleted after use). Setup: scenario 'paving', which has no held step, so the child completes in about 100 ms. A consumer service binding forwarded the first /start to the real producer, then returned 503 in place of its 200; the runtime-js dispatch's transport.start (runtime-js/src/execution/remote-subagent-executor.ts:435-440, state: call.input) retried it. Producer: the first run emitted its output 92 ms after its run_started, and the retry's run started about 1 s after that; both POSTs returned 200. The child session holds two researcher run_started frames (run turn 2, startSequence 34) and 14 persisted messages, with the user message {"query":"alpha"} and the whole turn repeated, so record_note executed twice. The final state is { notes: ['alpha'], progress: 0.5 }, not ['alpha', 'alpha'] / 1.0, so turn 2 rebuilt the state from the request's state instead of building on turn 1's commit. Consumer: the parent completed, with one subagent_end carrying the second turn's output. Its stream got all 54 child chunks with childSequence 1..54, both run_started frames and record_note's tool_start twice. Its afterSubAgent fold received the second turn's state; inferred: (unverified) with a real model the second turn is a new sample over a transcript in which the user asked twice, so the output the parent receives can differ from the first run's, and a child tool with an external side effect performs it twice. The scripted model in the probe returned the same output both times. Not exercised: a 'failed' child (the same gate admits it, so a retry would re-run a failed child as a new turn), and a DOStubTransport consumer (it does not retry /start, runtime-cloudflare/src/do/do-stub-transport.ts:103-122, so it cannot hit this); core/src/orchestration/remote-sub-agent-dispatch.ts:235-266 with :599-621 (fix evidence, WE MR C1, the CP-95 interim: readChildAfterUnknownStart reads the child's status once after a start whose outcome is unknown; a child that answers is attached to, otherwise the call fails typed; /start is never re-sent), the same read at the runtime-js, DBOS and CF Workflows dispatch sites (their unit pairs: 'outcome is unknown'); live: e2e/src/__tests__/cross-service-do-do-recovery.integ.test.ts:150-214 (WE MR C1, 'dispatch-driven lost /start answer → status read → attach (CP-95 interim)', formerly the D7 lost-POST test): red on C1 before the status read (CI job 17251692054, the parent got no output), green 3 of 3 at !387's head 53055298a3, and red again with runtime-js's status-read attach disabled (WE-2's lock run, 2026-10-17); contrast: agent-server/src/agent-server.ts:386-400 (an agent-server producer, which hosts the JS, Temporal, DBOS and CF Workflows executors, answers /start on a completed or failed session with 409 ALREADY_COMPLETED). There the retry fails the delegation loudly instead of running it twice; that is CP-78; related: CP-78 (whose DO contrast left "what a retried /start does to a completed DO session" unassessed; this record answers it). Also docs/guide/cross-service-remote-agents.md "At-least-once POSTs and the ALREADY_RUNNING → attach rule" (says the protocol makes a lost-response retry safe, which holds only while the child is still running), docs/dev/follow-ups.md FU-EVICT-02 (CI-health-3, !292: the original wake-race e2e hit this shape when its child finished in ~15 ms, and the test was re-gated) and the design spec docs/superpowers/specs/2026-07-09-cross-service-remote-agents-design.md §3.2 (ALREADY_RUNNING → attach; it specifies no answer for a completed session). The fix contract is an open user decision for the cross-service bucket: what /start should answer for a session that has finished (attach to the completed result, or a typed ALREADY_COMPLETED that CP-78 then attaches to), and how it tells a retry from a deliberate new turn; severity S1: the consumer silently gets a second run's result in place of the first. The child's work runs twice (its model calls are paid for twice and its tools run twice), the first run's committed state is overwritten, its transcript holds a duplicated user turn, and the parent's stream carries both runs' chunks under distinct provenance keys, so the design spec's 'no duplicate effective chunks' requirement does not hold. Nothing reports an error, unlike CP-78's loud failure (S2). The trigger is narrow: a lost /start response, then a child that finishes within the retry backoff. But the effect is a silently wrong result reaching the consumer, plus repeated side effects; core/src/transport/http-transport.ts:105-111 and :283-325 (fix evidence, WE MR C1, the double-run half: start() and resume() go through postEntry, which POSTs an entry exactly once; a thrown fetch is a RemoteEntryOutcomeUnknownError and a non-2xx goes through remoteEntryProtocolError, a 5xx that is not the host's typed retryable answer being outcome-unknown) with core/src/orchestration/remote-sub-agent-dispatch.ts:524-531 (fix evidence, WE MR C1, the D-2 guard: a child the ref says was started is re-attached, never re-POSTed) and :576-579 (the started marker is recorded right after the start); the same guard on runtime-js, DBOS and CF Workflows (WE MR C1 commits de98730072, 2136ab8525, 56e6ae9ad3). There are no transport retries for /start or /resume any more, so a retried /start can no longer land after the child completed; related: what remains is peer-reported by WE-2 (WE MR C1, 2026-10-16): the outcome-unknown failure and the lost-identity re-POST, both closed by request keys on the remote transports (D-2 MR 2); runtime-cloudflare/src/do/durable-object-agent-base.ts honours request keys after !366; core/src/orchestration/remote-sub-agent-dispatch.ts:241-269 (fix evidence, D-2 MR 2 !404, origin/main 9a27ac7da3) (startRemoteChild: a /start carries the delegation's request key, and an answer that is lost (RemoteEntryOutcomeUnknownError) is re-sent with the SAME key at most KEYED_START_RESENDS times; the producer answers a keyed repeat from its key, attaching to the run it already started, never a second run) with core/src/store/child-request-key.ts:30 and :51 (origin/main 9a27ac7da3) (childStartRequestKey, childResumeRequestKey: the keys are derived from the delegation); runtime-js/src/execution/remote-subagent-executor.ts:475 (fix evidence, origin/main 9a27ac7da3), runtime-dbos/src/steps/execute-remote-subagent.ts:506 (fix evidence, origin/main 9a27ac7da3) and runtime-cloudflare/src/steps.ts:4596 (fix evidence, origin/main 9a27ac7da3) (every runtime's dispatch starts the child through startRemoteChild); Temporal through core's executeRemoteSubAgentDispatch (runtime-temporal/src/remote-sub-agent-activity.ts:96 (fix evidence, origin/main 9a27ac7da3) → core remote-sub-agent-dispatch.ts:579); live: e2e/src/__tests__/cross-service-do-do-recovery.integ.test.ts:174 and :245 (D-2 MR 2, !404, merged at 9a27ac7da3): a dispatch-driven lost /start answer → keyed re-send → attach, and a lost answer whose child already FINISHED → the keyed re-send replays its output (peer-reported, WE-2); related: so (a) and (b) are closed in code by D-2 MR 2 (a re-sent /start is keyed and attaches; the DO's missing stream identity, CP-141, now re-sends a keyed /start instead of starting the child again); (c), a re-sent /resume, uses childResumeRequestKey. The record stays open until a conformance cell witnesses it: the suite has no cross-service producer topology yet; related: (a) and (b) fixed in code by 9a27ac7da3 (D-2 MR 2: remote sub-agent entries carry a request key), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): core/src/orchestration/remote-sub-agent-dispatch.ts:267-280 (origin/main fb9150cc16) (a lost /start is re-sent with the same key; used by runtime-js, DBOS, core and CF Workflows), both transports send X-Helix-Request-Key (core/src/transport/http-transport.ts:311 (origin/main fb9150cc16); runtime-cloudflare/src/do/do-stub-transport.ts:162 (origin/main fb9150cc16)), and both producers honour it; runtime-cloudflare/src/steps.ts:6261 (origin/main fb9150cc16) ((c), still open on CF Workflows: the resumeDeadProducer path calls transport.resume with only the session id, no request key, then throws StreamDropError at :6274 (origin/main fb9150cc16), so a platform retry of the step sends a second unkeyed /resume; runtime-js, DBOS and core key the same re-send); related: Fixer: WE-2 for (c) (orchestrator, 2026-10-24). The earlier note that (c) uses childResumeRequestKey does not hold for the CF Workflows site | source, probe, peer-reported | verified | open |
| CP-96 | S1 | A stale resume / retry entry runs on a session that finished after its status pre-check: the entry's status gate is not atomic with its run-start commit, so B reads A's fresh run id while the session still reads interrupted/failed, A completes, and B's commit (expecting that run id) passes the CAS and the idle-owner guard — B runs a second turn on the completed session (resume overwrites A's output; retry rewinds A's completed turn). | runtime-js/src/__tests__/cp96-stale-entry.test.ts (fix evidence, !295: the JS repro, red before the fix — runtime-js + InMemoryStateStore with a gate on appendMessages({ startRun }) holding A right after its commit until B reached its own, then B until A completed: resume continue / with_message gave runs [interrupted, completed, completed] plus a second finish turn, and the output went a→b; retry rewound A's turn, leaving the transcript [user, assistant(b), tool(b)]; execute is benign, a later new turn being allowed on a completed session anyway); runtime-js/src/js-agent-executor.ts:3212-3216 (pre-fix, origin/main 5bb1eaaf2b) (resumeSession's status gate refuses completed / failed on its plan-time read) and :3221 (pre-fix, origin/main 5bb1eaaf2b) (assertNoLiveOwner runs only for a running session, so an interrupted session whose next run was just committed passes); :3373 (pre-fix, origin/main 5bb1eaaf2b) (the commit: commitRunStart); :3468 (pre-fix, origin/main 5bb1eaaf2b) (the entry state, written after the commit: writeEntryState); :3862 (pre-fix, origin/main 5bb1eaaf2b) (retry's gate: status !== failed); :3969 (pre-fix, origin/main 5bb1eaaf2b) (retry's commit); :4000 (pre-fix, origin/main 5bb1eaaf2b) (retry's entry state); runtime-js/src/run-start.ts:248 (pre-fix, origin/main 5bb1eaaf2b) (commitRunStart's appendMessages: the startRun expects the seen run by id only, so B's commit passes the CAS once A's run is idle); runtime-temporal/src/activities.ts:1083-1084 (pre-fix, origin/main 5bb1eaaf2b) (startRunCommit's resume plan gate, on its plan-time loadState) and :1134-1135 (pre-fix, origin/main 5bb1eaaf2b) (the retry gate); :1195 (pre-fix, origin/main 5bb1eaaf2b) (the commit: commitEntryRun); :1279 (pre-fix, origin/main 5bb1eaaf2b) (the entry state, written after the commit: writeEntryState); runtime-dbos/src/steps/run-commit.ts:649-650 (pre-fix, origin/main 5bb1eaaf2b) (the resume plan gate) and :705-706 (pre-fix, origin/main 5bb1eaaf2b) (the retry gate); :764 (pre-fix, origin/main 5bb1eaaf2b) (the commit: commitDbosEntryRun); :860 (pre-fix, origin/main 5bb1eaaf2b) (the entry state, written after the commit: writeEntryState); runtime-cloudflare/src/steps.ts:947-948 (pre-fix, origin/main 5bb1eaaf2b) (the CF Workflows resume plan gate) and :1004-1005 (pre-fix, origin/main 5bb1eaaf2b) (the retry gate); :1032 (pre-fix, origin/main 5bb1eaaf2b) (the commit: commitCfwEntryRun); :1141 (pre-fix, origin/main 5bb1eaaf2b) (the entry state, written after the commit: writeEntryState) and :1150 (pre-fix, origin/main 5bb1eaaf2b) (its active status); contrast: runtime-cloudflare/src/do/durable-object-agent-base.ts:603 (pre-fix, origin/main 5bb1eaaf2b) commitRunStartSync re-checks entry.allowedStatuses (:632-633 (pre-fix, origin/main 5bb1eaaf2b), entry_status_changed) and writes the entry state inside the same transactionSync as the commit (:617 (pre-fix, origin/main 5bb1eaaf2b) and :647-648 (pre-fix, origin/main 5bb1eaaf2b), R31) — the CF DO was already safe; live: e2e redis-cas-races.integ.test.ts › "Concurrent Resume Races › should allow exactly 1 executor to succeed when 5 race with 10ms stagger" — expected 2 to be 1 (!290 job 16919048157); intermittent, never red on main in the CI-health-2 sweep; runtime-temporal/src/__tests__/integration/cp96-entry-interleave.integ.test.ts and runtime-dbos/src/__tests__/integration/cp96-entry-interleave.integ.test.ts (probe: the interleave on real Temporal and DBOS was green before and after the fix — while A's workflow is open an owner check refuses B before its commit (Temporal: the executor's assertNoLiveOwner; DBOS: the entry step's assertDbosOwnerless), and when B's held commit lands after A finished the run-id CAS refuses it (expected_mismatch_idle → the C3 conflict)); runtime-temporal/src/__tests__/cp96-entry-gates.test.ts, runtime-dbos/src/__tests__/cp96-entry-gates.test.ts, runtime-cloudflare/src/__tests__/cp96-entry-gates.test.ts and runtime-cloudflare/src/__tests__/cp96-entry-gates.wf-noiso.test.ts (probe: the durable plan→commit unit tests — Temporal, DBOS, CF Workflows incl. the real Workflows engine — were red before the fix); conformance: control.stale-entry-loses-after-winner on js-memory, js-redis and js-postgres — both variants (resume continue vs resume continue on an interrupted session; retry vs retry on a failed session) hold A right after its run-start commit and B before its own (hold-run-start-commit), prove B's commit expects A's fresh run and that A is terminal in the store before B is released, then judge .b-rejected-typed, .a-output-intact, .one-new-run, .transcript-untouched and .stream-untouched: B loses as the C3 AgentAlreadyRunningError (expected_mismatch) and writes nothing; stable 5/5 on each of the three lanes. Red before the fix: with both CP-96 layers disabled locally (runtime-js commitRunStart's seen-status check and core applyRunStartRules' run-status CAS + session gate), every JS cell fails exactly .b-rejected-typed, .a-output-intact, .one-new-run, .transcript-untouched and .stream-untouched in both variants — B completes, the output goes ALPHA-WINNER → BRAVO-STALE, a second run record and B-stamped chunks appear, resume appends B's finish turn and retry rewinds A's turn away (assistant calls [retry-b-finish]); green again with the layers restored. The durable and CF Workflows lanes are declared gaps (no hold-run-start-commit; the probes above cover them); core/src/store/run-start.ts:241-242 (fix evidence, !295: the run-start commit carries the run status the entry saw and the session statuses it planned from, and every store decides both inside the commit — applyRunStartRules' run-status CAS, a mismatch being the C3 loser) and :293-295 (applyRunStartRules' session gate, a mismatch being RunStartRejectedError(entry_status_changed)); core/src/store/message-commit.ts:96 (pre-fix, origin/main e96f88bc98) (StartRun.expectCurrentRunStatus) and :105 (pre-fix, origin/main e96f88bc98) (StartRun.entryStatuses); store-memory/src/in-memory-state.ts:550 (pre-fix, origin/main 487090d3fb) (fix evidence, !295: each store feeds the gate its in-commit session status read), store-redis/src/redis-state.ts:2036 (pre-fix, origin/main 487090d3fb) (with store-redis/src/commit-script.ts:673 (pre-fix, origin/main 487090d3fb) re-validating it unconditionally in the script), store-postgres/src/state/postgres-state-store.ts:1379 (pre-fix, origin/main 487090d3fb) (the locked row), store-cloudflare/src/d1-state.ts:1207 (pre-fix, origin/main 487090d3fb) and runtime-cloudflare/src/do/do-state-store.ts:1307 (pre-fix, origin/main 487090d3fb) (DO-SQLite, one raw read); runtime-js/src/run-start.ts:298 (fix evidence, !295: commitRunStart checks the seen run by id AND status, matchesSeenRun, before committing); the durable builders runtime-temporal/src/run-entry.ts:314-315, runtime-dbos/src/run-entry.ts:299-300 and runtime-cloudflare/src/run-entry.ts:586-587 (origin/main fdc56520a5) (fix evidence, !295: expectCurrentRunStatus from the builder's own expected read, entryStatuses from the plan's session status); related: FU-STALE-ENTRY-RUNS-ON-COMPLETED-SESSION (docs/dev/follow-ups.md); related: CP-100 (the same entry's failure path on CF Workflows: an entry that fails BEFORE its run-start commit records an existing session failed, where Temporal leaves a session that already has a run untouched (its pre-commit failure path writes nothing to an existing session)); severity S1: a stale entry runs a whole extra turn on a finished session — resume overwrites A's output (a→b) and retry rewinds A's completed turn so it is lost from the transcript — and the caller is told nothing | source, live-repro, probe | verified | fixed (3 cells) |
| CP-97 | S1 | CF Workflows drops a stop pressed while the run's TERMINAL LLM step is in flight, and the run persists 'completed', not 'interrupted'. interrupt() sets the durable interrupt flag, but the workflow reads that flag only at the top of each loop iteration, before the next step starts. The in-flight model call is never aborted (CP-16), so the step finishes normally. When that step completes the run (a text answer with a normal stop, isComplete), the workflow commits it and goes straight to finishRun('completed') without reading the flag again. stop() reports success, the run's outcome and the persisted session both say completed, and the flag is left set. This is CP-56's mechanism (Temporal) on CF Workflows. | runtime-cloudflare/src/workflow.ts:2182-2194 (origin/main 4620e90228) (the main loop reads the durable flag with checkInterruptFlag at the top of each iteration, before the next step, and ends the run interrupted only from there). The only other reads in the workflow body are :2076-2083 (origin/main 4620e90228) (the resume path, before a deferred finishWith) and :2735-2738 (origin/main 4620e90228) (before spawning sub-agents); runtime-cloudflare/src/workflow.ts:2244-2257 (origin/main 4620e90228) (the LLM step) then :2328-2347 (origin/main 4620e90228) (result.isComplete: commitCurrentStep, the memory-extraction step, then finishRun('completed') at :2347, with no flag read in between). completeWithFinishWithOutput (:2004-2028 (origin/main 4620e90228)) and the stopWhen tail (:4351 (origin/main 4620e90228)) contain no flag read either; runtime-cloudflare/src/steps.ts:596 and :634 (as before WA MR2, origin/main cae966c0a7) (loadAgentState samples the flag into AgentState.aborted once, when the step starts) and :1470-1479 (origin/main 4620e90228) (executeAgentStep returns aborted only for a flag that was already set when the step began); runtime-cloudflare/src/steps.ts:2004-2025 (origin/main 4620e90228) (generateStep is called with no abortSignal, so a stop cannot end the held call: CP-16) and :2248-2256 (origin/main 4620e90228) (a text step with a normal stop reason on an agent without outputSchema returns isComplete: true); runtime-cloudflare/src/executor.ts:1881-1903 (origin/main 4620e90228) (interruptWorkflow sets the durable flag, then sends an interrupt event; only the sub-agent waits race that event, runtime-cloudflare/src/workflow.ts:2957 (origin/main 4620e90228), so a held LLM step never sees it); runtime-cloudflare/src/steps.ts:1138-1150 (origin/main 4620e90228) (a later root turn, an explicit resume or a retry clears a leftover flag at its run start; a child session's turn keeps it) and :5343 (origin/main 4620e90228) (writeRunInterrupted clears it at a run's end, on the interrupted path only). So after a lost stop the flag stays set on the completed session until the next such entry; live: probe (plan 0c MR-2 dry run, origin/main 4c3cc736b9, the cfw-workflows-d1 executor lane run in the working tree only, ScriptedModel; the cited code is unchanged at this record's base). stop.mid-llm-call failed .outcome-interrupted and .persisted-interrupted: the held call was released after stop() returned { ok: true }, and the run ended completed. The same held-terminal-step stop failed lifecycle.resume-with-message .resume-setup-interrupted, lifecycle.resume-continue .continue-setup-interrupted, lifecycle.resume-after-turn .setup-interrupted, lifecycle.from-checkpoint .setup-interrupted, stop.continued-turn .outcome-interrupted and stop.resumed-with-message-turn .setup-interrupted. The two resume cells' .resume-completes / .resume-message-reaches-model and .continue-completes / .continue-reaches-model then failed because resume() correctly refuses a session that is already completed ('Cannot resume: Agent already completed'). A second run of these cells failed the same ids; contrast: CP-56 (the same lost stop on Temporal, ledgered on temporal-* with these scenarios); stop.mid-llm-call passes on js-memory, js-redis and js-postgres, whose interrupt aborts the in-flight call; related: CP-56 (the Temporal analogue; this record is the CF Workflows half, a different fix site); CP-16 (no abortSignal reaches the model on CF Workflows, which is why the step runs to completion); CP-19 (a flag left set after a run ends: on CF Workflows a later root turn, resume or retry clears it, so a lost stop is not expected to interrupt the next root run; a child session keeps it. inferred: (unverified), no cell reached a next entry after a lost stop); severity S1, as CP-56: the user's stop is acknowledged ({ ok: true }) and then silently discarded. The run is reported and persisted completed, and its interrupted-only paths (run_interrupted, the interrupt context, resume()) never happen; resume() on the session then throws 'already completed'. Nothing tells the caller the stop was lost. The trigger is ordinary: any stop pressed while the last model call of a run is in flight; conformance: on cfw-workflows-d1 (the plan 0c MR-2 lane flip: 5 full lane runs over origin/main d2e95ad181 product code, the same failed ids in every run): stop.mid-llm-call .outcome-interrupted and .persisted-interrupted (stop() returned { ok: true }; the outcome and the persisted status are completed); lifecycle.resume-with-message .resume-setup-interrupted, lifecycle.resume-continue .continue-setup-interrupted, lifecycle.resume-after-turn .setup-interrupted, lifecycle.from-checkpoint .setup-interrupted, stop.continued-turn .outcome-interrupted and stop.resumed-with-message-turn .setup-interrupted (each setup run's stop was lost and the run persisted completed). The other ids of these cells fail only because of it. lifecycle.resume-after-turn, lifecycle.from-checkpoint and stop.resumed-with-message-turn fail theirs as preconditions of the setup id (failPreconditions). lifecycle.resume-with-message .resume-completes / .resume-message-reaches-model and lifecycle.resume-continue .continue-completes / .continue-reaches-model are not precondition-linked: they fail because resume() correctly refuses the session this record left completed ('Cannot resume: Agent already completed'), so no resumed run reaches the model; runtime-cloudflare/src/steps.ts:4999-5009 (pre-fix, origin/main 487090d3fb) (fix evidence, WA MR2 (!355) R2b, drive spec §4.3 / CP-59: commitTerminalState honours the run's stop; the run's own plan commits under refuseIfStopped, so a stop that lands while the terminal LLM step is in flight refuses the planned completed and the stop's terminal is committed instead) with runtime-cloudflare/src/workflow.ts:915-935 (pre-fix, origin/main e96f88bc98) (the workflow ends the run with what landed, interrupted for an interrupt) and :1236-1260 (pre-fix, origin/main e96f88bc98) (handleStop, the canonical stop terminal); conformance: fixed. Every cfw-workflows-d1 cell this record ledgered now passes its CP-97 ids, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12): lifecycle.resume-with-message, lifecycle.resume-continue, lifecycle.resume-after-turn, lifecycle.from-checkpoint, stop.continued-turn, stop.resumed-with-message-turn, and stop.mid-llm-call's .outcome-interrupted and .persisted-interrupted. stop.mid-llm-call itself still fails .provider-call-aborted, which is CP-16, so it is not in provenBy | source, probe | verified | fixed (6 cells) |
| CP-98 | S2 | CF Workflows builds its step.do step names by interpolating agent names, tool names, companion names and tool-call ids with no length bound, but the Workflows engine refuses a step name longer than 256 characters with a WorkflowFatalError. The engine throws before the step runs, so the run fails mid-flight. The longest names join two unbounded names and a provider call id: ${agentType}-spawn-${subAgentType}-${callId} for a sub-agent, ${agentType}-continue-persistent-${childName}-${callId} for a companion, ${agentType}-tool-${toolName}-${callId} for every server tool. Names the framework itself permits can cross the limit: core allows a companion name of up to 128 characters. CP-90's fix encoded the workflow instance ids and event types, not the step names, and no doc states the limit. | runtime-cloudflare/src/workflow.ts:2807 (origin/main 62d3661b45) (${agentType}-spawn-${subAgentType}-${subAgentCall.id}), :2945 (origin/main 62d3661b45) (…-wait-${subAgentType}-${subAgentCall.id}-${waitAttempt}) and :3112 (origin/main 62d3661b45) (…-record-${subAgentType}-${subAgentCall.id}): every sub-agent step name joins the parent and child agent names and the call id; runtime-cloudflare/src/workflow.ts:3560 (origin/main 62d3661b45) (${agentType}-spawn-persistent-${childName}-${companionCall.id}), :3931 (origin/main 62d3661b45) (…-continue-persistent-${childName}-${companionCall.id}) and :3965 (origin/main 62d3661b45) (…-wait-continue-${childName}-${companionCall.id}): the companion steps join the agent name, the companion name and the call id; :1447 (origin/main 62d3661b45) (…-respawn-${childInfo.agentType}-${childInfo.parentToolCallId}) and :2492 (origin/main 62d3661b45) (${agentType}-tool-${toolCall.name}-${toolCall.id}, every server tool) have the same shape; runtime-cloudflare/src/instance-id.ts:96 and :120 (CP-90's fix: buildWorkflowInstanceId and buildWorkflowEventType bound and encode instance ids and event types; no helper bounds a step name, and no step name in runtime-cloudflare/src is built through one); runtime-cloudflare/src/workflow.ts:2807 (origin/main 62d3661b45) reaches the engine's check: Miniflare 5.20260925.0-alpha (the runtime-cloudflare and e2e devDependency), dist/src/workers/workflows/binding.worker.js line 1430 (MAX_STEP_NAME_LENGTH = 256), lines 1442-1444 (isValidStepName) and lines 2217-2222 (step.do throws WorkflowFatalError 'Step name … exceeds max length (256 chars) or invalid characters found' before running the step); runtime-cloudflare/src/workflow.ts:4626-4631 (origin/main 4620e90228) (the workflow body's outer catch fails the run with the engine's message, finishRun('catch')); core/src/tools/companion/helpers.ts:44-57 (MAX_CHILD_NAME_LENGTH = 128: the comment says runtimes with tighter id rules encode their ids, which holds for CF Workflows instance ids but not for its step names); live: probe (plan 0c MR-2 dry run, origin/main 4c3cc736b9, the cfw-workflows-d1 executor lane run in the working tree only, ScriptedModel; the cited code is unchanged at this record's base). history.checkpoints-paired-after-subagent failed .parent-completes (then .subagent-call-paired and .every-checkpoint-paired as preconditions): the parent run failed with WorkflowFatalError: Step name "…-spawn-…-sub-1" exceeds max length (256 chars); that name was 261 characters. The harness's per-cell agent names (the cell id joined to the base name, about 70-80 characters each) and cell-prefixed call ids made it that long; inferred: (unverified) production Workflows enforces the same 256-character limit: Miniflare's engine is built from Cloudflare's Workflows source, but Cloudflare's published Workflows limits page (developers.cloudflare.com/workflows/reference/limits, read 2026-10-05) lists no step-name limit. Not run: a companion or respawn step over the limit, or an agentType long enough to fail every step name; related: CP-90 (unbounded CF Workflows instance ids; its fix bounded instance ids and event types only); RM-69 (this record makes history.checkpoints-paired-after-subagent fail before it can show the sub-agent half of RM-69); severity S2, as CP-90: a loud failure, not a silent one (the run fails and says why), for names the framework accepts and no doc forbids. It fires mid-run, inside the parent's workflow (for a sub-agent or companion call, after the step that issued it has committed), so the caller learns only when that call is reached. Two ordinary names and a provider call id are enough: two agent names of about 110 characters each, or a companion name within core's own 128-character bound beside an agent name of about 80, push a sub-agent or companion step name past 256; conformance: history.checkpoints-paired-after-subagent on cfw-workflows-d1 (the plan 0c MR-2 lane flip: 5 full lane runs over origin/main d2e95ad181 product code, the same failed ids in every run) — .parent-completes: the parent run fails with WorkflowFatalError: Step name "…-conf-cp-sub-parent-spawn-…-conf-cp-sub-child-…-sub-1" exceeds max length (256 chars); .subagent-call-paired and .every-checkpoint-paired fail only as its preconditions. The cell reproduces because the harness's per-cell agent names are long; real agent names and call ids can be as long, so the cell stays as is; related: fixed in code by !429 (WE-2's RM-80, which bounds every CF Workflows step name with boundWorkflowStepName, runtime-cloudflare/src/instance-id.ts:212, through guardLostClaims, run-entry.ts:1615-1650 (pre-fix, origin/main 656743ace8)): history.checkpoints-paired-after-subagent passes .parent-completes and .subagent-call-paired on cfw-workflows-d1 (CI pipeline 2961079548, and WE-2's local run at 70b8c2bd5e, product code e3783d111a); with boundWorkflowStepName returning the raw name the same cell fails all three of CP-98's former ids again (WE-2's red, working tree only). Its .every-checkpoint-paired is RM-69's (the sub-agent half, reached now that the fatal no longer stops the run); live: another site of the same class (owner slot c4, 2026-10-18; first filed as CP-151, now withdrawn as this record's duplicate): on !418 (847784ce9d), transcript.co-emitted-subagent-dispatched failed on cfw-workflows-d1 with WorkflowFatalError: the sub-agent park's step name (runtime-cloudflare/src/workflow.ts:4256-4262 (pre-fix, origin/main c4c0708011) with :5739 (pre-fix, origin/main c4c0708011) at origin/main c4c0708011: the parent's agent type, the sub-agent's type and the call id joined, plus -deadline) reached 259 characters with the cell's per-cell agent names; related: fixed in code by !429 (WE-2's RM-80, which bounds every CF Workflows step name with boundWorkflowStepName, runtime-cloudflare/src/instance-id.ts:212, through guardLostClaims, run-entry.ts:1615-1650): history.checkpoints-paired-after-subagent passes .parent-completes and .subagent-call-paired on cfw-workflows-d1 (CI pipeline 2961079548, and WE-2's local run at 70b8c2bd5e, product code e3783d111a); with boundWorkflowStepName returning the raw name the same cell fails all three of CP-98's former ids again (WE-2's red, working tree only). Its .every-checkpoint-paired is RM-69's (the sub-agent half, reached now that the fatal no longer stops the run); runtime-cloudflare/src/instance-id.ts:212-216 (origin/main 7daf87301b) (fix evidence, !429: boundWorkflowStepName keeps a step name of at most 256 characters as is, and shortens a longer one to its head plus an fnv1a64 hash of the whole, so long names stay distinct and engine-valid); conformance: history.subagent-long-step-names (owner batch 46af; per-cell agent names and a 140-character call id, so the sub-agent steps' names pass 256 characters) passes .outcome-settles, .parent-completes and .subagent-call-paired on every executor lane, 3 of 3 (owner slot s46 part 2, 2026-10-19, owner batch da88f26f50 on main a83323af26). With boundWorkflowStepName returning the raw name (instance-id.ts:213, runtime-cloudflare rebuilt, working tree only) it fails .parent-completes and .subagent-call-paired on cfw-workflows-d1 3 of 3: the parent fails with WorkflowFatalError: Step name … (the pre-fix fatal) | source, probe, live-repro | partly-inferred | fixed (1 cells) |
| CP-99 | S3 | CF Workflows handle.result() reports a finished run up to about half its run time late. It learns the outcome by polling the workflow instance's status, and the interval starts at 100 ms and grows ×1.5 after every poll (to a 30 s ceiling, with no overall timeout). The polls therefore land at about 0, 0.1, 0.25, 0.48, 0.81, 1.32, 2.08, 3.22, 4.93, 7.49, 11.3, 17.1, 25.7, 38.7 and 58.2 s after the first one, and a run that ends just after a poll is reported only at the next one: about 0.5·t + 0.1 s late when the poll it just missed came at t, and up to 30 s late once the interval reaches its ceiling (from about 58 s on). The schedule starts at the first result() call, not when execute() returns, so a run that ends between about 25.75 s and 30 s after result() is first called is reported at about 38.7 s (a caller that, like the conformance harness, calls result() straight after execute() returns sees the same window measured from execute()), so a caller that bounds result() at 30 s sees a run that finished inside its bound as still running. The result itself is correct, only late. | runtime-cloudflare/src/executor.ts:949-959 (origin/main 2424cd0180) (awaitInstanceTerminal: pollUntil over instance.status() with initialInterval 100, maxInterval 30000 and timeout Number.POSITIVE_INFINITY; it passes no backoffFactor, so the default 1.5 applies) and :941-948 (origin/main 2424cd0180) (its comment: no overall timeout, and a caller that wants a bound races result() with its own timer); runtime-cloudflare/src/executor.ts:1936-1951 (origin/main 2424cd0180) (the resumed handle's result()) and :2035-2047 (origin/main 2424cd0180) (the active handle's result()): both wait on awaitInstanceTerminal, so every CF Workflows result() call reports the outcome on that poll schedule; runtime-cloudflare/src/utils/polling.ts:71-77 (pollUntil defaults: initialInterval 100, maxInterval 30000, backoffFactor 1.5), :94-102 (check, then sleep the current interval) and :105 (interval = min(interval × 1.5, maxInterval)): the gap after the poll at time t is about 0.5·t + 0.1 s until the ceiling: the fifteenth sleep, after the 58.2 s poll, is already 29.2 s, and every sleep from the sixteenth (after the 87.4 s poll) is 30 s. The module header at :9-17 already notes the alternative: wait for the stream's end or fail frame instead of polling; live: probe (plan 0c MR-2 dry run, origin/main 4c3cc736b9, the cfw-workflows-d1 executor lane run in the working tree only, ScriptedModel, with a throwaway poller reading every executor-created instance's raw status every 20 ms to time its real terminal; the cited code is unchanged at this record's base). The result() polls landed at about 0, 0.1, 0.25, 0.48, 0.81, 1.32, 2.08, 3.22, 4.93, 7.49, 11.3, 17.1, 25.7 and 38.7 s after execute() returned (the executor driver calls result() as soon as the handle resolves, e2e/src/conformance/drivers/executor-driver.ts:234-239 (origin/main fdba7888ef)), within about 5 ms of the computed schedule. Measured lateness (the instance's terminal status to result() returning): 100-200 ms for ordinary runs, which close in 40-120 ms; about 0.5 s for runs of about 0.2 s; 0.66-0.74 s on lifecycle.loop-throw-fails-run; 1.1 s on stream.step-discarded-on-cancel; and 2.58 s on stop.mid-tool-honors-abort, whose instance went terminal about 5.06 s after create, just after the 4.93 s poll, so result() returned at the 7.49 s poll. No run in the lane lasted longer than 7.1 s, so no 30 s bound was crossed and no cell failed on it; contrast: runtime-js/src/execution/handle-factory.ts:202-207 (result() returns the in-process run's promise, which settles when the loop ends) and runtime-temporal/src/executor.ts:1171-1176 (as before WA MR2, origin/main cae966c0a7) (result() awaits the Temporal client's workflowHandle.result() for the workflow's close): neither waits on a growing client-side poll interval; related: DI-18 (the same awaitInstanceTerminal poll: its fix made the wait run through every non-terminal status with no overall timeout; this record is about its schedule, not its stop condition); CP-97 and CP-98 (the other CF Workflows findings of the same dry run). Also the plan 0c design spec docs/superpowers/specs/2026-09-30-conformance-plan-0c-cfw-lane-design.md, section 2 (the result() poll is one of the product waits that can outlast a named settle for a run that finished in time) and its MR-2 pre-flip decisions of 2026-10-05: catalogued, with no bound change (a user decision); severity S3 (a user decision, 2026-10-05): the outcome is correct and only late, by up to about half of the time already waited, at most 30 s. It is latent at today's run lengths: it crosses a bound only when a run ends in the last third or so of it. The conformance settles RUN_OUTCOME_BOUND_MS and OUTCOME_BOUND_MS are 30 s (e2e/src/conformance/scenarios/shared.ts:21 and e2e/src/conformance/scenarios/stop.ts:27), so a cfw run that really ends between about 25.75 s and 30 s after result() is first called would fail them at about 38.7 s, and a caller that races result() with its own 30 s timer would report the same run as timed out; conformance: lifecycle.result-latency (owner batch 28, executor tier, every lane): the run calls a gated tool held until 27 s after the start returned, then the gate is released and the run finishes one model step later. On cfw-workflows-d1 (main 8dc3a5c967 plus batches 24 to 27) the outcome settles 12.3 s, 12.1 s and 12.1 s after the release in 3 of 3 runs, at the 38.7 s poll, against a 6 s bound (.reported-promptly fails; .reports-completed passes). Green on js-memory | source, probe | verified | open |
| CP-100 | S2 | CF Workflows records an EXISTING session failed when a turn, resume or retry entry fails BEFORE its run-start commit, overwriting the session's real outcome although nothing ran. The entry plans inside the workflow (it loads the session, reads its history through the count-checked loader, and parses the input state) and only then commits its run. When that fails for good (a history read failure, or a store error that outlasts the run-start step's two retries), the workflow's outer catch finds no run and calls persistPreCommitFailure, which saves the session failed with the entry's error whenever no run is live. Only a planning validation_error and an existing session's completion-support refusal are exempt. So a completed session (its output still stored) now reads failed, and an interrupted session can no longer be resumed with continue / with_message, which refuse a failed session. The caller does get the failure, through the handle's result. Temporal leaves a session that already has a run untouched. runtime-js writes nothing on such a failure, except for a state_history_incomplete history read on execute() (and on a retry() whose plan re-read fails), which it recorded on purpose until the user ruling below. The user's ruling (2026-10-06) is that the session is left untouched everywhere. CF Workflows is to be fixed to follow Temporal's rule (a session that already has a run is never touched) by CF-D2 MR1 (decided, not yet opened). The JS half is a defect too and is catalogued separately as CP-101, because it has a different fix site; it is to be fixed in the same MR (CF-D2 MR1, its Task 9, under the ruling that a pre-commit entry failure leaves the session untouched). | runtime-cloudflare/src/workflow.ts:4538-4540 (origin/main 4620e90228) (the outer catch: did this entry's run commit, from runCommitted or a runExists read), :4541 (origin/main 4620e90228) (no run: the pre-commit branch), :4553-4556 (origin/main 4620e90228) (writesNothing only for a completion-support refusal of an existing session) and :4558-4560 (origin/main 4620e90228) (otherwise steps.persistPreCommitFailure); :4571 (origin/main 4620e90228) returns the failed result with the entry's errorDetail, which the caller's handle reports; runtime-cloudflare/src/steps.ts:1355-1400 (origin/main 4620e90228) (persistPreCommitFailure): it returns early only when the current run is the entry's own (:1363 (origin/main 4620e90228)); otherwise, unless the code is validation_error and while no run is live (:1364-1367 (origin/main 4620e90228)), it saves the loaded session with status 'failed', error and errorDetail set to the entry's (:1368-1380 (origin/main 4620e90228)), under the idle fence only (:1378 (origin/main 4620e90228)). Nothing checks whether the session already had a run, so a completed, failed or interrupted session is overwritten. The docstring at :1342-1354 (pre-fix, origin/main 4620e90228) states this intent; runtime-cloudflare/src/steps.ts:942-945 (pre-fix, origin/main 65061fbeb5) (the entry plan loads the session and reads its history with loadLLMHistory, the count-checked loader, before the commit) and :992 (pre-fix, origin/main 65061fbeb5) (a turn that brings its own state parses it with the agent's stateSchema inside the plan), and runtime-cloudflare/src/workflow.ts:233-235 (pre-fix, origin/main e96f88bc98) (RUN_START_STEP_CONFIG: the run-start step is retried twice, 1 s apart, before the throw reaches the outer catch); runtime-cloudflare/src/executor.ts:472-499 (origin/main 4620e90228) (awaitEntryRunStart: an instance that closed with no run, and no refusal or conflict, returns at :499 to the caller, whose handle's result() reports the failed workflow output), :1435-1441 (origin/main 4620e90228) (resume() in continue / with_message mode refuses a failed session) and :1589-1594 (origin/main 4620e90228) (retry() accepts only a failed session); contrast: runtime-temporal/src/activities.ts:6958-6961 (commitTerminalState finds no run with the entry's id and calls persistPreCommitFailure) and :7530-7551 (persistPreCommitFailure: 'When the session already has a run, the session is NOT touched (M1: a completed / failed / suspended session keeps its outcome)'; it returns at :7551 when getCurrentRun finds any run). Only a session that never had a run is saved failed; contrast: runtime-js/src/js-agent-executor.ts:3224-3228 (origin/main 4620e90228) (resume: nothing is written before the run-start commit) and :2856-2902 (origin/main 4620e90228) (execute's run-start catch rethrows every pre-commit failure without a write, except state_history_incomplete). That exception was deliberate (runtime-js's rule that an execute() whose history read hits state_history_incomplete marks the existing session failed) until the ruling of 2026-10-06, which makes it a defect (CP-101): runtime-js/src/js-agent-executor.ts:2663-2676 (origin/main 4620e90228) and :2874-2895 (origin/main 4620e90228) send a state_history_incomplete history read on execute() through persistRunFailure (:1331-1475 (origin/main 4620e90228): the status CAS and saveState, then the reactivation and failure of the session's ended stream at :1412-1419 (origin/main 4620e90228) and onAgentFail at :1456-1466 (origin/main 4620e90228)), a CAS from paused / interrupted / completed / failed to failed (:1516 (origin/main 4620e90228) and :2891 (origin/main 4620e90228)), and a retry() whose first read passes but whose plan's re-read fails re-marks an already failed session the same way (:4191 (origin/main 4620e90228); a retry whose first read already fails writes nothing). So for that one failure on execute() CF Workflows and runtime-js agree today, and both are to change: CF Workflows by this record's fix, runtime-js by CP-101's; runtime-cloudflare/src/__tests__/i5-pre-commit-failure-fence.test.ts:28-67 (origin/main 4620e90228) (the existing unit tests: a run that commits between the read and the save keeps its session, and a validation_error is not recorded). None starts from a completed or interrupted session; related: the user's ruling (2026-10-06): leave the session untouched everywhere. For CF Workflows the fix is decided (routed to CF-D2 MR1, which is not opened yet): it follows Temporal's rule: a session that already has a run is never touched; only a session that never had a run is marked failed; the caller still gets the typed failure. Its red-first test extends runtime-cloudflare/src/__tests__/i5-pre-commit-failure-fence.test.ts, plus one real-engine case. The runtime-js half is CP-101 (tracked as FU-JS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED, filed by CF-D2 MR1), which is to be fixed in CF-D2 MR1 too (its Task 9, under the same ruling); each record flips only when its own cell proves it, so this one can flip on its cfw-workflows-d1 cell while CP-101 waits on its 0d cells; related: CP-101 (the runtime-js half of the same ruling: its execute() and retry() mark an existing session failed on a state_history_incomplete history read before the run-start commit); related: !288 (bd26dc9b57) introduced persistPreCommitFailure when it moved the entry's planning and claim into the workflow's run-start commit. Before that, !279's executor.resume() claimed the session itself and rolled the claim back if a later pre-instance step threw; its changeset said a from_checkpoint failure that saved nothing, a history read failure included, restores the previous status. Since !288 that no longer holds on CF Workflows (runtime-js's resume still writes nothing before its commit). CP-35 records that !279 claim as fix evidence; CP-96 is the run-start commit's own status gate; related: the !279 changeset as merged (.changeset/llm-history-runtime-cloudflare.md, the from_checkpoint bullet) claims that a from_checkpoint failure which saved nothing, a history read failure included, restores the session's previous status; on CF Workflows that is false (above). The release-prep MR !305 (merged, 002deb4c33) corrected that text before release: the released runtime-cloudflare CHANGELOG (packages/runtime-cloudflare/CHANGELOG.md:318-320) says the session is recorded failed with the classified errorDetail; related: peer-reported (the release session, 2026-10-05, from source): a CF Workflows resume, retry or turn whose entry fails before its run-start commit, for example on a history read, records an existing session failed, while Temporal leaves it untouched and runtime-js writes nothing, contrary to the !279 changeset. Verified here against source: the CF Workflows and Temporal halves hold, and the runtime-js half holds except for a state_history_incomplete history read on execute() (and a retry() whose plan re-read fails), described above; inferred: (unverified) a schema-invalid state passed to execute() on an existing session is a caller-reachable trigger: the plan's stateSchema.parse throws a ZodError, which carries no code and so is not the exempt validation_error, and the completed session is recorded failed with the Zod message. Not exercised: any trigger. A retry() of a completed session wrongly marked failed would start a new run on its completed transcript; severity S2: the caller learns that its entry failed, so the failure is not silent to it, but the durable session's real outcome is overwritten for every other reader (getStatus, the snapshot, a reconnect, a parent reading a child): a completed session reads failed, and an interrupted one loses resume() in continue / with_message mode. A later turn or retry() works on it again. The trigger is rare: an entry that fails before its run commits, past the run-start step's retries; runtime-cloudflare/src/steps.ts:1484-1521 (pre-fix, origin/main 487090d3fb) (fix evidence, !316, reworked by !334: AgentSteps.persistPreCommitFailure follows Temporal's M1 rule, as its docstring at :1450-1483 (pre-fix, origin/main 487090d3fb) states: an entry that fails before its run-start commit never touches a session that already has a run. It reads the session's current run first (:1490-1492 (pre-fix, origin/main 487090d3fb); since !334 a failed read rejects so that its step retries it, where !316 wrote nothing, :1413-1422 (pre-fix, origin/main a07e5aa1fb)) and returns at :1495 (pre-fix, origin/main 487090d3fb) when the session has any run, the entry's own or another, live or ended, so a completed, failed or interrupted session keeps its state, run, stream and hooks. Only a session that never had a run is saved failed with the entry's detail, through core failSessionWithoutRun under the idle fence (:1500-1507 (pre-fix, origin/main 487090d3fb); a planning validation_error is still not recorded, :1496 (pre-fix, origin/main 487090d3fb)); since !334 (user ruling R43) that save is its only write, where !316 then restored its stream (:1455-1460 (pre-fix, origin/main a07e5aa1fb)). No lifecycle hook fires on any path) and runtime-cloudflare/src/workflow.ts:4975-5036 (pre-fix, origin/main 487090d3fb) (fix evidence, !316, re-pointed for !334: the workflow's outer catch still calls it for an entry whose run never started, :5011-5026 (pre-fix, origin/main 487090d3fb), in a step that retries the write and, once its retries run out, logs it and still answers the entry's own failure (!334), except for an existing session's completion-support refusal, :4998-5001 (pre-fix, origin/main 487090d3fb), and returns the failed result with the entry's errorDetail at :5036 (pre-fix, origin/main 487090d3fb), so the caller still gets the typed failure); runtime-cloudflare/src/__tests__/i5-pre-commit-failure-fence.test.ts:163-344 (proof, fix evidence, !316 and !334: the M1 unit cases on AgentSteps.persistPreCommitFailure. A session that ended completed, failed or interrupted keeps its state, run and stream byte-identical and no hook fires, :164-190; a session whose run ended with its stream still active does not have that stream ended either, :192-206; a completion-support refusal of a session that has a run writes nothing, :208-224; a session that never had a run is still marked failed with the detail, :226-243, and (!334, R43) its stream manager is never called, :253-288; a failed current-run read rejects (the step retries) and writes nothing, :290-309, where under !316 it resolved, :245-264 (pre-fix, origin/main a07e5aa1fb); a save failure other than a newer run winning rejects, :311-326) and e2e/src/__tests__/chat-pre-commit-failure-cfw-engine.wf.test.ts:34-90 (fix evidence, !316, on the real Workflows engine, the test:cfw-engine lane: after a completed turn 1, a SEND whose instance's entry history read fails with state_history_incomplete before its run-start commit gets a typed 409, and the session, its current run and its stream info all equal what turn 1 left, :86-89). Neither is a conformance cell; related: the record stays OPEN (owner's authorisation, 2026-10-07): !316's fix evidence above is unit-level and engine-level only, so status and pending are unchanged until the cfw-workflows-d1 executor cell described in pending.reason proves the fix. No conformance scenario that !316 adds or changes makes an entry fail before its run-start commit; conformance: fixed (owner batch 18b, on !334's head fedd857efb, merged as 137b07d57a with the same code; the fix is !316 as reworked by !334, above). control.pre-commit-failure-keeps-session arms the fault-entry-history-read control (the cfw worker's D1 store reads the armed session's message count one too many, so the entry plan's loadLLMHistory throws state_history_incomplete before the run-start commit). Leg A: a completed session, then a new turn; leg B: an interrupted session (stopped while a tool is held: leg B deliberately avoids CP-97's path, where a stop pressed during the run's terminal model step is lost and the run completes, so it could not set up an interrupted session), then resume({ mode: 'continue' }), then, disarmed, a second resume. Each faulted entry fails with the typed code, and the session's status, output, error and errorDetail, and its stream's status and head, equal what they were before; leg B's second resume completes. Green 3 of 3 on cfw-workflows-d1. Red with one mutation (named here, applied to the working tree only and reverted): persistPreCommitFailure restored to the pre-!316 rule (runtime-cloudflare/src/steps.ts:1362-1378 (pre-fix, origin/main 4620e90228): skip only the entry's own run or a live run, and save an ENDED run's session failed, fenced on that run). The cell then fails .a-status-kept (failed, not completed), .b-status-kept (failed, not interrupted) and .b-resumable-after ('Cannot resume: Agent has failed'). Dropping the hasRun guard alone is NOT a valid red: the save that follows is fenced idle on currentRunId null, which refuses a session that has a run, so that mutation writes nothing and the cell stays green (observed in this batch); the red must restore the pre-!316 rule, fenced on the ended run. The other lanes have no carrier for the fault (a declared gap); CP-101 is the runtime-js half | source, peer-reported | verified | fixed (1 cells) |
| CP-101 | S2 | runtime-js records an EXISTING session failed, and fails its ended stream, when an execute() or retry() entry hits a state_history_incomplete history read BEFORE its run-start commit, although nothing ran. The user's ruling (2026-10-06) is that an entry that fails before its run-start commit leaves the session (and its stream) untouched everywhere; this path was deliberate until that ruling. Three seams reach it. (1) execute() of a paused, interrupted, completed or failed session whose very first history read finds the durable log short of the store's live count (a log already damaged before the entry). (2) execute() when that first read passes and the continuation's plan re-reads the history inside the run-start step and fails. (3) retry() when its first read passes and the plan's re-read inside its try fails; a retry whose first read already fails throws with no write. On seams 2 and 3 the first read succeeded, so the log may be healthy and the overwrite is gratuitous. At each, persistRunFailure moves the session to failed with the entry's error and errorDetail, reactivates and fails its ended stream, and fires onAgentFail. So a completed session (its output still stored) reads failed and its stream carries a failure chunk, and an interrupted session can no longer be resumed with continue / with_message. On retry() the session is already failed, and the same seam rewrites its error and errorDetail with the retry's, replacing the original failure's detail. The Durable Object runtime inherits all of this: its base class runs a JSAgentExecutor for execute and retry. The caller does get the typed failure. Other pre-commit failures (and every resume() failure) already write nothing, so this one failure class on two entries is the only runtime-js departure from the rule. It is the runtime-js half of CP-100: CF Workflows' half is to be fixed (CF-D2 MR1), and the two have different fix sites. Tracked as FU-JS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED (filed by CF-D2 MR1), which the ruling (an entry that fails before its run-start commit leaves the session untouched) pulls into CF-D2 MR1 as its Task 9. Owner: CF-D2 MR1. Fixed on runtime-js and the CF DO by fb7c1e12f5 (WA MR1, after CF-D2 MR1's !316): persistPreCommitFailure writes nothing to a session that has a run (FU-JS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED is done); open until its cells. | runtime-js/src/js-agent-executor.ts:2659-2677 (origin/main 2424cd0180) (execute's first history read: a state_history_incomplete from loadAgentState on an existing session reads the raw status, throws AgentAlreadyRunningError for a live one, and otherwise returns failExecuteWithHistoryIncomplete with the version it read) and :1490-1520 (origin/main 2424cd0180) (failExecuteWithHistoryIncomplete: persistRunFailure with fromStatuses ['paused', 'interrupted', 'completed', 'failed'] at :1516, the comment at :1514-1515 stating it never includes 'active'), then a pre-projected failed handle; runtime-js/src/js-agent-executor.ts:1331-1475 (origin/main 2424cd0180) (persistRunFailure): a compareAndSetStatus from fromStatuses to 'failed' (:1351-1358 (origin/main 2424cd0180)), then, when it applied, a read-modify-write saveState with status 'failed', error and errorDetail set to the entry's (:1367-1376 (origin/main 2424cd0180)). Nothing checks whether the session already had a run, so a completed or interrupted session is overwritten. Still when the failure was persisted and the session is a root session (:1399 (origin/main 2424cd0180)), it reads the stream, reactivates a paused, ended or failed one (:1412-1417 (origin/main 2424cd0180)), then initStream and failStream with the detail (:1418-1419 (origin/main 2424cd0180)), so an ended stream of a completed run is reopened and failed. When it was persisted (:1430 (origin/main 2424cd0180)) it fires onAgentFail (:1456-1466 (origin/main 2424cd0180)). Its docstring at :1309-1330 (pre-fix, origin/main 2424cd0180) states this as the intent, for an entry that has not started a run on an existing session; runtime-js/src/js-agent-executor.ts:2856-2902 (origin/main 2424cd0180) (execute's run-start catch): the first history read at :2659-2662 (origin/main 2424cd0180) passed, and the continuation's plan inside the run-start step re-reads the history (initializeSession, :4644 (origin/main 2424cd0180), through loadFromSession, :4813 (origin/main 2424cd0180)) and fails before the commit. For state_history_incomplete only, persistRunFailure runs with fromStatuses ['paused', 'interrupted', 'completed', 'failed'] for an existing non-branch session (:2889-2892 (origin/main 2424cd0180)) pinned to the version this entry read (:2893 (origin/main 2424cd0180)), then the error is rethrown at :2902 (origin/main 2424cd0180). Every other error in that catch is rethrown with no write (:2857-2863 (origin/main 2424cd0180) maps StaleStateError to AgentAlreadyRunningError; nothing else writes); runtime-js/src/js-agent-executor.ts:3885 (origin/main 2424cd0180) (retry's first loadAgentState, count-checked through loadLLMHistory, outside the try that opens at :3922: a log already damaged before the entry throws there and writes nothing), :3931 (origin/main 2424cd0180) (planRetry's re-read of the history inside the try) and :4172-4200 (origin/main 2424cd0180) (retry's catch): a state_history_incomplete from that re-read, before the retry's own run-start (retryRunStarted false, set at :4043 (origin/main 2424cd0180) after the commit at :4019 (origin/main 2424cd0180)), calls persistRunFailure with fromStatuses ['failed'] (:4191 (origin/main 2424cd0180)). The session is already failed, so the CAS applies and saveState rewrites status, error and errorDetail with the retry's, and the stream is failed again. retry() accepts only a failed session, so this is a re-mark rather than a status change, but the original failure's error and errorDetail are lost. It is reached only when the first read succeeds and the re-read fails, as in the unit test below; runtime-js/src/js-agent-executor.ts:3224-3228 (origin/main 2424cd0180) (resume: the comment states nothing is written before the run-start commit) and :3419 (origin/main 2424cd0180) (commitRunStart is its first write); persistRunFailure has only three callers (:1501 (origin/main 2424cd0180), :2876 (origin/main 2424cd0180), :4180 (origin/main 2424cd0180)), none in resumeSession (:3140 (origin/main 2424cd0180)), so a resume() whose history read fails writes nothing. Confirmed by runtime-js/src/__tests__/js-agent-executor-history.test.ts:479-508 (origin/main 2424cd0180) (resume() with an incomplete history rejects before any claim: no active session, no run, no LLM call); runtime-js/src/__tests__/js-agent-executor-history.test.ts:100-122 (origin/main 2424cd0180) (execute on an existing session fails with state_history_incomplete and the persisted session carries that errorDetail), :151-190 (origin/main 2424cd0180) (the failed stream chunk and onAgentFail on a non-live session) and :511-545 (origin/main 2424cd0180) (retry persists state_history_incomplete; read #1 passes and every read from #2 on is broken, :521-534 (origin/main 2424cd0180), which is how it reaches the re-mark). These unit tests pin the current behaviour, so the fix has to change them. No conformance scenario starts a completed or interrupted session, fails its entry before the run-start commit and judges the session or its stream; runtime-cloudflare/src/do/durable-object-agent-base.ts:2326 and :2345 (origin/main 2424cd0180) (the DO's execute path builds a JSAgentExecutor over its DOStreamManager and calls executor.execute) and :3089 and :3106 (origin/main 2424cd0180) (its retry path builds one and calls executor.retry). The DO runtime therefore runs the same persistRunFailure seam: the defect and its fix are the runtime-js ones, and a DO session is marked failed, with its stream failed, by the same two entries; contrast: runtime-temporal/src/activities.ts:7530-7551 (persistPreCommitFailure: 'When the session already has a run, the session is NOT touched (M1)'; it returns when getCurrentRun finds any run); the docstring at :7529-7536 adds that the session's stream is not touched either ("nor is its stream", :7532-7533). This is the rule the ruling extends to every runtime, stream included; related: CP-100 (the CF Workflows half of the same ruling: its fix is decided, and routed to CF-D2 MR1, which is not opened yet. The ruling (a pre-commit entry failure leaves the session untouched) puts the fix for this record in the same MR, as its Task 9, but each record flips only when its own cell proves it: CP-100 on cfw-workflows-d1, this one on its 0d cells). CP-100 recorded this runtime-js behaviour as deliberate; the ruling of 2026-10-06 ("leave the session untouched everywhere") makes it a defect, which is why it is catalogued here and not left as a note on CP-100. CP-35 (its !279 claim is about CF Workflows' resume rollback) and CP-96 (the run-start commit's own status gate) do not cover this path: neither writes the session on a history failure; related: follow-up FU-JS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED (filed by CF-D2 MR1; not on main yet, so it is cited by name only). Fix decided: CF-D2 MR1 (not yet opened), its Task 9, under the ruling of 2026-10-06, red-first on runtime-js and on the CF Durable Object. Owner: CF-D2 MR1. The fix applies the ruling: a pre-commit entry failure on a session that already has a run writes nothing, so persistRunFailure on execute() / retry() makes no session write, does not reactivate or fail its stream, and does not fire onAgentFail (a fresh session or a branch clone created by this call is its own write and can still be marked failed). The caller still gets the typed failure, and the unit tests above flip to assert an untouched session and stream; inferred: (unverified) the trigger needs a durable log shorter than the store's live message count (a skipped or invalid row, as the history unit tests seed). On the first-read seam (execute's :2659-2677) the log is already damaged and fails every later read of that session, so marking it failed is arguably informative, though the ruling still prefers untouched. On the other two seams the first read passed, so a transient store fault or a concurrent writer correcting the log could be the cause and the session may be healthy; no probe; severity S2, consistent with CP-100: the caller learns that its entry failed, so the failure is not silent to it, but the durable session's real outcome and its stream are overwritten for every other reader (getStatus, the snapshot, a reconnect, a parent reading a child, a stream reader): a completed session reads failed with a failure chunk on its stream, an interrupted one loses resume() in continue / with_message mode, and a retried failed session loses its original errorDetail. The trigger is narrower than CP-100's (one error code, on two entries). On the first-read seam of execute() the log was already damaged, which argues for a lower weight there; but on the run-start-catch and retry seams the first read succeeded, so the log may be healthy and the session is overwritten for a failure that may be transient, which keeps it at S2 and not lower; runtime-js/src/js-agent-executor.ts:2290-2366 (origin/main e3f03e8196) (fix evidence, !316, rewritten by WA MR1 (!340) under spec I-D6 / M1 and rulings R35, R40, R43: JSAgentExecutor.persistPreCommitFailure, the M1 seam, as its docstring at :2290-2319 (origin/main e3f03e8196) states: an entry that fails before its run-start commit, whatever the failure class, never touches a session that already has a run. It reads the session's current run, with up to 3 attempts (:2323-2332 (origin/main e3f03e8196)), and returns the classified detail with NO write when the session has any run (:2340 (origin/main e3f03e8196)) or when the read still fails (:2333-2339 (origin/main e3f03e8196)). Only a session that never had a run is recorded failed, through core failSessionWithoutRun (:2341-2364 (origin/main e3f03e8196): the session only, fenced on no run, with no stream write and no hook, R43 / R49), so a session that already has a run gets no session write, no stream reactivation, failure or chunk, and no onAgentFail. Every pre-commit seam calls it: execute()'s first history read through failExecuteWithHistoryIncomplete (:4417 (origin/main e3f03e8196), which calls it at :2482 (origin/main e3f03e8196)), execute()'s run-start catch (:4649 (origin/main e3f03e8196), any failure class) and retry()'s catch before its own run-start (:6488 (origin/main e3f03e8196); after the run-start, :6503 (origin/main e3f03e8196) ends the run failed through failRetryRun, fenced on this run, which is at :2384 (origin/main e3f03e8196), WA MR2 Ruling 69). The CF Durable Object runs this JSAgentExecutor for execute and retry, so the same code fixes the DO half); runtime-js/src/__tests__/pre-commit-entry-failure-m1.test.ts:30-298 (proof, fix evidence, !316, extended by WA MR1 (!340): execute() on a completed, failed or interrupted session whose later history read fails rejects typed and writes nothing, :136-155; one whose first history read fails (C9) returns a failed handle and writes nothing, :157-177; retry() on a failed session whose plan history read fails rejects typed and keeps its original failure, :180-200; a failed current-run read writes nothing, :202-221; a session that never had a run is still marked failed, on its session only, with no stream and no onAgentFail (R43), :223-297), runtime-js/src/__tests__/js-agent-executor-history.test.ts:100-127, :155-189 and :509-548 (fix evidence, !316: the three unit tests that pinned the overwrite now assert that the session, its stream and onAgentFail are left alone), runtime-cloudflare/src/__tests__/pre-commit-entry-failure-do.cf.test.ts:152-231 (fix evidence, !316, extended by WA MR1 (!340), real workerd over a real DOStateStore: execute() on a completed or failed session whose run-start plan or first history read fails, and retry() of a failed session whose retry-plan read fails, each fail with state_history_incomplete and leave the session, runs, latest checkpoint, messages, stream and onAgentFail count unchanged; a session that never had a run is still recorded failed, with no stream and no onAgentFail, R43) and runtime-cloudflare/src/__tests__/integration/do-multiturn-execute-continuation.integ.test.ts:435-506 and :508-591 (fix evidence, !316, through the real runtime-cloudflare/src/do/durable-object-agent-base.ts under Miniflare: /start on a completed or failed session and /retry on a failed one answer 409 state_history_incomplete with no LLM call, no onAgentFail and an unchanged session snapshot). None is a conformance cell; related: the record stays OPEN (owner's authorisation, 2026-10-07): !316's fix evidence above is unit-level and integration-level only, so status and pending are unchanged until the 0d executor cells on js-memory, js-redis and js-postgres described in pending.reason prove the fix. No conformance scenario that !316 adds or changes makes an entry fail before its run-start commit; related: fixed in code by fb7c1e12f5 (WA MR1, !340, rewriting CF-D2 MR1's !316, whose own commit predates main's earliest reachable commit 1c5caeaa23), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/js-agent-executor.ts:2341-2387 (origin/main 56d564d073) (persistPreCommitFailure reads the current run up to 3 times and writes nothing when the session has a run or the read fails; only a never-run session is failed, on its session only, through failSessionWithoutRun), runtime-js/src/js-agent-executor.ts:2503 (origin/main 56d564d073) (execute's first history read, through failExecuteWithHistoryIncomplete, called at :4477 (origin/main 56d564d073)), runtime-js/src/js-agent-executor.ts:4709 (origin/main 56d564d073) (execute's run-start catch, any failure class) and runtime-js/src/js-agent-executor.ts:6548 (origin/main 56d564d073) (retry's catch before its own run-start); persistRunFailure no longer exists, and the CF DO runs this executor for execute and retry | source | verified | open |
| CP-102 | S3 | When a chat client disconnects mid-stream, handleChatStream's fresh-turn and resume responses do not stop reading the run's stream promptly. Live attach and terminal replay do stop: their onEnd closes the reader. streamHandle builds the SSE body with no onEnd, so a cancelled body only asks the source iterator to return(), and while a read is pending that request waits until the run writes its next chunk. On the JS path the reader is closed then. On the Durable Object path the source is the DO execution handle's /sse generator, whose finally only releases the reader lock and never cancels the DO's response body, so nothing in the product tells the DO the client left. Its /sse writer is released only by the DO's heartbeat eviction (backpressure or 60 s staleness), the terminal broadcast at the stream's end, or (inferred) the runtime. The request's abort signal does not help: handleChatStream passes it only to the passive-attach run-start wait, and neither StreamHandleOptions nor the DO handle's stream() accepts one. On Node, the documented pipeWebResponseToExpress helper has no 'close' handling, so it reads the body to the run's end after the client leaves. Each mid-stream refresh can leave such a reader (and on the DO an SSE connection) behind, at most until the run ends. No data is lost, and no effect beyond the leak is established. | ai-sdk/src/handler/handle-chat-stream.ts:2451-2539 (origin/main 4620e90228) (streamHandle, used by the resume path at :971 and the fresh-turn path at :1186: it reads handle.stream() at :2486-2488 and builds buildSSEResponse(events, { logger }) at :2535 with no onEnd) vs :1476-1487 and :1649-1656 (origin/main 4620e90228) (streamLiveResponse and replayTerminalStream pass onEnd: closeReader, which closes their reader when the body is cancelled); ai-sdk/src/response/sse-builder.ts:267-285 (origin/main 4620e90228) (the body's cancel(): it requests iterator.return() and awaits it after cleanup(), which runs onEnd; its comment says an async generator queues return() behind a pull's pending await, which settles only when the stream reader closes, and that onEnd is what closes it. With no onEnd, nothing closes it); runtime-cloudflare/src/do/do-clients.ts:1349-1371 (origin/main 4620e90228) (DOExecutionHandleImpl.stream() takes no options and calls stub.fetch('https://internal/sse?...') at :1356-1363 with no signal) and :1373-1464 (origin/main 4620e90228) (parseSSEToChunks: its finally, :1461-1463, only calls reader.releaseLock(), never reader.cancel(), so the DO's /sse response body is never cancelled) vs :683-698 (origin/main 4620e90228) (DOStreamManagerClient's resumable reader close() cancels its reader, the reader the live-attach and replay paths close through onEnd); runtime-cloudflare/src/do/durable-object-agent-base.ts:4141-4168 (origin/main 4620e90228) (handleSSE: a TransformStream per /sse request, its writer handed to setupSSEConnection) and :4526-4679 (origin/main 4620e90228) (setupSSEConnection registers the writer at :4620); core/src/stream/sse-connection-manager.ts:99-150 (origin/main 4620e90228) (addConnection drops a connection on the request's abort signal or when writer.closed settles), :193-233 (broadcast keeps writing to every registered writer) and :296 (broadcastTerminal closes them when the stream ends); ai-sdk/src/handler/handle-chat-stream.ts:397-408 (origin/main 4620e90228) (the request's signal is passed, at :407, only to the passive-attach streamLiveResponse with waitForLive) and :1366-1381 (origin/main 4620e90228) (the only use: firstChunkWithin(reader, ..., params.signal) answers 499 when the client leaves during the run-start wait). The other streamLiveResponse calls (:352, :910, :962 (origin/main 4620e90228)) and streamHandle take no signal; core/src/types/runtime.ts:588-597 (origin/main 4620e90228) (StreamHandleOptions has only fromSequence) and ai-sdk/src/types/minimal-interfaces.ts:56 (origin/main 4620e90228) (MinimalExecutionHandle.stream(options?: StreamHandleOptions)); ai-sdk/src/adapters/express.ts:153-175 (origin/main 4620e90228) (pipeWebResponseToExpress, whose JSDoc example serves handleChatStream and which createWebResponseExpressMiddleware, :203-214, uses; docs/frontend/frameworks.md documents it: no 'close' listener, so after the client leaves it keeps reading the body, res.write into a closed response, until the body ends with the run) vs :52-121 (origin/main 4620e90228) (pipeToExpress cancels its reader on 'close', :77-82); contrast: agent-server/src/adapters/express.ts:105-119 (origin/main 4620e90228) (pipeStreamToResponse cancels its reader on 'close'); runtime-js/src/execution/handle-factory.ts:114-129 (origin/main 4620e90228) (the JS handle's stream iterator closes its StreamManager reader in return(), unlike the DO generator; on the JS path return() is still reached only after the pending read resolves); live: probe (origin/main 4620e90228, no product change; a throwaway ai-sdk unit test in Node, deleted after the run): DOFrontendExecutor.execute over a stub DO whose /sse body was a ReadableStream the test controlled, its handle.stream() chunks fed through buildSSEResponse with no onEnd, as streamHandle does. After the client read one chunk and the next pull was pending on the /sse body, reader.cancel() did not resolve and the source generator did not finish until the stub wrote another chunk. The generator then finished, but the /sse body's cancel() was never called and the body still accepted writes. Cancelling before the next pull started finished the generator at once, again without cancelling the /sse body. Not run under workerd; live: probe (Cloudflare's edge, 2026-10-08; two throwaway workers.dev Workers, deleted afterwards; 5 runs per row; raw results archived with the reliability orchestrator at handoffs/ci-health-3-data/cp102-edge-probe/results-2026-10-08/): with enable_request_signal, a client that leaves a QUIET stream (one frame, then silence) fires the Worker's request.signal 5/5, seen by the client within 208-288 ms of its abort (a ticking stream 5/5 within 197-615 ms), and when the Worker pipes the DO's quiet /sse with stub.fetch(..., { signal: request.signal }) the DO's own request.signal fires 5/5, seen within 68-114 ms of the abort (upper bounds: the client polled the Worker's records every 250 ms). Without the flag nothing fires within 30 s on any endpoint, not even a stream writing every 50 ms. The response body's cancel never fired in any configuration. So on the edge the prompt close works with the flag, and without it a disconnected client's readers stay open until the run ends; inferred: (unverified) the likely trigger of FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT (docs/dev/follow-ups.md): the refresh-spam tests leave orphaned DO /sse connections; the uncaught 'Unable to enqueue' those runs show is thrown by OpenNext's own wrapper (third-party), not by this code; related: fixed by !332: core StreamHandleOptions.signal + abortableChunks; handleChatStream threads the client's signal through every SSE path and streamHandle owns an upstream AbortController, so a body cancel alone closes the fresh-turn / resume reader; every runtime's handle.stream({ signal }) closes its reader; DOExecutionHandleImpl.stream() passes the signal to its /sse fetch and cancels the body when it stops reading (parseSSEToChunks' finally cancels instead of only releasing the lock); pipeWebResponseToExpress cancels on 'close'. Proven, outside conformance, by runtime-cloudflare/src/__tests__/do-stream-signal-contract.test.ts (all three DO handles: the body is cancelled on abort and on return()) and the real-workerd integration test packages/e2e/src/helpers/do-sse-disconnect-suite.ts (the DO's own /sse connection count reaches 0 after the handle stops; with enable_request_signal at once, without it at the DO's next write); both fail on main; conformance: stream.client-disconnect-closes-upstream (the prompt cell: the cut lands while the run is held silent) on js-chat-host passes with the fix and is red with runtime-js/src/execution/handle-factory.ts reverted to origin/main 88e5230179, at .upstream-closed and .resume-upstream-closed (one upstream reader still open about 5 s after the client disconnected, both phases). On cf-do it is a declared gap (gaps.ts): Miniflare, like wrangler dev, does not report a quiet cut to the Worker within the cell's 5 s close bound, and the held cell writes nothing after it; conformance: stream.client-disconnect-closes-upstream-at-write on cf-do (host tier) cuts while step 1 is held, lets the run write past the cut while it is still live (step 2 holds after one delta), and requires the worker's own open upstream /sse bodies (counted at the chat handler's DO-namespace seam, e2e/src/harness/cf-do/worker/upstream-reader-counter.ts) to reach 0 within 5 s of that write. It is red at .upstream-closed-after-write with both of the DO handle's guards reverted (DOExecutionHandleImpl.stream() ignoring the signal AND closing by releaseLock() only, i.e. main's handle): the worker counter was still 1, 5,022 ms after the write. With only the signal ignored, or only the close reverted, it is green: either guard alone closes the reader at the write, so the cell proves the pair, not either one. Each guard has its own unit red/green in runtime-cloudflare/src/__tests__/do-stream-signal-contract.test.ts (the abort cases for the signal, the return()-without-signal case for the close). On js-chat-host the same scenario is a contrast cell, outside provenBy (green with the runtime-js change reverted too); conformance: on js-chat-host the fresh-turn and resume readers close at the run's next write even without the fix: stream.client-disconnect-closes-upstream-at-write|js-chat-host|host stays green with runtime-js/src/execution/handle-factory.ts reverted to origin/main 88e5230179, while the prompt cell (the run held silent after the cut) goes red under the same revert. So the JS defect is the quiet-run window, proven by the prompt cell only; live: probe (the e2e workerd pool, the real DurableObjectAgentBase, raw stub.fetch('/sse') with the run held mid-text): cancelling the DO's /sse body while a read is pending (reader.cancel(), releaseLock() + body.cancel(), or aborting the fetch's signal) reaches the DO only at its next write (a chunk, or the 30 s SSE heartbeat) unless the Worker has enable_request_signal, with which the DO's own request.signal fires and the connection drops at once; with no read pending the cancel reaches it at once either way. So the fixed DO path closes the connection promptly only with the flag the docs require; related: the prompt DO-side close needs enable_request_signal on the Worker that hosts the DO (the documented config; docs/runtimes/cloudflare-do.md, docs/frontend/ai-sdk.md). Without it, even with the fix, a disconnect while the DO's /sse read is pending holds the connection until the DO's next write (at most the SSE heartbeat interval, 30 s): the two-pool test packages/e2e/src/__tests__/client-disconnect-do-sse.cf.test.ts (default flags: closes at the DO's next write) and client-disconnect-do-sse.request-signal.cf.test.ts (flag on: closes at once). That next-write fallback is local workerd's; on Cloudflare's edge without the flag the Worker never learns of the disconnect at all (the edge probe above), so the flag is required, and handleChatStream warns once per isolate when the runtime reports it off (core requestSignalSupport()); live: probe (examples/opennext-cloudflare-do under real wrangler dev): the incoming request.signal fires on a client disconnect with enable_request_signal (3/3) and never without it (3/3); but a silent stream (one write, then nothing) never fired it within 5 s even with the flag (3/3), so local workerd does not report a quiet disconnect promptly. Whether and when it does was not measured there; W1's cf-do probes (2026-11-13, local workerd, a cut made before any response existed, no write after it) saw request.signal abort about 10 s after the cut on a macOS host, and not within 30 s on the Linux CI runner. Cloudflare's edge reports it promptly; see the edge probe above; severity S3: a resource leak, bounded by the run, with no data loss and no wrong outcome. Each mid-stream refresh can leave a server-side reader (and on the DO an SSE connection) running until the run writes again or ends, and they accumulate under refresh-spam. No production effect beyond the leak itself is established | source, probe, peer-reported | verified | fixed (2 cells) |
| CP-103 | S1 | When companion__sendMessage resumes an INTERRUPTED persistent companion and that resume is then refused (a completion-support refusal of the child's agent) or loses the child to another entry (child_busy, or a lost run-start), or on runtime-js cannot be started at all (no persistent-agent config for the child's type, or no child session), the parent's SubSessionRef is left running. Core's handleSendMessage writes the ref running (with completionDelivered: false) before the runtime starts the child's continuation run, and nothing restores it when that start fails, while the child session stays interrupted with no run. companion__getChildStatus and companion__listChildren then report running for a child nothing drives. On runtime-js the tool result says why (a tool error, or delivered: false, reason: 'child_busy'). On Temporal the resume is a child workflow whose start the parent awaits but whose result it does not, and on CF Workflows a child instance whose refused entry only sends the parent instance an errored event; in both the tool result is recorded before the child's outcome is known and the parent's ref is never restored to interrupted (on CF Workflows a companion__waitForResult in the same instance receives the child's errored event and rewrites the ref to failed, delivered, though the child is interrupted and resumable), so the LLM is told delivered: true for a message the child never received, and on Temporal an untimed companion__waitForResult on it keeps polling, because without a liveness probe only a ref reading interrupted counts as awaiting a resume. DBOS is not affected: it writes the ref only after the child took the message. The CF Durable Object's own companion tools are not affected either: they write the ref only after the child DO's /resume succeeded. | core/src/orchestration/companion-tool-dispatch.ts:808-810 (pre-fix, origin/main 93cb5c49bf) (wasInterrupted from the child session's own status) and :870-876 (pre-fix, origin/main 93cb5c49bf) (the ref is written status 'running', plus completionDelivered: false for an interrupted child) before :878-887 (pre-fix, origin/main 93cb5c49bf) return { delivered: true, resumeMetadata } for the runtime to start the child's continuation run; the message rides that run's run-start commit, so nothing has reached the child yet; runtime-js/src/run-loop.ts:5718-5780 (pre-fix, origin/main 93cb5c49bf) (resumeChildAgent: the completion refusal at :5720 throws before any child read or write; a child no longer interrupted returns 'child_busy' at :5740; a lost run-start claim returns 'child_busy' at :5779) called from :6466-6496 (pre-fix, origin/main 93cb5c49bf), which maps 'child_busy' to { delivered: false, reason: 'child_busy' }, while a refusal throws into the catch at :6498 (pre-fix, origin/main 93cb5c49bf) (a failed tool result). A child session that is not found throws the same way (:5732 (pre-fix, origin/main 93cb5c49bf)), and a child type with no persistent-agent config is { delivered: false, reason: 'no_resume_config' } with no resume (:6487-6492 (pre-fix, origin/main 93cb5c49bf)). None of these paths writes the parent's ref back.; runtime-temporal/src/workflow-dispatch.ts:889-924 (pre-fix, origin/main 93cb5c49bf) (sendMessage to an interrupted child: wf.startChild of a fresh detached child workflow, PARENT_CLOSE_POLICY_ABANDON; its start is awaited, its result never is; a start failure is only logged at :921-926) with runtime-temporal/src/activities.ts:8724-8738 (pre-fix, origin/main 93cb5c49bf) (the activity strips resumeMetadata and leaves the LLM's tool result { delivered: true }). The child's entry refuses before its run-start commit (activities.ts:9331 (pre-fix, origin/main 93cb5c49bf) initializeAgentState, :1184 (pre-fix, origin/main 93cb5c49bf) startRunCommit), and :7206-7252 (pre-fix, origin/main 93cb5c49bf) (persistPreCommitFailure) leaves a session that already has a run untouched, so the child stays interrupted; a lost run-start ends the child workflow without touching the session (runtime-temporal/src/workflow.ts:1815-1830 (pre-fix, origin/main 93cb5c49bf)); runtime-cloudflare/src/workflow.ts:4113-4174 (origin/main fdba7888ef) (sendMessage to an interrupted child: createChildInstance of a fresh-id resume instance, then the parent records the tool result at once) with runtime-cloudflare/src/steps.ts:6930-6943 (origin/main fdba7888ef) (that result is { delivered: true }). The child instance's completion refusal (runtime-cloudflare/src/workflow.ts:1066-1092 (origin/main fdba7888ef)) ends through the catch at :4540-4570 (origin/main fdba7888ef): a refusal on an existing session writes nothing (:4553-4560 (origin/main fdba7888ef)) and only sends a sub-agent-complete 'errored' event to the parent instance (:4562-4569 (origin/main fdba7888ef)); no ref write. A companion__waitForResult in the same parent instance then receives that 'errored' event: resolvePersistentChildRefStatus maps it to 'failed' (runtime-cloudflare/src/companion-ref-status.ts:44-57), and the record step writes the ref 'failed' with completedAt and completionDelivered: true (runtime-cloudflare/src/workflow.ts:3818-3826 (origin/main fdba7888ef), :3862-3868 (origin/main fdba7888ef)), so the ref is corrected away from 'running' but not to 'interrupted', and the resumable child is reported failed; core/src/orchestration/companion-tool-dispatch.ts:983-1027 (pre-fix, origin/main 93cb5c49bf) (listChildren and getChildStatus return the ref's status), :938-958 (pre-fix, origin/main 93cb5c49bf) (the lazy sync maps only a completed or failed child session onto the ref; an interrupted child's ref is left as it is) and :1234-1246 (pre-fix, origin/main 93cb5c49bf) (isAwaitingResume: with no isChildExecutionLive probe only ref.status === 'interrupted' counts, :1238-1240 (pre-fix, origin/main 93cb5c49bf); with the probe, a 'running' ref whose child is interrupted and not live counts). runtime-js/src/run-loop.ts:6405-6412 (pre-fix, origin/main 93cb5c49bf) is the only runtime that passes the probe, so on runtime-js an untimed waitForResult returns the interrupted guidance, and on Temporal it does not; inferred: (unverified) how long Temporal's untimed waitForResult polls: the companion activity runs under the workflow's default activity options (runtime-temporal/src/workflow.ts:285-295 (pre-fix, origin/main 93cb5c49bf), startToCloseTimeout 5 minutes, maximumAttempts 3), so it presumably ends in an activity timeout and its retries rather than a result; inferred: (unverified) what a later CF Workflows companion__waitForResult on that child sees: it waits for a sub-agent-complete event (runtime-cloudflare/src/workflow.ts:3752-3810 (origin/main fdba7888ef)); the refused child's 'errored' event went to the instance that ran the sendMessage, so a wait in that same instance may receive it, and one in a later turn's instance waits until subAgentEventTimeout; runtime-dbos/src/steps/execute-companion-tool.ts:1372-1378 (pre-fix, origin/main 487090d3fb) (a child workflow that is not live is answered { delivered: false, reason: 'child_not_running' } with no write) and :1408-1418 (pre-fix, origin/main 487090d3fb) (after the inbox send, a message the child did not take is { delivered: false } with no write); the ref is written 'running' only at :1425-1429 (pre-fix, origin/main 487090d3fb), after the child took it; contrast: runtime-cloudflare/src/do/persistent-companion-tools.ts:715-740 (pre-fix, origin/main 137b07d57a) (the CF DO's own sendMessage: the child DO's /resume first, which throws on a non-2xx answer, :215-233, and only then the ref write at :734-737); a DO without subAgentNamespace never reaches core's handleSendMessage, because runtime-cloudflare/src/do/durable-object-agent-base.ts refuses any agent declaring persistentAgents with 400 persistent_agents_no_subagent_namespace (runtime-cloudflare/src/do/durable-object-agent-base.ts:2369-2381 (pre-fix, origin/main 137b07d57a), and :2822 (pre-fix, origin/main 11b3caf8a0) / :3197 (pre-fix, origin/main 137b07d57a) on the other entry routes); related: CP-46 (Temporal: a wf.startChild that throws for a companion spawn or resume is only logged, the tool reports success, the ref stays 'running'; runtime-temporal/src/workflow-dispatch.ts:880-886 (pre-fix, origin/main 93cb5c49bf), :921-927 (pre-fix, origin/main 93cb5c49bf)) and CP-72 (Temporal: two sendMessage calls to one interrupted child in one step share a child workflow id, so the second start throws). The ledger split: CP-46 owns a Temporal child workflow that never starts (any cause, CP-72's id collision among them); CP-103 owns a resume whose child workflow or instance does start, or on runtime-js is attempted in process, and whose entry then refuses or loses the child. The two share Temporal's log site but not the failure: in CP-103 wf.startChild succeeds; related: CP-68 (fixed: the resume now keys on the child session's own status, so a later sendMessage still resumes the interrupted child, and is refused again while the cause holds); CP-23 (a parent interrupt does not reach blocking companions); MA-22 (a CF DO child's own stream left open after some drives). The refusal at resume needs the child's agent or adapter to change after its spawn (the spawn applies the same check, runtime-js/src/run-loop.ts:5641 (pre-fix, origin/main 93cb5c49bf)), e.g. a deploy between turns; related: fix: !325 (the completion-strategies test-hardening MR), its Task 16 / 16B (A1) and Task 16 Step 5: a refused or child_busy interrupted-companion resume never leaves the parent's ref running, on runtime-js, Temporal and CF Workflows. Core no longer writes the ref before the resume; each runtime's resume entry carries StartRun.resumedCompanionRef, so the store writes the ref inside the child's run-start commit; Temporal and CF Workflows refuse an unsupported child model parent-side, before the child starts (the fix evidence below). A child_busy resume's delivered: true on Temporal and CF Workflows is CP-105; core/src/orchestration/companion-tool-dispatch.ts:921-936 (pre-fix, origin/main 487090d3fb) (fix evidence, core: !325 Task 16 / 16B) (handleSendMessage writes the ref only when the message is not deferred, :928 (pre-fix, origin/main 487090d3fb); a deferred message to an interrupted child with no live run writes none) with core/src/store/message-commit.ts:186-204 (as before WA MR2, origin/main cae966c0a7) (StartRun.resumedCompanionRef: the store turns the parent's ref 'running', completionDelivered: false, in the same atomic step as the child's run-start commit; a refused or lost start writes nothing). Each runtime's resume entry passes it: runtime-js/src/run-loop.ts:6235-6239 (pre-fix, origin/main 487090d3fb) (resumeChildAgent's run-start plan, sent by runtime-js/src/run-start.ts:332 (pre-fix, origin/main 487090d3fb)) (!325 Task 16 / 16B); runtime-temporal/src/activities.ts:9731-9739 (pre-fix, origin/main 487090d3fb), :9792 (pre-fix, origin/main 487090d3fb) and :9799 (pre-fix, origin/main 487090d3fb) (initializeAgentState for a companionResume child workflow) (!325 Task 16 / 16B); runtime-cloudflare/src/workflow.ts:1445-1452 (pre-fix, origin/main e96f88bc98) (a companionResume instance's entry), runtime-cloudflare/src/steps.ts:1176-1184 (pre-fix, origin/main 487090d3fb) and runtime-cloudflare/src/run-entry.ts:1082 (pre-fix, origin/main 487090d3fb) (!325 Task 16 Step 5). The parent-side completion refusal, a tool error before any child starts: runtime-temporal/src/activities.ts:9073-9098 (pre-fix, origin/main 487090d3fb) (!325 Task 16 / 16B) and runtime-cloudflare/src/steps.ts:7700-7712 (pre-fix, origin/main 487090d3fb) with :7741-7751 (pre-fix, origin/main 487090d3fb) (companionResumeRefusal; WA MR1 (!340) changes only its unregistered-type check to core isAgentNotFoundError, :7746 (pre-fix, origin/main 487090d3fb)) (!325 Task 16 Step 5); severity S1 (re-argued on consequence, per the owner's ruling that severity follows consequence, not frequency): wrong model behaviour and a silently lost message, by the spec's S1 definition. On Temporal and CF Workflows the parent's LLM is told delivered: true for a message the companion never received, so it goes on as if the companion had it (it waits for, or relies on, work that was never requested), and the message itself is dropped with no error; the ref then reports the companion running though nothing drives it, and on Temporal an untimed waitForResult on it does not return. On runtime-js the tool result is truthful (a tool error or delivered: false) and only the ref lies, which alone would be S2. CP-46, rated S2 at 5bd30e20da, has the same consequence since !288 (bd26dc9b57) made an interrupted child's message ride the continuation's run-start instead of being appended to the child's log first, and is re-rated S1 with this record The companion stays interrupted and resumable, so a later sendMessage can still reach it once the cause is gone; conformance: companion.resume-refused-keeps-ref and companion.resume-busy-keeps-ref (executor; the companion seeded interrupted, owner ruling R9; the refusal cell's control companion resumes on the same seed, R1). Pre-fix (the CP-103 core, runtime-js and Temporal fix reverted over the 16A part 2 harness) every cell failed .ref-not-running (the parent's ref read 'running') on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres (both Temporal servers) and cfw-workflows-d1, and the refusal cell also failed .no-false-delivery ({ delivered: true }) on temporal-* and cfw-workflows-d1. On the fix (!325 Task 16 / 16B and Task 16 Step 5) every id of all 14 cells passes, 3 of 3 runs per lane (Temporal on both HELIX_TEMPORAL_BACKEND servers: the test-server and a Temporal dev server), -t 'companion\.resume-(refused|busy)-keeps-ref\|', no helix-agents pipeline running. Each cell went red under a minimal working-tree mutation of the fix mechanism its pass depends on (never an MR revert; the mutated package's dist rebuilt, one run per lane, then reverted, rebuilt and green again on every lane): (M-core) core/src/orchestration/companion-tool-dispatch.ts:928 (pre-fix, origin/main 487090d3fb), the deferred 'running' write restored (the !deferMessage condition made always true): exactly .ref-not-running fails on all 14 cells (js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres on both servers, cfw-workflows-d1); (M-js, two sites) runtime-js/src/run-loop.ts:6239 (pre-fix, origin/main 487090d3fb) resumedCompanionRef dropped from the run-start plan AND a pre-start 'running' ref write added at the top of resumeChildAgent, :6184 (pre-fix, origin/main 65061fbeb5), before its completion refusal: exactly .ref-not-running fails on both cells on js-memory, js-redis and js-postgres; (M-temporal, two sites) runtime-temporal/src/activities.ts:9738 (pre-fix, origin/main 487090d3fb) resumedCompanionRef dropped from initializeAgentState's plan AND a pre-start 'running' ref write added before its completion check, :9725 (pre-fix, origin/main 487090d3fb): exactly companion.resume-busy-keeps-ref.ref-not-running fails on temporal-memory, temporal-redis and temporal-postgres on both servers, while the refusal cells pass (the parent activity's completion refusal, :9083 (pre-fix, origin/main 487090d3fb), stops the resume before any child entry, so the child-entry write is never reached; M-core reddens those cells); (M-cfw-precheck) runtime-cloudflare/src/steps.ts:7709 (pre-fix, origin/main 487090d3fb), the parent pre-check's result discarded (always undefined): exactly companion.resume-refused-keeps-ref.no-false-delivery fails on cfw-workflows-d1, the busy cell passes; (M-cfw-write, two sites) runtime-cloudflare/src/steps.ts:1179-1184 (pre-fix, origin/main 65061fbeb5), resumedCompanionRef dropped from the entry plan AND a pre-commit 'running' ref write added in that plan: exactly companion.resume-busy-keeps-ref.ref-not-running fails on cfw-workflows-d1, the refusal cell passes (the pre-check refuses before any child instance exists). dbos-postgres is a declared gap for both scenarios, and that gap is not an unproven lane: DBOS was never affected (the contrast above: it writes the ref only after the child took the message), it has no completion refusal before completion strategies MR 3 (R2), and it never resumes an interrupted companion from sendMessage, so there is no resume commit to hold | source, peer-reported | partly-inferred | fixed (14 cells) |
| CP-104 | S1 | On CF Workflows, a client-tool answer that the chat handler has already stored is stranded when the resume that should consume it is refused: the session stays paused with the answer stored and nothing continues the turn without a new client POST. A refreshed client's auto-send POST takes handleChatStream's resume path. Its submit (Phase A) stores the answer on the pending entry (submittedResult / submittedAt), which stays in pendingClientToolCalls. Its resume() (Phase B) first waits, up to priorInstanceSettleTimeoutMs (30 s by default), for the session's previous instance to close; if that instance is still live it throws AgentAlreadyRunningError ('the previous run (…) is still finishing') having written nothing. The handler's catch reloads the session and treats the call as still pending because its entry is still in the map, though it is answered, so it skips its attach fallback and returns only a data-resume-rejected frame. CF Workflows does not resume on a submit and has no recovery for a paused session owed its continuation, and the client does not send again (its auto-send fires once per tool call), so the user's answer is accepted and the turn does not continue until the client POSTs again. A page reload recovers it: a reloaded client has a fresh auto-send set, so re-answering the call POSTs again, the submit is 'already_completed' and the resume, once the previous instance has closed, consumes the stored answer. The trigger is rare and load-dependent: the previous instance must still be live 30 s after its suspension commit. Fixed on CF Workflows by 62f880c3bf: a complete batch's submit now starts its own submit continuation (new_run), and a refused caller_resumes resume is a typed HTTP error, not a rejection frame; open only for its chat-handler cell. | ai-sdk/src/handler/handle-chat-stream.ts:723-788 (origin/main fdba7888ef) (Phase A: each intent is submitted; 'accepted' and 'already_completed' both count as success) and :803-827 (origin/main fdba7888ef) (the partial-batch check: a whole batch answered goes on to the resume), then :835-922 (origin/main fdba7888ef) (Phase B: callExecutorResume; its catch reloads the session, :863-868 (origin/main fdba7888ef), and computes stillPending at :869-871 (origin/main fdba7888ef) as 'the entry is still in pendingClientToolCalls', which a submitted, not yet drained entry is; the attach fallback runs only when !stillPending, :884-889 (origin/main fdba7888ef); otherwise :918-922 (origin/main fdba7888ef) returns rejectedOnlyResponse with the error's message); runtime-cloudflare/src/executor.ts:1678-1790 (origin/main fdba7888ef) (persistDurableSubmit: the answer is written onto the pending entry, submittedResult / submittedError and submittedAt at :1736-1750 (origin/main fdba7888ef), and the entry stays in the map; a second submit of the same call is 'already_completed', :1724-1733 (origin/main fdba7888ef)) and :1792-1835 (origin/main fdba7888ef) (submitToolResult: a durable write only, no event to the workflow and no resume; the drain happens in the resume workflow); runtime-cloudflare/src/executor.ts:1407-1419 (origin/main fdba7888ef) (resume: a session that is not 'active' first awaits the previous instance) and :1372-1391 (origin/main fdba7888ef) (awaitPriorInstanceSettled polls the session's current instance until it is no longer live, bounded by priorInstanceSettleTimeoutMs, default 30 000 ms at :247 (origin/main fdba7888ef), and throws AgentAlreadyRunningError(sessionId, 'the previous run (<instance id>) is still finishing') when the bound lapses; nothing is written before it); ai-sdk/src/react/send-automatically-when.ts:60-80, 116-119 (createHelixSendAutomaticallyWhen auto-sends once per tool call id, the sent-set check at :116-119; the same answered call never triggers another POST) and core/src/client-tool/pending-batch.ts:64-76 (pre-fix, origin/main 16de0b648c) (hasUnconsumedSubmit: a paused session whose whole batch carries submittedAt, 'a session owed its continuation'). Its only runtime callers are the CF DO's (runtime-cloudflare/src/do/durable-object-agent-base.ts:520 (origin/main fdba7888ef), :1229 (origin/main fdba7888ef), :1431 (origin/main fdba7888ef)), also imported by runtime-cloudflare/src/do/wake-recovery.ts:240 (pre-fix, origin/main 16de0b648c); nothing in CF Workflows reads it; live: CI, e2e packages/e2e/src/__tests__/refresh-c1.integ.test.ts, 'C-1 refresh mid a multi-step / HITL response (RM-08) — cfw-d1 > ai 6.0.49: refresh while the client tool is pending: client B runs it once, sends the output, char-exact' failed with 'snapshot of c1-pd-muyfoqu7-u7lev not terminal within 30000ms (last: paused)' in job 17009383743 (pipeline 2923488532, !310 at 5d8db7cf3c): the paused end state this path leaves. The job log does not name the rejection, so which refusal it was is inferred from source. It passed 60/60 locally for the reporter; no local repro; inferred: (unverified) the same test's ai 6.0.281 failure in job 17007855812 (pipeline 2923294110, !310 at f5150d9d2b), 'not terminal within 30000ms (last: active)', is a different end state and may be a different stall; and why the previous instance is still live 30 s after its suspension commit is unexplained. The diagnostics that name the stuck layer are on !300's branch (04d945265a), with FU-FLAKE-REFRESH-C1-CFW-PENDING-CLIENT-TOOL-NOT-TERMINAL; contrast: the CF Durable Object is not stranded by the same check. Its submit wakes the run itself (runtime-cloudflare/src/do/durable-object-agent-base.ts:3536 (origin/main fdba7888ef)), and a submit that lands while a run is live is kept as a pending wake that is re-checked when that run settles (runtime-cloudflare/src/do/durable-object-agent-base.ts:793-815 (origin/main fdba7888ef)), with hasUnconsumedSubmit recovery on wake (runtime-cloudflare/src/do/durable-object-agent-base.ts:520 (origin/main fdba7888ef)). So a refused explicit resume there still leaves the turn continuing; the client sees only the rejection frame instead of the continuation, which is CP-53; inferred: (unverified) runtime-js, Temporal and DBOS: the handler's stillPending check is runtime-agnostic, and none of them resumes on a submit, so a resume of theirs refused after the submit would strand the answer the same way. Whether their resume can be refused in that window was not established (Temporal's resume of a session whose previous workflow has not closed yet is the nearest case, cf. CP-84); related: CP-53 (the same stillPending read on the CF DO, where the woken run continues with no listener), CP-52 (fixed: a DO submit's wake dropped while a run tears down), CP-81 (a submitted result never consumed, the session stranded paused), CP-102 (the chat handler's fresh-turn and resume responses), DI-39 (a chat resume refused for completion support also ends in this rejectedOnlyResponse); related: fix planned (user decision): CF-D2, after !316. A server-side retry of the refused resume, then a typed retryable error instead of the 200 rejection frame; stillPending treats a submitted entry as pending but answered; and a cross-runtime check of the same path; severity S1: lost user action. The user's tool answer is accepted and stored, but the turn never continues and nothing tells the user it will not: the client gets one rejection frame for a transient condition, does not retry, and the session stays paused. The trigger is rare and load-dependent (the previous instance still live past the 30 s settle window), which bounds how often it happens, not what it costs when it does; related: fixed in code by 62f880c3bf (W1 MR2b, the claim-driven submit continuation), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-cloudflare/src/executor.ts:4995-5025 (origin/main fb9150cc16) (a complete batch's submit starts its own __submit__ continuation and answers new_run) and :4206-4226 (origin/main fb9150cc16) (a paused session with a suspended run and a ready batch always gets that instance); ai-sdk/src/handler/handle-chat-stream.ts:1375-1467 (origin/main fb9150cc16) (the handler attaches on new_run / deferred and makes no resume() call). Residual: in the caller_resumes case resume() can still be refused by the prior-instance wait (executor.ts:3321-3337 (origin/main fb9150cc16)), typed 409. Open until its cfw-workflows-d1 cell (the pending reason's submit.continuation-kind cell does not exist yet); related: fixed in code by 62f880c3bf (W1 MR2b, the claim-driven submit continuation), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/executor.ts:5093-5123 (origin/main 56d564d073) (a submit that completes the batch ensures the root's __submit__ continuation and answers new_run; caller_resumes only when nothing is owed), ai-sdk/src/handler/handle-chat-stream.ts:1379-1404 (origin/main 56d564d073) (on new_run / deferred the handler attaches to the session stream and makes no resume() call; the stillPending read is gone), ai-sdk/src/handler/handle-chat-stream.ts:1473-1492 (origin/main 56d564d073) (on caller_resumes a refused resume() is answered as a typed HTTP error with inputCommitted, not a data-resume-rejected frame) | source, live-repro, peer-reported | partly-inferred | open |
| CP-105 | S1 | Temporal and CF Workflows: a sendMessage resume whose started child loses its run-start (child_busy) reports delivered: true. companion__sendMessage to an interrupted persistent companion starts the resumed child fire-and-forget (Temporal: a detached wf.startChild; CF Workflows: createChildInstance of a __resume__ instance), and the parent records the tool result at once, before the child's entry runs. When that entry's run-start commit then LOSES to another entry that took the child (child_busy), the parent's ref correctly stays interrupted (CP-103's fix) but the LLM was already told delivered: true for a message the companion never received: neither workflow learns the entry's outcome. runtime-js reports { delivered: false, reason: 'child_busy' } in the same case. Fixed on CF Workflows by 62f880c3bf (W1 MR2b, !344); still open on Temporal. | runtime-temporal/src/workflow-dispatch.ts:889-935 (sendMessage to an interrupted child — wf.startChild of a fresh detached __resume__ child workflow with companionResume: true, PARENT_CLOSE_POLICY_ABANDON; its start is awaited, its result never is) with runtime-temporal/src/activities.ts:8750-8775 (origin/main 16de0b648c) (the companion activity strips resumeMetadata and leaves core's tool result { delivered: true }, core/src/orchestration/companion-tool-dispatch.ts:889). The child's lost run-start ends the child workflow as a start_conflict failure without touching the session or notifying the parent (runtime-temporal/src/workflow.ts:1798-1833 (as before WA MR2, origin/main cae966c0a7)); runtime-cloudflare/src/workflow.ts:4154-4222 (pre-fix, origin/main 16de0b648c) (sendMessage to an interrupted child: createChildInstance of the __resume__ instance in its own step.do, :4181-4211, then the parent records the companion tool result at once, :4213-4222) with runtime-cloudflare/src/steps.ts:7059-7070 (pre-fix, origin/main 16de0b648c) (that result is { delivered: toolResult.delivered }, core's true); nothing waits for the child's entry outcome; contrast: runtime-js/src/run-loop.ts:5770 and :5814 (origin/main 16de0b648c) (resumeChildAgent returns 'child_busy' for a child no longer interrupted or a lost run-start claim) and :6518-6520 (origin/main 16de0b648c) (mapped to { delivered: false, reason: 'child_busy' }); runtime-dbos/src/steps/execute-companion-tool.ts:1198-1204 (origin/main 16de0b648c) (DBOS answers a sendMessage to an interrupted, not-live child { delivered: false, reason: 'child_not_running' } and never resumes it) and :878-889 (origin/main 16de0b648c) (its continuation outcome: delivered: false with child_busy, child_start_unconfirmed or child_entry_failed once the child's run-start outcome is known or the wait lapses); conformance: companion.resume-busy-delivery (executor) on temporal-memory, temporal-redis, temporal-postgres (both HELIX_TEMPORAL_BACKEND servers) and cfw-workflows-d1 fails .tool-result-not-delivered ({ delivered: true }) both before and after CP-103's fix: the parent's resume of a seeded interrupted companion is held at its run-start commit (hold-entry-commit, before-commit) while a direct resume() of the child takes it and is held live at a tool gate; released, the held commit settles as a typed run-start loser (.resume-lost, the store's own answer), and the parent's send-1 result still reads { delivered: true }. The js-* cells pass ({ delivered: false, reason: 'child_busy' }) as the contrast; dbos-postgres is a declared gap (no resume from sendMessage). The cells stay failing until the Temporal wf.startChild resume and the CFW createChildInstance resume learn their child's entry outcome; related: FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW (docs/dev/follow-ups.md: the shared delivery contract both halves must land with — the child signals its entry outcome, the parent applies a bounded patch-gated wait, the tool result reports child_busy or child_start_unconfirmed); CP-103 (fixed for the ref; this record owns the tool result of a child_busy resume on Temporal and CF Workflows); CP-46 (Temporal: a wf.startChild that throws is only logged and reported as success — the child never starts; here it starts and then loses); related: fix owners: Temporal — the next Temporal bucket (orchestrator); CF Workflows — CF-D2 (confirmed by the orchestrator); severity S1: wrong model behaviour and a silently lost message. The parent's LLM is told its message was delivered, so it goes on as if the companion had it, and the message itself is dropped with no error. The trigger needs another entry to take the child inside the resume's window, which bounds how often it happens, not what it costs; runtime-cloudflare/src/workflow.ts:4390-4415 and :5030-5111 (as before WA MR2, origin/main cae966c0a7) (fix evidence, W1 MR2b !344, the CF Workflows half: after creating the __resume__ child instance, the parent no longer records core's delivered: true at once. awaitCompanionEntryOutcome polls the store in the workflow body, a pollInBody of 100 ms to 5 s bounded by companionEntryOutcomeTimeoutMs, 60 s by default, for the child's deterministic run id: the run existing is the run-start commit that took the message, so delivered: true; a closed instance with no run whose failure output is state_run_start_conflict is delivered: false, reason child_busy, at :5065-5070; any other failure is child_entry_failed with its errorDetail; the bound lapsing is child_start_unconfirmed. The parent records that outcome as the tool result at :4407-4415, and the detached sendMessage continue does the same at :4291-4312) with core/src/types/companion-delivery.ts:10-39 (as before WA MR2, origin/main cae966c0a7) (the shared CompanionDeliveryOutcome union and its schema), proven by runtime-cloudflare/src/__tests__/cfw-companion-entry-poll.test.ts:70 (pre-fix, origin/main e96f88bc98) (closed with a lost run start: child_busy), :54 (pre-fix, origin/main e96f88bc98), :137 (pre-fix, origin/main e96f88bc98), :233 (pre-fix, origin/main e96f88bc98) and :252 (pre-fix, origin/main e96f88bc98); conformance: narrowed by W1 MR2b: companion.resume-busy-delivery|cfw-workflows-d1|executor passes (owner slot on W1 MR2b's post-fix head 61ca81d8f8 (branch reliability-cf-w1-mr2b-t8b: MR2b + WA MR1 8e4921809a + WE-2 9e9c382563; dist built by W1, 39/39 turbo tasks), 2026-10-11; the ledgered .tool-result-not-delivered reported UNEXPECTED PASS). The Temporal lanes stay ledgered (Temporal is parked); related: fixed in code by 62f880c3bf (W1 MR2b, !344; the CF Workflows half), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/workflow.ts:4338-4352 (origin/main 56d564d073) (a sendMessage to an interrupted companion, after creating the __resume__ child instance, polls in the body for that child's run-start outcome), runtime-cloudflare/src/workflow.ts:4357-4366 (origin/main 56d564d073) (the tool result is core's delivery from that entry outcome, never delivered: true before it), runtime-cloudflare/src/workflow.ts:7773 (origin/main 56d564d073) (a lost run start answers child_busy); still open on Temporal: runtime-temporal/src/workflow-dispatch.ts:889-935 (origin/main 56d564d073) (a detached wf.startChild resume whose start error is only logged and whose entry outcome is never read, so core's delivered: true stands) | source, live-repro | verified | open |
| CP-106 | S2 | An interrupt that discards a step which already spawned a persistent companion leaves the companion's sub-session ref, and on some runtimes a running child, that no persisted message records. Core handleSpawnAgent writes the ref and starts the child directly, outside the step's staged commit. When the runtime then discards the step (runtime-js and the CF Durable Object: the abort check between the tools and the commit; DBOS: the step commit refused on interrupt), the parent ends interrupted with no companion__spawnAgent call or result in its history, but keeps a running ref whose parentToolCallId is in no message. The model never saw the spawn, yet listChildren lists the child, a re-spawn of the name collides, and on the CF DO and DBOS the child runs on. | core/src/orchestration/companion-tool-dispatch.ts:593 (pre-fix, origin/main 487090d3fb) (handleSpawnAgent: addSubSessionRefs, status running, parentToolCallId = the LLM call id) and :613 (pre-fix, origin/main 487090d3fb) (deps.spawnPersistentChild), both direct store / runtime writes outside the step commit; core/src/orchestration/step-iterator.ts:2260-2274 (the abort check between tool execution and the commit returns abortedOutcome(): the step's messages are never committed) with runtime-js/src/run-loop.ts:1268-1300 and :4940-5000 (as before WA MR2, origin/main cae966c0a7) (the soft-interrupt path truncates to the latest checkpoint; refs are not touched). The CF DO runs this executor, and its child is a sibling Durable Object started over /start (runtime-cloudflare/src/do/persistent-companion-tools.ts:703 (pre-fix, origin/main e96f88bc98)), so it runs on; on runtime-js the in-process child's controller is chained to the parent's signal (runtime-js/src/run-loop.ts:6092-6101 (as before WA MR2, origin/main cae966c0a7)), so the child is interrupted but its ref is still left behind; runtime-dbos/src/steps/execute-companion-tool.ts:736 (pre-fix, origin/main 487090d3fb) (ref written) and :799 (pre-fix, origin/main 487090d3fb) (child workflow started) run in the workflow body (phase 3), before the step commit, whose first attempt passes refuseIfInterrupted (runtime-dbos/src/workflows/shared.ts:1866 (as before WA MR2, origin/main cae966c0a7); the RM-61 discard). A stop() through the active handle cancels the workflow at once (runtime-dbos/src/lifecycle/interrupt.ts:94-98 (as before WA MR2, origin/main cae966c0a7)), so phase 3 never runs; only a flag-only interrupt (a reconnect handle) reaches the window; contrast: Temporal reads the interrupt flag only at iteration boundaries (runtime-temporal/src/workflow.ts:1144 (as before WA MR2, origin/main cae966c0a7)), never between dispatchCompanionTools (:1395) and commitStep (:1561): the spawn is always committed. Unverified: a workflowHandle.cancel() (abort) mid-step, with no CancellationScope, could leave the same shape; contrast: CF Workflows commits the step first (runtime-cloudflare/src/workflow.ts:3000 (pre-fix, origin/main e96f88bc98)) and runs companions afterwards in their own step.do (:3891 (pre-fix, origin/main 487090d3fb)), recording each with an append commit. The inverse case, a pre-subagent interrupt (:2914-2930 (as before WA MR2, origin/main cae966c0a7)) leaving a committed spawn call with no result, no ref and no child, is repaired by the C5 trailing heal on the next turn, resume or retry (core transcript.ts findTrailingUnpairedToolCalls / buildTrailingHealMessages; runtime-cloudflare/src/steps.ts:952 (pre-fix, origin/main 487090d3fb), :1104-1107 (pre-fix, origin/main 487090d3fb); runtime-cloudflare/src/run-entry.ts:1833 (pre-fix, origin/main e96f88bc98)), as C5 allows; live: WE-2 (cf-we-do-streams-2, 2026-10-10, main a07e5aa1fb): on the CF DO an interrupt sent right after /start left the parent interrupted with only the user message in its history, a sub_session_refs row {name: 'writer-1', parent_tool_call_id: 'sp-1', status: 'running'}, and the child DO completed (1 of 5 unheld runs; 3 of 3 in isolated runs of the old interrupt test). Controls: an interrupt before the spawn left no ref and no child; one while the child was held recorded the spawn; conformance: companion.spawn-discarded-on-stop and companion.spawn-discarded-on-flag-interrupt (executor tier): the parent step calls companion__spawnAgent and a gated tool together, and the interrupt lands while the gate holds. The CF DO half is proven by its host-tier sibling below (the cf-do worker now binds its namespace as subAgentNamespace, FU-CONF-CFDO-SUBAGENT-NAMESPACE); related: fix direction (WD backlog): at the core seam, never per runtime: make the spawn part of the step commit (stage the ref with the step and start the child only after the commit), or compensate on discard (remove or settle the ref, stop the child); conformance: companion.spawn-discarded-on-flag-interrupt fails .refs-recorded on js-memory, js-redis and js-postgres at WA MR2's pre-run head b43b8e1a31 and at its pre-merge head 53395d4bbf, and passed at main cae966c0a7 with the old flag scenario (WA investigation, Ruling 65). The cause is runtime-js/src/run-loop.ts:3868 (pre-fix, origin/main 65061fbeb5): the step commit is refused by refuseIfStopped once the stop is recorded while the gated tool holds the step, and the spawn's ref was written outside that commit. A mutation that turns refuseIfStopped off at that commit only makes the cell pass again. core handleSpawnAgent is unchanged since main. Ledgered under this record, not a new finding; conformance: Temporal now has the same shape at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos): companion.spawn-discarded-on-stop and companion.spawn-discarded-on-flag-interrupt fail .refs-recorded and .no-unrecorded-child-finishes on temporal-memory, temporal-redis and temporal-postgres (the parent's history holds only the user message, and the spawned child ran on and finished its turn). WA MR2's refuseIfStopped now refuses the spawning step's commit when the stop lands mid-step (Ruling 65), so the spawn is no longer always committed, as the contrast line above says it was before WA MR2; conformance: companion.spawn-discarded-on-stop-host (owner batch 26, host tier, cf-do): the parent's step calls companion__spawnAgent and a gated tool, and the host's stop route interrupts while the gate holds; the companion runs in a sibling DO (the worker's own namespace as subAgentNamespace). On main 8dc3a5c967 plus batches 24 and 25 it fails .refs-recorded and .no-unrecorded-child-finishes, 3 of 3: the parent is interrupted with only the user message in its history, keeps a ref to helper, and the child DO finishes its turn, WE-2's live repro as a cell. js-chat-host is a declared gap (no tool-gate port); conformance: at WA MR2 the CF DO's stop cascades to the stopped run's children (Ruling 68), so the unrecorded companion is stopped and no longer finishes its turn: companion.spawn-discarded-on-stop-host fails only .refs-recorded on cf-do, 4 of 4 at MR2's final head (WA's re-run, one under load and three alone); .child-settles and .no-unrecorded-child-finishes pass. The spawn's ref is still written outside the parent's history, so the record stays open; conformance: companion.blocking-spawn-stopped-during-wait (C-3; owner batch 29, executor tier and its cf-do -host sibling; WA spec 2026-10-14-cp106-post-commit-child-phase-design.md §9): the parent's step spawns a BLOCKING companion whose child is held inside its gated tool, and stop() lands while the parent waits. On main 34898cacaa plus batch 29, 3 of 3: js-memory, js-redis and js-postgres fail .refs-recorded and .spawn-call-paired (the parent's history is ["user:go"]: the step was discarded, CP-106's shape); temporal-memory, temporal-redis, temporal-postgres and cfw-workflows-d1 record and pair the spawn but fail .child-ended-by-stop (the child completes after the stop); dbos-postgres fails .refs-recorded, .spawn-call-paired and .child-ended-by-stop (the child stays active; its .parent-interrupted is CP-57's); cf-do fails .refs-recorded, .spawn-call-paired and .child-ended-by-stop (the child DO completes). js-chat-host is a declared gap (no tool-gate port); conformance: C-3 on main e96f88bc98 (!355, WA MR2) plus batch 29, 3 of 3: js unchanged. Temporal now discards the stopped step too, so temporal-memory, temporal-redis and temporal-postgres also fail .refs-recorded and .spawn-call-paired (the history is ["user:go"]; the contrast above, that Temporal always commits the spawn, no longer holds after !355's run-scoped stop) as well as .child-ended-by-stop. On cfw-workflows-d1 the child completes (.child-ended-by-stop), and since !355 it also leaves the spawn call unpaired, a regression filed as CP-118. On cf-do the DO's stop cascade now stops the child (.child-ended-by-stop passes), and .refs-recorded and .spawn-call-paired still fail; conformance: companion.blocking-spawn-stopped-during-wait-reason (+ -host; owner batch 29): the same setup judged only by .paired-not-executed-run-stopped (the spawn call's result is core's not-executed result with reason run_stopped, what a client reads). On main 34898cacaa plus batch 29 it fails on every executor lane and on cf-do, 3 of 3: on js, dbos and cf-do the spawn call is not in the parent's history; on temporal and cfw it is paired with the child's real completed result. On main e96f88bc98 plus batch 29 it still fails on every executor lane and on cf-do, 3 of 3; conformance: the dbos-postgres cell's red (companion.blocking-spawn-stopped-during-wait .refs-recorded, .spawn-call-paired and .child-ended-by-stop) is unledgered since the CP-57 MR (owner, orchestrator ruling 2026-10-16): the cell is a declared gap there (it needs 'ordered-stop-settle', which the lane does not claim: its post-stop handle outcome is a race, FU-CONF-CP57-DBOS-STOP-SETTLE-ORDER); it is ledgered again from observed runs once the lane claims it; conformance: on CF Workflows, fixed as a consequence of W1's CP-16 + CP-23 fix (branch reliability-cf-w1-cp16-cp23-cfw-stop at 2f0bf8dc89)'s CP-23 park fix (the parent's park hears its own stop and passes it to the child through the sub-agent stop path): companion.blocking-spawn-stopped-during-wait .child-ended-by-stop and -reason .paired-not-executed-run-stopped pass on cfw-workflows-d1 (slot W3); the other lanes' rows stay open; core/src/orchestration/child-phase.ts:176-239 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (runChildPhase: a step's companion calls run after its commit, starting one at a time in call order) and core/src/orchestration/step-iterator.ts:1886-1893 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (the step iterator runs them through the phase once the step is committed), with runtime-js/src/run-loop.ts:9587-9596 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (runtime-js's phase) and runtime-cloudflare/src/do/persistent-companion-tools.ts:2046-2077 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (the CF DO's spawn in the phase: the start fence, then onStarted): on runtime-js and the CF DO a spawn is never outside the step's commit; conformance: fixed on runtime-js and the CF DO by !523 (CP-106 MR A). In !523's MR pipeline 2987684089 at its head b06417e355 (jobs 17455241724 and 17455241720) companion.spawn-discarded-on-stop, companion.spawn-discarded-on-flag-interrupt and companion.blocking-spawn-stopped-during-wait with its -reason and -paired siblings were unexpected-pass on js-memory, js-redis and js-postgres, and companion.spawn-discarded-on-stop-host, companion.blocking-spawn-stopped-during-wait-host and -reason-host on cf-do (host), so their js and cf-do ledger rows are removed. Temporal and DBOS keep theirs: their companion calls still run in the step (MR B); conformance: with WE-2's #181 (!564), proven on #181 ALONE (owner slot s205, 2026-10-29, r6 b0fcf7c185, archived at orch-archive/batch-2026-10-29-1-11774a6828; landed in batch 3's squash f85872de1e) and on the batch-3 tree (tree 9398be4d36), interleaved, 3 of 3 identical on both: companion.spawn-discarded-on-stop passes on temporal-memory, temporal-redis and temporal-postgres; companion.spawn-discarded-on-flag-interrupt passes on those and dbos-postgres; companion.blocking-spawn-stopped-during-wait-paired and -reason pass on the three Temporal lanes (the call paired not_executed / run_stopped, the one shape createNotExecutedToolResult writes); companion.blocking-spawn-stopped-during-wait now fails only .child-ended-by-stop on Temporal (the child still completes after the parent's stop), and -paired / -reason still fail on dbos-postgres. Those rows left the ledger; the finding stays open on the rest | source, live-repro | verified | open |
| CP-107 | S3 | A chat client seeded from the WHOLE snapshot (snapshotResumeOptions(snapshot, { replayPrefix: false }), the profile the product documents for a client that cannot suppress replayed client tools) re-submits a client-tool output the server already consumed: any whole-snapshot-seeded client, on every ai 6.x, through the real useHelixChat on both cf-do and js-chat-host (not the C-1 seed). Client A answers a client tool and its continuation consumes the result; client B, seeded from the whole snapshot mid-continuation, then POSTs the same output again. The server answers 200 SSE carrying only data-resume-rejected { reason: 'unknown-tool-call' } for that call, so nothing breaks visibly and the render stays exact, but the client sends a stray, already-consumed submit after every such refresh. The example page (seeded without the resume message, resumed under the replayed prefix) does not. | live: WA MR1 (reliability-cf-wa-do-drive, 2026-10-10): adding a no-resume-rejected check (expectNoResumeRejected, e2e/src/helpers/refresh-char-exact-scenario.ts, commit f4dd1ac1ae on its branch) to the e2e C-1 refresh mid hitl-continuation cells (e2e/src/__tests__/refresh-c1.integ.test.ts) turned 5 cells red, on js-memory, js-redis, js-postgres, cfw-memory and cfw-d1 (ai 6.0.49): client B's re-submit of tc-ask-1 was answered data-resume-rejected { reason: 'unknown-tool-call', toolCallId: 'tc-ask-1' }. Not run with the check on cf-do or DBOS. The check stays off those cells until this is fixed; ai-sdk/src/react/send-automatically-when.ts:68 and :117-122 (origin/main b6b3eeb141) (the Helix sendAutomaticallyWhen predicate's dedup set starts empty per instance and only learns ids it sends itself) with ai-sdk/src/react/use-helix-chat.ts:381-389 (origin/main b6b3eeb141) (useHelixChat installs the predicate without seeding that set from its initial messages) and :363-373 (origin/main b6b3eeb141) (its replay filter only drops calls seen inside a replayed prefix). ai evaluates sendAutomaticallyWhen after a resume-stream on both 6.0.49 and 6.0.281, so a whole-snapshot seed's already-answered client-tool part (output-available) is POSTed once more, and the server answers data-resume-rejected unknown-tool-call. The C-1 seed is immune: snapshotResumeOptions drops the in-flight message and its replay re-emits the call inside the replay markers, which the filter drops (traced by WA, 2026-10-16, read-only at origin/main b6b3eeb141); related: the e2e refresh-c1 'refresh mid hitl-continuation' cells still run without the check (e2e/src/helpers/refresh-char-exact-scenario.ts:1328-1394 (origin/main b6b3eeb141): no expectNoResumeRejected there, unlike :1307 and :1424), so the e2e red is masked, not gone. The e2e helper is faithful to the product: headless-chat.ts:441 (origin/main b6b3eeb141) uses createHelixSendAutomaticallyWhen and :495-503 (origin/main b6b3eeb141) copies useHelixChat's replay filter verbatim; related: FU-AISDK-SEED-RESUBMITS-CONSUMED-TOOL-OUTPUT (filed in WA MR1, c668173614; owner WC, unowned). CP-53 (the server half of the same refresh-mid-continuation window, fixed by WA MR1) is a different mechanism; severity S3: the server refuses the stray submit cleanly and the render is unaffected, but every seeded refresh after an answered client tool makes a needless, already-consumed submit, and on a runtime whose late-submit answer is not a clean unknown_tool_call the stray submit is untested; conformance: submit.seeded-client-no-resubmit (owner batch 19, main f12eef982d; client tier, the real useHelixChat): client A answers a client tool, the continuation's model call is held (the answer consumed), the page reloads (client B seeded from the snapshot); after the release B POSTs to the chat route (.no-resubmit) and reads a data-resume-rejected frame (.no-resume-rejected), on both cf-do and js-chat-host; conformance: narrowed (owner, 2026-10-16). The client harness mounted the pre-S5 example ChatClient (seeded from the whole snapshot, continuing the last assistant message, no replay prefix) until the owner MR that made it mount the S5 page (use-snapshot-chat.ts + snapshot-load.ts). Re-run at main 124149eacb, ×3 per host, cf-do and js-chat-host: with the old harness submit.seeded-client-no-resubmit fails .no-resubmit and .no-resume-rejected 3 of 3; with the S5 page it passes every id 3 of 3 (the only verdict change across the whole client tier). So the red was the whole-snapshot seed's, not the example page's: submit.seeded-client-no-resubmit is now a regression guard for the page (not in this record's ledger or provenBy), and submit.whole-seed-client-no-resubmit reloads with the whole-snapshot profile (reload seed 'whole-snapshot', snapshotResumeOptions(snapshot, { replayPrefix: false })). At main 8d53593a3a plus the owner MR (with its two harness fixes: the cell wrapper forwards reload's options, and the cf-do worker's chat GET passes the request, so X-Helix-Replay-Prefix reaches the handler), ×3 per host, cf-do and js-chat-host (host load 46 to 49 on 14 cores, verdicts identical across runs): submit.seeded-client-no-resubmit passes every id, and submit.whole-seed-client-no-resubmit fails .no-resubmit and .no-resume-rejected: the whole-snapshot client POSTs the consumed output again and reads data-resume-rejected. An earlier run of the whole-seed cell that passed is void: the cell wrapper dropped the seed option, so it reloaded with the page's seed; ai-sdk/src/react/send-automatically-when.ts:74-112 and :140 (fix evidence, W1's !383: isClientSendableToolAnswer is the one rule the predicate and its new history seed share, and createHelixSendAutomaticallyWhen({ history }) pre-marks every answer the seed already shows as sent) with ai-sdk/src/react/use-helix-chat.ts:366-376 (useHelixChat reads its own seed's client-tool answers when it creates the Chat, so they are history and never auto-sent; a seeded pending call answered after mount still is); conformance: fixed by W1's !383 (W1's lock window, 2026-10-17, 3 runs per host, cf-do and js-chat-host, filter submit.(whole-seed|seeded)-client-no-resubmit): at !381's head e17288c938, submit.whole-seed-client-no-resubmit is expected-fail on .no-resubmit and .no-resume-rejected 3 of 3 on both hosts; with !383's 3 commits cherry-picked onto it, the same cell passes every id 3 of 3 on both hosts. The page's regression guard submit.seeded-client-no-resubmit passes 3 of 3 in both | source, live-repro, peer-reported | verified | fixed (2 cells) |
| CP-108 | S2 | A stop pressed while a CF Workflows session is PAUSED on client tools is erased by the next continuation, so the model runs after the user pressed stop. interrupt() of a paused session only sets the durable flag (the session stays paused), and the next non-recovery resume entry clears that flag in its run-start step, before the loop's first interrupt check. The executor's own batch-gate docstring says a pending interrupt makes the resume only honour the stop, mirroring runtime-js, which does honour it. DBOS honours it for a complete batch but not for a partial one: the resume re-suspends instead of ending interrupted. | runtime-cloudflare/src/steps.ts:1186-1192 (pre-fix, origin/main 16de0b648c) (the run-start step clears the interrupt flag for entry.kind === 'resume' && !entry.recovery) against runtime-cloudflare/src/workflow.ts:2339-2341 (as before WA MR2, origin/main cae966c0a7) (the loop's first check-interrupt step, which then finds no flag); runtime-cloudflare/src/executor.ts:790-797 (as before WA MR2, origin/main cae966c0a7) (assertClientToolBatchComplete: a pending durable interrupt bypasses the gate, the resume only honours the stop, and mirrors runtime-js / Temporal); contrast: runtime-js/src/js-agent-executor.ts:887-889 (as before WA MR2, origin/main cae966c0a7) (a pending durable interrupt: the resume only honours the stop, interrupted at its first boundary); the conformance cell passes on js-* and temporal-*; conformance: stop.paused-flag-then-continued (owner batch 19, main f12eef982d): the durable interrupt flag is set while paused; full leg (then the answer is submitted, then resume continue) and partial leg (c1 answered, flag set, resume continue past the R44 gate). cfw-workflows-d1 fails both legs: the model is called after the stop and the session completes (full), and the partial resume re-suspends instead of ending interrupted. dbos-postgres fails the partial leg only (it never settles: re-suspended). js-* and temporal-* pass; related: reported by W1 (W1 MR2b): MR2b's owed-continuation (__submit__) entry stops clearing the flag; an explicit resume on a partial batch still clears it (FU-CFW-EXPLICIT-RESUME-CLEARS-PAUSED-INTERRUPT, owner WB). CP-109 is the separate handle-stop no-op (the stop never reaches the flag on runtime-js and Temporal); CP-18 is the DO / agent-server paused stop (a 400 / 504), a different mechanism; CP-19 is the inverse (a leftover flag stopping the next run). The DBOS partial-batch half is an FU filed by W1 (owner: the parked DBOS bucket); runtime-cloudflare/src/executor.ts:3365-3383 (origin/main 487090d3fb) (fix evidence, the CF Workflows half, WA MR2 (!355) R2b, Ruling 18 / 57: a continue or with_message over a live run that has a stop record is decided by the stop first, before the owed-continuation adoption and the R44 partial-batch refusal; the run is settled through the stop path, settleRecordedStop, and a continue answers the stopped run's own outcome with no new instance and no model call); conformance: stop.paused-flag-then-continued passes on cfw-workflows-d1 (both legs, the full leg's .submit-refused-stopped included), at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12), so its cfw ledger entry is removed; runtime-dbos/src/workflows/shared.ts:2494-2498 (fix evidence, the DBOS half, WA MR2 (!355) R2c: a suspended run woken for its continuation reads its own stop record after the wake and before the answers' chunks, and a recorded stop ends the suspension by the stop, endSuspensionByStop, so the continuation never runs the model; the record is run-scoped and never cleared, runtime-dbos/src/steps/stop-request.ts:14-23) with runtime-dbos/src/lifecycle/resume.ts:422-430 (settleStoppedSuspension: a recorded stop never leaves a paused run refusing its own continuation); conformance: fixed. stop.paused-flag-then-continued passes on dbos-postgres at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos), and on cfw-workflows-d1 (the CF Workflows half, above) | source, live-repro, peer-reported | verified | fixed (2 cells) |
| CP-109 | S2 | stop() on the handle of a run that has already suspended (paused on client tools) reports success but records nothing on runtime-js and Temporal. runtime-js's interrupt callback only aborts the finished run's in-memory controller and returns; Temporal signals a workflow that already completed at its suspension and treats the resulting not-found as nothing to interrupt. No durable flag is set and the session stays paused, so the stop is lost: the next continuation runs as if it had never been pressed. CF Workflows and DBOS record it (both set the durable flag; DBOS also cancels the parked workflow and marks the session interrupted). | runtime-js/src/execution/handle-factory.ts:204-212 (as before WA MR2, origin/main cae966c0a7) (handle.interrupt awaits the executor's interruptCallback) with runtime-js/src/js-agent-executor.ts:3296-3312 (as before WA MR2, origin/main cae966c0a7) (interruptCallback: sets in-memory interrupt state, aborts the run's AbortController, and returns at once when execution is already done; no stateStore.setInterruptFlag); runtime-temporal/src/executor.ts:1215-1240 (as before WA MR2, origin/main cae966c0a7) (the execute() handle, createActiveHandle at :1158: interrupt signals the workflow; a WorkflowNotFoundError / 'Completed workflow' answer, which is what a run that suspended gives, is logged as temporal.interrupt.workflow_already_completed and returns without writing anything); contrast: runtime-cloudflare/src/executor.ts:4373-4394 (as before WA MR2, origin/main cae966c0a7) (interruptWorkflow: setInterruptFlag, then an interrupt event) and runtime-dbos/src/lifecycle/interrupt.ts:79-127 (as before WA MR2, origin/main cae966c0a7) (setInterruptFlag, a control message, cancelWorkflow, then the session force-updated to interrupted); inferred: (unverified) a chat or app stop button that keeps the run handle and calls stop() after the stream ended paused loses the stop on these runtimes; the stop is honoured only through the durable flag (CP-108's contract, green on js-* and temporal-*); related: CP-108 (the durable-flag paused stop, erased on CF Workflows by the next resume; distinct: here the stop never reaches the flag at all). Found by the owner while authoring batch 19 (the first form of the handle cell failed on all eight lanes; the per-runtime mechanisms above split it into this record and CP-108); conformance: stop.paused-handle-stop-recorded (owner batch 19, main f12eef982d; a handle stop() of a run that suspended on a client call, judged right after the stop): .stop-recorded fails on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres (the stop reported ok, the session stayed paused, no flag); cfw-workflows-d1 and dbos-postgres pass. Fix routing (orchestrator, 2026-10-11): the runtime-js half to W1 MR3 (the handle-tells-the-truth contracts), the Temporal half to the parked Temporal bucket; runtime-js/src/js-agent-executor.ts:1332-1387 (fix evidence, origin/main 487090d3fb) (handleStop, the runtime-js half, WA MR2 (!355) Task 9, D3: a handle's interrupt() / abort() stop the handle's OWN run through core stopRun; a run not executing, suspended or stranded, is ended at once, so a stop on a run that already suspended is recorded and honoured); conformance: stop.paused-handle-stop-recorded .stop-recorded passes on js-memory, js-redis and js-postgres, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12), so they leave the ledger; runtime-temporal/src/executor.ts:1055-1059 (fix evidence, the Temporal half, WA MR2 (!355): a handle's interrupt() stops its own run through stopRunById, :1033-1047, which calls core stopRun, core/src/run/stop-run.ts:114: the stop is recorded durably first, and a run with nothing executing it, as a run suspended on client tools is, gets its stop terminal committed at once, so the session no longer stays paused with nothing recorded); conformance: fixed. stop.paused-handle-stop-recorded passes on temporal-memory, temporal-redis and temporal-postgres at WA MR2 head 7d4aef1be8 (owner window 2026-10-13, every executor lane run in full: js 276 cells as ledgered, temporal x3, dbos), and on js-memory, js-redis and js-postgres (the runtime-js half, above) | source, live-repro | verified | fixed (6 cells) |
| CP-110 | S2 | CF Workflows: two client-tool submits answering different calls of one batch that land concurrently (two tabs) fail one of them with an untyped error, and its answer is lost. persistDurableSubmit loads the session, sets one pending entry and saves the whole row pinned to the version it read; the later writer's saveState hits StaleStateError, which persistDurableSubmit rethrows (it re-reads only on RunSupersededError). StaleStateError is a plain Error with no code. runtime-js, Temporal and DBOS re-read and re-apply on StaleStateError. | runtime-cloudflare/src/executor.ts:2960-3061 (pre-fix, origin/main f12eef982d) (persistDurableSubmit: loadState, the entry updated, saveState fenced on the entry's run; only RunSupersededError is caught and re-read, at :3051-3061); store-cloudflare/src/d1-state.ts:3452-3510 (origin/main f12eef982d) (saveState is version-pinned on the caller's loaded state.version; a mismatch is StaleStateError at :3495) and core/src/errors/agent-errors.ts:54-64 (origin/main f12eef982d) (StaleStateError extends plain Error: no code, no retryable); contrast: runtime-temporal/src/executor.ts:2248 (origin/main f12eef982d) and runtime-js/src/js-agent-executor.ts (applySubmitToOwner retries on StaleStateError); the conformance cell passes on js-*, temporal-* and dbos-postgres; conformance: submit.concurrent-batch (owner batch 19, main f12eef982d; the hold-submit-write control holds c1's submit write after its read, c2's submit lands, c1 is released): on cfw-workflows-d1 c1's submit throws (.c1-accepted) and its answer is lost (.no-answer-lost); every other executor lane passes; related: narrowed from W1's candidate FU-SUBMIT-CONCURRENT-BATCH-LOST-UPDATE (ruled out as a silent lost update on every store: each saveState is version-pinned). W1 MR2b adds the re-read-and-apply (its executor-submit-stale-state test holds the same point); typing StaleStateError with an ErrorCode across stores is a separate follow-up (owner WF); runtime-cloudflare/src/executor.ts:4669-4671 (origin/main 487090d3fb) and :4758-4770 (fix evidence, W1 MR2b !344: persistDurableSubmit, at :4457 (origin/main e96f88bc98), now loops on one budget of 5 attempts, SUBMIT_WRITE_ATTEMPTS, shared with the superseded re-read at :4775-4780 (origin/main 487090d3fb). Each attempt re-reads the owner with loadState and re-applies the submit to the fresh entry; a StaleStateError from the version-pinned saveState, meaning a concurrent submit of another call of the batch wrote first, continues the loop instead of being rethrown, so both answers land. Past the budget the last error is rethrown), proven by runtime-cloudflare/src/__tests__/executor-submit-stale-state.test.ts:112 (both answers land and the batch is complete) and :164 (a StaleStateError that persists past the bound is rethrown, never a loop); conformance: fixed by W1 MR2b: submit.concurrent-batch|cfw-workflows-d1|executor passes every id (owner slot on W1 MR2b's post-fix head 61ca81d8f8 (branch reliability-cf-w1-mr2b-t8b: MR2b + WA MR1 8e4921809a + WE-2 9e9c382563; dist built by W1, 39/39 turbo tasks), 2026-10-11; the ledgered .c1-accepted and .no-answer-lost reported UNEXPECTED PASS): c1's stale write is re-read and applied, and both answers survive | source, live-repro | verified | fixed (1 cells) |
| CP-111 | S1 | A chat message sent while the run is paused on an UNANSWERED client call (chat path 4: the call is abandoned and a new run answers the message) never renders its reply in the product client. On the CF Durable Object the send is refused with a typed 409 state_already_running from the DO's /start: the per-call abandon submits wake a continuation that the send then collides with. On the JS chat host the send never settles and no reply renders. | conformance: chat.send-over-pending-client-calls (owner batch 19, main f12eef982d; client tier, the real useHelixChat): the first turn ends paused on an unanswered client call, then a new message is sent. cf-do: the client ends in error with {code: 'state_already_running'} ('Agent session … is already running (status: 409 from DO /start: Agent already running, ALREADY_RUNNING)'), and no reply is rendered (.no-client-error, .reply-rendered). js-chat-host: the send does not settle within 30 s and no reply is rendered (.send-settles, .reply-rendered); its mechanism is not yet traced; inferred: (unverified) the cf-do half: handleChatStream's path 4 posts an abandon submit per pending call before the send's entry; on the DO a submit wakes the continuation (the CP-53 path), so the send's /start finds a live run and is refused; related: W1 MR2b replaces the per-call path 4 (abandon submits, then a run) with one D11 entry, which removes this collision and hang by construction; what remains on MR2b's head is the foreign tool-output ai-reader throw W1's 5b-r23 probe found, fixed by WE-2's standalone transformer MR. CP-53 (the DO submit continuation), DI-47 (the untyped DO submit refusal); ai-sdk/src/handler/handle-chat-stream.ts:662-675 and :1563-1649 (fix evidence, W1 MR2b !344, D11: path 4 sends a new message over pending client-tool calls as ONE entry when the executor declares supportsAbandonOnResume and the session is suspended on client-tool calls alone, so no per-call abandon submit wakes a continuation that the send then collides with. handleAbandonAndSendPath makes one resume with mode with_message and abandonPending: true at :1590-1595, answers a refusal typed through typedEntryErrorResponse at :1601-1608, and streams from the entry run's own startSequence at :1630-1648), with core planWithMessageEntry at core/src/client-tool/with-message-entry.ts:47-70 (as before WA MR2, origin/main cae966c0a7) as the one decision; runtime-js/src/js-agent-executor.ts:454 and runtime-js/src/with-message-entry-rows.ts:259-266 and :293-300 (as before WA MR2, origin/main cae966c0a7) (fix evidence, W1 MR2b !344, the js-chat-host half: JSAgentExecutor declares supportsAbandonOnResume, and the entry answers every unanswered awaited call client_tool_abandoned and commits those results, then the marked message, in its own run-start commit) with runtime-cloudflare/src/do/do-clients.ts:1000 (pre-fix, origin/main 7a390aaf8d) and runtime-cloudflare/src/do/drive/request-schemas.ts:74-80 (pre-fix, origin/main 7a390aaf8d) (fix evidence, W1 MR2b !344, the cf-do half: DOFrontendExecutor declares supportsAbandonOnResume, and the DO's /resume accepts with_message plus abandonPending, which its embedded runtime-js executor runs as that same one entry); ai-sdk/src/transformer/stream-transformer.ts:220-252 (fix evidence, the other half: WE-2's DI-48 fix, merged to main as 16de0b648c. An output for a call an earlier message holds is never forwarded, since ai would throw No tool invocation found and fail the whole response; the response carries one data-tool-output-elsewhere re-anchor trigger per call instead); conformance: fixed by W1 MR2b together with WE-2's DI-48 fix (9e9c382563, in the same tree; path 4 needs both): chat.send-over-pending-client-calls passes every id on cf-do and js-chat-host, client tier (owner slot on W1 MR2b's post-fix head 61ca81d8f8 (branch reliability-cf-w1-mr2b-t8b: MR2b + WA MR1 8e4921809a + WE-2 9e9c382563; dist built by W1, 39/39 turbo tasks), 2026-10-11; every ledgered id reported UNEXPECTED PASS) | live-repro | partly-inferred | fixed (2 cells) |
| CP-112 | S2 | Cloudflare Durable Object: a stop pressed while the session is PAUSED on client tools is erased by the submit that answers the whole batch, so the model runs after the user pressed stop. EndpointContext.stop('interrupt') on a paused session only sets the durable interrupt flag (nothing ends at once). The submit that answers the last pending call starts a continuation, and its run-start commit clears the flag, because DOAB's R39 clear keeps the flag only while the batch is still partial (honoursStop). The continuation then runs as if the stop had never been pressed. The DO drive pipeline's upgrade guide says the opposite: that the next continuation, a submit or a resume, honours the stop. CP-108 is the same shape on CF Workflows, through a different mechanism (its run-start step's clear). | runtime-cloudflare/src/do/durable-object-agent-base.ts:615-625 (origin/main fb7c1e12f5; R39: the winning run-start commit clears the interrupt flag unless the session is paused over a PARTIAL batch, via isClientToolBatchPartial; a submit answering the whole batch makes it no longer partial); core/src/client-tool/pending-batch.ts:46-52 (isClientToolBatchPartial: true only while some awaited call is unanswered); runtime-cloudflare/src/do/durable-object-agent-base.ts:2037-2042 (origin/main fb7c1e12f5; EndpointContext.stop: interrupt sets the flag through interruptRun, on a paused session too) and :2359-2362 (origin/main fb7c1e12f5; interruptRun); contrast: docs/upgrade-guides/do-drive-pipeline.md:91-94 (origin/main fb7c1e12f5; the guide says a custom endpoint's stop on a paused run is honoured by the next continuation, a submit or a resume, which ends that run interrupted at its first boundary); conformance: stop.paused-whole-batch-submit|cf-do|host (owner batch 21): a one-call paused batch, EndpointContext.stop('interrupt') through a production custom endpoint, then a submit-direct answering the batch. Observed on main fb7c1e12f5 and on main + W1 MR2b a09ba78b9d, the same verdict: the session ends completed and the model is called once after the stop (.ends-interrupted, .no-model-call-after-stop); related: found by WA MR2's survey (design note 2026-10-11-do-stop-as-one-signal-design.md §1.2 item 1, from reading the code; this cell is its red). Fix direction: WA MR2 §3.2, a stop on a paused run ends it at the stop with K5 interrupted, so no later submit can erase it, and R39's clear is deleted; severity S2: a stop the user pressed is silently lost and the model runs, as CP-108; runtime-cloudflare/src/do/drive/stop.ts:12-18 (fix evidence: a stop on a paused run ends it at the stop with its K5 interrupted, so no later submit can continue it; WA MR2 (!355), design note 2026-10-11 §3.2, user ruling D-1: every DO stop path, /interrupt, /abort, EndpointContext.stop and interrupt_previous, goes through handleStop at :112-183, core stopRun: a running run is recorded and signalled and answered 202 stopping at once; a stranded or suspended run is recorded and ended by the stop's K5 now and answered 200 with its terminal; no live run is 409 state_not_stoppable); conformance: fixed. stop.paused-whole-batch-submit passes on cf-do: the run ends interrupted and the model is not called after the stop, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12) | source, peer-reported | verified | fixed (1 cells) |
| CP-113 | S2 | Cloudflare Durable Object: /interrupt and EndpointContext.stop('interrupt') on a running drive do not answer until the run has ended. The durable flag is written first, but then the handler awaits the execution state's interrupt, which awaits the run itself. A run inside a tool that ignores its abort signal (or a slow step) keeps the stop request open for as long as it runs, with no bound, and the answer then says { status: 'interrupted' } whatever the run did. The user's D-1 ruling (2026-10-11) is: record the stop, signal the run, answer at once. CP-14 is the same wait on /abort. | runtime-cloudflare/src/do/durable-object-agent-base.ts:2382-2414 (origin/main fb7c1e12f5; handleInterrupt awaits interruptRun, then answers 200 status interrupted) and :2359-2362 (origin/main fb7c1e12f5; interruptRun writes the flag, then awaits this._executionState.interrupt); runtime-cloudflare/src/do/execution-state.ts:107-132 (origin/main fb7c1e12f5; interrupt awaits the handle's interrupt, else aborts the controller and awaits this._execution, the run itself); runtime-cloudflare/src/do/durable-object-agent-base.ts:2037-2042 (origin/main fb7c1e12f5; EndpointContext.stop goes through the same interruptRun); conformance: stop.running-host-stop-returns|cf-do|host (owner batch 21): while the run is inside a tool that ignores its abort, POST /do/:id/interrupt gives no answer within 2 s (.interrupt-answers-at-once); once the tool is released the run ends interrupted, which passes. stop.hung-tool-deadline|cf-do|host: the same, .stop-answered. Observed on main fb7c1e12f5 and on main + W1 MR2b a09ba78b9d, the same verdict; related: WA MR2's survey, design note 2026-10-11-do-stop-as-one-signal-design.md §1.1 (the /interrupt row) and §1.2 item 2 (the 200 says interrupted whatever the run did). Fix direction: §3.2, every stop path records, signals and answers 202 stopping at once; severity S2: the stop control blocks its caller for as long as the run runs; the run itself does stop at its next boundary; runtime-cloudflare/src/do/drive/stop.ts:12-18 (fix evidence, WA MR2 (!355), design note 2026-10-11 §3.2, user ruling D-1: every DO stop path, /interrupt, /abort, EndpointContext.stop and interrupt_previous, goes through handleStop at :112-183, core stopRun: a running run is recorded and signalled and answered 202 stopping at once; a stranded or suspended run is recorded and ended by the stop's K5 now and answered 200 with its terminal; no live run is 409 state_not_stoppable); conformance: fixed. stop.running-host-stop-returns .interrupt-answers-at-once and stop.hung-tool-deadline .stop-answered pass on cf-do: the stop answers 202 stopping while the tool still holds the run, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12) | source, peer-reported | verified | fixed (2 cells) |
| CP-114 | S1 | Cloudflare Durable Object: a stopped run held by a tool that ignores its abort signal never ends, so the session is wedged. Nothing bounds a stop: no deadline ends the run, and the drive stays registered as executing for as long as the tool runs, so the run stays the session's live run. The session stays active, and a new chat turn sent meanwhile never settles (its exact path through the chat route is not traced). The interrupt-poll alarm subscriber cannot help: its deadline is always now + 30 s, so it never fires. | runtime-cloudflare/src/do/durable-object-agent-base.ts:584-591 (origin/main fb7c1e12f5; while the drive executing the current live run is registered, every run-start commit for the session is refused run_active; nothing ends that registration but the run returning); runtime-cloudflare/src/alarm-scheduler.ts:93-102 (origin/main fb7c1e12f5; the scheduler's own caveat: a subscriber whose deadline is relative to now never fires under real alarms) and runtime-cloudflare/src/do/durable-object-agent-base.ts:1178-1192 (origin/main fb7c1e12f5; interrupt-poll returns now + SSE_HEARTBEAT_INTERVAL_MS while executing); conformance: stop.hung-tool-deadline|cf-do|host (owner batch 21): a tool that ignores its abort, held at a gate; after the interrupt the session is still active 30 s + 15 s later (.deadline-ends-run, and .ends-interrupted on it), and a new turn sent while the tool is still held does not settle within 30 s (.new-send-settles, and .new-send-completes on it). Observed on main fb7c1e12f5 and on main + W1 MR2b a09ba78b9d, the same verdict. .late-write-refused passes on main: once released, the stopped run discards the tool's result; related: WA MR2's survey, design note 2026-10-11-do-stop-as-one-signal-design.md §3.4 (no deadline today; a run held by a hung tool stays active) and §1.2 item 3 (interrupt-poll never fires). Fix direction: §3.4, a durable deadlineAt per stop (default 30 s, user ruling Q1); at the deadline the stop's terminal is committed and the stuck drive detached, its later writes refused; severity S1: one tool that ignores its abort wedges the session; nothing short of the tool returning, or an eviction, frees it; runtime-cloudflare/src/do/drive/stop.ts:23-29 (fix evidence, WA MR2 (!355) §3.4, user ruling Q1: every stop records deadlineAt = requestedAt + stopDeadlineMs, default 30 s, validated by resolveStopDeadlineMs at durable-object-agent-base.ts:692 (pre-fix, origin/main 487090d3fb)) with :386 (endRunAtStopDeadline: a run still live past its deadline is ended with the stop's terminal and its stuck drive is detached) and durable-object-agent-base.ts:1741-1753 (pre-fix, origin/main 487090d3fb) (the stop-deadline alarm subscriber fires at the ABSOLUTE deadlineAt, so it genuinely fires and survives an eviction); conformance: fixed. stop.hung-tool-deadline passes on cf-do: past the deadline the run ends interrupted while the tool still ignores its abort, and a new turn sent meanwhile settles and completes, at WA MR2 pre-run head b43b8e1a31, whose product code equals the final head outside runtime-dbos and runtime-temporal (WA matrix, 2026-10-12) | source, peer-reported | verified | fixed (1 cells) |
| CP-115 | S1 | runtime-js: a direct execute() over a session whose run is suspended on its children (suspended_awaiting_children) or on a partly committed step (suspended_step_partial) is accepted and silently supersedes that run, so the children's results are lost and no error is raised. runtime-js refuses a live suspended run only when the session reads running / waiting_tool, or when it is paused with pending client-tool calls; a session paused on children has neither, and its suspended run released its lease, so the store's owner guard counts it ownerless and the new run-start commit supersedes it. (Narrowed by WE-2's !377: the CF Durable Object, which runs this executor, now refuses every request-driven /start over a suspended run itself.) | runtime-js/src/js-agent-executor.ts:3077-3094 (origin/main 6ab56e113a) (the refusal of a live suspended run, AgentAlreadyRunningError 'session is suspended awaiting a client-tool result or children', sits inside the branch for a session that reads running or waiting_tool) and :3102-3116 (origin/main 6ab56e113a) (a paused session is refused only when pendingClientToolCalls is non-empty); a session paused on children reaches the run-start commit, whose owner guard treats the suspended run's released lease as ownerless and supersedes it; live: peer-reported (WE-2, 2026-10-13, from the !352 code review). A probe on built runtime-js and store-memory: after turn 1 completed, a run set suspended_awaiting_children with the session paused; execute(agent, 'hello?') was accepted and completed, and the suspended run became superseded. On the CF DO a real-workerd test (do-run-guards.integ on !352, with !352's DO refusal mutated off) answered a /start over a seeded suspended_awaiting_children run 200 started; conformance: the DO half, ledgered. do.start-over-awaiting-children-refused and do.start-over-step-partial-refused (owner batch 34, host tier, cf-do): turn 1 completes through the chat route, then the session's DO seeds a live suspended run over it through the store's own run-start commit (the session saved paused; the DO never reaches either status by itself, so a seed is the only way in, setup only per ruling R9), then a native POST /start with a new message and no chat marker. On main 487090d3fb plus owner batches 33 and 34, 3 of 3 on both cells (host load 5 to 10 on 14 cores): the /start answers 200 status started, the seeded run ends superseded, a third run completes, and the session holds the new message and its reply. The positive control .marked-send-refused passes 3 of 3: a chat send over the same seed is refused 409 state_already_running (!352's guard), so the DO reads the seed as a live suspended run and only the unmarked /start gets through; runtime-cloudflare/src/do/drive/run-drive.ts:415-430 (pre-fix, origin/main 7e49f42fd1) (fix evidence, the DO half, WE-2's !377, D-2 MR 1.5 item 2, ruling O-70) (checkConcurrency: with nothing executing, a request-driven /start over a live run that is not running, i.e. suspended on its children, a partial step or pending client-tool results, is lost with the run's status, answered 409 state_already_running, marked by the chat route or not); conformance: the DO half fixed by !377. do.start-over-awaiting-children-refused and do.start-over-step-partial-refused (host tier, cf-do) pass 3 of 3 on !377's head (WE-2's run): the native /start over the seeded suspended run is refused 409 state_already_running, the seeded run stays current and suspended, nothing is written; their cf-do ledger rows are dropped. The runtime-js half (direct execute()) stays open, its cell pending; related: !352 (WE-2) closes the chat route only (the handler refuses before execute(), and the DO refuses a /start the chat route marks); the direct execute() API stays open. Ruling O-70(b) (orchestrator, 2026-10-13): file it; the backlog decides the slot. Temporal, DBOS and CF Workflows are not probed; severity S1: a new message silently ends a run that is waiting on its children, whose results are then never delivered to it, with no error to the caller; runtime-js/src/js-agent-executor.ts:4462-4474 (origin/main fb9150cc16) (still true at main: a paused session is refused only when it holds pending client-tool calls, so a session paused on its children is accepted, and the run-start commit supersedes the suspended run; the suspended-run refusal at :4436-4453 (origin/main fb9150cc16) covers running / waiting_tool only). verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24). The CF DO half is fixed by 7e49f42fd1 (runtime-cloudflare/src/do/drive/run-drive.ts:457-462 (origin/main fb9150cc16)); runtime-js only, non-CF (parked); live: peer-reported (WE-2, 2026-10-26, from CP-106 MR B's CP-157 pin, runtime-js/src/__tests__/new-turn-stranded-remote-child.test.ts on its branch, memory store with a fake RemoteAgentTransport): a parent parked on a REMOTE child (a committed remote call, its live ref, the run suspended_awaiting_children, the session paused, remotePendingCalls only and no pendingClientToolCalls) takes the same path: execute() resolved instead of rejecting. The parked call is then never paired (the new turn's C5 heal and RM-81 settle run only after a completed / failed / interrupted prior run, js-agent-executor.ts:7704-7707 (origin/main 318045e4bd)), and the remote ref stays running with its child never interrupted. On the CF DO the same entry is refused (run-drive.ts:484-490 (origin/main 318045e4bd), checkConcurrency 'suspended_run'), so it is runtime-js direct only. CP-106 MR B keeps suspended runs out of its new-turn takeover, so this case is unchanged there (verified by the owner at origin/main 318045e4bd: js-agent-executor.ts:4829-4847 (origin/main 318045e4bd) (the suspended-run refusal, inside the running / waiting_tool branch only) and :4855-4868 (origin/main 318045e4bd) (a paused session refused only with pending client-tool calls)); conformance: the runtime-js half: subagent.new-turn-over-children-park-refused (owner, executor tier, the lanes with 'subagent-client-tool'): the parent parks on its sub-agent child suspended on a client call, then execute() with TURN-2. At main f184eb6b1b (owner slot s170, 2026-10-27, 3 of 3 identical, .parent-parked green) it fails .new-turn-refused, .park-unchanged, .no-input-appended and .child-result-delivered on js-memory, js-redis and js-postgres: the entry is admitted and supersedes the parked run, so the child's result is never delivered; conformance: the CF DO half on a real park: subagent.new-turn-over-remote-park-refused-host (owner, host tier, cf-do: a CP-157 remote park, no seed) passes every id 3 of 3 at main f184eb6b1b (owner slot s170): the chat send and the DO's native /start are both refused 409 state_already_running and the park is unchanged; conformance: the probe (owner slot s170, one force-claimed run of the executor cell on the other lanes): on Temporal and CF Workflows the new turn is refused and the park kept; Temporal's parent holds no ref for the awaited call (RM-110, its own cell); DBOS has no committed children-park (its parent's workflow blocks live on the child's); see the declared gaps; related: fixer: WA (runtime-js interim 409), after !555 (orchestrator, 2026-10-27); conformance: Temporal is red too in the settled order. runtime-temporal/src/executor.ts:505-547 (origin/main 6a2dc68aaf) (execute() refuses a paused session only for its own pending client calls, and assertNoLiveOwner only a running run): a new turn over a parked parent is refused only by the run-start commit's ownerless proof while the parent's workflow is still open, and supersedes the park once it closed. !561's CI (pipeline 2997143017) admitted it on temporal-postgres (.new-turn-refused, .park-unchanged, .no-input-appended and .child-result-delivered failed) while owner slot s179 (3 of 3) and CI's temporal-memory / temporal-redis refused: the cell is a declared gap there until the lane can tell the workflow closed (observe-owner-closed, FU-CONF-TEMPORAL-OWNER-CLOSED-OBSERVE). Fixer: WA, with runtime-js (orchestrator, 2026-10-27); runtime-js/src/js-agent-executor.ts:2485-2494 (fix evidence, WA's CP-115 member, e1cce433e7) (refuseNewTurnOverSuspendedRun: a new turn over a session whose current run is live and not running is refused AgentAlreadyRunningError with its status and suspendedRunRefusalHint, nothing written; a run with a recorded stop is not refused, Ruling 18) with :5271-5277 (same pin, e1cce433e7) (called for a running, waiting_tool or paused session, so a session paused on children no longer reaches the run-start commit); conformance: the runtime-js half is proven by subagent.new-turn-over-children-park-refused on js-memory, js-redis and js-postgres (.new-turn-refused, .park-unchanged, .no-input-appended, .child-result-delivered): UNEXPECTED PASS 3 of 3 on each at WA's 42cd736ce5 merged with the owner's cells (WA slot s1, 2026-10-28, peer-reported, logs verified by the owner), and plain PASS 3 of 3 on each with this record applied at WA's member head eb0a93f6cc (WA slot 1b, 2026-10-28, peer-reported, logs verified by the owner: the records unit catalogue / drift / docs 125/125; the cf-do cells below pass 1 of 1 in the same slot, no unhealthy teardown); the DO half by do.start-over-awaiting-children-refused and do.start-over-step-partial-refused on cf-do (fixed by !377). Temporal, CF Workflows and DBOS stay declared gaps (Temporal: FU-CONF-TEMPORAL-OWNER-CLOSED-OBSERVE; WA's Temporal unit covers the settled order) | source, peer-reported | partly-inferred | fixed (5 cells) |
| CP-116 | S1 | A stop that discards a step which already ran a sub-agent leaves that sub-agent unrecorded: an ephemeral sub-agent call runs its child during the step's tool phase and writes the parent's sub-session ref straight to the store when the child returns, and a remote sub-agent call writes its ref after the child's /start, neither inside the step commit. When the stop refuses or skips the step's commit, the parent keeps a ref whose parentToolCallId is in no persisted message, and the child ran (the sub-agent half of CP-106). | runtime-js/src/run-loop.ts:3557 (origin/main 34898cacaa) (the ephemeral SubSessionRef write after the child outcome, a direct store write in the tool phase) and core/src/orchestration/remote-sub-agent-dispatch.ts:404 (origin/main 34898cacaa) (the remote ref written after /start); conformance: subagent.spawn-discarded-on-stop (owner batch 29, executor tier, every lane): the parent's one step calls its gated tool and an ephemeral sub-agent whose child answers at once; with the gate held the cell waits (bounded, not judged) until the child has finished its turn, then stop(). On main e96f88bc98 plus batch 29, 3 of 3: js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres and dbos-postgres fail .refs-recorded and .unrecorded-child-stopped (the parent's history is ["user:go"], it keeps a ref to the child, and the child finished or was still running at the bound; DBOS's .parent-interrupted is CP-57's). cfw-workflows-d1 passes (it starts the sub-agent only after the tool phase). The wait was added after a train run on CI stopped DBOS before it had dispatched the child (it runs a step's tools and sub-agents in one Promise.all, runtime-dbos/src/workflows/shared.ts:3466), so no ref was written and both ids passed; without it the stop raced the child's dispatch on every runtime that runs the child beside the tool; conformance: subagent.remote-spawn-discarded-on-stop-host (owner batch 29, host tier, cf-do; the DO runs every sub-agent through its remote rewrite in a sibling DO): with the same wait for the child, it fails .refs-recorded and .unrecorded-child-stopped, 3 of 3 on main e96f88bc98 plus batch 29; related: CP-106 (the companion-spawn half, same class); WA spec 2026-10-14-cp106-post-commit-child-phase-design.md §9 C-2 (MR B gates on these cells); conformance: the dbos-postgres cell's red (subagent.spawn-discarded-on-stop .refs-recorded and .unrecorded-child-stopped) is unledgered since the CP-57 MR (owner, orchestrator ruling 2026-10-16): the cell is a declared gap there (it needs 'ordered-stop-settle', which the lane does not claim: its post-stop handle outcome is a race, FU-CONF-CP57-DBOS-STOP-SETTLE-ORDER); it is ledgered again from observed runs once the lane claims it; related: Fixer: W1 (orchestrator, 2026-10-24), after CP-213; real at main fb9150cc16 (its ledger rows are CI-enforced); conformance: with WE-2's #181 (!564), proven on #181 ALONE (owner slot s205, 2026-10-29, r6 b0fcf7c185, archived at orch-archive/batch-2026-10-29-1-11774a6828; landed in batch 3's squash f85872de1e) and on the batch-3 tree (r6 x !572 x !574 x !575, tree 9398be4d36), interleaved, 3 of 3 identical on both: subagent.spawn-discarded-on-stop passes every id on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres, and subagent.remote-spawn-discarded-on-stop-host passes every id on cf-do (batch 3's pipeline 3002323872 saw the same as UNEXPECTED PASS) — proves the fix; related: the fix is #181's stop semantics (a stopped step keeps only the writes its rows record, and the kept row's calls are paired, RM-29), which leave no unrecorded child behind a discarded step; the fixer of record was W1 (above), the cells moved with #181 | source, probe | verified | fixed (7 cells) |
| CP-117 | S2 | Same-step companion calls race: one step that spawns a companion and then sends it a message runs both tool calls concurrently, so whether companion__sendMessage sees the spawn's ref depends on store timing. On several runtimes the send fails 'No child agent found with name: helper' though the spawn was emitted first. Contract (Ruling O-72): same-step companion calls execute in emission order and a later call sees the refs earlier calls committed; a send to a name spawned LATER in the step still gets the typed not-found. | core/src/orchestration/step-iterator.ts:1761-1795 (origin/main 34898cacaa) (a step's tool calls, companion tools included, are awaited concurrently) with core/src/orchestration/companion-tool-dispatch.ts:713-735 (origin/main 34898cacaa) (handleSendMessage throws 'No child agent found with name' when the ref is not there yet); live: owner batch 29 cell runs (companion.spawn-then-send-same-step, executor tier, every lane, and -host on both hosts; withdrawn as a race, below): executor tier, every lane) and -host (both hosts): one step calls companion__spawnAgent then companion__sendMessage to the same name. On main 34898cacaa plus batch 29 it fails .send-accepted on js-memory, js-redis, temporal-postgres, temporal-redis, cf-do and js-chat-host, 3 of 3 each ({"error":"No child agent found with name: helper"}, helixToolFailed); it passes on js-postgres, temporal-memory, cfw-workflows-d1 and dbos-postgres; live: the same cells on main e96f88bc98 (!355) plus batch 29, 3 of 3: js-postgres now fails .send-accepted too (it passed 3 of 3 on 34898cacaa), the race the note below predicts; temporal-memory, cfw-workflows-d1 and dbos-postgres still pass; inferred: the verdict per lane is a race outcome (store latency decides whether the spawn ref lands before the send reads it): each lane was consistent 3 of 3 locally, but on CI temporal-postgres won the race (an unexpected pass, train pipeline of !357), so the cells were withdrawn before they could be ledgered. A deterministic cell holds the spawn's ref write until the in-step send has answered, forcing the losing order on a runtime that runs the two calls concurrently; conformance: companion.spawn-then-send-same-step (owner batch 31, executor tier, every lane but cfw) and -host (cf-do, js-chat-host), with the spawn's sub-session ref write HELD (hold-ref-write: until a refs read that started while it was held has answered, or 5 s) so the losing order is forced rather than left to store timing. On main e96f88bc98 plus batches 29 to 31, 3 of 3: .send-accepted fails ('No child agent found with name: helper') on js-memory, js-redis, js-postgres, temporal-postgres, temporal-redis, cf-do and js-chat-host. The hold opened at-bound on the js lanes (the send had read before the spawn reached its write) and after-read or at-bound on Temporal; either way a concurrent runtime cannot win. dbos-postgres and temporal-memory pass with the hold engaged (the cell refuses to judge a write that never reached the hold), so they run the send after the spawn. cfw is a declared gap (its spawn writes in the workflow body); conformance: withdrawn on Temporal as a race (orchestrator ruling 2026-10-16, no-race-ledgers rule). With the hold engaged, temporal-redis passed .send-accepted unexpectedly in two pipelines whose MRs touch no companion dispatch: !364's 2955540773 (job 17235637813) and !370's 2955508395 (job 17235365198), while temporal-postgres stayed red in the first and temporal-memory always passes. The hold opens after-read or at-bound on Temporal, and a send that still sees the spawn's ref there means the losing order is not forced on that runtime. The Temporal lanes no longer claim hold-ref-write, so the cell is a declared gap on all three until it anchors on an observed event (FU-CONF-CP117-TEMPORAL-ORDER-ANCHOR); conformance: not covered on Temporal. Whether Temporal runs same-step companion calls concurrently (the defect) or in emission order is unobserved deterministically; the parked Temporal fix (O-72) and its cell wait on FU-CONF-CP117-TEMPORAL-ORDER-ANCHOR; related: Ruling O-72 (orchestrator, 2026-10-14): a defect, runtime parity; fixed by MR A (JS and the DO: child-phase calls run in emission order after the step commit) and checked on CFW by MR B; Temporal and DBOS get FUs. WA spec §9 C-6; core/src/orchestration/child-phase.ts:176-239 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (runChildPhase: a step's companion calls run after its commit, one at a time in call order, so a same-step send starts only once the spawn's child started) with runtime-js/src/run-loop.ts:9587-9596 (fix: !523, WE-2; as on branch reliability-we2-cp106-child-phase-mr-a b8b37e035e) (runtime-js, which the CF DO runs): fixed on runtime-js and the CF DO; conformance: fixed on runtime-js and the CF DO by !523 (CP-106 MR A): in !523's MR pipeline 2987684089 at its head b06417e355 (jobs 17455241724 and 17455241720) companion.spawn-then-send-same-step was unexpected-pass on js-memory, js-redis and js-postgres, and companion.spawn-then-send-same-step-host on cf-do and js-chat-host, so both ledger rows are removed. CF Workflows, Temporal and DBOS still run a step's companion calls in the step, concurrently (MR B), where no lane can force the losing order (no hold-ref-write): open there, with no cell | source, probe | verified | open |
| CP-118 | S2 | CF Workflows: a stop that lands while the parent waits on a BLOCKING companion spawn leaves the spawn's tool_use unpaired in the persisted history, a regression of !355. The step that calls companion__spawnAgent commits before the wait, and the wait's outcome is recorded by steps.appendCommitted, which !355 guards with refuseIfStopped. So the recorded stop refuses the result row (CommitStoppedError, nothing written), and the run ends interrupted with the call unanswered, breaking the RM-29 pairing invariant until a later turn's trailing heal. Before !355 the child's result was appended and the pair held. | runtime-cloudflare/src/workflow.ts:4088-4101 (pre-fix, origin/main 2f2ce3fdab) (the blocking spawn records the child outcome as the spawn call's result row through steps.appendCommitted, after the wait) with runtime-cloudflare/src/steps.ts:2686-2708 (pre-fix, origin/main 2f2ce3fdab) (appendCommitted passes refuseIfStopped: true since !355, so a stop recorded for the run refuses the row atomically, CommitStoppedError, nothing written); conformance: companion.blocking-spawn-stopped-during-wait on cfw-workflows-d1 (owner batch 29, C-3). On main 34898cacaa plus batch 29 it failed only .child-ended-by-stop, 3 of 3: .spawn-call-paired passed, the history pairing the spawn call with the child's completed result. On main e96f88bc98 plus batch 29 it also fails .spawn-call-paired, 3 of 3: the history is the user row and the assistant row holding the spawn call, with no result. In 34898cacaa..e96f88bc98, db8d0763ba (!352) changes only ai-sdk and the DO, so the change is !355's; related: CP-106 C-3 (the target contract: a stop during the wait pairs the spawn call with a not-executed result whose reason is run_stopped; WA's spec 2026-10-14-cp106-post-commit-child-phase-design.md §9); RM-29 (the pairing invariant; the C5 trailing heal repairs only trailing calls, on the next turn). The fix belongs to WA (the CFW half of the child-phase work); conformance: companion.blocking-spawn-stopped-during-wait-paired (owner batch 31, executor tier): the same setup judged only by .spawn-call-paired; on main e96f88bc98 plus batches 29 to 31 it fails on cfw-workflows-d1, 3 of 3 (and on every other executor lane, CP-106's discarded-step shape); runtime-cloudflare/src/steps.ts:2694-2724 (pre-fix, origin/main 487090d3fb) (fix evidence, WA's CP-118 MR: appendCommitted takes recordsCompletedWork, and an append of completed work is run-fenced but not stop-guarded, Ruling O-73) with runtime-cloudflare/src/workflow.ts:4145-4164 (pre-fix, origin/main 487090d3fb) (the blocking spawn's result row is appended with RECORDS_COMPLETED_WORK); conformance: fixed on WA's CP-118 MR (head 0f56f263ab plus this record): companion.blocking-spawn-stopped-during-wait-paired on cfw-workflows-d1 is its provenBy cell; WA's lock window runs it ×3 with this flip applied (the record is judged by that window: a provenBy cell that fails is a verify failure). The C-3 shape cell's cfw entry loses .spawn-call-paired with this ledger; its .child-ended-by-stop stays CP-106's | source, live-repro | verified | fixed (1 cells) |
| CP-119 | S1 | An approved tool that finishes after its run was stopped loses its real result, and the model is later told it never ran. The approval continuation executes the call; a stop recorded for that run before the result is committed makes the result's append refused (refuseIfStopped) and swallowed, so the run ends interrupted with the call unanswered. The session's next turn then pairs it through the continuation heal as NOT executed: a tool with real side effects is reported to the model as not run. Ruling O-73: a stop refuses new work, never the record of completed work. | conformance: approval.approved-tool-result-survives-stop (owner batch 31, executor tier, every lane): the responder approves the call, the approved tool runs until its gate, a stop is recorded for the session's CURRENT run (requestStop: the approval continuation is a new run, so a stop through the pre-approval handle names a run that has ended and records nothing), and the gate is released 500 ms after the stop request. On main e96f88bc98 plus batches 29 to 31, 3 of 3: js-memory, js-redis, js-postgres, cfw-workflows-d1, temporal-memory, temporal-redis and temporal-postgres end interrupted (.stop-took-effect passes) with no result row for the call (.result-recorded fails), and after the next turn the call is paired not-executed (.next-turn-keeps-result fails). On dbos-postgres the recorded stop is not honoured at all (.stop-took-effect fails; CP-120), so its result ids fail as preconditions; live: WA (reliability-cf-wa-do-drive, 2026-10-14/15, the CP-118 survey on main e96f88bc98): CF Workflows' approve drain refuses the executed result's appendCommitted at a stop and swallows it (WA's red-first unit test, the stop recorded inside the approved tool's execute); related: Ruling O-73 (orchestrator, 2026-10-14): a stop refuses new work, never the record of completed work; results of work that finished are recorded after a stop (run-fenced), and the run_stopped pairing covers only calls with no result. WA's CP-118 MR fixes CF Workflows and runtime-js (the CF DO runs runtime-js); Temporal is FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS; related: CP-118 (the same refusal on a blocking companion spawn's result row); CP-66 (the approval-deny shape, whose cell the same submit-approval driver capability unblocks); inferred: an earlier version of the cell stopped through the start's handle and passed on js and cfw: that stop named the pre-approval run and recorded nothing (a vacuous green, caught before ledgering); .stop-recorded and .stop-took-effect now guard the window; runtime-js/src/js-agent-executor.ts:2919-2930 (origin/main 487090d3fb) (fix evidence, WA's CP-118 MR: the resuming run appends a drained result, an approved tool that ran included, run-fenced and not stop-guarded, Ruling O-73) and runtime-cloudflare/src/steps.ts:6966-6971 (pre-fix, origin/main 487090d3fb) (the CF Workflows approve drain appends the approved tool's result with recordsCompletedWork); conformance: narrowed on WA's CP-118 MR (head 0f56f263ab plus this record): runtime-js and CF Workflows record the approved result through the stop, so js-memory, js-redis, js-postgres and cfw-workflows-d1 leave this ledger; WA's lock window runs the cell ×3 on those lanes and on temporal with this flip applied. Temporal stays open (FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS); dbos-postgres is CP-120 first (FU-DBOS-APPROVED-RESULT-SURVIVES-STOP); related: fixed in code by dab67a452b (WA's CP-118 MR, Ruling O-73) on runtime-js, the CF DO and CF Workflows, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/js-agent-executor.ts:3896-3907 (origin/main 56d564d073) (the resume drain appends a drained result, an approved tool that ran included, run-fenced and not stop-guarded), runtime-cloudflare/src/steps.ts:9208-9213 (origin/main 56d564d073) (the CF Workflows approve drain appends the approved tool's result with recordsCompletedWork); still open on Temporal: runtime-temporal/src/activities.ts:5741-5745 (origin/main 56d564d073) (applyResultsAndReload appends the approved tool's executed result with refuseIfStopped: true) and :5750 (origin/main 56d564d073) (the CommitStoppedError is rethrown, so the completed work's result is lost) | source, live-repro | verified | open |
| CP-120 | S1 | DBOS ignores a stop recorded for a run while its approved tool executes: the approval continuation runs the tool to completion, then calls the model again and completes the run, although the stop (WA MR2's run-scoped StopRequest, the cross-process stop every runtime reads) was recorded for that run before the tool returned. The user's stop is silently lost. | conformance: approval.approved-tool-result-survives-stop (owner batch 31, executor tier): the responder approves the call, the approved tool runs until its gate, a stop is recorded for the session's current run (requestStop answers { recorded: true }), and the gate is released 500 ms later. On main e96f88bc98 plus batches 29 to 31, on dbos-postgres, 3 of 3: the run ends completed, not interrupted (.stop-took-effect fails), having made its next model call; CP-119's result ids then fail as preconditions. Every other executor lane ends interrupted; inferred: the seam is not identified yet: DBOS's approval continuation either does not read the run's StopRequest between the approved tool and the next model step, or reads it for another run id than the one requestStop recorded. A DBOS stop pressed through the active handle cancels the workflow at once (CP-57), so only a recorded stop reaches this window; related: CP-119 (its DBOS cell cannot be judged until this is fixed); FU-DBOS-APPROVED-RESULT-SURVIVES-STOP (WA, the DBOS approval continuation) | live-repro | partly-inferred | open |
| CP-121 | S2 | CF Workflows ignores companion__waitForResult's timeout: the wait on a child that does not finish runs to the runtime's own subAgentEventTimeout (60 s by default) whatever the caller asked for, and then answers a tool ERROR ({"error":"Unknown error"}, helixToolFailed) instead of the typed { name, status: 'timeout' }. A companion that ends interrupted while the wait is parked is answered as a tool error too ({ error: "Child agent … was interrupted …" }), not the typed { name, status: 'interrupted', message } every other executor lane answers. | runtime-cloudflare/src/workflow.ts:651 (pre-fix, origin/main 7a390aaf8d) (subAgentEventTimeout = options?.subAgentEventTimeout ?? 60000); runtime-cloudflare/src/workflow.ts:4117-4175 (pre-fix, origin/main 7a390aaf8d) (the waitForResult step: totalTimeout = subAgentEventTimeout at :4123; the caller's timeout argument is never read); runtime-cloudflare/src/workflow.ts:4177-4248 (pre-fix, origin/main 7a390aaf8d) (the record step: a timed-out wait builds resultPayload { name, status: 'timeout' } at :4194 but appends it with success false and no error (:4236-4245 (pre-fix, origin/main 7954d01cd5)), so createToolResultMessage writes { error: 'Unknown error' }; an interrupted child is appended success false with interruptedCompanionError at :4202-4210); core/src/orchestration/message-builder.ts:163 (pre-fix, origin/main 7a390aaf8d) (a failed tool result with no error is { error: "Unknown error" }); conformance: companion.wait-for-result-timeout-honored (owner batch 32, executor tier): the parent spawns its non-blocking companion, held in its gated tool, and waits with timeout 50. On main 7a390aaf8d plus batches 31 and 32, on cfw-workflows-d1, 3 of 3: the wait answered 60486 to 61058 ms after the child entered its gate (.wait-answered-promptly, bound 10 s) with {"error":"Unknown error"} (.wait-reports-timeout). js ×3 and temporal ×3 answer a timeout promptly; dbos-postgres ×3 passes this cell; conformance: companion.wait-for-result-interrupted-typed (owner batch 32): a stop recorded for the waited-on child, then its gate released. On cfw-workflows-d1, 3 of 3: the wait answered {"error":"Child agent 'helper' was interrupted before completing. It is resumable: use companion__sendMessage to continue it."} with helixToolFailed (.wait-result-typed). js ×3 and temporal ×3 answer the typed result; dbos-postgres is CP-124; related: WA's wait-timeout MR (reliability-cf-wa-wait-timeout-one-shape): one typed wait result shape (core CompanionWaitResultSchema) and the caller's timeout honoured on CF Workflows; runtime-cloudflare/src/workflow.ts:4174-4341 (pre-fix, origin/main 4d355e1eca) (fix evidence, WA's wait-timeout MR, FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES: the waitForResult park runs to the caller's deadline, waitDeadlineAt from its timeout, bounded by subAgentEventTimeout, and every outcome is core's typed CompanionWaitResultSchema: a timeout is { name, status: 'timeout', currentStatus }, an interrupted child { name, status: 'interrupted', message }); conformance: fixed on WA's wait-timeout MR (head 9ee9f2d967 plus this record): companion.wait-for-result-timeout-honored and companion.wait-for-result-interrupted-typed on cfw-workflows-d1 pass every id, 3 of 3 each in WA's lock window (unexpected-pass against the batch 32 ledger, failedIds []); the timeout cell's run took about 32 s end to end with the wait answered inside its 10 s prompt bound | live-repro, source | verified | fixed (2 cells) |
| CP-122 | S3 | At the HTTP boundary a missing session or checkpoint answers 409, not 404: core's status table maps every non-retryable state_* code to 409, so the CF Durable Object's /resume answers 409 state_checkpoint_not_found for another session's checkpoint and 409 state_session_not_found for a session that never started, while its own /interrupt and the chat snapshot route answer the same missing session 404. Orchestrator ruling 2026-10-15: not-found codes answer 404 at every HTTP route, a foreign checkpoint exactly like a missing one (in-process executor calls keep state_checkpoint_foreign). | core/src/errors/http-error.ts:56-67 (pre-fix, origin/main 7a390aaf8d) (httpStatusForHelixError → statusFor: framework_agent_not_found 404, validation 400, retryable 503, category state 409, else 500; no not-found group); runtime-cloudflare/src/do/drive/stop.ts:295-301 (pre-fix, origin/main 7a390aaf8d) (the stop routes' own sessionNotFound, answered 404 per the module's table at :17); conformance: wire.checkpoint-not-found-masked-host (owner batch 32, host tier, cf-do): two sessions each complete a turn; A's /resume names B's latest checkpoint, and /resume and /interrupt reach a session that never started. On main 7a390aaf8d plus batches 31 and 32, 3 of 3: .foreign-checkpoint-404 gets 409 with code state_checkpoint_not_found (the code is already the masked one), .missing-session-resume-404 gets 409 state_session_not_found; .missing-session-stop-404 passes (404). .foreign-detail-hidden passes once the caller's own echoed checkpoint id is set aside (the first window's red was the cell's: that id embeds B's session id); conformance: wire.missing-session-chat-404-host (owner batch 32): the chat snapshot of a never-started session answers 404 state_session_not_found on cf-do (a guard); related: W1 MR3 (!358), the 404 group in core's table and the foreign mask in typedErrorBody; js-chat-host is a declared gap for the DO-route cell (only the DO's native routes take a checkpoint id); core/src/errors/http-error.ts:67-80 (fix evidence, W1 MR3 !358: a not-found group, state_session_not_found / state_checkpoint_not_found / state_checkpoint_foreign, answers 404 ahead of the state category's 409) and :117-126 (wireError: over HTTP a foreign checkpoint is answered exactly as a missing one; the foreign detail is logged, never sent); conformance: fixed on W1 MR3 (188562d2f5 plus this record): wire.checkpoint-not-found-masked-host on cf-do is its provenBy cell (.foreign-checkpoint-404 and .missing-session-resume-404 answer 404 with the masked codes; .foreign-detail-hidden and .missing-session-stop-404 stay green); W1's lock window runs it ×3 with this flip applied, and wire.missing-session-chat-404-host as the guard | live-repro, source | verified | fixed (1 cells) |
| CP-123 | S2 | CF Workflows decides that a step.waitForEvent slice merely timed out by its error MESSAGE (includes('timeout') || includes('Timeout')). An engine whose slice rejection says something else ("timed out") would be read as a wait error: the companion wait would answer that the still-running child failed and record its ref failed and delivered, a zombie no sendMessage reaches and a same-name spawn restarts. Two rejections can reach the match: a slice's timeout, and a paused instance's "Aborting engine: User called pause". Neither reproduces on Miniflare 5 (its timeout text crosses workerd RPC with the class name WorkflowTimeoutError, which matches 'Timeout'; a pause records no failed ref); the production engine's texts are untested. | runtime-cloudflare/src/workflow.ts:4159-4172 (pre-fix, origin/main 7a390aaf8d) (waitForResult: a slice rejection is a timeout only if its message includes 'timeout' / 'Timeout', else { status: 'errored', WaitError }), :4027 and :3360-3361 (pre-fix, origin/main 7a390aaf8d) (the same text match at the other wait sites); runtime-cloudflare/src/companion-ref-status.ts:44-56 (pre-fix, origin/main 7a390aaf8d) (an 'errored' outcome resolves the ref 'failed'); conformance: companion.wait-slice-timeout-keeps-child-live (owner batch 32, executor tier, cfw-workflows-d1): the child held in its gated tool, the parent waits with timeout 40000 (past one 30 s slice). On main 7a390aaf8d plus batches 31 and 32, 3 of 3 plus one probe run: .child-ref-live-after-wait passes (the ref stays running); the probe saw the wait answer at 62 s with CP-121's tool error, the full 60 s bound, so both slices were taken as timeouts; inferred: WA (code-read, 2026-10-15): Miniflare 5's waitForEvent rejects a timed-out slice with a WorkflowTimeoutError ("Execution timed out after Nms"), and the error reaching the workflow body across workerd RPC very likely carries the class name, so includes('Timeout') matches by accident; a paused instance's waitForEvent rejects with "Aborting engine: User called pause" (raceAgainstAbort on the engine's pause signal), which the match does not take for a timeout (node_modules miniflare 5.20260925.0-alpha, workers/workflows/binding.worker.js); conformance: companion.wait-paused-parent-keeps-child-live (owner batch 33, executor tier, cfw-workflows-d1; anchored on the engine's own step log through Miniflare's local explorer, workflowInstances.runOwnerSteps: the parent's wait step logged its waitForEvent and has not finished it): the run owner is paused, the parent's ref sampled 10 s while paused and 15 s after the resume with the companion still held. On main 2f2ce3fdab plus batches 32 and 33, green 3 of 3 (host load 28 to 74): no failed ref while the companion lives, so the pause rejection's misclassification does not land as a recorded ref on Miniflare either (a guard there); live: peer-reported (WA, 2026-10-15, the wf-noiso real-engine park test, on its wait-timeout MR !364): on main the timeout path answers an untyped error, { error: 'Unknown error' } after 60 170 ms; green at head; related: WA, the wf-noiso real-engine park test (the arbiter for the production engine); CP-121 (the same wait ignores the caller's timeout); orchestrator ruling 2026-10-15: no red ledger while the Miniflare lane does not reproduce it | source | partly-inferred | open |
| CP-124 | S2 | DBOS: companion__waitForResult never sees a persistent companion's turn end. It polls the child's SESSION status, which a DBOS persistent child keeps 'active' between turns (CP-69's root), so a companion stopped while the parent waits (its run ends 'interrupted') is reported timeout (current: running) at the caller's timeout, or never with no timeout. This is the path the DBOS docs give as CP-69's workaround (non-blocking spawn + companion__waitForResult). Every other executor lane answers the typed { name, status: 'interrupted', message }. | runtime-dbos/src/steps/execute-companion-tool.ts:1671-1758 (pre-fix, origin/main 7a390aaf8d) (handleWaitForResult: each poll resolves the child through resolveChildStatus, answers a terminal status, else 'timeout (current: <status>)' at :1748 once the caller's timeout passed); runtime-dbos/src/steps/execute-companion-tool.ts:1605-1653 (pre-fix, origin/main 7a390aaf8d) (resolveChildStatus: a non-terminal ref is resolved from the child's SESSION status, :1617 loadState, :1626 only a terminal session status counts; the child's run record is never read); runtime-dbos/src/workflows/persistent-workflow.ts:392 and :425 (pre-fix, origin/main 7a390aaf8d) (each persistent turn runs with skipSessionStatusUpdate: true, so the session stays 'active' between turns); conformance: companion.wait-for-result-interrupted-typed (owner batch 32, executor tier): the parent spawns its non-blocking companion, held in its gated tool, and waits with a 30 s timeout; a stop is recorded for the child's current run (requestStop answers { recorded: true }) and the gate released. On main 7a390aaf8d plus batches 31 and 32, on dbos-postgres, 3 of 3: the wait answers {"name":"helper","status":"timeout (current: running)"} at 30 s (.wait-result-typed fails). The child's run record is 'interrupted' (completion_reason interrupted) while its session row stays 'active', step_count 0, in all three runs (read from the lane's Postgres after each run). js ×3 and temporal ×3 pass it; related: CP-69 (the blocking spawn polls the same session status and stalls to the idle timeout; its evidence names waitForResult as the recovery path); docs/dev/follow-ups.md FU-DBOS-BLOCKING-SPAWN-SEMANTICS; inferred: a companion whose turn COMPLETES is reported the same way (its session also stays 'active'); not run here; conformance: fixed by !601 (CP-69's MR). companion.wait-for-result-interrupted-typed passes .wait-result-typed (WE-2 slot w-cp69b-1031-022828, dbos-postgres, 3 of 3, identical sets, at CP-69's fix head b03061b924 (restacked as !601 72f71bcf0d on main a203fec499)); conformance: re-confirmed at !601 de0c3029fd (its Greptile fixes; WE-2 slot w-601g-1031-050650, dbos-postgres, 3 of 3, identical to the w-cp69b sets) | live-repro, source | verified | fixed (1 cells) |
| CP-125 | S1 | CF DO: a persistent companion that is alive but unreachable for the poll-failure window (30 s of failing /status polls) is recorded failed and its completion marked delivered BEFORE the parent tries to stop it, and that stop is best-effort. When the stop does not reach the child either, the child keeps running as a zombie: its ref already says failed and delivered, so its real completion is never delivered to the parent, and a same-name re-spawn deletes its session. Both the blocking spawn's park and companion__waitForResult give up this way. | runtime-cloudflare/src/do/persistent-companion-tools.ts:131 (pre-fix, origin/main 7a390aaf8d) (DEFAULT_POLL_FAILURE_WINDOW_MS = 30_000); runtime-cloudflare/src/do/persistent-companion-tools.ts:185-213 (pre-fix, origin/main 7a390aaf8d) (markChildFailedAndAbort: updateSubSessionRef status 'failed', completionDelivered true, then stopChildDO kind 'abort'), :863-877 (pre-fix, origin/main 7a390aaf8d) (stopChildDO logs a failed stop and never throws); runtime-cloudflare/src/do/persistent-companion-tools.ts:1539-1600 (pre-fix, origin/main 7a390aaf8d) (pollChildToTerminal, the blocking spawn park: once every poll failed for the window, :1594 calls markChildFailedAndAbort and throws) and :1405 (pre-fix, origin/main 7a390aaf8d) (the same give-up in companion__waitForResult); related: WA (DO companion group, with CP-39 / CP-40 / CP-23), follow-up FU-DO-BLOCKING-SPAWN-UNREACHABLE-ZOMBIE (on WA's branch, not on main yet); orchestrator ruling 2026-10-15 to file it with a cf-do witness; conformance: companion.unreachable-child-blocking-host and companion.unreachable-child-wait-host (owner batch 33, host tier, cf-do; the harness control fault-child-do-route, harness/cf-do/worker/child-route-fault-namespace.ts: the parent DO's requests to the child's /status and stop routes fail as an unreachable DO's would, armed for 40 s, then disarmed and the child's gate released). On main 2f2ce3fdab plus batches 32 and 33, 3 of 3 (host load 8 to 14): both fail .ref-not-failed-while-live (the parent records the companion's ref failed while its session is active) and .completion-reaches-parent (the spawn / wait is answered 'Persistent child … has been unreachable for 30000ms …' / '… while waiting for its result'), while .child-completes passes: the companion finishes its turn after the release, its completion never delivered; conformance: companion.unreachable-child-stop-reaches-host (owner batch 33, the same window with only /status unreachable): green 3 of 3, the parent's ref matching the companion's own outcome (a guard for the stop that reaches); core/src/orchestration/companion-tool-dispatch.ts:353-354 (pre-fix, origin/main 2e5afe376b) (fix evidence, WA's MR A, D1: ChildTerminalResult's unreachable arm; nothing is written and the ref stays live) with :512-520 (pre-fix, origin/main 2e5afe376b) (applyParkResult answers a typed unreachable and writes no ref from a transport failure); conformance: fixed by WA's MR A (!389; the owner's MR A window 2026-10-17, 3 runs, at 9fd0cc5daf plus the owner's cell commit): companion.unreachable-child-blocking-host and companion.unreachable-child-wait-host pass every id 3 of 3 on cf-do (red on main: .ref-not-failed-while-live). Their .completion-reaches-parent (the result on the spawn / wait call's own row) was removed: under D1 the park answers typed unreachable at the window, and the completion reaches the parent on a later turn. The delivery is witnessed by the siblings companion.unreachable-child-blocking-next-turn-host and companion.unreachable-child-wait-next-turn-host (the result reaches the parent's history exactly once) | source, peer-reported, live-repro | verified | fixed (2 cells) |
| CP-126 | S1 | CF Workflows: companion__waitForResult can answer with a companion's PREVIOUS run's completion, a silent wrong answer. A companion run's sub-agent-complete:<child> event that no wait consumed stays buffered in the parent instance, and the wait takes the first event of that type it dequeues: the blocking spawn and continuation parks filter by the parent consult (I-6), the wait does not. So in one parent turn, after a companion run R1 ended with nothing waiting on it and a companion__sendMessage started its run R2, the parent's wait returns R1's output as completed while R2 is still running. | runtime-cloudflare/src/workflow.ts:4117-4175 (pre-fix, origin/main 2f2ce3fdab) (the waitForResult park: each slice's waitForEvent takes any sub-agent-complete:<child> event, parsed at :4148 with no check of which run or consult sent it); runtime-cloudflare/src/workflow.ts:5168-5176 (pre-fix, origin/main 2f2ce3fdab) (the I-6 filter the blocking spawn and continuation parks apply: an event of another consult is skipped; the wait does not use it); runtime-cloudflare/src/steps.ts:7495-7499 (pre-fix, origin/main 2f2ce3fdab) (sendMessage to a completed companion writes no ref; the continuation's run-start commit turns it running, run-entry.ts:1705-1711, so after a delivered send the wait parks on the event rather than answering from the ref); conformance: companion.wait-not-answered-by-stale-completion (owner batch 34, executor tier, cfw-workflows-d1; anchored on the engine's step log: the parent holds in its own gate until R1's run owner logged notify-parent-complete finished, and R2 is released once the wait parked or answered). On main 2f2ce3fdab plus batches 32 to 34, 3 of 3 (host load 15 to 18): .r1-sent, .send-delivered (delivered: true) and .wait-settled pass; .wait-not-stale and .wait-answers-r2 fail: the wait answered {"name":"helper","status":"completed","result":{"ok":"r1"}} while R2 was still held in its gate; conformance: companion.wait-after-spawn-continue-not-stale (owner batch 34, the same anchors; WA's m-4 trigger: a non-blocking companion__spawnAgent that continues the completed companion and the wait in ONE step). On main 2f2ce3fdab plus batches 32 to 34, 3 of 3 (host load 8 to 9): .wait-not-stale and .wait-answers-r2 fail with R1's output, and the parent's step log shows the wait PARKED (a waitForEvent step), so it was answered by R1's buffered event, this record's path, not from a stale ref (the m-4 candidate's path did not answer here); live: WA (reliability-cf-wa-do-drive, 2026-10-15): the mechanism and the delivered-send guard; the fix (an event filter on the companion run in the four parks) is WA's, in the wait-timeout MR or the next; inferred: the Workflows engine's event buffering in production is unconfirmed; the cell proves Miniflare 5's; runtime-cloudflare/src/workflow.ts:5727-5777 (pre-fix, origin/main 4d355e1eca) (fix evidence, WA's !364, the wait-timeout one-shape MR) (the wait's park takes ONLY an event of the consult it waits on, fail closed, and drops any other consult's or an earlier run's event, then re-reads the child's committed outcome), with the same consult filter in the spawn and continuation parks (:4299 (pre-fix, origin/main 4d355e1eca), :4071 (pre-fix, origin/main 4d355e1eca), :3485 (pre-fix, origin/main 4d355e1eca)) and a terminal ref of an older consult not taken as this wait's answer (:4361 (pre-fix, origin/main 4d355e1eca)); conformance: fixed by !364. companion.wait-not-answered-by-stale-completion and companion.wait-after-spawn-continue-not-stale (cfw-workflows-d1) pass 3 of 3 on !364's head (WA's run): the wait no longer answers R1's buffered completion, and answers R2's own; their cfw ledger rows are dropped. Red 3 of 3 before it (owner batch 34, the ledgered runs above) | source, peer-reported, live-repro | verified | fixed (2 cells) |
| CP-127 | S1 | runtime-js: a persistent companion that SUSPENDS (on a client-executed tool, or awaiting an approval) is recorded in its parent's ref as completed, with no output, while it is paused. The companion's in-process loop returns a suspended_* outcome; runSubAgentLoop throws only for a failed or superseded one and returns for everything else, and the spawn (and continuation) wrapper then writes the ref completed. So a blocking spawn answers completed with no result, a later companion__waitForResult reads the same ref, and the push notifier treats the paused companion as finished: a silent wrong answer. | runtime-js/src/run-loop.ts:6141-6174 (origin/main 65ebe4a50c) (runSubAgentLoop: throws only for outcome.kind 'superseded' or 'failed'; every suspended_* outcome falls through to return undefined, the comment at :6167-6172 naming the suspension paths as not handled); runtime-js/src/run-loop.ts:6562-6584 (origin/main 65ebe4a50c) (spawnPersistentChild: after runSubAgentLoop returns, updateSubSessionRef(... status: 'completed') and { status: 'completed', output: completed?.output }, the output of a paused session being undefined); runtime-js/src/run-loop.ts:6874-6887 (origin/main 65ebe4a50c) (the continuation wrapper: the same completed ref write after runSubAgentLoop); core/src/orchestration/run-outcome.ts:44-46 (origin/main 65ebe4a50c) (RunOutcome's suspended_client_tool / suspended_awaiting_children / suspended_step_partial kinds); live: WA (reliability-cf-wa-do-drive, 2026-10-15): the candidate, from the DO companion design work; the planned fix (a typed suspended answer) is that design's MR C; conformance: companion.client-tool-suspended-not-completed and companion.approval-suspended-not-completed (owner batch 34, executor tier): a BLOCKING persistent companion suspends on a client-executed tool / an approval, the ref read once the parent turn settles. On main 5fb9f6f543 plus owner batch 34, 3 of 3 on every executor lane (host load 6 to 12 on 14 cores): .ref-not-completed fails on js-memory, js-redis and js-postgres (the parent's ref reads completed while the companion is paused); temporal-memory, temporal-redis, temporal-postgres, cfw-workflows-d1 and dbos-postgres pass both cells; live: fix owner (orchestrator ruling 2026-10-15): WA's DO companion design MR C (a typed suspended answer, status suspended with pendingCalls naming the call); runtime-js/src/run-loop.ts:7990-7994 (fix: CP-40 C2 T4, WA; as on branch reliability-cf-wa-cp40-c2 3b864439b0) (runSubAgentLoop reports a committed suspended_client_tool outcome instead of returning undefined) and :7853-7868 (fix: CP-40 C2 T4, WA; as on branch reliability-cf-wa-cp40-c2 3b864439b0) (recordSuspendedChild: the ref is written paused_awaiting_client with no completedAt, never completed, and the answer is suspended with the awaited calls; :8177-8186 is the resumed turn's site); conformance: fixed. companion.client-tool-suspended-not-completed and companion.approval-suspended-not-completed pass .ref-not-completed on js-memory, js-redis and js-postgres (executor), 3 of 3, at WA's C2 T4 head 24b615b21f plus the owner's C2 cells (owner slot s146, 2026-10-25), and still pass on cfw-workflows-d1 after C2 T5 (owner slot s147, 65f16e8d01); s146 / s147 / s148b are the C2 car's lane evidence (MR pipelines skip the conformance lanes since !525) | source, peer-reported, live-repro | verified | fixed (6 cells) |
| CP-128 | S1 | runtime-js: a persistent companion that suspends (on a client-executed tool, or awaiting an approval) while its parent's step is open FAILS the parent. The companion's pending call writes its ownership entry onto the ROOT session with a plain saveState, which moves the root's version; the parent's open step then loses its commit's version CAS (StaleStateError), and the run loop ends the parent failed with an uncoded "Executor superseded for session … - another executor has taken over" (no newer run exists). The step's rows (its tool results) are lost, and no later step runs. | runtime-js/src/client-tool-resolver.ts:221-243 (origin/main 65ebe4a50c) (registerPending: rootId = the session's rootSessionId; for a child, ownership is written on the root FIRST, :241-243) and :427-458 (origin/main 65ebe4a50c) (writeOwnership: loadState(root), set clientToolCallOwnership, saveState(rootId, root), a version-bumping write to a session whose run may be mid-step); runtime-js/src/run-loop.ts:1253-1306 (origin/main 65ebe4a50c) (a step commit that threw StaleStateError: the comment reads it as 'this run is still current and live and another writer moved the session's version'; the run ends failed with ExecutorSupersededError's message, its errorDetail resolved from that error, no code); core/src/errors/agent-errors.ts:155-170 (origin/main 65ebe4a50c) (ExecutorSupersededError: 'Executor superseded for session … - another executor has taken over'); related: CP-40 is the CF DO's side of the same ownership write (it fails silently there); this record is runtime-js's, where the write lands and breaks the parent's commit; conformance: companion.client-tool-suspend-during-parent-step and companion.approval-suspend-during-parent-step (owner batch 34, executor tier): the parent is held in its gated tool until its non-blocking companion holds its client-tool / approval call pending, then released. On main 5fb9f6f543 plus owner batch 34, 3 of 3 on every executor lane (host load 6 to 12 on 14 cores): .parent-completes and .gated-row-kept fail on js-memory, js-redis and js-postgres (the parent ends failed, 'Executor superseded for session … - another executor has taken over', with no errorDetail code, and the gated call's result row is lost); temporal-* and cfw-workflows-d1 pass both cells. dbos-postgres fails .parent-completes for a different defect, CP-130 (the parent's result answers suspended early); its gated row is there once its session settles; live: fix owner (orchestrator ruling 2026-10-15): WA's MR D (a field-scoped client-tool ownership write that never moves the version, and a typed StepCommitConflictError on every surface); conformance: owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 3 of 3 identical: companion.answer-after-parent-turn-runs (its root held live in a gated tool while the companion suspends) ends the root's own turn failed on js-memory, js-redis and js-postgres (.parent-completes: 'the root's session after its turn: failed'), and every later id fails after it; companion.answer-on-paused-child-session's turn 1 fails on js-memory only ('Executor superseded … another executor has taken over'), js-redis and js-postgres passing it; conformance: companion.send-to-paused-child-keeps-calls on js-memory (owner slot s217, 2026-10-31, tree b596b9e320, 3 of 3 identical): turn 1 fails ('Executor superseded for session …', .turn1-completes), as its control's does, and every later id fails after it; conformance: no longer ledgered here (orchestrator ruling A3 on batch 3 !613, 2026-10-31): companion.answer-on-paused-child-session and companion.send-to-paused-child-keeps-calls held the race only by timing (WA's proof: at the batch head their js-redis cell passed 5 of 5 free runs, and held orders are deterministic both ways), so their shared builder now holds the helper's first step until turn 1 has settled; this record stays ledgered on companion.client-tool-suspend-during-parent-step, which holds the losing order on purpose; conformance: fixed by MR D (WA, !616, !620, !622, !628, !629, !630): client-tool ownership has one writer, the store's field-scoped updateClientToolCallOwnership, which moves no version, and full-state writes keep the stored map and markers. A step commit that does lose a version race is a typed StepCommitConflictError (state_concurrency_conflict): the stream fails and onAgentFail fires; conformance: owner slot s231 (2026-11-01, MR D through its D6 pins on main 5eef81d2db), 3 of 3 identical on js-memory, js-redis and js-postgres: companion.client-tool-suspend-during-parent-step, companion.approval-suspend-during-parent-step and companion.answer-after-parent-turn-runs pass, read as unexpected passes against the old ledger; conformance: the paused-child builder's hold and its 'companion-runs-in-parent-process' flag are gone. companion.answer-on-paused-child-session and companion.send-to-paused-child-keeps-calls pass unheld at owner slot s231, 3 of 3 on the same lanes | source, live-repro | verified | fixed (9 cells) |
| CP-129 | S1 | runtime-js: a non-blocking persistent companion still running when its parent's turn ends FAILS. The companion writes into its parent's stream (it never claims it), and the turn's end closes that stream, so the companion's next write throws "Cannot write to stream … in 'ended' state": its session and the parent's ref read failed, its work is lost, and the parent's next turn is told it failed. The sub-agents guide documents the opposite contract for Temporal (a detached child's live chunks to an ended or paused parent stream are dropped instead of failing the child). | runtime-js/src/run-loop.ts:6118-6127 (origin/main 65ebe4a50c) (a companion child's run starts with ownsStream: false: 'a child writes into its parent's stream — never claims it'); store-memory/src/in-memory-stream.ts:270-280 (origin/main 65ebe4a50c) (a writer throws "Cannot write to stream ${streamId} in '${status}' state" for a non-active stream); related: docs/guide/sub-agents.md:223-229 (origin/main 65ebe4a50c) ('A child that writes into its parent's stream never changes that stream's liveness … On Temporal, a detached child (a non-blocking companion) that is still running after the parent's turn ended or paused the stream has its live chunks to that stream dropped instead of failing the child'); severity S1 (orchestrator ruling 2026-10-16, raised from S2): the parent never learns the companion's outcome, a silent loss of the companion's answer (runtime-js fails it on the ended stream; CF Workflows stalls it for minutes, then takes its error path); CF backlog §7 lists it at S1; conformance: companion.outlives-parent-turn, reworked to the documented contract (owner batch 35, executor tier): turn 1 spawns the non-blocking helper, whose first step holds in its gated tool, and completes; the companion is released and its outcome awaited as an event (a terminal session status, or on CF Workflows its engine step log showing the error path); then turn 2, which must carry exactly one completed notice for it, after which the parent's ref reads completed. On main 5fb9f6f543 plus batches 34 and 35, 3 of 3 per lane (host load 5 to 13 on 14 cores): js-memory, js-redis and js-postgres fail .child-completes, .completion-delivered and .ref-completed-after-delivery (the companion fails 'Cannot write to stream … in ended state'); cfw-workflows-d1 fails .child-completes (its run took the error path after its tool-starts step, the write into the ended parent stream, retried about 60 s) and turn 2's ids as preconditions; temporal-memory, temporal-redis and temporal-postgres pass all 7 ids (the documented contract holds there); dbos-postgres fails for a different mechanism, CP-132; conformance: W1's CP-129 part 1 (the stream managers drop a child's chunk to an ended / failed / paused parent stream), the same cells 3 of 3 on its branch: js-memory, js-redis, js-postgres and cfw-workflows-d1 pass all 7 ids; temporal-* stays green; dbos-postgres is unchanged (CP-132); live: fix owner (orchestrator ruling 2026-10-16): W1's CP-129 MR, part 1 (a non-owning companion never writes to an ended parent stream, on every runtime); CF Workflows half in CLOUDFLARE-BACKLOG.md §7; core/src/stream/run-markers.ts:98-129 (origin/main af9dfde232) (fix evidence, W1's CP-129 part 1, !378: decideStreamWrite, the one stream-write rule every manager applies: a chunk is OWN when its agentId is the stream's or it carries the owning run; a guest chunk to an ended, failed or paused stream is dropped, not thrown); conformance: fixed by !378 (owner batch 35, main af9dfde232 plus batch 35, 3 of 3 per lane, host load at or below 14): companion.outlives-parent-turn passes every id on js-memory, js-redis, js-postgres and cfw-workflows-d1 (red before !378, ledgered above in the record's history); temporal-memory, temporal-redis and temporal-postgres pass as before; dbos-postgres stays red on CP-132's ids | source, live-repro | verified | fixed (4 cells) |
| CP-130 | S1 | DBOS: a parent's result() reports suspended_client_tool for a call that belongs to its NON-BLOCKING companion while the parent's own run is still active. The active handle races the workflow result against a poll that treats every unsubmitted pending call reachable from the root session, sub-agent-owned ones included (through clientToolCallOwnership), as the handle's own suspension. That is right for an ephemeral sub-agent the parent blocks on, but a non-blocking companion's pending call does not suspend its parent. So the caller is told the parent is suspended awaiting a client tool while the session reads active and the parent's step is still running: the result and the session disagree, the parent's real outcome is never reported through that handle, and a caller that acts on the answer (submits or resumes the parent) acts on a run that never suspended. | runtime-dbos/src/handles/active-handle.ts:221-227 (origin/main 5fb9f6f543) (_fetchResult races the workflow result against _pollForPendingClientTool) and :272-320 (origin/main 5fb9f6f543) (a poll win returns status suspended_client_tool with the observed toolCallIds); runtime-dbos/src/handles/active-handle.ts:484-497 (origin/main 5fb9f6f543) (the poll enumerates every unsubmitted pending toolCallId visible from the root, root-owned and sub-agent-owned, the comment naming the ephemeral sub-agent case) with :622-627 and :676-704 (origin/main 5fb9f6f543) (the enumeration follows clientToolCallOwnership into the owner session's pendingClientToolCalls, with no check that the owner is a child the parent blocks on); conformance: companion.client-tool-suspend-during-parent-step and companion.approval-suspend-during-parent-step (owner batch 34, executor tier) on dbos-postgres, main 5fb9f6f543 plus owner batch 34, 3 of 3 each (host load 6 to 12 on 14 cores): the parent is held in its gated tool until its non-blocking companion holds its client-tool / approval call pending, then released. The parent's outcome is {terminal: suspended, source: suspended_client_tool} while the persisted session reads active with no error (.parent-completes), and the parent's gated call has no result row yet when that early answer comes. Read once the parent's session has left active (.parent-session-settles), the gated row is there (.gated-row-kept passes, 3 of 3 at the re-run, main 5fb9f6f543 plus batch 34 and its fix): the early answer, not a lost step, is the defect. .gated-row-kept read at the answer was a race (red 3 of 3 locally, green on !373's CI), so it is judged after the session settles. .parent-completes was a race too while the parent was released as soon as the companion's call was pending (red 3 of 3 locally, an unexpected pass on !373's train pipeline 2956990146, when the parent completed before the pending poll's next read, at most 500 ms apart): the parent is now held until its handle answers or EARLY_ANSWER_HOLD_MS (10 s) passes, so the early answer always comes while the parent is held. runtime-js fails the same ids for CP-128 (the parent fails 'Executor superseded'); temporal-* and cfw-workflows-d1 pass both cells; related: CP-128 (runtime-js, the same cells: the companion's root ownership write fails the parent's step); CP-93 (another DBOS status split, a held batch reads active). Filed by the owner from the batch 34 every-lane run; orchestrator ruling 2026-10-16: a separate S1 record, parked with DBOS; conformance: fixed. companion.client-tool-suspend-during-parent-step and companion.approval-suspend-during-parent-step: .parent-completes fails on dbos-postgres before the fix (owner batch 34 above, 3 of 3) and passes 3 of 3 at !589 05a804cda2 (WE-2's fix: the root's handle reports its own run, never its companion's pause; WE-2 slot w-589fix-1030-223459, 2026-10-30, with the owner cells 4f978f4ce0 / 6ac17a2b50 alongside), every other id passing too; severity S1: a wrong terminal answer for a live run. The caller is told the parent suspended on a client call it never made, the session says active, and the parent's real outcome never reaches that handle; acting on the answer (a submit or resume of the parent) targets a run that did not suspend; related: scope DBOS only (WE-2's cross-runtime read, 2026-10-30, spot-checked by the orchestrator): runtime-js's handle reads its own pendingClientTools (core/src/orchestration/step-iterator.ts:2561-2565), Temporal its own step (runtime-temporal/src/workflow.ts:1502 and :1543-1549), CF Workflows its own output (runtime-cloudflare/src/executor.ts:2676), the CF DO one session per DO (runtime-cloudflare/src/do/durable-object-agent-base.ts:4970 and :5013-5018); it confounds CP-233's dbos .wait-answers-suspended (see CP-233); fixer WE-2, in the CP-247 MR; related: since this record's fix (!589), DBOS's root handle no longer surfaces a client call raised by a blocking companion's own EPHEMERAL sub-agent (the over-broad poll did, by accident), and the companion's readers see no pause, so that call waits silently until its clientToolTimeout: DBOS's documented-unsupported doubly-nested HITL (docs/runtimes/dbos.md, GitLab #73; FU-DBOS-NESTED-HITL-UNDER-COMPANION, WE-2, 2026-10-31); no cell pinned it, so not a regression id | source, live-repro | verified | fixed (2 cells) |
| CP-131 | S1 | CF Workflows: a companion__waitForResult in a LATER parent turn does not answer a companion that finishes during the wait with its result. The wait waits for the child's sub-agent-complete event inside the CURRENT turn's instance, while the child sends that event to the instance that spawned (or continued) it. Observed on main: the companion completes (its session completed, every row committed, the parent's ref completed), and the wait's row is a failure with no error text, {"error":"Unknown error"} (the message builder's default for a failed result with no message; the payload's own status is not kept in the row), so the later turn is told the wait failed while the companion's result is committed. | runtime-cloudflare/src/workflow.ts:4186-4212 (origin/main 5fb9f6f543) (waitForResult: a step.waitForEvent on 'sub-agent-complete' for the child session, inside the current instance) with :4013 and :4401 (origin/main 5fb9f6f543) (the spawn and the continuation give the child parentWorkflowId: event.instanceId, the spawning turn's own instance); live: peer-reported (W1, 2026-10-16, with the orchestrator's Q4 ruling on CP-129): one instance per call, so after the spawning turn's instance closes a later turn's wait can only time out (subAgentEventTimeout), even when the child has long finished; related: CP-126 (the same wait answered by a previous run's buffered completion, closed by WA's !364, whose consult keying assumes the same instance); CP-129 (a companion outliving its parent's turn). Fixed by WA's !364 rather than the planned W1 event-addressing MR: its rewritten wait reads the child's committed outcome after each rejected slice. The cell judges correctness, not latency: the answer can wait up to one 30 s wait slice; severity S1: the later turn is told the companion timed out while its result is committed, a wrong answer the parent model acts on (it may re-spawn or give up on work that is done); conformance: companion.wait-later-turn-child-finishes-during-wait (owner batch 36, executor tier; the wait's start observed through 'observe-companion-wait-start': on cfw the parent's run owner parked in its waitfor-event step for the helper) on cfw-workflows-d1, main af9dfde232 plus owner batches 35 and 36, 3 of 3: .wait-answers-result fails (the wait's answer {"error":"Unknown error"}, not the companion's completed result). A diagnostic run printed the persisted state after turn 2: the companion's session completed with rows [user, assistant, tool, assistant, tool], the parent's ref completed, and the wait's row content {"error":"Unknown error"} with metadata.helixToolFailed. js-memory, js-redis and js-postgres pass every id 3 of 3; Temporal and DBOS are declared gaps (no observable companion wait start); core/src/orchestration/message-builder.ts:156-164 (origin/main af9dfde232) (encodeResult: a failed result with no error message is stored as {"error":"Unknown error"}) with runtime-cloudflare/src/workflow.ts's record-waitfor step (origin/main af9dfde232): success only for a completion event whose status is 'complete', the error text only from that event's error.message; conformance: fixed by WA's !364 (merged as 4d355e1eca): at its head 14a3b5c6e5 plus owner batches 35 and 36, 3 of 3 on cfw-workflows-d1, the same cell passes every id, and so does the already-finished guard companion.wait-later-turn-reaches-finished-child (!364 rewrites this wait: it reads the child's committed outcome after each rejected slice) | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-132 | S1 | DBOS: a finished non-blocking persistent companion never reaches a terminal status, whether or not its parent's turn has ended. Its whole run commits (its gated tool's result and its __finish__ call's result are both in its history), yet its session stays active indefinitely and the parent's ref reads running, so the companion's outcome is never delivered: the parent's next turn gets no completion notice and a wait can never answer it. Unlike CP-129 (runtime-js / CF Workflows, the companion's write into the parent's ended stream), the stream write is not the cause here: W1's CP-129 part 1, which drops a child's chunks to an ended parent stream, leaves this unchanged. | conformance: companion.outlives-parent-turn (owner batch 35, executor tier) on dbos-postgres, main 5fb9f6f543 plus owner batches 34 and 35, 3 of 3 plus 2 diagnostic runs (host load 5 to 13 on 14 cores): turn 1 completes with the companion held in its gated tool; released, the companion runs to its end, its history reading user, assistant, the gated tool's result, assistant, the __finish__ result, while its session stays active past the 180 s bound (.child-settles, .child-completes: 'undecided', status active), so turn 2 never runs (.turn2-settles, .completion-delivered, .ref-completed-after-delivery as preconditions); conformance: the same cells on W1's CP-129 part 1 branch (the stream managers drop a child's chunk to an ended / failed / paused parent stream), 3 of 3: js-* and cfw-workflows-d1 turn green, dbos-postgres is unchanged, so the DBOS cause is not the ended-stream write; inferred: the companion's terminal commit (its K5 / run finalization) or its completion message to the parent never lands once the parent's turn has ended; the DBOS workflow-side cause is not yet traced (runtime-dbos/src/steps/execute-companion-tool.ts carries the companion's DBOS.send completion delivery, a candidate); related: CP-129 (the runtime-js / CF Workflows half of the same scenario, a different mechanism); CP-130 (another DBOS companion status split). Filed by the owner from the CP-129 rework's every-lane run; orchestrator ruling 2026-10-16: its own record, parked with DBOS; severity S1 (orchestrator ruling 2026-10-16): the parent never learns a finished companion's outcome, a silent loss of the companion's answer, the same class as CP-129; conformance: broadened (owner batch 35, orchestrator ruling 2026-10-17): companion.wait-later-turn-reaches-finished-child on dbos-postgres, main af9dfde232 plus batch 35, 3 of 3. Turn 1 spawns the companion and then holds in its own gated step, so the parent's turn is still LIVE while the companion runs; the companion never reads completed within 60 s (.child-completes; every later id fails its precondition). A diagnostic run printed the child's persisted state while turn 1 was held: status active, rows [user, assistant, the __finish__ call's result], so its whole run is committed; 5 s after the release the parent had completed and the child was still active. The parent's turn ending is therefore not the trigger; conformance: fixed by !601 (CP-69's MR: a DBOS persistent turn writes its terminal to the session, so a finished companion's session reads terminal per turn). companion.outlives-parent-turn and companion.wait-later-turn-reaches-finished-child pass every id on dbos-postgres at owner slot s226 (2026-10-31, tree b4be0f11bc = batch 4 !617's head: main 47f0892469 plus !601 / !602 / !609-!612 / !614), 3 of 3, identical sets; batch 4's own test:conformance:dbos job 17632129076 showed both as unexpected passes first. The failed-run shape passes there too (companion.child-error-not-parent-outcome, the CP-255 member's cell, same slot, its tree B, 3 of 3) | live-repro | partly-inferred | fixed (2 cells) |
| CP-133 | S2 | The CF Durable Object refuses a chat send that arrives after the previous turn's reply has ended but before that turn's drive has entered its tail: 409 state_already_running for a run that is already terminal. The executor's terminal order ends the stream before the drive's handle.result() resolves, and the DO marks the drive settled (the state in which a new entry waits for it instead of losing) only after that. A send in the window meets a registered, unsettled drive and is refused outright. A real client lands there: sendAutomaticallyWhen posts the moment a stream ends, and so does a fast follow-up. | runtime-cloudflare/src/do/drive/run-drive.ts:409-436 (origin/main f93791269b) (checkConcurrency: a registered drive that is settled gets a bounded wait, then idle; one that is not yet settled loses a chat send outright, reject_concurrent_entry, which the route answers 409 state_already_running) with :753-761 (origin/main f93791269b) (the drive's tail: host.enterTail(execution) runs only after handle.result() resolved) and runtime-cloudflare/src/do/durable-object-agent-base.ts:1075-1078 (origin/main f93791269b) (enterTail sets the registration's settled flag); live: CI, !386's pipeline 2957791227 (job 17251605874, test:conformance 1/5): client.retry-reuses-key|cf-do|client failed .retry-resent with the TURN-2 sends [409, 409]: the product client's retry(), posted right after turn 1's reply ended on the pre-S5 page, was refused 409 like the send that ran while turn 1 was live. The cell now waits for the host to report the run no longer executing before its retry (owner, !381), so it no longer races; inferred: js-chat-host has no such window. Its entry gate is the run record, whose terminal status is committed before the stream ends (the terminal order: K5, terminal run status, stream end); severity S2 (orchestrator ruling 2026-10-17): a send the client is right to make (the turn is over) is refused as if a run were live; it is the DO's start reliability, and an automatic send after a stream ends (sendAutomaticallyWhen) meets it by design; conformance: chat.send-in-run-end-window-host (owner batch 36, host tier; hold-run-end holds turn 1's run after its stream's terminal write, turn 2 is sent in that window and judged PARK_MS in): on main (af9dfde232 product plus owner batches 35 and 36), 3 of 3, cf-do fails .send-not-refused (turn 2's send answered 409 at once) and .turn2-completes; js-chat-host passes every id. At WA's !390 head fc57abe417 (the drive-409 fix) with the control and the cell applied, cf-do passes every id 3 of 3; conformance: fixed by WA's !390 (merged as 68cad4d1d2: DO drive settle 409, stage 1): at main 68cad4d1d2 plus owner batches 35 and 36, 3 of 3, chat.send-in-run-end-window-host passes every id on cf-do (host load 18 to 20 on 14 cores during the cells) | source, live-repro | verified | fixed (1 cells) |
| CP-134 | S3 | A same-name companion__spawnAgent for a companion whose parent ref reads running (or interrupted) is an untyped tool error, Agent '<name>' is already running. Use sendMessage to communicate with it., not the typed { status: 'not_spawned', reason: 'child_busy', delivered: false } answer of the spawn contract (CompanionSpawnResult). Core's resolveChildName throws before any dispatch, so the model gets an error string it has to parse. On CF Workflows a non-blocking child's ref stays running until the parent's next iteration syncs it, so a busy continue there always meets this refusal; runtime-js answers the same continue typed (its lost reopen leaves the ref completed). | core/src/tools/companion/helpers.ts:78-81 (origin/main da336553a6) (resolveChildName throws the untyped refusal when the existing ref's status is in ACTIVE_STATUSES, :6, running or interrupted); conformance: companion.spawn-continue-not-spawned (WA's DO companion MR B cells, executor tier; the owner's MR B window 2026-10-17, 3 of 3 per side) on cfw-workflows-d1 fails .answer-typed on main and at MR B's head (the spawn row is {"error":"Agent 'helper' is already running. Use sendMessage to communicate with it."}); the same cell passes every id on js-memory, js-redis and js-postgres at MR B's head; related: FU-COMPANION-ALREADY-RUNNING-TYPED (on WA's DO companion MR B / MR C branches, not yet on main; fix owner WA, MR C2, the typed refusal in resolveChildName); FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY (docs/dev/follow-ups.md; MR B, the continue answers on the contract); severity S3: the outcome is right (nothing is spawned or wiped), only its shape is untyped: a model reading the tool error may retry or misreport, and callers cannot branch on a typed reason; core/src/tools/companion/helpers.ts:95-145 (fix: CP-40 C2 T1, WA, a3df1a3ba2; as on branch reliability-cf-wa-cp40-c2 bc5a339945) (resolveChildName returns { kind: 'resolved' | 'refused' }: a running name is refused child_busy, an interrupted one child_interrupted, a paused one child_awaiting_client with its calls; the dispatcher answers the spawn contract's typed not_spawned, never a thrown tool error; CF Workflows and the CF DO spawn through it); conformance: fixed. companion.spawn-continue-not-spawned passes on cfw-workflows-d1 (executor), 3 of 3, at WA's CP-40 C2 T1 head a3df1a3ba2 (owner slot s145, 2026-10-25): the same-name spawn answers typed; js-memory, js-redis and js-postgres still pass. The C2 car's head is proven by its own pipeline | source, live-repro | verified | fixed (1 cells) |
| CP-135 | S2 | A DO session owed its continuation whose every wake fails with a lost-class entry failure (e.g. a run-start commit rejected entry_status_changed) ends failed with a bare framework_auto_resume_exhausted (retryable: true) and no cause: a lost wake is a noop that records no lastFailure, so the crash-loop exhaustion carries nothing of the failure that kept the run from resuming. An operator sees only that it gave up after 5 wakes. | core/src/run/classify-entry-failure.ts:23 and :72-82 (origin/main f7a78e05d1) (a RunStartRejectedError with cause entry_status_changed is classified lost); runtime-cloudflare/src/do/drive/failure-policy.ts:7 and :20 (origin/main f7a78e05d1) (lost -> noop; only transient, :9, records lastFailure) with runtime-cloudflare/src/do/drive/run-drive.ts:1719-1727 (origin/main f7a78e05d1) (the noop branch debug-logs and reports, recording no lastFailure); runtime-cloudflare/src/do/durable-object-agent-base.ts:392-402 and :2544 (origin/main f7a78e05d1) (the exhaustion detail takes its cause only from record?.lastFailure); conformance: wake.paused-submit-lost-bounded (owner batch 38, host tier, cf-do; fault-do-store runStartRejected entry_status_changed) at main f7a78e05d1 plus !386 and owner batches 37 and 38, 3 of 3: .bounded passes (the session ends failed after 5 alarm wakes) and .cause-kept fails, the errorDetail {"message":"Run could not be auto-resumed after 5 wake attempts","code":"framework_auto_resume_exhausted","category":"framework","retryable":true} with no cause; related: the same cell's auto-send submit is answered 409 by the chat route (source http:409), the client's answer refused while the session is still paused; not judged here; severity S2: the run does end (bounded), but its terminal detail hides the failure that ended it and calls it retryable, so an operator or a retrying client gets no actionable cause; runtime-cloudflare/src/do/drive/run-drive.ts and runtime-cloudflare/src/do/drive/failure-policy.ts (fix evidence, !411, 22cb4e513a) (a lost wake refusal that the store disproves carries into the exhaustion's cause); conformance: fixed. wake.paused-submit-lost-bounded passes .cause-kept on cf-do 3 of 3 at owner batch 1f937ca39c (main 22cb4e513a), 2026-10-18 (an unexpected pass of the ledgered failure); W1's gate was red on the red tree (the exhaustion with no cause) | source, live-repro | verified | fixed (1 cells) |
| CP-136 | S1 | The chat route's worker-side reader of a run's DO stream (DOStreamManagerClient.createResumableReader, wire v2) bounds its read-ahead by chunk COUNT, not bytes: its pump keeps reading the DO's /sse until 1024 chunks are unread (READER_BUFFER_HIGH_WATER). When the browser stops reading (a backgrounded tab, a stalled link), the pull-based chat response stops pulling, and the worker holds up to 1024 unread chunks per connection, whatever their size. A chunk is as large as the DO row it came from (a tool_end carries the whole tool result, up to the DO's ~2 MB row limit), so a few dozen large tool results can exceed a 128 MB Workers isolate and kill it, taking every other request on that isolate down with it. CP-26's sibling on the worker hop. Fixed on the CF DO's reader by ebd86ba9df (a byte budget, 8 MiB by default); the initial backlog is still read whole up to 32 MiB. | runtime-cloudflare/src/do/do-clients.ts:490 (origin/main 75f5ccb13f) (READER_BUFFER_HIGH_WATER = 1024, documented as unread chunk slots) with :719-728 (origin/main 75f5ccb13f) (unread() counts slots; releaseSpace resumes the pump at half) and :735-744 (origin/main 75f5ccb13f) (the pump reads the connection until 1024 slots are unread; nothing counts bytes); ai-sdk/src/response/sse-builder.ts:192 and :213 (origin/main 75f5ccb13f) (the chat response is a pull-based ReadableStream: an unread browser stops its pulls, so the reader above fills to its bound and stays there); runtime-js/src/run-loop.ts:3096-3110 (origin/main 75f5ccb13f) (a tool_end chunk carries the full tool result: chunk size is bounded only by the DO store row it is written to); inferred: no cell yet. The bound follows from the source; the isolate's memory under it is not measured. Reported by the reliability orchestrator (2026-10-17) from WE MR C1's reader; verified at source by the owner; related: CP-26 (the same unread client on the DO hop: the DO's own SSE buffer); fix owner WE-2 (likely C2a: a byte budget on the reader's read-ahead); severity S1: by source, one unread browser on a run that streams large chunks can grow the worker isolate past its memory limit, and an isolate OOM kills every in-flight request it hosts, not just the slow one; related: its cf-do cell (stream.unread-client-worker-read-ahead-host) could not reach its judgement in owner slots 2b / c2 (2026-10-18): the run did not complete within 90 s, and neither did the no-hold control stream.large-reply-completes-host, so the bound, not a held reader, was the limit (FU-CONF-LARGE-REPLY-BRIDGE-LATENCY); the bound is now 240 s. Fixed in code by WE-2's C2a (!410, ebd86ba9df); the flip waits for the cell; related: correction (owner slots c3 / c4, 2026-10-18): the cell and its control never completed on cf-do because the reply was one 17.6 MB assistant message, past the DO's row limit: the step commit failed the run (MA-31), and the cells waited only for completed. Not bridge latency (FU-CONF-LARGE-REPLY-BRIDGE-LATENCY withdrawn; js-chat-host streamed it in 381 ms). Both cells now stream the reply over 11 steps of about 1.64 MB (owner batch 43r), and the control completes on both hosts; conformance: the 11-step cell does not discriminate yet (owner slot c4, 2026-10-18, 3 of 3 each, not ledgered): with !410's C2a in the base (f7d4b73f43) it fails its .backpressure-reaches-worker precondition (the worker's upstream reads total 18.4 MB over 5 bodies, 4 of them ending in an error and re-opened, while the held client read 50 B), and with C2a's product diff reversed it passes every id (the worker read less than the reply, within the 12 MiB budget). An evicted body's unparsed tail is sent again by the re-opened one (WE-2), so summing bodies over-counts: the cell now counts each stream sequence's frame once (owner batch 43x), its precondition is .client-held (which an unbounded reader cannot break), and its red is WE-2's mutation (the reader pump never waits on its byte budget); conformance: the distinct-sequence cell (owner batch 43y) does not discriminate either (owner slot c5, 2026-10-18, 3 of 3 each, not ledgered): on the C2a base (f7c059ca09) the worker took in 18 397 524 B of DISTINCT frames (the whole stream: no frame was sent twice) over 3 to 8 bodies, all but the last ending in an error, while the held client had read 50 to 85 B, so .read-ahead-bounded fails; WE-2's unbounded-pump mutation gives the same figure. Either C2a's byte budget does not bound the worker's reader here, or the driver's client hold does not reach the worker (a buffer between them absorbs the stream); with WE-2; related: why it cannot judge (WE-2, 2026-10-18): the worker's reader holds at most its own 8 MiB budget, and the transport between the chat route and the held Node client legitimately absorbs about 10 MB (this Mac's loopback autotunes 4 MiB send and receive buffers, plus workerd's and undici's queues), so the whole 18 MB reply passes through a bounded reader and an unbounded one alike; a larger reply would make the bound a guess about the CI kernel. Orchestrator ruling: the cell is a declared gap on cf-do ('observe-reader-buffer'), re-seated on the reader's own buffered-byte high-water once WE-2's reader seam (after RM-80) logs it at close; conformance: fixed (the byte budget on the reader's read-ahead, runtime-cloudflare/src/do/do-clients.ts:904, origin/main cfed7b8ffa; WE-2's C2a with !430's close report): stream.unread-client-worker-read-ahead-host passes 3 of 3 on cf-do (owner slot s49, 2026-10-19, trial eff5b86592 on main cfed7b8ffa; owner batch 46ap's cell: 2,000 custom events of 16 KiB from one tool, ~32.8 MB with a small transcript, read through the resumable reader its .resumable-reader-ran precondition proves), and fails .read-ahead-bounded 3 of 3 with the budget removed (budget.full() → false): the reader held 31 252 015 to 32 321 459 B against the 8 405 351 B bound. The pre-fix bound (1024 chunk slots, ~16.7 MB of these chunks) is past that bound too; related: not CI-proven yet (owner, 2026-10-20). The cell's first CI run, !464 job 17302397122, never completed the run within .run-completes' 120 s bound (the cell ran 204.9 s, against about 35 s per local run in s49). The same job's no-hold control stream.large-reply-completes-host took 77.2 s for ~18.4 MB (18 s locally), so a slow runner is likely, but a stall (a held client's backpressure reaching the DO's producer) is not ruled out. The cf-do cell is a declared gap until its progress diagnostics decide (FU-CONF-READ-AHEAD-CI-TIMEOUT). The fix rests on the s49 local evidence above (3 of 3 green, red with the budget removed) and on WE-2's unit tests, as it did before the batch; related: fixed in code by ebd86ba9df (WE C2a: reader buffers bounded in bytes) with 327d8395dc, verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-cloudflare/src/do/do-clients.ts:994 (origin/main fb9150cc16) (the pump waits while the budget is full) and :844 (origin/main fb9150cc16) (a ReaderBufferBudget); core/src/stream/reopen-policy.ts:122 and :186-187 (origin/main fb9150cc16) (full at 1024 chunks or 8 MiB). Residual: the initial backlog is read whole, capped at 32 MiB (reopen-policy.ts:133 (origin/main fb9150cc16), FU-C2A-BACKLOG-BEYOND-READER-BUDGET). Open until its cf-do cell (declared a gap, e2e/src/conformance/gaps.ts:966 (origin/main fb9150cc16)); related: fixed in code by ebd86ba9df (WE-2 C2a, !410) with 327d8395dc (!430, the close report), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/do-clients.ts:848 (origin/main 56d564d073) (the resumable reader builds a ReaderBufferBudget), runtime-cloudflare/src/do/do-clients.ts:998 (origin/main 56d564d073) (the pump waits while the budget is full), core/src/stream/reopen-policy.ts:122 (origin/main 56d564d073) (the 8 MiB byte bound, the budget's default), core/src/stream/reopen-policy.ts:187 (origin/main 56d564d073) (full at 1024 chunks or at the byte bound); conformance: calibrated on WE-2's slot w-c136 (2026-11-04, local, the gap lifted in a scratch tree of e0df73fcc7 on main aa4f781a6c): stream.unread-client-worker-read-ahead-host passes on cf-do 3 of 3 (the run ended completed after 63.9, 37.7 and 33.0 s, maxBufferedBytes 8,389,926 each), and with the budget removed (budget.full() to false, rebuilt) it fails only .read-ahead-bounded 3 of 3 (maxBufferedBytes 24,233,395, 24,500,771 and 24,216,223 against the 8,405,351 B bound, about 2.9 times over; the worker read 66,974,932 B). That red peak is lower than s49's 31.25 to 32.32 MB: the cause is not attributed (a shallow search found no second limiter, which does not rule one out); conformance: fixed. In CI, jobs 17761624437, 17762456338 and 17763113921 (test:conformance 1/7, pipeline 3030641083, merge a49f3d0617 of the G1 head that landed as 98fb3973d4): stream.unread-client-worker-read-ahead-host passes on cf-do 3 of 3, the first passes of it in CI: the run ended completed after 79.2, 79.9 and 95.1 s, the cell took 110, 111 and 138 s, inside its 360 s run bound, with no harness unhealthy line. The red stays w-c136's calibration above; its lower peak is still not attributed | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-137 | S1 | store-cloudflare's binding-side resumable reader of a stream DO (durable-stream.ts, the CF Workflows chat path's stream manager) reads the DO's /sse eagerly with no backpressure into a buffer of at most 10000 unread chunks (DEFAULT_MAX_BUFFER_SIZE), and past that it DROPS each further chunk with only a logger.error (and an optional onBufferOverflow callback). The consumer's drain then skips the empty slot, so a slow or paused consumer (a backgrounded tab while a long answer streams word by word) silently loses chunks from the middle of the stream: no gap marker, no error, no resync. Fixed by ebd86ba9df (WE C2a, !410): the reader applies backpressure past its backlog, buffers every chunk, and ends an oversized backlog with StreamTruncatedError; open until its cell is registered. | store-cloudflare/src/durable-stream.ts:65 (origin/main 75f5ccb13f) (DEFAULT_MAX_BUFFER_SIZE = 10000) with :461-472 (origin/main 75f5ccb13f) (the read loop reads the body as fast as it arrives, whatever the consumer has taken); store-cloudflare/src/durable-stream.ts:346-368 (origin/main 75f5ccb13f) (a chunk frame is buffered only while fewer than maxBufferSize are unread; otherwise it is dropped: logger.error("Buffer overflow, dropping chunk") and onBufferOverflow) with :485-497 (origin/main 75f5ccb13f) (drainBuffered skips undefined slots "left by truncation pruning or buffer overflow"); inferred: no cell yet. Reported by the reliability orchestrator (2026-10-17, WE-2's assessment); verified at source by the owner. A turn's text streams one delta per word (core MockLLMAdapter / ScriptedModel split, and real providers' small deltas), so 10000 chunks is one long answer; related: CP-136 (the DO-side reader on the cf-do chat path: bounded by count, not bytes, but it applies backpressure and drops nothing); CP-26 (the DO hop). Fix owner WE-2 (C2a); severity S1: silent data loss. A client reading through this reader is handed a stream with chunks missing from its middle and no sign of it; the transcript a UI renders from the stream is wrong; store-cloudflare/src/durable-stream.ts (fix evidence, !410, ebd86ba9df) (the binding-side reader's buffer is bounded in bytes and a drop re-opens, never a silent drop of chunks); conformance: stream.paused-consumer-every-chunk (owner batch 40h, rewritten in 42q to count from the stream head) fails .every-chunk-once on cfw-workflows-d1 3 of 3 at owner batch 01b2eb6a0c (main 826d9665af, before !410), 2026-10-18 (sequences 1..10504: the reader yields 10001 and is missing 10002 onward), passing on every other lane; and passes on cfw-workflows-d1 3 of 3 at owner batch 1f937ca39c (main 22cb4e513a, with !410), with every other lane; related: the provenBy cell stream.paused-consumer-every-chunk is UNREGISTERED (orchestrator ruling, 2026-10-19), so CP-137 is open again, its fix evidence and observed red / green kept: on the CI runner the cell crashed at its cell end with a binding fault (cfw: binding fault (status instance=…): fetch failed) after running 34-58 s, in !422's jobs 17268393859, 17272345391 and 17273937439 (and a fourth sighting, per the orchestrator), while it passes alone and in-suite locally in about 2.5 s (FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD). It flips again when the cell is re-registered with the cfw lane's Miniflare restart between heavy cells and passes x3 in CI; related: fixed in code by ebd86ba9df (WE C2a), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): store-cloudflare/src/durable-stream.ts:578-583 (origin/main fb9150cc16) (the read loop applies backpressure past the backlog), :446-448 (origin/main fb9150cc16) (every chunk is buffered: the overflow drop and onBufferOverflow are gone) and :427-443 (origin/main fb9150cc16) (a backlog past 32 MiB ends with StreamTruncatedError, never a silent loss). A stale code comment at :665 (origin/main fb9150cc16) still says buffer overflow. Open until its cfw-workflows-d1 cell (not in scenario-manifest.ts yet); related: fixed in code by ebd86ba9df (WE C2a, !410), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): store-cloudflare/src/durable-stream.ts:578-583 (origin/main 56d564d073) (past the backlog the read loop waits while the unread budget is full, so nothing is dropped), store-cloudflare/src/durable-stream.ts:446-448 (origin/main 56d564d073) (every chunk frame is buffered; no overflow drop or onBufferOverflow is left in the file), store-cloudflare/src/durable-stream.ts:424-444 (origin/main 56d564d073) (a backlog over READER_BACKLOG_MAX_BYTES ends with StreamTruncatedError, never a silent loss); a stale comment at store-cloudflare/src/durable-stream.ts:665 (origin/main 56d564d073) still says buffer overflow; conformance: stream.paused-consumer-every-chunk, registered with laneRebuild 'around' (!693, batch !697, main de10dbb892), passes on cfw-workflows-d1 3 of 3 in CI with no binding fault: test:conformance 7/7 job 17746476118 (MR pipeline 3028484662, the cell 80.9 s), job 17747540198 (train pipeline 3028615021, 27.2 s) and job 17748432030 (the MR job retried, 33.8 s), each logging the lane's rebuild before and after the cell. FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD stays open for the fault itself; related: the fault recurred with the rebuild in place (owner, 2026-11-05): !712's train pipeline 3031996667, test:conformance 7/7 jobs 17771786644 (56.9 s) and 17773061477 (44.0 s), each logging the lane's rebuild before the cell, each crashing at endCell on a status read (binding fault, fetch failed). The observation was discarded both times, so .every-chunk-once was not judged: a harness crash, not this finding. The rebuild is disproved as a cure: the lane was fresh, so the fault is not left over from earlier cells. CP-137 stays fixed, and the cell stays registered as its only witness. FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD holds the fault | source, peer-reported | verified | fixed (1 cells) |
| CP-138 | S1 | A run suspended on a client-tool call that no client answers never advances at the call's deadline (clientToolTimeoutMs): nothing fires at the deadline, so the run sits suspended until some later request (a reload, the user's next message) runs the resume-time expiry. The user-reported chat stall (2026-10-17): the browser's send stream dropped, the run then emitted 5 client-tool calls nobody received, and nothing happened at their deadline; only the user's request 22 s later recovered the run. Seen on the CF Durable Object, CF Workflows and runtime-js. | runtime-cloudflare/src/do/durable-object-agent-base.ts:1939-1985 (origin/main 790256dfc7) (registerAlarmSubscribers: the alarm scheduler serves heartbeat (:1943 (origin/main 790256dfc7)), stop-deadline (:1962 (origin/main 790256dfc7)), parent-notice (:1971 (origin/main 790256dfc7)) and child-verify (:1985 (origin/main 790256dfc7)); nothing schedules or serves a client-tool deadline); runtime-js/src/js-agent-executor.ts:2769-2781 (origin/main 790256dfc7) (the resume drain is the only expiry: expireOverdue runs when a resume reaches the drain; its comment's "Mirrors the CF runtime's AlarmScheduler-driven expiry path" names a path the DO does not have); conformance: clienttool.deadline-advances-unattended (owner batch 40g, host tier cf-do + js-chat-host, executor tier every lane) at main 790256dfc7 plus owner batches 37-41 (trial 7d66d06620), 3 of 3 on each lane: on cf-do (host), js-chat-host (host), cfw-workflows-d1, js-memory, js-redis and js-postgres (executor) the run suspends on the call (a 3 s clientToolTimeoutMs) and no second model call starts within 15 s past the deadline, with no request or executor call from outside: .advanced-at-deadline fails, and .call-resolved-as-timeout and .run-completes fail as its preconditions; live: the incident (orchestrator worktree tmp/reliability-investigation/incidents (the 2026-10-17 chat incident file), SigNoz evidence, app on SDK 7.0.0): 5 editBlock calls with deadlineAt 21:36:27.674; nothing at the deadline; client_tool.orphan_recovery_started and 5 client_tool.timeout only at 21:36:57, on the user's next request; conformance: the cfw-workflows-d1 executor row is fixed by WE-2's CP-138 MR (branch reliability-we-cp138-cfw-deadline, head 2980f99be5 over origin/main fb9150cc16; the fix b0d98fb530: a suspension creates a __deadline__ watcher instance that sleeps until the earliest deadline, expires the lapsed call as client_tool_timeout and continues the root once). WE-2's slot: clienttool.deadline-advances-unattended on cfw-workflows-d1 passes with failedIds [] at that head, and with the suspension's watcher create disabled fails exactly .advanced-at-deadline, .call-resolved-as-timeout and .run-completes (peer-reported, WE-2; orchestrator relay 2026-10-24). Unit red: runtime-cloudflare/src/__tests__/client-tool-deadline.test.ts (2e8ebc2340). The js-chat-host, js-* and temporal-* rows stay ledgered (FU-JS-CLIENT-TOOL-DEADLINE-NO-TIMER; Temporal parked); related: the Temporal lanes did not run (no Temporal server in the evidence slot; owner slot 2 re-runs them); DBOS waits on the 5-minute default instead (CP-139). Fix owner WA; severity S1: a run whose client left is stuck indefinitely: the user sees no progress, and the run resumes only if someone happens to send a request; conformance: clienttool.deadline-advances-unattended fails .advanced-at-deadline, .call-resolved-as-timeout and .run-completes on temporal-memory, temporal-redis and temporal-postgres (executor; a private Temporal dev server), 3 of 3 at owner batch 38c21d844e (product code origin/main d23f9afcb9), 2026-10-17: no deadline fires on Temporal either; the Temporal rows are ledgered with the others (Temporal is parked under the Cloudflare-first program); conformance: the cf-do host row is fixed by S-A (!406): clienttool.deadline-advances-unattended passes on cf-do 3 of 3 at owner batch 197c038727 (main 826d9665af, which includes !406), 2026-10-18 (an unexpected pass of the ledgered failure); js-chat-host still fails the three ids 3 of 3 (ledgered); related: the CF Workflows row was fixed by WE-2 (!500, merged as 1965fd8934; see the line above). The js-chat-host, js-* and temporal-* rows stay open (FU-JS-CLIENT-TOOL-DEADLINE-NO-TIMER, out of scope; Temporal parked); runtime-js/src/client-tool-deadline-timer.ts:86-185 (fix: !536, WE-2; as on branch reliability-we2-cp138-js-deadline-timer 96805fc866) (ClientToolDeadlineTimer: one in-process fire per session at its earliest owed deadline; the fire expires the lapsed calls and continues the owed batch, then re-arms) with runtime-js/src/js-agent-executor.ts:1045-1058 (fix: !536, WE-2; as on branch reliability-we2-cp138-js-deadline-timer 96805fc866) (armClientToolDeadline: armed for a ROOT session's suspended run only; a sub-agent session's own client call has no timer, FU-JS-CLIENT-TOOL-DEADLINE-SUBAGENT); conformance: fixed on runtime-js FOR ROOT RUNS by !536 (CP-138's JS half): in !536's MR pipeline 2988006230 at its head 96805fc866 clienttool.deadline-advances-unattended was unexpected-pass on js-memory, js-redis and js-postgres (executor) and js-chat-host (host), so those rows are removed. A local sub-agent suspended on its own client call still has no timer (FU-JS-CLIENT-TOOL-DEADLINE-SUBAGENT, Greptile on !536; no cell drives it yet). The Temporal rows stay (parked). The CF DO and CF Workflows rows were fixed earlier (!406, !500) | source, live-repro, peer-reported | verified | open |
| CP-139 | S3 | DBOS ignores the agent-level client-tool timeout (AgentConfig.clientToolTimeoutMs): its client-tool wait takes only the tool's own clientToolTimeoutMs, else the 5-minute default, where core's step loop (and so runtime-js and both CF runtimes) falls back from the tool to the agent. An agent that sets its deadline once at the agent level gets 5 minutes on DBOS. | runtime-dbos/src/workflows/shared.ts:668 and :761 (origin/main 790256dfc7) (deadlineMs = tool.clientToolTimeoutMs ?? DEFAULT_CLIENT_TOOL_TIMEOUT_MS: no agent fallback); contrast: core/src/orchestration/step-iterator.ts:1669 (origin/main 790256dfc7) (timeoutMs = tool.clientToolTimeoutMs ?? deps.agent.clientToolTimeoutMs ?? undefined); core/src/types/agent.ts:273-275 (origin/main 790256dfc7) (the agent-level option, overridden per tool); conformance: clienttool.deadline-advances-unattended on dbos-postgres (owner batch 40g, trial 7d66d06620 on main 790256dfc7), 3 of 3: with the agent's 3 s clientToolTimeoutMs no second model call starts within 15 s past it (.advanced-at-deadline, and its two dependents, fail): DBOS waits its 5-minute default. Whether DBOS's wait itself advances at a deadline it does honour is not judged here; severity S3: a configuration silently not applied on one runtime: the deadline the app set is not the one enforced, so a stuck client call holds the run for 5 minutes instead | source, live-repro | verified | open |
| CP-140 | S1 | runtime-js's remote sub-agent reconciliation (and the D4 orphan sweep) reads the remote refs from state.subSessionRefs, and a resume that restores a checkpoint rebuilds that list from the checkpoint's state snapshot: a remote ref registered after the checkpoint (a remote call dispatched mid-step) is missing, so reconciliation never sees it and the ref stays running forever, whatever the child's transport says, and the parent awaits it indefinitely. On the CF Durable Object (which runs runtime-js) the snapshot carries no refs at all: they live in their own table. (S-C, WE-2's analysis, first named it the 2026-10-17 chat incident's cause; a stranded-run recovery on the DO does not restore a checkpoint, so the incident's DO path is CP-141's, not this record's.) Fixed on runtime-js and the CF DO by 88b6e983ae: reconciliation and the lost-step sweep read the store's refs. | runtime-js/src/js-agent-executor.ts:2422 (origin/main d23f9afcb9) (reconcileRemoteSubAgents filters the running remote refs out of state.subSessionRefs) with :4878-4885 (origin/main d23f9afcb9) (a checkpoint restore sets state.subSessionRefs from target.state.subSessionRefs, the snapshot) and :4924 (origin/main d23f9afcb9) (reconciliation runs on that restored state); contrast: runtime-js/src/js-agent-executor.ts:2113 (origin/main d23f9afcb9) (a non-restoring resume builds its state from stateStore.getSubSessionRefs: the live refs); runtime-cloudflare/src/do/do-state-store.ts:3145 (origin/main d23f9afcb9) (writeCommitCheckpointSync: the checkpoint state is the session row; DOStateStore never writes subSessionRefs into it, the refs live in sub_session_refs); live: WE-2's red test on we2-s-c-wip, red on main's line and green with the fix: runtime-js remote-orphan-reconciliation.test.ts 'S-C / D4 on a REWIND: an orphan registered after the resume checkpoint is still swept' (a from_checkpoint resume to a checkpoint taken before the orphan's call); related: correction (WE-2, 2026-10-17): the S-C DO integ (do-eviction-recovery.integ.test.ts 'S-C: a stranded parent whose remote child DO never persisted…') witnesses CP-141 and the typed lost-child settlement, NOT this record: a stranded-run recovery on the DO does not rebuild from a checkpoint, so its checkpoint-state-refs mutation stays green there; likewise remote.lost-child-settles-host (cf-do) is CP-141's cell, not this record's; related: WE-2 scanned runtime-js for other post-resume readers of the rebuilt refs and found none (the trailing heal and the run loop read the store); CF Workflows and Temporal read refs from the store (not yet witnessed at cell level). The typed settlement of a child that is found lost is the S-C typed-failure record. Fix owner WE-2 (we2-s-c-wip 42c20fe8dc: reconcile reads stateStore.getSubSessionRefs); severity S1: a parent run waits forever on a child that no longer exists, and nothing short of a manual reset ends it; the user sees a turn that never finishes; related: fixed in code by WE-2's S-C (!412, merged at its tested head ff11cfe40f, main 88b6e983ae; orchestrator, 2026-10-18); the record stays open until a conformance cell witnesses it: a resume from_checkpoint after a mid-step remote dispatch on cf-do (owner to-do); related: fixed in code by 88b6e983ae (S-C), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-js/src/js-agent-executor.ts:2790 (origin/main fb9150cc16) (reconcileRemoteSubAgents reads the store's getSubSessionRefs, not the snapshot's refs) and :2687 (origin/main fb9150cc16) (the lost-step sweep reads the store too, before reconcile at :5700 / :5704 (origin/main fb9150cc16)); CF Workflows loads refs from the store (runtime-cloudflare/src/steps.ts:982 (origin/main fb9150cc16)). Open until its cell; related: fixed in code by 88b6e983ae (WE-2's S-C, !412), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/js-agent-executor.ts:2799-2812 (origin/main 56d564d073) (reconcileRemoteSubAgents filters the owned remote refs from the store's getSubSessionRefs, never the checkpoint-restored state.subSessionRefs), runtime-js/src/js-agent-executor.ts:2707 (origin/main 56d564d073) (the lost-step sweep reads the store's refs too); conformance: fixed. remote.rewind-resume-reconciles-ref-host (host tier, cf-do): a root parked on a remote child (CP-157, one hop) is stopped with its run_stopped pairing refused (fault-do-store appendFailing), the child finishes on its own DO, and a from_checkpoint resume to the root's suspension commit (named precondition .checkpoint-holds-call: its messageCount covers the call's row, with no result row in its prefix) pairs the call with the child's real result right after its tool_use and settles the ref completed: 3 of 3 identical on cf-do|host (WA slot slot-rmb v3, lock A, 2026-11-04, tree fc24e8d68a) and 1 of 1 at dacb3dd2e5 (slot-cp140c). Red with both 88b6e983ae's rule and CP-157 L1's re-attach off (WA slot slot-cp140c, lock A, 2026-11-04, tree dacb3dd2e5): .real-result-paired, .ref-completed and .history-paired fail (the call left unpaired, the ref left paused_awaiting_client with its stopOwed). Green with 88b6e983ae's rule alone restored (reconcile filters the restored snapshot's refs, empty on the DO; slot-rmb v3, tree fc24e8d68a): that single-rule run is recorded as masked by L1 (below); L1 disabled alone was not run; related: the pre-fix rule is masked today on the reachable shapes: the orphan (its call rewound away) is settled by the D3 lost-step sweep, which reads the store (CP-194, 3a55689d5d; runtime-js/src/js-agent-executor.ts:3479-3487 (origin/main fcb82e788e)), and a parked call by CP-157 L1's re-attach, driven by the stored session's suspendedAwaitingChildren (runtime-js/src/run-loop.ts:1405-1548 (origin/main fcb82e788e)). Reconcile's store read (88b6e983ae) is the remaining defence; no reachable shape was found where it alone decides; related: no checkpoint snapshot carries refs except on the memory store: subSessionRefs is an AgentState field (core/src/types/state.ts:441 (origin/main fcb82e788e); SessionState has none), and a commit checkpoint snapshots the stored SessionState (core/src/store/commit-checkpoint.ts:32 (origin/main fcb82e788e); the CF DO's at runtime-cloudflare/src/do/do-state-store.ts:4153 (origin/main fcb82e788e), the session row, whose refs live in sub_session_refs). Only the memory store, which keeps the whole saved object, can carry them. So the divergence is any durable store's, not only the DO's (on the js lanes: js-postgres, js-redis); there is no js cell because the cell's entry state (a stopped parked parent whose call stays unpaired) needs fault-do-store appendFailing, a cf-do control: a runtime-js stop pairs the call (CP-118), and the observable there is covered by the same D3 and L1 paths | source, peer-reported | verified | fixed (1 cells) |
| CP-141 | S1 | The CF Durable Object's store (DOStateStore) has no column for a sub-session ref's remote stream identity (SubSessionRef.remote): every remote ref it holds reads back without remote, where D1, Postgres and Redis keep it. Every runtime-js path that keys on ref.remote is then blind on the DO: reconciliation never sees a DO remote ref (a lost child is never settled and a D4 orphan never swept, even with CP-140 fixed); the dispatch's re-attach guard and lastSequence resume never fire (a recovered DO parent re-POSTs /start and re-reads its child from 0); and resume's and retry's trailing heal, which spares only a running remote call whose ref carries remote, records a still-running DO child's call as not executed, so the child's real outcome never reaches the transcript. The store contract case that would catch it is skipped for the DO, a skip that looked like a design boundary. | runtime-cloudflare/src/do/do-state-store.ts:2007-2016 (origin/main d23f9afcb9) (addSubSessionRefs inserts parent_session_id, sub_session_id, agent_type, parent_tool_call_id, status, started_at, mode, name, completion_delivered, parent_run_id: no remote column) with :634 (origin/main d23f9afcb9) (the ref keeps only location, local | remote); runtime-cloudflare/src/__tests__/integration/do-state-store-contract.integ.test.ts:54 (origin/main d23f9afcb9) (the store contract skip: remoteStreamMetadata, DOStateStore does not persist SubSessionRef.remote (no column)) with core/src/testing/sub-session-operations.ts:339-345 (origin/main d23f9afcb9) (the contract checks ref.remote only when the store does not skip it); runtime-js/src/js-agent-executor.ts:2422 (origin/main d23f9afcb9) (reconcileRemoteSubAgents keeps only refs with ref.remote) with core/src/orchestration/remote-sub-agent-dispatch.ts:429-433 (origin/main d23f9afcb9) (the re-attach guard and the lastSequence resume read existingRef.remote); runtime-js/src/js-agent-executor.ts:4613-4621 (origin/main d23f9afcb9) (resume's trailing heal excludes a call only when its ref is running AND has ref.remote) with :5424-5430 (origin/main d23f9afcb9) (retry's heal, the same filter over getSubSessionRefs): on the DO a running remote child's trailing call gets a not_executed synthetic result (owner's store-contract skip audit, 2026-10-17, source only); runtime-js/src/execution/remote-subagent-executor.ts:437-451 (origin/main d23f9afcb9) (executeSingleRemoteSubAgentCall's D-2 guard: when the getStatus pre-check failed without a definitive answer, :420-435, it re-attaches only if the ref's remote.streamId is set, else it POSTs /start again; on the DO it is never set, so a re-entered call whose pre-check fails transiently can start a child that already ran, a narrow window; source only); contrast: store-cloudflare's D1StateStore and store-postgres persist it (remote_json), store-redis as its remote field (orchestrator / WE-2's survey, 2026-10-17); live: WE-2's S-C DO integ on we2-s-c-wip (do-eviction-recovery.integ.test.ts 'S-C: a stranded parent whose remote child DO never persisted settles the call typed…'): with CP-140's live-refs fix in, the parent completes but the remote call gets no result row: reconciliation still never saw the ref; contrast: stop propagation is unaffected on the DO: its stop cascade is the drive's cascadeStop -> host.stopChildren (runtime-cloudflare/src/do/drive/settle.ts:435-457 (origin/main d23f9afcb9) and drive/stop-children.ts, origin/main d23f9afcb9), keyed on the ref's mode, not on ref.remote; runtime-js's in-process cascade (runtime-js/src/execution/in-process-runs.ts:278 (pre-fix, origin/main 826d9665af), :292 (pre-fix, origin/main 826d9665af)) keys on ref.remote but never runs on the DO (no in-process run registry is passed there: run-loop.ts:5754 (pre-fix, origin/main 826d9665af) returns at once), and every store it runs over persists remote. WE-2's do-companion-stop.integ.test.ts ('two live ephemeral (remote) children') stops both; related: CP-140 (the same blindness from the checkpoint-restored list; the two flip together when the second lands). Fix owner WE-2 (the S-C MR: a DO-SQLite remote_json column, and the contract skip removed); severity S1: on the DO a lost remote child is never settled (the parent awaits it forever), a recovered parent restarts its child from scratch (a duplicate run), and a resumed or retried parent drops a running child's real outcome for a not-executed result; conformance: store.sub-session-ref-keeps-remote (owner batch 41k, the store tier: the store contract's remote round-trip, core subSessionRefCases, never skipped) fails .remote-round-trips on store-do-sqlite 3 of 3 at owner batch 42a 5e02699d49 (product code origin/main d23f9afcb9), 2026-10-17: the remote ref read back without remote (expected { streamId: 'child-stream', … }, got undefined). It passes on store-memory, store-redis and store-d1, 3 of 3 (store-postgres pending the lane's table-prefix fix, owner batch 42f); conformance: remote.lost-child-settles-host (owner batch 41g; the child's /start held, the host evicted, the parent woken) fails .ref-failed and .call-result-typed on cf-do (host), 3 of 3 at owner batch 38c21d844e (product code origin/main d23f9afcb9), 2026-10-17: the parent completed, but the lost child's ref still reads running and the call has no result row: reconciliation never saw the ref (it has no remote on the DO). WE-2: its S-C DO integ witnesses this record and the typed lost-child settlement, not CP-140; related: remote.lost-child-settles-host's .call-result-typed was wrong for its shape (WE-2 and the orchestrator, 2026-10-18): the step evicted mid-dispatch committed nothing, its call included, so no result row for it may exist (RM-29); it is restated as .uncommitted-call-no-row and .history-paired (owner batch 43p, not ledgered), and the typed row for a lost child moves to a cell with a committed call. .ref-failed stays the witness: a remote ref with no ref.remote yet (its /start never answered) is skipped by the reconcile, which keys on ref.remote (WE-2, fixed in S-C by keying on location === 'remote'); conformance: remote.lost-child-settles-host (restated, owner batch 43q) fails .ref-failed on cf-do 3 of 3 at owner batch f7d4b73f43 (main 22cb4e513a), 2026-10-18 (owner slot c4); its .uncommitted-call-no-row, .history-paired and .parent-completes pass. On WE-2's S-C tree (49506b2f1d, reconcile keyed on location === 'remote', merged onto the same base) every id passes 3 of 3, and remote.step-boundary-no-foreign-read-host passes too; conformance: fixed by WE-2's S-C (!412, merged at its tested head ff11cfe40f, main 88b6e983ae): on the owner's next-batch head (this batch plus later record commits; these cells unchanged) merged with main 88b6e983ae, remote.lost-child-settles-host passes every id on cf-do and store.sub-session-ref-keeps-remote passes on store-do-sqlite, 3 of 3 each (owner slot c10, 2026-10-18); before it both failed 3 of 3 (its former ledger); related: remote.lost-child-settles-host is retired (owner, 2026-11-04; orchestrator rulings of 2026-11-03): G1 (!673) commits the delegation before the call starts, so a child whose start was never confirmed is owed, not lost, and its .ref-failed and .uncommitted-call-no-row no longer describe the shape. Its six ids are never reused. remote.lost-child-started-once-host is the re-proof; a ref with stream identity whose child is not found starts nothing (runtime-js delegation-commit-reattach.test.ts R6b); conformance: remote.lost-child-settles-host at the G1 batch 0343342300, landed as 98fb3973d4 (owner slot s276, 2026-11-04, cf-do), 3 of 3 identical: .uncommitted-call-no-row fails (the call is committed, and it has a result row); its other five ids pass, .ref-failed included. remote.lost-child-started-once-host replaces it; conformance: remote.lost-child-started-once-host, the re-proof: it passes every id on cf-do 3 of 3 at 58b2e07907 on the G1 batch, landed as 98fb3973d4 (owner slot s277, 2026-11-04): one child session, the ref's, with one model call, the call's row holding the child's real output, the ref completed, history paired, the parent completed. Without G1 (main de10dbb892 plus the cell, same slot) it fails exactly .delegation-committed, .one-child, .call-has-real-output and .ref-completed 3 of 3: the reply is not committed before the child starts, no child runs, and the ref reads failed. The old cell's .ref-failed still passed on the batch only because it scripted no child, which then failed; conformance: in CI, jobs 17761624437, 17762456338 and 17763113921 (test:conformance 1/7, pipeline 3030641083, merge a49f3d0617 of the G1 head that landed as 98fb3973d4): remote.lost-child-started-once-host passes 3 of 3 | source, peer-reported | verified | fixed (2 cells) |
| CP-142 | S2 | On the CF Durable Object, re-spawning a persistent companion whose previous child FAILED or was TERMINATED does not start it fresh: it runs a new turn on the old child's session, with its whole history and custom state. Core's re-spawn deletes the old child session first, but the DO's store refuses deletes by design (framework_not_supported), and the dispatcher swallows that refusal at warn ('the child session is reused') and goes on to spawn under the same deterministic child session id. The DO's spawn then POSTs /start to the child's Durable Object, whose session still exists, so the new spawn message is a new turn of the failed conversation. CLAUDE.md promises that 'failed/terminated children still re-spawn fresh'. | core/src/orchestration/companion-tool-dispatch.ts:639-662 (origin/main d23f9afcb9) (a failed / terminated ref, or a runtime without continue support, takes the delete-and-respawn branch: deleteSession(childSessionId), and a framework_not_supported refusal is tolerated and logged at warn, 'Re-spawn deleteSession refused by the store; the child session is reused', then the ref is set running again) with :621 (origin/main d23f9afcb9) (the child session id is generatePersistentSubSessionId(parentSessionId, childName): the same id for every spawn of that name); runtime-cloudflare/src/do/do-state-store.ts:3350-3361 (origin/main d23f9afcb9) (DOStateStore.deleteSession always throws framework_not_supported: each session is its own Durable Object); runtime-cloudflare/src/do/persistent-companion-tools.ts:979-1030 (origin/main d23f9afcb9) (spawnPersistentChild POSTs /start to the child DO named by that same id) with :743-762 (origin/main d23f9afcb9) (childStartBody: sessionId, agentType, message, parent and root ids; no fresh-session or reset field); related: CLAUDE.md, the RM-29 companion continuation paragraph: a COMPLETED companion is re-consulted by continuing its preserved session, but 'failed/terminated children still re-spawn fresh'. On the DO only the first half holds; related: fix direction (orchestrator, 2026-10-17): the DO refuses deleteSession by design, so a re-spawn should get a FRESH child session id (a new Durable Object), not a delete, and the swallowed refusal should go; the old child's ref and session stay as history; contrast: every other runtime runs this branch over a store whose deleteSession succeeds (memory, Redis, Postgres, D1), so the re-spawned child starts with an empty session; severity S2: the re-spawned companion's model sees the failed conversation (and possibly the failure's own output) and reads the old custom state, so the re-spawn's result can depend on the failure it was meant to leave behind; nothing is lost and no error is raised; related: found by the owner's store-contract skip audit (2026-10-17, the DO deleteSession skip, source only; not observed); conformance: companion.respawn-after-failure-fresh-host (owner batch 42l; a non-blocking companion writes a note into its state, then fails; the parent re-spawns it by name) fails .fresh-history and .fresh-state on cf-do (host), 3 of 3 at owner batch 7be28c1611 (main 22cb4e513a), 2026-10-18: the re-spawned child runs on the failed session's own id, its first model call's user messages are ['FIRST-SPAWN-WORK', 'SECOND-SPAWN-WORK'] and its state still holds 'OLD-CHILD-STATE'. It passes on js-chat-host (the store deletes the failed session first: the contrast); related: fix direction, approved (WE-2, 2026-10-18): the DO child wipes its own session in place on a re-spawn's freshSession start (its stream identity and usage reset), guarded parent-only with a typed refusal; companion.respawn-after-failure-fresh-host stays the witness. DBOS reaches the same outcome by a different cause (CP-149); conformance: fixed by !419 (merged at its tested head 7c09bffb69, main squash 2e680f2c15): on this batch's head 8e05dd2524 merged with main 2e680f2c15, companion.respawn-after-failure-fresh-host passes every id on cf-do 3 of 3 (owner slot c7, 2026-10-18); before it, the same cell failed on cf-do 3 of 3 (its former ledger, owner slots rs and c4) | source, live-repro | verified | fixed (1 cells) |
| CP-143 | S1 | On Temporal, a chat message sent while the session is suspended on client-tool calls is refused for good: the user is stuck. Temporal does not declare supportsAbandonOnResume, so handleChatStream takes the per-call path: it submits client_tool_abandoned for every pending call, and Temporal's submit only stores the answer (the batch-completing call answers caller_resumes). The handler then calls execute(), whose paused-with-pending guard still sees the answered entries in pendingClientToolCalls and throws AgentAlreadyRunningError, so the send gets 409 state_already_running. Nothing ever drains those answered entries: the message is not saved, the session stays paused, every re-send gets the same 409, and the client's useHelixChat().retry() re-anchors on the still-paused snapshot instead of re-sending. | ai-sdk/src/handler/handle-chat-stream.ts:772-800 (origin/main d23f9afcb9) (path 4: only an executor that declares supportsAbandonOnResume gets the single resume({ mode: with_message, abandonPending }) entry; any other submits per call via abandonPendingTools, then executes) with :1741-1770 (origin/main d23f9afcb9) (abandonPendingTools: one client_tool_abandoned submit per pending entry; the submits' continuation is not acted on); runtime-temporal/src/executor.ts:2614-2617 (origin/main d23f9afcb9) (submitToolResult's startContinuation is caller_resumes: Temporal stores only, nothing resumes the run) with :524-539 (origin/main d23f9afcb9) (execute()'s paused-with-pending guard: a paused session with any pendingClientToolCalls entry, answered or not, throws AgentAlreadyRunningError); related: FU-TEMPORAL-ABANDON-SEND-STUCK (docs/dev/follow-ups.md; fix direction: implement ResumeOptionsWithMessage.abandonPending on Temporal and declare supportsAbandonOnResume, as runtime-js, the CF DO and CF Workflows do since W1 MR2b, !344). Before WA's S-B (branch reliability-cf-wa-chat-send-no-presubmit) the end state was the same after an extra 5 s poll; contrast: runtime-js, the CF DO and CF Workflows declare supportsAbandonOnResume, so the send is one resume entry that abandons the calls inside its run-start commit (CP-54, fixed for a run paused on client calls alone); related: found by WA while writing S-B's Temporal lane cell (peer-reported, 2026-10-17); witness to come: S-B's chat-send-over-pending-temporal.integ, an it.fails written against the correct behaviour (the calls drain and the re-send succeeds), not yet run; related: WA's check of the other runtimes (2026-10-17, peer-reported, WA's local note answered-pending-window.md): runtime-js (js-agent-executor.ts:3833-3851), the CF DO (its /start runs runtime-js execute(), 409 state_already_running) and CF Workflows (executor.ts:2857-2867) refuse execute() over a session whose pending calls are ALL answered with the same guard, but none is stuck: resume() drains the answered entries (runtime-js / DO drainOrphanClientToolCalls; CFW applyResultsAndReload), and their chat send is the one with_message / abandonPending entry. Low, a misleading refusal of an entry the CP-104 contract says not to use (caller_resumes means resume()); no record of its own. A shared fix, execute() draining a fully answered batch, would close this record's stuck path and that refusal together; severity S1: a user message is refused for good and the session is wedged paused; only an interrupt() or a submit by hand gets it out; related: parked with the Temporal items under the Cloudflare-first program | source, peer-reported | partly-inferred | open |
| CP-144 | S3 | A CF Durable Object agent with an ordinary (local) sub-agent and no subAgentNamespace is not refused: every call of that sub-agent fails as a tool error saying the child's session was not found (state_session_not_found). Persistent agents without a namespace are refused typed up front (framework_not_supported); local sub-agents are passed through unchanged, and runtime-js then starts the child in the PARENT's Durable Object store, which holds one session. Its sessionExists(childSessionId) answers true because DOStateStore.exists ignores the id it is given, so the child's createSession is skipped, and the child's run-start plan then finds no session under the child's id. Nothing is written to the parent's session (every commit path the child reaches checks the stored session id). Fixed on the CF DO by 5af6e15645 (a typed refusal at materialization, and exists compares the id). | runtime-cloudflare/src/do/durable-object-agent-base.ts:2973-2974 (origin/main d23f9afcb9) (rewriteSubAgentTools returns the agent unchanged when there is no subAgentNamespace, so its local sub-agent tools stay in-process) with runtime-cloudflare/src/do/drive/materialize.ts:104-114 (origin/main d23f9afcb9) (only persistentAgents are refused typed without the namespace); runtime-cloudflare/src/do/do-state-store.ts:3275-3280 (origin/main d23f9afcb9) (exists(_sessionId) ignores its argument: true once the object holds any session); runtime-js/src/run-loop.ts:3389-3436 (origin/main d23f9afcb9) (the local child start: sessionExists(childSessionId) on the parent's store, createSession only when false, then commitRunStart whose plan reads loadLLMHistory and sessionStatusOf for the child) with runtime-js/src/run-start.ts:734-743 (origin/main d23f9afcb9) (sessionStatusOf: loadState null → sessionNotFoundError, state_session_not_found) and runtime-cloudflare/src/do/do-state-store.ts:3670-3677 (origin/main d23f9afcb9) (loadState answers null for another session id, warn 'sessionId mismatch'); runtime-js/src/run-loop.ts:3445-3470 (origin/main d23f9afcb9) (the catch: subagent.dispatch_failed at error, then subagent_end and an error tool_end; the model gets a tool error); contrast: runtime-cloudflare/src/do/do-state-store.ts:2975-2980 (origin/main d23f9afcb9) (requireStateRowSync: the run-start, step and state commits refuse a session id other than the stored one, so no row lands in the parent's session); related: fix direction (orchestrator, 2026-10-17): this is an unsupported configuration (a DO store holds one session, so an in-process child has nowhere to live), so refuse it typed (framework_not_supported) at definition or dispatch time, as persistent agents are; and make DOStateStore.exists compare the id it is given, as loadState does; severity S3: a configuration error surfaces late and mislabelled (a not-found session, per call, instead of a refusal at start); nothing is written to the wrong session and nothing is lost; related: found by the owner's store-contract skip audit (2026-10-17, source only; not run). Replaces the follow-ups FU-DO-STATE-STORE-EXISTS-IGNORES-ID and FU-DO-LOCAL-SUBAGENT-NO-NAMESPACE-UNTYPED (orchestrator ruling: a product path, so a record); related: fixed in code by 5af6e15645 (CP-10 + CP-144), verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24): runtime-cloudflare/src/do/drive/materialize.ts:117-128 (origin/main fb9150cc16) (step 2b: a DO agent with any sub-agent tool and no subAgentNamespace is refused framework_not_supported, non-retryable, before any write); CP-188 (withdrawn) is its duplicate. Open until its cf-do cell; related: fixed in code by 5af6e15645 (CP-10 + CP-144), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/drive/materialize.ts:117-128 (origin/main 56d564d073) (step 2b: a DO agent with any sub-agent tool and no subAgentNamespace is refused framework_not_supported, non-retryable, before any write), runtime-cloudflare/src/do/do-state-store.ts:3851-3857 (origin/main 56d564d073) (exists compares the stored session_id with the id it is given) | source | partly-inferred | open |
| CP-145 | S1 | CF Workflows: a step that co-emits a client-tool call (or another phase-1 suspension) with sub-agent calls suspends before ANY of them is dispatched, and nothing dispatches them later. The phase-1 suspension commits and returns before the step commit and before both the local sub-agent registration / spawn / fold and the remote registration / dispatch, so no co-emitted sub-agent ref is registered and no child starts; after the client answers, the resume re-spawns only children recorded as awaited (which the phase-1 commit never sets), and the deferred-call path runs only companion / finishWith calls (and none at all when any trailing call is of another kind). A continue or with-message resume of the paused session then sends the model the sub-agent tool_use unpaired, which a pairing-enforcing provider rejects, and once an assistant row follows the gap sits mid-history. | runtime-cloudflare/src/workflow.ts:3864-3916 (origin/main d23f9afcb9) (phase-1 pending HITL calls: the suspended status, commitSuspendedStep, suspendRun, then return with status suspended_client_tool / suspended_step_partial); runtime-cloudflare/src/workflow.ts:4754-4768 (origin/main d23f9afcb9) (the remote sub-agent dispatch: remoteSubAgentCalls, register-remote-subsessions, after the return above) with :3655 (origin/main d23f9afcb9) (remoteSubAgentCalls is split out of the same step's pendingToolCalls) and :4748 (origin/main d23f9afcb9) (the local sub-agent fold, also after the return); runtime-cloudflare/src/workflow.ts:4595-4640 (origin/main d23f9afcb9) (the cascade-up suspension for suspended children also returns early; whether a remote call co-emitted there is dispatched first is not checked); inferred: on resume after the client's answer, nothing re-dispatches the step's remaining calls (no deferred dispatch for sub-agent or remote calls, only for finishWith, :4999-5009 (origin/main d23f9afcb9)); the next continuation's C5 heal pairs them not_executed (steps.ts:1048 (pre-fix, origin/main 968e106dcf)); related: found by WE-2's cross-runtime RM-29 order check (an explore read, peer-reported, 2026-10-17, unverified: no reproducing test); the suspension's early return re-read here at origin/main d23f9afcb9. The DO (runtime-js run loop) is not affected per WE-2; runtime-cloudflare/src/workflow.ts:3963-4016 (origin/main 0392f3d05b) (the phase-1 suspension: commit and return) before :4026 (pre-fix, origin/main ca98f9ed47) (the step commit), :4044-4778 (pre-fix, origin/main ca98f9ed47) (the local sub-agent registration, spawn and fold) and :4854-5017 (pre-fix, origin/main ca98f9ed47) (the remote registration, :4867 (pre-fix, origin/main ca98f9ed47), and dispatch): no co-emitted sub-agent call is registered or started (WE-2, re-checked at 0392f3d05b, 2026-10-18); runtime-cloudflare/src/steps.ts:7387-7439 (origin/main 0392f3d05b) (the resume re-spawns only childrenToRespawn, from suspendedAwaitingChildren, which the phase-1 commit never sets) with :6124-6152 (pre-fix, origin/main ca98f9ed47) (the deferred-call kinds are companion and finishWith only) and core/src/orchestration/transcript.ts:219-221 (origin/main 0392f3d05b) (findDeferredCalls returns nothing when any trailing call has another kind, so a co-emitted finishWith / companion is not run either); runtime-cloudflare/src/steps.ts:1283-1287 (origin/main 0392f3d05b) (the resume heal runs only from interrupted, so a paused session's continue / with_message resume pairs nothing) with :1810 (pre-fix, origin/main ca98f9ed47) and :1989 (pre-fix, origin/main ca98f9ed47) (the next LLM step builds its messages with the unpaired tool_use) and core/src/orchestration/transcript.ts:116-118 (origin/main 0392f3d05b) (once an assistant row follows, the gap is mid-history, which no trailing heal can repair); severity S1 (orchestrator, 2026-10-18): requested sub-agent work silently never runs, and the transcript the next model call gets is invalid (RM-29 class), so the run fails on a provider that enforces pairing. WE-2 is writing a CFW witness (red first) that pins the run-level outcome; live: WE-2's CFW witness runtime-cloudflare/src/__tests__/integration/cp145-phase1-subagent-workflow.integ.test.ts (ae543c7a2d on reliability-we-rm80-cp145, base main 0392f3d05b; runAgentWorkflow over a real Miniflare D1 with a mocked step and child binding), red 2 of 2: one LLM step emits a client-tool call and a sub-agent call; local sub-agent: 'the sub-agent child was started before the suspension: expected [] to have a length of 1' (the run ended suspended_client_tool with only the client call pending, the child never created); remote sub-agent: 'expected transport.start to be called 1 times, but got 0 times'. Asserted parity shape (orchestrator ruling D-b, as runtime-js and the CF DO): the child started, its real result committed, the run suspended mixed (suspended_step_partial) with the client call still pending (peer-reported, 2026-10-18); conformance: transcript.co-emitted-subagent-dispatched (owner batch 43n: one step emits a client-tool call and a local sub-agent call; the client answers at the pause) fails .child-ran, .child-result-committed and .history-paired on cfw-workflows-d1 (executor) 3 of 3 at owner batch f7d4b73f43 (main 22cb4e513a), 2026-10-18 (owner slot c4): the turn completed with no sub-session ref and the sub-agent call unpaired; it passes on js-memory, js-redis, js-postgres and the three Temporal lanes. On !418 (847784ce9d merged onto the same base) .child-ran passes, and the run then failed on CF Workflows' 256-character step-name limit (the sub-agent wait step's name joins both per-cell agent names and the call id); the cell's call ids are shortened (owner batch 43u) and its fix-tree run is repeated; conformance: with its call ids shortened (owner batch 43u), the cell fails the same three ids on cfw-workflows-d1 3 of 3 at owner batch f7c059ca09 (main 22cb4e513a) and passes every id on !418 (847784ce9d merged onto the same base) 3 of 3, 2026-10-18 (owner slot c5): the child ran, its real result was committed and the history paired; the other executor lanes pass on both; conformance: fixed by WE-2's !418 (merged at its tested head 847784ce9d, main 976e3ec25f): transcript.co-emitted-subagent-dispatched passes every id on cfw-workflows-d1 3 of 3 on that head merged onto owner batch f7c059ca09 (owner slot c5, 2026-10-18); before it, the same cell failed .child-ran, .child-result-committed and .history-paired 3 of 3 on the base (the ledger above, owner slots c4 and c5) | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-146 | S1 | On the CF Durable Object, the run loop reads a REMOTE sub-agent child's state through the parent's own store at every step boundary while the child runs: runtime-js's ephemeral / remote completion injection loads the child session by id, but a DO's store holds only the parent's session, so each read logs a 'sessionId mismatch' warn and answers null. For a running child the null only skips the injection (remote completion is reconciliation's job); for a child whose ref is already terminal while its call is still unpaired at a step boundary, the null would make the injection pair the call with an empty success or a generic 'Sub-agent failed' instead of the child's real outcome (witnessed by WE-2's unit tests, below). | runtime-js/src/run-loop.ts:925-927 (origin/main 826d9665af) (the step-boundary completion injection selects ephemeral AND remote refs) with :937 (pre-fix, origin/main 826d9665af) (a running ref: loadState(ref.subSessionId), childStatus undefined on a null, :946 (pre-fix, origin/main 826d9665af) continue) and :942-943 (pre-fix, origin/main 826d9665af) (a terminal ref: the same read for its output / error, then :949-966 (pre-fix, origin/main 826d9665af) inject a result for a call with no tool_result); runtime-cloudflare/src/do/do-state-store.ts:3670-3677 (origin/main 826d9665af) (loadState answers null for another session id, warn 'sessionId mismatch in DOStateStore.loadState'); conformance: remote.step-boundary-no-foreign-read-host (owner batch 41g; a turn that runs a remote sub-agent to completion, the parent DO's host logs observed) fails .no-foreign-session-read on cf-do (host), 3 of 3 at owner batch 38c21d844e (product code origin/main d23f9afcb9), 2026-10-17: the parent DO logged a 'sessionId mismatch' warn; the turn itself completes. js-chat-host is a declared gap (in-process sub-agents); related: WE-2's analysis (peer-reported, 2026-10-18): of the four foreign child-state reads in runtime-js and core, only this one runs on the DO. runtime-js's persistent-companion path (run-loop.ts:1152-1184 (pre-fix, origin/main 826d9665af)) reads the store only when the companion notifier answers undefined, which the DO's never does; core record-sub-session-result.ts:412 already skips remote refs; core completion-delivery.ts:64 (pre-fix, origin/main cfed7b8ffa) / :76 (pre-fix, origin/main cfed7b8ffa) (DBOS and Temporal only) has no remote skip, unverified whether a remote persistent companion reaches it there (parked). Fix: WE-2's S-C MR (!412), dbcf965af1 (the injection's filter skips refs with remote) with b61075d766 (CP-141: the DO store keeps remote, without which that filter never fires on the DO); unit witness runtime-js/src/__tests__/js-agent-executor-resume.test.ts:2745 on !412; live: WE-2's red tests on we2-s-c-wip (peer-reported, 2026-10-18): runtime-js remote-orphan-reconciliation.test.ts, the four 'CP-146: …' tests, red at 71c14f4871 (before the fix) and green at dea01a6583: a remote ref already completed / failed whose call never got its result is, on main, paired by the DO's step-boundary injection with result null (the child's real output replaced). DO integ witnesses do-eviction-recovery.integ.test.ts 'CP-146: a resume pairs a remote call whose ref is already COMPLETED with the child's REAL output…' (da52c5715a) and 'CP-146: a DO parent that runs past a remote child's call never reads the child from its own store…' (38f6357d85), their red proofs in WE-2's next slot; severity S1 (raised from S3 once the :942 case was witnessed red, the orchestrator's ruling): the parent's model silently receives an empty success (or a generic failure) in place of a remote child's real result, the CP-95 class of a silently wrong result reaching the consumer; the warn and the foreign read are the visible symptom. Fix: S-C (!412), dea01a6583 (reconcile and the resume / retry heal own a trailing unpaired remote call whatever its ref's status; the child's real outcome from transport.getStatus, a gone child settles typed framework_subagent_lost); inferred: the reachability of the :942 shape is unverified (WE-2): likely only through a crash between the remote ref turning terminal and the call's result being appended. Its witnesses (WE-2's unit tests and DO integ) seed that state; no natural path to it has been observed; conformance: fixed by WE-2's S-C (!412, merged at its tested head ff11cfe40f, main 88b6e983ae): on the owner's next-batch head merged with main 88b6e983ae, remote.step-boundary-no-foreign-read-host passes on cf-do 3 of 3 (owner slot c10, 2026-10-18); before it, it failed 3 of 3 (its former ledger) | source, live-repro, peer-reported | partly-inferred | fixed (1 cells) |
| CP-147 | S2 | The CF Durable Object ends an announced-terminal stopped run (R77) without clearing its pending client-tool calls, breaking the stop contract every other path keeps. Its interrupted end goes through endSessionWithoutRun, which commits the session with its pending map as it is, where core commitStopTerminal clears it. A later continue resume then answers the call runtime_restarted (a fiction: the user stopped it) where the contract pairs it not_executed, and a late submit for it is not answered run_stopped. | runtime-cloudflare/src/do/drive/announced-terminal.ts:223-227 (origin/main 826d9665af) (the DO's R77 announced-terminal end: endSessionWithoutRun with status interrupted; the session's pendingClientToolCalls are kept); contrast: core/src/run/commit-stop-terminal.ts:108 (origin/main 826d9665af) (commitStopTerminal's stopped session sets pendingClientToolCalls: undefined) and runtime-cloudflare/src/steps.ts:5407 (origin/main 826d9665af) (CF Workflows' commitTerminalState clears it on a stop resolution); Temporal and DBOS clear it too (W1, peer-reported); related: core/src/client-tool/stopped-submit.ts:20-24 (origin/main 826d9665af) (the stop contract, case (b): the stop's K5 cleared the pending call, so a late submit answers run_stopped); WA MR2 Ruling 56 (a stopped run's pending client calls stay unpaired until the next turn's heal, which pairs them not_executed); related: runtime-js's in-loop stop K5 (runtime-js/src/run-loop.ts:4536 (origin/main 826d9665af) at 826d9665af, commitTerminalState spreads ...current) also keeps whatever pending map it loads, but W1 found no reachable path where it holds one: a stop of a PAUSED run ends through endEntryRunByStop → commitStopTerminal (the map empty after; a continue resume pairs the call not_executed), and a stop that lands in-step never reached the suspension commit that writes pending entries (W1's two single-file probes, peer-reported, 2026-10-18). W1's fix adds an invariant comment there; the only reachable interrupted-with-pending shape is the DO's R77 end; related: found by W1 (peer-reported, 2026-10-18, by reading and the probes above); fix owner W1, its (ii) MR after RM-81 (red first in the announced-terminal test); severity S2: no data is lost, but the transcript records a restart that never happened instead of the user's stop, and a late answer for a stopped call is not refused as such; live: W1's unit red (peer-reported, 2026-10-18): core fail-session-without-run.test.ts 'R77 and its interrupted twin': endSessionWithoutRun({ status: 'interrupted' }) over a session with a pending map keeps it today (expected ['c1'] to deeply equal []); inferred: the DO-level path to that state is reasoned, not driven (W1, from announced-terminal.ts:1-33 at 826d9665af): a resume run adopts a submitted call's pending entry, a stop lands during its drain and the stop's K5 commitState is refused, so runtime-js finalizes the interrupt (run interrupted, stream ended) while the session stays active with the entry, and the next wake commits R77's interrupted end over it. Unverified whether that stop takes the K5-failed branch at all; if it is unreachable, this record is defensive, like the runtime-js K5 | source, peer-reported | partly-inferred | withdrawn — No reachable entry leaves an interrupted session holding a pending client call, so R77's announced-terminal end never carries one (W1, orchestrator ruling, 2026-10-18; the same class as RM-82). Every stop's end clears pending in its K5 (core commitStopTerminal, run/commit-stop-terminal.ts:108, verified by the owner at origin/main 6de1f045e4; Ruling 56). A stop K5 that does not land keeps the run live and never announces interrupted: runtime-js endEntryRunByStop throws on not_committed (js-agent-executor.ts:1502, verified at origin/main 6de1f045e4), and the DO's stop deadline (do/drive/stop.ts endRunAtStopDeadline) treats not_committed as the run staying live and retries. The run loop's own stop exit comes after the drain, which under Ruling O-73 records every answered call before the loop's stop boundary, and a stop-refused with_message stash takes the endEntryRunByStop path. W1's DO pin never got past its precondition for exactly these reasons and is reverted on (ii) at 0e316a040a |
| CP-148 | S2 | The CF Durable Object's store drops a sub-session ref's status and completedAt when the ref is added: addSubSessionRefs maps each ref without them, and the insert writes running, so a ref added completed or failed reads back running, with no completedAt, where the other stores keep what they are given. runtime-js adds a settled LOCAL ephemeral sub-agent's ref completed / failed with completedAt, so on a DO that runs its sub-agents in process (no subAgentNamespace) a finished child's ref reads running forever: the parent's refs report it live, and anything that selects live local refs (a stop cascade, a children-still-running check) treats it as running. | runtime-cloudflare/src/do/do-state-store.ts:4216-4228 (origin/main 826d9665af) (addSubSessionRefs maps subSessionId, agentType, parentToolCallId, startedAt, mode, name, completionDelivered, location: no status, no completedAt) with :2007-2014 (origin/main 826d9665af) (addSubSessionRefsInternal's INSERT writes status 'running'); contrast: the in-memory store keeps the status the caller passes (WE-2, peer-reported); the DO state-store contract has no 'addSubSessionRefs keeps status' case, so it passes 278 green; related: found by WE-2 while proving an S-C witness (peer-reported, 2026-10-18); its test seed now updates the status after the add (8c721f90d6) and asserts the precondition; runtime-js/src/run-loop.ts:4042-4072 (origin/main 459fa678ca) (after a local ephemeral sub-agent settles, its ref is recorded with addSubSessionRefs, status completed or failed and completedAt; 'running' only for a suspended child): the product path the original 'latent' reading missed (WE-2, 2026-10-19); contrast: runtime-cloudflare/src/do/durable-object-agent-base.ts:3290-3291 (origin/main 459fa678ca) (the DO rewrites sub-agent tools to remote ones only when subAgentNamespace is configured; without it they run in process through runtime-js, which reaches the path above); related: every other production caller of addSubSessionRefs passes 'running' (core remote / companion dispatch, the runtime-js remote executor, DBOS, Temporal, CF Workflows), so a DO with the remote rewrite is not affected; the conformance cf-do host runs every sub-agent remotely, so no cell drives the in-process path live (source-verified, not observed); severity S2 (owner, 2026-10-19; was S3 'latent, no known product path'): a finished child's ref reads running forever on the DO's store, reachable on a DO that runs its sub-agents in process. Folds in an unmerged owner draft of the same defect (filed from WE-2's report before this record was matched, under an id withdrawn before it landed; RM-103 now names a different finding); conformance: store.sub-session-ref-keeps-status fails .status-round-trips on store-do-sqlite 3 of 3 at owner batch 1f937ca39c (main 22cb4e513a), 2026-10-18 (the added completed ref read back running); it passes on store-memory, store-redis, store-postgres and store-d1 | source, peer-reported, live-repro | verified | open |
| CP-149 | S1 | DBOS: re-spawning a persistent companion whose previous child FAILED or was TERMINATED starts it under its first spawn's workflow id and run id (${childSessionId}-r1), so DBOS dedupes the start (it hands back the existing workflow) and NO turn runs: the re-spawn's message is silently dropped. A blocking re-spawn parks to its bound and answers 'running'; a non-blocking one answers 'spawned' for a turn that never runs. CP-142's contract (failed and terminated children re-spawn fresh), broken by a different cause. | runtime-dbos/src/steps/execute-companion-tool.ts:710-720 (origin/main 0392f3d05b) (the re-spawn branch after FAILED / TERMINATED: updateSubSessionRef to running, reusing the initial session-scoped workflow id; no deleteSession) with runtime-dbos/src/workflow-ids.ts:46-48 (origin/main 0392f3d05b) (persistentSession(agentType, childSessionId): the same id for every spawn of that child); related: a DBOS workflow id cannot be re-used for a new run, so the fix needs both the delete (or a fresh child session) and a fresh workflow id; CP-142 (the CF DO, whose store refuses deletes); found by WE-2 (peer-reported, 2026-10-18, by reading), parked under the Cloudflare-first program (orchestrator ruling D-g); runtime-dbos/src/steps/execute-companion-tool.ts:734-745 (origin/main e3eee732d0) (the re-spawn branch after FAILED / TERMINATED resets the ref to running and, per its comment, reuses the initial session-scoped workflow id), :594 (origin/main e3eee732d0) (deriveChildIds: initialWorkflowId), :767 (origin/main e3eee732d0) (initialRunId = ${childSessionId}-r1, the first spawn's) and :823 (origin/main e3eee732d0) (startPersistentWorkflow(initialWorkflowId, …)); inferred: DBOS dedupes a start on an existing workflow id (the handle of that workflow comes back and nothing new runs), and the reused run id would match the first run's idempotency anyway; so the earlier reading of this record (the child runs on the failed session's history) is superseded: no turn runs at all (WE-2's reading, 2026-10-31, verified by the owner at the cited lines); related: fixer WE-2, the CP-69 MR (we2-cp69): a re-spawn launches on its own deterministic workflow and run (${child}-respawn-${toolCallId}) after stopping a still-live previous loop (a recorded inbox shutdown and a bounded wait, else a typed retryable not_spawned child_busy); its unit reds: companion-turn-truth.test.ts (the loop-live shape, and main's shape: a failed or terminated companion whose loop already exited); severity S1 (orchestrator, 2026-10-31; was S2): a success-shaped answer ('spawned' / 'running') for a turn that never runs, with the message dropped; DBOS S1s are unparked; conformance: fixed by !601 (CP-69's MR: a DBOS re-spawn starts a fresh run under a run id of its own). companion.respawn-after-failure-fresh passes every id on dbos-postgres at !601's own base (WE-2 slot w-rsx-1031-031939, 3 of 3, CP-69's fix head b03061b924 with the owner cells): after the companion fails, the re-spawn runs, its model call holds only the new message, and its state carries none of the failed child's (red at main before !601: owner slots s215 / s215b, where CP-69 kept the failed child's session active). The TERMINATED half is !601's own runtime-dbos tests (a re-spawn after a failed and after a terminated child each start a fresh turn); no conformance cell terminates a companion before re-spawning it | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-150 | S2 | A remote sub-agent ref whose child has not yet answered its /start carries location: 'remote' but no ref.remote (the dispatch records the child's stream identity only once the child answers), and three sites take ref.remote to mean "is remote". runtime-js's stop cascade skips such a child, so stopping the parent while the child's /start is in flight never interrupts it: once the /start lands the child runs on to its end, unstopped and its outcome never consumed (an orphan run and its cost). The same cascade counts the ref as an in-process child and reads, and may stop, its run through the parent's own store; and CF Workflows' ended-ephemeral reconcile treats it as a local child read from the parent's D1. S-C's reconcile had the same discriminator (WE-2 keys it on location === 'remote'). Fixed on runtime-js and the CF DO by 7daf87301b and on CF Workflows' reconcile by 656743ace8: every site keys on core isRemoteSubSessionRef (location === 'remote'). | runtime-js/src/execution/remote-subagent-executor.ts:206-209 (origin/main c4c0708011) (the dispatch adds the ref with location: 'remote' before the child can exist) with :189-197 (origin/main c4c0708011) (the stream identity, ref.remote, is recorded only once the child answers); runtime-js/src/execution/in-process-runs.ts:276-279 (origin/main c4c0708011) (stopChildren's remote pass skips a ref when ref.remote is unset or the child is not live, so a remote child with no identity gets no interrupt or abort) with :225-226 (origin/main c4c0708011) (the cascade's call site); runtime-js/src/execution/in-process-runs.ts:292 (origin/main c4c0708011) (the same ref is added to the in-process children, whose run is read with getCurrentRun from the parent's store and stopped through it at :296-300 (origin/main c4c0708011)); core/src/orchestration/record-sub-session-result.ts:411-414 (origin/main c4c0708011) (reconcileEndedEphemeralSubSessionRefs skips only refs with ref.remote, so a remote ref with no identity is reconciled as a local ephemeral child from the parent's store; CF Workflows' caller); related: found by WE-2 while fixing S-C's reconcile (peer-reported, 2026-10-18; verified at source by the owner); the CF Workflows site is fixed in WE-2's RM-80 MR with the same core predicate (isRemoteSubSessionRef), the runtime-js sites after RM-80 (orchestrator ruling); severity S2: in a plain deployment a stopped parent's in-flight remote child runs to its end unstopped (cost, and work the user stopped); the foreign reads write nothing unless the child's producer shares the parent's store, where a stop or a ref settlement bypasses the producer; related: a residual window after the fix (WE-2, 2026-10-18; FU-REMOTE-STOP-BEFORE-START-TOMBSTONE): a parent that is NOT executing, stopped while an earlier execution's remote /start is still in flight, sends its interrupt before the child exists (not found / state_not_stoppable); the /start then lands and the child runs to its end unless a later parent entry settles it. Fix direction: a child-side stop tombstone keyed on the D-2 request key. The CF Workflows parent has no such cascade at all (CP-152); related: fixed in code by 7daf87301b (CP-150 runtime-js, the stop cascade) and 656743ace8 (WE-2's RM-80 MR, the core reconcile CF Workflows calls), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/types/session.ts:739-741 (origin/main 56d564d073) (isRemoteSubSessionRef: remote by location === 'remote', never by ref.remote), runtime-js/src/execution/in-process-runs.ts:322 (origin/main 56d564d073) (the cascade's local pass skips every remote-located ref, so it never reads or stops it through the parent's store), runtime-js/src/execution/in-process-runs.ts:383 (origin/main 56d564d073) (the remote pass stops every live remote ref by location, through isLiveRemoteChildRef at core/src/run/stop-remote-child.ts:44-49 (origin/main 56d564d073), identity or not), core/src/orchestration/record-sub-session-result.ts:483 (origin/main 56d564d073) (reconcileEndedEphemeralSubSessionRefs skips a remote-located ref); the CF DO runs this cascade through runtime-js | source, peer-reported | partly-inferred | open |
| CP-151 | S3 | CF Workflows builds a sub-agent wait's step names from both agent names and the call id with no length bound, so a long enough agent name, sub-agent name or call id fails the run with a fatal engine error. The local sub-agent park's prefix is the parent's agent type, the sub-agent's type and the call id joined (its first step adds '-deadline'), and Cloudflare Workflows refuses a step name over 256 characters with a WorkflowFatalError, which ends the instance; the persistent-companion waits build theirs the same way. Instance ids and event types go through buildWorkflowInstanceId / buildWorkflowEventType (bounded and hashed); step names do not. | runtime-cloudflare/src/workflow.ts:4256-4262 (origin/main c4c0708011) (the sub-agent park: ParkClock.start with prefix agentType-wait-subAgentType-callId) with :5739 (origin/main c4c0708011) (its first step is named prefix plus -deadline), :5774 and :5819 (origin/main c4c0708011) (the slice-end and floor steps add an attempt number); runtime-cloudflare/src/workflow.ts:2537-2539 and :2966-2967 (origin/main c4c0708011) (the persistent-companion and continue waits join the agent type, the child's name and the call id the same way); contrast: runtime-cloudflare/src/instance-id.ts:19 and :96 (origin/main c4c0708011) (MAX_WORKFLOW_INSTANCE_ID_LENGTH = 100; buildWorkflowInstanceId bounds and hashes ids, and event types likewise); live: owner slot c4 (2026-10-18), transcript.co-emitted-subagent-dispatched on !418 (847784ce9d merged onto owner batch f7d4b73f43) 3 of 3: the run failed with WorkflowFatalError: Step name "…-conf-ce-parent-wait-…-conf-ce-child-…-sub-deadline" exceeds max length (256 chars) or invalid characters found (259 characters: the cell's per-cell agent names and call id; the cell's ids are now shortened); severity S3 (orchestrator, 2026-10-18): a hard failure, but it needs agent names and call ids summing past ~240 characters; provider call ids are about 30 and agent names are the app's own | source, live-repro | verified | withdrawn — A duplicate of CP-98 (the same defect: CF Workflows builds step.do names from unbounded agent names, tool names and call ids, and the engine refuses a name past 256 characters with a WorkflowFatalError). Filed by the owner on 2026-10-18 without checking CP-98; its one new observation (the co-emitted sub-agent cell on !418 failing on the sub-agent wait step's 259-character name) is carried into CP-98 as related evidence. |
| CP-152 | S2 | CF Workflows: stopping a parent whose instance is not executing never stops its live remote sub-agent child. The stop reaches a remote child only from inside an executing remote dispatch, which polls its abort controller and then sends the remote transport's abort or interrupt; no other CF Workflows path sends either. So when the parent's instance is closed or not executing (the executor's abort of a closed owner, a reader settling a closed owner's run), its remote child keeps running to its end until the parent's next entry settles the ref, or indefinitely if none comes. The CF Durable Object and runtime-js each have a stop cascade that reaches remote children. | runtime-cloudflare/src/steps.ts:4878-4887 (origin/main 6856de1efd) (the remote dispatch: if its abort controller fired (not a timeout), transport.abort for a recorded abort, else transport.interrupt; the only remote stop in the package outside the DO drive); contrast: runtime-cloudflare/src/do/drive/stop-children.ts (origin/main 6856de1efd) (the DO's stop cascade reaches remote children) and runtime-js/src/execution/in-process-runs.ts:276-288 (origin/main 6856de1efd) (stopChildren's remote pass); related: found by WE-2 while fixing CP-150 (peer-reported, 2026-10-18; source-read, no probe); the owner verified at origin/main 6856de1efd that runtime-cloudflare has no transport.interrupt or transport.abort call outside the executing dispatch and the DO drive. RM-80's settle on the parent's next entry ends the ref, not the child's run; severity S2: a stopped CF Workflows parent leaves a remote child running (cost, and work the user stopped) whenever the parent was not mid-dispatch at the stop; related: fix owner W1 (CF Workflows stop lane, after CP-16 and CP-23, then MA-16; orchestrator, 2026-10-18); conformance: subagent.remote-child-stopped-with-parent (owner batch 46at, the remote-child capability) fails .child-stopped on cfw-workflows-d1 3 of 3 on main (owner slot s54, 2026-10-19, trial 3d8cd09371 = owner batch ef609b29ab + main 92fb81362a): the parent's instance terminated while its remote call waits, its handle stopped, the child (on the lane's producer worker, held on a gate that honours its abort signal) still reads running 40 s later; and passes every id 3 of 3 on W1's fix (CP-152 0f729c2634 via CP-156 858080e961; trial 252eac9c4e), the child stopped within about a second and the ref's stopOwed absent; conformance: fixed by CP-152's MR (main a8afa649bd): subagent.remote-child-stopped-with-parent passes .child-stopped and its other ids on cfw-workflows-d1 3 of 3 (owner slot s64, 2026-10-19, owner batch 4ab4ffd345 = main cd2d6e3169 + the owner batch), where it failed .child-stopped 3 of 3 on main before (owner slot s54, above) | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-153 | S1 | DBOS: a model's call to a tool its agent does not define runs ANOTHER registered agent's tool of that name instead of being refused. DBOS keeps one process-global tool registry keyed by tool name for every agent it registers, and the tool step resolves a call by its name alone, so it refuses only a name no registered agent defines: agent A's call to a tool only agent B has executes B's tool, with A's arguments, inside A's run. (Two agents that each define a tool of the same name did run their own tool in the owner's cell: that half of the first reading did not reproduce.) | runtime-dbos/src/lifecycle/shared-registration.ts:69-70 (origin/main 51a1cc7d90) (for every effective tool of every registered agent: ToolRegistryHolder.registry[tool.name] = tool, one map for the process) with runtime-dbos/src/steps/execute-tool.ts:246-252 (origin/main 51a1cc7d90) (the tool step looks the call up as deps.toolRegistry[toolCall.name] and refuses only a name no agent registered); related: W1's first reading (peer-reported, 2026-10-18) also inferred a same-name collision (the last registered implementation answering both agents); the owner's cell did not reproduce it, so the record states only the observed cross-agent execution. DBOS is parked under the Cloudflare-first program; severity S1 (orchestrator, 2026-10-18; reproduced since): a model's call can run code its agent never offered, from another agent in the same process (whether it gets a security exception to the DBOS parking is with the user); conformance: tools.agents-isolated-by-name (owner batch 46ac) fails .foreign-tool-refused on dbos-postgres 3 of 3 at owner batch 46 merged with main 51a1cc7d90, 2026-10-18 (owner slot c11): agent A's call to ti_b_only (only agent B defines it) got B's body's answer ({"who":"B-only"}, metadata.helixToolFailed false) instead of a refusal; js-memory, js-redis, js-postgres and the Temporal lanes refuse it. Agent A's and agent B's same-named ti_shared calls each answered with their own agent's value on dbos-postgres (the cell's .a-runs-own-tool / .b-runs-own-tool checks were mis-decoded in that run and are corrected, owner batch 46ae); conformance: fixed. tools.agents-isolated-by-name (owner slot s211): .foreign-tool-refused fails on dbos-postgres at main e34b96e2c2 (expected-fail, 1 run) and passes 3 of 3 with !585 84cbc50232 (WE-2: a tool call resolves among the executing agent's own tools); the stopped-partial cells, preparing their agent once per cell, pass 3 of 3 under its name guard; related: pending widening (orchestrator, 2026-10-30): DBOS's companion spawn resolves the type by name from the process-level registry, never against the parent's declared companions (runtime-dbos/src/steps/execute-companion-tool.ts:600-612 and :905-907 (origin/main e34b96e2c2)); this record widens to that site only once companion.spawn-undeclared-type-refused (which first runs the declaring parent's turn) is red on dbos-postgres at main and green in the batch pipeline's dbos lane with !585 (owner slot s211's run was vacuous: the type was never registered; owner slot s212, the fixed cell, 3 of 3: .not-spawned and .refusal-recorded fail on dbos-postgres at main e34b96e2c2, every other lane passes; the batch lane's green is owed) | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-154 | S2 | An approved call whose tool the agent no longer has is recorded as a SUCCESSFUL tool result, so the model is told a tool that never ran succeeded (CF Workflows, runtime-js and the CF Durable Object that runs it, and Temporal). Each resume drain classifies a pending entry as an approval only by looking its tool up, so with the tool gone (a deploy that removed or renamed an approval-gated tool between the suspension and the submit) the approval falls into the submitted client-tool result branch and its { approved: true } becomes the call's success result. CF Workflows does not even persist the entry's approvalId; runtime-js and Temporal persist it but never read it there. Fixed on CF Workflows, runtime-js and the CF DO by a971537989 (CF Workflows persists approvalId at registration; every drain classifies an approval by its approvalId and pairs a gone tool's approval as an error); still open on Temporal, which persists approvalId but still classifies by the tool lookup; DBOS unproven. | runtime-cloudflare/src/steps.ts:7058-7068 (origin/main 7daf87301b) (the drain: isApprovalSubmission needs tool !== undefined && isApprovalGatedTool(tool)) with :7110-7141 (origin/main 7daf87301b) (otherwise a submitted entry with no submittedError is paired as createToolResultMessage({ result: entry.submittedResult, success: true })); runtime-cloudflare/src/approval-gate-workflow-helper.ts:51-80 (origin/main 7daf87301b) (registerPendingApproval writes toolName, input, stepCount, suspendedAt, deadlineAt and runId, but no approvalId; only the tool_approval_request frame carries it); contrast: core/src/types/state.ts:200-209 (origin/main 7daf87301b) (PendingClientToolCall.approvalId: set when the call awaits an approval, absent on runtimes that do not write it yet) and core/src/orchestration/step-iterator.ts:1648-1658 (origin/main 7daf87301b) and runtime-temporal/src/activities.ts:3944-3954 (origin/main 7daf87301b), which write it; live: W1's reds (peer-reported, 2026-10-18; each on product code at origin/main 7daf87301b, branch reliability-cf-w1-rm87-refused-call-paired). CF Workflows (dee1cb4ba7, runtime-cloudflare steps-applyResultsAndReload.test.ts, describe AgentSteps.applyResultsAndReload: an approval is classified by its approvalId (CP-154)): the registration persists no approvalId (expected undefined to be the frame's), an approved call to a gone tool is paired as a success (an error result, not a success: expected false to be true), and a gated tool's entry without its approvalId is run (the tool spy called once); live: runtime-js and the CF DO, which runs it (W1, 8b0eb9d0e8, runtime-js approval-id-classification.test.ts, a real flow: suspend on the approval, so core writes the real approvalId, submit approved, then resume under a definition without the tool): the resume drain (runtime-js/src/js-agent-executor.ts:3050-3056 (origin/main 7daf87301b), its toolByName.has gate) and the with_message drain (runtime-js/src/with-message-entry-rows.ts:207-209 and :270 (origin/main 7daf87301b)) each pair the gone tool's approval as a success (expected false to be true), and a gated tool's answered entry without its approvalId is classified by the lookup and run (expected 1 to be +0); live: Temporal (W1, a local probe, not committed): GenericActivities.applyResultsAndReload (runtime-temporal/src/activities.ts:5298-5305 (origin/main 7daf87301b), CF Workflows' twin), with an approved entry for a tool the agent no longer has that carries its approvalId, persisted { role: tool, toolName: removedTool, content: {approved:true}, metadata.helixToolFailed: false }: the misread reproduces. Temporal is parked: a follow-up on this record; inferred: (unverified) DBOS has no drain for this: runtime-dbos/src/workflows/shared.ts:3172 and :3398 (origin/main 7daf87301b) classify LIVE calls at dispatch against liveToolsByName, so a gone tool arises only when a recovery after a redeploy replays the workflow without it (unproven; a follow-up on this record); related: RM-87 (W1 found this from its reds). Fix plan (orchestrator ruling, 2026-10-18): inside RM-87's MR, tagged CP-154: persist approvalId and classify an approval by it only, with no tool-lookup fallback, and refuse a gone tool's approval as an error result. W1's cross-runtime sweep decides the other rows: a runtime-js / CF DO gap is fixed in the same MR, a Temporal / DBOS one becomes a follow-up on this record; severity S2 (orchestrator, 2026-10-18; proposed by W1 and the owner): a silent misreport of a tool that never ran, reachable only through a deploy that removes or renames an approval-gated tool while an approval is pending; related: fixed in code by !435 (merged as main a971537989, 2026-10-19: an approval is classified by its approvalId on CF Workflows, runtime-js and the CF DO; CF Workflows now persists approvalId at registration; a gone tool's approval is paired as an error). Temporal and DBOS stay open as follow-ups on this record. It flips on its cfw-workflows-d1 and cf-do cells (the agent-swap control), not written yet; related: fixed in code by a971537989 (W1, !435, RM-87 + CP-154), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/approval-gate-workflow-helper.ts:78-81 (origin/main 56d564d073) (registerPendingApproval persists approvalId), runtime-cloudflare/src/steps.ts:8897-8902 (origin/main 56d564d073) (the CF Workflows drain classifies an approval by its approvalId only) and :8907-8945 (origin/main 56d564d073) (a gated tool's answered entry without one is refused typed and paired), runtime-js/src/js-agent-executor.ts:3550-3567 (origin/main 56d564d073) (the runtime-js resume drain classifies by approvalId and pairs an approved gone tool as an error), runtime-js/src/with-message-entry-rows.ts:173-180 (origin/main 56d564d073) (the with_message drain the same), core/src/client-tool/new-turn-settlement.ts:122 (origin/main 56d564d073) (new-turn settlement the same); still open on Temporal: runtime-temporal/src/activities.ts:5341-5347 (origin/main 56d564d073) (applyResultsAndReload still classifies an approval by tool !== undefined && isApprovalGatedTool(tool), so a gone tool's approval falls to the success branch); related: W1 (peer-reported, 2026-10-25): CF Workflows child approvals now get a snapshot part through the root's clientToolCallOwnership (CP-186's fix), and the replay of a child's tool_approval_request must stay idempotent on that snapshot-rendered part (one part, approval-requested, no re-announce); whoever finishes CP-154 must keep W1's CFW-order guard (local-child-pending-reload-real-chat.test.ts, on W1's CP-186 branch) green | source, peer-reported | partly-inferred | open |
| CP-155 | S1 | runtime-js (and the CF Durable Object that runs it): re-consulting a finished persistent companion reopens the child (CAS completed → active) and resets the parent's ref to running BEFORE the child's continued run has started, so a failure in between strands the companion. Three classes. (1) A failure after the CAS but before the ref reset (the prior-result hold, the history read's state_history_incomplete, a store read) leaves the session active and the ref completed: every later re-consult loses the CAS and answers child_busy, getChildStatus reports completed with the OLD output, and only a new child name escapes, losing the child's memory: the name is wedged for good. (2) A refused run-start after the reset leaves the session active, the ref running and no live run: sendMessage answers child_not_running and getChildStatus reports running forever; the only escapes (waitForResult's orphan detection, a terminate) end in a respawn that wipes the child's memory. (3) A run-start that committed but whose call threw (a lost commit acknowledgement) leaves a live run nothing drives: a later sendMessage appends under it and answers delivered: true, a silent loss. The continue also throws instead of answering typed, so the model sees a raw error string. | runtime-js/src/run-loop.ts:7140 (origin/main 6a2a66817c) (the reopen CAS completed → active) with :7167 (origin/main 6a2a66817c) (holdUndeliveredPrior) and :7180 (origin/main 6a2a66817c) (loadLLMHistory) before :7204-7208 (origin/main 6a2a66817c) (the ref reset to running, completionDelivered false): a throw between them is class 1; runtime-js/src/run-loop.ts:7290 (origin/main 6a2a66817c) (startChildRun, the child's run-start commit, after the reset: its refusal is class 2); inferred: (unverified) class 3, a committed run-start whose call threw, from the fence rule (a later append under the run's id is accepted while nothing drives the run); WA's trace, not reproduced; related: WA's trace for CP-155's design (peer-reported, 2026-10-19, at main 6a2a66817c; WA's local note cp155-investigation.md), verified by the owner for classes 1 and 2 at that pin. Fixer: WA (design approved by the orchestrator); severity S1 (owner re-rate, 2026-10-19, orchestrator: classes 1-2 heavier than S2; was S2): a companion's name can be wedged permanently (class 1) or its memory lost to a forced respawn (class 2), and class 3 answers a lost consult as delivered; conformance: (owner slot s59, 2026-10-19, cells pin cb1333fc79 = main 6a2a66817c product, no product diff) companion.continue-window-no-stale-notice fails .no-stale-notice on cfw-workflows-d1 3 of 3: two completion notices reached the parent in turn B, while the child's continue commit was held (the stale-notice class reproduced); its .child-b-settles poll crashed on a reopened child's empty output (a cell defect, fixed at 077aa17f7f, unrun). companion.continue-unreachable-not-stranded-host passes .answer-typed and .child-not-reopened on cf-do 3 of 3 and fails .re-consult-answered only on a cell defect (it judged the non-blocking send's row; turn C now waits, 077aa17f7f, unrun); conformance: the fix (!461, WA, gated head 39c9ba8f75, final 630b7141f3): companion.continue-window-no-stale-notice (cells e76fc12bfe, its parent the wait parent) passes every id on cfw-workflows-d1 3 of 3 on the fix and fails .no-stale-notice and .outcome-notified 3 of 3 on main cd2d6e3169 (turn B received a 'No output' completion notice; the only notice after turn A did not carry the turn-B answer), and red 1 of 1 with core's run-first read reverted to the session-only sync (WA's re-slot, 2026-10-19); companion.continue-refused-not-stranded fails on main and passes on the fix on js-memory, js-redis and js-postgres 3 of 3 each (WA's gate on aace0064aa + 077aa17f7f); the two cf-do -host cells are contrasts (green at main and on the fix); runtime-js/src/run-loop.ts:6830, :7303 and runtime-js/src/run-start.ts:215 (fix evidence, !461, origin/main d919c9b584) (a re-consulted companion is reopened only by its own run-start commit, StartRun.resumedCompanionRef; a refused or failed entry leaves the child completed and the ref unchanged, which removes classes 1 and 2 by construction), with core/src/orchestration/companion-continue-outcome.ts:78 (continueStartOutcome) and core/src/orchestration/child-terminal-outcome.ts:90, :186 (origin/main 841ce4c960) (readChildTerminalOutcome / readChildNotice: the outcome and the notice are read from the child's run); runtime-cloudflare steps.ts / workflow.ts take the same run-first read; conformance: companion.continue-refused-not-stranded on js-memory, js-redis and js-postgres and companion.continue-window-no-stale-notice on cfw-workflows-d1 prove the fix. JS cells proven by !464's MR pipeline on main d919c9b584 (merged result; the pipeline id is in !464's description); WA's local JS run was on pre-final aace0064aa + 077aa17f7f. The CF Workflows cell is proven by WA's re-slot on 39c9ba8f75 + e76fc12bfe (the line above). No cell reaches class 1 (a throw between the reopen and the ref reset): the harness has no store read-fault seam, and !461 removes that window by construction. Class 3 stays inferred | source, peer-reported | partly-inferred | fixed (4 cells) |
| CP-156 | S2 | runtime-js (and, by source, the CF Durable Object in a narrow shape, since it runs runtime-js): a stop that cascades to a REMOTE child waits on that child with no time bound, so an unreachable remote child can keep the parent's stop from ever answering. stopChildren's remote pass awaits each live remote ref's transport interrupt or abort in turn, with no timeout and no signal (the transport takes none), so a remote endpoint that never answers (a hung connection, an unresponsive service binding) blocks stopChildren, the in-process run stop that awaits it, and the stop's caller. Fixed on runtime-js and the CF DO by 0723d4f5c5 (each remote child's stop bounded at 10 s, all stopped together); the in-flight dispatch's own stop is still unbounded by source. | runtime-js/src/execution/in-process-runs.ts:281-293 (origin/main 6ef75cb6c9) (if opts.remote: for each live remote ref, await transport.interrupt / transport.abort, with no bound; only a thrown error moves on); contrast: runtime-cloudflare/src/do/drive/stop-children.ts:294-307 (origin/main 6ef75cb6c9) bounds each remote child stop at CHILD_STOP_TIMEOUT_MS and moves on, the outcome unknown (stopRemoteChild); runtime-js/src/run-loop.ts:5906-5930 (origin/main a971537989) (cascadeStop: a run that ends by its stop calls stopChildren(..., { remote: true }) with remoteTransportsOf(input.tools), reaching the unbounded pass); runtime-js/src/execution/in-process-runs.ts:226-227 (origin/main a971537989) (stopInProcessRun: stopChildren with remote: true whenever the run is not already stopping); runtime-js/src/js-agent-executor.ts:1531-1532 (origin/main a971537989) (endEntryRunByStop: stopChildren with remote: true); runtime-cloudflare/src/do/durable-object-agent-base.ts:3261-3265 (origin/main a971537989) (the CF DO rewrites its sub-agents into remote sub-agent tools over DOStubTransport, so a DO run's stop reaches the same unbounded pass through the runtime-js run loop it runs; the DO's own drive cascade, stop-children.ts, is the bounded one); related: W1 (peer-reported, 2026-10-19, at the orchestrator's request), verified by the owner at main. Fix direction: CP-152 lifts the DO's bounded stopRemoteChild into a core primitive with an optional timeoutMs (runtime-js passes none there, to keep that refactor behaviour-neutral, orchestrator ruling); this record's fix is runtime-js passing the bound; severity S2 (owner, 2026-10-19): a stop that never answers, reachable through one unreachable remote child; related: fixer W1, right after CP-152 (orchestrator, 2026-10-19): CP-152's shared stop primitive takes the bound, so the fix is runtime-js passing timeoutMs, with a red; related: the CF DO path traced by W1 (2026-10-19), verified by the owner at main. Fix in progress on W1's branch reliability-cf-w1-cp156-js-remote-stop-bound (stacked on CP-152; unopened): a shared core REMOTE_CHILD_STOP_TIMEOUT_MS (10 s); every live remote child stopped together, each bounded; outcomeUnknown logged past the bound; conformance: the CF DO half did NOT reproduce (owner slot s50, 2026-10-19, 3 of 3 on main 397a2a5fa0, diagnostic): with a DO child held mid-step and its stop route held unanswered (subagent.remote-child-stop-bounded-host), the stopped parent's next turn still completed within one 10 s bound, so in the executing-parent (202) shape only the DO's bounded settle cascade reached the child, though its ref is location 'remote'. The DO line is back to inferred until W1's re-trace (orchestrator, 2026-10-19); the runtime-js half stands on its source and needs its own runtime-js executor cell; related: the DO's reach is narrow (W1, 2026-10-19, after owner slot s50): with a remote dispatch in flight, the dispatch itself stops its child on the parent's abort and marks the ref failed, so the run loop's cascade finds no live ref; the unbounded cascade wait needs a live remote ref with no dispatch in flight (a run recovered after an eviction, before it re-attaches its owed call: RM-80's window). The in-flight dispatch's own stop (remote-subagent-executor.ts:1033-1050, an awaited transport call) has no bound either by source, but did not wedge the next turn in s50; related: fixed in code by 0723d4f5c5 (W1), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/execution/in-process-runs.ts:361-401 (origin/main 56d564d073) (stopChildren's remote pass stops every live remote child together, each through core's stopRemoteChild bounded at remoteChildStopTimeoutMs, else REMOTE_CHILD_STOP_TIMEOUT_MS, and logs outcomeUnknown past the bound), core/src/run/stop-remote-child.ts:30 (origin/main 56d564d073) (the 10 s bound); the CF DO inherits it through the runtime-js run loop; still open on runtime-js and the CF DO (the secondary in-flight shape, by source): runtime-js/src/execution/remote-subagent-executor.ts:1306-1311 (origin/main 56d564d073) (the in-flight dispatch's own stop on the parent's abort awaits transport.interrupt or transport.abort with no bound) | source, peer-reported | partly-inferred | open |
| CP-157 | S1 | On runtime-js, the CF Durable Object and CF Workflows, a REMOTE sub-agent child that pauses (e.g. waiting on a human approval deep in its own tree) fails the parent's call after a few seconds of stream retries, where a LOCAL sub-agent in the same position suspends the parent (suspended_awaiting_children) and the parent waits as long as it takes. The paused child's stream ends at once; the dispatch's recovery getStatus reads paused, and with no output the call retries under its short stream-retry budget and then fails, so the parent's turn moves on with a failed call and never receives the child's real result when the approval lands minutes later. Parity needs the parent to park on a paused remote child and wake when it completes: a design question (a notification or poll back from the producer), not a local fix. Temporal's and DBOS's half is CP-230. | runtime-js/src/execution/remote-subagent-executor.ts:723-735 (origin/main c0e8bd6ebd) (recovery: a child status 'interrupted' or 'paused' is recorded as lastKnownStatus; only an output ends the call); runtime-js/src/execution/remote-subagent-executor.ts:846-858 and :875-882 (origin/main c0e8bd6ebd) (the final getStatus: paused without output leaves receivedEnd false, and the call throws 'Remote agent stream connection lost after N retries … last observed status: paused'); related: CP-45 (a paused / interrupted remote child with stale output folded as success); after CP-45's fix this failure is typed (transport_stream_drop, cause remote_child_paused, retryable), so a retry of the call re-attaches to the same child under the D-2 key, but the parent still does not park (W1); related: W1 (peer-reported, 2026-10-19, at the orchestrator's request; not in CP-45), verified on runtime-js by the owner at main c0e8bd6ebd; runtime-cloudflare/src/do/drive/materialize.ts:119 with do/durable-object-agent-base.ts:3224-3279 (origin/main cfed7b8ffa) (on the CF Durable Object EVERY sub-agent is a remote child: rewriteSubAgentTools turns each sub-agent tool into a DOStubTransport remote tool, DOStateStore holding one session), so a DO parent never parks on a paused child of any depth (WE-2's CP-157 design research, 2026-10-19); related: the producer's awaitingClientTool covers only the child's OWN pending calls, so a pause deeper in its tree skips the fail-fast (WE-2: agent-server.ts:833; DO :4292-4306, not re-pinned by the owner); after the failure the runtimes mark the ref failed and the stop cascade skips failed refs, so the paused child is never interrupted and stays parked on its producer; fixed on runtime-js, the CF DO and CF Workflows by CP-157's first MR (!504's 3d3eca56c5 interrupts a failed call's child before its ref settles failed), still open on Temporal and DBOS (CP-230); severity S1 (owner re-rate, 2026-10-19, orchestrator: S1 for the DO; was S2): on the CF DO every sub-agent is remote, so any sub-agent that pauses (a human approval anywhere in its tree) fails the parent's call within its retry budget, and the child's real result never reaches the parent's turn; related: fixer WE-2 (the remote lane), after RM-90; a design first, to the orchestrator (orchestrator, 2026-10-19); runtime-js/src/execution/remote-subagent-executor.ts:929-933 (origin/main fb9150cc16) (still true at main: no end frame and a paused child throw RemoteChildPausedError, a failed call; nothing turns it into a parent suspension), reached on the CF DO, which rewrites every sub-agent into this transport (runtime-cloudflare/src/do/durable-object-agent-base.ts:3561-3566 (origin/main fb9150cc16)), and on CF Workflows, which fails the call once its retries are spent (runtime-cloudflare/src/workflow.ts:2764-2779 (origin/main fb9150cc16)). verified by the owner at origin/main fb9150cc16 (S1 sweep, 2026-10-24); related: Fixer: WE-2 (orchestrator, 2026-10-24); core/src/orchestration/remote-pause.ts:32-45 (origin/main 47dd092fb9) (fix: !504, WE-2, merged as 47dd092fb9: planRemotePause parks the parent on a paused child whose /status lists its tree's pending calls, untruncated, over a transport that can submit; otherwise the CP-45 failure is kept); runtime-js/src/execution/remote-subagent-executor.ts:1000-1046 (origin/main 47dd092fb9) (fix: !504: runtime-js, and so the CF DO, parks on that plan, a remote wait with the ref written paused_awaiting_client at :1028, so the parent suspends suspended_awaiting_children; the root forwards an answer for a parked call to the producer through core routeParkedRemoteSubmit); runtime-cloudflare/src/steps.ts:6159, :6546 and :6580 (origin/main 47dd092fb9) (fix: !504: CF Workflows parks first and refuses only when planRemotePause does not park, at depth 1 too; its re-park CAS expects the status the step read); conformance: red at main's product. subagent.remote-child-approval-parks-parent-host, subagent.remote-child-approval-parks-parent-client and subagent.remote-child-approval-parks-parent-reload-client on cf-do (host, client) and subagent.remote-child-approval-parks-parent on cfw-workflows-d1 (executor) fail .parent-parked and the ids after it, 3 of 3, with every product source at origin/main 64110bd5ac and the owner's cells (owner slot s118 R, 2026-10-24); the js-chat-host host and client cells pass there (the local contrast); conformance: fixed. The same four cells pass (failedIds empty), 3 of 3, at !504's dfbe79516c (owner slot s119) and at 5d4422556b (s126, every cf-do teardown healthy), and on !504's final MR pipeline 2986595385 at 5326e65bcf (its test:conformance shards). The js-chat-host reload-client cell stays expected-fail there, as CP-186; related: the spin bound (a second no-progress re-park of the same child and call fails the call typed) and the CFW re-park are proven by unit tests only, runtime-js/src/__tests__/remote-park.test.ts and runtime-cloudflare/src/__tests__/cfw-cp157-remote-park.test.ts (origin/main 47dd092fb9): no cell drives a producer that re-parks on the same call; related: a remote child answered on the producer's own side does not wake its parked root: CP-185, open, due with CP-157 MR 2 (FU-CP157-MR2-DURABLE-POLL, docs/dev/follow-ups.md); related: split by the owner on 2026-10-25 (orchestrator ruling at filing, 2026-10-24: CP-157 fixed for the CF-first runtimes, a sibling for Temporal and DBOS): !504 touches neither runtime-temporal nor runtime-dbos; their half is CP-230; related: CP-279 (2026-11-06) narrows this record's premise that a terminal run's parked calls can never be woken: it holds for the session's OWN parks, but a park root also records its descendants' parks (CP-245), and a descendant whose run is still suspended keeps its entries through every terminal commit (core/src/orchestration/remote-pause.ts:149-160, as on branch reliability-w1-cp279-companion-outlives-root a80d04453c). The stores' status CAS off live still clears the map whole (FU-STORE-TERMINAL-CAS-DROPS-DESCENDANT-PARK) | source, peer-reported, live-repro | verified | fixed (4 cells) |
| CP-158 | S3 | CF Durable Object (narrow reach): a stop of a run a drive is executing can send a live remote child TWO stops, through two cascades in sequence (the runtime-js run loop's cascadeStop, then the drive's settle cascade, both over the same live remote refs), with nothing to de-duplicate them. It needs a remote ref that is still LIVE with NO dispatch in flight when the executing run is stopped, e.g. a run recovered after an eviction that has not yet re-attached its owed remote call (RM-80's window). With a dispatch in flight, the dispatch itself sends its child the stop on the parent's abort and writes the ref failed, so both cascades find no live ref and the child gets ONE stop (the common shape; owner slot s50). Where it applies: a duplicate stop (an idempotent child answers the second as already ended), a doubled wait for a hung child, the producer claim held through both. | runtime-cloudflare/src/do/drive/stop.ts:197 (origin/main cfed7b8ffa) (handleStop: the stop route answers through core stopRun directly, not runtime-js stopInProcessRun); runtime-js/src/run-loop.ts:1735 and :5906-5930 (origin/main cfed7b8ffa) (cascade 1: the run ends by its stop; cascadeStop calls stopChildren with { remote: true } over remoteTransportsOf(agent.tools), writing no ref); runtime-cloudflare/src/do/drive/settle.ts:465-493 (origin/main cfed7b8ffa) (cascade 2: after the drive returns, settle's cascadeStop calls host.stopChildren, runtime-cloudflare/src/do/durable-object-agent-base.ts's stopDOChildren); runtime-cloudflare/src/do/drive/stop-children.ts:144 (origin/main cfed7b8ffa) (its remote set is every live non-persistent ref: the same refs cascade 1 stopped); runtime-js/src/js-agent-executor.ts:7079 (origin/main cfed7b8ffa) (the run loop is given inProcessRuns on every run, the DO included, so its remote pass runs there too); contrast: runtime-js/src/execution/in-process-runs.ts:226-227 (origin/main cfed7b8ffa) (stopInProcessRun passes remote: answer.status !== 'stopping', one cascade per stop; not on the DO's path); inferred: (unverified) an entry whose commit the stop refused also cascades twice: js-agent-executor.ts:1531-1532 endEntryRunByStop with remote: true, then the drive's settle cascade (W1, by source, not traced end to end); related: CP-156 (the run-loop cascade is the unbounded one; with its fix a hung child costs 2 × 10 s here); CP-152 (CF Workflows needed the same one-cascade-per-stop guards); related: W1 (peer-reported, 2026-10-19, traced at main cfed7b8ffa and on its cp156 branch at 91fdfc0745: the same flow, only cascade 1's bound and parallelism differ), verified by the owner at main. Fix direction: on the DO only the settle cascade (the durable one: its mark, owed-record retries from the alarm, consult scoping) stops remote children; the drive tells its executor to skip the run loop's remote pass, and the refused-entry path likewise; severity S3 (owner, 2026-10-19): a duplicate stop and a doubled wait, no wrong outcome (an idempotent child answers the second stop as already ended); related: fixer WA (the DO drive lane), grouped with CP-40 and the persistent-consult stranding after MA-31 (orchestrator, 2026-10-19); conformance: NOT reproduced in the 202 shape (owner slot s50, 2026-10-19, 3 of 3 on main 397a2a5fa0, diagnostic): subagent.remote-child-stopped-once-host counted exactly ONE stop request on the child's stop route for one stop of an executing DO parent. The cell is kept; the record waits on W1's re-trace of the two cascades (orchestrator, 2026-10-19); runtime-js/src/execution/remote-subagent-executor.ts:1033-1050 and :1056-1062 (origin/main 49c3daae63) (on the parent's abort the in-flight dispatch sends its child the stop, awaiting transport.interrupt / abort, then writes the ref failed: no cascade then finds it live); related: narrowed by W1 (2026-10-19, after owner slot s50 counted one stop; verified by the owner at main 49c3daae63). The dispatch's failed ref is CP-157's leak (e): every later cascade skips a failed ref, harmless today only because that same dispatch sent the stop first; CP-157's fix interrupts a still-paused child before marking it failed | source, peer-reported | partly-inferred | open |
| CP-159 | S2 | On CF Workflows, Temporal and DBOS a stop of a parent PARKED on its suspended children (suspended_awaiting_children) never reaches those children, local ones included: the children stay suspended, their own pending calls answerable, after the parent's run has ended stopped. CF Workflows propagates a stop only to the children still in its in-body pendingSubAgents map, and a child is removed from it once it suspends; once the parent is parked, its stop goes through core stopRun with no child cascade. Temporal's and DBOS's stop paths have no child cascade at all. runtime-js and the CF DO do cascade. Fixed on CF Workflows by 079b079413 (W1, !452): its executor's stop cascades to the run's children through stopChildrenOfRun; still open on Temporal and DBOS (parked). | runtime-cloudflare/src/workflow.ts:4598-4605 and :4792-4800 (origin/main cfed7b8ffa) (a child that suspends is collected as suspended and DELETED from pendingSubAgents); runtime-cloudflare/src/workflow.ts:4905-4928 (origin/main cfed7b8ffa) (the in-body stop propagates only to pendingSubAgents); runtime-cloudflare/src/executor.ts:5010 (origin/main cfed7b8ffa) (interruptRun: a parked run is stopped through core stopRun, run-level only; the executor names no child cascade); runtime-temporal/src/executor.ts:1056-1090 (origin/main cfed7b8ffa) (stopRunById / interruptRun: core stopRun, no child cascade); runtime-dbos/src/lifecycle/interrupt.ts:106-137 (origin/main cfed7b8ffa) (stopRun → cancelWorkflow of the owner; the module never names a child); contrast: runtime-js/src/execution/in-process-runs.ts:249-293 (origin/main cfed7b8ffa) (stopChildren) and runtime-cloudflare/src/do/drive/stop-children.ts:127-223 (origin/main 841ce4c960) (stopDOChildren) cascade to children; related: WE-2 (peer-reported, 2026-10-19, from CP-157's design research, two traced explorations), verified by the owner at main cfed7b8ffa; severity S2 (owner, 2026-10-19): a stop the caller saw succeed leaves its children suspended and resumable (a later answer to a child's pending call runs it on with no parent), across three runtimes; related: fixer W1 (the CF Workflows stop lane), after CP-152 / CP-156, reusing CP-152's shared cascade primitive for parked children, local and remote; the Temporal and DBOS halves are follow-ups on this id, parked (orchestrator, 2026-10-19); related: the CF Workflows half is fixed by !452 (W1, merged as 079b079413): a stopped run's children, local and remote, are stopped through runtime-cloudflare/src/stop-children.ts:122 (origin/main 079b079413) (stopChildrenOfRun; a local child's current run is stopped by kind, its own terminal awaited, bounded, and the parent's ref written from it), called from the executor's stop at runtime-cloudflare/src/executor.ts:2099-2106 (origin/main 079b079413). Its witnesses are runtime-cloudflare/src/__tests__/cfw-cp159-suspended-child-stop.wf-noiso.test.ts (describe 'CP-159: a stop reaches a suspended local child (real workerd)': interrupt, abort, a suspended grandchild, a second stop writes nothing more) and cfw-cp159-wave-stop.test.ts; CP-23's history.companion-continue passes on cfw-workflows-d1 on it. The Temporal and DBOS halves stay open, parked; related: fixed in code by 079b079413 (W1, !452) on CF Workflows, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/executor.ts:2080-2085 (origin/main 56d564d073) (a stop that committed the run's terminal with nothing executing it cascades to the run's children), runtime-cloudflare/src/executor.ts:2100 (origin/main 56d564d073) (the cascade calls stopChildrenOfRun), runtime-cloudflare/src/stop-children.ts:163 (origin/main 56d564d073) (stopChildrenOfRun stops each live child, local and remote, by kind); still open on Temporal and DBOS: runtime-temporal/src/executor.ts:1055-1070 (origin/main 56d564d073) (stopRunById is core stopRun only, with no child cascade), runtime-dbos/src/lifecycle/interrupt.ts:72-83 (origin/main 56d564d073) (stopDbosRun is core stopRun only; the module never names a child) | source, peer-reported | verified | open |
| CP-160 | S2 | CF Workflows and Temporal commitSuspendedStep overwrite a child's OWN paused status (a child suspended on its own client tool or approval) with status failed, failureReason parent_suspended, contradicting their own C2 comment: its CAS treats paused as non-terminal and rewrites it, the very clobber the comment says the re-read exists to avoid ('clobber a child's own client-tool suspension and lose its toolCallId'). The child's pending entries survive; its status does not, so the parent's resume drain reads a suspended child as a parent-suspended failure and re-spawns it. | runtime-cloudflare/src/steps.ts:6860-6915 (origin/main cfed7b8ffa) (the C2 comment, then NON_TERMINAL = { 'active', 'paused' } and saveState(child, { status: 'failed', failureReason: 'parent_suspended' }) for any child not terminal); runtime-temporal/src/activities.ts:6476-6520 (origin/main cfed7b8ffa) (the same: NON_TERMINAL includes 'paused', the child is saved failed / parent_suspended; the comment names the clobber of a child's own client-tool suspension); related: WE-2 (peer-reported, 2026-10-19, from CP-157's design research), verified by the owner at main cfed7b8ffa; severity S2 (owner, 2026-10-19): a suspended child's own state is overwritten as failed, so its suspension is treated as a failure to retry; related: fixer WA, !583 (CF Workflows and Temporal together; the Temporal half un-parked with the Temporal / DBOS S1s, 2026-10-29); the re-spawn the earlier record inferred is a resume-mode entry that takes over the child's pending calls (WA, from source), so no answer is lost: that inference is dropped; runtime-cloudflare/src/steps.ts:9116 (fix: CP-160, WA, !583 70deed3b2e) (commitSuspendedStep marks a child failed / parent_suspended only while it is active; a paused child keeps its own suspension) and runtime-temporal/src/activities.ts:6838 (fix: CP-160, WA, !583 70deed3b2e) (the same); conformance: fixed. lifecycle.child-own-suspension-kept (owner slot s210, 3 of 3, identical sets): .child-status-kept fails on cfw-workflows-d1, temporal-memory, temporal-redis and temporal-postgres at main e34b96e2c2 (the child reads failed, not paused) and passes on all four with !583 (70deed3b2e); js-memory, js-redis and js-postgres pass on both | source, peer-reported | verified | fixed (4 cells) |
| CP-161 | S2 | runtime-js (and the CF Durable Object that runs it): a persistent companion's FIRST spawn answers { started: true } before its child's run-start commit. The child's state init, run-start commit and loop run in a detached task, and the spawn returns at once, so a spawn whose child run-start is refused reads as spawned to the model, and its ref turns failed only later. The same untyped-answer gap as CP-155's continue, on the spawn side. Fixed on runtime-js and the CF DO by e3f03e8196 (the spawn answers from the child's entry outcome). | runtime-js/src/run-loop.ts:7010-7077 (origin/main 6a2a66817c) (the first spawn: an un-awaited async task runs initializeSubAgentState, then runSubAgentLoop with the child's run-start commit; the spawn returns { started: true } at :7077 without awaiting it); related: WA (peer-reported, 2026-10-19, side record (a) of the CP-155 trace), verified by the owner at main 6a2a66817c. Fixer: the orchestrator assigns (WA's companion lane); severity S2 (owner, 2026-10-19): the model is told a companion started that never ran; its later outcome reads failed; related: fixer WA, right after CP-155 in the companion lane: await the child's run-start commit before answering started (orchestrator, 2026-10-19); related: fixed in code by e3f03e8196 (CP-161: a companion's first spawn and respawn answer from the child's entry outcome), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/run-loop.ts:7581-7747 (origin/main 56d564d073) (startPersistentChildEntry: the completion-support refusal, createSession, the guarded run-start commit and run_started, awaited before the spawn answers; only the child's loop runs detached), runtime-js/src/run-loop.ts:8293-8330 (origin/main 56d564d073) (the first spawn awaits that entry, and a not_started entry answers typed, is logged companion.spawn.not_started and fails the never-run child session); on the CF DO the companion tools are the DO's own (they carry _companionDispatchOverride, so runtime-js's companion dispatch is bypassed) and its first spawn answers the same way: runtime-cloudflare/src/do/persistent-companion-tools.ts:1389-1499 (origin/main 6a2f8b288a) (spawnPersistentChild, the core dispatcher's spawn dep: the consult recorded, the bounded /start awaited, its answer returned only after childEntryOutcome maps it) and :1068-1154 (origin/main 6a2f8b288a) (childEntryOutcome: started on the child's 2xx start answer, or for a replay on the busy 409 of its own child; else a typed not-started reason, child_start_unconfirmed / child_busy / child_terminated / child_entry_failed); related: D2's window (a second execution of the same companion call, refused at its own entry) is unreachable on runtime-js after T0 and CP-106 (owner ruling 2026-10-26, read at origin/main 318045e4bd): every runtime-js companion call runs in the child phase (core/src/orchestration/step-iterator.ts:1747 (origin/main 318045e4bd), core/src/orchestration/child-phase.ts:25-27 (origin/main 318045e4bd)), so its start is keyed on the committing run (core/src/orchestration/companion-tool-dispatch.ts:1902 (origin/main 318045e4bd)), and a repeat that loses attaches to the run serving the call (runtime-js/src/run-loop.ts:7867 (origin/main 318045e4bd), keyedRunServingCall). Residual: a lease-lapse zombie racing the recovered run AND a failed getRun in keyedRunServingCall (runtime-js/src/run-loop.ts:7878 (origin/main 318045e4bd)) at the same instant, its claim release fenced by expect.parentRunId. W1's diagnostic (2026-10-27) showed the original m-4 "replay" was a re-emission (the first execution had already answered the call), which T0 turns into a new call. The genuine same-call window IS reachable on runtime-js: a first execution that stalls or crashes inside its child phase (after the child's run started, before the call's answer commits) and its successor's B2 re-entry, which replays the call under its committed id; the store refuses its keyed start request_key_taken, and CP-106's keyed rule (keyedRunServingCall) or CP-161 §2.3's child_busy attach answers it, never a refusal cause. So fix 3 (consultRunExists: a refused entry mapped to child_entry_failed beside a serving run) has a runtime window that is not reachable on runtime-js today, because the keyed attach and §2.3 answer first: it is witnessed at core's seam only (a4a33dc68b, companionSpawnNonStartRefUpdate / companionSpawnNonStartFailsSession, red on target), the remaining guard if a later change removes an attach layer. The production window itself is witnessed by W1's m-4, 9b0b5c9ad3 (runtime-js companion-first-spawn-entry; it supersedes aef0faaab5 / 68865e1596): a first execution stalled at childPhaseHooks.afterChildStart, then its successor's B2 recovery of the same call; the store refuses the successor's keyed start request_key_taken (core maps it to child_busy), and the replay ATTACHES to the run serving the call (CP-106's keyedRunServingCall and CP-161 §2.3's replay attach): R1 keeps the ref unwritten and the verdict is delivered once (red with both attach layers bypassed: the replay answers not_spawned / child_busy). Both are on reliability-cf-w1-t0-tasks456 (T0, not yet an MR). WE-2 and the orchestrator agree | source, peer-reported | verified | open |
| CP-162 | S2 | runtime-js (and the CF Durable Object that runs it): a terminate racing a companion's continue is lost. The continue's ref reset writes status running with no check, so it overwrites a just-written terminated ref, and the child's end later writes completed: the companion the caller terminated runs on and is recorded completed. | runtime-js/src/run-loop.ts:7204-7208 (origin/main 6a2a66817c) (the continue's updateSubSessionRef(status running, completionDelivered false), unconditional: no read of the ref's current status); related: WA (peer-reported, 2026-10-19, side record (b) of the CP-155 trace), verified by the owner at main 6a2a66817c. CP-155's fix closes it (the run-start commit's resumedCompanionRef refuses a terminated ref, typed); WA tests it in that MR; severity S2 (owner, 2026-10-19): a terminate the caller saw succeed is undone by a concurrent continue; related: fixer WA, closed inside CP-155's MR (the resumedCompanionRef refusal of a terminated ref) with its own red; that MR names both ids (orchestrator, 2026-10-19); contrast: companion.terminate-during-continue-not-lost-host passes on cf-do 3 of 3 (owner slot s59, 2026-10-19, cells pin cb1333fc79 = main 6a2a66817c product, no product diff): the conformance cf-do host runs companions in child DOs (the remote dispatch), which never takes run-loop.ts's local continue path above, so the DO host cell is a contrast, not a witness; the defect's witness is the JS-lane executor cell this record is pending on; conformance: companion.terminate-during-continue-not-lost fails on main and passes on CP-155's fix (!461, which refuses a terminated ref at the continue's run-start commit) on js-memory, js-redis and js-postgres 3 of 3 each (red at main in owner slot s56; green in WA's CP-155 gate on aace0064aa + 077aa17f7f, 2026-10-19); its cfw-workflows-d1 and temporal lanes are declared gaps; runtime-js/src/run-loop.ts:7303 and runtime-js/src/run-start.ts:215 (fix evidence, !461, origin/main d919c9b584) (the continue's run-start commit carries StartRun.resumedCompanionRef, which a terminated ref refuses, so a concurrent terminate is never overwritten); conformance: companion.terminate-during-continue-not-lost on js-memory, js-redis and js-postgres proves the fix. JS cells proven by !464's MR pipeline on main d919c9b584 (merged result; the pipeline id is in !464's description); WA's local JS run was on pre-final aace0064aa + 077aa17f7f; conformance: the window this record's cell forced is unreachable since !523 (CP-106 MR A: a step's companion calls run after its commit, one at a time in call order, on runtime-js and the CF DO, the only runtimes its cells ran on): a same-step terminate now starts only after the continue's run-start commit, so companion.terminate-during-continue-not-lost{,-host} (which held that commit) deadlocked in !523's MR pipeline 2987684089 (jobs 17455241724 and 17455241720) and are retired. Their siblings companion.terminate-during-live-continue-not-lost (js lanes) and -host (cf-do) guard the reachable order: the terminate runs while the continued child's model call is held, and must stop it. They cannot red this record (the pre-fix overwrite needs the retired window), so the proof is the store contract case below | source, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-163 | S1 | A chat send over a suspended run still wakes the OLD run on a self-waking runtime (the CF Durable Object), and the send's message is dropped, through two paths CP-54's fix does not cover. (1) The pre-routing deadline sweep: on every request, a send with a new message included, handleChatStream submits client_tool_deadline_exceeded for each LAPSED pending call before its routing, and the submit that completes the batch wakes the old run, which continues its plan while the send's own entry then loses (the 2026-10-17 chat incident took this path: its calls had lapsed, CP-138). (2) A mixed suspension (pending client calls AND awaited children) is not input_after_drain, so the send falls through to the per-call abandonPendingTools, the same per-call-submit wake. | ai-sdk/src/handler/handle-chat-stream.ts:657-675 (origin/main 6a2a66817c) (the deadline sweep: for every expired pending call, submitOrIgnore(client_tool_deadline_exceeded), before routing); ai-sdk/src/handler/handle-chat-stream.ts:772-795 (origin/main 6a2a66817c) (the send's routing: the abandon-and-send path only when planWithMessageEntry is input_after_drain; otherwise abandonPendingTools submits per call); related: CP-54 (fixed: the non-lapsed, client-only send over a pending call; its provenBy cells, chat.send-over-pending-old-plan, exercise only that case) and CP-138 (the missed deadline that made the incident's calls lapse); related: WA (peer-reported, 2026-10-19, from the orchestrator's DoD audit of the incident MRs), verified by the owner at main 6a2a66817c. Fixer: WA in !409 (S-B, draft): with a trailing user message the sweep does not run (the entry's own drain expires lapsed calls to client_tool_timeout inside its run-start commit, no submit before routing), and a mixed suspension answers a typed 409 state_already_running before any write (the DO's /snapshot now carries suspendedAwaitingChildren); severity S1 (owner, 2026-10-19): the user's message is dropped while the old run continues, CP-54's own consequence, reached by the incident's own path; related: agreed (orchestrator and WA, 2026-10-19): a sibling of CP-54 (which stays fixed for its own shape), fixer WA via !409; the owner writes its two cells against WA's branch, red on main and green on !409's head before !409 un-drafts (held for the user's CLAUDE.md go); whichever lands second carries the flip. WD's incident replay e2e covers the same lapsed path at the e2e tier; conformance: chat.send-over-mixed-suspension is a CONTRAST, not a witness (owner slots s56 / s58, 2026-10-19, main 6a2a66817c plus the cells): green 3 of 3 on js-chat-host, where runtime-js already refuses a send over the mixed suspension (409 state_already_running, nothing written); the mixed defect is the CF DO's self-waking abandon, where the cell is a declared gap. The mixed half's only witness is S-B's seeded real-DO integ. The lapsed half's red: chat.send-over-lapsed-pending fails .paired-as-timeout 3 of 3 on js-chat-host on main (the row's code is client_tool_deadline_exceeded, the pre-routing sweep's); conformance: fixed by !409 (WA S-B: a chat send never submits before it is routed; main cd2d6e3169): chat.send-over-lapsed-pending passes .paired-as-timeout and its other ids on js-chat-host 3 of 3 (owner slot s64, 2026-10-19, owner batch 4ab4ffd345 = main cd2d6e3169 + the owner batch), where it failed .paired-as-timeout on main before !409 (owner slot s56, 2026-10-19, main 6a2a66817c plus the cells); cf-do is a declared gap for it (the DO's own client-tool deadline alarm expires a lapsed call before a send can meet it; S-B's seeded DO-wire test covers that routing), and the mixed half's witness is S-B's own MR tests (see above); related: the CF DO cannot build CP-163's mixed state through any real flow (WM's T1 work, 2026-10-20, verified by the owner at origin/main 46db645e27): a remote child that suspends on a client call makes the parent fail fast (core/src/orchestration/remote-sub-agent-dispatch.ts:541 (origin/main 46db645e27), RemoteSubAgentClientToolUnsupportedError, #107; the DO's status surfaces it at runtime-cloudflare/src/do/durable-object-agent-base.ts:4408-4413 (origin/main 46db645e27)), and a merely running remote child keeps the parent active with the step uncommitted. So !409's DO-side mixed refusal is defensive: its proof stays the stub-wire test, and the cf-do mixed cell is a declared gap. Not a finding (nothing behaves wrongly). It becomes reachable only if the DO gains local, in-isolate children, at which point the refusal needs a real-flow cell | source, peer-reported | verified | fixed (1 cells) |
| CP-164 | S2 | runtime-js: a persistent companion whose process crashed mid-run can only be recovered by a respawn that WIPES its memory. When the process dies while a companion child's run is live, nothing resumes that run, so the child's run stays running on a lapsed lease and the parent's ref stays running. The only recovery is core's waitForResult orphan compensation, which marks the ref failed (and delivered); a later same-name spawn then respawns, deleting the child session with its whole transcript. Before compensation, a sendMessage to the crashed child appends under its undriven run and answers delivered: true, a silent loss. | core/src/orchestration/companion-tool-dispatch.ts:1409-1432 (origin/main 6a2a66817c) (waitForResult's orphan detection: after ORPHAN_CONFIRMATIONS it compensates the ref to status failed, completionDelivered true); runtime-js/src/run-loop.ts:7085-7113 (origin/main 6a2a66817c) (a same-name spawn over a failed child respawns it: deleteSession wipes what the child session holds); inferred: (unverified) runtime-js has no child-run reconciler that resumes a crashed companion's live run, and a sendMessage before compensation answers delivered: true to the undriven run (WA's reading); related: CP-155 (a companion stranded by a failed continue: its class 3 is the same undriven-run silent loss in its own window; CP-155's FU pointer cites this record); WA (peer-reported, 2026-10-19, at the orchestrator's request), the compensation and the respawn's delete verified by the owner at main 6a2a66817c; related: the reviewer's fix direction: on a confirmed orphan, end the child's run interrupted (core stopRun / commitStopTerminal) and set the ref interrupted instead of failed, so a later sendMessage / spawnAgent resumes the companion with its memory (runtime-js's interrupted-resume path, resumeChildAgent). Parked with the JS runtimes; severity S2 (owner, 2026-10-19): a crash costs a companion its whole memory, and one path answers a lost message as delivered | source, peer-reported | partly-inferred | open |
| CP-165 | S1 | useHelixChat's dropped-send re-attach swallows a failed snapshot read: when a send's stream drops and the snapshot route then fails (a 503 past its retries, or a 404), the re-attach's gate read is caught and dropped, so nothing re-attaches and nothing reaches resyncError / onError while the run continues; the user sees a bare network error and the live run is lost to them (useReanchorOnError reports the same failure). Fixed in the shared ai-sdk client by 5cf0bf0ae9 (!444): the one re-attach controller reports a failed gate read to onError / resyncError. | ai-sdk/src/react/use-helix-chat.ts:581-587 (origin/main 6a2a66817c) (the re-attach: fetchSnapshotWithRetry at :582, the whole block ending .catch(() => undefined) at :587); contrast: ai-sdk/src/react/use-reanchor-on-error.ts:92-95 (origin/main 6a2a66817c) (the same snapshot failure reported); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !401 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S1 (owner, 2026-10-19); ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-165 among the findings it fixes). Not witnessed: no conformance cell yet (FU-CONF-444-CLIENT-CELLS); related: fixed in code by 5cf0bf0ae9 (WD, !444), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/client/reattach.ts:280-300 (origin/main 56d564d073) (startIfServerHolds reports a failed snapshot read through report(err) instead of swallowing it), ai-sdk/src/react/use-helix-chat.ts:580-597 (origin/main 56d564d073) (useHelixChat builds the shared controller and wires onReport to its reattach error and to resync onError), ai-sdk/src/react/use-reanchor-on-error.ts:103 (origin/main 56d564d073) (useReanchorOnError uses the same controller) | source, peer-reported | verified | open |
| CP-166 | S2 | useHelixChat's send-drop check serverHoldsTurn compares user-message COUNTS only, never ids or content: if another tab committed a turn while this tab's send never committed, the counts match and the re-anchor (or retry()) silently replaces this tab's unsent message with the other tab's turn. | ai-sdk/src/client/send-drop.ts:32-37 (origin/main 6a2a66817c) (return users(server) >= users(client)); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !401 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S2 (owner, 2026-10-19); conformance: client.retry-after-other-tab-turn fails .a-message-not-replaced on cf-do and js-chat-host, 3 of 3 each (owner slot s59, 2026-10-19, cells pin cb1333fc79 = main 6a2a66817c product, no product diff, its own invocation): after A's retry the session's user messages are [TABS-FIRST, TABS-B-MESSAGE], A shows its message false, A reported a failure; WD's gate (tree a1ca68c834) passes it 3 of 3 on js-chat-host (its cf-do run there was poisoned by an earlier cell's late traffic, not judged); ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-166 among the findings it fixes); conformance: client.retry-after-other-tab-turn, ledgered red above (.a-message-not-replaced), passes every id on cf-do and js-chat-host (owner slot s83, 2026-10-20, the owner batch 6f2acd67f8 merged with main 6ae9057c9e (!444), 2 of 2 each; reported UNEXPECTED PASS against the ledger, which is removed) | source, peer-reported, live-repro | verified | fixed (2 cells) |
| CP-167 | S1 | useHelixChat (and useReanchorOnError) decide that a dropped request's response had streamed from the RENDERED chat.status, which ai sets to streaming only on the first chunk that changes the message: a 2xx whose body fails before such a chunk (a start chunk with no id or metadata does not count), or a streaming / error pair batched into one render, is never re-attached, so the live run is lost to the client. Fixed in the shared ai-sdk client by 5cf0bf0ae9 (!444): both hooks now decide from the transport-recorded outcome of the failed request. | ai-sdk/src/react/use-helix-chat.ts:536-540 and :578 (origin/main 6a2a66817c) (requestStreamedRef set only when chat.status === 'streaming'; the re-attach gated on it); ai-sdk/src/react/use-reanchor-on-error.ts:62-66 and :74 (origin/main 6a2a66817c) (the same inference); inferred: (unverified) that this was the 2026-10-17 chat incident's stuck state (WD's reading; not checkable from source); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !401 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S1 (owner, 2026-10-19); ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-167 among the findings it fixes). Not witnessed: no conformance cell yet (FU-CONF-444-CLIENT-CELLS); related: fixed in code by 5cf0bf0ae9 (!444, one re-attach controller, CP-165..170), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/react/use-helix-chat.ts:580 (origin/main 56d564d073) (useHelixChat builds the shared createReattachController), ai-sdk/src/react/use-reanchor-on-error.ts:103 (origin/main 56d564d073) (useReanchorOnError builds the same controller), ai-sdk/src/client/reattach.ts:303-326 (origin/main 56d564d073) (observe decides from the transport-recorded ConnectionOutcome of the failed request: a failed send with a re-attachable outcome re-anchors when the server holds the turn, never from the rendered chat.status) | source, peer-reported | partly-inferred | open |
| CP-168 | S2 | useHelixChat classifies any error that is not an AI_* error and not a Helix body as a dropped stream (isDroppedStreamError by error.name prefix), so an app's own throw from onToolCall / onData (a TypeError) is re-anchored as a drop, hiding the app's bug; with the failed-resume re-anchor (no cap) a throw on a live call can repeat. Fixed in ai-sdk by 5cf0bf0ae9 (!444): re-attach is decided from the transport's recorded outcome. | ai-sdk/src/client/send-drop.ts:24-26 (origin/main 6a2a66817c) (return !error.name.startsWith('AI_') && parseHelixChatError(error) === undefined); ai-sdk/src/react/use-helix-chat.ts:398-405 (origin/main 6a2a66817c) (the onToolCall wrapper does not catch); inferred: (unverified) the loop: replay suppression blocks re-dispatch of replayed calls, so it needs the throw on a live, non-replayed call; related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !401 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S2 (owner, 2026-10-19); ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-168 among the findings it fixes). Not witnessed: no conformance cell yet (FU-CONF-444-CLIENT-CELLS); related: fixed in code by 5cf0bf0ae9 (!444, one re-attach controller), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/client/send-drop.ts:25-29 (origin/main 56d564d073) (re-attach is decided from the transport's recorded ConnectionOutcome, dropped, server_error or unreachable, and an app's own onToolCall or onData throw stays chat.error; the name-prefix classifier is gone from packages/*/src), ai-sdk/src/client/reattach.ts:325-330 (origin/main 56d564d073) (the shared re-attach controller applies that rule) | source, peer-reported | partly-inferred | open |
| CP-169 | S1 | useReanchorOnError: the re-anchor gets no isCurrent / signal, so it treats itself as current through its second snapshot fetch (up to 4 tries with backoff); a send made meanwhile is cut off by stop() or wiped by setMessages(snapshot), and a competing resumeStream starts. The user's new message and its reply are lost. | ai-sdk/src/react/use-reanchor-on-error.ts:79-90 (origin/main 6a2a66817c) (the takeover check runs once, before the second fetch; reanchorOnSnapshot is called with no isCurrent / signal); ai-sdk/src/client/resync.ts:119-135 and :344 (origin/main 6a2a66817c) (isCurrent defaults to !signal?.aborted: always true here; no exclusive lock); contrast: ai-sdk/src/react/use-helix-chat.ts (origin/main 6a2a66817c) (useHelixChat holds the exclusive lock); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !414 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S1 (owner, 2026-10-19); ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-169 among the findings it fixes); conformance: client.stock-reanchor-send-survives passes every id on cf-do and js-chat-host (owner slot s83, 2026-10-20, the owner batch 6f2acd67f8 merged with main 6ae9057c9e (!444), 2 of 2 each). its red: !464's cancelled MR pipeline 2965604898 (job 17302397122, merged result on main 11d0bde621, before !444, 2026-10-20) failed .send-survives on both lanes (the send lost: message false) | source, peer-reported | verified | fixed (2 cells) |
| CP-170 | S2 | useReanchorOnError has no cap: a live run whose resumed stream keeps sending an in-band error that is not a Helix body loops snapshot → resume → error for as long as the run lives (useHelixChat re-anchors after every failed resume with no cap either). | ai-sdk/src/react/use-reanchor-on-error.ts:69-75 (origin/main 6a2a66817c) (no attempt cap, no 'last request was a resume' skip); ai-sdk/src/react/use-helix-chat.ts:554-560 (origin/main 6a2a66817c) (skips this path but re-anchors after each failed resume, uncapped); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !414 incident MR. Fixer: WD (orchestrator routing, 2026-10-19); severity S2 (owner, 2026-10-19); conformance: client.reanchor-cap-stock and client.reanchor-cap-helix fail .reattaches-stop and .exhaustion-reported on cf-do and js-chat-host, 3 of 3 each (owner slot s59, 2026-10-19, cells pin cb1333fc79 = main 6a2a66817c product, no product diff): 7 to 20 resume GETs past the cap of 5, no ReanchorExhaustedError reported (the helix cell judged on useHelixChat's resync.onError since cb1333fc79); WD's gate (tree a1ca68c834, WD's fix + the cells) passes the stock cell 3 of 3 on both hosts; related: the in-band arm (a send or resume answered 200 with nothing but the route's in-band error, no progress) is covered by client.reanchor-cap-inband-stock / -helix on cf-do (the cf-do worker control control-upstream-read refuse + every; js-chat-host a declared gap). It is NOT a reproduction of this record's loop: at main the in-band answer makes 0 resume GETs in 60 s on both arms (the chat stays in its error; no re-attach, no typed exhaustion), while on !444 it makes 1 resume GET and reports ReanchorExhaustedError, 3 of 3 per arm (WD's re-gates on the owner's cells 1a7c51fff7, 2026-10-19, read from the cells' observe lines before an unrelated harness crash, fixed in dfaee856ec). Main staying in a visible error is not a silent failure, so it is not filed; the cells land as coverage of !444's cap; ai-sdk/src/client/reattach.ts:142 (fix evidence, !444, origin/main 5cf0bf0ae9) (createReattachController, the one re-attach controller useHelixChat and useReanchorOnError now share; !444 names CP-170 among the findings it fixes); conformance: client.reanchor-cap-stock and client.reanchor-cap-helix, ledgered red above (.reattaches-stop, .exhaustion-reported), pass every id on cf-do and js-chat-host (owner slot s83, 2026-10-20, the owner batch 6f2acd67f8 merged with main 6ae9057c9e (!444), 2 of 2 each; UNEXPECTED PASS against the ledger, which is removed). The in-band arm (client.reanchor-cap-inband-stock / -helix, cf-do; js-chat-host a declared gap) passes too in the same runs, and is not in provenBy until its red is cited; related: these cells run on e2e's ai 6.0.49 only. On ai 6.0.90 and later the re-anchor's failure was not seen at all (DI-83, a stall, not a loop); its fix keeps one handling per connection so this cap is spent once. Once e2e moves to ai 6.0.170 these cells must stay green; a red there is DI-83's, not this record reopening | source, peer-reported, live-repro | verified | fixed (4 cells) |
| CP-171 | S2 | handleChatStream submits a request's client-tool answers in order and returns on the first one that throws: later answers are never submitted, and because an earlier answer was stored the error is the non-retryable row (don't re-submit), so the never-stored answer is lost and the run waits on that call (on plain runtime-js, with no durable deadline timer, until another request arrives). Fixed in the shared handler by 16ae386528: every answer is submitted, a transient throw is re-submitted once, and a never-stored answer beside a stored one is the typed 409 answer_not_stored naming it. | ai-sdk/src/handler/handle-chat-stream.ts:1134-1152 (origin/main 6a2a66817c) (the catch returns typedEntryErrorResponse at once, inputCommitted: storedSubmit); ai-sdk/src/handler/handle-chat-stream.ts:1067 and :1082-1084 (origin/main 6a2a66817c) (storedSubmit set by an earlier accepted answer); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !403 incident MR. Fixer: W1 (orchestrator routing); severity S2 (owner, 2026-10-19); related: fixed in code by 16ae386528 (the CP-179 / CP-180 / CP-182 chat submit response contract MR), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/handler/handle-chat-stream.ts:1170-1172 (origin/main 56d564d073) (each submit's throw is collected, never returned on), ai-sdk/src/handler/handle-chat-stream.ts:1184-1199 (origin/main 56d564d073) (a transient throw is re-submitted once), ai-sdk/src/handler/handle-chat-stream.ts:1258-1263 (origin/main 56d564d073) (the loop goes on to the request's later answers), ai-sdk/src/handler/handle-chat-stream.ts:1271-1296 (origin/main 56d564d073) (a classified refusal is answered as itself), ai-sdk/src/handler/handle-chat-stream.ts:1305-1316 (origin/main 56d564d073) (a never-stored answer beside a stored one is the typed 409 answer_not_stored naming its call) | source, peer-reported | verified | open |
| CP-172 | S2 | Since !403, re-POSTing an answer to a client-tool call that was already drained is a 409 state_not_resumable instead of a harmless 200: extract-resume-intent checks only the pending map and ownership, never the completed markers, so the repeat is rejected as unknown-tool-call before core's idempotent already_completed path is reached. A lost-200 retry turns into an error mid-turn. Fixed in the shared ai-sdk handler by 16ae386528: a drained call's re-sent answer is submitted and answered already_completed, a 200. | ai-sdk/src/handler/extract-resume-intent.ts:465-469 and :332-335 (origin/main 6a2a66817c) (pendingAt reads only pendingMap / ownership; nothing reads completedClientToolCalls; where === undefined → unknown-tool-call); ai-sdk/src/handler/handle-chat-stream.ts:691-697 and :4502 (origin/main 6a2a66817c) (the rejection becomes state_not_resumable); related: CP-107 (fixed, client side: this request was a harmless 200 before !403); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !403 incident MR. Fixer: W1 (orchestrator routing); severity S2 (owner, 2026-10-19); related: fixed in code by 16ae386528 (W1, the chat submit response contract, CP-180 / CP-179 / CP-182), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/handler/extract-resume-intent.ts:472-480 (origin/main 56d564d073) (pendingAt reads completedClientToolCalls and answers drained for a call with a completed marker, so a re-sent answer is submitted, not rejected unknown-tool-call), ai-sdk/src/handler/handle-chat-stream.ts:1204-1212 (origin/main 56d564d073) (an already_completed answer counts as answered, and a consumed continuation never replaces one another answer decided); related: CP-257: the fix above reads the completed marker, and CF Workflows' drain never writes one, so its 'on every host' does not reach CF Workflows until CP-257 is fixed | source, peer-reported | verified | open |
| CP-173 | S2 | CF DO: the client-tool deadline expiry can store client_tool_timeout on a run that was just stopped. onClientToolDeadline reads the stop synchronously, then awaits expireOverdue, which waits on the resolver's session lock and saves with fenceRunId only (SaveStateOptions has no refuseIfStopped); a stop recorded meanwhile, before its terminal commits, is not seen, so the stopped run's calls are paired as timeouts (Ruling 56: they must stay unpaired). Fixed on the CF DO by 40a6012d92 (the expiry save is stop-guarded in the store's atomic step). | runtime-cloudflare/src/do/durable-object-agent-base.ts:1913-1919 (origin/main 6a2a66817c) (the sync stop read, then await expireOverdue); runtime-js/src/client-tool-resolver.ts:834-840 (origin/main 6a2a66817c) (the save with { fenceRunId } only); core/src/store/state-store.ts:182-188 (origin/main 6a2a66817c) (SaveStateOptions has no stop guard); related: once the stop's terminal commits, the fence refuses the write (do-state-store.ts assertRunIsCurrentLive), so the window is between the stop being recorded and its terminal committing; CP-138 (the deadline alarm), CP-147 (Ruling 56 by another path); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !406 incident MR. Fixer: WA (orchestrator routing); severity S2 (owner, 2026-10-19); related: fixed in code by 40a6012d92 (CP-173 + CP-174 + CP-197), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/durable-object-agent-base.ts:2164 (origin/main 56d564d073) (the DO's deadline alarm expires through expireOverdue fenced on its run), runtime-js/src/client-tool-resolver.ts:867-885 (origin/main 56d564d073) (the expiry save is { fenceRunId, refuseIfStopped: true }, and a CommitStoppedError answers { kind: 'stopped' } with nothing paired), core/src/store/state-store.ts:208-224 (origin/main 56d564d073) (SaveStateOptions.refuseIfStopped, which every store checks in its atomic step) | source, peer-reported | verified | open |
| CP-174 | S2 | CF DO: a client-tool deadline expiry (or its deadline read) that fails permanently is retried every 30 s forever, and a transient one every 1 s, with no attempt budget and no typed terminal: the run stays paused, the error is logged every 30 s, and the client is never told. Fixed on the CF DO by 40a6012d92: a fatal or refused failure ends the run typed at once, a transient one runs under the shared crash-loop budget and then ends framework_auto_resume_exhausted; only a terminal that cannot land is retried at 30 s (gave_up). | runtime-cloudflare/src/do/durable-object-agent-base.ts:1921-1934 and :1864-1874 (origin/main 6a2a66817c) (re-arm at EXECUTION_WATCHDOG_RETRY_MS when permanent, CLIENT_TOOL_DEADLINE_RETRY_MS otherwise; no budget); runtime-cloudflare/src/do/durable-object-agent-base.ts:376 and :395 (origin/main 6a2a66817c) (30_000 and 1_000); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !406 incident MR. Fixer: WA (orchestrator routing); severity S2 (owner, 2026-10-19); related: fixed in code by 40a6012d92 (WA, CP-173 + CP-174, !490 per the verifier), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/durable-object-agent-base.ts:2217-2297 (origin/main 56d564d073) (onClientToolDeadlineFailure classifies a failed deadline fire: fatal or refused ends the run typed at once, at :2246-2249 (origin/main 56d564d073), and transient re-arms under the shared crash-loop budget maxWakeResumeAttempts, at :2252-2289 (origin/main 56d564d073), then ends framework_auto_resume_exhausted), runtime-cloudflare/src/do/durable-object-agent-base.ts:2371 (origin/main 56d564d073) (the typed end is failCommittedRun with dropUnansweredPending), runtime-cloudflare/src/do/durable-object-agent-base.ts:2386-2391 (origin/main 56d564d073) (then settleRun via client_tool_deadline), runtime-cloudflare/src/do/durable-object-agent-base.ts:2462-2482 (origin/main 56d564d073) (a terminal that does not land is gave_up, retried at 30 s and logged at error once per run) | source, peer-reported | verified | open |
| CP-175 | S3 | CF DO: the deadline expiry times out every lapsed unsubmitted entry in the session, any run's, while the alarm's own deadline counts only the current run's: a leftover entry of another run (one that survived a rewind) is timed out under the current run's fence. | runtime-js/src/client-tool-resolver.ts:816-829 (origin/main 6a2a66817c) (filters on submittedAt and deadlineAt only; pre-submit stubs carry submittedAt, so they are skipped); contrast: runtime-cloudflare/src/do/wake-recovery.ts:298-304 (origin/main 6a2a66817c) (entry.runId !== currentRun.runId); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !406 incident MR. Fixer: WA (orchestrator routing); severity S3 (owner, 2026-10-19) | source, peer-reported | verified | open |
| CP-176 | S2 | runtime-js (and the CF DO): a remote child that answers interrupted or paused (or running with a dead producer) counts as still running, so every resume or retry that carries a message is refused awaiting_remote_child; only a thrown getStatus reaches the deadline rule, and reconcile leaves the ref running. A new turn escapes it (the orphan path), so it is a near-wedge for resume / retry. Fixed on runtime-js and the CF DO by 0889c0ee05 (!449): an interrupted child settles its call failed, and running or paused is owed only inside the child deadline. | core/src/orchestration/remote-children-before-input.ts:144, :163 and :181-196 (origin/main 6a2a66817c) (throw AgentAlreadyRunningError(... 'awaiting_remote_child')); runtime-js/src/js-agent-executor.ts:2601-2618 (origin/main 6a2a66817c) (the non-terminal branch leaves the ref running); related: CP-157 (a paused remote child fails the parent's call), CP-45; related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !412 incident MR. Fixer: WE-2 (orchestrator routing); severity S2 (owner, 2026-10-19); runtime-js/src/js-agent-executor.ts:2614 (fix evidence, !449, origin/main 0889c0ee05) (reconcileRemoteSubAgents) and core/src/orchestration/remote-children-before-input.ts:211 (origin/main 0889c0ee05) (the entry settle's remoteChildVerdict): !449 (Invariant S, one result per remote call) names CP-176 among the findings it fixes. Not witnessed: no conformance cell yet (FU-CONF-449-CELLS); related: fixed in code by 0889c0ee05 (WE-2, !449, Invariant S), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/lost-remote-child.ts:162-167 (origin/main 56d564d073) (an interrupted remote child is a failed verdict, so its call is settled, not owed) and :168-172 (origin/main 56d564d073) (running or paused is owed only inside the child deadline, then lost deadline_exceeded), so core/src/orchestration/remote-children-before-input.ts:273-288 (origin/main 56d564d073) (the awaiting_remote_child refusal) is bounded; runtime-js/src/js-agent-executor.ts:2882 (origin/main 56d564d073) (reconcile uses the same remoteChildVerdict, which the CF DO inherits through runtime-js) | source, peer-reported | verified | open |
| CP-177 | S2 | runtime-js (and the CF DO): one RemoteAgentNotFoundError settles a remote child lost at once, with no age check, though the DO answers not_found also while the child's /start has not committed yet: a parent that wakes while its child's /start is still committing marks the ref failed, and the child then runs to completion unsupervised, its result dropped. Fixed on runtime-js and the CF DO by 0889c0ee05 (!449): not_found is lost only past the child deadline or once the start is confirmed. | core/src/orchestration/lost-remote-child.ts:38-40 (origin/main 6a2a66817c) (not_found → lost); runtime-cloudflare/src/do/durable-object-agent-base.ts:4016-4021 (origin/main 6a2a66817c) ('its /start never committed (not yet, or never)'); related: CP-141; related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !412 incident MR. Fixer: WE-2 (orchestrator routing); severity S2 (owner, 2026-10-19); runtime-js/src/js-agent-executor.ts:2614 (fix evidence, !449, origin/main 0889c0ee05) (reconcileRemoteSubAgents) and core/src/orchestration/remote-children-before-input.ts:211 (origin/main 0889c0ee05) (the entry settle's remoteChildVerdict): !449 (Invariant S, one result per remote call) names CP-177 among the findings it fixes. Not witnessed: no conformance cell yet (FU-CONF-449-CELLS); related: fixed in code by 0889c0ee05 (!449, Invariant S), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/lost-remote-child.ts:139-143 (origin/main 56d564d073) (a RemoteAgentNotFoundError is lost only past the child deadline or once the start is confirmed, else owed), runtime-js/src/js-agent-executor.ts:2882-2890 (origin/main 56d564d073) (reconcileRemoteSubAgents passes startConfirmed only for a ref with a recorded remote or a non-live status), core/src/orchestration/remote-children-before-input.ts:217-219 (origin/main 56d564d073) (the entry settle uses the same verdict) | source, peer-reported | verified | open |
| CP-178 | S2 | runtime-js (and the CF DO): settling a remote child lost (including status_unreachable) never interrupts it, so a partitioned producer keeps running the child, outside the parent's stop cascade (the ref is failed), spending on a result nobody reads. Fixed on runtime-js and the CF DO by 0889c0ee05 (!449): every lost settle interrupts the child, best-effort and bounded, or keeps its stop owed. | runtime-js/src/js-agent-executor.ts:2619-2661 (origin/main 6a2a66817c) and core/src/orchestration/remote-children-before-input.ts:163-179 (origin/main 6a2a66817c) (no interrupt on the lost path); contrast: runtime-js/src/js-agent-executor.ts:2467-2470 (origin/main 6a2a66817c) (orphanRemoteChild interrupts); related: CP-152; related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !412 incident MR. Fixer: WE-2 (orchestrator routing); severity S2 (owner, 2026-10-19); runtime-js/src/js-agent-executor.ts:2614 (fix evidence, !449, origin/main 0889c0ee05) (reconcileRemoteSubAgents) and core/src/orchestration/remote-children-before-input.ts:211 (origin/main 0889c0ee05) (the entry settle's remoteChildVerdict): !449 (Invariant S, one result per remote call) names CP-178 among the findings it fixes. Not witnessed: no conformance cell yet (FU-CONF-449-CELLS); related: fixed in code by 0889c0ee05 (WE-2, !449, Invariant S), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/remote-children-before-input.ts:248-251 (origin/main 56d564d073) (a lost verdict, status_unreachable included, settles the ref failed and pushes it to the entry's interrupts), runtime-js/src/js-agent-executor.ts:2581-2603 (origin/main 56d564d073) (each settled-lost child gets a bounded best-effort interrupt through interruptLostRemoteChild), runtime-js/src/js-agent-executor.ts:2958-2966 (origin/main 56d564d073) (reconcileRemoteSubAgents interrupts a child it settles lost), runtime-js/src/js-agent-executor.ts:2738-2756 (origin/main 56d564d073) (a lost-step child is stopped, or its settle keeps stopOwed for a later retry); the CF DO runs this executor | source, peer-reported | verified | open |
| CP-179 | S2 | handleChatStream answers a 200 that carries only data-resume-rejected when one answer of a request is stored and another names a call that is STILL PENDING (a stale-run rejection, which fires only after the call was found pending, or a malformed approval id, rejected before any lookup): the continuation is awaiting_batch, so the run stays paused on the rejected call while the client, which reads no data-resume-rejected, believes it answered. useHelixChat never re-sends (its auto-send dedupes by toolCallId), so the run waits until the CF DO's client-tool deadline (client_tool_timeout), or unbounded on runtime-js; when EVERY answer is rejected the request is a typed 409 and the client re-reads the snapshot. Fixed in the shared ai-sdk handler by 16ae386528: such a request is now the typed 409 state_not_resumable / answer_not_stored naming the calls. | ai-sdk/src/handler/handle-chat-stream.ts:1159-1171 and :1188-1199 (origin/main 6a2a66817c) (typed only when every answer failed; otherwise the gathered rejections go to rejectedOnlyResponse when nothing continues); ai-sdk/src/handler/handle-chat-stream.ts:4577-4588 (origin/main 6a2a66817c) (rejectedOnlyResponse: a 200 of data-resume-rejected only); ai-sdk/src/handler/extract-resume-intent.ts:284-286, :290-305 and :350-363 (origin/main 6a2a66817c) (malformed id before the lookup; stale-run only for a call pendingAt found); ai-sdk/src/react/send-automatically-when.ts:142-155 (origin/main 6a2a66817c) (useHelixChat's auto-send dedupes by toolCallId: no re-send); inferred: (unverified) reachability: a stale-run needs a client part stamped with an older run while the call is pending under the current one (e.g. after StartRun.pendingRestamp moved a surviving call to a new run); a malformed id is a client bug only. Not traced end to end; related: the fix direction (orchestrator ruling 2026-10-19, a direction only; W1's design note decides the contract): when a rejected answer names a still-pending call, answer the request typed (409 state_not_resumable listing the rejected ids, the non-retryable row since a sibling is inputCommitted), so the client re-reads the snapshot, which shows the call still awaiting its answer; related: the half of WD's partial-200 item that is by design for the SERVER's state (orchestrator ruling): a rejection of a call pending nowhere (unknown-tool-call, including an already-drained call, CP-172) or of a stopped run (run_stopped, Ruling 49b / 56) leaves no wait on the server. Its client half is CP-180 (the stock re-POST loop); CP-03 (fixed: the all-failed case is typed); related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !403 incident MR. Fixer: W1 (orchestrator routing); severity S2 (owner, 2026-10-19); related: fixed in code by 16ae386528 (W1, the chat submit response contract, CP-180 / CP-179 / CP-182), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): ai-sdk/src/handler/handle-chat-stream.ts:1297-1316 (origin/main 56d564d073) (case C: a stale-run or malformed-approval-id rejection of a call the server still awaits, or a transiently failed submit, beside a stored sibling answer is the typed 409 answer_not_stored naming the calls, never a 200 of rejections only), ai-sdk/src/handler/handle-chat-stream.ts:4913 (origin/main 56d564d073) (answerNotStoredResponse builds that 409) | source, peer-reported | partly-inferred | open |
| CP-180 | S1 | A STOCK useChat with ai's lastAssistantMessageIsCompleteWithToolCalls auto-send loops forever when the chat route answers a client-tool answer with a 200 that changes nothing: ai re-sends after every non-error finish while the last step's client tools are all answered, and handleChatStream answers a request where nothing continues with a 200 of data-resume-rejected only (rejectedOnlyResponse) or an empty one (emptySSEResponse), so each re-POST gets the same 200. The loop never ends, never surfaces an error, sends one request per round trip, and appends one more data-resume-rejected part each round. The reachable state found on main is a step with a client call beside an approval-gated server call, on a page RELOADED while both pend: the rehydrated gated part is providerExecuted, so the stock predicate skips it and sends the client answer while the batch awaits the approval (awaiting_batch). | ai-sdk/src/handler/handle-chat-stream.ts:1168-1171, :1188-1199, :4577-4588 and :4594-4602 (origin/main 6a2a66817c) (the rejected-only and empty 200s); ai-sdk/src/handler/handle-chat-stream.ts:1188, :1217, :4461 and :4478 (pre-fix, origin/main b93c58c53d) (the same 200s before !403: the loop predates it); ai@6.0.49 dist/index.mjs:12307 and :12384-12400 (makeRequest re-sends when sendAutomaticallyWhen holds after a non-error finish; lastAssistantMessageIsCompleteWithToolCalls holds while the last step's non-provider-executed tool parts are output-available / output-error); live: owner, 2026-10-19, lock-free Node run: ai 6.0.49's AbstractChat + DefaultChatTransport + lastAssistantMessageIsCompleteWithToolCalls, an addToolOutput answer, against a fetch stub answering with the REAL sse-builder framing of rejectedOnlyResponse ('data: {type: data-resume-rejected, …}' then [DONE]) or emptySSEResponse ([DONE] only): 25 POSTs until the stub's cap answered 500, 25 data-resume-rejected parts on the message; the same chat with createHelixSendAutomaticallyWhen: 1 POST; a 409 answer: 1 POST. The rejected-only stream is well-formed for ai 6.0.49 (each chunk passes uiMessageChunkSchema; no start / finish is needed: the chat ends ready, no error). A stub, not the chat route: the conformance cell is owed; related: CP-179 (same response seam; fixer W1 designs both together, and the design must hold for stock useChat, not only useHelixChat, orchestrator ruling 2026-10-19); CP-03 (fixed: the all-failed case); stock compatibility is a program priority; severity S1 (owner): an unbounded, self-sustaining request loop on the stock client that surfaces nothing to the user, grows the message every round and loads the server, reachable from a lost-200 re-POST; related: WD's retro F4 review of the merged incident MRs (2026-10-19; orchestrator worktree tmp/reliability-investigation/audits/2026-10-19-incident-mrs-retro-f4-t3.md), re-verified by the owner at main 6a2a66817c; the !403 incident MR. Fixer: W1 (orchestrator routing); severity S1 (owner, 2026-10-19); related: the live stream does NOT reach it (owner slot s56, 2026-10-19: the cell's first, live shape never POSTed): a live tool part is providerExecuted only once its output arrives (the tool_end chunk's flag, forwarded on tool-output-available, ai-sdk/src/transformer/stream-transformer.ts:1075-1077 (origin/main 6a2a66817c)); its tool-input-available carries the flag only through StreamTransformerOptions.findTool (:976-1002 (origin/main 6a2a66817c)), which no product code passes. So a live approval-gated part in approval-requested is unflagged, the stock predicate counts it and waits (conservative). Rehydrated from the snapshot the same part is flagged (the converter, from the persisted call), which is what lets the stock predicate send; conformance: client.stock-autosend-no-repost-loop fails .no-repost-loop on cf-do and js-chat-host, 3 of 3 each (owner slot s59, 2026-10-19, cells pin cb1333fc79 = main 6a2a66817c product, no product diff): after the reload the stock app's one answer is POSTed 19 times (cf-do) and 8 times (js-chat-host) within 5 s, every one answered 200; .suspended and .answer-posted pass. The reload shape loops on the real route (it was inferred before this run); conformance: reshaped with the fix's design (cells 268a781b97: .answer-given, the reloaded app answered the client call locally, then .no-repost-loop asserts NO chat POST carries the answer within 5 s while the approval pends, as live): fails on main on cf-do and js-chat-host 3 of 3 each (owner slot s61, cells pin 268a781b97 = main 6a2a66817c product: dozens of answer POSTs, every one 200), and passes 3 of 3 on both hosts on W1's head 5bd9b127c0 with cells c7bb69e6ac (W1's gate gC180); the guard client.stock-autosend-approval-continues (the approval sent once given, the run completes) passes on main and on the fix; related: fixed by !453 (W1, merged as 16ae386528): a submit on which nothing continues for this client is answered by a 200 that ENDS the client's tool step (ai-sdk/src/handler/handle-chat-stream.ts:4320-4346 (origin/main 16ae386528), stepEndingResponse at :4347: start, start-step, the re-anchor triggers, finish-step, finish), so a stock useChat's sendAutomaticallyWhen predicate reads an empty step and does not send the same request again; and a reloaded approval part carries no providerExecuted, so the tool-calls predicate no longer sends a client answer beside a waiting approval (ai-sdk/src/handler/snapshot.ts:236-247 (origin/main 16ae386528), markPendingCalls); conformance: client.stock-autosend-no-repost-loop passes on cf-do and js-chat-host (client tier) on !453's fix: the cells were judged UNEXPECTED-PASS in !464's MR pipeline 2968169157 (job 17320844243, test:conformance 1/5, at 2372bd0e2f over main 16ae386528), so the ledger is replaced by provenBy | source, peer-reported, live-repro | verified | fixed (2 cells) |
| CP-181 | S3 | Persistent child session ids are not injective, so two different parents' companions can share one child session in a nested companion tree. The id is ${parentSessionId}-agent-${name} and a companion name is free text: when a parent p spawns a companion named x-agent-y while p's own companion x (session p-agent-x) spawns a child named y, both derive p-agent-x-agent-y, so two parents' entries read and write the same child session (its transcript, its run, its refs). | core/src/types/session.ts:1123-1125 (origin/main 459fa678ca) (generatePersistentSubSessionId: ${parentSessionId}-agent-${name}); core/src/tools/companion/spawn-agent.ts:11-19 (origin/main 459fa678ca) (the spawn's name is z.string().optional(): any text, a '-agent-' included); runtime-dbos/src/persistent/child-session-id.ts:5 (origin/main 459fa678ca) (DBOS derives the same {parentSessionId}-agent-{name} format); inferred: (unverified) the ephemeral sub-session id ${parentSessionId}-sub-${toolCallId} (core session.ts:1110-1112) can collide with a persistent one across kinds the same way (a companion named x-sub-c of p, and the ephemeral sub-agent call c of p's companion x, both derive p-agent-x-sub-c); provider tool-call ids rarely carry such text; related: WA (peer-reported, 2026-10-19, at the orchestrator's request: traced by reading in CP-161 fix round 2, design §7.3, not reproduced), the id format and the free-text name verified by the owner at main 459fa678ca; severity S3 (owner, 2026-10-19): the consequence is serious (two parents share one child session), but it needs a model-chosen name containing the separator inside a nested companion tree; an injective encoding (an escaped name, or a hashed one) closes it | source, peer-reported | partly-inferred | open |
| CP-182 | S2 | A stock useChat app on ai 6.0.0 to 6.0.91 that auto-sends on ai's lastAssistantMessageIsCompleteWithApprovalResponses never sends an approval given after a page reload: the rehydrated approval-gated part is providerExecuted: true, and those ai versions drop provider-executed parts from that predicate, so it never holds and the run stays paused on the approval. The live part carries no flag until its output arrives, so the same approval sends before a reload; ai 6.0.230 and later count the part. | ai-sdk/src/converter/helix-to-aisdk-converter.ts:195-206 (origin/main f0b3da371e) (a rehydrated tool part carries the persisted providerExecuted, true for a framework-executed, approval-gated server tool); ai@6.0.0 dist/index.mjs:11581, ai@6.0.49 dist/index.mjs:12373 and ai@6.0.91 dist/index.mjs:12876 (lastAssistantMessageIsCompleteWithApprovalResponses filters out part.providerExecuted before counting approval-responded parts); contrast: ai@6.0.230 dist/index.mjs:14022 and ai@6.0.281 dist/index.mjs:14786 (the same predicate without that filter); every one of these is a supported ai version (DI-20 pins 6.0.0, 6.0.49, 6.0.91, 6.0.230 and 6.0.281); related: ai's lastAssistantMessageIsCompleteWithToolCalls never counts an approval-responded part (ai@6.0.49 dist/index.mjs ~12384-12400: only output-available / output-error), so an approval-only step is sent by the approval predicate alone; related: CP-180 (the same rehydration seam: the rehydrated gated part is providerExecuted, so the stock tool-calls predicate skips it and re-sends a client answer); raised by W1 while designing CP-180 B2 (peer-reported, 2026-10-19), verified by the owner in the workspace node_modules; severity S2 (owner, 2026-10-19): a stock app that reloads while an approval is pending cannot continue the run from its UI on three supported ai versions (the workspace pins 6.0.49), with nothing reported; conformance: client.stock-approval-predicate-after-reload fails on main on js-chat-host 3 of 3 on its own ids (W1's gate gC180, 2026-10-19, cells c7bb69e6ac against main's ai-sdk product): .approval-shown holds, .approval-posted sees no chat POST carrying the approval within 30 s and .run-completes finds the session still paused; on cf-do (owner slot s61, cells pin 268a781b97 = main 6a2a66817c product) .approval-posted failed 3 of 3 the same way before the cell's own status check crashed (fixed at c7bb69e6ac, so cf-do's red on the fixed cell is owed); conformance: the fix (W1's CP-180 / CP-182 MR, head 5bd9b127c0 with cells c7bb69e6ac; the snapshot renders a pending approval part without providerExecuted, as live) passes the cell 3 of 3 on cf-do and js-chat-host, beside client.stock-autosend-no-repost-loop and client.stock-autosend-approval-continues (W1's gate gC180); conformance: the fixed cell (cells c7bb69e6ac) fails on main on cf-do and js-chat-host 3 of 3 each (owner slot s63, slot B, 2026-10-19, main 6a2a66817c product): .approval-shown holds, .approval-posted sees no chat POST carrying the approval within 30 s and .run-completes finds the session still paused; the third run finished after the slot's 5 min cap (the cap did not stop it), with the same verdicts; related: fixed by !453 (W1, merged as 16ae386528): the snapshot marks a reloaded approval part without providerExecuted, as on the live stream, so ai's stock sendAutomaticallyWhen approval predicate counts it after a reload (ai-sdk/src/handler/snapshot.ts:236-247 (origin/main 16ae386528), markPendingCalls); conformance: client.stock-approval-predicate-after-reload passes on cf-do and js-chat-host (client tier) on !453's fix: the cells were judged UNEXPECTED-PASS in !464's MR pipeline 2968169157 (job 17320844243, test:conformance 1/5, at 2372bd0e2f over main 16ae386528), so the ledger is replaced by provenBy | source, peer-reported, live-repro | verified | fixed (2 cells) |
| CP-183 | S1 | The CF Durable Object's remote-protocol routes /subagent/:agentType/sse, /status, /interrupt and /abort ignore :agentType: they act on whatever session the addressed DO holds. Behind createRemoteAgentWorkerHandler, whose agents allowlist is the documented type gate (and whose guide says the DO validates :agentType against the session's stored agent type), a caller admitted for type X can read the status and the live stream of, and interrupt or abort, a session of any other type on the same namespace by naming its session id in an allowlisted type's route. | runtime-cloudflare/src/do/durable-object-agent-base.ts:3996-4067 (origin/main cd2d6e3169) (handleSubAgentRoute: 'sse' → handleSSE(request) :4008-4009, 'status' → handleStatus() :4011-4012, 'interrupt' / 'abort' → stopRoute(kind, request, 'subagent') :4063-4067; none reads agentType); contrast: runtime-cloudflare/src/do/durable-object-agent-base.ts:4128 and :4233 (origin/main cd2d6e3169) (the snapshot and usage routes answer NOT_FOUND when the session's stored agent_type is not :agentType), and start / resume route by the agent type; related: the documented contract, docs/guide/cross-service-remote-agents.md:125-126 (origin/main cd2d6e3169) (the mount forwards every agentType to the DO, "which validates it against the session's stored agent_type"; the agents allowlist is the mount's own type gate); related: WE-2 (peer-reported, 2026-10-19, via the orchestrator), found while adding the type-gated /subagent/:type/submit-tool-result route (typed 409 state_agent_type_mismatch); verified by the owner at main cd2d6e3169; severity S1 (owner, 2026-10-19): a documented authorization boundary (the agentType allowlist, the DO's type validation) is bypassed for reading another type's live stream and status and for stopping its runs; the caller still needs the mount's auth and the target session id, so it is not an anonymous exploit; conformance: remote.subagent-type-gated-host (owner cell, conformance-owner-cp183-cell) fails .status-refused, .stream-refused, .interrupt-refused, .abort-refused and .run-untouched on cf-do 3 of 3 against main's DO base, and passes 3 of 3 on the fix's head 9e413f47a8 (!459, WE-2's gate, 2026-10-19 18:27-18:28); WE-2's per-route workerd case on main's base: status 200, sse 200 (the live stream served), interrupt 202 (the foreign type stopped the run), abort 409; runtime-cloudflare/src/do/durable-object-agent-base.ts:4009 and :4093 (fix evidence, !459, origin/main 3a7c0bf7e7) (handleSubAgentRoute calls subAgentTypeGate first: a session of another type answers 404 NOT_FOUND on /status, /sse, /interrupt and /abort, and 409 state_agent_type_mismatch on the entries), with runtime-cloudflare/src/do/subagent-routes.ts (the route actions); conformance: remote.subagent-type-gated-host passes every id on cf-do in !459's MR pipeline 2965571250 (job 17302195268, merged result on main, 2026-10-20), and in WE-2's slot 3 of 3 at !459's head 9e413f47a8. With only durable-object-agent-base.ts reverted to main's (the gate reversed, runtime-cloudflare rebuilt) it fails .status-refused, .stream-refused, .interrupt-refused, .abort-refused and .run-untouched 3 of 3, and .run-held passes (WE-2, 2026-10-19, peer-reported and verified by the owner from its result files). Its js-chat-host lane is a declared gap | source, peer-reported | verified | fixed (1 cells) |
| CP-184 | S3 | On CF Workflows a stop cascade's write of a local child's terminal to the parent's sub-session ref is not atomic with the child's own terminal: the cascade awaits the signalled child's terminal, then writes the parent's ref in a separate store write, and stamps stopOwed only when the wait times out. A process that dies in between (an executor-site stop dying mid-wait, or the workflow body's stop-children step lost with its instance after the child ended) leaves the ref running with no stopOwed, and nothing heals it: a later stop reads the child's run as not live and writes nothing, the companion sync leaves an interrupted child's ref running, and a later companion__sendMessage meets a running ref over an ended child. | runtime-cloudflare/src/stop-children.ts:342 (origin/main 12e684af5f) (fix evidence, !452, 8c0e6a8248) (awaitChildTerminal: the cascade waits for a signalled local child's own terminal); runtime-cloudflare/src/stop-children.ts:286-335 (origin/main 12e684af5f) (fix evidence, !452, 8c0e6a8248) (recordChildTerminalRef: the parent ref is written from that terminal in its own updateSubSessionRef, after the child's terminal committed); runtime-cloudflare/src/stop-children.ts:368-412 (origin/main 12e684af5f) (fix evidence, !452, 8c0e6a8248) (stampLocalStopOwed: stopOwed is stamped only past the wait bound, so a crash during the wait leaves no stamp); related: W1 (peer-reported, 2026-10-19, filed as FU-CFW-STOP-CASCADE-CHILD-REF-CRASH-WINDOW at the orchestrator's request, next to FU-CFW-STOP-CASCADE-SPAWN-IN-FLIGHT); verified by the owner at !452's head 8c0e6a8248. Fix direction (W1): the generation-scoped ref compare-and-set of WE-2's !449 / WA's CP-161 carrying the terminal, or a reader-side reconcile of a local ref whose child run ended; severity S3 (owner, 2026-10-19): crash-only and CF Workflows only, but the stale running ref over an ended child is permanent and can wedge that companion's name for later consults; conformance: companion.stop-cascade-ref-write-lost (pins 2b, the cascade retry) and companion.stop-cascade-ref-write-retries-spent (pins 2a, the reader fold, with the cascade's 3 ref-write attempts spent; its precondition .ref-stranded green) are red on cfw-workflows-d1 3 of 3 each in the owner's batch gate s92 (2026-10-20, the batch head c3e7f5f492 over !464 / main 16ae386528): -lost fails exactly .ref-not-stranded (the parent's ref for the companion that ended interrupted stays running) and .later-consult-heals (after a later consult the turn completes but the ref still reads running), -retries-spent fails exactly .later-consult-heals. WA's x1 on CP-161's head efd2866c08 matched. The fixer W1 runs both as the red-first witness of the CP-184 fix (design C); conformance: fixed by this MR (!492, W1; the fix head 4d942b2dda on branch reliability-cf-w1-cp184-ref-heal-v2): companion.stop-cascade-ref-write-lost (2b, the cascade retries the parent-ref write) and companion.stop-cascade-ref-write-retries-spent (2a, the later consult's reader heal once the retries are spent) pass on cfw-workflows-d1 3 of 3 at the fix (W1's gate g184, judged on meta.conformance.failedIds: failedIds [] each run), and fail exactly their former ledgered ids at the pre-fix product (-lost: .later-consult-heals and .ref-not-stranded; -retries-spent: .later-consult-heals). Red-first unit tests: 2a 3773438daf / 001f9d0e01, the always-bump contract 16aa6cd856 / 2ac24badd6, 2b 9ccf2c17db / c8b4bc8e36, runtime-js 9331280726 / f637ea828b (peer-reported by W1) | source, peer-reported | verified | fixed (2 cells) |
| CP-185 | S3 | CP-157 MR 1 (remote-child suspension, merged as !504): when a root parked on a paused remote child forwards a submit that the producer stores and continues, but the root's own continuation never runs and nobody re-sends, a reload re-presents the call (the root snapshot still lists it), and a DIFFERENT re-answer (deny after an earlier approve) is forwarded, answered already_completed and silently discarded: the producer keeps the first answer and nothing tells the user the second choice was ignored. Closed by CP-157 MR 2's durable poll. | core/src/client-tool/route-submit.ts:643-701 (origin/main 56d564d073) (routeParkedRemoteSubmit, merged with CP-157 MR 1 as 47dd092fb9: a root stop wins first, then an entry whose run is gone is refused unknown_tool_call (reason parked_call_stale, :674-681 (origin/main 56d564d073)), then the root stamps answeredAt (:696-701 (origin/main 56d564d073)) and forwards the submit to the producer; the stamp feeds only the re-park spin bound, and nothing compares a later re-answer with the first, so a different re-answer is still forwarded and answered already_completed); related: docs/superpowers/specs/2026-10-19-cp157-remote-child-suspension-design.md §4.3 step 2, the crash windows, window 3 (as on branch reliability-we-cp157-remote-park fec5d5c49e): forwarded and the producer continued, then the root continuation fails or the request dies with no re-send; nothing re-drives the root in MR 1 (no producer to consumer callback), a reload re-presents the call, and the re-answer is answered already_completed; core/src/__tests__/client-tool/route-submit-remote.test.ts:594 (origin/main 56d564d073) (WE-2's MR 1 unit pin, merged: 'window 3 (MR 1, due with the MR 2 poll): a CONFLICTING re-answer is answered already_completed and the producer keeps the first answer': today's behaviour, pinned until MR 2); related: orchestrator ruling (2026-10-19): file it S3 with CP-157, due with CP-157 MR 2's durable poll, not blocking MR 1; fixer WE-2; severity S3 (owner, 2026-10-19): needs a forward that lands, a root continuation that never runs, no re-send and a reload with a different answer; the run itself continues on the first answer | source, peer-reported | verified | open |
| CP-186 | S1 | A reload of a session whose LOCAL sub-agent awaits an approval errors the stream in a stock useChat, so Approve / Deny never shows and the approval cannot be answered: the root snapshot has no tool part for the child's approval-gated call (buildSnapshot converts only the root's messages and marks only the root's pendingClientToolCalls), while the resume re-sends the child's tool-approval-request from its skip range, and ai's getToolInvocation then throws 'No tool invocation found for tool call ID'. The root's own approval-gated call reloads fine. Also on CF Workflows (WE-2's real-engine test, red at main's snapshot.ts): the record first called CFW unaffected. | ai-sdk/src/handler/snapshot.ts:189-191 (origin/main 4980e8a8fb) (markPendingCalls runs over the root messages with view.session.pendingClientToolCalls, the root session only), :239-244 (origin/main 4980e8a8fb) (markPendingCalls); ai-sdk/src/handler/handle-chat-stream.ts:4543-4558 (origin/main 4980e8a8fb) (flushSkippedApprovals re-sends a tool_approval_request at or before the resume cursor, after the skip range; the skip range drops the tool-input-available the transformer announced from that request, so no part precedes it), called at :4670 and :4684 (origin/main 4980e8a8fb); live: WE-2's probe (2026-10-19) on real code: a runtime-js run (MockLLMAdapter, store-memory), the real buildSnapshot / snapshotResumeOptions / handleChatStream and a stock ai AbstractChat with DefaultChatTransport (replayPrefix true), on ai 6.0.49 and 6.0.281. Root approval-gated call: chat ready, part approval-requested. A local child's (parent, subagent__child, deploy with requireApproval): root suspended_awaiting_children, snapshot paused, NO part for the child's call (only the parent's sub-agent call as input-available), and chat.resumeStream() ends in status error with 'No tool invocation found for tool call ID'. The child wrote tool_approval_request with no tool_start before it; related: WE-2's ai-sdk real-chat red (orchestrator relay, 2026-10-19): 9 of 9 'No tool invocation found' across the aliased ai 6.0.0 to 6.0.281, and useHelixChat the same; not re-run by the owner; related: ai 6.0.49 dist/index.mjs getToolInvocation throws for an unknown tool call id (peer-reported by WE-2, not re-read by the owner); related: peer-reported by WE-2 (2026-10-19; orchestrator: incident-class, likely S1); the snapshot and skip-phase source verified by the owner at main 4980e8a8fb. Fixer WE-2 (a red in ai-sdk on its own branch, then the fix MR). Side observation, not filed: the parent writes subagent_end while the child is still suspended; live: CF Workflows IS affected (correcting the record's first contrast, which called it unaffected from WE-2's 2026-10-19 run): WE-2's real-engine e2e, ported by W1 in !517 (e2e/src/__tests__/sub-agent-approval-reload-cfw-engine.wf.test.ts:78-100, as on branch reliability-cf-w1-cp186-local-child-approval-reload 316c3d252e: the real CF Workflows engine through the chat route, 3 ai profiles x approve / deny), is red 6 of 6 with main's snapshot.ts as a mutation and green 3 of 3 with the fix (W1 slot g186, 2026-10-25, peer-reported); no conformance lane drives a CF Workflows chat host, so this test is the CFW witness; related: CP-10 (a local child's answered call is never driven by the root's resume, fix W1's !421): CP-186's .run-completes stays red on WE-2's CP-186 fix until CP-10 is fixed; the CP-186 MR stacks after it (orchestrator, 2026-10-19); severity S1 (owner, 2026-10-19): a reload, the first thing a user does when a chat looks stuck, makes a pending approval of a local sub-agent unanswerable in a stock useChat, with only a client-side stream error to show for it; conformance: subagent.remote-child-approval-parks-parent-reload-client (CP-157's reload cell; on js-chat-host its child and grandchild run in process, so the approval is a LOCAL grandchild's, two levels deep) fails .approval-renders-after-reload, .approval-posted and .turn-completes on js-chat-host (client, a stock useChat app), 3 of 3 at WE-2's CP-157 head dfbe79516c with owner cells d6e009926c (owner slot s119, 2026-10-24), and the same 3 of 3 at 00134b3a9c (s118) and with the product at main 64110bd5ac (s118 red phase): the approval renders before the reload and not after it; its preconditions .parent-parked and .approval-renders pass; ai-sdk/src/handler/snapshot.ts:493-612 and :626-657 (fix: !517, W1, 50cbf62afd and 6ed0278b2e; as on branch reliability-cf-w1-cp186-local-child-approval-reload 316c3d252e) (localChildPendingCalls: the snapshot renders every local sub-agent's pending call, at any depth, placed by the root's sub-session ref and found through the root's clientToolCallOwnership, so a reload shows Approve / Deny exactly when the submit router can take the answer; insertLocalChildCallParts inserts it as the part the live stream would have); conformance: fixed. subagent.remote-child-approval-parks-parent-reload-client passes on js-chat-host (client, a stock useChat app), 3 of 3, at !517's head 316c3d252e (owner slot s141, 2026-10-25): the approval renders after the reload, is posted and the turn completes; CF Workflows is witnessed by the engine test above (no CFW client lane) | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-187 | S3 | A parent whose LOCAL sub-agent suspends on an approval writes the sub-agent's subagent_end on its stream while the child is still suspended (its ref paused_awaiting_client, the root suspended_awaiting_children), so a client that reads subagent_end as the sub-agent finishing renders the delegation as done while it awaits an answer. | core/src/orchestration/record-sub-session-result.ts:310-321 (origin/main 4980e8a8fb) (the parent stream's subagent_end, written with the sub-session's result before its tool_end; which caller reaches it for a suspended child is not traced by the owner); live: WE-2's probe (2026-10-19) on a real runtime-js run, parent, subagent__child (local, ephemeral), approval-gated deploy: the root parks suspended_awaiting_children with the child's ref paused_awaiting_client, and the root stream carries, in order, the parent's run_started, step_start, step_end and subagent_start (sub1), the child's run_started, step_start, step_end, tool_approval_request and step_committed, then the parent's subagent_end (sub1), step_committed and run_paused; related: peer-reported by WE-2 (2026-10-19, on the orchestrator's go), found beside CP-186; orchestrator: not fixed in CP-186's MR unless its red shows it on that path; severity S3 (owner, 2026-10-19): a wrong live activity state (a delegation shown finished while it awaits an approval); the approval itself, the run and the persisted history are not affected by it | peer-reported, live-repro | partly-inferred | open |
| CP-188 | S2 | A Durable Object agent configured WITHOUT subAgentNamespace accepts an ephemeral createSubAgentTool silently: the DO neither rewrites it into a remote child nor refuses it, so the sub-agent would run in process on the parent's DOStateStore, which holds one session per DO and ignores sessionId, writing the child's state over the parent's (the code's own comment: it would corrupt the single-session SQLite state). A persistent companion is refused there (framework_not_supported); an ephemeral sub-agent is not. | runtime-cloudflare/src/do/durable-object-agent-base.ts:3282-3291 (origin/main 4980e8a8fb) (rewriteSubAgentTools returns the agent unchanged when subAgentNamespace is absent; its doc comment at :3286-3288 states that a local sub-agent would run in the same DO instance and corrupt the single-session SQLite state); runtime-cloudflare/src/do/do-state-store.ts:3580-3584 (origin/main 4980e8a8fb) (exists ignores sessionId: one session per DO, state row id = 1), :3595-3598 (origin/main 4980e8a8fb) (the session-per-DO model); inferred: (unverified) that the in-process child's writes land on the parent's single state row and corrupt it; no run has been observed. Peer-reported by WE-2 (2026-10-19, code read beside CP-186), the accept-without-refusal verified by the owner at main 4980e8a8fb; related: orchestrator (2026-10-19): file it; CP-186 (the local sub-agent reload, found in the same review); severity S2 (owner, 2026-10-19): silent corruption of a session if it happens, but it needs a DO configured with sub-agent tools and no subAgentNamespace | source, peer-reported | partly-inferred | withdrawn — A duplicate of CP-144 (open, its cf-do cell pending): the same configuration, a DO agent with createSubAgentTool and no subAgentNamespace, filed against a pin (4980e8a8fb) older than CP-144's fix 5af6e15645c (materializeAgent step 2b refuses it framework_not_supported before any write, runtime-cloudflare/src/do/drive/materialize.ts:117-128 (origin/main fb9150cc16)), which was already on main when this record was filed. Its corruption premise does not hold: at the pin each sub-agent call failed typed (state_session_not_found, a tool error) before the child's run-start commit, and every commit path of DOStateStore refuses a foreign session id (requireStateRowSync, runtime-cloudflare/src/do/do-state-store.ts:3496-3501 (origin/main fb9150cc16)); the claim came from a stale code comment (durable-object-agent-base.ts:3529-3531) and docs/runtimes/cloudflare-do.md:2661-2662, which a docs change corrects (orchestrator ruling, 2026-10-24) (traced by WA in a local design note; verified by the owner at origin/main fb9150cc16) |
| CP-189 | S1 | runtime-js: resuming a ROOT parked suspended_awaiting_children after a submit to its suspended LOCAL child never drives the child. The submit is accepted and stored on the child (caller_resumes), but resume() runs the root's own next model step: the child never runs its approved (or answered) call, its ref stays running with the submitted result unconsumed, and the root ends completed (no outputSchema), with its sub-agent call left without a tool_result in history, or failed (outputSchema). | runtime-js/src/run-loop.ts:2652 (origin/main 4980e8a8fb) (the run records suspendedAwaitingChildren on the session when it parks on a suspended child; no runtime-js code reads it back on resume); core/src/run/reconcile-closed-owner-run.ts:183 (origin/main 4980e8a8fb) (the only reader of suspendedAwaitingChildren: the closed-owner reconcile, not the resume path); runtime-js/src/__tests__/submit-tool-result-routing.integ.test.ts:140-160 (origin/main 4980e8a8fb) (the routing test resumes the CHILD by hand first, :149-157, although its comment, :140-144, states that resuming the parent drives the child); live: WE-2's probe (2026-10-19, a plain runtime-js executor on main's source, also re-run as a temporary vitest file): parent with createSubAgentTool(child), the child's requireApproval deploy (also execute: 'client') suspends it and the root parks suspended_awaiting_children; submitToolResult(approval-response, approved) on the root answers accepted with continuation caller_resumes; resume(parent, root) runs the root's next model step. With no outputSchema the root completes with history [user, assistant(sub1), assistant(next step), tool], sub1 unpaired, the child ref running, its entry keeping submittedResult unconsumed and deploy never run; with an outputSchema the root fails and the child is never resumed; related: peer-reported by WE-2 (2026-10-19), the runtime-js source verified by the owner at main 4980e8a8fb; it blocks CP-186's .run-completes on WE-2's fix. Other runtimes not checked yet (CF Workflows next, WE-2). Also violates RM-29's pairing contract (a sub-agent tool_use without its tool_result in a completed session); severity S1 (owner, 2026-10-19): an approved action never runs while the run reports completed, and the persisted history is left with an unpaired call | source, peer-reported, live-repro | partly-inferred | withdrawn — A duplicate of CP-10 (S1, open; fixer W1, draft !421): CP-10 already records that runtime-js resumes only the ROOT, whose resume has no step for an awaited local child and never reads suspendedAwaitingChildren. Filed by the owner from WE-2's probe without a catalogue search; WE-2's two new facts (the approval variant, and a root without outputSchema reporting completed with its sub-agent call unpaired) are folded into CP-10's evidence (orchestrator ruling, 2026-10-19) |
| CP-190 | S3 | runtime-js: a parent process that crashes after its companion spawn writes the name's ref claim and before the child's run starts strands the name: the ref stays running for a child that never started, spawnAgent answers 'already running' for it, and only a later waitForResult's process-local orphan detection frees it (JS child starts carry no parentRunFence, so no store fence settles it). CP-161's reclaim of the same case is CF Workflows only. | core/src/orchestration/companion-tool-dispatch.ts:1163 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (the spawn writes the ref claim) with runtime-js/src/run-loop.ts:7464 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (startPersistentChildEntry starts the child's run in process, after it); core/src/orchestration/companion-tool-dispatch.ts:2004-2030 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (waitForResult's orphan detection, the only release: isChildExecutionLive reads the in-memory registry, runtime-js/src/run-loop.ts:8091 and :577-606, as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f; three not-live reads fail the ref) and core/src/orchestration/helpers.ts:79-82 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (spawnAgent on a running ref answers already running); inferred: (WA's source read, 2026-10-19, CP-161 cross-runtime table, peer-reported) only CF Workflows sets parentRunFence on a child start (runtime-cloudflare/src/workflow.ts:2960, as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f); the crashed step's tool_use is never committed (step.atomic), so CP-161's reclaim predicate would need the step's in-memory call ids on runtime-js; related: CP-161 (the same strand on CF Workflows, fixed there by the reclaim; the CF DO frees it through the spawn tool's pre-dispatch reconcile); WA's proposed name FU-JS-STRANDED-SPAWN-CLAIM. Orchestrator 2026-10-19: park it with the JS work unless rated higher; severity S3 (owner, 2026-10-19): a process crash in a narrow window; the name is wedged for spawnAgent until a waitForResult frees it | source, peer-reported | partly-inferred | open |
| CP-191 | S3 | Temporal: a parent workflow CLOSED (terminate, timeout, reset) between the dispatch activity that writes a companion's ref claim and wf.startChild strands the name: the ref stays running for a child that never started, with no orphan detection, no parentRunFence on the child, and child runs recording no parent, so nothing frees it. A worker crash is replayed, not a strand. | runtime-temporal/src/workflow-dispatch.ts:778 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (wf.startChild after the dispatch activity that wrote the ref) with runtime-temporal/src/activities.ts:8958 (pre-fix, origin/main 841ce4c960) (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (spawnPersistentChild is a no-op in the activity); inferred: (WA's source read, 2026-10-19, CP-161 cross-runtime table, peer-reported) no orphan detection reads the ref; child runs record no parent (FU-TEMPORAL-COMPANION-RUN-PARENT), so CP-161's 'no live child run serves the claim' test cannot be decided strictly; a non-blocking child (PARENT_CLOSE_POLICY_ABANDON) whose start was issued before the close may still be starting, so a reclaim needs a fence; related: CP-161 (CF Workflows), CP-190 (runtime-js); WA's proposed name FU-TEMPORAL-CLOSED-SPAWNER-STRANDS-CLAIM; parked under the Cloudflare-first direction; severity S3 (owner, 2026-10-19): narrow (a workflow closed from outside in one activity-to-command window); permanent for that name | source, peer-reported | partly-inferred | open |
| CP-192 | S3 | DBOS: a parent workflow cancelled from outside (cancelWorkflow), or whose recovery attempts are exhausted, between its companion spawn's ref insert and the child workflow's start strands the name: the ref stays running for a child that never started. A kill is not a strand (recovery replays the spawn by its call id and deterministic workflow id), and a failed start compensates the ref failed; DBOS uses a plain updateSubSessionRef, not CP-161's compare-and-set. | runtime-dbos/src/steps/execute-companion-tool.ts:751-830 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome a2dac83f2f) (the spawn in the workflow body: the ref insert :751, createSession about :770, startPersistentWorkflow :814; a failed start compensates the ref failed, :820-830), replayed by its call id (replayRef, :534) on recovery; inferred: (WA's source read, 2026-10-19, CP-161 cross-runtime table, peer-reported) the residual window is an external cancel or exhausted recovery between the insert and the start; the ref writes are plain updateSubSessionRef (:727), outside CP-161's ref CAS; related: CP-161 (CF Workflows), CP-190 (runtime-js), CP-191 (Temporal); parked under the Cloudflare-first direction; severity S3 (owner, 2026-10-19): needs an external cancel or recovery exhaustion inside one step | source, peer-reported | partly-inferred | open |
| CP-193 | S1 | CF Durable Object: a run producing stream output without I/O monopolizes the isolate; status, stop and every other request wait for the whole run (the stop arrives after it ends). | runtime-cloudflare/src/do/do-stream-manager.ts:302-408 (origin/main 6ae9057c9e) (DOStreamWriter.write is async but awaits nothing: DO-SQLite inserts and the non-blocking SSE / WebSocket broadcasts, all synchronous. So each await writer.write() / ctx.emit() resolves as a microtask, and a run with no real I/O is one unbroken microtask chain that never returns the isolate to its event loop until it ends); live: WE-2's red runtime-cloudflare/src/__tests__/integration/do-large-output-responsive.integ.test.ts with helpers/do-large-output-worker.ts (branch reliability-we-do-busy-stream f059b813f0 = main 6ae9057c9e + the test; WE-2's slot B, 2026-10-20, 3 of 3, peer-reported, the result relayed by WE-2). A real DurableObjectAgentBase with DOStateStore in Miniflare runs a tool that awaits ctx.emit() 100,000 times (200 B custom events, no I/O of its own). /start answers in 18 ms, but a probe sent 300 ms later (an in-memory counter read) is dispatched only after the whole flood (1,949 to 1,967 ms, reporting 100,000 of 100,000 emitted); /status then reads completed and /interrupt answers 409 (the run is over); live: owner slot s81 (2026-10-20, f04c5fc74a, the cf-do host lane with CP-136's cell un-parked): while one run's tool emits 2,000 custom events of 16 KiB to a held client, no harness read through the session's DO (observe's store reads, streamInfo) is answered until the run ends: 26 s unconstrained, 102 to 106 s under taskpolicy -b, and one run's listRuns read failed with 'fetch failed'; inferred: a real LLM yields between network reads, so a model stream bounds the stall per read rather than per run; the unbounded case is any producer that streams without real I/O (a tool emitting progress, a large text step whose deltas arrive in one network read, in-isolate sub-agents); related: CP-136's cell timed out on its first CI run (!464 job 17302397122) while its run streamed ~32.8 MB through one tool; this finding is the likely cause (FU-CONF-READ-AHEAD-CI-TIMEOUT); related: fixed by !466 (WE-2, merged as 46db645e27): a run of DO stream writes yields to the event loop on a budget, at runtime-cloudflare/src/do/do-stream-manager.ts:262-274 (origin/main 46db645e27) (spendYieldBudget: after DO_STREAM_YIELD_EVERY_WRITES writes or the byte slice, a smaller slice while an SSE reader is attached) and :284-288 (origin/main 46db645e27) (yieldNow: a setTimeout(0) macrotask, so a queued /status, stop or submit is dispatched mid-run); conformance: no lane witnesses it (orchestrator ruling, 2026-10-20; owner slots s86-s88). The cf-do host lane's Miniflare runs the worker entry and the session DO in one isolate, so at main a no-I/O flood starves the worker routes too (s88: the first worker-route answer came 250.6 s in), and on the fix a new connection to workerd is reset (fetch failed, cause ECONNRESET). The cell scenarios/do-flood-responsive.ts stays unregistered (FU-CONF-CFDO-FLOOD-HARNESS-COST), so the flip uses provenByException on WE-2's real-DO integ tests; severity S1 (owner, 2026-10-20): a Stop the user sends during such a run is not even dispatched until the run has finished, so it can never stop it, and status, snapshot and every other request to the session wait for the whole run (the incident's 'work stops' class) | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-194 | S2 | runtime-js (and the CF Durable Object that runs it): a new turn after a run that died mid remote-sub-agent call never reconciles the child. The child's ref is registered and the child started before the step commits its tool_use and result together, so after a crash, supersede or stop in that window the ref reads running with no call in any committed row. resume / retry / resumeLoop reconcile such a ref, but a new turn's settle skips it twice over: it runs only for a terminal prior session (a run that died live leaves the session active), and even when it runs it settles only refs whose call is among the history's trailing unpaired calls (a call in no committed row never is). The child runs on unwatched and its ref stays running. A third cause is on resume / retry / resumeLoop: their reconcile, for such a ref, only logs an unreadable child and leaves the ref running. | runtime-js/src/execution/remote-subagent-executor.ts:210-215 (origin/main aaf34606c5) (addSubSessionRefs registers the remote ref) and :480-486 (origin/main aaf34606c5) (startRemoteChild starts the child), both inside the tool call, before core/src/orchestration/step-iterator.ts:2304 (origin/main aaf34606c5) (hooks.commitStep commits the step's tool_use and its result together); runtime-js/src/js-agent-executor.ts:5241, :5931 and :7411 (origin/main aaf34606c5) (resume, retry and resumeLoop each call reconcileRemoteSubAgents, which pairs a running remote ref with the child's real outcome); runtime-js/src/js-agent-executor.ts:6652-6665 (origin/main aaf34606c5) (cause 1, the gate: a new turn calls settleRemoteChildrenBeforeInput only when priorStatus is completed / failed / interrupted; a run that crashed or was superseded mid-call leaves the session active, so the settle is skipped) with :4279 (origin/main aaf34606c5) (the new turn's orphan pass, fed only by that settle's result); core/src/orchestration/remote-children-before-input.ts:134 (origin/main aaf34606c5) (cause 2, the trailing-only filter: even with the gate open, the settle loop skips a ref unless trailing.has(ref.parentToolCallId), the history's trailing unpaired calls; a lost step's call is in no committed row, so its ref is skipped). Reachable with the gate open: a stop whose K5 commits interrupted while the stopped step's remote call never committed (the session reads interrupted, and :134 still drops the ref); related: W1 and WE-2's joint note (peer-reported via the orchestrator, 2026-10-20), verified against main aaf34606c5 and !449 07012bb291 by both; the owner verified the sites above at aaf34606c5. The note cited the reconcile calls at !449's line numbers (js-agent-executor.ts :5436 / :6123 / :7596 at 07012bb291). Cause 2 is WE-2's correction (peer-reported via the orchestrator, 2026-10-20, verified by the owner at aaf34606c5): core isReconcileOwnedRemoteRef (core/src/orchestration/remote-children-before-input.ts:64-74, origin/main aaf34606c5) claims every running remote ref, but its caller at :134 adds the trailing filter; severity S2 (owner, 2026-10-20): needs a crash or a supersede inside one remote call; the child keeps running and spending unobserved, and its result is never delivered; runtime-js/src/js-agent-executor.ts:2713-2715 (origin/main 0889c0ee05) (cause 3, the reconcile's unreadable arm: resume / retry / resumeLoop's reconcileRemoteSubAgents, for a live remote ref whose call did not survive (toolCallSurvived false, :2632 (origin/main 0889c0ee05)), only logs an unreadable verdict (logUnreadableRemoteChild) and continues, so the ref stays running and nothing settles or stops the child), beside :2701-2711 (same pin, origin/main 0889c0ee05) (its owed / lost siblings orphan the child with a ref write that is not compare-and-set). WE-2 (orchestrator ruling, 2026-10-20, peer-reported), verified by the owner; the fix is CP-194's single post-commit sweep, which takes over every live ref whose call is in no committed row; related: fixed by !473 (WE-2, merged as 3a55689d5d): every runtime-js / CF DO entry stops and settles the remote children of a lost step (D3): core/src/orchestration/lost-step-remote-refs.ts:30 (origin/main 3a55689d5d) (lostStepRemoteRefs: a live remote ref whose call is in no assistant row) applied by runtime-js/src/js-agent-executor.ts:2678 (same pin, origin/main 3a55689d5d) (sweepLostStepRemoteChildren, after each entry's run-start commit, whatever the prior status: it covers all three causes). Temporal and DBOS are FU-D3-TEMPORAL-DBOS-LOST-STEP; related: Save-First Tools MR 1 (!754): a new turn over a DO parent demoted after a mid-dispatch eviction now pairs the lost remote child's call outcome_unknown, not unpaired_on_continuation: the call is on the parent's saved delegation row and its child's start was never confirmed, so it may have started (runtime-cloudflare/src/__tests__/integration/do-eviction-recovery.integ.test.ts:782-785, as on branch reliability-cf-wa-sft-mr1 a9ef6b573d; green in WA's slot slot-b14c, 23 of 23). A demotion with no saved row keeps unpaired_on_continuation | source, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-195 | S2 | CF Workflows: CP-152's stop cascade writes a remote child's ref back with the status it read before the stop wait, unconditionally. A child result settled during that wait (the ref turned completed / failed) is reverted to running, so the ref contradicts the committed result. | runtime-cloudflare/src/stop-remote-children.ts:134-152 (origin/main aaf34606c5) (after the bounded stop, up to REMOTE_CHILD_STOP_TIMEOUT_MS, updateSubSessionRef writes status: ref.status, the value read before the wait, with stopOwed cleared at :138-142 or stamped at :146-150; no compare-and-set on the ref's current status); related: W1 and WE-2's joint note (peer-reported via the orchestrator, 2026-10-20), verified against main aaf34606c5 by both; the owner verified the site above at aaf34606c5. CP-152's fix introduced the cascade (!441, origin/main a8afa649bd, SubSessionRef.stopOwed). Fixer: W1, in !458; severity S2 (owner, 2026-10-20): needs a child settle landing inside the cascade's stop wait; the reverted ref then reads running over a committed result (Invariant S); related: fixed by !458 (W1, merged as 67d41c3bf5): every cascade write that re-states the read status is a CAS on it, runtime-cloudflare/src/stop-children.ts:308-330 (origin/main 67d41c3bf5) (stampLocalStopOwed passes expectStatus: ref.status and answers false on a SubSessionRefConflictError, logged stop.child_ref_settled_meanwhile) and :485-516 (origin/main 67d41c3bf5) (the remote branch's stopOwed stamp and clear, the same); the store refuses a changed status atomically (core/src/types/session.ts:1057 (origin/main 67d41c3bf5), updateSubSessionRef's expectStatus, in memory, D1, Postgres, Redis and DO-SQLite) | source, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-196 | S3 | runtime-js (and the CF Durable Object): a remote sub-agent's lost-child deadline omits the in-run dispatch's backoff waits between stream retries, so a resume or reconcile after that deadline can declare a child lost (framework_subagent_lost, and the child interrupted) while the normal dispatch would still have been retrying it. runtime-js and the CF Durable Object (which runs it); Temporal is not affected. | core/src/orchestration/lost-remote-child.ts:28-34 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome 07012bb291, !449) (remoteChildDeadline = startedAt + (streamRetries + 1) × timeoutMs + RECONCILE_GRACE_MS, 60 s at :17; read by the lost-child decision at :73); runtime-js/src/execution/remote-subagent-executor.ts:833 (origin/main aaf34606c5) (between stream retries the dispatch waits min(streamRetryBaseMs × 2^attempt, MAX_RETRY_DELAY_MS), with MAX_RETRY_DELAY_MS = 5,000 at :74 and the base defaulting to 100 ms at :183); the deadline adds none of these waits; runtime-js/src/js-agent-executor.ts:2684 (as on branch reliability-cf-wa-cp161-spawn-entry-outcome 07012bb291, !449) (the reconcile's remoteChildVerdict) and core/src/orchestration/remote-children-before-input.ts:211 (same pin, as on branch reliability-cf-wa-cp161-spawn-entry-outcome 07012bb291) (the entry settle's remoteChildVerdict), the deadline's only callers; related: Temporal is not affected (WE-2's retraction, verified by the owner): core's dispatch makes one attempt per call, bounded by timeoutMs (core/src/orchestration/remote-sub-agent-dispatch.ts:642-643, as on branch reliability-cf-wa-cp161-spawn-entry-outcome 07012bb291), the streamRetries loop and its backoff live only in runtime-js's executor, and Temporal retries the activity on its own policy (runtime-temporal/src/workflow-dispatch.ts:1277-1282 (origin/main aaf34606c5): 3 attempts, 1 s initial, x2, 10 s cap). Its per-attempt startToCloseTimeout ((streamRetries + 1) x timeoutMs + 60 s, :1163-1168 (origin/main aaf34606c5)) is over-generous, never short, and Temporal never calls remoteChildDeadline / remoteChildVerdict. CF Workflows' remote call retries through the engine's step.do retries, not streamRetries, so the deadline does not model CFW's watch window in either direction (its entry settle at runtime-cloudflare/src/steps.ts:1208, as on branch reliability-cf-wa-cp161-spawn-entry-outcome 07012bb291, which !458 deletes); related: found by !449's Greptile review and confirmed by WE-2 in !449's code (peer-reported, 2026-10-20); verified by the owner at the sites above. Example with valid settings (timeoutMs 1000, streamRetries 50, streamRetryBaseMs 5000): the deadline is about 111 s, and the waits alone are up to 50 × 5 s = 250 s. It affects runtime-js and the CF DO; severity S3 (owner, 2026-10-20): the default retry settings (3 retries, a 100 ms base) wait 0.7 s in total, well inside the 60 s grace, so the window needs a non-default retry configuration; then a still-retrying child is declared lost and its work discarded; related: after !449 merged, the same code on main: core/src/orchestration/lost-remote-child.ts:28-34 (origin/main 0889c0ee05) (remoteChildDeadline, unchanged), runtime-js/src/js-agent-executor.ts:2685 (origin/main 0889c0ee05) (the reconcile's remoteChildVerdict) and core/src/orchestration/remote-children-before-input.ts:211 (origin/main 0889c0ee05) (the entry settle's); the branch-pinned refs above stay as written (WE-2 re-read the lines at 0889c0ee05; verified by the owner); related: fixed by !467 (WE-2, merged as c008a5358e): the deadline is the runtime's own watch budget plus the grace, core/src/orchestration/lost-remote-child.ts:116-118 (origin/main 16ae386528) (remoteChildDeadline(ref, watchMs)), with runtime-js / the CF DO's budget at :67 (origin/main 16ae386528) (streamRetryWatchBudget: the attempts, each recovery status read, and the remoteStreamRetryDelayMs waits) passed by the reconcile at runtime-js/src/js-agent-executor.ts:2704 (origin/main 16ae386528). CF Workflows' entry settle used cfwRemoteWatchBudget at runtime-cloudflare/src/steps.ts:1256 (origin/main 16ae386528), which !458's main merge removes with its only consumer (the RM-80 entry settle; orchestrator ruling, peer-reported by WE-2); related: !458 (W1, merged as 67d41c3bf5) removed the CF Workflows budget as ruled: cfwRemoteWatchBudget (runtime-cloudflare/src/remote-watch-budget.ts is gone) and core's stepRetryWatchBudgetMs no longer appear in runtime-cloudflare or core source at 67d41c3bf5, with CFW's RM-80 entry settle, their only consumer; on CFW a lost-step child is D3's (settleLostStepRemoteChildren), with no entry-time deadline to be wrong. The runtime-js / CF DO budget (streamRetryWatchBudget) is unchanged | source, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-197 | S3 | A resumed run's drain records a stopped run's lapsed client-tool deadline as an answer (client_tool_timeout), so the call ends up paired. Under the 2026-10-20 ruling (Q1: a lapse after a stop is new work; the first durable write wins) a stop recorded first must leave the call pending, so that a late submit is still answered unknown_tool_call / run_stopped (Ruling 56). runtime-js (and the CF DO that runs it) and CF Workflows; CP-173's sibling. | runtime-js/src/js-agent-executor.ts:3451-3462 (origin/main 8e2924ed46) (the drain's append commit: run-fenced but deliberately NOT stop-guarded under Ruling O-73, which counted a drained result, a lapsed deadline included, as completed work); runtime-cloudflare/src/steps.ts:7912-7917 (origin/main 8e2924ed46) (the CF Workflows drain: "drained answers are the record of completed work (a client result, a denial, a lapsed deadline): recorded even after a stop", appendCommitted with recordsCompletedWork); related: CP-173 (the CF DO alarm's own window: its expiry stores client_tool_timeout on a just-stopped run). The orchestrator's 2026-10-20 ruling Q1 makes a lapse after a stop new work for both paths. WA filed this sibling (peer-reported via the orchestrator, 2026-10-20; trace: WA's local investigation note, §1.2), verified by the owner at the sites above. Fixer: WA, in the CP-173 + CP-174 MR (branch reliability-cf-wa-cp173-174-expiry-seam; design docs/superpowers/specs/2026-10-20-cp173-174-deadline-expiry-seam-design.md at 929d978f8d); severity S3 (owner, 2026-10-20): it needs a stop that lands before the lapsed call is drained; nothing is lost, but the stopped run's call reads answered (timed out) and a late submit gets the wrong answer; conformance: stop.drain-lapse-after-stop (owner slot s90, 2026-10-20) is red at main 46db645e27 on js-memory and cfw-workflows-d1, 3 of 3 each, on exactly .no-timeout-row, .late-submit-run-stopped and .next-turn-heals, with its window proof .stop-recorded-live green on every run (R2 running with a stop record at the release); green on WA's CP-173 + CP-174 tip 1c28613186, 3 of 3 on both; temporal-memory passes (unreachable there: Temporal's drain append carries refuseIfStopped). dbos-postgres is a declared gap. js-redis and js-postgres (owner gate s91, 2026-10-20, redis-stack and postgres on private ports, at the batch head ff15126b3c over main 16ae386528): red 3 of 3 each on the same three ids with the window proof green, while js-memory and cfw-workflows-d1 were judged expected-fail; related: CP-138 (!500) takes the cfw-workflows-d1 cell off the ledger (a declared gap, owner 2026-10-24): CF Workflows' __deadline__ watcher expires a lapsed call of a suspended run with no inbound request and then advances the run; a stop recorded first wins there: runtime-cloudflare/src/client-tool-deadline.ts:154 (as on branch reliability-we-cp138-cfw-deadline e6881d4e2c) (a run with a stop record is 'stopped', before any write), so on !500's MR pipeline 2985130121 the cell's precondition .lapsed-pending read the run completed (wrong-reason). A continuation started BEFORE the lapse makes no run-start commit while the call is pending, on CF Workflows and runtime-js alike (owner slot s121, 2026-10-24: the after-commit hold was never entered on cfw-workflows-d1, js-memory, js-redis, js-postgres), so the drain-over-a-lapsed-call path stays reachable on CF Workflows only when the watcher never expired the call (its create failed, or its rounds ran out, before any reader backstop): no suite control reaches that, so CF Workflows' drain over a lapsed call is now UNEXERCISED by any lane (it still records the lapse after a stop: steps.ts recordsCompletedWork, unchanged by !500); a unit witness for it is FU-CFW-CP197-DRAIN-LAPSE-UNIT-WITNESS, to land with CP-197's CF Workflows fix; related: the CF DO runs the same runtime-js drain, but it has no cell: the scenario is executor tier, and on the cf-do host no control holds a DO run right after its run-start commit for a caller's stop to reach (orchestrator ruling, 2026-10-20); runtime-js/src/js-agent-executor.ts:3310-3333 and :3444-3446 (as on branch reliability-cf-wa-cp173-174-expiry-seam 2c7f6030c4) (fix: !490, WA's CP-173 + CP-174 expiry seam, 6e91489256 and b07f8b5048: the resume drain's expiry save is stop-guarded, and with the expiry refused by a stop, or failing after one, an unanswered lapsed call stays pending and the run ends by the stop; the CF DO runs this drain); runtime-cloudflare/src/steps.ts:8073 and :8340 (as on branch reliability-cf-wa-cp173-174-expiry-seam 2c7f6030c4) (fix: !490, f1782340f2 and 1c4a25a31b: CF Workflows' persistLapsedDeadlines writes the lapse onto the pending entry in one stop-guarded saveState before the drain writes any frame or row, so a stop recorded first keeps the call pending); conformance: fixed on runtime-js. stop.drain-lapse-after-stop passes on js-memory, js-redis and js-postgres (executor) on !490's MR pipeline 2985877649 (head 2c7f6030c4, merged result c1e11dcf22), job 17444954200: the three ledgered ids .no-timeout-row, .late-submit-run-stopped and .next-turn-heals no longer reproduce (reported UNEXPECTED PASS against the open record), the other 475 cells of conformance-js.integ.test.ts pass; related: CF Workflows has no cell (the cfw-workflows-d1 cell is a declared gap since CP-138, above); its half is proven by !490's runtime-cloudflare unit witnesses, packages/runtime-cloudflare/src/__tests__/steps-applyResultsAndReload.test.ts:724 (as on branch reliability-cf-wa-cp173-174-expiry-seam 2c7f6030c4) (a stop recorded before the drain: the lapsed call is not answered client_tool_timeout and stays pending, added with f1782340f2) and :737 (a stop that lands once the drain has started, added with 1c4a25a31b), which close FU-CFW-CP197-DRAIN-LAPSE-UNIT-WITNESS; both landed in their fix commits, not red-first, and were not re-run against the pre-fix drain (WA, 2026-10-25), so this half rests on the fix's source and those green unit tests, not on a red/green pair | source, peer-reported | verified | fixed (3 cells) |
| CP-198 | S2 | CF Durable Object: a duplicate client-tool submit after a lost answer is answered caller_resumes, not deferred. The first submit stored the answer and its wake started the continuation, but the answer back to the chat handler was lost (a retryable transport_error); the re-submit gets already_completed, which the DO passes through with the JS executor's continuation unchanged. The chat handler then calls resume() against the live continuation: under concurrentEntry 'reject' (the default) a wrong 409 state_already_running for a running turn, and under 'interrupt_previous' the live continuation is stopped and rerun. | runtime-cloudflare/src/do/durable-object-agent-base.ts:3813-3814 (origin/main 0889c0ee05) (as on branch reliability-cf-w1-cp180-submit-contract 1089168988, !453) (an already_completed local submit only reschedules the alarm; the deferred mapping, continuationFromWake / isSubmitContinuationOwed, runs only for an accepted submit that woke); runtime-cloudflare/src/do/do-clients.ts:1754-1771 (same pin, as on branch reliability-cf-w1-cp180-submit-contract 1089168988) (submitToolResult: a rejected stub fetch is a typed, retryable transport_error, so the handler or a client re-submits an answer the DO already stored); related: W1, found in !453's idempotency check (peer-reported via the orchestrator, 2026-10-20), verified by the owner at the sites above. It is reachable by any client-side duplicate submit before !453, and !453's automatic one re-submit of a transient failure makes it automatic. CF DO only: runtime-js and Temporal resume once per request, CF Workflows adopts its deterministic __submit__ instance, and DBOS its <run>:continuation id; severity S2 (owner, 2026-10-20, the orchestrator's estimate): a lost answer plus one re-submit gives either a wrong 409 for a live turn the client should have attached to, or a stopped and rerun continuation; related: fixer W1, in !453 (the DO maps an already_completed local submit through the same live-drive check: a registered drive or a live current run answers deferred, otherwise unchanged; red first for both concurrentEntry modes and the not-live case); related: fixed by !453 (W1, merged as 16ae386528): the DO tracks every wake in flight (runtime-cloudflare/src/do/durable-object-agent-base.ts:576 (origin/main 16ae386528), _inflightWakes, added from before the producer-claim wait at :2119-2123 (origin/main 16ae386528)), and an already_completed local submit whose continuation is caller_resumes while a wake is in flight answers from that wake (:3858-3864 (origin/main 16ae386528), answerFromContinuation over continuingWakeOutcome, runtime-cloudflare/src/do/drive/submit-answer.ts:298 (origin/main 16ae386528)), so the duplicate gets the first submit's own answer: deferred or new_run while the continuation runs, already_completed consumed once it finished, the same typed failure when the wake fails; related: the residual FU-DO-DUPLICATE-SUBMIT-DURING-REQUEST-DRIVE (W1, docs/dev/follow-ups.md:3906 (origin/main 16ae386528)): a duplicate whose stored answer a REQUEST drive (not a wake) is about to drain, with no wake in flight, still answers caller_resumes; no red is reachable today | source, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-199 | S3 | CF Workflows aborts a failed parent's live persistent companions. When a reader settles a run whose Workflows instance closed mid-run with no stop record and nothing committed (fail_uncommitted), it runs the stop cascade with 'abort', and the cascade takes every live local child ref, persistent companions included. The user's ruling (2026-10-20: companions are session-scoped, C3) says a parent's failure leaves its companions running and their results deliver on the next turn; runtime-js, the CF DO and CF Workflows' own in-body failure paths cascade nothing. | runtime-cloudflare/src/executor.ts:2312-2313 (origin/main 16ae386528) (the closed-owner settle: on reconciled.outcome 'failed' it calls stopChildren(sessionId, runId, 'abort', 'parent run failed: its owner closed')); runtime-cloudflare/src/stop-children.ts:115-120 (origin/main 16ae386528) (isLiveLocalChildRef: any local ref that is running or paused_awaiting_client, whatever its mode, so a non-blocking persistent companion is stopped with the ephemeral children); contrast: runtime-js/src/run-loop.ts:1813-1815 (origin/main 16ae386528) (an ordinary failure runs no cascade: it is gated on stopResolution) and runtime-cloudflare/src/workflow.ts:2441-2447 (origin/main 16ae386528) (Deliver-on-next-turn, C3: the parent's exit, completion or failure, leaves its live persistent children running); related: found by W1 in the companion-lifetime trace for the user's session-scoped ruling (peer-reported via the orchestrator, 2026-10-20), verified by the owner at the sites above. The orchestrator's ruling detail: a Stop of a run whose record reads running but whose owner died is a live-run Stop, so the cascade stays for the closed-owner settle_stopped branch (executor.ts:2310-2311, same pin); only the failure branch is wrong; related: fixer W1, a small fix-forward MR after !421's re-merge: the cascade takes a cause ('stop' | 'failure'), and a failure skips persistent companions; red first; severity S3 (owner, 2026-10-20): it needs a CF Workflows instance that closes mid-run without its terminal commit (terminated from outside, errored), then a reader; the companion's work and memory are discarded where the ruling keeps them; related: fixed by !472 (W1, merged as b0394f2e5f): the cascade takes a cause, runtime-cloudflare/src/stop-children.ts:116 (origin/main b0394f2e5f) (StopChildrenInput.cause: 'stop' | 'failure'), and a failure leaves the run's persistent companions running, :179-180 (same pin, origin/main b0394f2e5f) (stopsThisChild: not for a top-level 'failure' cascade's persistent refs), applied to both the remote and the local selection (:184, :188 (origin/main b0394f2e5f)); the closed-owner settle's failure branch passes 'failure', runtime-cloudflare/src/executor.ts:2316 (origin/main b0394f2e5f) | source, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-200 | S2 | CF Durable Object: an SSE reader that (re-)opens with a cursor inside a stop's mid-log rewind skips the discarded chunks silently. A stop discards the uncommitted step's stream chunks (cleanupToStep) and tells only the readers attached at that moment (the rewound control frame). A reader that opens afterwards from a cursor inside the deleted range, such as a slow follower re-opened after its eviction, gets no rewound frame and no typed truncation: it jumps straight to the later frames (run_interrupted, then a clean end), so a client cannot tell that content it had partly shown was discarded. CF Workflows' stream DO has the same open check (inferred). | runtime-cloudflare/src/do/durable-object-agent-base.ts:4964-4965 (origin/main 16ae386528) (the /sse open refuses a cursor only below the retention floor while a truncation marker is set: fromSequence < getOldestRetainedSequence() && hasTruncationMarker(); a cursor inside a mid-log rewind passes with no rewound frame); runtime-cloudflare/src/do/do-stream-manager.ts:1518 (origin/main 16ae386528) (cleanupToStep deletes the discarded step's chunks, called by the stop at runtime-js/src/run-loop.ts:5797, same pin), persisting only truncated_at_step at :1528-1530 (same pin, origin/main 16ae386528), and telling only attached readers through the rewound control frame at :1557 (origin/main 16ae386528) (same pin, broadcastControl); contrast: runtime-cloudflare/src/do/do-clients.ts:2069-2074 (origin/main 16ae386528) (an ATTACHED DOStreamManagerClient reader throws a typed StreamTruncatedError on a rewound frame past its cursor; a re-opened one never gets the frame, so it ends clean); live: WE-2's repro, 8 of 8 at 16ae386528 (peer-reported, 2026-10-20): a slow follower evicted twice mid-flood, then /interrupt; its third re-open from about sequence 3218 received only run_interrupted (80367) and end, and /sse?fromSequence=0 afterwards returned 2 chunks (the stop deleted the rest). First seen in main pipeline 2966843082 (job 17310853766): the CP-193 integ test's re-opened reader got 1..15469 then 31515 then a clean end (do-large-output-responsive.integ.test.ts:186, same pin; read by the owner in the job log); intermittent there; inferred: the same open check on CF Workflows' stream DO, store-cloudflare/src/stream-durable-object.ts:683 (origin/main 16ae386528) (retention floor only), with the rewound frame only broadcast at :1567 (same pin, origin/main 16ae386528); not run (WE-2); related: not new against the published 7.0.0, whose DO stream manager has no rewound frame at all, so every reconnecting reader already skipped silently there (WE-2); the C2a / C2b re-open path (!410 / !416) made a re-open over a rewind routine for slow readers. Fix direction (WE-2): persist { atStep, atSequence, minSequence } with the marker and write rewound on /sse open when minSequence < fromSequence < atSequence; severity S2 (owner, 2026-10-20, on WE-2's analysis; first filed S1 provisional): the skipped chunks are the discarded step's own content, so no committed data is lost; the defect is the missing typed signal to a reader that opened after the rewind; related: fixed by !471 (WE-2, merged as cf569d8d20): a cleanup records its rewind and a re-open inside it gets that cleanup's rewound frame first, on both stream DOs: core/src/transport/stream-frame.ts:189 (origin/main cf569d8d20) (rewoundForReopen, with parseRewoundRecord at :160, same pin), runtime-cloudflare/src/do/do-stream-manager.ts:1480 (same pin, origin/main cf569d8d20) (the DO's rewoundForReopen over the persisted last_rewound record) written by the /sse open at runtime-cloudflare/src/do/durable-object-agent-base.ts:4997 (same pin, origin/main cf569d8d20), and store-cloudflare/src/stream-durable-object.ts:714 (same pin, origin/main cf569d8d20) (CF Workflows' stream DO, the same); the CF Workflows half is now verified by its own red tests, not inferred | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 4 tests) |
| CP-201 | S3 | A run that fails in its own body or loop leaves its EPHEMERAL children running behind a running ref: CF Workflows' body failure path, runtime-js's onFailed and the CF DO's run settle stop no children (only a stop, or CF Workflows' reader settle of a closed owner, cascades). An ephemeral local child of the failed run, such as one suspended on its own client tool, stays live forever with its parent's ref reading running, after the parent's call to it has already been paired not-executed. Its result can be recorded only by the parent run's own wait, so no second row is reachable: a leak, not a silent loss. Fixed on runtime-js and the CF DO by a80920762b and on CF Workflows by e1a3d9ab51 (a failed run stops its ephemeral children, companions kept). | runtime-cloudflare/src/workflow.ts:5730 (origin/main 0a74f4e6eb) (the run body's outer catch: its failure path calls no stopChildren; the only in-body cascade is at a recorded stop, :1328, same pin); runtime-js/src/run-loop.ts:1812-1815 (origin/main 0a74f4e6eb) (after onFailed, the child cascade runs only if (stopResolution); an ordinary failure stops no child), which the CF DO runs too; live: W1's unit probe on !465's rebased branch 417503cfc9 (peer-reported, 2026-10-20): runtime-cloudflare/src/__tests__/cfw-landed-step-emit-failure.test.ts:207 (as on branch 417503cfc9) ('a landed cascade suspension: its assistant row once, and the awaited child is paired not-executed exactly once (RM-91)'): after the parent fails, the local ephemeral child's ref s1 reads running while its call is paired once (CF Workflows' body failure path, after pairOwedCallsOnFailure, workflow.ts:6066 as on branch 417503cfc9); related: found by W1 (orchestrator ruling, 2026-10-20), filed separately from CP-199 (a failed parent's live persistent companions aborted on CF Workflows); verified by the owner at the main sites above. The user's session-scoped ruling (C3) keeps COMPANIONS running past a failed parent; ephemeral children have no next-turn delivery, so they should stop with it; related: fixer W1, after CP-199 lands and reusing its cause selection rule: CF Workflows' body failure path and runtime-js / the CF DO's onFailed run the cascade with cause 'failure', which stops ephemeral children only (companions untouched); severity S3 (owner, 2026-10-20): a failed parent strands its ephemeral children live (a client-tool child waits forever, its ref running), spending nothing further but leaving the session's child view wrong; nothing committed is lost; related: the DO has two failure seams (W1, 2026-10-23, withdrawing its 2026-10-22 note that the DO never reaches runtime-js's cascade; verified by the owner at origin/main 8cd8beb434): the DO's executor is a JSAgentExecutor (runtime-cloudflare/src/do/drive/build-executor.ts:93 (origin/main 8cd8beb434)) with its own InProcessRunRegistry (runtime-js/src/js-agent-executor.ts:649 (origin/main 8cd8beb434)), so a failure inside the run loop reaches runtime-js's cascade, which stops no child for an ordinary failure (the run-loop line above); a failure outside the loop (a wake-terminal, failCommittedRun) can cascade only through the DO's own settle cascade, runtime-cloudflare/src/do/drive/settle.ts:466-493 (origin/main 78b21c1683) (cascadeStop), which answers 'none' for a failed run with no stop record (:493 (origin/main 78b21c1683)). So a failed DO parent stops no ephemeral child on either seam; related: CP-206 (the runtime-js STOP path leaves a stopped ephemeral child's ref running: the cascade writes only persistent refs), filed separately (orchestrator ruling, 2026-10-22); W1 fixes both in CP-201's MR a; related: fixed in code by a80920762b (W1, CP-201 (a) + CP-206: runtime-js and the CF DO) and e1a3d9ab51 (W1, CP-201 (b): CF Workflows), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/run-loop.ts:6972-6993 (origin/main 56d564d073) (endFailedRun cascades with cause 'failure' for a run that failed with no stop, which stops its ephemeral children only), runtime-cloudflare/src/do/drive/settle.ts:522-531 (origin/main 56d564d073) (the DO's settle cascades a failed run with no stop record as an abort with cause 'failure'), runtime-cloudflare/src/workflow.ts:6862-6880 (origin/main 56d564d073) (the CF Workflows body's error-stop-children step before the failure K5, cause 'failure' unless a stop was recorded) | source, peer-reported | verified | open |
| CP-202 | S2 | A superseded run's in-run remote dispatch rewrites its child's sub-session ref with no compare-and-set, so a ref another writer already settled is reverted: after the next entry's lost-step sweep (D3), a stop cascade or a reconcile settles the ref failed / interrupted, a late write from the old dispatch sets it back to running (stream identity, progress) or to completed. The parent's view of its children (getSubSessionRefs, reconcile) then shows a settled lost child as live or completed. runtime-js (and the CF DO), CF Workflows (its own dispatch) and Temporal (core's dispatch helper). | core/src/orchestration/remote-sub-agent-dispatch.ts:480 (origin/main 0a74f4e6eb) (the stream-identity write, re-stating the read status), :707 (same pin, origin/main 0a74f4e6eb) (chunk-cadence progress, running) and :915 (same pin, origin/main 0a74f4e6eb) (progress on a drop, running): plain updateSubSessionRef calls with no expectStatus (the file names expectStatus nowhere at this pin); this helper serves Temporal (runtime-temporal/src/remote-sub-agent-activity.ts:49, same pin); runtime-js/src/execution/remote-subagent-executor.ts:231 (origin/main 0a74f4e6eb) (identity), :603 and :711 (same pin, origin/main 0a74f4e6eb) (progress), :319 and :964 (same pin, origin/main 0a74f4e6eb) (completed, CP-146's terminal write before the step commit), :1195 (same pin, origin/main 0a74f4e6eb) (failed): the same, no expectStatus in the file; runtime-cloudflare/src/steps.ts:5785, :5881, :5994 and :6160 (origin/main 0a74f4e6eb) (CF Workflows' own dispatch, executeRemoteSubAgentCall: four status 'running' ref writes, none with expectStatus). WE-2 reported CF Workflows as using core's helper; at this pin it has its own dispatch with the same unguarded writes (owner); related: found by Greptile on !473 (CP-194) and confirmed by WE-2 (peer-reported via the orchestrator, 2026-10-20); the sites verified by the owner at main. The CAS it needs exists since CP-195 (!458: UpdateSubSessionRef.expectStatus, core/src/types/session.ts:1057 (origin/main 67d41c3bf5), origin/main 67d41c3bf5); related: fixer WE-2, in !473 (CP-194; unmerged): each of these writes passes expectStatus, and a conflict is skipped and logged; red first on runtime-js and core; severity S2 (owner, 2026-10-20, WE-2's estimate): a settled lost child flips back to live or completed in the parent's child view and in reconcile's; the transcript is not corrupted, since the old run's step commit is fenced; related: fixed by !473 (WE-2, merged as 3a55689d5d) on all three dispatches: runtime-js/src/execution/remote-subagent-executor.ts:184 (origin/main 3a55689d5d) (writeOwnRef: every in-run ref write is a CAS on the status it re-states), core/src/orchestration/remote-sub-agent-dispatch.ts:488, :725 and :942 (same pin, origin/main 3a55689d5d) (expectStatus on the identity and progress writes; Temporal), and runtime-cloudflare/src/steps.ts:3358 (same pin, origin/main 3a55689d5d) (persistOwnRemoteRef: CF Workflows' four writes, expectStatus 'running'); a conflict is skipped and logged | source, peer-reported | verified | fixed (no lane can reach it: 3 tests) |
| CP-203 | S2 | A TERMINAL remote sub-session ref that carries an owed stop (stopOwed) is never retried, on any runtime: the stop cascades select only live remote refs, and the lost-step sweeps (D3) settle a lost ref failed WITH stopOwed when its interrupt was not confirmed and then select only live refs too. So a remote child whose stop failed (a transient outage during the interrupt) can run on unstopped, spending on its producer, after the parent starts more turns or is stopped; nothing ever re-sends the stop. | runtime-cloudflare/src/stop-children.ts:162-165 (origin/main 0a74f4e6eb) (CF Workflows' cascade takes only isLiveRemoteChildRef refs) and runtime-cloudflare/src/steps.ts:3391-3393 (same pin, origin/main 0a74f4e6eb) (D3, settleLostStepRemoteChildren: an unconfirmed interrupt settles the ref with stopOwed set, so the ref is terminal and owed); runtime-js/src/execution/in-process-runs.ts:287 (origin/main 0a74f4e6eb) (runtime-js's cascade, which the CF DO runs too, skips a ref that is not isLiveRemoteChildRef); runtime-js's own D3 sweep arrives with CP-194's fix (!473, unmerged) and selects live refs as well (WE-2); related: FU-CFW-REMOTE-CHILD-STOP-OWED-RETRY (docs/dev/follow-ups.md:3263, origin/main 0a74f4e6eb) names the CF Workflows half as CP-152's limit and notes that D3 keeps the owed stop on the failed ref; this finding is that limit on every runtime. The wording in !458 and CP-194 that a later sweep acts on such a ref is not true today; related: found by Greptile on !473 (CP-194) and confirmed by WE-2 (peer-reported via the orchestrator, 2026-10-20); the selectors verified by the owner at main. Fixer: WE-2, a small MR right after !473: a core retryOwedRemoteStops that re-stops any remote ref with stopOwed (any status) at each entry's sweep and in both stop cascades, clearing the debt (stopOwed null, with expectStatus) only on a confirmed stop; severity S2 (provisional, owner, 2026-10-20, WE-2's estimate): an unstopped remote child keeps running and spending after its parent moved on; nothing in the parent's transcript is wrong; related: fixed by !480 (WE-2, merged as e4a79cda2a): core retryOwedRemoteStops (core/src/run/stop-remote-child.ts:161 (origin/main e4a79cda2a)) re-stops every remote ref that still owes a stop, whatever its status, with backoff, and abandons a debt once after REMOTE_STOP_OWED_RETRY_WINDOW_MS (24 h, :118 (origin/main e4a79cda2a); the rule at :211 (origin/main e4a79cda2a)). It runs at each entry (runtime-js / the CF DO, runtime-js/src/js-agent-executor.ts:2693 (origin/main e4a79cda2a); CF Workflows, runtime-cloudflare/src/steps.ts:3399 (origin/main e4a79cda2a)) and in the stop cascades (runtime-js/src/execution/in-process-runs.ts:290 (origin/main e4a79cda2a); CF Workflows' stopChildrenOfRun, runtime-cloudflare/src/stop-children.ts:195 and :477 (origin/main e4a79cda2a)). The CF DO's own settle cascade is not wired: its SQL reads live refs only (WE-2, stated in the guide). A dispatch never re-opens a ref that owes a stop. !480 also filed FU-REMOTE-OWED-STOP-RETRY-SERIALISE and FU-JS-WRITEOWNREF-REOPEN-RULE | source, peer-reported | verified | fixed (no lane can reach it: 3 tests) |
| CP-204 | S3 | CF Durable Object: a parent turn's companion completion notifier runs before the parent's owed verified reads, so a settled companion's outcome can reach the parent one turn late. The child's settle notice only records the parent's verify debt and starts the verified read in the background; a new parent turn never runs owed reads before its notifier reads the refs, so a message sent after the child ended and its notice arrived, but before the background read landed, starts a turn that sees the ref still running and delivers the outcome a turn later, against the guide's next-turn contract. | runtime-cloudflare/src/do/companion-settle-notice.ts:349-382 (origin/main 5af6e15645) (handleChildSettledNotice: setChildVerifyOwed records the debt, the alarm is re-armed, then verifyOwed() runs under waitUntil in the background and the route answers without waiting for it); runtime-cloudflare/src/do/durable-object-agent-base.ts:1142-1143 (origin/main 5af6e15645) (verifyOwedChildren, the owed-read runner of :647-649, same pin), called only from the alarm at :2336 (origin/main 5af6e15645) (same pin, alarm.fire_expired) and the notice route at :3212 (same pin, origin/main 5af6e15645); no turn entry runs it before the run loop's completion notifier reads the refs; contrast: docs/guide/sub-agents.md:268 (origin/main 5af6e15645) ('A child can outlive its parent's turn': the parent learns the outcome on its next turn); live: the incident replay spec's companion test K3 (examples/opennext-cloudflare-do, !474) asserts the parent's next turn carries the companion's verdict exactly once; its delivery assertions are parked under FU-OPENNEXT-COMPANION-K3-DELIVERY-RACE until the fix lands (WM, peer-reported, 2026-10-20; surfaced by Greptile on !474 and validated in code); related: found via !474's review, filed at the orchestrator's direction (WM, 2026-10-20), verified by the owner at the sites above. Fixer: WE-2 (reassigned from WA by the orchestrator, 2026-10-23); its MR re-enables K3's delivery assertions as the e2e witness; severity S3 (provisional, owner, 2026-10-20): the outcome is not lost, only delivered one turn later, and only when a send lands inside the background verified read's window; related: the order that opens the window (WE-2, 2026-10-23; verified by the owner at origin/main 8cd8beb434): the child's settle finalizes its own stream first, runtime-cloudflare/src/do/drive/settle.ts:396-397 (origin/main 8cd8beb434) (step 2: the child's snapshot reads ended), and only then rings the parent, :406-408 (same pin, origin/main 8cd8beb434) (step 5a, ringParent); the parent's handleChildSettledNotice (runtime-cloudflare/src/do/companion-settle-notice.ts:356 (origin/main 8cd8beb434)) starts the verified read under waitUntil at :382 (same pin, origin/main 8cd8beb434), and runtime-js's notifier delivers only a ref already completed / failed with a recorded outcome (runtime-js/src/run-loop.ts:1522-1524 (pre-fix, origin/main 8cd8beb434), same pin). So a send between the child's ended stream and the parent's recorded outcome is delivered a turn late; runtime-js, CF Workflows (checkCompletedChildren) and Temporal / DBOS (core completion-delivery) read the child at the notifier pass over a shared store instead. WE-2's fix: the run's first notifier pass makes its own verified read of every outstanding consult; related: fixed by !485 (WE-2, merged as 5966973e80): before each completion-notifier pass the run loop refreshes the notifier, telling it the run's first pass (runtime-js/src/run-loop.ts:1457-1460 (origin/main 5966973e80)); on the CF DO the refresh makes its own verified read of every outstanding consult, waiting for the owed reads up to COMPANION_REFRESH_WAIT_MS (runtime-cloudflare/src/do/companion-settle-notice.ts:630 (origin/main 5966973e80), refreshCompanionOutcomes at :662 (origin/main 5966973e80), the notifier's refresh at :729 (origin/main 5966973e80)), wired in runtime-cloudflare/src/do/durable-object-agent-base.ts:916 (origin/main 5966973e80). So a send right after a companion ended carries its outcome in that turn, whether the notice never reached the parent or its background read had not landed. runtime-js in-process is CP-207's; related: the e2e opennext incident-replay K3 delivery assertions are re-enabled as a regression guard only: they passed at the pre-fix product too (the race did not occur there), so they are not this record's witness (WE-2) | source, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-205 | S3 | CF Durable Object: re-spawning a persistent companion leaves that companion's OWN companions' sessions behind, so the fresh companion's first spawn of the same-named child continues the old child's history. When parent P re-spawns companion x, the DO stops x's live children and wipes x's session (its refs included), but each child y keeps its own DO session; x-new's first spawnAgent of y then starts a new turn on y's old session, and the model gets a 'spawned' child that remembers a previous incarnation. On runtime-js and CF Workflows the re-spawn's session delete removes y with x, so y starts fresh. Fixed on the CF DO by a7fb2d8841: every companion start, a first spawn included, asks the child DO for a fresh session, so the child wipes a session an earlier incarnation left. | runtime-cloudflare/src/do/drive/run-drive.ts:516-520 (origin/main 78b21c1683) (planRespawn: before the wipe, the old generation's live companions are only STOPPED, host.stopChildren(…, 'abort', 'parent_respawned'); the comment at :517-518 (same pin, origin/main 78b21c1683) states the divergence: 'the other stores' re-spawn deletes them with the session'); runtime-cloudflare/src/do/drive/run-drive.ts:461-475 (origin/main 78b21c1683) (CP-142's respawn plan: the session is wiped by DriveHost.wipeSessionForRespawn; nothing deletes a child DO's session); inferred: runtime-js and CF Workflows delete the respawned companion's session through core deleteSession, whose cascade deletes its child sessions, so x-new's spawn of y starts y fresh (WA; not run against both by the owner); related: found by WA in CP-161's Greptile round (!476; peer-reported, 2026-10-22), verified by the owner at the DO sites above; it predates CP-161, which only made the re-spawned spawn succeed again (the generation floor). WA files it as FU-DO-RESPAWNED-PARENT-CHILD-CONTINUES-OLD-SESSION in !476's follow-ups.md, citing this id; severity S3 (owner, 2026-10-22): it needs a companion that has companions of its own and is then re-spawned; the stale child is silent (its answers draw on a previous incarnation's history), nothing is lost; related: why the first spawn lands on the old session (WE-2, 2026-10-23; verified by the owner at origin/main 8cd8beb434): the child id is name-derived, core/src/orchestration/companion-tool-dispatch.ts:745 (origin/main 8cd8beb434) (generatePersistentSubSessionId(parentSessionId, childName)); a re-spawned parent holds no ref, so it takes the first-spawn branch (:752 (pre-fix, origin/main 8cd8beb434) isFirstSpawn, :805 (pre-fix, origin/main 8cd8beb434), same pin); and the DO sends the fresh-session headers only for a respawn, runtime-cloudflare/src/do/persistent-companion-tools.ts:1376-1377 (same pin, origin/main 8cd8beb434), so the first spawn posts a plain /start onto the child's old session. Fixer: WE-2 (orchestrator, 2026-10-23); its witnesses (branch reliability-we-cp205-first-spawn-fresh, unpushed): runtime-cloudflare cp142-companion-respawn-fresh.integ.test.ts 'CP-205: a re-spawned parent's first spawn of a name starts that companion on a fresh session' with a first-spawn guard, do-persistent-agents.test.ts, and runtime-js cp205-respawn-tree-parity.test.ts; related: fixed in code by a7fb2d8841 (WE-2, !484 per the code comments), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/persistent-companion-tools.ts:1451-1478 (origin/main 56d564d073) (every companion start, a first spawn included, sends the fresh-session and reject-concurrent-entry headers; only a claim replay whose probe found no child starts plainly, so the child's planRespawn wipes a session an earlier incarnation left), runtime-cloudflare/src/do/persistent-companion-tools.ts:1416 (origin/main 56d564d073) (a replay that finds another consult's run, an earlier incarnation's child, also starts fresh); its unit and integration witnesses runtime-cloudflare/src/do/__tests__/cp142-companion-respawn-fresh.integ.test.ts and runtime-js/src/__tests__/cp205-respawn-tree-parity.test.ts are on main | source, peer-reported | partly-inferred | open |
| CP-206 | S3 | runtime-js: a stop that cascades to an EPHEMERAL in-process child ends the child's run but leaves the parent's ref for it reading running: the cascade writes the parent's ref only for a persistent child. A parent suspended on an ephemeral child (itself suspended on a client tool) that is aborted leaves the child's run failed and its ref { mode: 'ephemeral', status: 'running' }; CF Workflows' cascade writes every local ref. Fixed on runtime-js and the CF DO by a80920762b: the cascade now writes the ref of every child it ends. | runtime-js/src/execution/in-process-runs.ts:326-336 (origin/main 78b21c1683) (stopChildren: after a child it stopped settles, the parent's ref follows the settled status only if ref?.mode === 'persistent'); live: W1's lock-free probe on !465's tree (peer-reported, 2026-10-22, an untracked test, in-process-runs unchanged): executor.stop(parent, { kind: 'abort' }) over a parent suspended on an ephemeral child suspended on a client tool left the child's run failed and the parent's ref { mode: 'ephemeral', status: 'running' }; contrast: CF Workflows' cascade writes every local child's ref from its terminal (runtime-cloudflare/src/stop-children.ts:269 (origin/main 78b21c1683), recordChildTerminalRef, called for every stopped local child at :217 (origin/main 78b21c1683)); related: CP-201 (the FAILURE path leaves ephemeral children running behind a running ref); this is the STOP path's ref, filed separately as a pre-existing defect (orchestrator ruling, 2026-10-22). Fixer W1, in CP-201's MR a (runtime-js + DO), with its own red on the stop path; severity S3 (owner, 2026-10-22): the child's run did end, so nothing keeps running; the parent's child view (getSubSessionRefs) reads a stopped ephemeral child as live; related: fixed in code by a80920762b (W1, CP-201 (a) + CP-206), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-js/src/execution/in-process-runs.ts:425-442 (origin/main 56d564d073) (the stop cascade writes the parent's ref for every child it ends, companion or ephemeral, scoped to the ref's generation; the CF DO inherits it through runtime-js), runtime-js/src/execution/in-process-runs.ts:415 (origin/main 56d564d073) (the persistent-mode check survives only to decide whether the owner must still be current) | source, peer-reported | verified | open |
| CP-207 | S3 | runtime-js in-process: a persistent companion's parent ref turns terminal only after the child's run has ended (its K5 committed and its stream finalized), and the parent's completion notifier skips a ref that still reads running before it reads the child. A parent turn whose notifier pass runs between the two, two awaits apart in one process, delivers the companion's outcome one turn late. Core's completion delivery (Temporal / DBOS) and CF Workflows' checkCompletedChildren read every undelivered persistent ref's child and sync a stale ref; the CF DO's wider window of the same class is CP-204. | runtime-js/src/run-loop.ts:7369 (origin/main 8cd8beb434) (the spawn path awaits companionCtx.runSubAgentLoop, which returns once the child's run has ended) and :7385-7387 (same pin, origin/main 8cd8beb434) (only then updateSubSessionRef(parent, { status: 'completed' })); the continue path the same, :7733 and :7740-7742 (same pin, origin/main 8cd8beb434); a failed child's ref is written after the loop throws too, :7402 and :7757 (same pin, origin/main 8cd8beb434) (recordPersistentChildNonCompletion); runtime-js/src/run-loop.ts:1522 (origin/main 8cd8beb434) (the notifier pass skips a persistent ref whose status is not completed / failed, before any read of the child at :1523-1530, same pin); contrast: core deliverPendingChildCompletions (core/src/orchestration/completion-delivery.ts; Temporal / DBOS) and CF Workflows checkCompletedChildren read every undelivered, non-terminated persistent ref through readChildNotice and lazy-sync a stale one (WE-2; not re-read by the owner); live: WE-2's red, runtime-js/src/__tests__/run-loop-persistent.test.ts 'CP-204: a child whose K5 landed while the parent ref still reads running is delivered in the next turn and its ref synced' (branch we2-backup-cp204-inproc-red, commit a46a7c906b; red at 8cd8beb434, with the guard 'a running ref whose child is still live is left for a later pass' green; peer-reported, 2026-10-23); related: found by WE-2 while fixing CP-204 and filed per the orchestrator's ruling (peer-reported, 2026-10-23); verified by the owner at the sites above. WE-2 cited :7410 / :7765, which are the failure branches; the completed writes are :7385-7387 / :7740-7742. Widening runtime-js's notifier read as core does would break C16 deliver-then-continue (a child settled under a running ref is delivered by the re-consult call as priorResult; ruled unchanged by the orchestrator), so WE-2's direction is to write the parent's ref terminal from the child's own terminal sequence, before its stream finalizes, minding the terminate race noted at runtime-js/src/run-loop.ts:7376-7384 (origin/main 8cd8beb434); related: CP-204 (the CF Durable Object: the child ends its stream before it rings the parent). Fixer WE-2, after its CF work, as FU-JS-INPROC-COMPANION-REF-AFTER-STREAM-END; severity S3 (owner, 2026-10-23; WE-2 proposed S3 or S4, and the scale stops at S3): the window is two awaits in one process, and the outcome arrives a turn late, never lost | source, peer-reported | partly-inferred | open |
| CP-208 | S2 | runtime-js: retry() runs its loop with no sub-agent resolver, so a retried run whose model calls a LOCAL sub-agent gets that call failed ('no resolver configured') as its tool result, the child never runs, and the parent carries on (in W1's repro, to completion). execute(), resume() and the resume loop all pass the resolver; retry, the recovery path after a failure, silently loses delegation. Remote sub-agents, the CF DO (every sub-agent tool is remote there) and CF Workflows (children are workflow instances) are unaffected. | runtime-js/src/js-agent-executor.ts:6441-6451 (origin/main e4a79cda2a) (retryEntry, :6053 (origin/main e4a79cda2a), calls runWithNewRunLoop with no resolveSubAgent), against executeEntry :4747 and :4757 (origin/main e4a79cda2a), resume :5957 and :5977 (origin/main e4a79cda2a) and the resume loop :8016 (origin/main e4a79cda2a), which pass resolveSubAgent: subAgentResolver; runtime-js/src/js-agent-executor.ts:7661-7666 (origin/main e4a79cda2a) (runWithNewRunLoopInner's own comment: 'retry() may omit them and a sub-agent call surfaces RunOutcome.failed'), and runtime-js/src/run-loop.ts:3703-3711 (same pin, origin/main e4a79cda2a) (runSubAgentCall answers failed: 'runLoop: sub-agent call … has no resolver configured'); runtime-js/src/run-loop.ts:2512 (origin/main e4a79cda2a) (a remote sub-agent call is routed to executeSingleRemoteSubAgentCall, which needs no resolver): so remote sub-agents and the CF DO, which makes every sub-agent tool remote, are unaffected (W1); live: W1's repro during CP-201's tests on !481's branch (peer-reported, 2026-10-23; in-memory store): a parent with a local createSubAgentTool child fails its first model call; executor.retry(parent, sid) runs, its model calls subagent__<child>, no child model call happens, the call's tool result is the 'no resolver configured' error, and the parent completes. The same call in a new turn (execute()) dispatches the child; related: found by W1 and ruled a finding, not a known limit, by the orchestrator (2026-10-23); verified by the owner at the sites above. CF Workflows has no resolver (its children are workflow instances whatever the entry); Temporal and DBOS retry were not checked (parked); severity S2 (owner, 2026-10-23; W1 proposed S2 / S3): retry is the recovery path after a failure, and it silently degrades an agent that delegates to a local sub-agent, with no error reaching the caller; the run can complete on the failed delegation; related: fixed by !487 (W1, merged as a6b42c95a3): retryEntry builds the user hook manager and passes it with the sub-agent resolver and the persistent-child adapter resolver, as resume() does, runtime-js/src/js-agent-executor.ts:6448-6461 (origin/main a6b42c95a3); the same MR fixes CP-209 (the retry's missing userHookManager) | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-209 | S3 | runtime-js: retry() runs its loop with no user hook manager, and the loop's child inheritance falls back to the parent's agentHooks only when there is no usageContext, so under usage tracking a retried run's in-process sub-agent children inherit none of the parent's user hooks (onAgentStart / onAgentComplete / tracing stay silent for them). execute() and resume() pass one; the CF DO's /retry runs runtime-js. Reachable once CP-208's fix let a retried run reach a local child at all. | runtime-js/src/js-agent-executor.ts:6441-6451 (origin/main 71f55dac3a) (retryEntry calls runWithNewRunLoop with no userHookManager), against resume, which builds one at :5955 (origin/main 71f55dac3a) (buildUserHookManager(agent)), and execute, which passes one; runtime-js/src/run-loop.ts:4028-4032 (origin/main 71f55dac3a) (runSubAgentCall: the child's parent user hooks are input.userHookManager, else the parent's agentHooks only when usageContext is undefined) and :2364-2366 (origin/main 71f55dac3a) (the awaited-children resume, the same rule); related: found by W1 while fixing CP-208 (peer-reported, 2026-10-23; the orchestrator asked W1 to file it with the owner); verified by the owner at the sites above. Temporal, DBOS and CF Workflows were not traced: their children are child workflows / instances with their own hook wiring; severity S3 (owner, 2026-10-23): observability only — user hooks silently missing for a retried run's children; nothing persisted is wrong; related: fixed by !487 (W1, merged as a6b42c95a3, with CP-208): retryEntry builds the user hook manager, runtime-js/src/js-agent-executor.ts:6448 (origin/main a6b42c95a3), and passes it to the loop at :6455 (origin/main a6b42c95a3) | source, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-210 | S2 | CF Durable Object (regression from !484): since CP-205's fix every companion /start except a replay that found no child asks the child to start on a fresh session, and the child's respawn plan wipes any non-live session of the same parent. A duplicate delivery of one consult's /start therefore wipes the child it already started: when a parent's first spawn is evicted before the answer, its replay finds no child and starts it plainly, and if that run completes before the original headered /start is delivered, the late delivery wipes the completed child's history, state and output. Before !484 the same delivery ran a new turn on the child. The parent sees nothing. Fixed on the CF DO by 3d126a6728 (!488): a duplicate of a served consult is answered busy, never a wipe. | runtime-cloudflare/src/do/persistent-companion-tools.ts:1475-1477 (origin/main e3f03e8196) (the fresh-session and reject-concurrent headers go on every child /start except a plain replay that found no child, CP-161 fix 4); runtime-cloudflare/src/do/drive/run-drive.ts:516-517 (origin/main e3f03e8196) (planRespawn refuses only a LIVE run, busy) and :542 (origin/main e3f03e8196) (otherwise it wipes the session): nothing compares the current run's consult with the /start's, so a duplicate of the same consult reads as a re-spawn; live: WE-2's Miniflare cases on !488 (runtime-cloudflare/src/do/__tests__/cp142-companion-respawn-fresh.integ.test.ts, the three '(b)' cases; peer-reported, 2026-10-24): at main e3f03e8196 the duplicate answers 200 and the completed child's messages, customState and output are gone; related: filed as FU-DO-COMPANION-FRESH-START-DUPLICATE by !484's independent review (LOW, owner WE-2); the orchestrator ruled it a finding (2026-10-24): silent data loss introduced by a merged MR (!484, merged as a7fb2d8841). The FU stays and cites this id. Its other order, two same-name first spawns in one step (a), is already closed on main by CP-161's claim (!476): a claim lost to another call's live ref answers child_busy before any child IO, and one lost to a COMPLETED ref continues the child without fresh-session headers (WE-2's trace); its pin is WE-2's guard MR after !488; related: fixer WE-2. !488 (b): planRespawn refuses to wipe a session whose current run serves the SAME consult and generation as the /start, answering it busy (a duplicate delivery), so a re-spawn (always a new consult, or generation g + 1) still resets; severity S2 (orchestrator, 2026-10-24): a narrow window, but a finished companion's history, state and output are lost silently; related: (b) fixed by !488 (WE-2, merged as 3d126a6728): planRespawn answers a start whose consult the current run already served as a duplicate (never a wipe) unless it names a newer parent-ref generation than the one that consult started at, runtime-cloudflare/src/do/drive/run-drive.ts:540-545 (origin/main 3d126a6728) (the 'duplicate' plan, :494 (origin/main 3d126a6728)). The record stays open until (a)'s guard pin (WE-2's guard MR) and the cf-do cell land (orchestrator, 2026-10-24); related: fixed in code by 3d126a6728 (WE-2, !488), with (a)'s guard pin 395462f20a, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/do/drive/run-drive.ts:540-545 (origin/main 56d564d073) (planRespawn answers a start whose consult the current run already served as a duplicate, never a wipe, unless it names a newer parent-ref generation), runtime-cloudflare/src/do/drive/run-drive.ts:1368 (origin/main 56d564d073) (the drive answers a duplicate busy, duplicate_consult) | source, live-repro, peer-reported | verified | open |
| CP-211 | S3 | CF Workflows: a SUBMITTED ERROR on an approval-gated client tool is recorded as framework_internal_error ('… has no approvalId') instead of the submitted error, and the tool's afterTool never runs. The approval-answer check needs an approval-response result, which a submitted error has none of, so the answered entry falls into CP-154's 'approval-gated call without its approvalId' breach before the error-result branch that records a plain client tool's submitted error. The chat handler's deadline sweep submits exactly that error for a lapsed approval, as can any client. runtime-js and the CF DO record the error row and run afterTool. Fixed on CF Workflows by 40a6012d92 (WA, !490): a stored error is classified before any approval-answer check and drains as that error. | runtime-cloudflare/src/steps.ts:8368-8371 (origin/main e3f03e8196) (applyResultsAndReload: isApprovalSubmission requires submittedAt, an approvalId and an approval-response submittedResult; a submitted error has no submittedResult) and :8376-8385 (origin/main e3f03e8196) (an approval-gated tool's answered entry that is not an approval answer is refused as a framework_internal_error breach and paired); related: the breach branch came with a971537989 (RM-87 + CP-154, 2026-10-19), which made an approval classified by its stored approvalId instead of by guessing; a submitted error was never an approval answer, so it was caught by the guard meant for a missing id; contrast: runtime-js (and the CF DO, which runs it) builds the error result whenever the drained outcome is not ok, before any approval classification (runtime-js/src/js-agent-executor.ts:3418-3421 (origin/main e3f03e8196)), so the error row lands and afterTool runs; Temporal handles a submitted error too (WA); related: found by WA in !490's Greptile round (CP-173) and ruled a separate pre-existing defect by the orchestrator (2026-10-24); verified by the owner at the sites above. NOT CP-173's own lapse regression (its fix round 1, 1c4a25a31b, made the CF Workflows drain save a lapse first, which reached the same branch for an unsubmitted lapsed approval; fixed in the same commit); severity S3 (owner, 2026-10-24): the model receives a framework error in place of the submitted one and the tool's afterTool hook is skipped; the call is still paired, nothing is lost; related: fixed in code by 40a6012d92 (WA, !490), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/steps.ts:8897-8902 (origin/main 56d564d073) (applyResultsAndReload classifies a stored error, a lapse or a submitted error, before any approval-answer check, so it is never an approval submission), runtime-cloudflare/src/steps.ts:8907-8913 (origin/main 56d564d073) (the approval-without-approvalId breach now skips a stored error, which drains through the error-result branch with its row and afterTool); its red-first unit test is runtime-cloudflare/src/__tests__/steps-applyResultsAndReload.test.ts:859 (origin/main 56d564d073) | source, peer-reported | verified | open |
| CP-212 | S3 | A persistent companion whose first spawn fails, then is spawned again under the same name, never gets the parent's kind:'subagent' usage pointer: core's dispatcher records the pointer only for a first spawn (no existing ref) that started, and a failed first spawn keeps its failed ref, so the later spawn is a respawn and records nothing. The respawned companion's usage is silently missing from the parent's usage view. runtime-js and Temporal (through core's dispatcher) and CF Workflows (its own first-spawn gate, the same assumption); and DBOS (its own first-spawn gate, the same assumption: a failed first spawn keeps its ref, marked failed); the CF DO has no hook here. Fixed on runtime-js, Temporal and CF Workflows by ad1c0d90a7 (every spawn entry that starts the child session records the pointer); still open on DBOS. | core/src/orchestration/companion-tool-dispatch.ts:1223 (origin/main e3f03e8196) (isFirstSpawn = !existingRef) and :1759 (origin/main e3f03e8196) (the pointer is recorded only if isFirstSpawn, after the entry's outcome, so a spawn that did not start leaves no pointer); core/src/orchestration/companion-tool-dispatch.ts:217 (origin/main e3f03e8196) (releaseSpawnClaim: a failed first spawn's ref is kept, status failed and completionDelivered, generation 0), so the next spawn of the name finds a ref and is a respawn; related: pre-existing, not from !476: at 71f55dac3a a failed first spawn already kept its ref (core/src/orchestration/companion-tool-dispatch.ts:819 (origin/main 71f55dac3a), the ref added before the start, and :886-888 (origin/main 71f55dac3a), updated to failed), and the pointer was recorded only when isFirstSpawn (:914 (origin/main 71f55dac3a)); related: found by WA while researching CP-161's cells (peer-reported, 2026-10-24); verified by the owner at both pins; ruled a finding by the orchestrator (2026-10-24). Fixer WE-2 (CF Workflows first), after its CP-210 guard MR and before CP-207. Fix direction: record the pointer on the first spawn that STARTS a given child session (keyed by the child session), red-first on runtime-js and CF Workflows; severity S3 (orchestrator, 2026-10-24): a silent usage undercount (the companion's model usage never reaches the parent's view); the transcript and the companion's own session are correct; runtime-cloudflare/src/steps.ts:9537 (origin/main e3f03e8196) (CF Workflows' own gate: recordUsage only when entryKind.kind === 'first_spawn', with the same assumption that a respawn's ref already has the pointer), so its fix is separate from core's (WE-2, verified); runtime-dbos/src/steps/execute-companion-tool.ts:636-642 (origin/main e3f03e8196) (DBOS's own gate: isFirstSpawn = !existingRef) and :830-856 (origin/main e3f03e8196) (a start error marks the ref failed with completionDelivered, it does not remove it), so DBOS's next spawn of the name is a respawn and records no pointer: affected (Greptile on !496, verified by the owner at e3f03e8196 and 3d126a6728; it corrects WE-2's earlier 'unaffected' contrast); related: fixed in code by ad1c0d90a7 (WE-2, CP-212) on runtime-js, Temporal and CF Workflows, verified by the owner at origin/main 56d564d073 (audit 2026-10-25): core/src/orchestration/companion-tool-dispatch.ts:1150-1173 (origin/main 56d564d073) (one best-effort pointer write per child session, also on a recorded replay) and :1782 (origin/main 56d564d073) (every spawn entry that is not a continue records it, a respawn included), runtime-cloudflare/src/steps.ts:10065-10070 (origin/main 56d564d073) (CF Workflows marks every started spawn entry recordUsage) and runtime-cloudflare/src/workflow.ts:3580-3587 (origin/main 56d564d073) (the body writes the pointer for it); still open on DBOS: runtime-dbos/src/steps/execute-companion-tool.ts:642 (origin/main 56d564d073) (isFirstSpawn = !existingRef) and :866 (origin/main 56d564d073) (the pointer is written only for a first spawn) | source, peer-reported | verified | open |
| CP-213 | S1 | runtime-js: an agent with workspaceOpenStrategy 'eager' whose first step is terminal (a plain-text final answer, or finish) ends failed with ExecutorSupersededError ('Executor superseded … another executor has taken over') instead of completing. The eager open persists the workspace ref out-of-band before the first step, and the registry's ref writes are adopted (ruling R38) only by the tool-step and suspension commits, so a run that reaches its terminal commit without one CASes on a stale version and reads its own write as a foreign one. Deterministic: through the executor every such run fails, top-level and as an in-process persistent companion child; through the chat route (the CF Durable Object, js-chat-host) a structured-output run ends completed but its output is never persisted (a plain-text run there completes normally). | runtime-js/src/run-loop.ts:936-940 (origin/main 3d126a6728) (the eager open: openAll() then get() before the first step, which persists the workspace ref and bumps the session version); core/src/orchestration/step-iterator.ts:2041-2042 (origin/main 3d126a6728) (the suspension commit adopts the drained ref writes) and :2240-2241 (origin/main 3d126a6728) (the tool-step commit adopts them): the only adoption sites (git grep adoptWorkspaceRefWrites, takeWorkspaceRefWrites at that pin); core/src/workspace/ref-writes.ts:32 (origin/main 3d126a6728) (adoptWorkspaceRefWrites: adopts exactly the versions the run wrote, over the version the commit would CAS on); live: reproduced by the owner at 3d126a6728 (runtime-js unit, real InMemoryStateStore, temporary tests): JSAgentExecutor.execute() of an eager-workspace agent whose step 1 is a text final answer → failed 'Executor superseded for session …' (the workspaceRef was persisted); the same agent with a tool call on step 1, then text → completed; an in-process persistent companion child with an eager workspace whose step 1 is __finish__ → failed 'Executor superseded for session sess-1-agent-helper', and with a lazy open (a tool calling ctx.workspaces.get()) → completed; related: reported by WE-2 while building CP-207's witness (peer-reported, 2026-10-24; the companion repro); widened to top-level runs by the owner; widened to the chat route by the owner's s106 cells. Fixer: W1 (orchestrator, 2026-10-24); core/src/orchestration/step-iterator.ts:1467 (origin/main 3d126a6728) (the thrower: the terminal step's own commitStep → saveStateAndPromoteStaging raises StaleStateError expected version 1, store has version 2, which runtime-js maps to ExecutorSupersededError; pinned by W1's stack trace, peer-reported); severity S1 (orchestrator, 2026-10-24, raised from S2 after s106): on the CF Durable Object's chat path, the one production DO users take, the run ends completed with its structured output never persisted, a silent wrong success; through the executor it is a loud wrong failure (failed instead of completed), deterministic; related: Temporal and DBOS eager workspaces not checked; CF Workflows refuses a workspace agent (framework_not_supported); conformance: fixed by W1's MR (branch reliability-cf-w1-cp213-eager-workspace-terminal, the fix bc6b659d27, head 2936e8e746: right after the eager open, runLoop adopts the registry's R38 ref writes into its state through core adoptWorkspaceRefWrites; peer-reported, W1). Owner slot s106: at main 3d126a6728 + the cells, materialize.eager-workspace-text-first and materialize.eager-workspace-finish-first fail on js-memory|executor (.run-completes: failed 'Executor superseded for session …') and materialize.eager-workspace-finish-first fails on cf-do|host and js-chat-host|host (.run-completes: completed, the persisted output undefined); at 2936e8e746 + the cells all four pass, with the eager open's in-memory workspaceRef persisted. materialize.eager-workspace-text-first on cf-do|host and js-chat-host|host passes at both shas, so it is not a witness; related: since CP-128's MR D, ExecutorSupersededError and the executor_superseded chunk are gone: a step commit that loses a version race is StepCommitConflictError (state_concurrency_conflict). The pinned lines above keep the old names | source, peer-reported | verified | fixed (4 cells) |
| CP-214 | S2 | CF Workflows answers a persistent companion whose run suspends PARTIAL (suspended_step_partial: a step with a server tool beside a client or approval call) as failed with no errorDetail: the child notifies its parent 'suspended', and the parent's ref-status resolver maps every status but complete / terminated to failed, so a companion that is only waiting on a person is reported as having failed. | runtime-cloudflare/src/workflow.ts:1244-1252 (origin/main 64110bd5ac) (the child notifies its parent status 'suspended', with suspension.childStatus, for any of the three suspensions); runtime-cloudflare/src/companion-ref-status.ts:44-57 (origin/main 64110bd5ac) (resolvePersistentChildRefStatus: complete → completed, terminated → interrupted / the caller's, everything else, suspended included, → failed); inferred: the parks that consume it (WA's trace, not run by the owner): the spawn / respawn park answers parkTerminalOf failed with no errorDetail, and the blocking continue and companion__waitForResult read it the same way; related: FU-CFW-COMPANION-WAIT-SUSPENDED-CHILD-AS-FAILED (follow-ups.md, LOW, owner WB) narrows to this id; WA found it with C2; related: narrowed to suspended_step_partial (orchestrator, 2026-10-29): suspended_client_tool is answered suspended since CP-40 (C2) (runtime-cloudflare/src/companion-ref-status.ts:72-88 (origin/main e34b96e2c2)); suspended_awaiting_children is CP-249 (a different fix: the companion parked on its descendant is also never re-driven); the fixer is WA's !584, step_partial only; related: runtime-js's twin (the same suspended_step_partial companion answered completed with no output) is CP-248; conformance: fixed by !584 (2ea34599a2, batch 3). companion.suspends-mixed-step (owner slot s214, 2026-10-31, 3 of 3, identical sets): .spawn-answers-suspended and .ref-paused fail on cfw-workflows-d1 with the cell over main e3eee732d0 (tree A: the owner's member fa94fe1729) and pass every id with !584 2ea34599a2 merged in (tree B); related: CP-279 (2026-11-06) rules this record's deferred reachability for one case: a suspended_awaiting_children event whose child has a REMOTE child parked on client / approval calls is answered suspended on those calls (runtime-cloudflare/src/companion-ref-status.ts:85-90, as on branch reliability-w1-cp279-companion-outlives-root a80d04453c); a wait on LOCAL children keeps its failed mapping (Q3) | source, peer-reported | partly-inferred | fixed (1 cells) |
| CP-215 | S2 | On runtime-js a persistent companion that fails with a terminal model error writes its error chunk into its PARENT's chat stream mid-turn, so a stock useChat (which stops reading at the first error event) reads the parent's turn as failed although the parent goes on and completes. A companion's run writes into the parent's stream (its streamId is the parent's), and its one terminal error chunk is written there like the parent's own. | runtime-js/src/js-agent-executor.ts:4506 (origin/main 64110bd5ac) (an entry's stream is options.parentStreamId when given: a companion's run writes into its parent's stream); runtime-js/src/execution/persistent-subagent-executor.ts:22 (origin/main 64110bd5ac) (the companion execution context carries the parent run's parentStreamId); conformance: companion.child-error-not-parent-error-host and companion.child-error-not-parent-error-non-blocking-host (owner batch 49) fail .no-child-error-event and .turn-completes on js-chat-host (host), 3 of 3 each at owner batch 4ab60e4105 (main fb9150cc16 + the owner batch), 2026-10-24 (owner slot s109), and again 3 of 3 in owner slot s110 (batch 3c06825871): the parent's chat stream ended on { terminal: failed, source: stream:error, error: 'conf CP-215 companion failure' }; preconditions .parent-held and .child-failed pass. On cf-do both pass (the contrast: the DO runs each companion in its own DO with its own stream); conformance: companion.respawn-tree-first-spawn-fresh-host (CP-205's scenario) fails .turn-completes, .first-generation-failed and .fresh-grandchild on js-chat-host, 5 of 5 in owner slot s108 and 3 of 3 in s110: its first scribe generation fails with a model error, which ends the editor's chat stream; that row is ledgered here (it is CP-205's cell; the unexpected pass after this fix makes its owed re-check); inferred: CF Workflows (code-traced, not run: a companion child writes into the parent's stream on the parent's instance; runtime-cloudflare/src/workflow.ts:3111-3112 and steps.ts:2611-2620 (origin/main fb9150cc16)); on CLOUDFLARE-BACKLOG as suspected; related: owner disposition at filing (2026-10-24): Needs a fixer: a companion's terminal failure reaches its parent as the spawn / wait answer, never as an error chunk on the parent's stream (runtime-js; CF Workflows to be checked). A cfw cell needs a lane that observes a chat stream.; ai-sdk/src/transformer/stream-transformer.ts:732-745 (!820, as on origin/main 41ea410715) (the fix, shared with CP-307: a child's error chunk, here a companion's guest chunk on the parent's stream, is dropped from the parent's response with a debug log at :722-728; the stored chunk is unchanged); conformance: fixed. WE-2's judged slot w-cpj (2026-11-10) at 741e01ce8b, 3 of 3 runs: companion.child-error-not-parent-error-host, companion.child-error-not-parent-error-non-blocking-host and companion.respawn-tree-first-spawn-fresh-host pass on js-chat-host (host), where the red-first record w-cpr at 2eae8504b9 (main 7b92366025 plus the tests) had them fail exactly their ledgered ids; under MF (the fix's guard removed) they fail those ids again; related: CF Workflows: covered by construction (the fix is in the ai-sdk transformer every chat host uses); not witnessed, since no conformance lane observes a CF Workflows chat stream (ruling D3, orchestrator, 2026-11-10) | source, live-repro | partly-inferred | fixed (3 cells) |
| CP-216 | S2 | A parent whose step is lost while it waits on a remote sub-agent (its process dies mid-wait) bills a SECOND child for the same delegation: the child's ref and /start precede the step's commit, so the recovered run truncates the step, stops the old child (parent_step_lost) and calls its model again, and a real provider mints a new call id, so the re-delegation starts, and bills, another child. The old child's paid-for work is discarded. Documented as D3; its cost (GitLab #132: a clip child is about $0.63 mean, $1.13 max, a median of 34 model calls) is the defect. | runtime-js/src/js-agent-executor.ts:2799-2801 (origin/main 64110bd5ac) (reconcile: a ref whose call did not survive truncation is a genuine orphan, "the re-rolled LLM mints a NEW toolCallId (a NEW child session)"); runtime-js/src/js-agent-executor.ts:5698-5700 (origin/main 64110bd5ac) (every entry sweeps the lost step's remote children, D3 / CP-194: stops them and settles the refs failed parent_step_lost before reconcile); related: docs/guide/remote-agents.md:1103-1110 (origin/main 64110bd5ac) (D3 as documented: the ref is registered and the child started BEFORE the step commits; a run that ends before that commit leaves the child to be stopped and its ref settled failed); conformance: subagent.remote-lost-step-one-child-host (owner batch 49 62b38a2c38) fails .one-billed-child and .billed-result-delivered on cf-do (host), 3 of 3 at owner batch 3c06825871 (main fb9150cc16 + the owner batch), 2026-10-24 (owner slot s110): child A's held call (precondition .child-a-billed) was billed before the evict, the recovered parent re-delegated (call B) and completed with child B's answer; its preconditions .child-a-held, .child-a-billed and .turn-completes pass; inferred: runtime-js has the same order (the CF DO runs its executor); no JS lane can lose its process mid-wait (js-chat-host has no evict), so the JS half is code-traced only; inferred: CF Workflows (partly): a same-instance replay re-uses the cached LLM step.do (workflow.ts:4302) and re-attaches the child (steps.ts:5849-5873), but an instance that dies reaches the same sweep; not determinable on the cfw lane; related: GitLab #132 (the app's G1, via the orchestrator's intake, 2026-10-24); two fix directions keep the cell green: commit the delegation before the wait, or adopt the orphaned child for the re-emitted call; related: owner disposition at filing (2026-10-24): Needs a design decision (commit the delegation before the wait, or adopt the orphan for the re-emitted call) and a fixer; the cf-do cell is in place and ledgered.; core/src/orchestration/step-iterator.ts:2048-2067 (fix: !673, WE-2; G1) (save the call first: a reply that delegates to a remote sub-agent is committed before any call of the step starts, marked as a delegation row), runtime-js/src/run-loop.ts:4093-4100 (the commit) and :1590-1601 (an entry re-attaches each remote call of the row, and pairs its other unpaired calls parent_step_lost); runtime-cloudflare/src/steps.ts:4705 (!676), runtime-temporal/src/workflow.ts:1357 (!677, behind the helix-delegation-commit-v1 patch) and runtime-dbos/src/workflows/shared.ts:3631 (!681) do the same; conformance: subagent.remote-lost-step-one-child-host passes on cf-do at the G1 batch 0343342300, landed as 98fb3973d4 (owner slot s276, 2026-11-04), 3 of 3 identical: one child billed for the delegation, its result delivered; .one-billed-child and .billed-result-delivered were this record's ledger row; conformance: a regression inside G1, never on main: on runtime-js and the CF DO a message-carrying entry over a lost delegated row left the row's calls unpaired mid-history (!673, fixed before landing). transcript.lost-delegation-message-entry-paired-host, -retry-host and -active-host pass on cf-do 3 of 3 at 944125fd63 (WE-2's slot w-g1c2), and with the pre-fix rule restored at the three js-agent-executor.ts sites each fails exactly .every-call-paired and .pairs-precede-message 3 of 3. That red comes from a missing result, so the after-the-message order is shown by the self-test lost-delegation-message-entry.test.ts at 3cbbf3138a; the cells were not re-run for that extraction; related: the same regression's active door on Temporal and DBOS (a with_message resume over a lost row of an active session skipped the heal), fixed in G1. No lane can run it there (no remote-child producer, no lost-run control), so the proof is real resume() tests, delegation-commit.integ.test.ts on each runtime: 'a workflow terminated mid-wait, then a with_message resume of the active session: every call paired before the message' (runtime-temporal) and 'a workflow cancelled mid-wait, then a with_message resume of the active session: every call paired before the message' (runtime-dbos; local only, as !681 states), WA's slot slot-g1t at 6a0a8df205, red by the probe at c2e3cc7309 and M-t-guard / M-b-guard. CF Workflows guards both entry sites (run-entry.ts, steps.ts), pinned by W3c and W3d; related: fixed in G1, never on main: DBOS stored a remote tool's remote agent type in SubSessionRef.agentType where every other runtime stores the local name, so G1's lookups missed a renamed tool. !681 stores the local name (execute-remote-subagent.test.ts 'a renamed remote tool: the ref holds the local name, the start the remote one', red by m-b-name; settle and stop pins in delegation-entry-settle.test.ts), and the field's comment states its meaning; conformance: in CI, jobs 17761624437, 17762456338 and 17763113921 (test:conformance 1/7, pipeline 3030641083, merge a49f3d0617 of the G1 head that landed as 98fb3973d4): subagent.remote-lost-step-one-child-host passes 3 of 3, with transcript.lost-delegation-message-entry-paired-host, -retry-host and -active-host, and no harness unhealthy line; related: Save-First Tools MR 1 (!754; spec §6, the owner's rulings 2026-11-06): a stop committed with no executing owner now pairs a saved reply's own unfinished calls run_stopped, and leaves to the re-attach or continue a call that a sub-session ref of this run's call names (the same parentToolCallId, and dispatchId when both carry one), a __finish__, and pending client calls (core/src/run/commit-stop-terminal.ts:193 and :295, as on branch reliability-cf-wa-sft-mr1 8d06533c7b). So transcript.lost-delegation-message-entry-paired-host's interrupted entry asserts a new precondition, .lost-row-stop-paired (the regular call paired once, run_stopped; the delegated call unpaired), in place of .lost-row, which the -retry and -active entries keep: it fails exactly .lost-row-stop-paired and its precondition chain at origin/main 3dda2d7f2e's product (main's own scenario with only the id ported, the port equal to the head's change) and passes at 8d06533c7b, -retry and -active passing at both (WA's slot slot-c2, read by the owner). The exclusion's witness is WA's MEX mutation (slot drv-s9, re-confirmed in slot-s10r): with the ref exclusion removed, core's delegated-left unit, DBOS's B1+B2 integ and CF Workflows' registered-call case each fail; with it, they pass. A __finish__ beside a delegated child is honoured with no model call on continue and on retry (runtime-cloudflare/src/__tests__/cfw-g1-delegation-commit.test.ts:554 and :535, W3g and W3b, green in WA's slot slot-s10r); related: Save-First Tools MR 1 (!754) after merging CP-280: a lost step's recovery removes the lost attempt's frames past the saved cursor (D7), CP-280's announcement of the delegated call among them, and a re-attach wrote none of its own, so the recovered call had no tool_start. The lost-row settle stage now announces it once before the re-attach (runtime-js/src/run-loop.ts:1683, as on branch reliability-cf-wa-sft-mr1 f46938af8b); a parked wait's re-attach (CP-157, no D7) is unchanged. Held by subagent.remote-lost-step-one-announcement-host on cf-do (exactly one tool_start and one tool_end for the call in the stored stream, start before end, after .child-held, .step-lost and .child-reattached): red once at the pre-fix merge head 220552c010 on exactly .one-tool-start and .start-before-end, green 3 of 3 at f46938af8b (WA's slot slot-m2, read by the owner); js-chat-host a declared gap, as for its sibling. An internal fix of the MR, so no id | source, peer-reported | verified | fixed (1 cells) |
| CP-217 | S2 | A parent's stop that cannot reach its REMOTE ephemeral child at first is never retried, so the child the stop missed runs on, and bills, to its own end under an interrupted parent. On the CF Durable Object a remote child's failed stop is logged and never owed (a companion's is owed and the alarm retries it); runtime-js's in-flight dispatch sends the stop once, best-effort, then records the ref failed, so no cascade finds it live. | runtime-cloudflare/src/do/drive/stop-children.ts:98-99 (origin/main 64110bd5ac) ("A REMOTE ephemeral child keeps the old rule: a failed stop is logged and never owed (FU-DO-CASCADE-REMOTE-STOP-UNCONFIRMED)"); runtime-js/src/execution/remote-subagent-executor.ts:1196-1214 (origin/main 64110bd5ac) (on the parent's abort the in-flight dispatch sends the child its stop once; a failure is logged at info as best-effort, and the ref is then written failed, so the stop cascade skips it); conformance: subagent.remote-child-stop-retried-host (owner batch 49 3c06825871) fails .stop-retried and .child-stopped on cf-do (host), 3 of 3 at owner batch 3c06825871 (main fb9150cc16 + the owner batch), 2026-10-24 (owner slot s110): the parent's stop hit the child's faulted stop route (.stop-missed-child, served >= 1) and the parent ended interrupted (.parent-interrupted), but once the route reached the child again its session never read interrupted within 30 s (the owed-stop backoff starts at 1 s), and after the held call was released the child made a second model call and completed; inferred: CF Workflows stamps an unconfirmed remote stop stopOwed on the ref (runtime-cloudflare/src/stop-children.ts:447 (origin/main 64110bd5ac)) with no timer; a later entry retries it (CP-203), so a quiet session never does; not run on the cfw lane; related: GitLab #136 (the app's G5): the app wants no NEW model call after a stop is accepted (one in-flight step allowed); supersedes FU-DO-CASCADE-REMOTE-STOP-UNCONFIRMED; related: fix direction: retry only an UNCONFIRMED stop (owe it like a companion's, paced by the alarm). CONFLICT for the fix MR (orchestrator ruling, 2026-10-24): CP-158's contrast cell subagent.remote-child-stopped-once-host asserts exactly ONE stop request while the child's stop route is armed to fail for the whole cell; a retry of unconfirmed stops sends more, so that cell must be restated in the fix MR (count only stops after the child CONFIRMED one, or before the first retry was due; the owner supplies the records + cell commit); related: owner disposition at filing (2026-10-24): Needs a fixer: owe an unconfirmed remote stop and retry it (DO alarm; runtime-js / CF Workflows equivalents), with CP-158's once-host cell restated in the same MR.; related: fixed on runtime-js and the CF DO by WA's !555 (CP-217 MR 1, head 11dea269e7): an unconfirmed stop of a remote child is owed on its ref (core remoteStopDebtUpdate) and retried until the child confirms it, the DO's stop-cascade alarm and runtime-js's per-session pacer retrying every debt on core's schedule (DEFAULT_OWED_STOP_SCHEDULE: 30 s doubling to 1 h, given up after 24 h). Witnesses (WA, red at the pre-fix commits): runtime-js remote-subagent-executor.test.ts (describe 'CP-217: a remote child's unconfirmed stop is owed on its ref'), stop-children-owed-retry.test.ts, lost-step-remote-sweep.test.ts, remote-orphan-reconciliation.test.ts, remote-owed-stop-arming.test.ts; core retry-owed-remote-stops.test.ts; runtime-cloudflare do/drive/__tests__/stop-children.test.ts and child-stop-owed.test.ts, integration do-stop.integ.test.ts and do-companion-stop.integ.test.ts; conformance: with !555 (head 11dea269e7) subagent.remote-child-stop-retried-host passes on cf-do (host), every id: !555's MR pipeline 2995844407 (job 17515059901, one run; the cell read UNEXPECTED PASS against this record's ledger). Owner ×3 confirmation pending in the !555 green-proof slot; related: remainder (WA, 2026-10-27): CF Workflows is not closed: its entries and cascades retry a SETTLED debt, but its in-flight dispatch still sends its stop once and owes nothing on a live ref, and no timer runs between entries (FU-CFW-REMOTE-STOP-PACER, CP-217 MR 2, WA). Temporal and DBOS are parked (FU-TEMPORAL-REMOTE-CHILD-STOP-OWED, FU-DBOS-REMOTE-CHILD-STOP-OWED); related: the cf-do lane runs the production owed-stop schedule (no lane override): a cell that faults a child's route settles the parent's debt before it ends (scenarios/stop-debts.ts), and the cf-do cell-end guard names a debt still owed as the ending cell's own harness failure (harness/cf-do/owed-stop-guard.ts) | source, peer-reported | partly-inferred | open |
| CP-218 | S2 | On Temporal and DBOS, a remote sub-agent call that times out (createSubAgentTool's timeoutMs) fails the call but never stops the child: the dispatch returns or throws the timeout and writes the ref failed with no stop of the child, and the stop cascades skip a failed ref. The child runs on, and bills, to its own end. runtime-js, the CF DO and CF Workflows stop it since !504 (CP-157 §5). | runtime-js/src/execution/remote-subagent-executor.ts:699-703 (origin/main 64110bd5ac) (a timeout throws 'Remote sub-agent … timed out after Nms' with no stop of the child); runtime-js/src/execution/remote-subagent-executor.ts:1196-1214 (origin/main 64110bd5ac) (the catch sends the child a stop only when batchAbortSignal.aborted; a timeout is not an abort of the batch); core/src/orchestration/remote-sub-agent-dispatch.ts:800-813 (origin/main 64110bd5ac) (core's dispatch returns { kind: 'failed', failureReason: 'timeout' } on the timeout, with no stop either); conformance: subagent.remote-child-timeout-stops-child-host (owner batch 49 3c06825871) fails .child-stopped and .no-call-after-timeout on cf-do (host), 3 of 3 at owner batch 3c06825871 (main fb9150cc16 + the owner batch), 2026-10-24 (owner slot s110): the parent's row names the timeout (.timeout-row) and its turn completes (.turn-completes), but the child's session never reads interrupted, and once its held call is released it makes a second model call and completes; related: the app expects a TYPED timeout result (not a throw or a parent failure) plus a durable stop of the child (orchestrator, 2026-10-24); today's row is an untyped error string. The result's shape is the fix design's (no id asserts a code); related: owner disposition at filing (2026-10-24): Needs a fixer: a timed-out remote call stops its child (owed until confirmed, as CP-217), on every runtime; the typed timeout result is part of its design.; runtime-dbos/src/steps/execute-remote-subagent.ts:618-622 (origin/main 15f24344b3) (DBOS's own dispatch throws 'Remote sub-agent … timed out after Nms' on the timeout; the step calls no stop of the child on any failure); runtime-js/src/execution/remote-subagent-executor.ts:1321-1340 (as on branch reliability-we-cp157-remote-park d438ffb3a7) (fix for runtime-js and the CF DO: !504 3d3eca56c5 interrupts the child of a call that failed for any reason other than the batch's own stop, a timeout included, through core stopRemoteChild, before its ref settles failed); runtime-cloudflare/src/steps.ts:5925-5965 and workflow.ts:3110-3120 (as on branch reliability-we-cp157-remote-park d438ffb3a7) (fix for CF Workflows: !504 3d3eca56c5 adds interruptFailedRemoteChild, called in its own step.do before a failed remote call's ref settles; no cell witnesses it, the scenario is host tier on cf-do only); conformance: subagent.remote-child-timeout-stops-child-host passes (failedIds empty) on cf-do (host), 3 of 3 at !504's d438ffb3a7, 2026-10-25 (WE-2 slot A, the one cell); with only 3d3eca56c5's runtime-js stop disabled (a 1-line mutation) it fails .child-stopped and .no-call-after-timeout again, and at main 15f24344b3 it fails both as before: the stop alone turns both ids. The js-chat-host cell of the same scenario stays expected-fail on !504's pipeline (CP-219); related: narrowed by the owner on 2026-10-25 (orchestrator ruling (b)): !504's cf-do cell passed unexpectedly on its pipeline 2986036157 (jobs 17445744962 and 17445744971), and the ledger's only row, cf-do host, was removed. The finding stays open for Temporal (core's remote-sub-agent-dispatch.ts, above) and DBOS, which !504 does not touch | source, peer-reported | verified | open |
| CP-219 | S2 | createSubAgentTool's timeoutMs is silently ignored everywhere except the CF Durable Object: only the DO's remote rewrite reads the tool's _timeoutMs, so on runtime-js (in-process sub-agents), Temporal, DBOS and CF Workflows a sub-agent call has no wall-clock limit at all, though the guide presents timeoutMs as a per-tool wall-clock limit. | runtime-cloudflare/src/do/durable-object-agent-base.ts:3567 (origin/main 64110bd5ac) (the only reader of _timeoutMs in packages/*/src: the DO rewrite passes it to the remote tool, default 10 minutes); core/src/types/tool.ts:892 (origin/main 64110bd5ac) (createSubAgentTool stores options.timeoutMs as _timeoutMs; nothing else reads it); related: docs/guide/sub-agents.md:47 (origin/main 64110bd5ac) ("timeoutMs: 60_000, // Optional: per-tool timeout in ms"); related: docs/guide/sub-agents.md:1794-1805 (origin/main 64110bd5ac) ("use timeoutMs to enforce wall-clock limits"); conformance: subagent.remote-child-timeout-stops-child-host fails .turn-completes (and its dependents .timeout-row, .child-stopped, .no-call-after-timeout) on js-chat-host (host), 3 of 3 at owner batch 3c06825871 (main fb9150cc16 + the owner batch), 2026-10-24 (owner slot s110): with a 3 s timeoutMs and the in-process child's call held (.child-running green), the parent's turn did not settle within 60 s; inferred: Temporal, DBOS and CF Workflows (code-traced only: none of their sources reads _timeoutMs); executor-tier cells on those lanes come later (orchestrator, 2026-10-24); related: owner disposition at filing (2026-10-24): Needs a fixer: honour timeoutMs on every runtime (or document it as DO-only and refuse it elsewhere), with executor-tier cells on the other lanes. | source | partly-inferred | open |
| CP-220 | S3 | A model call that a stop aborts, but whose provider finished anyway and REPORTED its usage, is not booked on CF Workflows (and on DBOS when the workflow cancel lands): the LLM step's stop branch returns before it records tokens, so usage the provider billed goes missing. (No usage is known before the provider's finish part, so an aborted stream with nothing reported books nothing, which is correct.) | runtime-cloudflare/src/steps.ts:2769-2781 (origin/main 64110bd5ac) (CF Workflows: a stop that aborted the call returns aborted: true before afterLLMCall and before recordTokens at :2861); runtime-dbos/src/workflows/shared.ts:2973 (origin/main 64110bd5ac) (DBOS's own rule: a discarded step's tokens are recorded, "the provider billed them"; its readStop step throws after a landed cancel, before recordStepUsage); inferred: llm-vercel's undefined usage on an abort is correct: ai 6.0.49 surfaces no usage before the finish part (an abort with no finished step rejects totalUsage), so only a call that finished despite the abort has known usage; conformance: usage.stopped-call-reported-usage-booked (owner batch 49 7b3d348878) fails .reported-usage-booked on cfw-workflows-d1 (executor), 3 of 3 at owner batch 13935ac62a, 2026-10-24 (owner slot s116): a held call scripted to ignore its abort (ScriptedModel ignoreAbort) answered usage 4321 / 21 after the run's stop; the session has no tokens entry at all; preconditions .call-held, .stop-reached-call, .run-interrupted pass; conformance: usage.completed-call-usage-booked (the positive control) passes on cfw-workflows-d1, 3 of 3 in the same run (s116): a completed call is booked and readable through observe-usage, so the red is the stop branch, not the channel; related: contract ruling (orchestrator, 2026-10-24): an aborted call's KNOWN usage is booked on every runtime, never estimated; JS / the CF DO / Temporal-interrupt book it once the adapter reports it (code-traced), CF Workflows and Temporal-abort drop it, DBOS drops it when the cancel lands; related: owner disposition at filing (2026-10-24): Needs a fixer (CF-first): CF Workflows books a stopped step's reported usage before the stop branch returns; Temporal's abort and DBOS (parked) later, with observe-usage wired on their lanes.; runtime-cloudflare/src/steps.ts:3097-3115 (origin/main 3dfd151570) (fix evidence: batch 1's squash of W1's CP-242 MR !563, the same stop branch) (the stop branch reports the call that answered: afterLLMCall with the stopped result and its usage, staged under the step so the abort discard drops its writes) with :3073-3096 (origin/main 3dfd151570) (reportModelCall: the hook, and in a finally the call's usage, so a throwing hook loses no usage) and :3040-3066 (origin/main 3dfd151570) (bookModelCallUsage: best-effort, cacheWrite included); red-first unit test runtime-cloudflare/src/__tests__/cfw-stopped-call-usage.test.ts (origin/main 3dfd151570) (W1's red-first test on !563: red over the pre-fix product, 9 green, with mutations re-reddening it); conformance: usage.stopped-call-reported-usage-booked passes on cfw-workflows-d1 at W1's !563 head (its product landed on main as batch 1's squash 3dfd151570) (owner slot s190, 2026-10-28, 3 of 3), so its ledger row is removed. The record stays open: DBOS drops the usage when its cancel lands and Temporal's abort drops it too, and neither lane runs the cell (no observe-usage there; parked under the Cloudflare-first program) | source | partly-inferred | open |
| CP-221 | S2 | A parent's wait on a persistent companion whose Durable Object class has autoResumeOnWake: false polls forever once a wake demotes the companion's stranded run: the demote leaves the session paused and the run interrupted with no terminal checkpoint, the parent's consult reads the companion as live (its announced outcome needs the session itself to read interrupted), so companion__waitForResult (untimed) and a blocking park never answer, and the model is never told the companion needs a resume. | runtime-cloudflare/src/do/types.ts:558 (origin/main 64110bd5ac) (autoResumeOnWake: DO-class scope; with false "the stranded run is demoted to paused … until an explicit /resume"); core/src/run/committed-run-outcome.ts:38-43 (origin/main 64110bd5ac) (an interrupted run counts as committed (announced) only when the session status is also interrupted; a demoted session reads paused); runtime-cloudflare/src/do/persistent-companion-tools.ts:745 (origin/main 64110bd5ac) (readChildWord: no committed outcome → kind 'live'); conformance: companion.no-auto-resume-wait-answers-host (owner batch 49 c99e4b1d74; parent-first observe f33c3b9ccd) fails .wait-answers and .answer-not-completed on cf-do (host), 3 of 3 at owner batch f33c3b9ccd, 2026-10-24 (owner slot s115): the 'nar-scribe' companion ran in AGENTS_NAR (ConformanceNoAutoResumeDO), the wake demoted it (.companion-demoted: paused), and turn 2's untimed waitForResult did not answer within 60 s; related: contract ruling (orchestrator, 2026-10-24): a wait / park never polls forever; within a bound it answers a typed, not-completed result (interrupted / paused: the companion needs a resume); whether the framework re-drives the companion is G2's open design (the cell asserts no re-drive); related: the docs say "stays dead until an explicit /resume" (docs/runtimes/cloudflare-do.md:2116) but not who sends it, and nothing for companions; related: owner disposition at filing (2026-10-24): Needs a fixer: the parent's consult reads a demoted companion as not-completed and the wait / park answers typed within a bound; G2 decides any re-drive. | source | verified | open |
| CP-222 | S2 | A stop that reaches a run after its last stop read, and so lands at its terminal commit over a planned completion, stops none of the run's children on CF Workflows, Temporal and DBOS: the run ends interrupted while a live companion keeps running (and billing). runtime-js stops its in-process children at the stop request, so it is not affected. Fixed on CF Workflows by e1a3d9ab51 (!482); still open on Temporal and DBOS. | runtime-cloudflare/src/workflow.ts:968 (origin/main 64110bd5ac) (finishRun's late-stop branch, k5.committed && k5.stopped, ends the run with the stop's terminal and runs no cascade); runtime-cloudflare/src/workflow.ts:888-899 (origin/main 64110bd5ac) (onAgentComplete fires in its own complete-hooks step, after the run's last stop read and before the terminal commit: the window); runtime-js/src/execution/in-process-runs.ts:226-231 (origin/main 64110bd5ac) (the contrast: stopInProcessRun cascades to in-process children at the stop request, cause 'stop'); conformance: companion.late-stop-at-terminal-stops-companions (owner batch 49 5610397ae8) fails .companion-stopped on cfw-workflows-d1, temporal-memory, temporal-redis, temporal-postgres and dbos-postgres (executor), 3 of 3 at owner batch fe7aa8c669, 2026-10-24 (owner slot s112): the parent's onAgentComplete was held at a tool gate, the product's stop() recorded, the hook released, the parent ended interrupted, and the companion's session still read active; it passes on js-memory, js-redis and js-postgres; related: W1's !482 Greptile P1 (2026-10-24); its RED D (1f805b1aa5, runtime-cloudflare cfw-cp201-failure-cascade.test.ts 'CP-201 review 4 …') is the unit witness of path (2), a body failure whose catch runs before the terminal commit; !482 @9534bd44fb is the CF Workflows fix evidence for both paths (its stop-children-after-stop step) once it merges; related: the CF Workflows row is fixed by W1's !482 (merged as e1a3d9ab51; its fix commit 9534bd44fb, refs/merge-requests/482/head): a stop that lands at the terminal commit stops the run's companions after that commit. Its unit witness is the RED D strengthening 4ff60e26f8 (merged with !502): with 9534bd44fb's workflow.ts hunk reversed, cfw-cp201-failure-cascade.test.ts 'CP-201 review 4 (!482, Greptile P1): a Stop recorded AFTER the failure cascade's read lands at the terminal commit, and the run's companions are stopped by it' fails at :343 expected 'running' to be 'interrupted' (the companion's run), its precondition at :341 passing (W1's gate g213b). Temporal and DBOS (parked) still need the same cascade after a late-landed stop; conformance: companion.late-stop-at-terminal-stops-companions passes on cfw-workflows-d1 (executor), 3 of 3 with failedIds [] (an unexpected pass of the ledgered failure) at owner batch 800edd140a over origin/main 1be1ca68e5, which includes !482, 2026-10-24 (owner slot s127); its cfw-workflows-d1 row is dropped from the ledger; related: fixed in code by e1a3d9ab51 (W1, !482; the CF Workflows row), verified by the owner at origin/main 56d564d073 (audit 2026-10-25): runtime-cloudflare/src/workflow.ts:1078-1095 (origin/main 56d564d073) (after a stop landed at the terminal commit, the stop-children-after-stop step cascades with cause 'stop', companions included); still open on Temporal and DBOS: runtime-temporal/src/activities.ts:7121-7159 (origin/main 56d564d073) (the K5 commits the stop's terminal through commitTerminalHonouringStop and runs no child cascade after it), runtime-dbos/src/steps/run-commit.ts:1503-1531 (origin/main 56d564d073) (the same, with no cascade) | source, peer-reported | verified | open |
| CP-223 | S1 | A run's input custom state on a CONTINUATION silently replaces the state its session committed: runtime-js (and the CF Durable Object, which runs it) and CF Workflows take the input state over the loaded state on every entry, and keys the input omits reset to the schema's defaults. With interrupt_previous, an app route that read state, then sent a keyed /start with initialState, wipes everything the interrupted run committed after the read. history is honoured only on an empty log; the input state has no such gate, though the docs call it 'for a fresh session'. Temporal's half is CP-228. | runtime-js/src/js-agent-executor.ts:7210 (origin/main 64110bd5ac) (resolvedState = initialState !== undefined ? initialState : baseState, on every entry; :7366 parses it through the schema, so omitted keys take their defaults); runtime-js/src/js-agent-executor.ts:7192-7198 (origin/main 64110bd5ac) (history is honoured only when the durable log is empty; the input state has no such gate); runtime-cloudflare/src/do/drive/request-schemas.ts:59 (origin/main 64110bd5ac) (the native /start body accepts input.state whether or not the session exists); runtime-cloudflare/src/do/drive/run-drive.ts:1384-1397 (origin/main 64110bd5ac) (interrupt_previous stops the live run and waits for it before executing, so the replaced baseState is what the interrupted run committed); runtime-cloudflare/src/steps.ts:1487-1496 (origin/main 64110bd5ac) (CF Workflows: entry.initialState ahead of the session customState); core/src/types/executor.ts:66-67 (origin/main 64110bd5ac) (AgentInputObject.state: "Optional initial custom state", nothing about continuations); docs/frontend/ai-sdk.md:285 and docs/reference/ai-sdk.md:185 (origin/main 64110bd5ac) call it "for a fresh session", enforced nowhere; conformance: state.continuation-input-state-keeps-committed (owner batch 49 13935ac62a) fails .committed-state-kept on js-memory, js-redis, js-postgres, cfw-workflows-d1, temporal-memory, temporal-redis and temporal-postgres (executor), 3 of 3 at owner batch 13935ac62a (main 64110bd5ac's product + the owner batch), 2026-10-24 (owner slot s116): turn 1's tool committed count 1, turn 2 sent state { note }, and count read 0 (the schema default). It passes on dbos-postgres only because DBOS reads no input state at all (CP-224); that pass is not a fix; conformance: state.interrupting-start-keeps-committed-host fails .committed-state-kept on cf-do (host), 3 of 3 in owner slot s116: on an interrupt_previous DO, turn 1 committed count 1 and was held, a native POST /start with input.state answered 2xx, stopped it and completed turn 2, and count read 0; its preconditions .turn1-held, .turn1-committed, .start-answered and .turn2-completes pass; related: GitLab #144 (the SDK-upgrade app, 2026-10-24): its image-search route sends initialState = { ...state, ...actor } after getSessionState (api/chat/[sessionId]/route.ts:98-99) and its plan route sends the PlanDoc as initialState.plan (writer/articles/[articleId]/plan/route.ts:159-167, 209), both on keyed /start with interrupt_previous; the app's own mitigation (Task 9b) avoids it, residual races inferred; related: contract ruling (orchestrator, 2026-10-24): the input state seeds a FRESH session only, as history; on a continuation it is not applied, with a WARN naming the ignored keys and explicit docs (core AgentInputObject.state JSDoc + the two ai-sdk pages); a merge mode is a separate feature; related: owner disposition at filing (2026-10-24): Needs a fixer (TBD, orchestrator): gate the input state on a fresh session on every runtime (runtime-js / CF DO, CF Workflows, Temporal), WARN on the ignored keys, document it.; runtime-js/src/js-agent-executor.ts:7207-7214 (fix: !508, W1, 1579bf1044: input state seeds a FRESH session only; on a continuation it is dropped with a js.state_dropped_on_continuation warn naming its keys; the CF DO runs this executor); runtime-cloudflare/src/steps.ts:1467-1472 (fix: !508, 1579bf1044: CF Workflows seeds only a genesis log from entry.initialState and warns cf.state_dropped_on_continuation otherwise); core SendOptions removed, handle.send(input) takes no state (09486b7952); conformance: fixed on runtime-js, CF Workflows and the CF DO. On !508's MR pipeline 2985775451 (before !505 ledgered these rows) state.continuation-input-state-keeps-committed passes on js-memory (job 17444504203), js-redis, js-postgres and cfw-workflows-d1 (executor), and state.interrupting-start-keeps-committed-host passes on cf-do (host); the same pipeline still fails .committed-state-kept on temporal-memory, temporal-redis and temporal-postgres (job 17444504201), Temporal's half, split to CP-228; related: split (owner, 2026-10-25; orchestrator route (a)): the three Temporal rows moved to CP-228, open, parked with Temporal (FU-TEMPORAL-INPUT-STATE-ON-CONTINUATION). DBOS reads no input state at all, so its pass is not a fix (CP-224, unchanged) | source, peer-reported | verified | fixed (5 cells) |
| CP-224 | S2 | DBOS ignores a run's input custom state entirely, even on a FRESH session: its workflows read only the input's new messages and history, so a caller's documented way to seed a session's state does nothing there (the run starts from the schema's defaults). | runtime-dbos/src/workflows/shared.ts:341 (origin/main 64110bd5ac) (the workflow destructures only { newMessages, history } from the parsed input); runtime-dbos/src/lifecycle/execute.ts:261-267 (origin/main 64110bd5ac) (the session is created with no initial state); conformance: state.fresh-session-input-state-seeds (owner batch 49 13935ac62a) fails .input-state-seeded on dbos-postgres (executor), 3 of 3 at owner batch 13935ac62a, 2026-10-24 (owner slot s116): a fresh session started with state { count: 5 } read count 0; it passes on every js, cfw and Temporal lane; related: found with CP-223 (GitLab #144's trace): DBOS's pass of state.continuation-input-state-keeps-committed is this record's defect, not a fix of CP-223; related: owner disposition at filing (2026-10-24): DBOS is parked (CF-first); needs a fixer when it resumes: DBOS seeds a fresh session from the input state, and, with CP-223, ignores it on a continuation. | source | verified | open |
| CP-225 | S2 | A failed session's chat snapshot (what a page reload reads) carries no error: FrontendSnapshot has no error field and buildSnapshot drops the session's persisted errorDetail, so a reloaded UI sees only status 'failed' with no code or message for the error that ended the turn. Live, a stock useChat gets the code only through a transient data-error-metadata part; an adapter THROW's error chunk carries no code at all. The snapshot half is fixed (FrontendSnapshot.errorDetail, W1's CP-225 fix); the adapter-throw error chunk's missing code is tracked as FU-CP225-THROWN-LLM-ERROR-CHUNK-NO-CODE. | ai-sdk/src/types.ts:829-916 (origin/main 64110bd5ac) (FrontendSnapshot: sessionId, state, messages, streamSequence, status, checkpointId, stepCount, resumeMessageId?, latestTruncateCheckpointId?; no error field); ai-sdk/src/handler/snapshot.ts:197-213 (origin/main 64110bd5ac) (buildSnapshot has view.session, errorDetail included, and copies none of it); ai-sdk/src/transformer/stream-transformer.ts:1287-1310 (origin/main 64110bd5ac) (live: only a transient data-error-metadata { code, recoverable } plus error { errorText }; category, retryable and cause never reach the browser); core/src/orchestration/step-iterator.ts:1133-1143 (origin/main 64110bd5ac) (an adapter that THROWS writes an error chunk with no code or errorDetail: the sibling, no code even live); conformance: chat.failed-snapshot-carries-error-detail (owner batch 49 01045f627a) fails .snapshot-carries-code on cf-do and js-chat-host (host), 3 of 3 at owner batch e1cfcdf4d7 (main 64110bd5ac's product + the owner batch), 2026-10-24 (owner slot s117): the turn failed, the persisted errorDetail carried provider_error, and the snapshot read failed with only sessionId / state / checkpointId / stepCount / latestTruncateCheckpointId besides its standard fields; related: GitLab #142 (the SDK-upgrade app, 2026-10-24); the fix must NAME the snapshot field and keep it additive and optional (a stock useChat reads the snapshot JSON as is), scrubbed as the 409 stream_failed body is (handler/handle-chat-stream.ts:3967-3990); also fix the dangling HelixStreamError.fromDataEvent doc reference (ai-sdk/src/types.ts:328) in the same MR; related: owner disposition at filing (2026-10-24): Needs a fixer: a failed snapshot carries the session's scrubbed errorDetail in a named, optional field (both hosts), and an adapter throw's error chunk carries its classified code.; ai-sdk/src/handler/snapshot.ts:216-217 (fix: W1, c6233db0f9; as on branch reliability-cf-w1-cp225-failed-snapshot-error c6233db0f9) (buildSnapshot adds errorDetail to a failed snapshot: the session's persisted errorDetail through core sanitizeErrorDetail, so code, category and retryable are kept and free text is scrubbed; FrontendSnapshot.errorDetail is optional and present only when status is failed; both hosts build their snapshot here); conformance: fixed. chat.failed-snapshot-carries-error-detail passes on cf-do and js-chat-host (host), 3 of 3, at W1's CP-225 head c6233db0f9 (owner slot s144, 2026-10-25): the reloaded failed session's snapshot carries the error's code. The title's second clause (an adapter throw's error chunk carries no code) is not witnessed by this cell and stays with W1 as FU-CP225-THROWN-LLM-ERROR-CHUNK-NO-CODE (orchestrator, 2026-10-25: no new id) | source, peer-reported | verified | fixed (2 cells) |
| CP-226 | S2 | An agent resolver cannot refuse a session with a typed error: the DO's materialize rewraps ANY resolver throw as framework_agent_resolution_failed (its message prefixed 'could not be materialized'), losing the thrower's code, and a resolver that returns nothing reads 404 framework_agent_not_found, the same answer as deploy skew. A plain Error is retryable and answers 503; only a non-retryable one answers 500. | runtime-cloudflare/src/do/drive/materialize.ts:50-51 (origin/main 64110bd5ac) (an AgentResolutionError passes; anything else becomes AgentResolutionError.failed, framework_agent_resolution_failed, the code lost); runtime-cloudflare/src/do/durable-object-agent-base.ts:2623-2624 (origin/main 64110bd5ac) (a resolver that returns undefined throws AgentResolutionError.notFound: 404 framework_agent_not_found); core/src/errors/http-error.ts:116-129 (origin/main 64110bd5ac) (statusFor: 404 for not-found, 503 for a retryable error, else 500; typedErrorBody drops cause); conformance: entry.resolver-refusal-keeps-code-host (owner batch 49 e1cfcdf4d7) fails .refusal-code-kept on cf-do (host), 3 of 3 at owner batch e1cfcdf4d7, 2026-10-24 (owner slot s117): a resolver throwing HelixError validation_error (retryable false) was answered http:500 framework_agent_resolution_failed ("Agent 'conf-smoke' could not be materialized: …"); .refusal-not-retryable and .no-model-call pass; related: GitLab #138 (the SDK-upgrade app, 2026-10-24); its evidence said a plain throw answers 500: a plain Error is unclassified, so retryable, and answers 503 (owner trace); related: design first (orchestrator, 2026-10-24): the fixer writes a short note on a typed refusal (e.g. an AgentResolutionError.refused code + status, its reason as cause) for the orchestrator to rule; related: owner disposition at filing (2026-10-24): Needs a design note (a typed resolver refusal), then a fixer: the refusal's own code and status reach the caller through /start and the chat route. | source, peer-reported | verified | open |
| CP-227 | S3 | afterLLMCall hooks (and the usage they feed) are told the CONFIGURED model, never the model the provider SERVED: llm-vercel never reads the response's model id and StepResult has no field for it, so an alias or a proxy rewrite is invisible to usage and ledger hooks. | core/src/hooks/types.ts:205-220 (origin/main 64110bd5ac) (AfterLLMCallPayload.model: "Model identifier that was used"); core/src/orchestration/step-iterator.ts:733 (origin/main 64110bd5ac) (the payload model comes from the configured model, extractModelIdHelper(effectiveLLMConfig.model, …); every other runtime does the same); llm-vercel/src/vercel-adapter.ts:218 and :419 (origin/main 64110bd5ac) (the adapter reads a model identity only from config.model); live: owner unit witness served-model.witness.test.ts (a local file, run in llm-vercel's __tests__ at owner batch e1cfcdf4d7, 2026-10-24) fails: a response-metadata modelId 'served-model-u22' (configured claude-sonnet-5-5) never reaches the adapter's step result; handed to the fixer to adopt; related: GitLab #143 (the SDK-upgrade app, 2026-10-24); design: an optional servedModel on StepResult, forwarded into AfterLLMCallPayload | source, live-repro, peer-reported | verified | open |
| CP-228 | S1 | On Temporal, a run's input custom state on a CONTINUATION silently replaces the state its session committed: the workflow takes input.initialState over the persisted state on every entry, and keys the input omits reset to the schema's defaults. CP-223's Temporal half, left open when !508 made the input state seed a fresh session only on runtime-js, the CF DO and CF Workflows. | runtime-temporal/src/activities.ts:9752-9755 (origin/main 15f24344b3) (resolvedInitialState = input.initialState when set, else the persisted customState: on every entry, a continuation included); conformance: state.continuation-input-state-keeps-committed fails .committed-state-kept on temporal-memory, temporal-redis and temporal-postgres (executor), 3 of 3 at owner batch 13935ac62a (main 64110bd5ac's product + the owner batch), 2026-10-24 (owner slot s116): turn 1's tool committed count 1, turn 2 sent state { note }, and count read 0 (the schema default); still failing after !508 on its MR pipeline 2985775451, job 17444504201, while the runtime-js, CF Workflows and CF DO cells pass (CP-223); related: CP-223 (the parent, fixed for runtime-js / CF DO / CF Workflows by !508); split by the owner on 2026-10-25 (orchestrator route (a)). The contract ruling is the same: the input state seeds a FRESH session only, as history; on a continuation it is not applied, with a WARN naming the ignored keys; related: FU-TEMPORAL-INPUT-STATE-ON-CONTINUATION (docs/dev/follow-ups.md): parked with Temporal (the program is Cloudflare-first); the fix gates the input state on a fresh log in the workflow entry, as runtime-js / CF Workflows do; runtime-temporal/src/activities.ts:10821-10824 (pre-fix, origin/main 638aaed36c) (the continuation's state: input.initialState when given, else the committed base, so a continuation entry adopted the caller's state over the committed one); runtime-temporal/src/activities.ts:11063-11064 (as on branch reliability-cf-wa-cp228 f3b7f73bd9) (fix: CP-228, WA: honorInputState only when the entry is fresh, rowsBeforeEntry from the committed run; otherwise the committed state is kept and the dropped keys are logged, temporal.state_dropped_on_continuation, keys only); conformance: fixed. state.continuation-input-state-keeps-committed passes on temporal-memory, temporal-redis and temporal-postgres at the fix f3b7f73bd9 and fails exactly .committed-state-kept with the base's activities.ts, in WA's slot drv-tc2 (part A), 3 of 3; the mutation m228 is exact; js, cfw and dbos pass at both heads, guards. DBOS's half of the shape is CP-224 (open); FU-TEMPORAL-INPUT-STATE-ON-CONTINUATION is closed by !794 | source, peer-reported | verified | fixed (3 cells) |
| CP-229 | S2 | On runtime-js, a conversation whose first turn was a KEYED execute(agent, message, { sessionId, requestKey }) cannot continue through that turn's handle: handle.send() re-uses the execute's D-2 requestKey for the new turn, so the store refuses the send's run-start commit as a key conflict (RequestKeyConflictError, state_request_key_conflict, non-retryable) and the second message is never run. The workaround is a fresh execute() on the session instead of handle.send(). | runtime-js/src/js-agent-executor.ts:7393-7399 (pre-fix, origin/main ca70115dbb) (createSendCallback re-executes with { ...originalOptions, sessionId }: the keyed execute's requestKey is carried into the send's execute); core/src/errors/agent-errors.ts:613-626 (origin/main ca70115dbb) (RequestKeyConflictError: a key reused for a DIFFERENT request, state_request_key_conflict, non-retryable 409); runtime-temporal/src/executor.ts:460 (origin/main ca70115dbb) (Temporal logs and ignores requestKey on every execute, so its send cannot conflict); conformance: session.send-after-keyed-execute-continues (executor tier; new executor-driver capabilities 'request-key', a pass-through of execute()'s requestKey, and 'send', the session's latest handle's own send(message)) fails exactly .second-turn-completes on js-memory, js-redis and js-postgres at !508's red 22fa724145 plus the cell (owner slot s135, 2026-10-24): the send's outcome is threw { code: 'state_request_key_conflict', category: 'state', retryable: false }, 'Request key already used for a different request in session …'; its preconditions .first-settles, .send-settles and .first-completes pass; conformance: the same cell passes (failedIds empty) on js-memory, js-redis, js-postgres and cfw-workflows-d1 at !508's fix 7010be85b9 plus the cell (owner slot s135); the only product difference between the two trees is 7010be85b9's send callback, which clears requestKey beside branch. Temporal and DBOS are the CI matrix's (no local slot, orchestrator ruling 2026-10-24); related: found by the orchestrator's audit of !508 (U23 / CP-223, W1), 2026-10-24; red 22fa724145 and fix 7010be85b9 are runtime-js unit tests. The source audit found no other runtime affected (Temporal ignores the key; the CF Workflows and DBOS sends pass only the session id); this cell is that audit's evidence on every executor lane. A distinct finding from CP-223 (it predates U23). Whichever of the owner's records MR and !508 lands second flips it; related: the CF Durable Object runs runtime-js, so !508's fix covers it, but no cell witnesses the DO: no DO route calls handle.send(), and the one DO path is app code in the public afterEntry hook, which receives the run's handle (runtime-cloudflare/src/do/drive/host.ts:214-220 (origin/main ca70115dbb)); a cf-do witness is sized in FU-CONF-CFDO-SEND-AFTER-KEYED-START (docs/dev/follow-ups.md); runtime-js/src/js-agent-executor.ts:7398-7410 (origin/main 31d4a7b23c) (fix: !508, W1, 7010be85b9, merged as 31d4a7b23c: createSendCallback re-executes with requestKey: undefined beside branch: undefined, so a send is its own request on the handle's session; the CF DO runs this executor); conformance: fixed. session.send-after-keyed-execute-continues passes on js-memory, js-redis and js-postgres (executor) at 7010be85b9 plus the cell (owner slot s135, above); flipped at the records MR's main merge, after !508 landed (the owner's records MR landed second). The merged head's run is this records MR's pipeline | source, peer-reported, live-repro | verified | fixed (3 cells) |
| CP-230 | S2 | On Temporal and DBOS, a REMOTE sub-agent child that pauses (an approval or client tool deep in its own tree) still fails the parent's call: the dispatch reads the child paused after its stream drops and ends the call as a typed, retryable RemoteChildPausedError, so the call is retried and then fails within its retry budget, where runtime-js, the CF DO and CF Workflows park the parent since !504. CP-157's Temporal and DBOS half. | core/src/orchestration/remote-sub-agent-dispatch.ts:883-891 and :960-972 (origin/main 47dd092fb9) (Temporal's dispatch, executeRemoteSubAgentDispatch: a post-drop paused status sets childPaused, and the call returns kind 'failed' with RemoteChildPausedError's message and detail; no park); runtime-dbos/src/steps/execute-remote-subagent.ts:671-675 and :734-739 (origin/main 47dd092fb9) (DBOS's remote step: a paused child throws RemoteChildPausedError for the step retry; no park); core/src/orchestration/remote-sub-agent-dispatch.ts:986 (origin/main 47dd092fb9) (the dispatch's only stop of the child is transport.interrupt when the PARENT is interrupted; a failed call, a paused child's included, sends the child none); runtime-dbos/src/steps/execute-remote-subagent.ts:802-811 (origin/main 47dd092fb9) (DBOS's step catch writes the ref failed and sends the child no stop); related: the leak (WE-2's CP-157 research, peer-reported; not re-traced by the owner on Temporal): once the ref is failed the stop cascades skip it, so the paused child is never interrupted and stays parked on its producer; runtime-js, the CF DO and CF Workflows interrupt a failed call's child first since !504 (3d3eca56c5), Temporal and DBOS do not; inferred: (unverified) the call fails once the activity retry policy (Temporal) or the step's retries (DBOS) are spent: W1's reading, not traced by the owner; related: CP-157 (the parent, fixed for runtime-js / CF DO / CF Workflows by !504); split by the owner on 2026-10-25. Not silent: neither runtime writes remotePendingCalls, so the submit router's remote branch is unreachable there, and the failure is typed (CP-45); related: FU-CP157-TEMPORAL-DBOS-REMOTE-PARK (docs/dev/follow-ups.md): parked with Temporal and DBOS (the program is Cloudflare-first); the fix is the CP-157 park classification in both dispatches, their remotePendingCalls write and the router's remote resolver | source, peer-reported | partly-inferred | open |
| CP-231 | S2 | A pending APPROVAL of a server tool answered with a plain tool RESULT (a submit of kind 'tool-result' whose payload is not { approved, reason }) is recorded as that tool's SUCCESSFUL output on runtime-js (and the CF Durable Object that runs it), Temporal and DBOS, though the tool never ran: a client-supplied payload becomes the server tool's result in history and in every later model call. No submit path compares the submit's kind (or an approval id) with the pending entry, and runtime-js's resume drain classifies an entry as an approval only when the stored answer has the approval shape, so the forged payload falls through to the client-result branch. | runtime-js/src/js-agent-executor.ts:6811-6820 (origin/main 56d564d073) (validateLocalOwnerSubmit: an approval-response is not checked, and a tool-result for any pending entry is checked only against the tool's outputSchema, never against the entry being an approval); runtime-js/src/js-agent-executor.ts:6948-6970 (origin/main 56d564d073) (the submit stores a tool-result's payload as the entry's submittedResult whatever the entry is; the entry keeps its approvalId); runtime-js/src/js-agent-executor.ts:3526-3533 and :3547-3553 (origin/main 56d564d073) (the resume drain's two approval branches: an approval-gated entry WITHOUT an approvalId is refused, and one WITH an approvalId is an approval only if isApprovalResponseResult(result); an entry with its approvalId and any other payload matches neither); runtime-js/src/js-agent-executor.ts:3781-3792 (origin/main 56d564d073) (so the drain builds the row with success: outcome.ok and the forged payload as the result: a successful tool row for a server tool that never ran); contrast: runtime-cloudflare/src/steps.ts:8898-8913 (origin/main 56d564d073) (CF Workflows' drain: an answered entry that is not an approval submission, of a tool that is approval-gated, is refused typed and paired as a failure, never run or recorded as success; its message says the entry has no approvalId, which is misleading for this shape); contrast: core/src/client-tool/new-turn-settlement.ts:120-166 (origin/main 56d564d073) (a new turn's settlement of the same entry refuses it typed: framework_internal_error, success false); inferred: (unverified) CF Workflows still records the payload as a success when the tool is gone from the agent (the refusal needs the tool lookup); the Temporal and DBOS drains are witnessed by the cell, their sites not traced; conformance: approval.forged-tool-result-not-success (owner batch 50 f566845200, executor tier) fails .no-forged-success-row and .model-never-told-forged-success on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres and dbos-postgres, 3 of 3 at owner batch f566845200 (main 56d564d073 + the owner batch), 2026-10-25 (owner slot s137): the forged submit is answered accepted (caller_resumes; new_run on DBOS), and the continuation's drain persists a SUCCESS row for the gated call carrying the forged payload, which the next model call is given; cfw-workflows-d1 passes 3 of 3 (accepted new_run, the drain refuses the entry, no forged row); related: reported by WE-2 (peer-reported, 2026-10-25, code-traced at main 56d564d073), relayed by the orchestrator; the runtime-js fall-through verified by the owner at the cited lines. The CF Durable Object runs runtime-js's executor (its /submit-tool-result reaches the same submitToolResult and drain), and the CP-157 forward in core/src/client-tool/route-submit.ts carries the kind unchecked to the producer; related: CP-154 (an approved call whose tool is gone recorded as success): the same drain classification; fixer W1 with CP-154, after CP-186, on every runtime, Temporal and DBOS included, not parked (orchestrator ruling, 2026-10-25: a security bug); severity S2 (owner, 2026-10-25): a client that can submit can make the model believe a server tool ran and returned its payload; nothing runs, so no server side effect, but every later decision of the model rests on the forged result; core/src/client-tool/route-submit.ts:337-380 and :550-558 (fix: !518, W1, 790fa9c069; as on branch reliability-cf-w1-cp231-submit-kind-binding ff2d2aacb3) (assertSubmitAnswersPendingKind, called by routeSubmitToolResult on every runtime's submit before anything is stored or forwarded: a tool result for a pending approval, an approval-response for a client call, or an approval-response whose approvalId is not the entry's is a typed, non-retryable validation_error, HTTP 400); runtime-js/src/js-agent-executor.ts:3532 (fix: !518, W1, 12e35f063a; as on branch reliability-cf-w1-cp231-submit-kind-binding ff2d2aacb3) (and every other drain, as on the same branch: CF Workflows runtime-cloudflare/src/steps.ts:8908, Temporal runtime-temporal/src/activities.ts:4013 and :5358, DBOS runtime-dbos/src/workflows/shared.ts:1154 and :1358-1361, and core's new-turn settlement: an approval entry holding a tool result is refused typed and paired as a failure, never the tool's success; Temporal and DBOS now register the entry's approvalId; 8c7ba67780); conformance: fixed. approval.forged-tool-result-not-success passes on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres and dbos-postgres (executor), 3 of 3, at !518's head ff2d2aacb3 (owner slot s140, 2026-10-25): the forged submit is refused typed at submitToolResult, nothing is stored and no success row or model call carries the payload; cfw-workflows-d1 still passes 3 of 3. The merged head's run is !518's pipeline | source, peer-reported | verified | fixed (7 cells) |
| CP-232 | S3 | useHelixChat: after a dropped send whose re-attach could not read the snapshot, the lost-connection error (resyncError, with its Retry) stays up after the user's retry() succeeds: the hook returns reattachError ?? resyncError, and reattachError is cleared only by a new send record or by the re-attach controller's own run succeeding, never by a retry() that re-anchors or re-sends. The finished turn shows in full under a stale error until the next send (GitLab #170). | ai-sdk/src/react/use-helix-chat.ts:565 (origin/main 8e352e91ef) (reattachError, set only through the re-attach controller onReport); ai-sdk/src/react/use-helix-chat.ts:716 (origin/main 8e352e91ef) (retry() re-anchors through reanchorRef or re-sends through regenerateSameKey; neither path clears reattachError); ai-sdk/src/react/use-helix-chat.ts:788 and :804 (origin/main 8e352e91ef) (resyncError: reattachError ?? resyncError); related: GitLab #170 (the SDK-upgrade app, 2026-10-25): observed live on two surfaces, the snapshot held the complete reply shown once and the error line stayed; fixer W1 in !516 (with #173), whose hook test drives a real useHelixChat over ai's AbstractChat with a send that drops while the snapshot route answers 503 (use-helix-chat-retry-state.test.tsx, as on branch reliability-cf-w1-helix-chat-retry-state); severity S3 (owner, 2026-10-25): the turn completed and renders in full; the stale error and Retry mislead the user until the next send, and nothing is lost; conformance: client.retry-clears-lost-connection (owner batch 50, client tier, the product's useHelixChat; new harness control 'fault-snapshot-route') fails exactly .error-cleared on cf-do and js-chat-host, 3 of 3 at owner branch 607950ff59 (main 56d564d073 plus CP-231 and this cell; owner slot s142, 2026-10-25): resyncError still holds the SnapshotUnavailableError ('the snapshot route is down') 5 s after the retried turn rendered in full and the session completed; its preconditions .dropped and .lost-connection-shown (a faulted snapshot read and resyncError set) pass. With !516's head 237a42c589 merged in, the same cells pass 3 of 3 (s142); ai-sdk/src/react/use-helix-chat.ts:738-745 (origin/main 60ee77949d) (fix evidence, !516, W1; #170: retry()'s re-anchor that resolved clears the re-attach controller's last report, reanchorClearingReport, used on both re-anchor paths; a re-send's new send record clears it in the controller); conformance: fixed. client.retry-clears-lost-connection passes every id on cf-do and js-chat-host (client tier), 3 of 3, at owner branch 0b5760d315 (the cell's branch plus !516; owner slot s142, 2026-10-25): resyncError is null after the successful retry(). The records MR's own lane run at main is its CI-once lane evidence | source, peer-reported, consumer-documented | verified | fixed (2 cells) |
| CP-233 | S2 | Temporal: a parent's companion tools do not answer a non-blocking companion paused on its own client calls typed. companion__waitForResult polls to its deadline and answers timeout with the child read as running (never suspended with its calls). On DBOS the wait and sendMessage halves are fixed by !589 (the paused companion is answered typed). Temporal is not on the CP-40 (C2) contract (runtime-js, the CF DO and CF Workflows are). | core/src/orchestration/companion-tool-dispatch.ts:1050-1059 (as on branch reliability-cf-wa-cp40-c2 3b864439b0) (the dispatcher records paused_awaiting_client only from a suspended child word) and :2626-2633 (as on branch reliability-cf-wa-cp40-c2 3b864439b0) (the wait's one deadline decision: a ref still running answers timeout / running); Temporal's and DBOS's companion paths never hand the dispatcher a suspended word, so the ref stays running; runtime-dbos/src/steps/execute-companion-tool.ts:1779 (as on branch reliability-cf-wa-cp40-c2 3b864439b0) (DBOS's own wait answers waitForResultTimeout with the ref's status at the deadline); conformance: companion.paused-child-answers-typed (owner, CP-40 C2 sibling, executor tier) at owner-c2-cells 65f16e8d01 = WA's C2 head 3b864439b0 plus the owner's cells (owner slot s147, 2026-10-25), 3 of 3: on temporal-memory, temporal-redis and temporal-postgres .wait-answers-suspended fails (the wait answers {"name":"helper","status":"timeout","currentStatus":"running"}); on dbos-postgres .wait-answers-suspended and .send-answers-awaiting-client fail (neither row is in the parent's history after turn 2). runtime-js (js-memory, js-redis, js-postgres) and cfw-workflows-d1 pass every id; conformance: companion.stale-paused-child-not-suspended (owner, the stale-pause ruling) at owner-c2-cells 1312c7d2e0 (owner slot s148b, 2026-10-25), 3 of 3: on dbos-postgres .send-answers-busy fails (the send answers {"delivered":true}: the root's submit continued the child, which holds in its gated next step); on temporal-memory, temporal-redis and temporal-postgres .answers-stored fails (the root's ownership never names the calls within the cell's 10 s read, and the submit to the child's own session that follows is not accepted), so every later id fails as a precondition. In s148 (9ee6b2beea) the same submit made at once after the pause was accepted and every id passed on Temporal. runtime-js and cfw-workflows-d1 pass every id; conformance: companion.paused-child-answers-typed passes every id on dbos-postgres 3 of 3 at !589 05a804cda2 (WE-2's fix: a paused companion is answered suspended / awaiting-client typed; WE-2 slot w-589fix-1030-223459, 2026-10-30, with the owner cells 4f978f4ce0 / 6ac17a2b50 alongside), so its dbos ledger row is removed; the Temporal rows stand; inferred: on Temporal the root ownership write (runtime-temporal/src/activities.ts:6505-6511, writeRootOwnership :6634-6668 at origin/main 8e352e91ef) does not land for a companion paused after its parent turn ended, and the child stops accepting answers within 10 s; candidate cause: the parent workflow closing ends the child workflow (parent-close policy), unverified; related: FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY (Temporal / DBOS are not on the companion spawn contract); CP-40 (the DO side, closed by C2); CP-127 (the runtime-js side, fixed by C2); the late-pause routing gap (owner candidate, companion.late-pause-root-routes); runtime-dbos/src/steps/execute-companion-tool.ts:1349-1364 (origin/main e34b96e2c2) (the sendMessage C-4 guard refuses only when the child's SESSION status is 'paused' with pending calls; a DBOS persistent child keeps its session 'active' while paused, its pause being on its run and stream (WE-2, 2026-10-30), so the guard never refuses a send to a paused companion; the guard's own comment says such a send would orphan the pending calls: companion.send-to-paused-child-keeps-calls measures that); conformance: a correction (owner, 2026-10-30, the orchestrator's doorbell ruling): DBOS's { delivered: true } to the continued child (s148b above) is the doorbell contract (§4.5: a live target takes the message as its next turn; core handleSendMessage answers the same for a live run), not a defect: the root's submit continues the child at once, so at the send it runs in its gated tool with an empty pending map (WE-2's CP-247 slot, 2026-10-30). The stale cell no longer sends (it asserts the wait and never-terminate only); the busy answer moved to companion.draining-child-send-busy, which holds a started continuation's run-start commit before it commits, so the child is draining (answered, not consumed) when the parent sends; no dbos-postgres cell (no entry-commit hold, and its continuation drains before its run-start commit, FU-DBOS-CONTINUATION-ENTRY-ORDER); related: CP-130 confounds this record's dbos .wait-answers-suspended (the root's handle settles turn 2 early as suspended_client_tool on the companion's calls and the cell's end cancels it): that ledger entry is re-attributed to CP-130 if it stays red with that shape once this record's DBOS wait fix lands (WE-2's diagnostic, 2026-10-30); conformance: owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 3 of 3 identical: companion.draining-child-send-busy fails on temporal-memory, temporal-redis and temporal-postgres from .child-draining on (the child's answers are not stored as the draining state needs: this record's Temporal submit shape), passing on js-memory, js-redis, js-postgres and cfw-workflows-d1 (dbos-postgres is a declared gap); companion.stale-paused-child-not-suspended matches its Temporal row and passes on js, cfw-workflows-d1 and dbos-postgres (the withdrawn dbos send row stays withdrawn); conformance: the Temporal rows of companion.stale-paused-child-not-suspended and companion.draining-child-send-busy are fixed by !615 (WE-2, CP-255's fix): both pass every id on temporal-memory, temporal-redis and temporal-postgres at owner slot s228 (2026-11-01, tree 9672fe7064 = batch 5 !624's head), 3 of 3 identical; batch 5's own test:conformance 3/7 job 17638684194 showed all 6 as unexpected passes first. !615 is the batch's only change to Temporal's behaviour, so the attribution is by elimination. companion.paused-child-answers-typed stays red there 3 of 3 (.wait-answers-suspended), so this record stays open for the wait; inferred: the child stopped accepting answers because it had already failed at its suspension marker (CP-255's mechanism), not the parent-close policy the inferred line above named as a candidate; not observed directly in these two cells | source, live-repro | partly-inferred | open |
| CP-234 | S2 | runtime-js and Temporal: a NON-blocking companion that pauses on a client-executed call after its parent's turn ended leaves that call unroutable through the root: the root's clientToolCallOwnership never names it, so a client's submit against the root (where clients submit) answers unknown_tool_call, and nothing writes the routing later. The call is answerable only through the child's own session id, which a client does not normally hold, and its frames reach no live stream (the parent's stream ended; a guest chunk to it is dropped, CP-129). CF Workflows and DBOS route it; on the CF DO such a companion is refused at spawn (CP-40, C2). | core/src/client-tool/route-submit.ts:490-505 (origin/main 8e352e91ef) (the router reads only the root's clientToolCallOwnership, then the root's remotePendingCalls, filled only for remote children a run parks on, else answers unknown_tool_call, logged client_tool.submit_unknown; it never scans the root's sub-session refs); runtime-js/src/run-loop.ts:2620 (origin/main 8e352e91ef) (an in-process companion runs with rootRunId: input.rootRunId ?? input.runId, the parent's run at spawn time) and :3251-3262 (origin/main 8e352e91ef) (the root ownership write is fenced on that run, else the root's live run, else skipped with a warning); runtime-js/src/run-loop.ts:3295-3299 (origin/main 8e352e91ef) (the fenced saveState onto the root, which throws RunSupersededError once the parent run ended) and :3308-3313 (origin/main 8e352e91ef) (the throw is swallowed as persistSuspension: failed to write ownership to root; the child keeps its own pending entry); runtime-temporal/src/activities.ts:6508-6514 (origin/main 8e352e91ef) (a sub-agent's commitSuspendedStep writes the root ownership only when the session's rootSessionId names another session) and :6637-6671 (origin/main 8e352e91ef) (writeRootOwnership, an unfenced saveState onto the root); the observed root never gains the entry within 10 s (below): why the write does not land for a companion paused after its parent's turn is unverified; runtime-dbos/src/steps/client-tool.ts:218-220 (origin/main 8e352e91ef) (the sub-agent path writes the root ownership first) and :438 (origin/main 8e352e91ef) (an unfenced saveState): DBOS routes the call; runtime-cloudflare/src/client-tool-workflow-helper.ts:107-134 (origin/main 8e352e91ef) (CF Workflows writeOwnership, an unfenced saveState onto the root) and runtime-cloudflare/src/approval-gate-workflow-helper.ts:94-102 (origin/main 8e352e91ef) (the same for an approval): CF Workflows routes the call; runtime-js/src/local-tree-root.ts:12-17 (origin/main 8e352e91ef) (localTreeRootOf answers the child's own id when the root is not in its store, so a CF DO child, which runs runtime-js in its own DO, never writes the parent's map: CP-40's (a) half, closed on the DO by C2 refusing such a companion at spawn); conformance: companion.late-pause-root-routes (owner, executor tier): turn 1 spawns a non-blocking companion whose first step holds in a gated tool until the parent's turn completed, then pauses on one client call; a bounded settle waits up to 10 s for the root's ownership to name the call, and the root's submit is then checked. At main f9e74ecd9a plus the cell (owner slot s150r, 2026-10-25), 3 of 3: on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres .root-owns-call (the root's ownership ids stay [] at the bound), .root-submit-accepted (the root's submit answers unknown_tool_call) and .answer-reaches-child (precondition) fail; cfw-workflows-d1 and dbos-postgres pass every id. Also observed at !520's head 3dc4ca03d9 (owner slot s149, x1 per lane), the same split; related: CP-128 (the same root ownership write, fenced since, made a parent step fail when the companion suspended while that step was open: a different window and outcome); FU-JS-COMPANION-SUSPENDED-CONTINUATION (who drives a runtime-js companion after its answers arrive, MR D); the user picked spec Q-MR3-1 (a), route such a call rather than refuse the companion (orchestrator, 2026-10-25); the planned fix is the tree client-calls design (tree-client-calls-design.md, draft v2, not yet approved), which finds that no runtime supports a background companion's client call end to end, so the fix waits on that spec's approval; severity S2 (orchestrator, 2026-10-25): the companion's call cannot be answered the way a client answers calls, and nothing reports it; the parent's own turn is unaffected; conformance: companion.answer-after-parent-turn-runs (owner, CP-250's cell, which holds the root LIVE in a gated tool until its ownership names the companion's call) at owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 3 of 3 identical: on temporal-memory, temporal-redis and temporal-postgres .root-owns-call never settles ("the root's ownership, the parent held: {}") and every later id fails after it. So on Temporal the root ownership write does not land at all within the bound, not only for a companion paused after its root turn ended (this record's inference above widens to both); inferred: on runtime-js any failure of the root ownership write is only logged (runtime-js/src/run-loop.ts:3600 (origin/main e34b96e2c2), 'persistSuspension: failed to write ownership to root'), so a transient store error on a LIVE root strands the call the same way as the fenced refusal this record names; related: CP-253 (the window BEFORE a root ownership write that does land: a submit on the frame is refused); CP-254 (the companion's own-session route, refused on CF Workflows and DBOS); related: CP-250 (the other half: a companion that paused while its root was live IS routable, and the answer through the root is accepted, but its continuation never runs the call once the root's turn ended; cell companion.answer-after-parent-turn-runs, which holds the root live until its ownership names the call); conformance: no longer red on runtime-js (batch 3 !613: !594's idle-root ownership write, with the js lanes' agentRegistry, orchestrator ruling A1): companion.late-pause-root-routes passes every id on js-memory, js-redis and js-postgres (owner slot s218, 2026-10-31, tree 04f2d6252a = batch 3 head 9bfa305c8c + the owner's fixes, 3 of 3); the js backends leave its row, and Temporal stays; inferred: Temporal's cause, read not run: a companion's child workflow starts with no rootSessionId (runtime-temporal/src/workflow-dispatch.ts:848-866 (origin/main cb8175c1d5), and :964 (origin/main cb8175c1d5) on a resume), its session is created without one (runtime-temporal/src/activities.ts:10134 (origin/main cb8175c1d5), set only from the input at :10341 (origin/main cb8175c1d5)), so its suspension reads as self-rooted (runtime-temporal/src/activities.ts:6597 (origin/main cb8175c1d5)). Sub-agents do get one (runtime-temporal/src/workflow-dispatch.ts:428 (origin/main cb8175c1d5)); related: WA's relay, the child's root write has two more arms that only log, read not reproduced: runtime-js/src/run-loop.ts:3827-3833 (origin/main dcda6c8798) (no live root run to fence on: warn, return) and :3851-3860 (origin/main dcda6c8798) (another session owns the id: warn, skip). Neither is this record's mechanism, a companion never takes the first. CP-253's fix decides them (orchestrator, 2026-11-01): a foreign owner and a thrown write fail the step; with no rootRunId and no live root a non-companion writes nothing and logs at info, its call answered on the child's own id and unknown_tool_call at the root, a named exception; conformance: fixed on Temporal by !637 (WE-2's CP-234-T, batch 9 !642: a Temporal companion's child workflow carries its tree root, so its suspension writes the call's ownership on the root). Since then the root's ownership names the call and the root route reaches the companion, where Temporal's answer is the CP-250 ruling's typed refusal (owner_not_continuable, until CP-246), so the positive cell's last two ids cannot pass there by design. The Temporal half is judged by the sibling companion.late-pause-root-refused-typed (.root-owns-call, .root-submit-refused-typed), which passes every id on temporal-memory, temporal-redis and temporal-postgres at owner slot s242b (2026-11-02, batch 9's head f961445d11 plus the owner's member), 3 of 3; companion.late-pause-root-routes is a declared gap there ('companion-self-wake') | source, live-repro | partly-inferred | fixed (6 cells) |
| CP-235 | S3 | CF Workflows: an LLM error that ends the run without a step.do retry writes TWO error chunks to the run's stream: the step's held report (core holdLLMErrorReport, settled as terminal) and the terminal seam's failRunStream chunk. Nothing deduplicates them, so E5 (one LLM failure is one error chunk) breaks for every classified non-retryable model error (e.g. provider_auth_error); stock useChat stops at the first, and a stream reader sees the failure twice. | runtime-cloudflare/src/steps.ts:2606-2612 (origin/main f9e74ecd9a) (executeAgentStep holds the adapter onError report: holdLLMErrorReport) and :2816-2823 (origin/main f9e74ecd9a) (errorReport.settle(rawStepResult, terminal) with terminal true for an error step step.do will not retry, which writes the held error chunk); runtime-cloudflare/src/workflow.ts:1170-1180 (origin/main f9e74ecd9a) (the failed terminal calls steps.failRunStream for a run that owns its stream) and runtime-cloudflare/src/steps.ts:7675-7713 (origin/main f9e74ecd9a) (failRunStream writes its own type: error chunk, :7698-7708, then failStream): a second error chunk for the same failure; conformance: stream.one-error-chunk-classified (owner, executor tier): a classified non-retryable provider_auth_error the adapter REPORTS through onError and returns as the error step (the mock adapter's error response; the scripted fail reports nothing, FU-CONF-SCRIPTED-FAIL-REPORTS-ONERROR) ends the run failed. At main f9e74ecd9a plus the cell (owner slot s150r, 2026-10-25), 3 of 3, .one-error-chunk fails on cfw-workflows-d1 (two error chunks: the held report stamped with the LLM attempt, then the terminal one); js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres and dbos-postgres pass; live: found by W1 from source while gating !520 (2026-10-25); its run-level test error-chunk-run-level.workflow proves the classified case red at main; related: !520 (MA-26 / DI-35) sends an unclassified LLM error down the same non-retried path; CP-215 (a CF Workflows child writes its model error into the parent's stream mid-turn) shares the held-report seam; runtime-cloudflare/src/steps.ts:2627-2636 (fix: !520, W1; as on branch reliability-cf-w1-ma26-llm-step-retry 94a90437a2) (E5 on CF Workflows: the step never writes the held report; the run's terminal seam, failRunStream, writes the one chunk with the classified detail) and :2829 (fix: !520, W1; as on branch reliability-cf-w1-ma26-llm-step-retry 94a90437a2) (errorReport.settle(rawStepResult, false)); conformance: fixed. stream.one-error-chunk-classified passes on cfw-workflows-d1 (executor), 3 of 3, at !520's head 94a90437a2 plus the cell (owner slot s152, 2026-10-25): one error chunk; at main f9e74ecd9a it failed 3 of 3 (above). This batch trains after !520 | source, peer-reported, live-repro | verified | fixed (1 cells) |
| CP-236 | S2 | runtime-js (and the CF Durable Object that runs it): a state write the agent's onAgentStart hook makes (ctx.updateState) lives only in the run's memory until step 1's commit; it is not part of the run-start commit. A run that dies before step 1 commits is recovered on the DO's wake as a resume, and a resume never re-fires onAgentStart, so the write is lost. A retry() after a first-step failure restores the run-start commit and, being a resume of that checkpoint, does not re-fire the hook either: the write is lost again. Nothing reports either loss. | runtime-js/src/run-loop.ts:1013-1058 (pre-fix, origin/main 378f141792) (onAgentStart fires only when !input.isResume, inside the run loop, after the entry's run-start commit); runtime-js/src/run-loop.ts:4885-4902 (pre-fix, origin/main 378f141792) (buildHookContextOptions: the hook context's updateState replaces the in-memory state.customState, produce(), with no store write); runtime-js/src/run-loop.ts:1423 and :1689 (pre-fix, origin/main 378f141792) (the only store writes before step 1 are message appends, child results and notices, never customState) with :4847 (pre-fix, origin/main 378f141792) (the first customState write: saveStateAndPromoteStaging, the step commit); runtime-cloudflare/src/do/drive/run-drive.ts:816 (pre-fix, origin/main 378f141792) (the DO's wake recovery calls executor.recover) with runtime-js/src/js-agent-executor.ts:5088-5101 (pre-fix, origin/main 378f141792) (recover is resumeSession) and :6059 (pre-fix, origin/main 378f141792) (resumeSession runs the loop with isResume: true); runtime-js/src/js-agent-executor.ts:6438 and :6551 (pre-fix, origin/main 378f141792) (retry: isResume is checkpointForHooks !== null; after a first-step failure planRetry's target is the run-start commit, a non-null checkpoint, so the hook is not re-fired and the restored customState lacks its write); live: GitLab #177 (the user's report); the owner's source survey at origin/main f9e74ecd9a (owner runs gl177/survey.md), re-verified line by line at origin/main 378f141792 (2026-10-26); related: MA-25 (on DBOS no hook can write state: its hook context's updateState is undefined), so DBOS shows no loss here because it allows no write; the orchestrator's ruling 2026-10-25 holds two sibling questions for the user's contract ruling: a retry() with no checkpoint re-firing onAgentStart, and Temporal re-firing it on every retry (once per turn vs once per run); severity S2 (owner, 2026-10-26): state an app's hook seeded is silently missing for the rest of the session after an ordinary crash recovery or retry; related: fixed on runtime-js, the CF DO and CF Workflows by WE-2's !554 (#177 MR 1: one entry state per turn, the start commit, one start signal per run), merged at main d262d86c52; the owner's green runs were at its 812fb53f18 (main 3d009e1afb by patch-id). Temporal and DBOS are #177 MR 2's (the ledger below); conformance: the #177 cells (owner, 2026-10-27), red at main and green with !554's 812fb53f18 (main 3d009e1afb by patch-id), 3 of 3 each (owner slots s172 / s174; every set identical across its three runs). hooks.start-writes-durable-host (cf-do: the first model call held, the backend evicted, the session's DO woken by its alarm): at main .start-write-kept fails (customState.startWrites 0: the hook's write lost with the run); with !554 every id passes (startWrites 1, onAgentStart once). hooks.retry-start-work (executor, every lane): leg A (the first model call fails, then retry({ message })) fails at main .a-retry-signal, .a-retry-message-hook and .a-start-write-kept (no onAgentRetry, no onMessage for the retry's message, the write lost); leg B (retry({ checkpointId: the turn's input checkpoint, message }), the message required by design: the checkpoint's prefix already ends at the trigger) fails .b-signal-then-start and .b-start-write-once; with !554 both legs pass on js-memory, js-redis, js-postgres and cfw-workflows-d1 (leg A: onAgentRetry only, startWrites restored to 1; leg B: onAgentRetry, then onAgentStart with startSignal false, startWrites 1). hooks.start-signal-fresh-turn (executor, every lane): a fresh turn's run-signal hooks are [onAgentStart startSignal true], red at main (no startSignal), green with !554 on js-*, cfw-workflows-d1 and temporal-*; conformance: the entry patch, the same slots: entry.state-patch-not-overwritten (executor, every lane) and -host (cf-do, the DO's native /start input.statePatch): at main the patch is ignored (.tool-saw-patch and .patch-kept fail); with !554 every id passes on js-*, cfw-workflows-d1 and cf-do. Temporal and DBOS refuse a statePatch typed (framework_not_supported) until #177 MR 2, and their retry / start-hook ids stay red there (Temporal re-fires onAgentStart on every retry and calls no onAgentRetry; DBOS calls no onAgentRetry and passes no startSignal, and its hooks cannot write state, MA-25): ledgered below, fixer #177 MR 2 | source, peer-reported | verified | open |
| CP-237 | S2 | runtime-js: a new turn (resume with_message, no abandonPending) over a parent paused on suspended_awaiting_children is accepted and runs, and nothing in that entry settles the awaited child: the child's run stays suspended on its own client call, its session paused, and the parent's sub-session ref to it stays running under a parent that then completes. The child is orphaned. Before child identity MR 1 this was masked only when the new turn re-issued the same call id (the derived child id re-entered the old child). | core/src/client-tool/with-message-entry.ts:63-110 (origin/main 378f141792) (planWithMessageEntry: only abandonPending over suspended_step_partial / suspended_awaiting_children is refused, :91-96; a plain with_message falls through to input_in_run_start, :110); runtime-js/src/js-agent-executor.ts:5228-5500 (origin/main 378f141792) (the with_message entry plans with planWithMessageEntry, :5319, and settles pending client calls, but no awaited child: the executor's only stopChildren call is the stop path, :1669); live: W1 (peer-reported, 2026-10-26): a probe in runtime-js stop-request-scenarios (childScenario: the child suspends on its client tool, the tool is answered at the root, an interrupt stop is recorded on the child's run, then resume(parent, { mode: 'with_message' })): the parent completed, the child's run stayed suspended_client_tool with its stop, its session paused, and the refs read [[child, sub1, running], [fresh child, sub1, completed]]; related: FU-WITH-MESSAGE-MIXED-CHILDREN-ORDER (a send over a suspension that also awaits children still puts the message first: row order only, not this); CP-159 (a stop of a parent parked on its suspended children never reaches them, on CF Workflows, Temporal and DBOS); related: the ruling covers LOCAL (in-process) children parks only (orchestrator, 2026-10-28). A with_message over a REMOTE park (the CF DO, whose children are always remote sibling DOs) is refused busy by S-C ruling (B), 2026-10-17 (core/src/orchestration/remote-children-before-input.ts, running: 'refuse'): state_already_running, nothing written, park, ref and child unchanged. That is the deliberate contract, not this finding, so the same user action is delivered on runtime-js and refused on the DO; whether a remote park should deliver too is a design item (FU-REMOTE-PARK-WITH-MESSAGE-CONTRACT, docs/dev/future-work.md); conformance: with WA's CP-237 fix (!574, landed in batch 3's squash f85872de1e; its pre-restack red 1ae4057f73 and fix 6f7e70dd66 are archived at wa-archive-cp237-dc5d032831; WA slot s2, 2026-10-28, ran at the pre-restack fix e4f75f6167 and red b4806f783a, archived at wa-archive-cp237-e4f75f6167, whose tree the pushed head reproduces): the cell passes every id on js-memory, js-redis and js-postgres in 3 of 3 runs (UNEXPECTED PASS on exactly .child-completed, .child-result-delivered and .ref-settled), and at the red commit fails exactly the ledgered ids (expected-fail) — proves the fix; related: the root cause the cell found (WA diag2 / diag3): the conformance driver wires recorder hooks into every agent, and with onAgentStart / onMessage hooks runtime-js writes a with_message turn's start-work commit (run-loop.ts run-start path) before its awaited-children drive; that commit's read-modify (persistViaSaveAndPromote) kept pending, ownership and tracing but not suspendedAwaitingChildren, so the drive found no waits and the parent completed over its unpaired child call. The fix keeps the stored park on every commit except the step and suspension commits, so a continue that owes start work (a turn with no recorded turnStart, e.g. hooks added mid-turn) is covered too; the other session-only fields such commits drop are FU-RJS-COMMIT-DROPS-SESSION-ONLY-FIELDS; severity S2 (owner, 2026-10-26): a child left suspended under a completed parent holds an answerable call no turn will consume, and its ref reads live forever; conformance: subagent.with-message-over-children-park-delivers-child (owner, executor tier, the lanes with 'subagent-client-tool'): the parent parks on its sub-agent child (suspended_awaiting_children; the child waits on its own client call), a resume with_message TURN-2 is sent, the child's call is answered through the root, and the parent must end completed with the child's result. Owner slot s188 (2026-10-28, 3 of 3 identical; .parent-parked, .message-settles, .message-reaches-model, .turn-settles and .turn-completes green): it fails exactly .child-result-delivered, .ref-settled and .child-completed on js-memory, js-redis and js-postgres (the parent completes without its child's result). The variant where the new turn re-issues the call id is not written; conformance: the same cell under a slot's scratch 'subagent-client-tool' claim (owner slots s185 and s188, 3 of 3 each) passes every id on cfw-workflows-d1 once .message-reaches-model is read after the turn settles (CF Workflows re-suspends on the child before its model call, then its submit continuation answers the message with the child delivered); the lane stays a declared gap until the claim lands. On Temporal (s185, 3 of 3) the with_message resume makes no model call and the parent stays active, a workflow-close dependence; DBOS has no children-park (gaps); related: owed cells (owner): the JS executor variant where the new turn re-issues the call id, and a cf-do host cell (its native /resume with_message) to decide the CF DO, whose settle path cascades stops (FU-CONF-CP237-OWED-CELLS) | source, peer-reported | verified | fixed (3 cells) |
| CP-238 | S2 | On Temporal and DBOS, a parent stop during an in-flight REMOTE sub-agent call does not reach the remote child. Temporal: an interrupt only signals the workflow, which acts at its next boundary, so the remote activity is never cancelled and the child is never told; an abort cancels the activity, whose cleanup sends one best-effort interrupt (even for an abort) and logs a failure at info. DBOS: a stop is the run's stop record plus cancelWorkflow, and nothing sends the remote child any stop. The child runs on, and bills, to its own end. | runtime-temporal/src/executor.ts:1042-1043 (origin/main 378f141792) (a stop signals INTERRUPT_SIGNAL_NAME; only an abort then cancels the workflow); runtime-temporal/src/workflow.ts:404-414 (origin/main 378f141792) (the interrupt handler only records the stop for the next boundary, stopAtBoundary) with runtime-temporal/src/workflow-dispatch.ts:1288 (origin/main 378f141792) (the remote call is an ordinary executeRemoteSubAgentCall activity) and runtime-temporal/src/remote-sub-agent-activity.ts:126-132 (origin/main 378f141792) (the dispatch learns of a stop only through activity cancellation); core/src/orchestration/remote-sub-agent-dispatch.ts:982-996 (origin/main 378f141792) (on cancellation: one transport.interrupt, a failure logged at info, the call failed); runtime-dbos/src/steps/execute-remote-subagent.ts:545-629 (origin/main 378f141792) (the remote step's only abort is its own timeout controller) with runtime-dbos/src/lifecycle/interrupt.ts:106-137 (origin/main 378f141792) and abort.ts:68 and :273 (origin/main 378f141792) (a stop is cancelWorkflow); no transport.interrupt, transport.abort or stopOwed anywhere in runtime-dbos/src; live: WA (peer-reported, 2026-10-26, side records of the CP-217 trace, key lines spot-checked by WA); verified by the owner at origin/main 378f141792; related: CP-217 (the CF DO and runtime-js send a missed stop once and never retry it), CP-218 (a remote call's timeout never stops the child on Temporal and DBOS), CP-159 (a stop of a parent parked on suspended children), CP-17 (DBOS interrupt calls cancelWorkflow immediately); parked under the CF-first direction; severity S2 (owner, 2026-10-26): a stopped parent's remote child keeps running and billing to its end | source, peer-reported | verified | open |
| CP-239 | S3 | CF Durable Object (candidate; reproduced in local workerd only, production unverified): the clock workerd measures timers from advances only on an INCOMING event (an RPC reply, a request) or true idleness; binding awaits (D1, storage), outgoing writes and 0 ms timers never move it. The trigger is a long stretch with no incoming event: while a D1-busy stretch runs no timer fires at all, and a timer armed after one is measured from the stale clock, so once the isolate is otherwise idle it fires early by the stretch. The sharpest product site is a companion wait's deadline timer, which could then cut the wait short with a typed timeout before its deadline. | runtime-cloudflare/src/do/persistent-companion-tools.ts:2398-2439 (origin/main 378f141792) (waitForResult: setTimeout(() => deadlineCut.abort(), Math.max(1, deadline - now())) at :2404; an abort answers waitForResultTimeout, :2426-2429); live: WE-2 (peer-reported, 2026-10-26; local workerd, Miniflare via @cloudflare/vitest-pool-workers 0.9.14, lock-free diagnostic): after a 1.5 s D1-only stretch a 1 s timer armed in the same flow, with the isolate otherwise idle, fired after 1 ms of wall time, 2 of 2 (the lag 1490 ms at arm time; the fire advanced the clock by exactly its delay); over 70 s of D1-only awaits Date.now() moved 70000 ms and performance.now() 10 ms; an incoming RPC reply reset the lag to 0, which is why an earlier red with an incoming event between its steps passed; related: FU-CF-TIMER-CLOCK-PRODUCT-TIMERS (WE-2: the other DO / Worker sites that arm a timer or AbortSignal.timeout after a possible binding-only stretch); no product code measures elapsed time with performance.now() (WE-2's grep); fix direction: a timeout re-checks Date.now() against its deadline when it fires and re-arms if early; live: WE-2 (peer-reported, a later slot, 2026-10-26; local workerd): during a D1-busy stretch timers do not fire at all (the clock stays frozen until an incoming event or true idleness), and in-isolate fixes failed; so the early fire above needs the isolate idle after the stretch, and the product effect (a companion-wait timer cut early after binding-only awaits) stays plausible, not shown; severity S3, a candidate (owner, 2026-10-26): reproduced in local workerd only; production workerd may advance the clock on I/O (unverified), so the production effect is unknown until a deployed probe says otherwise | peer-reported, live-repro | partly-inferred | open |
| CP-240 | S2 | handleChatStream: a client-tool request with no new user message that carries a call's REAL output loses it when the call is past its deadline. The chat handler's deadline sweep runs on such a request before its answers are stored, and answered every lapsed pending call client_tool_timeout, the very call the request answers included; the first stored answer wins, so the real output was dropped and the turn continued on a timeout (SDK-1, the 2026-10-20 incident: a reload POST after a DO replacement carrying the late real answer). | ai-sdk/src/handler/handle-chat-stream.ts:694-720 (pre-fix, origin/main 765fdf0d76) (step 4, the deadline sweep: for a request without a trailing user message every expired pending entry, findExpiredPending at :695, is submitted client_tool_timeout at :698-703 before the request's own client-tool outputs are stored); live: the 2026-10-20 production incident (reported by the app session, relayed by the orchestrator, file tmp/reliability-investigation/incidents/2026-10-26-client-tool-deadline-and-drain-usage-write.md in its worktree): the late real answer of a lapsed call was discarded and the turn continued on client_tool_timeout; conformance: chat.lapsed-call-real-answer-kept (owner, host tier, js-chat-host; cf-do is a declared gap, FU-CONF-CFDO-HOLD-DEADLINE-ALARM): two client calls lapse with the session's deadline fires held (hold-client-tool-deadline), then the client posts c1's real output (submit-direct, c2 unanswered). At main 765fdf0d76 (owner slot s166, 2026-10-26) it fails exactly .real-answer-kept, 3 of 3: c1's row is a failed client_tool_timeout row, not its output; .sibling-timed-out (c2 timed out) and the preconditions pass. With !548 (60d006b784) merged into the cell's branch every id passes, 3 of 3; ai-sdk/src/handler/handle-chat-stream.ts (fix, !548, WA): the sweep never times out a call its own request answers, and it runs only after the request's own answers are stored (the Greptile round, fd92f277e8); conformance: fixed. chat.lapsed-call-real-answer-kept passes every id on js-chat-host, 3 of 3, with !548 merged (owner slot s166); the records MR's own lane run at main is its CI-once lane evidence; severity S2 (owner, 2026-10-26): a real client answer is silently replaced by a timeout and the model continues on it; nothing reports the loss | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-241 | S2 | CF DO (CP-106 ruling (ii)): a non-blocking companion re-consult that HELD its prior consult's undelivered outcome, then crashed (an eviction) after starting the child's continue and before its own answer committed, is answered on recovery without that outcome: the re-entered call never reclaims its own hold, its row carries no priorResult, and the next notifier pass delivers the outcome as a separate notice. The outcome is delivered exactly once, but on a channel the crash chose, not as the re-consult's priorResult. With !544's first fix the gap stayed on the C14 replay path (the child had already recorded the re-consult's outcome): the replay answered before the reclaim. | core/src/orchestration/companion-tool-dispatch.ts:1946 (pre-fix, origin/main 38b50b66bc) (handleSpawnAgent: the answer's priorResult is entry.priorResult ?? heldPrior; a re-entered call whose earlier attempt held the prior and started the child carries neither, and nothing reclaims the slot that names this call) and :1332-1338 (pre-fix, origin/main 38b50b66bc) (C14: a replayed spawn whose child outcome is recorded, recordedConsultOutcome, which only the CF DO supplies at runtime-cloudflare/src/do/persistent-companion-tools.ts:1665 (pre-fix, origin/main 38b50b66bc), answers a non-blocking spawn { started: true } with no prior); runtime-js/src/run-loop.ts:1763-1788 (pre-fix, origin/main 38b50b66bc) (the held-prior notifier pass: a row for the call that carries no prior is no delivery, so the held outcome is appended as a notice and the slot released); conformance: companion.reconsult-evicted-keeps-prior-host (owner, host tier, cf-do; js-chat-host has no evict / wake, a declared gap): the parent spawns helper, holds its second model call while helper's first consult finishes (R1 recorded, undelivered), answers it with a same-name re-consult whose child phase is held at afterChildStart (hold-child-phase: after the hold and the child's continue started), lets helper finish the re-consult (R2), evicts the backend and wakes the parent by its alarm. At 5228a6e10d on origin/main 765fdf0d76 (owner slot s168, 2026-10-26) it fails exactly .prior-result (the re-consult row {"name":"helper","status":"spawned","delivered":true}, no priorResult) and .delivered-once (one notifier notice carries R1), 3 of 3, every precondition passing. With !544's first head (8edf54af50) merged it fails the same two ids, 3 of 3: the recovery took the C14 replay branch, which returned before !544's reclaim. With !544's head e51fce2483 (the C14 reclaim, e79ff27a0f) merged every id passes, 3 of 3 (owner slot s168g, 2026-10-27, green tree 180c7f8fbc); core/src/orchestration/companion-tool-dispatch.ts:1303 (fix evidence, !544, origin/main d05119f568) (reclaimOwnPrior: a re-entered child-phase call's own held prior) used at :1371 (origin/main d05119f568) (the C14 replay branch, both answers) and :1992 (origin/main d05119f568) (the spawn path); !544's commits on main: ddd056e1f6 / 74d512fb97 (a re-entered call reclaims its own held prior: core ownHeldPrior, the CF DO's holdPriorResult and reclaimOwnHeldPrior, runtime-js) and, from this cell, 279d2cec01 (the C14 replay branch reclaims too; red 85dbffa90c). runtime-js supplies no recordedConsultOutcome, so its C14 half never applied there; conformance: fixed. companion.reconsult-evicted-keeps-prior-host passes every id on cf-do, 3 of 3, with !544's head merged (owner slot s168g); the records MR's own lane run at main is its CI-once lane evidence; severity S2 (owner, 2026-10-27): an eviction at one point of a re-consult moves the prior outcome from the call's answer to a separate notice; nothing is lost, but the model reads it where the contract says it does not | source, live-repro | verified | fixed (1 cells) |
| CP-242 | S2 | A model call that a stop cuts short mid-stream reports no usage to afterLLMCall, so the tokens the provider already billed (an Anthropic message_start's input tokens) are never booked. The Vercel adapter answers a cut stream with the partial text and no usage, and CF Workflows' LLM step returns on a stop before afterLLMCall and before it records tokens, so even a reported usage is dropped there. | llm-vercel/src/vercel-adapter.ts:853-881 (origin/main 31c79c31e5) (usage is read only from await result.usage, which rejects with the abort reason when a stop cuts the stream short: the catch at :879 leaves it undefined, so the cut stream's text result, returned from the normal path with the text streamed so far, carries no usage; the AI SDK reports usage on the finish part, :805, which a cut stream never reaches, so a provider's partial count, Anthropic's message_start input, is never seen) (W1, 2026-10-27: a stop mid-text answers text, not an error, before and after the fix); runtime-cloudflare/src/steps.ts:2864-2879 (origin/main 31c79c31e5) (CF Workflows' LLM step: a stop that aborted the call returns aborted: true before the afterLLMCall hook at :2928-2936 and before recordTokens at :2944-2956, so an aborted call's reported usage reaches neither; W1's reading of :2938 / :2953 as recording usage is the non-stop error path, which a stop never reaches); contrast: core/src/orchestration/step-iterator.ts:1108 (origin/main 31c79c31e5) (runtime-js and the CF DO pass an error result's usage to afterLLMCall, fired at :1404 for a returned error), and core usage-hooks.ts books whenever usage and the model are present, error or not; live: GitLab #180 (the Broadcast app): a model call cut short by Stop / interrupt reports usage: undefined to afterLLMCall, so partially billed tokens are never booked; conformance: usage.aborted-call-usage-reported and usage.aborted-call-usage-booked (owner, harness MR CP-242: ScriptedModel usageOnAbort, the recorded afterLLMCall channel, observe-usage on both hosts): a call streams one delta and pauses, a stop aborts it, and it answers as the Vercel adapter answers a cut stream (W1, !560): a text result carrying the text streamed so far, stopReason unknown, and usage { promptTokens: 4711 }. The scripted model replaces the adapter, so these cells witness each RUNTIME's forwarding, not the adapter. Owner slot s179 (2026-10-27, 3 of 3 identical), run with the earlier cancelled-error form of the stop answer: js-memory, js-redis and js-postgres pass .hook-fired / .hook-saw-usage (prompt 4711, completion absent); cf-do and js-chat-host pass every id of both cells and the completed-call control; cfw-workflows-d1 fails exactly .hook-fired and .hook-saw-usage (no afterLLMCall fires for the stopped call) and .usage-booked, while usage.completed-call-usage-booked passes there (the channel reads). Re-run on the cut-stream form (owner slot s182, 2026-10-27, 3 of 3 identical): every verdict is the same (js-* and both hosts pass; cfw-workflows-d1 fails exactly the ids ledgered below; the completed-call control passes there): CF Workflows' stop branch returns before afterLLMCall whatever the call answered; related: ruling (W1, 2026-10-27): an aborted call books its prompt tokens only; provisional output is ignored, so its completion count is unknown (absent), never a small number. The cell's .hook-saw-usage pins it; related: on Temporal and DBOS a stop does not cancel the in-flight call (CP-16), so no call is cut short there: the reported cell is ledgered under CP-16 on those lanes (owner slot s182, 3 of 3) until CP-16 is fixed; related: fixers: the adapter half is W1's (#180, llm-vercel: report a stopped call's provider-reported usage; witness: W1's red-first llm-vercel unit test, a mock stream with a raw message_start usage, then an abort); the CF Workflows half is TBD (the stop branch must fire afterLLMCall with the error result's usage and record its tokens; ledgered below). CP-220 (a stopped call whose provider finished anyway) shares the CF Workflows stop branch; severity S2 (owner, 2026-10-27): a production app's usage silently under-counts tokens the provider billed on every stopped call; runtime-cloudflare/src/steps.ts:3097-3115 (origin/main 3dfd151570) (fix evidence: batch 1's squash of W1's CP-242 MR !563) (the stop branch reports the call that answered: afterLLMCall with the stopped result and its usage, staged under the step so the abort discard drops its writes) with :3073-3096 (origin/main 3dfd151570) (reportModelCall: the hook, and in a finally the call's usage, so a throwing hook loses no usage) and :3040-3066 (origin/main 3dfd151570) (bookModelCallUsage: best-effort, cacheWrite included); red-first unit test runtime-cloudflare/src/__tests__/cfw-stopped-call-usage.test.ts (origin/main 3dfd151570) (W1's red-first test on !563: red over the pre-fix product, 9 green, with mutations re-reddening it); conformance: the CF Workflows half is proven by usage.aborted-call-usage-reported (.hook-fired, .hook-saw-usage) and usage.aborted-call-usage-booked (.usage-booked) on cfw-workflows-d1, which pass at W1's !563 head (its product landed on main as batch 1's squash 3dfd151570) with this record applied (owner slot s190, 2026-10-28, 3 of 3 on cfw-workflows-d1, with the completed-call control green; the other provenBy cells green in the same slot: .hook-fired / .hook-saw-usage on js-memory, js-redis and js-postgres, both cells on cf-do and js-chat-host); the adapter half (llm-vercel reports a cut stream's provider-reported usage) landed with W1's #180 MR !560 and is witnessed by its llm-vercel unit test (no cell runs the adapter: the scripted model replaces it) | source, peer-reported | verified | fixed (9 cells) |
| CP-243 | S1 | CF Workflows: an afterLLMCall hook that writes state (context.updateState) fails the step that follows with 'State version mismatch'. The hook is invoked with no stage, so its write is saved directly (the session's version moves on), and the LLM step then saves its own in-memory state at the old version; the run fails, the hook's later writes and the tool's write are lost. Any app whose afterLLMCall writes state fails every tool step on CF Workflows. | runtime-cloudflare/src/steps.ts:3100-3112 (origin/main 316c92e61b) (the LLM step invokes afterLLMCall through invokeHookWithStateTracking with no stage argument); runtime-cloudflare/src/steps.ts:1282-1293 (origin/main 316c92e61b) (invokeHookWithStateTracking: a hook with state changes and no stage saves the state directly, saveAgentState at :1292, instead of staging its writes under the step); runtime-cloudflare/src/steps.ts:3404-3405 (origin/main 316c92e61b) (the LLM step then saves its own state object, whose version predates the hook's save); contrast: on runtime-js and Temporal the same hook's writes ride the step's commit (the cell passes there, below); the CF DO runs runtime-js's executor but has no executor lane, so it is not observed here; live: peer-reported (WE-2's #181 CF Workflows slot, 2026-10-28): 'State version mismatch … expected version 1, but store has version 2' on a tool step after an afterLLMCall that wrote state; conformance: state.after-llm-call-write-then-tool-step (owner, executor tier, every lane): the agent's afterLLMCall hook counts each model call in state (hookCalls) and its one tool sets toolRan; the script is one tool step, then text. Owner slot s192 (2026-10-28, main 316c92e61b plus the cell, 3 of 3 identical): on cfw-workflows-d1 the run fails with 'State version mismatch for session …', persisting { hookCalls: 1 }, so .run-completes, .tool-write-kept and .hook-writes-kept fail; js-memory, js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres pass; dbos-postgres fails the same ids for MA-25's cause (its hook context has no updateState); related: fixer WE-2 (#181, which removes the hook's direct save on CF Workflows: its writes are staged under the step), with its own red-first unit test (cfw-step-hook-state, 'a state-writing afterLLMCall never fails a tool step'); this record flips with that member's batch; conformance: with the fix (owner slot s198, 2026-10-28, WE-2's #181 at 5565eee652 on main afd2cf80c6, archived at orch-archive/batch-2026-10-29-1-11774a6828): state.after-llm-call-write-then-tool-step passes on cfw-workflows-d1 in 3 of 3 runs, and on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres (dbos-postgres stays under MA-25's rows) — proves the fix; related: the fix is WE-2's #181, landed in batch 3's squash f85872de1e: CF Workflows' afterLLMCall writes are staged under the step (reportModelCall(stepResult, plan, { stepId, key: 'afterLLMCall' }) in runtime-cloudflare/src/steps.ts), red-first in cfw-step-hook-state.test.ts (the member commits fix 446d11c9ec and red 30da56789e are archived at orch-archive/batch-2026-10-29-1-11774a6828); severity S1 (owner, 2026-10-28): a typed, documented hook write fails every tool step of the run on one runtime, so the run fails and its state writes are lost | source, peer-reported | verified | fixed (7 cells) |
| CP-244 | S2 | A new turn (execute() with a new message) over a run suspended on client calls whose stop is only RECORDED (another process's stop, or the window between the record and its terminal) is refused as already running on runtime-js, CF Workflows and DBOS, so the stopped run stays suspended with its call unpaired and the user's new message is refused. Ruling 57a (orchestrator, 2026-10-28, extending Ruling 57 to execute()): the new turn settles the recorded stop before the pending-client-call gate and proceeds; the stopped run ends interrupted and its call is paired once. Temporal does so. | runtime-js/src/js-agent-executor.ts:5244-5256 (origin/main 95ca5fe5cf) (C-4: a paused session with pending client-tool calls is refused AgentAlreadyRunningError 'Cannot execute: session is already running with unresolved client-tool calls', before the run's stop record is read; the CF Durable Object runs the same executor); runtime-cloudflare/src/executor.ts:2944-2956 (origin/main 95ca5fe5cf) (CF Workflows execute(): the same pending-call refusal, before any stop read); runtime-dbos/src/lifecycle/run-start-event.ts:334-343 (origin/main 95ca5fe5cf) (assertNoOpenOwner: a live current run whose owner workflow is open is refused AgentAlreadyRunningError 'active'; the parked workflow stays open while it waits on its client call, and no stop record is read); contrast: runtime-temporal/src/executor.ts:532 (origin/main 95ca5fe5cf) settles a stopped suspension before its pending-calls check (settleStoppedSuspension, :1193), pinned by runtime-temporal/src/__tests__/integration/stop-request.integ.test.ts:638 (origin/main 95ca5fe5cf) ('Ruling 18: a new turn over a stopped suspension settles it and proceeds (the call is paired)') (peer-reported, WA); related: the contract. docs/superpowers/specs/2026-10-11-do-stop-as-one-signal-design.md:324-327 (origin/main 95ca5fe5cf) (Ruling 18: a new-turn entry refused expected_run_stopped settles the stopped run and proceeds), :328-330 (Ruling 41: a continuation settles a recorded stop BEFORE the partial-batch refusal) and :48-50 (Ruling 57: a with_message over a paused run with a recorded stop settles it BEFORE the batch gate); Ruling 57a (orchestrator, 2026-10-28) extends Ruling 57 to execute(); C-4 still refuses a paused run with pending calls and NO recorded stop; live: peer-reported (WA, 2026-10-28, by reading): runtime-js refuses the new turn; the owner's cell found CF Workflows and DBOS refuse it too; conformance: stop.paused-record-then-new-turn (owner, executor tier, every lane with request-stop): the agent suspends on one client call, a stop is recorded for the run (request-stop: the store's requestStop, no terminal), then a new turn TURN-2 is started. Owner slot s195 (2026-10-28, main 95ca5fe5cf plus the cells, 3 of 3 identical; .suspended and .stop-recorded green): it fails exactly .new-turn-accepted (the refusal, state_already_running, arrives in the new turn's outcome), .new-turn-completes, .stopped-run-interrupted (the first run's record stays suspended_client_tool) and .call-paired (the call has no result row) on js-memory, js-redis, js-postgres, cfw-workflows-d1 and dbos-postgres; temporal-memory, temporal-redis and temporal-postgres pass. With the fix (owner slot s196, 2026-10-28, W1's !568 at ff28e0f7f8, a pre-restack head archived at owner-archive/cp244-s196-ff28e0f7f8, 3 of 3 identical) every id passes on all 8 executor lanes, which proves the js-memory, js-redis, js-postgres, dbos-postgres and temporal rows; on !568's final head (W1's CF Workflows-only review commits after s196; landed in batch 2's squash afd2cf80c6) batch 2's lanes prove the cfw-workflows-d1 row (!571 pipeline 3001450587: every id passes on cfw-workflows-d1, job 17557299348, and on the js lanes and dbos-postgres, jobs 17557299346 / 17557299349) — proves the fix; related: the fix is W1's !568, landed in batch 2's squash afd2cf80c6 (its member commits are archived at orch-archive/batch-2026-10-28-2-4ea2bdc630): execute() settles a recorded stop on the paused run before its pending-client-call refusal on runtime-js (and so the CF Durable Object), CF Workflows and DBOS, and CF Workflows' settle skips a branch, reads the session at the gate and settles only a suspended run (the member's review round); related: the cell drives STANDARD sessions on every lane (the DBOS lane starts root sessions in standard mode); a DBOS PERSISTENT-mode execute() over a paused run with a recorded stop (runtime-dbos/src/lifecycle/execute.ts's persistent restart path) falls under the same contract but no cell witnesses it (FU-CP244-DBOS-PERSISTENT-MODE, W1); severity S2 (owner, 2026-10-28): a user's message after a stop pressed elsewhere is refused and the stopped run is left suspended with an unanswerable call, until something settles it; no data is lost | source, peer-reported | verified | fixed (8 cells) |
| CP-245 | S1 | runtime-js: a LOCAL sub-agent whose own REMOTE child pauses on a client call or an approval parks that call on itself, not on the root, so the call cannot be answered through the root (where a client sends it): the submit answers unknown_tool_call and nothing is forwarded to the producer. The local child's park looks for its tree's root through the parent's sub-session ref, and runtime-js writes an in-process child's ref only after the child's call returned, so at the park the ref is missing and the walk stops at the child. | core/src/orchestration/remote-pause.ts:128-140 (origin/main e34b96e2c2) (parkRootOf walks up through each parent's sub-session refs and stops at :137 when the ref is missing, treating the session as its own root); runtime-js/src/run-loop.ts:3418-3424 (origin/main e34b96e2c2) (the suspending run parks its remote entries on parkRootOf(partialState); parksOnOwnRoot when the walk stopped at the child itself); runtime-js/src/run-loop.ts:5056-5078 (origin/main e34b96e2c2) (the in-process child's SubSessionRef is persisted only after the child's call settled, a suspended child recorded running at :5074-5078), so it is absent while the child parks; live: WA's probe at main e34b96e2c2 (runtime-js / memory; its local probe log): the mid child reads paused with the remote park on itself, the submit of g1 through the root answers {"status":"unknown_tool_call"}, and 0 calls reach the producer; related: fixer WA, draft MR !580 (RunLoopInput.parkRootSessionId passed down at both in-process dispatch sites); the CF DO does not expose the shape (every DO sub-agent is remote); CF Workflows pending (WA); severity S1 (owner, 2026-10-29): a client or approval call of a nested remote child can never be answered through the root, so the turn can never complete; conformance: subagent.nested-remote-park-answerable-through-root (owner, executor tier, the js lanes' remote producer) at owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 4 of 4 identical: on js-memory, js-redis and js-postgres the approval submitted at the root is not accepted (.answer-accepted; the parked entry sits on the mid, so the root answers unknown_tool_call), and .turn-settles and .remote-answer-delivered fail after it; cfw-workflows-d1 passes every id (the contrast); runtime-js/src/run-loop.ts:3220 and :5414 (fix: CP-245, WA then W1; as on branch reliability-cf-wa-cp245 423f79a66f) (an in-process parent passes its park root, parkRootOfRun at :6715 (as on branch reliability-cf-wa-cp245 423f79a66f), down to its in-process children, so a remote grandchild parks its calls on the tree root); conformance: fixed. subagent.nested-remote-park-answerable-through-root passes every id 3 of 3 on js-memory, js-redis and js-postgres at a5919fc8e0 (423f79a66f plus test files only; W1's slot w1-cp279-slot-20261106-063944, phase 3, read by the owner from its result files), against its ledger from owner slot s213. W1's slot w1-cp245-slot-20261106-061804 at 423f79a66f: the unit file red 4 of 4 and the Redis / Postgres integ red 8 of 8 at ee97857f8d, both green at the fix; the full runtime-js unit 1953 passed; two mutations (the ephemeral site, the companion site) red as predicted; related: fixer moved from WA (draft !580) to W1 (orchestrator, 2026-11-06); a blocking companion that pauses under the same tree outlives its root's turn unanswerable: CP-279 | source, live-repro, peer-reported | verified | fixed (3 cells) |
| CP-246 | S2 | Temporal: a blocking companion spawn or continue whose child run ends SUSPENDED (suspended_client_tool, suspended_step_partial, suspended_awaiting_children) is answered as a successful result {name, status} (status: the raw run status) with no pending calls, and the parent's ref stays running: the child workflow returns its suspension with its awaited calls, the dispatch keeps only name / status / output, and the ref write skips a non-terminal child. So the model is never told the companion waits on a person, and the ref never says it is paused (no C2 suspended answer, CP-40). | runtime-temporal/src/workflow.ts:1543-1570 (origin/main e34b96e2c2) (a suspending run returns status suspended_* with suspended.toolCallIds / suspended.children); runtime-temporal/src/workflow-dispatch.ts:887-922 (origin/main e34b96e2c2) (the blocking spawn awaits childHandle.result() and writes the tool result {name, status, output?, error?} from it, dropping suspended; an error only for 'failed'); runtime-temporal/src/workflow-dispatch.ts:1067 and :1121-1125 (origin/main e34b96e2c2) (the blocking continue: the same result shape); runtime-temporal/src/activities.ts:8624-8630 (origin/main e34b96e2c2) (markPersistentChildCompleted maps only completed / failed and returns for any other child status, so the ref is left running with completionDelivered unset); inferred: (unverified) what follows: whether the child's call is still answerable through the root after the parent's turn ends, and what companion__waitForResult and the completion notifier read from the running ref; related: WA (peer-reported, 2026-10-29, from CP-214), verified by the owner at main e34b96e2c2; FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY (Temporal / DBOS are not on the companion entry-outcome contract); CP-40 (C2) is the contract runtime-js, the CF DO and CF Workflows answer; severity S2 (orchestrator, 2026-10-29): the model is not told a false completed (CP-127's S1), but it cannot tell the user what the companion needs and the ref misreports; S1 if the cell shows the child's call is never answerable afterwards (the turn settles while the call stays stranded); conformance: companion.blocking-spawn-answers-suspended-client-tool and -approval (owner slot s212, 3 of 3, identical sets): .spawn-answers-suspended and .ref-paused fail on temporal-memory, temporal-redis and temporal-postgres at main e34b96e2c2; js-memory, js-redis, js-postgres and cfw-workflows-d1 pass (C2); related: its ref half (the ref left running) shares CP-73's cause (markPersistentChildCompleted maps only completed / failed, runtime-temporal/src/activities.ts:8624-8630 (origin/main e34b96e2c2)); the answer half (the dispatch drops the suspension) is this record's own; one fix of the mapping covers CP-73's interrupted child and this suspended one; conformance: companion.suspends-mixed-step (owner slot s214, 2026-10-31, 3 of 3, identical sets; a blocking companion whose step runs a server tool beside a client call and suspends partial): .spawn-answers-suspended and .ref-paused fail on temporal-memory, temporal-redis and temporal-postgres both over main e3eee732d0 and with !584 2ea34599a2 merged in (CP-214 / CP-248's fix does not reach Temporal); .child-step-partial passes; related: this record's Temporal fix also carries CP-250's remainder: a submit for a paused companion's call whose parent's turn ended is refused typed on Temporal (owner_not_continuable) until FU-TEMPORAL-SUBMIT-SELF-WAKE lets a Temporal companion continue itself (the CP-250 ruling ties the two; runtime-temporal's executor names both at its startContinuation). CP-250 stays open on that alone (orchestrator, 2026-10-31); related: when this record's Temporal fix lands, both companion-answer sibling pairs flip together: the temporal-* lanes gain 'companion-self-wake' and drop 'companion-self-wake-refused' (drivers/capabilities.ts), so companion.late-pause-root-routes and companion.answer-after-parent-turn-runs run there and their *-refused-typed siblings become declared gaps; CP-234's provenBy then moves its Temporal cells back to companion.late-pause-root-routes (orchestrator, 2026-11-02) | source, peer-reported | partly-inferred | open |
| CP-247 | S2 | DBOS: a blocking companion spawn or continue whose child pauses on its own client call or approval is answered {status: 'running'} once its wait bound passes, with no C2 suspended answer and no pending calls: the spawn polls the child SESSION for completed / failed / interrupted, a paused child never reaches one, and at the bound (the child's idle timeout plus 30 s) the poll returns running. So the model is never told the companion waits on a person (CP-40's contract). | runtime-dbos/src/steps/execute-companion-tool.ts:979-1009 (origin/main e34b96e2c2) (pollChildSessionTerminal: completed / failed / interrupted end the poll; any other status polls on, and at maxWaitMs it returns { status: 'running' }, :1008-1009); runtime-dbos/src/steps/execute-companion-tool.ts:921-925 (origin/main e34b96e2c2) (the blocking spawn polls with idleTimeoutMs + 30 s; idleTimeoutMs defaults to 24 h, :765) and :701-704 (origin/main e34b96e2c2) (the blocking continue, the same poll); inferred: (unverified) the child's call is answerable through the root meanwhile (the root's submit forwards by clientToolCallOwnership, runtime-dbos/src/handles/active-handle.ts), so an answered child may complete within the wait; related: WA (peer-reported, 2026-10-29, from CP-214), verified by the owner at main e34b96e2c2; the length of the wait itself is CP-69 (any blocking child: a persistent child session ends only at its idle-out), not this id; CP-40 (C2) is the contract runtime-js, the CF DO and CF Workflows answer; severity S2 (orchestrator, 2026-10-29): the misreport of a paused child; the long wait is its own cause (CP-69); conformance: companion.blocking-spawn-answers-suspended-client-tool and -approval (owner slot s212, 3 of 3, identical sets): .spawn-answers-suspended and .ref-paused fail on dbos-postgres at main e34b96e2c2 (the companion idles out in 3 s, so the poll answers within the cell); conformance: fixed by !599 (batch 2, squash a203fec499, carrying !589's DBOS companion fix). companion.blocking-spawn-answers-suspended-client-tool and companion.blocking-spawn-answers-suspended-approval: .spawn-answers-suspended and .ref-paused fail on dbos-postgres before the fix (owner slot s212 above, 3 of 3) and pass 3 of 3 at !589 05a804cda2 (WE-2 slot w-cp69-1030-230811, conformance-bs-results-1..3, with the owner cells 4f978f4ce0 / 6ac17a2b50 alongside), every other id passing too; conformance: fixed by !599, a third cell: companion.suspends-mixed-step (a blocking companion spawn whose child pauses on a client call beside a server tool in the same step): .spawn-answers-suspended and .ref-paused fail on dbos-postgres at main e3eee732d0 and pass 3 of 3 at !589 cd9f41e8b1 (WE-2 slot w-cp69b-1031-011925, with the owner cells 4f978f4ce0 / 6ac17a2b50 / 5474a5350c alongside; owner-verified from its jsonl), every other id passing too | source, peer-reported | partly-inferred | fixed (3 cells) |
| CP-248 | S1 | runtime-js: a persistent companion whose run suspends PARTIAL (suspended_step_partial: a step with a server tool beside a client or approval call) is recorded and answered as completed with no output: runSubAgentLoop reports only suspended_client_tool, so a suspended_step_partial outcome returns undefined and the spawn, continue and resume sites write the ref completed. A blocking spawn or continue answers {name, status: 'completed'} with nothing in it, and a non-blocking ref reads completed while the companion waits (CP-127's residual: CP-127's fix covered suspended_client_tool only). | runtime-js/src/run-loop.ts:8677-8681 (origin/main e34b96e2c2) (runSubAgentLoop: only outcome.kind suspended_client_tool is reported; every other suspended_* outcome returns undefined, the comment naming the others as legacy semantics); runtime-js/src/run-loop.ts:9183 and :9192 (origin/main e34b96e2c2) (the spawn site: after an undefined loop result, the ref is written completed and the answer is { status: 'completed', output } with no output), :9712 and :9721 (origin/main e34b96e2c2) (the continue site, the same) and :8900 (origin/main e34b96e2c2) (the resume site's completed ref); live: WA's CP-214 red on runtime-js (unit, run-loop-persistent.test.ts:7372, :7400 and :7434 on branch reliability-cf-wa-cp214, red commit 349d549efe on origin/main e34b96e2c2): a child whose current run is suspended_step_partial is answered {name:'m1', status:'completed'} with no output by the blocking spawn, its non-blocking ref reads completed, and the continue answers {name:'again', status:'completed'}; related: CP-127 (fixed; its suspended_client_tool / approval cells are unaffected) is the same site and mechanism for the client-call-only suspension; the same site's suspended_awaiting_children answer is FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN (follow-ups.md), still open pending WA's probe B (its reach decides whether this record widens, a sibling id is minted, or the FU closes as unreachable; orchestrator, 2026-10-29); fixer WA, CP-214's MR (fix 3a809203ee on reliability-cf-wa-cp214, step_partial only); severity S1 (orchestrator, 2026-10-29; CP-127's grade): the parent's model is told the companion completed with no result while it waits, a silent wrong answer; conformance: fixed by !584 (2ea34599a2, batch 3). companion.suspends-mixed-step (owner slot s214, 2026-10-31, 3 of 3, identical sets): .spawn-answers-suspended and .ref-paused fail on js-memory, js-redis and js-postgres with the cell over main e3eee732d0 (tree A: the owner's member fa94fe1729) and pass every id with !584 2ea34599a2 merged in (tree B) | source, live-repro, peer-reported | verified | fixed (3 cells) |
| CP-249 | S2 | CF Workflows: a blocking persistent companion whose run ends suspended_awaiting_children (its own sub-agent waits on a client call) is answered to its parent as failed with no detail, and the parent's turn goes on as if the companion had failed: the companion ref-status resolver keeps the failed mapping for every suspension but suspended_client_tool. (The user's later answer to that call is then never acted on: CP-250.) | runtime-cloudflare/src/companion-ref-status.ts:49-62 and :72-82 (origin/main e34b96e2c2) (resolvePersistentChildRefStatus maps any event but complete / terminated to failed; suspendedChildAwaitedCalls answers only a suspended_client_tool child, a suspended_awaiting_children one keeps its failed mapping, Q3); live: WA's probe B at main e34b96e2c2 (CF Workflows, real workerd; its local probe log): parent → blocking companion mid → mid's sub-agent leaf → leaf's client call: companion answer {"name":"mid","status":"failed"}, mid ref failed, mid session paused, mid run suspended_awaiting_children, leaf run suspended_client_tool, the root owns the leaf's call, the parent completed; the submit through the root answered {"status":"accepted","continuation":{"kind":"caller_resumes"}} and the companion did not settle within 60 s; related: the strand that follows (the answer accepted with caller_resumes, the caller's resume() of the completed root refused, the call never run: WA's probe 3 at !583's head 70deed3b2e, 0 result rows in 60 s) is CP-250, a separate cause (the submit's continuation routing); runtime-js cannot reach this shape until CP-251 is fixed (a companion's own sub-agent is unresolvable there); related: CP-214 (the step_partial half of the same failed mapping, fixed by !584, which leaves awaiting_children out); CP-160 (the leaf reads failed at the base by CP-160's clobber, fixed by !583: a cross-reference, not this cause); runtime-js is unproven (probe B inconclusive: the leaf was never reached; FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN stays open); contrast: the CF DO cannot reach the shape (CP-40 (C2), D2 (c)): a companion whose resolved tree can call a client-executed or approval-gated tool is refused at spawn before anything is written, runtime-cloudflare/src/do/persistent-companion-tools.ts:1482-1495 (origin/main e34b96e2c2) (refuseChild), with the child's own backstop at runtime-cloudflare/src/do/drive/materialize.ts:140-159 (origin/main e34b96e2c2) (step 2c, framework_companion_not_supported); WA, verified by the orchestrator and the owner; related: a bordering runtime-js site, not this cause (WA, from !584's review, verified by the owner at f53834282a on reliability-cf-wa-cp214): runtime-js/src/run-loop.ts:8577-8598 (awaitAttachedChildTerminal) answers a suspended_step_partial companion child whose client calls are all answered but which still awaits its own child as suspended with no pendingCalls once the stale-pause bound passes, so its parent is not told the child waits on a descendant (unproven by a cell); severity S2 (owner, 2026-10-30, re-graded from S1 when the lost answer moved to CP-250): the parent is told a waiting companion failed (CP-214's grade for the step_partial half); related: DBOS's face of the same shape (a blocking companion whose own ephemeral sub-agent waits on a client call): unsurfaced since CP-130's fix, the call waits until its clientToolTimeout; documented-unsupported there (GitLab #73, FU-DBOS-NESTED-HITL-UNDER-COMPANION); runtime-js's is FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN | source, live-repro, peer-reported | verified | open |
| CP-250 | S1 | A user's answer to a call owned by a PAUSED persistent companion of a COMPLETED root is accepted but never acted on. The parent's turn ends while its companion waits on a client call the root owns; the submit through the root stores the answer and names the continuation as the root's (caller_resumes on runtime-js and Temporal; on CF Workflows the root's continuation, and a completed root owes none, so caller_resumes), but the caller's resume() of the completed root is refused (state_not_resumable), and nothing continues the companion: the call never runs. | runtime-js/src/js-agent-executor.ts:7715-7718 (origin/main e34b96e2c2) (the submit router's startContinuation always answers { kind: 'caller_resumes' }: 'the caller continues the turn with resume()'); runtime-temporal/src/executor.ts:2678-2681 (origin/main e34b96e2c2) (the same: 'Temporal stores only; the caller resumes'); runtime-cloudflare/src/executor.ts:5164-5196 (origin/main e34b96e2c2) (startContinuation ensures only the ROOT's submit continuation; a root that owes none answers caller_resumes); live: WA's probe 3 (its local run log): on CF Workflows at !583's head 70deed3b2e (a companion parked on its sub-agent's client call) and on runtime-js at !584's head 2ea34599a2 (a suspended_step_partial companion, now answered suspended): the submit through the root answered {"status":"accepted","continuation":{"kind":"caller_resumes"}}, the caller's resume() of the root threw 'cannot be resumed (status: completed)', and the call's result rows within 60 s: 0; inferred: (unverified) Temporal (its startContinuation answers the same; no probe) and the plain suspended_client_tool companion (CP-40 (C2)'s own shape) on every runtime: the continuation routing is the same, the companion's status does not enter it; DBOS's in-workflow continuation is unprobed; related: CP-249 (CF Workflows answers an awaiting_children companion failed: a separate cause, the same strand follows it); CP-40 (C2) made the companion's answer suspended with its awaited calls, so the model is told to wait for an answer this path loses; the CF DO is not affected: it refuses such a companion at spawn (D2 (c), CP-249's contrast); severity S1 (orchestrator, 2026-10-30; owner concurs): an accepted user answer is silently lost; the fix needs a contract ruling (which run continues a companion of a completed root); related: CP-234 is the other half: a companion that pauses AFTER its root's turn ended is not routable through the root at all on runtime-js and Temporal (the root's ownership never names the call, unknown_tool_call; ledgered from companion.late-pause-root-routes); this record is the half where the call IS routable (the companion paused while its root was live) and the answer's continuation never runs it. Owner slot s212's companion.answer-after-parent-turn-runs mixed the two (its companion's pause raced the root's end) and is not ledgered; the cell now holds the root live until its ownership names the call (owner, 2026-10-30, correcting the same day's widening, which had restated CP-234); conformance: companion.answer-after-parent-turn-runs (owner, executor tier; the root held live until its ownership names the call) at owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 3 of 3 identical: on cfw-workflows-d1 and dbos-postgres the root's submit IS accepted (CF Workflows caller_resumes, DBOS new_run) but the companion stays paused (CF Workflows) or active (DBOS) and never runs the call (.child-settles, .call-runs-once, .companion-completes). On runtime-js (js-memory, js-redis, js-postgres) the root's own turn fails first (CP-128), and on Temporal its ownership never names the call (CP-234), so this record's subject is not reached there; conformance: no longer red on CF Workflows (batch 3 !613: !594 makes a paused companion's answered call continue the companion itself): companion.answer-after-parent-turn-runs passes .child-settles, .call-runs-once and .companion-completes on cfw-workflows-d1 (owner slot s218, 2026-10-31, tree 04f2d6252a, 3 of 3; first seen in !613's pipeline 3011600936); cfw-workflows-d1 leaves the row, and dbos-postgres stays; conformance: the dbos-postgres half is fixed by !594 + !601. companion.answer-after-parent-turn-runs passes every id on dbos-postgres at owner slot s226 (2026-10-31, tree b4be0f11bc = batch 4 !617's head: main 47f0892469 plus !601 / !602 / !609-!612 / !614), 3 of 3, identical sets (batch 4's test:conformance:dbos job 17632129076 showed it as an unexpected pass first), so its dbos row is dropped. The record stays open for Temporal alone: the ruled typed refusal (owner_not_continuable) until FU-TEMPORAL-SUBMIT-SELF-WAKE, which the CP-250 ruling ties to CP-246 (orchestrator, 2026-10-31) | source, live-repro, peer-reported | partly-inferred | open |
| CP-251 | S2 | runtime-js: a persistent companion cannot call its OWN sub-agents. The companion's run loop is handed its parent's sub-agent resolver, which knows only the ROOT agent's sub-agent tools and persistent agents, so the companion's call to a sub-agent it declares fails with 'runLoop: sub-agent type '…' is not registered', returned to its model as a tool error, and the companion goes on without it. The child-scoped resolver rebuild exists only for ephemeral sub-agents. | runtime-js/src/run-loop.ts:2837 (origin/main e34b96e2c2) (the persistent companion child runLoop gets resolveSubAgent: input.resolveSubAgent, the parent's, unchanged); runtime-js/src/run-loop.ts:4853-4870 (origin/main e34b96e2c2) (Triage #7: the child-scoped resolveSubAgent rebuilt from the child's own tools, on the ephemeral sub-agent path only) and :4486-4491 (origin/main e34b96e2c2) (an unresolved sub-agent type is a failed outcome: 'runLoop: sub-agent type … is not registered'); runtime-js/src/js-agent-executor.ts:5572, :6947 and :7488 (origin/main e34b96e2c2) (buildSubAgentResolver(agent) from the ROOT agent only); live: WA's probe 3 (its local run log, runtime-js): the companion's log holds that error as its sub-agent's result and the companion finishes normally; related: makes CP-249's shape (a companion awaiting its own children) unreachable on runtime-js until fixed; WA (peer-reported, 2026-10-30, from CP-214's probe), verified by the owner at main e34b96e2c2; severity S2 (owner, 2026-10-30): a declared capability of a companion does not work on runtime-js; the failure reaches the companion's model as a tool error, never the app, and nothing is lost or misreported to the parent; conformance: companion.own-subagent-runs (owner slot s212, 3 of 3, identical sets): .sub-result-delivered fails on js-memory, js-redis and js-postgres at main e34b96e2c2; temporal-*, cfw-workflows-d1 and dbos-postgres pass. Host sibling companion.own-subagent-runs-host: .sub-ran fails on js-chat-host and passes on cf-do, so the CF DO is not affected (scope: runtime-js; the DO's companions resolve their own sub-agents) | source, live-repro, peer-reported | verified | open |
| CP-252 | S1 | runtime-js as a remote PRODUCER (agent-server, or any runtime-js host serving a remote /start): a remote-bound child's client-tool call that nobody answers never advances at its deadline after the child's first suspension. The deadline timer is armed only for a parentless run, and every suspension after the first reads the stored parent, so the child's timer is never armed; the root's timer cannot cover it either (it reads only its own session's pending calls, and a remote child's calls live on the producer). The run sits suspended indefinitely: CP-138's observable, for a remote-bound child. | runtime-js/src/js-agent-executor.ts:1149-1157 (origin/main e34b96e2c2) (armClientToolDeadline returns when parentSessionId is defined: it arms only a parentless run); runtime-js/src/js-agent-executor.ts:5636, :1567 and :7617 (origin/main e34b96e2c2) (the later arming sites pass the state's parentSessionId, which a bound child's later runs load from the stored row: never armed); core/src/client-tool/deadline.ts:70-84 (origin/main e34b96e2c2) (nextClientToolDeadlineAt iterates only the given session's own pendingClientToolCalls) and runtime-js/src/client-tool-deadline-timer.ts:51 (origin/main e34b96e2c2) (readClientToolDeadlineTarget reads the armed session); inferred: (unverified) the remote child's calls are not in the ROOT's pending map (they live on the producer, CP-157's remotePendingCalls), so the consumer's timer never sees them; the first suspension is armed only by accident (MA-46: the first run has no parent); contrast: the CF DO is not affected (alarm-based client-tool deadlines, CP-174); related: CP-138 (the same observable for a root; the runtime-js timer is its fix); MA-46 (the first run's missing parent); W1 (peer-reported, 2026-10-30, M5 trace (f) at 0a95f13245), verified by the owner at main e34b96e2c2; fixer W1, !586 (a second red / fix: arm when the binding is remote); severity S1 (owner, 2026-10-30; CP-138's grade): a run whose client left is stuck indefinitely, the user sees no progress; conformance: subagent.remote-child-later-deadline-fires (owner, executor tier, the js lanes' remote producer) at owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 4 of 4 identical: on js-memory, js-redis and js-postgres the remote child's later client-call deadline never fires (.later-deadline-fires), its earlier call answered and its submit accepted; cfw-workflows-d1 passes every id (the contrast); related: correction (owner, 2026-11-06, from W1's probe 3): the timer gate above was changed by !600 (d97398047b) to arm a remote-bound child too, and the cell stayed red. The stall's real cause: the deadline continuation resumes the remote-bound child with mode continue alone, runtime-js/src/js-agent-executor.ts:1380 (origin/main 5f2aaba172), so the resume does not adopt the stored binding and the store refuses its run-start child_binding_mismatch; the catch then drops the turn as a lost race (CP-278); runtime-js/src/js-agent-executor.ts:1386-1392 (fix: CP-252, W1; as on branch reliability-w1-cp252-deadline-binding 4ab8cf8257) (the deadline continuation is owed to the run its expiry landed on, owedRunId, so it adopts a bound child's binding); conformance: fixed. W1's slot w1-cp252fix-slot-20261106-063856, read by the owner from its result files: subagent.remote-child-later-deadline-fires fails exactly .later-deadline-fires 3 of 3 on js-memory, js-redis and js-postgres at the red head 2127fc617a, as ledgered, and passes every id 3 of 3 at the fix 4ab8cf8257; cfw-workflows-d1 passes (the contrast). The unit file red then green; the full runtime-js unit 1951 passed | source, live-repro, peer-reported | verified | fixed (3 cells) |
| CP-253 | S1 | runtime-js and Temporal stream a sub-agent's client-call tool_start to the root's stream BEFORE they write the root's clientToolCallOwnership entry for it. A client that answers on the frame (a direct submitToolResult caller, an auto-answering transport, a server integration) submits against the root while the root does not yet name the call, and the router answers unknown_tool_call (ownership_missing_toolcall / ownership_map_absent) with no fallback and no retry, although the call is routable moments later. Even with the ownership present, an answer to a LOCAL owner whose suspension commit has not landed yet goes wrong too: on runtime-js the resolver writes a pre-submit stub with a plain saveState and answers accepted, the owner's version-CAS'd suspension commit then throws StaleStateError and the child run ends failed (executor_superseded), so the submitter was told accepted for an answer that never runs; on Temporal the owner has no entry yet and the submit answers unknown_tool_call. CF Workflows and DBOS write both before the frame. A self-rooted call has the same window: its tool_start is written before its own routing entry, which rides the suspension commit. | runtime-js/src/run-loop.ts:3860-3877 (origin/main e34b96e2c2) (the client call's tool_start, the suspension boundary, is written while the tool is prepared), :3472 (origin/main e34b96e2c2) (then the suspension step commit) and :3531-3605 (origin/main e34b96e2c2) (only then the root ownership write, fenced on the root run); runtime-temporal/src/activities.ts:4089-4100 (origin/main e34b96e2c2) (the tool activity writes the client call's tool_start), :6775-6780 (origin/main e34b96e2c2) (commitSuspendedStep commits and streams step_committed) and :6792-6798 (origin/main e34b96e2c2) (only then writeRootOwnership); runtime-js/src/client-tool-resolver.ts:715-763 (origin/main e34b96e2c2) (submitOnce: with no pending entry yet it writes a pre-submit stub, isPreSubmitStub, with a plain saveState at :752 and answers accepted at :763) and runtime-js/src/run-loop.ts:2027-2037 (origin/main e34b96e2c2) (the owner's suspension commit then fails its version CAS with StaleStateError and the run ends failed, executor_superseded); the stateless suspension path never merges a stub (WA's read; inferred until the cell runs); runtime-temporal/src/executor.ts:2740-2767 (origin/main e34b96e2c2) (applySubmitToOwner: no pending entry on the owner yet answers unknown_tool_call, nothing written); core/src/client-tool/route-submit.ts:548-562 (origin/main e34b96e2c2) (the router reads the root's clientToolCallOwnership, then its remotePendingCalls, else answers unknown_tool_call; no read of the sub-session refs, no wait, no retry); runtime-cloudflare/src/client-tool-workflow-helper.ts:649 (origin/main e34b96e2c2) (CF Workflows writes the root ownership first, then registers the pending entry, :650-658: no window; the pre-submit stub already serves that order); inferred: the CF DO runs runtime-js, so an in-process sub-agent there has the same window; unverified; live: WA's s3g integ failures (2026-10-30) were this window, seen by a test answering on the frame (WA's read-only check, relayed by orchestrator-6b); related: CP-234 (the same root ownership write, fenced and best-effort: a write that fails, e.g. for a companion paused after its root turn ended, leaves the call never routable; this record is the window before a write that does land); CP-128 (the same write, an earlier window and outcome); CP-10 (fixed: who continues an awaited child); severity S1 (orchestrator, 2026-10-31; S2 before the stub half was folded in): on runtime-js the submitter is told accepted while the child run ends failed (a success answer for work that never runs, and a failed run); elsewhere a typed but wrong unknown_tool_call loses the answer; a stock useChat answering an awaited child is normally safe (human latency); related: fix directions (WA, the fixer after (c)): (a) write the root ownership before the frame, as CF Workflows does, plus a bounded wait for the owner's suspension commit instead of the stub, with a typed retryable answer past the bound; (b) a router fallback through the root's sub-session refs; conformance: owed. Deterministic cells need holds on the root ownership write and on the owner's suspension commit (no such controls yet): answer on the frame while each is held, then release; never a race ledger; related: CP-254 (a submit on a paused companion's own session answers unknown_tool_call on CF Workflows and DBOS); runtime-js/src/run-loop.ts:4161 (origin/main dcda6c8798) and :3710-3714 (origin/main dcda6c8798) (a self-rooted call: its tool_start, then its own routing entry in the suspension commit; the same on Temporal, runtime-temporal/src/activities.ts:6634-6663 (origin/main dcda6c8798)); inferred: an answer on a self-rooted call's frame gets unknown_tool_call (WA's read); no cell has held that window yet; related: widened to self-rooted calls, no new id (orchestrator, 2026-11-01): every call's routing entry is written before its frame on runtime-js, the CF Durable Object and Temporal; related: CP-259 (the window after the commit: the answer is stored and the follow-up resume is refused); related: since CP-128's MR D, ExecutorSupersededError and the executor_superseded chunk are gone: a step commit that loses a version race is StepCommitConflictError (state_concurrency_conflict). The pinned lines above keep the old names; related: since CP-128's MR D, a live CF Durable Object child with no entry answers unknown_tool_call and stores nothing, where it answered accepted with a stub, until CP-253's fix. Pinned by packages/runtime-cloudflare/src/__tests__/do-submit-after-ended-run.sqlite.test.ts. For now a live child on other runtimes keeps the stub (read from the resolver at MR D's head, before CP-253, not a lasting promise); related: the orders on the other two runtimes, read since MR D, no cell: self-rooted on DBOS and CF Workflows, the route, then the pending entry, then the frame (runtime-dbos/src/steps/client-tool.ts:145 (origin/main bb9ffe5d46), runtime-cloudflare/src/client-tool-workflow-helper.ts:164 (origin/main bb9ffe5d46)); a DBOS sub-agent, the pending entry, then the route, then the frame (runtime-dbos/src/workflows/shared.ts:726 and :738 (origin/main bb9ffe5d46)). So a client answering on the frame is not in this window on either runtime; related: a different class from this record: a CF Workflows child's approval writes its pending entry, then the root's route (runtime-cloudflare/src/approval-gate-workflow-helper.ts:109-112 (origin/main bb9ffe5d46)), and its frame follows both (runtime-cloudflare/src/workflow.ts:5522 and :5608 (origin/main bb9ffe5d46)). Whether a snapshot can expose a child's pending call in that gap is unchecked | source | partly-inferred | open |
| CP-254 | S3 | CF Workflows and DBOS answer {status: 'unknown_tool_call'} to a client-tool submit made to a paused companion's OWN session id, although that session holds the call pending: the answer is refused typed, nothing is stored and the call never runs. runtime-js and Temporal accept the same submit. The root route (a submit to the root session) is accepted there (CP-250), so this is a secondary path. | conformance: companion.answer-on-paused-child-session (owner, executor tier; the control of companion.send-to-paused-child-keeps-calls: the same cell with no turn 2) at owner slot s213 (2026-10-31, tree 72db336dcc = the owner stack x main e34b96e2c2), 3 of 3 identical: on cfw-workflows-d1 and dbos-postgres the submit to the paused companion's own session answers {"status":"unknown_tool_call"} (.answer-accepted), and .child-settles and .call-runs-once fail after it; js-redis, js-postgres, temporal-memory, temporal-redis and temporal-postgres pass every id (js-memory fails earlier, at .turn1-completes: CP-128); conformance: the same refusal first seen in owner slot s212 (2026-10-30), 3 of 3 on cfw-workflows-d1 and dbos-postgres, in companion.send-to-paused-child-keeps-calls's child-session submit; the control id was added to separate it from that cell's send; inferred: the cause is not traced yet: both runtimes refuse the call by the child session's id while that session holds it pending (runtime-js and Temporal resolve the same submit against the child's own pending entry); related: CP-250 (the root route: the answer submitted at the root IS accepted on CF Workflows and DBOS, but its continuation never runs the call); CP-234 (on runtime-js and Temporal a companion paused after its root turn ended is answerable only through its own session, the path this record finds refused on CF Workflows and DBOS); !594 / !595 change CF Workflows' submit routing: re-observe on batch 3's head before it lands, and flip there if it accepts (orchestrator, 2026-10-31); severity S3 (orchestrator, 2026-10-31): loud (a typed refusal the caller sees) and the root route works on both runtimes, so it is not a silent loss; conformance: companion.send-to-paused-child-keeps-calls (the cell this control was split from) at owner slot s217 (2026-10-31, tree b596b9e320 = the owner stack x main a203fec499), 3 of 3 identical: on cfw-workflows-d1 and dbos-postgres its child-session submit answers {"status":"unknown_tool_call"} (.answer-accepted), and .child-settles and .call-runs-once fail after it, the same as the control; it passes js-redis, js-postgres and temporal x3; conformance: companion.resumed-after-parent-turn-ended (CP-255's cell: the companion pauses only after its parent's turn ended) at owner slot s222 (2026-10-31, tree 6124f75ae2 = main 47f0892469 plus the CP-255 member), 3 of 3 identical: on cfw-workflows-d1 and dbos-postgres the submit to the companion's own session answers {"status":"unknown_tool_call"} with the call pending (.answer-accepted), and .child-settles, .call-runs-once and .companion-completes fail after it. So the refusal does not depend on whether the parent's turn is still live; conformance: fixed by !637 (WE-2's CP-234-T, CP-254 (a), batch 9 !642: core's router routes a submit addressed to a rooted child's own session to that session, and validates it as the root route does). companion.answer-on-paused-child-session, companion.send-to-paused-child-keeps-calls and companion.resumed-after-parent-turn-ended pass every id on cfw-workflows-d1 and dbos-postgres at owner slots s242b (passes 1 and 2) and s242c (pass 3; s242b's third dbos-postgres pass was killed by an outside SIGTERM and is not counted; both slots ran the same head on the same built tree) (2026-11-02, batch 9's head f961445d11 plus the owner's member), 3 of 3; first seen passing in batch 9's lane run (pipeline 3018007295) | live-repro | partly-inferred | fixed (6 cells) |
| CP-255 | S1 | Temporal: a non-blocking companion resumed after its client call is answered, after its parent's turn ended, fails writing to the parent's ended stream; the answer is accepted and the call never runs. | runtime-temporal/src/activities.ts:6417 (origin/main 47f0892469) (emitSuspensionMarker) and :6451 (origin/main 47f0892469) (the marker is stamped agentId: streamId: for a companion the stream is its parent's, so the chunk is classified as the parent's OWN chunk, and over the parent's ended stream the stream manager refuses it); runtime-temporal/src/workflow.ts:1527 (origin/main 47f0892469) (the marker is written unguarded right after the suspension commit, so the refusal fails the companion with its call recorded as pending); live: owner slots s219 / s219p (2026-10-31, the paused-child builder with the helper held until the parent's turn settled): on temporal-memory, temporal-redis and temporal-postgres the companion, answered on its own session ({"status":"accepted","continuation":{"kind":"caller_resumes"}}) and resumed, ends failed with no row for its call; its errorDetail is 'Cannot write to stream' followed by the parent's session id 'in 'ended' state'. Owner slot s220 reached the same order unheld on temporal-postgres (1 and 2 of 3); related: the cause above was WE-2's static reading of main, confirmed by the s222 probe below (the refused chunk is the companion run's suspension marker stamped with the parent's session id); the fix is WE-2's !615 (a companion skips the root's suspension marker); related: CP-129 (the same mechanism, a companion still running when its parent's turn ends fails writing into the parent's ended stream: fixed on runtime-js and CF Workflows); CP-80 (Temporal's handle projects stream chunks); severity S1 (orchestrator, 2026-10-31): the user's answer is accepted, and then the call never runs; conformance: companion.resumed-after-parent-turn-ended (owner, executor tier) at owner slot s222 (2026-10-31, tree 6124f75ae2 = main 47f0892469 plus the CP-255 member), 3 of 3 identical: on temporal-memory, temporal-redis and temporal-postgres turn 1 completes with the helper held, the call is pending, the submit on the companion's own session answers {"status":"accepted","continuation":{"kind":"caller_resumes"}}, and .call-runs-once and .companion-completes fail (the companion's session failed, 0 rows for its call). It passes js x3; on cfw-workflows-d1 and dbos-postgres the submit is refused first (CP-254); live: probe (owner slot s222 phase 0, 2026-10-31, the cell x1 on temporal-memory with a scratch log at store-memory's refused write, local only): the refused chunk is type suspension_marker, agentId the PARENT's session id, runId the companion's run, over the parent's ended stream owned by its ended turn-1 run; stack emitSuspensionMarker (runtime-temporal/src/activities.ts) to emitChunk to the stream manager's write, 3 times (the activity's attempts). The companion had failed AT its suspension, so the caller's resume() after the accepted submit was refused state_run_start_conflict / entry_status_changed; conformance: fix check at owner slot s222's tree B (2026-10-31, the same tree plus a local merge of WE-2's !615 at ece9dc714b, its delta byte-identical to !615's diff) companion.resumed-after-parent-turn-ended passes on temporal-memory, temporal-redis and temporal-postgres, 3 of 3 (red to green); companion.child-error-not-parent-outcome passes there too; conformance: fixed by !615 (WE-2, rebased onto main cb8175c1d5 at 02b6e652fc: a companion writes no suspension marker into its parent's stream, and a marker task without the suspending session fails typed and non-retryable). companion.resumed-after-parent-turn-ended on temporal-memory, temporal-redis and temporal-postgres proves the fix (red to green at s222's tree B above) | source, probe, live-repro | verified | fixed (3 cells) |
| CP-256 | S2 | Every runtime: a run that ends failed keeps its unanswered pending client calls in the FAILED session. The failure terminal is built from the loaded session, so pendingClientToolCalls survives. No run can answer such a call: a late submit meets the run's fence. It stays recorded as awaited by a run that ended, and on Temporal and DBOS, whose new turn leaves pending ids alone, a later turn can send the model a tool_use with no tool_result. Seen live on Temporal, a run failed after its suspension committed; read at the same terminal on the others. | runtime-temporal/src/activities.ts:7331-7339 (origin/main 47f0892469) (persistTerminalState builds the terminal session as { ...loaded, status, output, error, errorDetail, … }: nothing clears or drops pendingClientToolCalls); runtime-temporal/src/activities.ts:7476 (origin/main ac42f447d5) and :7514-7517 (origin/main ac42f447d5) (the same at the own-plan failed K5 and its head-read-failure fallback save); runtime-cloudflare/src/steps.ts:7769 (origin/main ac42f447d5) and :7866-7872 (origin/main ac42f447d5) (CF Workflows' failed K5 clears the pending map only for a stop, and its fallback save keeps the map); runtime-dbos/src/steps/run-commit.ts:1515-1520 (origin/main ac42f447d5) and :1558-1565 (origin/main ac42f447d5) (the DBOS failed K5 and its fallback save, both spread the loaded session); runtime-js/src/run-loop.ts:6361-6374 (origin/main ac42f447d5) (the runtime-js K5, so the CF Durable Object too: the failed session spreads the loaded session); core/src/run/fail-session-without-run.ts:86-93 (origin/main ac42f447d5) (the core save for an ended run: nothing drops the pending map); runtime-temporal/src/workflow.ts:1527 (origin/main 47f0892469) (the suspension marker, the stream write that follows the suspension commit; any throw there fails the run with its pending calls committed); runtime-temporal/src/executor.ts:2872 (origin/main 47f0892469) (the late submit's save is fenced on the entry's own run, so a submit for the failed run is refused typed: RunSupersededError, state_run_superseded, non-retryable; nothing is stored); core/src/run/fail-committed-run.ts:120 (origin/main 47f0892469) and :254 (origin/main 47f0892469) (the contrast: core failCommittedRun({ dropUnansweredPending: true }), CP-174 / ruling Q3 on the CF DO deadline alarm path, drops a failed run's unanswered calls in the same failed write); conformance: submit.answer-after-failed-suspension (owner, executor tier; the run's suspension marker write failed on every attempt by the fault-post-suspension-commit control) at owner slot s225 (2026-10-31, tree 293c254191 = main 47f0892469 plus the owner's fault-post-suspension-commit member), 3 of 3 identical on temporal-memory, temporal-redis and temporal-postgres: the fault is served, the run's handle and session read failed (.run-failed), the failed session still holds the call as unanswered pending (.pending-dropped), the late submit throws state_run_superseded with retryable false (.late-submit-refused holds), and nothing is stored (.nothing-stored holds). The other lanes write no stream chunk after their suspension commit (declared gaps); conformance: owner slot s224 (2026-10-31, tree 0632e3493e, the cell before the orchestrator ruled any typed non-retryable refusal acceptable) gave the same .pending-dropped red 3 of 3 on temporal x3; conformance: the late-submit checks of those runs (.late-submit-refused, .nothing-stored, both passing on temporal x3) moved to their own scenario, submit.late-answer-after-failed-suspension-refused (orchestrator, 2026-10-31), so that this record's cells never carry an id a fix does not reach; submit.answer-after-failed-suspension keeps .run-settles, .run-failed and .pending-dropped; inferred: the dangling tool_use. Temporal's new turn excludes the session's pending ids from its continuation heal (FU-TEMPORAL-RM81-NEW-TURN-PENDING, RM-81's fix is not on Temporal), so the next turn's model call can carry the call's tool_use unpaired, a request providers reject; a retry() is RM-82's shape on Temporal, untraced; related: RM-82 (withdrawn on the premise that no public entry leaves a failed session holding an unanswered pending call: this record breaks it on Temporal behind a stream fault); RM-81 / FU-TEMPORAL-RM81-NEW-TURN-PENDING (the new-turn settlement Temporal lacks); CP-174 (the DO's drop); CP-255 (the same failed-at-the-marker state for a companion, where its submit was accepted instead; its fix is !615); FU-DBOS-CLIENT-TOOL-RECV-OUTLIVES-FAILED-TURN (the DBOS shape); inferred: the wider sites are read, not run, at this cell: only Temporal has a stream write after its suspension commit. A DBOS run that fails holding a call reaches settle.failTurn, which clears the call before its K5 (runtime-dbos/src/workflows/shared.ts:2298 (origin/main cb8175c1d5)), so the DBOS row of the widening is the terminal itself, not that seam. WE-2's !621 proves the others with unit tests on each terminal (runtime-js/src/__tests__/cp256-failed-terminal-pending.test.ts and the runtime-cloudflare, runtime-dbos and core tests named in !621's description); conformance: fixed by !621 (WE-2, rebased onto main ac42f447d5 at 821e0b5aae): core withoutUnansweredPendingOf is applied at every failed terminal, core failCommittedRun drops by default, and the next entry's C5 heal pairs a dropped call. submit.answer-after-failed-suspension on temporal-memory, temporal-redis and temporal-postgres proves it; the other lanes have no seam for this cell (gaps.ts); conformance: fix check at owner slot s229 (2026-11-01). Tree A a873361945 = main ac42f447d5 plus the owner's member, without !621; tree B f54f5a9d9e = A plus a local merge of !621 at 821e0b5aae (patch-id of the merge delta = ac42f447d5..821e0b5aae). submit.answer-after-failed-suspension on temporal-memory, temporal-redis and temporal-postgres, 3 of 3 identical: at A .pending-dropped fails, at B it passes (red to green). The late submit, submit.late-answer-after-failed-suspension-refused, passes at both: at A it throws state_run_superseded with retryable false, at B it answers unknown_tool_call. conf-unit 1947 and harness 304 + 1 skipped pass at A; related: since owner slot s240 (2026-11-02) submit.answer-after-failed-suspension also runs on dbos-postgres (the fault's DBOS seam, pauseStream) and passes 3 of 3, through settle.failTurn's own clear before the K5, not this record's fix, so it is not in provenBy; its late submit there is CP-262; severity S2 (orchestrator, 2026-10-31): it needs a stream or store fault after the suspension commit, but then the session is left holding a call no run will answer and a later turn can fail at the provider | source, live-repro, peer-reported | partly-inferred | fixed (3 cells) |
| CP-257 | S2 | CF Workflows: the drain never clears a client call's ownership entry or writes its completed marker, so re-sending an answer that was already drained is refused unknown_tool_call instead of answered already_completed. runtime-js, Temporal and DBOS do both. The same re-send through the chat handler is rejected before any submit: CP-172's symptom, by another mechanism. | runtime-cloudflare/src/steps.ts:9444-10047 (origin/main dcda6c8798) (applyResultsAndReload, the drain: it neither deletes an ownership entry nor writes a marker); runtime-cloudflare/src/steps.ts:1519-1520 (origin/main dcda6c8798) (the only marker path off the DO: a run-start commit's markCompleted, which a drain is not); runtime-cloudflare/src/client-tool-workflow-helper.ts:55-57 (origin/main dcda6c8798) (the comment gives the clear to the drain, which has none); runtime-cloudflare/src/workflow.ts:5478 (origin/main dcda6c8798) and runtime-cloudflare/src/client-tool-workflow-helper.ts:449 (origin/main dcda6c8798) (completedTombstoneRetentionMs is passed in, and the helper never reads it); ai-sdk/src/handler/extract-resume-intent.ts:475-482 (origin/main dcda6c8798) (read, not run: no pending entry and no marker for a root-owned call reads as unknown, so the handler rejects a re-send before any submit); contrast: runtime-temporal/src/activities.ts:6330-6342 (origin/main dcda6c8798) and runtime-dbos/src/steps/client-tool.ts:565-573 (origin/main dcda6c8798) clear the entry and write the marker at the drain; live: WA's probe (2026-11-01, main c806b247a8; step level, a mock binding, one root-owned call, no real Workflows instance): a re-send after the drain answers unknown_tool_call, the entry stays, no marker; inferred: a drained id stays in rootOwnedCallIds (runtime-cloudflare/src/steps.ts:6397 (origin/main dcda6c8798), :6511 (origin/main dcda6c8798), :9029 (origin/main dcda6c8798)), so a later remote child reusing the call id would meet parkedCallConflicts; and the map grows per call. Neither measured; related: CP-172 (the handler half of the same re-send, on the hosts that do write the marker); related: the fixer is WA, a small member after CP-128's MR D, using the field-scoped op's complete in the drain; severity S2 (owner, accepted by the orchestrator, 2026-11-01): a typed but wrong answer to a harmless re-send; nothing is lost or run twice; conformance: subagent.remote-park-after-call-id-finished (owner, executor tier) on cfw-workflows-d1 at owner slots s235 and s236 (2026-11-01), 3 of 3 each: a local child's drained call X, then a real remote child parking on its own X, fails the parent with 'Remote child's call id(s) X already name another call on root session': the first run of the rootOwnedCallIds inference above. The remote ref stays paused_awaiting_client under the failed parent. The error text was read at owner slot s236; the cell's later setup fix changed only turn 1, and at owner slot s237 (3 of 3) the same five ids fail, with the text not printed because a ledgered cell logs none; related: the re-send itself still has no cell: its own scenario on every executor lane (pause, submit, continue as the submit says, complete, re-send, expect already_completed and nothing stored) is owed, ledgered only from identical runs; conformance: on !658 (WE-2, cdd653d36a) the ledgered cell no longer reaches this finding: subagent.remote-park-after-call-id-finished passes every id on cfw-workflows-d1 at owner slot s248b (2026-11-03, tree 9fbf7751b6 = !658's head on !648 c40753d1c5), 3 of 3 identical. !658's drain completes the drained call on the tree root, which clears its route, so the reused id parks, and the router's owner-first rule routes the answer to the remote child; related: CP-260's fix goes before or with this one (orchestrator, 2026-11-01): clearing ownership and writing a marker would otherwise move CF Workflows from this typed failure onto CP-260's silent discard | source, probe, live-repro, peer-reported | partly-inferred | open |
| CP-258 | S2 | runtime-js (so the CF Durable Object) and DBOS: a late submit to a child session that already ended, failed, completed or interrupted, for a call with no pending entry, writes a pre-submit stub and answers accepted, although no run can consume it and resume() refuses a failed session. The stub branch is for a child still registering its call and never reads the session's status. CP-256's fix made this reachable by a real late answer: the failed terminal now drops the entry that used to meet the run's fence. | runtime-js/src/client-tool-resolver.ts:730 (origin/main dcda6c8798) and :741-745 (origin/main dcda6c8798) (no pending entry on a child: the stub branch, gated on the root id and the call's name, not on the session's status), :760-761 (origin/main dcda6c8798) (the stub is saved with no run fence) and :772 (origin/main dcda6c8798) (accepted); runtime-dbos/src/dbos-agent-executor.ts:1223-1228 (origin/main dcda6c8798), :1254-1255 (origin/main dcda6c8798) and :1262 (origin/main dcda6c8798) (the same branch on DBOS, through the root; WE-2's read has it followed by a DBOS.send); core/src/run/fail-committed-run.ts:180 (origin/main dcda6c8798) (withoutUnansweredPendingOf, !621: the failed terminal drops the entry, so the answer now reaches the branch above); contrast: Temporal answers unknown_tool_call for no entry on the owner (runtime-temporal/src/executor.ts:2777 (origin/main dcda6c8798)); WE-2's read has CF Workflows and any root session doing the same; related: reported by Greptile on !626 (note 4134972096) and read by WE-2 at main dcda6c8798. The branch predates !621 for an id with no entry on any ended child, so !621 widened who reaches it rather than creating it; related: CP-256 (fixed; its fix is the regression's cause, and that record's own mechanism stays fixed); CP-253 (the same stub, while the child's run is still committing); related: the fixer is WE-2, expected on main before CP-128's MR D (orchestrator, 2026-11-01): unknown_tool_call and nothing stored on an ended child; a child still active or paused keeps the stub; severity S2 (owner, accepted by the orchestrator, 2026-11-01): the run is already reported failed and a following resume() is refused typed, so the client learns; the wrong parts are the accepted answer and a stored stub nothing consumes; conformance: submit.late-answer-on-completed-child-refused, submit.late-answer-on-failed-child-refused and submit.late-answer-on-interrupted-child-refused (owner, executor tier) at owner slot s233 (2026-11-01, main dcda6c8798 plus these records, without the fix), 3 of 3 identical: on js-memory, js-redis and js-postgres the late submit on the ended companion's own session answered accepted, caller_resumes, and stored a stub (.answered-unknown and .nothing-stored fail). Temporal, CF Workflows and DBOS passed: the DBOS stub is on its path through the root, which these cells do not build; conformance: fixed by !631 (WE-2): an ended child answers unknown_tool_call and stores nothing, or already_completed where the id is a call its root finished, while an active or paused one keeps the stub. The same cells at owner slot s233 with its fix merged pass on all eight executor lanes, 3 of 3; related: WE-2's tests carry what these cells cannot. cf-do has no cell, as the harness worker cannot host a persistent companion: packages/runtime-cloudflare/src/__tests__/integration/do-submit-tool-result.integ.test.ts runs direct submits per ended status and a failed child through the root DO in real workerd (the status is seeded there), and packages/runtime-cloudflare/src/__tests__/do-submit-after-ended-run.sqlite.test.ts has a real failed run on DOStateStore. DBOS through the root: packages/runtime-dbos/src/__tests__/submit-tool-result.test.ts on a fake store. Also packages/runtime-js/src/__tests__/submit-after-ended-run.test.ts; inferred: not run anywhere: a real failed run on workerd, a child Durable Object's rootSessionId set by a real parent, and an ended DBOS child through the root against a real store; related: since CP-128's MR D, a live CF Durable Object child with no entry answers unknown_tool_call and stores nothing, where it answered accepted with a stub, until CP-253's fix. Pinned by packages/runtime-cloudflare/src/__tests__/do-submit-after-ended-run.sqlite.test.ts. For now a live child on other runtimes keeps the stub (read from the resolver at MR D's head, before CP-253, not a lasting promise) | source, live-repro, peer-reported | verified | fixed (9 cells) |
| CP-259 | S2 | runtime-js: a client answer that lands between a suspension's commit and the run's own suspended status write is stored and answered caller_resumes, and the follow-up resume is then refused state_already_running: its seen read took the run as running, the status write landed, and the run-start commit read that as the run having moved on. Over the chat route the client gets a 409 for a healthy turn, and the turn stays parked with the answer stored until the next message. | runtime-js/src/run-loop.ts:2568 (origin/main 36fea5f686) (the run writes suspended_client_tool after the suspension commit, which made the pending entry visible); runtime-js/src/js-agent-executor.ts:6095 (origin/main 36fea5f686) (the resume's seen read) and runtime-js/src/run-start.ts:364-366 (origin/main 36fea5f686) (the commit compares id and status, so the run's own forward step reads as a loss; the one retry needs the seen status to still be current); runtime-js/src/js-agent-executor.ts:7973-7974 (origin/main 36fea5f686) (the comment says a resume of a paused session is never refused transiently); ai-sdk/src/handler/handle-chat-stream.ts:1523-1540 (origin/main 36fea5f686) (caller_resumes calls resume once, and a refusal goes to the typed answer with the input committed; nothing else on this path continues the turn); conformance: submit.resume-across-suspension-status-write (owner, executor tier, hold-run-status) at owner slot s235 (2026-11-01, main 5eef81d2db plus these cells), 3 of 3 identical: on js-memory, js-redis and js-postgres the submit answers accepted, caller_resumes and the resume throws 'is already running (status: suspended_client_tool)' (.resume-admitted, .turn-completes fail). .order-held passes: the status write landed at readSeenRun's read, which kept 'running'; conformance: chat.answer-across-suspension-status-write (owner, host tier) at owner slot s235, 3 of 3 identical, on js-chat-host: the request ends http 409, code state_already_running, body "retryable":true; the session then reads paused with the call pending for the whole 30 s bound (.turn-continues fails). .next-message-drains passes: a next message ends completed with the stored answer in history; contrast: runtime-temporal/src/executor.ts:2113 (origin/main 36fea5f686) keeps only the run id from its read (owner's read of why), and submit.resume-across-suspension-status-write passes on temporal-memory, temporal-redis and temporal-postgres, 3 of 3 at owner slot s235, with the same order held; inferred: not shown on a stock useChat: the host cell posts through the driver's client model. A stock client is expected not to retry a 409 by itself (docs/reference/ai-sdk.md:259-261 documents status 'error' and the field for a refused send, so its constant was not decided for this path); inferred: the window is narrow, the commit to the status write, so a human answer would rarely land in it; a client tool that answers on its frame is the ordinary way in. CI saw this error on C-2 (job 17648031820, js-postgres, iteration 48, and a !557 car), with no timing in the log, so those runs are not proven to have taken this order; related: DBOS and CF Workflows are open: by the documented suspension order each writes its status before its commit, but neither is run here (gaps.ts). The CF Durable Object answers a submit through its wake, not caller_resumes (read); related: a fix must keep CP-96's control.stale-entry-loses-after-winner passing: that record owns the rule that fires here, and its stale entry must still lose; related: CP-253 (the window BEFORE the suspension commit); WE-2's probe first forced this order (2026-11-01, root session, memory store); severity S2 (orchestrator, 2026-11-01): a typed error, the answer kept, and the next message recovers, 3 of 3. The client sees an error on a healthy turn and the turn sits parked, so its fix goes ahead of the other S2s; ai-sdk/src/handler/handle-chat-stream.ts:3415-3428 (origin/main 36fea5f686) (read, not run: the body's retryable: true is alreadyRunningResponse's constant, returned at :3509 (origin/main 36fea5f686) before inputCommitted is read, so core/src/errors/http-error.ts:265 (origin/main 36fea5f686) does not run on this path; WA's read, confirmed by the owner). A fix decides what the field should say here; core/src/store/run-start.ts:78-91 (fix: !647, WA, ac6faf6133; as on branch reliability-cf-wa-cp259 ac6faf6133) (isOwnSuspensionStatusStep: the seen run read running, the current run is the same run now suspended_*, the session reads paused and the latest checkpoint is that run's own step commit: the run's own suspension status step, not a move the plan missed); runtime-js/src/run-start.ts:384 (fix: !647, WA, ac6faf6133; as on branch reliability-cf-wa-cp259 ac6faf6133) (the run-start commit accepts that step) and runtime-js/src/js-agent-executor.ts:6559 (fix: !647, WA, ac6faf6133; as on branch reliability-cf-wa-cp259 ac6faf6133) (so does the resume's seen check); conformance: fixed by !647 (WA, ac6faf6133). submit.resume-across-suspension-status-write passes every id on js-memory, js-redis and js-postgres, and chat.answer-across-suspension-status-write passes on js-chat-host (cf-do a declared gap), at owner slot s245 (2026-11-02, tree ac6faf6133 = !647's head on main 47ceddad1f), 3 of 3 identical | source, probe, live-repro, peer-reported | verified | fixed (4 cells) |
| CP-260 | S2 | A root's completed marker answers for a call it no longer is: when a root finished call X and a remote child later parks on its own call X, a root-addressed answer gets already_completed, consumed, is not forwarded and is discarded, and the child stays paused. It needs one id used twice in a tree. | core/src/client-tool/route-submit.ts:548-556 (origin/main 36fea5f686) (the marker short-circuit answers before :563 (origin/main 36fea5f686) reads remotePendingCalls); core/src/orchestration/remote-pause.ts:101-115 (origin/main 36fea5f686) (parkedCallConflicts checks parked and owned ids, not markers, though its comment gives the same reason: the router answers those first); conformance: subagent.remote-park-after-call-id-finished (owner, executor tier) at owner slot s235 (2026-11-01, main 5eef81d2db plus these cells), 3 of 3 identical: on js-memory, js-redis and js-postgres, with a real remote child over a real socket, the answer to the parked call gets {"status":"already_completed","continuation":{"kind":"consumed"}} (.answer-accepted, .turn2-settles, .turn2-completes, .remote-child-completes fail). So a real park leaves the marker in place; live: WA's router-level probe (2026-11-01): the same current park forwards 0 times with the marker and once without; inferred: the CF Durable Object path: cf-do has no cell, as the capability exists only on the runtime-js lanes and CF Workflows, so a DO remote child reaching this is a code read; inferred: it needs one id used twice in a tree, and whether real providers repeat ids is an open assumption; this cell repeats one on purpose. If they do, this is S1; core/src/client-tool/route-submit.ts:601-611 (fix: !658, WE-2, cdd653d36a; as on branch reliability-we2-late-answers-1b cdd653d36a) (owner first: a call's owner, else the remote child parked on it, decides the answer before any marker is read; the short-circuit at route-submit.ts:548-556 (origin/main 36fea5f686) is deleted); core/src/client-tool/route-submit.ts:653-655 (fix: !658, WE-2, cdd653d36a; as on branch reliability-we2-late-answers-1b cdd653d36a) (the root's marker answers only when nothing owns or parks the call); conformance: fixed by !658 (WE-2, cdd653d36a). subagent.remote-park-after-call-id-finished passes every id on js-memory, js-redis and js-postgres at owner slot s248b (2026-11-03, tree 9fbf7751b6 = !658's head on !648 c40753d1c5), 3 of 3 identical: the answer to the parked call is accepted, turn 2 settles and completes, and the remote child completes; related: CF Workflows reaches the conflict first, CP-257's row. CP-257's fix clears ownership and writes a marker, which would move CF Workflows from that typed failure onto this silent discard, so this fix goes before or with CP-257's (orchestrator, 2026-11-01); related: CP-185 (a re-answer of one call discarded after its forward; here a different call's answer is never forwarded); CP-181 (non-injective child session ids); severity S2 (orchestrator, 2026-11-01): a success-shaped answer and the real answer dropped, on an assumption about ids not yet shown on real providers | source, probe, live-repro, peer-reported | partly-inferred | fixed (3 cells) |
| CP-261 | S2 | runtime-js: a run whose suspension committed and whose own suspended_client_tool status write then fails on every attempt is reported failed by its handle and its stream, but writes no failed terminal: the session stays paused holding the call, and the run record stays running. A new execute() is refused as already running, inside the run's lease and after it, because the session still holds an unresolved client-tool call that nothing will answer: the client was told the run failed. | runtime-js/src/run-loop.ts:2583-2591 (origin/main 6741980fd2) (the suspension branch: a failed status write fails the stream and returns failed; no K5 is written, so the session keeps its paused status and its pending call); runtime-js/src/run-loop.ts:6688 (origin/main 6741980fd2) and :6690 (origin/main 6741980fd2) (writeRunStatus stops the lease's renewals before its write, then tries it 3 times; a failure leaves the run record running with its lease stopped but never released); runtime-js/src/js-agent-executor.ts:5334-5343 (origin/main 6741980fd2) (C-4: execute() over a paused session holding pending calls is refused AgentAlreadyRunningError; this check reads no lease); conformance: submit.suspension-status-write-fails (owner, executor tier, fault-run-status: every suspended_client_tool write of the session's current run fails) at owner slot s240 (2026-11-02, main bb9ffe5d46 plus the owner's member), 3 of 3 identical on js-memory, js-redis and js-postgres: the fault is served and the handle reads failed (.handle-failed); the stream reads failed (.stream-agrees passes); the session reads paused with the call pending (.session-agrees fails); the run record reads running (.run-terminal fails); conformance: re-proved on main 6741980fd2 (batch 9 changed runtime-js's run loop and executor under this cell): owner slot s243 (2026-11-02, this record's member on main 6741980fd2), 3 of 3 identical on js-memory, js-redis and js-postgres, the same four ids failing as ledgered (.session-agrees, .run-terminal, .entry-not-blocked, .entry-after-lease-not-blocked); conformance: the same runs: an execute() 5 to 17 ms after the handle failed, inside the lease (.entry-in-lease-window passes), and another 35003 to 35036 ms after it, once the lease has ended, are both refused with 'is already running (status: Cannot execute: session is already running with unresolved client-tool calls. …)' (.entry-not-blocked and .entry-after-lease-not-blocked fail); runtime-js/src/js-agent-executor.ts:5338-5343 (origin/main 6741980fd2) and core/src/errors/agent-errors.ts:26 (origin/main 6741980fd2) (C-4's sentence is passed as AgentAlreadyRunningError's status argument, so the message reads '(status: Cannot execute: …)'; wording only, not a finding); inferred: what clears the calls without the client, read at origin/main 6741980fd2, none run. The failed run arms no deadline: runtime-js/src/js-agent-executor.ts:9335-9346 (origin/main 6741980fd2) arms one only for a suspended outcome. Only a later touch arms it: getHandle on a paused session (:5709-5711 (origin/main 6741980fd2)) or a submit (:7889 (origin/main 6741980fd2)); armed, it fires at the call's deadline (core/src/constants.ts:117 (origin/main 6741980fd2), 5 minutes by default) while the process lives. Whether its owed check continues a run still recorded running is not traced; inferred: retry() is refused, since the session is not failed (runtime-js/src/js-agent-executor.ts:7307 (origin/main 6741980fd2)). A resume or a with_message entry passes C-4 and meets the run-start owner guard, which refuses it while the lease lasts (core/src/store/run-start.ts:308-313 (origin/main 6741980fd2)) and by reading admits it after; a chat client's next message is that with_message entry. A stop is answered as stopping, with nothing to honour it, while the run reads executing (runtime-js/src/execution/in-process-runs.ts:131-136 (origin/main 6741980fd2), up to the lease plus one ttl); a later stop should commit its terminal and clear the calls. None of these is run here: open; inferred: the CF Durable Object runs the runtime-js executor, so it is expected to write the same way; whether its settle path repairs this state is not traced. Temporal, DBOS and CF Workflows are not run (gaps.ts); related: CP-259 (the same status write landing late rather than failing); CP-256 (a failed terminal drops unanswered calls, which this run never writes); CP-96's control.stale-entry-loses-after-winner must keep passing under a fix; severity S2 (orchestrator, 2026-11-02): it needs a store failure after the suspension commit, then direct execute() callers stay refused until something clears the calls, while a chat client's next message is expected to recover after the lease (by reading) | source, live-repro | partly-inferred | open |
| CP-262 | S2 | DBOS: a run that fails while holding an unanswered client call clears that call by writing its completed marker, so a late submit for a call nobody answered is told already_completed, consumed: the client is told its answer was stored and drained when it never was. Unchanged by CP-128 MR D. | runtime-dbos/src/workflows/shared.ts:2303-2332 (origin/main 6741980fd2) (settle.failTurn clears every call the turn holds before its K5, through clearPendingClientToolCallStep); runtime-dbos/src/steps/client-tool.ts:424-433 (origin/main 6741980fd2) and :563-586 (origin/main 6741980fd2) (the clear goes through clearOwnership, the op's complete, which writes the completed marker unconditionally, as its comment says); core/src/client-tool/route-submit.ts:584-593 (origin/main 6741980fd2) (an active completed marker answers already_completed, consumed, before the ownership lookup); conformance: submit.late-answer-after-failed-suspension-refused (owner, executor tier; the run's pauseStream after its suspension commit fails on every attempt, fault-post-suspension-commit's DBOS seam) at owner slot s241 (2026-11-02, main bb9ffe5d46 plus the owner's member), 3 of 3 identical on dbos-postgres: the run fails (.run-failed), a late submit for the call answers {status: already_completed, continuation: {kind: consumed}} (.late-submit-refused fails), and nothing is stored (.nothing-stored passes); conformance: re-proved on main 6741980fd2 (batch 9 changed DBOS's client-tool step and core's router): owner slots s243 (passes 1 and 2) and s243b (pass 3, on the same head and built tree, after s243 hit its cap on a load wait) (2026-11-02, this record's member on main 6741980fd2), 3 of 3 identical on dbos-postgres with only .late-submit-refused failing as ledgered; the Temporal contrast passes every id on temporal-memory, temporal-redis and temporal-postgres, 3 of 3; contrast: the same cell passes on temporal-memory, temporal-redis and temporal-postgres at owner slot s241, 3 of 3: the late submit answers unknown_tool_call; inferred: the reach, by reading: every failTurn of a DBOS run that holds calls, the calls of its suspending step (runtime-dbos/src/workflows/shared.ts:3442 (origin/main 6741980fd2)) or calls restored for a resumed step (:2460 (origin/main 6741980fd2)); failTurn is the turn's settle for any application failure (:1510-1520 (origin/main 6741980fd2)). Only the pauseStream path is run; inferred: a fix's shape, for a fixer, not prescribed: a marker must not mark a call nobody answered; clearing an unanswered call removes its route and pending entry without writing a completed marker; related: CP-257 is its sibling and the opposite case (CF Workflows' drain writes no marker for an answer it drained), so a fixer reads both; CP-49 (a different DBOS already_completed: an owner that cannot be resolved); CP-260 (a marker answering for a call that is not its own); CP-256 (the failed-terminal drop: on DBOS submit.answer-after-failed-suspension passes through failTurn's own clear); severity S2 (orchestrator, 2026-11-02): a silent misreport to the client, behind a run failure while a call is held | source, live-repro, peer-reported | verified | open |
| CP-263 | S2 | Temporal: a client's real answer to a lapsed call, submitted while the resume drain records the lapse, is answered accepted and then lost. The drain writes the session's pending calls back from its own pre-drain read, so the stored answer is dropped, and the call's one result row is client_tool_timeout. | runtime-temporal/src/activities.ts:5492 (origin/main 47ceddad1f) (applyResultsAndReload reads the pending calls once); :5522 (origin/main 47ceddad1f) (a lapsed call counts as answered, so a single lapsed call is drained, not held as a partial batch); runtime-temporal/src/activities.ts:5758 (origin/main 47ceddad1f) and :5768 (origin/main 47ceddad1f) (a call past its deadlineAt with no submittedAt in that read gets a client_tool_timeout row, kind 'timeout'); :5788 (origin/main 47ceddad1f) (only calls still pending in that read go to remainingPending); :5854 (origin/main 47ceddad1f) (the rows are appended); runtime-temporal/src/activities.ts:6249 (origin/main 47ceddad1f) (the final write reloads the session) and :6272-6273 (origin/main 47ceddad1f) (but writes pendingClientToolCalls from remainingPending, the pre-drain read); :6290 (origin/main 47ceddad1f) (the whole-state save); runtime-temporal/src/executor.ts:2834 (origin/main 47ceddad1f) (applySubmitToOwner refuses only an entry that already carries submittedAt) and :2892 (origin/main 47ceddad1f) (it stores the answer and answers accepted): a submit between the drain read and its final save is accepted, and that save drops it; conformance: submit.lapsed-call-answer-kept (owner, executor tier, hold-lapse-write seam timeout-row: the drain's timeout-row append is held while the real answer is submitted) at owner slot s244b (2026-11-02, this record's member !648, stacked on !645, over main 47ceddad1f), 3 of 3 identical on temporal-memory, temporal-redis and temporal-postgres: the submit answers {"status":"accepted","continuation":{"kind":"caller_resumes"}}, and the call's one result row is the client_tool_timeout row (.accepted-answer-kept fails; .window-held, .submit-answered, .run-settles and .one-result-row pass); contrast: submit.lapsed-call-answer-kept-unheld (the same lanes, no hold: the drain records the lapse and the run settles first, then the answer arrives) passes 3 of 3 at the same slot: the late submit answers {"status":"already_completed","continuation":{"kind":"consumed"}} and the timeout row stands. The timeout wins only for an answer that arrives after the lapse is recorded; related: CP-264 is the DBOS counterpart (a different path: the recv give-up's clear); planned fix: late answers 1c (batch 10), whose drain becomes the tree root's complete with a timedOut flag, then one gated consumePending commit; related: an app report's T1 incident is SDK-2's late-answer path, a different path, not this record; severity S2 (orchestrator, 2026-11-02): an accepted user answer is silently replaced by a timeout result; conformance: at owner slot s247 (2026-11-02, !648's head 395e27bdfe with !656 6236c82557 merged), 3 of 3 on temporal-memory, temporal-redis and temporal-postgres, the held submit is answered {"status":"already_completed","continuation":{"kind":"consumed"}} and the call's one row is the client_tool_timeout row, so .accepted-answer-kept passes on its refused arm: the held window now falls after the lapse is stored, and the accepted-then-lost path is not exercised. The ledger rows are dropped (pending 1c); the unheld control passes on the same lanes and on dbos-postgres. Re-run on !656's next head at owner slot s247b (2026-11-03, !648's head 395e27bdfe with !656 b35ec579f1 merged), 1 of 1 on the same three lanes: the same refused arm, the same response and the timeout row; conformance: at owner slot s250 (2026-11-03, tree 202a09dd74 = !648 c40753d1c5 with !659 11abb81354 and !660 merged; !660 at 67485cadfb (product), head 11431f46e9 adds only CLAUDE.md), 3 of 3 identical on temporal-memory, temporal-redis and temporal-postgres, the held submit is answered {"status":"late_answer_after_timeout","consumedAt":…} (a timestamp, different per run) and the call's one row is the client_tool_timeout row: .accepted-answer-kept passes on its refused arm, and the accepted (replace) arm is not reached. The unheld control answers late_answer_after_timeout on the same lanes and {"status":"already_completed","continuation":{"kind":"consumed"}} on dbos-postgres, each with the coded timeout row. The cell does not observe whether the late answer is recorded or reaches the model; inferred: (read, not run) on !660 the held write is the drain's gated consume append, runtime-temporal/src/activities.ts:5956 (!660 67485cadfb, as on branch reliability-cf-wa-late-answers-1c-temporal 67485cadfb), and it runs after the tree root's complete, runtime-temporal/src/activities.ts:5938 (!660 67485cadfb, as on branch reliability-cf-wa-late-answers-1c-temporal 67485cadfb) (core completeBeforeGatedConsume), which clears the call's route and writes its timedOut marker, so the router finds no owner and answers from the marker at core/src/client-tool/route-submit.ts:653-655 (!658 cdd653d36a, as on branch reliability-we2-late-answers-1b cdd653d36a). A replace needs the submit to land after the timeout is stored and before that complete, which this cell's seam does not hold; runtime-temporal/src/executor.ts:2856 (fix: !660, WA, 3789d47333; as on branch reliability-cf-wa-late-answers-1c-temporal 3789d47333) (Temporal's owner path answers a submit over an answered entry through core answerOverAnsweredEntry, core/src/client-tool/late-answer-submit.ts:148 (fix: !658, WE-2, 3789d47333; as on branch reliability-cf-wa-late-answers-1c-temporal 3789d47333): an unconsumed timeout of a live run is replaced by the real answer and the submit answers accepted); runtime-temporal/src/activities.ts:5985-5996 (fix: !660, WA, 3789d47333; as on branch reliability-cf-wa-late-answers-1c-temporal 3789d47333) (the drain's gated consume, refused because the answer changed after its read, re-plans once from a fresh read, so the call's row is the real answer); conformance: fixed by !660 (WA, late answers 1c on Temporal; !660 at 3789d47333, its product unchanged since 4a354a5293), stacked on !658 (1b, WE-2) and !656 (piece B, WA): this flip depends on all three. submit.lapsed-call-timeout-replaced (owner, executor tier, hold-lapse-drain: the tree root's complete is held, so the stored timeout is still unconsumed and routed) passes every id on temporal-memory, temporal-redis and temporal-postgres at owner slot s255 (2026-11-03, tree 41415acfe1 = the member 4b33cdcf10 with !659 11abb81354 and !660 3789d47333 merged), 3 of 3 identical: the submit answers {"status":"accepted","continuation":{"kind":"caller_resumes"}}, the call's one result row is the real answer ({"failed":false,"content":"{\"answer\":\"REAL-ANSWER-IN-THE-LAPSE-WINDOW\"}"}), and the model's next input carries it. Red at the same slot on tree a755da4f05 (the member with !656 b35ec579f1 merged), 1 of 1 on the same lanes: {"status":"already_completed","continuation":{"kind":"caller_resumes"}} with the timeout row (.answer-accepted, .real-answer-row, .accepted-answer-kept fail). The driver checked every cell held the complete (the harness's hold-entered line, 3 per run); related: submit.lapsed-call-late-answer-delivered (the timeout-row append held, after the complete) passes on the same lanes at s252 (3 of 3) and s255 (1 of 1): the answer is late_answer_after_timeout, recorded once, and delivered to the model as one hidden notice; red on !656 (already_completed, no notice). The s250 line above is that window, the existing cell's; related: owner slots s252 (its timeout-replaced cells), s253 and s254 ran with a harness defect, the cell view's lapse-hold wrapper dropping the seam (fixed in this member), so those cells held the timeout-row append, not the complete: not evidence for this record | source, live-repro, peer-reported | verified | fixed (3 cells) |
| CP-264 | S2 | DBOS: a client's real answer to a lapsed call, submitted after the workflow's wait for it gave up and before the clear lands, is answered accepted and then lost. The clear deletes the pending entry with no check for a stamped answer, so the run continues with client_tool_timeout. | runtime-dbos/src/client-tool-resolver.ts:119 (origin/main 47ceddad1f) and :121 (origin/main 47ceddad1f) (recordOutcome: a real answer is recorded; on a give-up, winner null) and :132 (origin/main 47ceddad1f) (clearPendingClientToolCallStep) and :140 (origin/main 47ceddad1f) (the outcome is client_tool_timeout); runtime-dbos/src/steps/client-tool.ts:424 (origin/main 47ceddad1f) (clearPendingClientToolCall), :429 (origin/main 47ceddad1f) (clearOwnership first: the op's complete, :569 (origin/main 47ceddad1f) and :576 (origin/main 47ceddad1f), writes the completed marker unconditionally) and :439 (origin/main 47ceddad1f), :443 (origin/main 47ceddad1f) and :448 (origin/main 47ceddad1f) (then a re-read that deletes the entry and saves, with no submittedAt check); runtime-dbos/src/dbos-agent-executor.ts:1163 (origin/main 47ceddad1f) (markPendingSubmittedBestEffort stamps the real answer on the entry, :1244 (origin/main 47ceddad1f), and saves it, :1294 (origin/main 47ceddad1f), then sends it to a wait that already gave up): a submit between the give-up and the clear's marker is accepted, and the clear deletes it; conformance: submit.lapsed-call-answer-kept (owner, executor tier, hold-lapse-write seam complete-op: the clear's completed-marker op is held while the real answer is submitted; agent conf-lapse-client, whose tool sets its own 3 s deadline, CP-139) at owner slot s244b (2026-11-02, this record's member !648, stacked on !645, over main 47ceddad1f), 3 of 3 identical on dbos-postgres: the submit answers {"status":"accepted","continuation":{"kind":"new_run","runId":"<suspended run>:continuation"}}, and the call's one result row is the client_tool_timeout row (.accepted-answer-kept fails; .window-held, .submit-answered, .run-settles and .one-result-row pass); contrast: submit.lapsed-call-answer-kept-unheld (no hold: the clear lands and the run settles first, then the answer arrives) passes 3 of 3 at the same slot: the late submit answers {"status":"already_completed","continuation":{"kind":"consumed"}} (the clear's marker) and the timeout row stands. The timeout wins only for an answer that arrives after the lapse is recorded; inferred: a second window, not run: a submit between the clear's marker and its delete meets the marker and is answered already_completed, consumed; that answer refuses the client's answer and loses nothing; related: CP-263 is the Temporal counterpart (a different path: the resume drain's write-back); CP-262 (the same clear on a failed run: its unconditional marker answers a call nobody answered already_completed); planned fix: late answers 1c (batch 10), whose clear re-reads the entry so a stamped real answer wins and the delete is gated on submittedAt; related: an app report's T1 incident is SDK-2's late-answer path, a different path, not this record; severity S2 (orchestrator, 2026-11-02): an accepted user answer is silently replaced by a timeout result; runtime-dbos/src/steps/client-tool.ts:503 (fix: !662, WE-2, 72d4cdefde; as on branch reliability-we2-late-answers-1c-dbos 72d4cdefde) (settleLapsedClientToolCallStep: a lapsed wait is settled, not cleared: an answer already on the entry is the call's answer; else the tree root's complete, then a version-pinned removal that re-reads the entry, so an answer that landed in between restores the route and the marker and is the answer); runtime-dbos/src/client-tool-resolver.ts:163 (fix: !662, WE-2, 72d4cdefde; as on branch reliability-we2-late-answers-1c-dbos 72d4cdefde) (the recv give-up settles through it); conformance: fixed by !662 (WE-2, late answers 1c on DBOS, 72d4cdefde, stacked on !658 at 6bbd4a99bf): this flip depends on !662. submit.lapsed-call-answer-kept (the recv give-up's complete held while the real answer is submitted) passes every id on dbos-postgres at owner slot s258 (2026-11-03, tree 90acdfaca6 = the records head 0ebbdb6904 with !662 merged), 3 of 3 identical: the submit answers {"status":"accepted","continuation":{"kind":"new_run",…}} and the call's one result row is the real answer ({"failed":false,"content":"{\"answer\":\"REAL-ANSWER-AFTER-LAPSE\"}"}). Red at the same slot on tree da27b1fb25 (the records head with !658 6bbd4a99bf, !662's base), 1 of 1: accepted, then the timeout row (.accepted-answer-kept fails). The unheld control passes on both (on !662 the late submit answers late_answer_after_timeout, the coded timeout row standing); runtime-dbos/src/steps/client-tool.ts:510-600 (fix: !662, WE-2, 13957ef575; as on branch reliability-we2-late-answers-1c-dbos 13957ef575) (the settle at !662's Greptile-fix head: a retrying step, :510; its body :518-600, now confirming the tree root's timedOut marker before the removal and retrying the step if it is not active, :549-570; the version-pinned removal still restores an answer that landed, :572-579); runtime-dbos/src/client-tool-resolver.ts:163 (fix: !662, WE-2, 13957ef575; as on branch reliability-we2-late-answers-1c-dbos 13957ef575) (the give-up settles through it); conformance: re-checked on !662's Greptile-fix head 13957ef575 (on !658 b463cfd044) at owner slot s260 (2026-11-03, tree 1ea9d433df = the records head 957d81c944 with !662 13957ef575 merged), 3 of 3 identical on dbos-postgres: the held submit.lapsed-call-answer-kept answers {"status":"accepted","continuation":{"kind":"new_run",…}} with the real answer as the call's one row, and the unheld control answers {"status":"late_answer_after_timeout",…} with the coded timeout row standing | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-265 | S3 | Temporal's resume drain, and by reading CF Workflows' drain, build a client-tool call's timeout and submitted-error result rows without the clientToolErrorCode metadata, against core's documented contract for that key; DBOS and runtime-js stamp it. A consumer can tell such a row from any other failed result only by its text. | core/src/types/state.ts:832 (origin/main 47ceddad1f) (COMMON_METADATA_KEYS.CLIENT_TOOL_ERROR_CODE, clientToolErrorCode: documented as present on tool-result messages for failed client-executed tool calls, one of CLIENT_TOOL_ERROR_CODES, so replay, tracing and other consumers classify without parsing the text); runtime-temporal/src/activities.ts:5717-5722 (origin/main 47ceddad1f) (the drain's submitted-error row) and :5760-5765 (origin/main 47ceddad1f) (its client_tool_timeout row): both built with createToolResultMessage and no metadata, so they carry only helixToolFailed; contrast: runtime-dbos/src/workflows/shared.ts:2588-2590 (origin/main 47ceddad1f) and :3703 (origin/main 47ceddad1f), runtime-js/src/js-agent-executor.ts:4606-4608 (origin/main 47ceddad1f) (the drain, which the CF Durable Object runs too), runtime-js/src/with-message-entry-rows.ts:318 (origin/main 47ceddad1f) and core/src/client-tool/new-turn-settlement.ts:227 (origin/main 47ceddad1f) stamp it; conformance: submit.lapsed-call-answer-kept-unheld (owner, executor tier; the lapse is recorded and the late answer is refused, {"status":"already_completed","continuation":{"kind":"consumed"}}) at owner slot s244c (2026-11-02, this record's member !648, stacked on !645, over main 47ceddad1f), 3 of 3 identical: .timeout-row-coded (a Zod safeParse of the standing row's metadata for clientToolErrorCode client_tool_timeout) fails on temporal-memory, temporal-redis and temporal-postgres and passes on dbos-postgres. The same slot's mutation run on dbos-postgres (the row swapped for another failed result) turned only .timeout-row-coded red; inferred: CF Workflows builds the same rows without the key, read, not run: runtime-cloudflare/src/steps.ts:9744-9749 (origin/main 47ceddad1f) (the submitted-error row, which is also its lapse: the deadline is written onto the entry as submittedError client_tool_timeout at :9255 (origin/main 47ceddad1f)) and :9563-9568 (origin/main 47ceddad1f) (client_tool_abandoned). The cell is a declared gap there (no 'hold-lapse-write' on cfw-workflows-d1); related: the legacy exported executeSingleToolCall (runtime-js/src/execution/tool-executor.ts:445-462 (origin/main 47ceddad1f)) carries the code only as errorDetail.code, not row metadata; it is unreachable from the stock runtimes (:318-331 (origin/main 47ceddad1f): it throws unless a custom resolver implements waitForResult); related: CP-263 and CP-264 (the same cells' held scenario); the fix is a stamp at each of Temporal's two drain rows and CF Workflows' two, as DBOS does (orchestrator, 2026-11-02: to WA); severity S3 (orchestrator, 2026-11-02): no user-visible loss; the documented structured classification of a client-tool failure is missing on two runtimes; conformance: fixed on the Temporal lanes by !651 (WA, 32d1b4bd0f: Temporal's and CF Workflows' drained failure rows now carry the code). submit.lapsed-call-answer-kept-unheld passes every id, .timeout-row-coded included, on temporal-memory, temporal-redis and temporal-postgres at owner slot s246 (2026-11-02, !648's head 10e7a6e426 with !651 32d1b4bd0f merged), 3 of 3, and is unchanged on dbos-postgres. The fix also stamps CF Workflows' two rows, but that lane is a declared gap for this cell (no 'hold-lapse-write'), so fixed covers the ledgered Temporal lanes only | source, live-repro | partly-inferred | fixed (3 cells) |
| CP-266 | S2 | A regression of batch 10 (SDK-2 1b): on runtime-js, an afterSubAgent hook's state write is silently dropped in a step that also delivers a late client-tool answer's notice. The hook writes the live state object runtime-js hands the core step iterator; the iterator rebases its own state when it delivers late notices, so its afterSubAgent fold pick-up reads the rebased copy's stale customState, and the step commit drops the hook's write. No error is raised and the run completes. | runtime-js/src/run-loop.ts:5697-5702 (origin/main fcb82e788e) (the sub-agent hook context: updateState reassigns state.customState on the live object), :2906-2917 (origin/main fcb82e788e) (the hooks are bound to the object the iterator receives as its iteration param); core/src/orchestration/step-iterator.ts:1014 (origin/main fcb82e788e) (late-answer notice delivery before the model call rebases state = { ...state, messages }), :1252 (origin/main fcb82e788e) (the same on a retried model attempt); core/src/orchestration/step-iterator.ts:2590-2598 (origin/main fcb82e788e) (the afterSubAgent fold pick-up composes preStepCustomState with state.customState: after a rebase, the stale copy); related: origin (git log -S on the delivery line): c6d815d6c8 (late answers 1b, 2026-11-02) and 91bcc89ee7 (a retried attempt delivers the notices), landed in batch 10 b9cbd26272; at 47ceddad1f (main before batch 10) the step iterator has no lateAnswerNotices at all; inferred: the CF Durable Object runs runtime-js's executor (runtime-cloudflare/src/do/drive/build-executor.ts), so it should share the defect; its host-tier twin cell proves or refutes that; related: peer-reported by WE-2 (2026-11-03, while working on G1), routed by the orchestrator; the owner verified every site above. CF Workflows, Temporal and DBOS deliver late notices on their own paths: their cells decide whether they share the pattern. Fixer: WE-2 (its own small MR on main); related: SDK-2 (batch 10, late client-tool answers); RM-113 and CP-264 (the same class: a side effect silently lost); conformance: owner slot s266 (2026-11-03, tree bcbf962e2a = these records on main fcb82e788e, the pre-fix product), 3 of 3 identical: state.subagent-fold-kept-with-late-notice fails exactly .fold-kept-with-notice on js-memory / js-redis / js-postgres (the control .fold-kept-control and .notice-delivered pass, so the late notice is delivered in the step that calls the sub-agent and the fold is lost there); on temporal-memory / temporal-redis / temporal-postgres, dbos-postgres and cfw-workflows-d1 it passes every id, the control and .notice-delivered included: those runtimes deliver the notice on their own paths and keep the fold. Ledgered from that run; core/src/orchestration/step-iterator.ts:791 (fix: !671, WE-2; origin/main 8cd7c7013f) (runtimeState = input.state: the runtime's own state object, which a late-notice rebase of state never replaces), :2613-2618 (fix: !671, WE-2; origin/main 8cd7c7013f) (the afterSubAgent fold pick-up reads runtimeState.customState); core/src/orchestration/step-iterator.ts:1632-1637 (origin/main 8cd7c7013f) (the fix inside !671, whose regression never reached main: after a late notice the model hooks' customState writes land on the rebound state, so they are written back to runtimeState once, after the last model hook and before the step's baseline, and the hooks' writes and the afterSubAgent fold compose; bb701a30b7's pick-up alone kept the fold but dropped the hooks' writes, the review note on !671); conformance: fixed by !671 (WE-2, bb701a30b7, based on main fcb82e788e): this flip depends on !671. Owner slot s267 (2026-11-03, tree 3ddca9f5a6 = these records 133107f4bf merged with !671's head bb701a30b7, 3 of 3 identical): state.subagent-fold-kept-with-late-notice passes every id on js-memory / js-redis / js-postgres (unexpected-pass against the s266 rows: the fold is committed beside the late notice), and still passes on temporal-* / dbos-postgres / cfw-workflows-d1; severity S2 (orchestrator, 2026-11-03): an app state write is silently dropped; below the top severity (it needs a late notice and a state-writing afterSubAgent in one step), and the transcript and the model are unaffected; conformance: state.subagent-fold-kept-with-late-notice (executor tier) is the cell: leg A (the fold with no notice) is the control and gates leg B (a late notice delivered in the step that calls the sub-agent); .notice-delivered proves the shape before .fold-kept-with-notice is judged; core/src/orchestration/step-iterator.ts:1313-1317 and :1588 (!810, as on origin/main 41ea410715) (since Additive Sessions 4a, runtime-js and the CF DO deliver the late notice through the context pass, before each model call; the lateAnswerNotices path at :1161 serves a runtime with no context pass; the runtime's own state object is held at :916, const runtimeState = input.state, so the fix above still holds under the context pass) | source, peer-reported | verified | fixed (3 cells) |
| CP-267 | S2 | runtime-js / the CF DO, CF Workflows, and since G1 Temporal and DBOS: a remote sub-agent whose result is settled at a later entry skips the parent's afterSubAgent hook and its childCustomState fold. The reconcile and the entry settle write the child's result row and settle its ref, and nothing else, so a remote child that finished while its parent was away never reaches the hook or the parent's state, although its result row is in the log and the transcript is valid. | runtime-js/src/js-agent-executor.ts:3604 (origin/main de10dbb892) (reconcileRemoteSubAgents), :3726-3755 (origin/main de10dbb892) (a completed or failed verdict: settleRemoteChildCall with the result row) and :3813-3840 (origin/main de10dbb892) (settleRemoteChildCall appends the row with its ref settle; no hook, no fold); core/src/orchestration/remote-children-before-input.ts:151 (origin/main de10dbb892) (settleRemoteChildrenBeforeInput) with :220-250 (origin/main de10dbb892) (the same verdicts become rows and settles only), called at runtime-js/src/js-agent-executor.ts:6639, :7671 and :8849 (origin/main de10dbb892); runtime-js/src/run-loop.ts:3275-3276 (origin/main de10dbb892) (contrast: a remote call the run itself awaits goes through executeSingleRemoteSubAgentCall, which fires afterSubAgent) with core/src/hooks/types.ts:619 (origin/main de10dbb892) (the hook); inferred: no cell yet. Read from source by the owner (2026-11-04, re-read at de10dbb892). The hook's writes and the child's state are lost with no error; the model still sees the result row; related: Temporal and DBOS take the same core settle since G1 (runtime-temporal/src/activities.ts:1571, :1693 and :10864; runtime-dbos/src/steps/run-commit.ts:633), rows and settles only, so they have the same gap. CF Workflows does not call the core settle; its entry path is unverified, and a cell decides; related: fixers WE-2 (core and runtime-js) and WA (Temporal, DBOS), after G1 (CP-216); related: correction (owner, 2026-11-05): CF Workflows does call the core settle, without completing its children, so it has the same gap. runtime-cloudflare/src/steps.ts:1534 (origin/main 08f2daa182) (startRunCommit's settleRemote calls settleRemoteChildrenBeforeInput), with callers at :1580, :1860 and :1940 (origin/main 08f2daa182); a child still running at the entry is re-attached and fires the hook at :6485 (origin/main 08f2daa182). Read by WE-2 at its branch head ab77fe6132 and by the owner at main; core/src/orchestration/remote-children-before-input.ts:247 and :261 (fix: CP-267, WE-2; as on branch reliability-we2-cp267 f0cb451b8a) (completeChildren: the settle returns each finished child's afterSubAgent payload and fold) and runtime-js/src/js-agent-executor.ts:3342 (as on branch reliability-we2-cp267 f0cb451b8a) (the entry fires afterSubAgent before the commit that settles the child), passed completeChildren at :6880, :7937 and :9133 (as on branch reliability-we2-cp267 f0cb451b8a); conformance: fixed on runtime-js and the CF DO (!725). WE-2's slot w-cp267d at ab77fe6132 on origin/main 08f2daa182, read by the owner from its result files: remote.finished-child-settled-resume-host, remote.finished-child-settled-with-message-host and remote.finished-child-wake-host pass every id on cf-do (host) 3 of 3; their red (w-cp267c, the pre-fix product) fails .after-subagent-once and .state-write-stored 3 of 3, the wake cell included (the reconcile settles a finished child on the wake route too). The contrast remote.running-child-reattached-wake-host (a child still running at the wake, re-attached) passes 3 of 3 at the head and at the pre-fix product, and its double-fire mutation fails exactly .after-subagent-once 3 of 3; conformance: CF Workflows red, by run (WE-2's slot w-cp267e at 9830b067e0, !725 plus the red cells and no fix, on origin/main 08f2daa182, read by the owner from its result files): subagent.remote-finished-child-hook-retry and subagent.remote-finished-child-hook-new-turn on cfw-workflows-d1 fail exactly .after-subagent-once and .state-write-stored 3 of 3 each, every precondition and the pairing, ref and completion ids passing. The parent's instance is terminated, so it settles failed; the child completes while the parent is down; then the entry. The cells land with WE-2's CF Workflows fix; severity: a user hook and state fold are silently skipped on a path that needs the parent to be away when the child ends; the transcript stays valid and nothing a person sees is wrong at once; runtime-cloudflare/src/steps.ts:1613 (fix: CP-267 on CF Workflows, WE-2; as on branch reliability-we2-cp267-cfw 4c188f9e16) (the entry settle completes its finished remote children, and :1520 (as on branch reliability-we2-cp267-cfw 4c188f9e16) fires each one's afterSubAgent once for the run-start step); conformance: fixed on CF Workflows (!730). WE-2's slot w-cp267f at 88690d61cc on origin/main 08f2daa182, read by the owner from its result files: subagent.remote-finished-child-hook-retry and subagent.remote-finished-child-hook-new-turn pass every id 3 of 3 on cfw-workflows-d1 (red in w-cp267e, above); w-cp267g (the fold's result discarded) fails exactly .after-subagent-once and .state-write-stored 3 of 3 each | source, live-repro, peer-reported | partly-inferred | open |
| CP-268 | S2 | DBOS: a message-less resume() after a Stop of an inline continuation whose client-tool deadlines lapsed attaches to the cancelled workflow instead of continuing the session. lapsedContinuationOwner reads only the continuation's LAPSED event and never whether that run is still live, so the resume returns the stopped run's outcome: the turn does not continue, and the model is never called. | runtime-dbos/src/lifecycle/resume.ts:197-208 (origin/main de10dbb892) (a resume that is neither a rewind nor with_message asks lapsedContinuationOwner first and, on an answer, attaches to that workflow); runtime-dbos/src/lifecycle/resume.ts:544-559 (origin/main de10dbb892) (lapsedContinuationOwner: the current run, and its LAPSED ContinuationEvent; no read of the run status); runtime-dbos/src/lifecycle/retry.ts:76 (origin/main de10dbb892) (no retry() way out: it refuses a session that is not failed); conformance: transcript.stopped-partial-omitted-after-late-notice (owner slot s265, 2026-11-03, tree R = MA-48's records then numbered MA-47, on main b9cbd26272, the pre-fix product), 3 of 3 identical: on dbos-postgres .continue-omits-partial and .notice-reaches-model fail, because the message-less continue attaches to the cancelled workflow and no model input follows; every other lane passes both ids; conformance: re-proved by owner slots s274 (2026-11-04, tree 12bc6f097a on main de10dbb892) and s275 (tree f35f9dab35), 3 of 3 identical each: transcript.stopped-partial-omitted-after-late-notice fails exactly .continue-omits-partial and .notice-reaches-model on dbos-postgres, and passes every id on js-memory, js-redis, js-postgres, temporal-memory, temporal-redis, temporal-postgres (docker backend) and cfw-workflows-d1; related: workarounds that take another path (resume.ts:197-198): a with_message resume, and a from_checkpoint resume; related: found by the owner while witnessing MA-48; fixer to be named (batch 11); severity: the caller is handed a finished run with nothing continued, silently; it needs a client-tool deadline to lapse and a Stop of the continuation, and the two workarounds exist; related: since Additive Sessions 4a the cell's late notice can take either shape: a hidden user row, or on runtime-js and the CF DO a context row with source late-answer (packages/e2e/src/conformance/scenarios/stopped-partial-notices.ts:124 (!810, as on origin/main 41ea410715), isLateNoticeRow). The DBOS defect above and its ledger are unchanged | source | verified | open |
| CP-269 | S1 | agent-server over a Temporal producer: a /resume carrying a message whose entry fails before its run-start commit answers 200 as if the run started, and the message is never written. The route's only guard is the handle's preCommitFailure, which runtime-temporal never sets, and awaitEntryOutcome leaves such a failure to the handle. The caller is told the turn started and attaches to the stream of the previous turn. | agent-server/src/agent-server.ts:693-703 (origin/main de10dbb892) (resumeAgent: a resume that failed before its commit must not answer started; the only check is handle.preCommitFailure); core/src/types/runtime.ts:770 (origin/main de10dbb892) (preCommitFailure, optional on a handle); runtime-temporal/src/executor.ts:786-806 (origin/main de10dbb892) (awaitEntryOutcome: a conflict and a refusal are typed, and any other early workflow outcome is left to the handle; nothing in runtime-temporal/src sets preCommitFailure); inferred: read, not run. Found by WA tracing a busy refusal, confirmed at source by the owner (2026-11-04). No request has been observed answering 200 this way; related: G1 adds an ordinary way in: a with_message resume over a lost delegation whose child is still running is refused by core settleRemoteChildrenBeforeInput, which fails Temporal's startRunCommit activity; the handle's result() then reports failed with errorDetail { message } alone (WA's slot g1r at ca131b5be6). That refusal should answer the busy 409; related: agent-server /start has the same gap: it shares the guard at agent-server/src/agent-server.ts:604-605 (origin/main de10dbb892), and Temporal's execute() and retry() go through the same awaitEntryOutcome (WA, read from source). The chat handler does not reach this path on Temporal, which does not declare supportsAbandonOnResume. DI-04 is the lost code of the same failure; DI-39 is the chat host surface, fixed; related: reachable on main without G1 (WA, read from source): a history-load failure in startRunCommit's entry plan (state_history_incomplete from a corrupt row or a persistent count mismatch) is non-retryable and ends the workflow with no run, and awaitEntryOutcome special-cases only completion refusals and run-start conflicts. Not enumerated further; related: beside it, on any runtime: agent-server's resumeAgent has no conflict mapping, so a lost entry on /resume answers a 500 INTERNAL_ERROR, not the busy 409 (WA, read from source). Not silent, so no record of its own; it is fixed with this one; severity: a user message is dropped behind a success answer, with no sign to the caller; it needs agent-server hosting a Temporal producer and an entry that fails before its commit; runtime-temporal/src/executor.ts:312-320 and :783 (fix: !677, WA; G1) (awaitEntryOutcome returns the failure and execute(), resume() and retry() attach it as preCommitFailure; no workflow change), runtime-temporal/src/activity-failures.ts:52 (the busy refusal maps to the run-start conflict) and agent-server/src/agent-server.ts:636-646 (a lost entry on /resume answers the typed 409, with no streamId); related: reasoned, not run: an old workflow history replays its recorded failure, so the fix changes no determinism; the slot's replay leg covers two existing files only | source, peer-reported | verified | fixed (no lane can reach it: 3 tests) |
| CP-270 | S3 | DBOS: a stop of a persistent loop that lands after a turn committed its terminal state but before its run status is written is not honoured. The run still reads live, so the stop skips the idle-loop rule, cancels the loop and finds the K5: the session stays completed, the loop is not finalized and the stop request is never read. Ruled: such a stop ends the loop (session interrupted, loop finalized, stop consumed), and the turn keeps its committed rows and output. | runtime-dbos/src/lifecycle/interrupt.ts:217 (origin/main 98fb3973d4) (a live run skips the idle-loop branch), :297 (origin/main 98fb3973d4) (the loop is cancelled as a run owner) and :324 (origin/main 98fb3973d4) (settleCancelledOwner returns on a committed K5); runtime-dbos/src/workflows/shared.ts:2150 (origin/main 98fb3973d4) (the K5) and :2217 (origin/main 98fb3973d4) (finalizeRun, after it): the window; inferred: read at source only (WA's candidate, read by the owner at 6d6736abe8, re-read at 98fb3973d4, 2026-11-05). Persistent mode is DBOS only; related: CP-69 (fixed) owns the idle-loop stop; this window is not in it. CP-84 is the same K5-to-status window on other paths. CP-272 is its entry side; related: the witness is stop.persistent-loop-stop-after-k5, DBOS only, anchored on a hold of the run-status write (a hold-run-status member for the DBOS lane, shared with CP-268); no race ledger | source, peer-reported | partly-inferred | open |
| CP-271 | S1 | DBOS: a with_message resume of a session whose run is suspended, or was just woken, while its workflow is still open returns a handle on that workflow and never writes the message. The caller gets a handle and no error, and the input is lost. | runtime-dbos/src/lifecycle/resume.ts:238-245 (origin/main 98fb3973d4) (pending calls or a suspended run with an owner not closed: unless the resume rewinds it attaches, whatever the mode) and :561 (origin/main 98fb3973d4) (attachToPendingWorkflow takes no message); conformance: finish.with-message-over-deferred on dbos-postgres (owner slot s268, tree 132e3a6bef, 2026-11-03), 3 of 3 identical: the with_message user row was not persisted, and no model call followed the resume. The cell was then amended for the wake lanes, so this is not its current shape and no ledger row; inferred: the mechanism is read at source, re-read at 98fb3973d4 (2026-11-05); the anchored cell has not run; related: the witness is its own anchored cell: a woken continuation runs a gated regular tool and the with_message arrives while it is held. RM-74 is the same entry on another order; CP-64 is the submit-and-resume race on DBOS | source, live-repro | partly-inferred | open |
| CP-272 | S2 | DBOS: a new turn sent after a run committed its terminal state but before its run status is written is refused state_already_running. The fast refusal reads a live run with an open owner, though the turn has ended, so a caller that saw the result and answered at once gets a busy error. | runtime-dbos/src/lifecycle/execute.ts:302 (origin/main 98fb3973d4) (the read-only fast refusal) and runtime-dbos/src/lifecycle/run-start-event.ts:334 (origin/main 98fb3973d4) (assertNoOpenOwner: a live run status with an open owner refuses); runtime-dbos/src/workflows/shared.ts:2150 (origin/main 98fb3973d4) (the K5) and :2217 (origin/main 98fb3973d4) (finalizeRun): the window; conformance: seen once on the full DBOS lane in owner slot s270 (2026-11-03), on the with-message cell, as a wrong-reason failure of that cell, which was then amended to wait for no live run. One sighting of a timing window: not a ledger row; inferred: the cause is read at source, re-read at 98fb3973d4 (2026-11-05); no held cell has run; related: CP-270 is the stop side of the same window. The witness is a held cell on the hold-run-status member shared with CP-268 | source, live-repro | partly-inferred | open |
| CP-273 | S3 | A message-carrying resume over a remote call whose child's start was never confirmed is refused busy until the child's deadline: about 41 minutes at the defaults on runtime-js, the CF DO, Temporal and DBOS, about 68 on CF Workflows. The refusal is typed, writes nothing and ends: past the deadline every status settles, and a plain new turn proceeds throughout. But a child whose start died in its first second holds a with_message caller for the whole bound. | core/src/orchestration/remote-children-before-input.ts:304 (origin/main 98fb3973d4) (the awaiting_remote_child refusal) and core/src/orchestration/lost-remote-child.ts:158 (origin/main 98fb3973d4) (not_found inside the deadline is owed: CP-177, for a start still committing); runtime-js/src/js-agent-executor.ts:6668 (origin/main 98fb3973d4) (resume) and :7705 (origin/main 98fb3973d4) (retry), runtime-cloudflare/src/steps.ts:1534 and :1848, runtime-temporal/src/activities.ts:1573, runtime-dbos/src/steps/run-commit.ts:795 and :868 (origin/main 98fb3973d4): one core helper, refusing on every runtime; core/src/orchestration/lost-remote-child.ts:116 (origin/main 98fb3973d4) (the deadline: startedAt, the watch budget, 60 s grace) and runtime-cloudflare/src/remote-watch-budget.ts:49 (origin/main 98fb3973d4) (CF Workflows: 6 attempts of 10 min and 4 status reads, 310 s of engine waits and the grace, 4090 s at production engine defaults); inferred: read, not run (WA's trace and the owner's, 2026-11-05). Temporal and DBOS's budget argument is WA's table. A tool's timeoutMs scales the wait; related: a regression of G1 (CP-216, 98fb3973d4) through CP-177's existing grace: G1 did not add the rule. It commits the call, so an entry reaches the rule, where before the sweep settled a call with no row at once. That comparison is read for runtime-js and the CF DO only; inferred: reach: an ordinary chat send goes through execute and does not meet it. A chat send reaches it only through handleChatStream's with_message path, on a lost step holding both a client-tool and a remote call (runtime-js, the CF DO and CF Workflows; reasoned, not shown, and unverified on CF Workflows). Direct with_message callers reach it on every runtime, and retry with a message on runtime-js and DBOS; CF Workflows' and Temporal's retry entries are not checked; related: fix directions for its owner to test: the running: 'orphan' entry a new turn uses, a keyed re-POST, or a shorter grace for a start never confirmed (the long budget exists for a child that is working); related: CP-176 (the same refusal for an interrupted, paused or dead-producer child), CP-196 (the deadline arithmetic), RM-80 | source, peer-reported | partly-inferred | open |
| CP-274 | S3 | runtime-js / the CF DO: a companion call re-entered after its run crashed (reenterChildPhase) commits its result row but fires none of the row hooks a step's own child phase fires. On the DO the re-entered call fires neither onMessage nor afterTool; on in-process runtime-js only onMessage is missed, because afterTool fires inline in the call's body there. The row and the result are correct; a consumer that counts, bills or logs from those hooks undercounts after a crash. | runtime-js/src/run-loop.ts:10612 and :10703 (origin/main 08f2daa182) (reenterChildPhase hands the re-entered calls to runChildPhase with commitResults only: no onMessage, no deferred afterTool); core/src/orchestration/step-iterator.ts:2389 and :2398 (origin/main 08f2daa182) (contrast: a step's own child phase fires onMessage for each row it commits, then that call's deferred afterTool); runtime-cloudflare/src/do/durable-object-agent-base.ts:4233 (origin/main 08f2daa182) (the DO's companion tools carry _companionDispatchOverride, so the call's afterTool is deferred to the phase, which the re-entry skips); related: scope (W1, from source, peer-reported; a grep and a read of the two callers): runChildPhase has two callers, the step iterator and reenterChildPhase, both in the loop the DO runs. CF Workflows, Temporal and DBOS fire their row hooks from their own tool sites and have no such re-entry; crash replay on those three was not traced; related: CP-106 (the companion re-entry itself); an app session measured 0 fires for a re-entered call (peer-reported, relayed by the orchestrator, 2026-11-05); core/src/orchestration/child-phase.ts:274 (fix: CP-274, W1; as on branch reliability-w1-cp274-reentry-hooks 2d41b5e1ea) (runChildPhaseWithHooks: each row commits, then fires onMessage and its call's deferred afterTool); core/src/orchestration/step-iterator.ts:2360 (fix: CP-274, W1; as on branch reliability-w1-cp274-reentry-hooks 2d41b5e1ea) and runtime-js/src/run-loop.ts:10702 (as on branch reliability-w1-cp274-reentry-hooks 2d41b5e1ea) (the step iterator and reenterChildPhase both go through it); conformance: fixed. W1's slot w1-cp274-slot-20261105-091310 at 2d41b5e1ea on origin/main 08f2daa182, read by the owner from its result files: companion.reentered-call-hooks-host passes every id on cf-do (host), 3 of 3 identical. The red, m-reenter (the pre-fix rule: the re-entry passes no row hooks; its marker asserted in the dist), fails exactly .on-message-once and .after-tool-once 3 of 3, with .ordinary-call-hooks and the setup ids green; m-twice (the non-vacuity of once) fails exactly .ordinary-call-hooks and .after-tool-once 3 of 3. The whole cf-do shard at the head: 446 rows, 312 ran, every verdict pass or expected-fail. js-chat-host is a declared gap (no evict or wake); related: limits. In-process runtime-js is held by unit tests only (runtime-js/src/__tests__/child-phase.test.ts, 5 red under m-reenter), since no runtime-js cell can evict. m-after-phase (hooks only after the whole phase) leaves the cell green, because its phases carry one call each, so the per-row order is held by unit tests only. Hooks fired after a commit are at most once per row: a crash between a commit and its hooks loses them, with no durable marker (orchestrator ruling); in-process afterTool stays at least once. The table is in docs/internals/hooks-system.md; severity: a user hook is silently skipped after a crash; the row and the result the model sees are correct | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-275 | S2 | A run that fails after its delegation commit (G1 / CP-216), its step commit refused, cannot be retried without a message. On runtime-js and the CF DO the failure pairs the delegated call unpaired_on_continuation, which is false (the child may have run, and finished), the finished child's real result is lost, and a no-message retry() is refused. On Temporal and CF Workflows the child's real row is kept and only the no-message retry() is refused ('No user message found to retry with'). A retry with a message recovers the session; the billed result stays lost. | runtime-js/src/run-loop.ts:6159 (origin/main 08f2daa182) (onFailed pairs every call the run still owes unpaired_on_continuation, a delegated one included; the CF DO runs this loop); runtime-cloudflare/src/steps.ts:4370 (origin/main 08f2daa182) (pairOwedCallsOnFailure, the same reason); core/src/store/retry-plan.ts:374 (origin/main 08f2daa182) (a failure pairing commit becomes the retry boundary) and :743 (origin/main 08f2daa182) (the no-input case applies only when the kept log ends in the delegation row itself), so the no-message retry is refused at :901 (origin/main 08f2daa182); related: reachable with no fault, by reading (WA): a step commit over the store's size limit (MA-31) fails the run typed on runtime-js, the DO and CF Workflows; conformance: subagent.delegated-commit-failed-retry-host (cf-do, host; a stepCommitFailedOnce fault refuses the parent's step commit after its delegation commit, the remote child held, then finished): with both product commits reverted at !726's head cc8a725c03 on origin/main 08f2daa182 (WA drv-g5, read by the owner from its result files) it fails exactly .paired-honestly, .real-result, .retry-accepted and .turn-completes 3 of 3, while .child-held, .run-failed, .fault-served and .child-once hold. The same red at f8fa5024ce (drv-g2). js-chat-host is a declared gap; related: the other runtimes' shapes (WA, peer-reported). Temporal by run (a Temporal docker probe at 67f2515b4f): the log keeps the child's real row, and retry() with no message is refused. CF Workflows by test: its real-row case is red at the reverted fix. DBOS: a real failure pairs nothing and a no-message retry completes with the child's real result, child once; pinned at the head only, with no red, so its pre-fix behaviour is not shown; related: the workaround before the fix: retry() with a message completes (rows user, the delegation row, tool(unpaired_on_continuation), user, assistant); the false reason and the lost result stay; related: a regression of G1 (CP-216, 98fb3973d4): its D4 unit tests seeded failCommittedRun, which pairs nothing, so they never reached onFailed's pairing; core/src/orchestration/delegation.ts:60 (fix: CP-275, WA; as on branch reliability-cf-wa-g1-real-retry b370a6889a) (failurePairingReason: parent_step_lost, which says the tool may have run, when the trailing calls are delegated), used at runtime-js/src/run-loop.ts:6181 (as on branch reliability-cf-wa-g1-real-retry b370a6889a) and runtime-cloudflare/src/steps.ts:4374 (as on branch reliability-cf-wa-g1-real-retry b370a6889a); core/src/store/retry-plan.ts:270 and :414 (fix: CP-275, WA; as on branch reliability-cf-wa-g1-real-retry b370a6889a) (a failure pairing commit is no boundary for a retry with no input, which re-attaches the children itself) and :287 and :783 (as on branch reliability-cf-wa-g1-real-retry b370a6889a) (endsInDelegatedStep: an append boundary ending in the delegation row and its calls' rows needs no input; a step commit past the row keeps the needs-a-message rule, its own red: retry-plan.test.ts 1 of 94 with it reverted, drv-g5); conformance: fixed. subagent.delegated-commit-failed-retry-host passes every id on cf-do (host) 3 of 3 at 99faba431f (WA drv-g3, read by the owner); the product is unchanged since apart from the endsInDelegatedStep guard (core retry planning only). Unit reds at the reverted fix, peer-reported: core retry 3, runtime-js 4, CF Workflows 3 (one the real-row case), Temporal 1; related: limit, by reading and untested: on the size route a no-message retry re-attaches, re-folds and is refused state_size_exceeded again each time; a retry with a message escapes (it targets the pairing's checkpoint, whose state predates the fold). The unbounded fold is MA-31's concern; related: the reason becomes outcome_unknown when Save-First Tools MR 1 lands (its spec §5.6, D9; WA); RM-91 (the failing run pairs what it owes) and RM-116 (the re-attach path, which keeps parent_step_lost until that rename); severity: a model-visible row is false and a finished child's billed result is lost, and the obvious recovery is refused; a retry with a message recovers the session | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-276 | S3 | A step whose delegation commit landed (G1 / CP-216), then stopped or failed, gets step_discarded although its reply stands in the log. Core discardUncommittedSteps counts only step checkpoints as committed, so the delegated step reads uncommitted (runtime-js, the CF DO; Temporal and DBOS by reading); CF Workflows' discardStagedChanges writes the marker unconditionally on its failure path. A client that honours the marker hides a reply the model and the snapshot still hold. | core/src/run/discard-uncommitted-steps.ts:85 (origin/main 08f2daa182) (the run's committed steps are its step checkpoints only); runtime-cloudflare/src/steps.ts:10717 and :10730 (origin/main 08f2daa182) (discardStagedChanges writes step_discarded whenever it reads the session; a failure after a delegation commit with no later result picks discard. A stop mid-step commits the step instead, so that route is clean); conformance: subagent.stop-after-delegation-keeps-step-host (cf-do, host; the parent stopped after its delegation commit, the remote child held, scoped to the parent's run): with both product commits reverted at !726's head cc8a725c03 on origin/main 08f2daa182 (WA drv-g5, read by the owner from its result files) it fails exactly .no-step-discarded 3 of 3, so the marker reaches the parent's chat SSE; .child-held, .reply-committed, .run-stopped, .stream-ended and .reply-stands hold. The same red at f8fa5024ce (drv-g2). js-chat-host is a declared gap; related: a regression of G1 (CP-216, 98fb3973d4): before G1 a remote step's reply committed with the step; core/src/run/discard-uncommitted-steps.ts:101-102 (fix: CP-276, WA; as on branch reliability-cf-wa-g1-real-retry b370a6889a) (a delegation commit counts as its step committed) and runtime-cloudflare/src/steps.ts:10764 (as on branch reliability-cf-wa-g1-real-retry b370a6889a) (discardStagedChanges writes no marker once the step's reply committed); conformance: fixed on runtime-js, the CF DO and CF Workflows. subagent.stop-after-delegation-keeps-step-host passes every id on cf-do (host) 3 of 3 at 99faba431f (WA drv-g3, read by the owner). Unit reds at the reverted fix, peer-reported: core discard 1, and CF Workflows through its real catch-all; related: open on Temporal and DBOS (WA's probe at cc8a725c03 on a private Redis, peer-reported). Core matches a delegation commit by its stepCount: on DBOS, by a real run, the delegation checkpoint carries the previous step's count (stream step 1, checkpoint append:0:2), so the step is missed, by reading; no step_discarded was written in that run, whose mock streams no text. Temporal is untraced (the probe timed out waiting for interrupted). A second DBOS shape, a unit case only: the cancelled-owner settle (runtime-dbos/src/stop-owner-probe.ts:169 (origin/main 08f2daa182), runCommittedStep, counts step checkpoints only) keeps a second flagged copy of a reply already in the log (replies 2, flagged 1); S2 if a real run shows the two copies in the model's history; severity: an informational marker contradicts committed content; nothing the model sees changes | source, live-repro, peer-reported | partly-inferred | open |
| CP-277 | S2 | A stop's abort reason reaches tools as a plain object, so a call it cuts shows unreadable live error text, and on the CF DO a remote sub-agent call gets two terminal frames that disagree with its stored row. A tool whose awaited operation is aborted rethrows the reason: workerd surfaces it as #<Object>, Node as the raw StopRequest, and the call's live tool_end carries that text. A remote call on the DO also writes its own failed tool_end before the stop pairing writes the run_stopped one; the UI transformer keeps the first, so the live stream shows #<Object> while a reload shows the stopped pairing. runtime-js, the CF DO and CF Workflows; Temporal and DBOS hand tools no StopRequest. | runtime-js/src/execution/remote-subagent-executor.ts:1469 (origin/main 5f2aaba172) (the remote call's catch writes its own failed tool_end with the caught error's text) and runtime-js/src/stop-pairing.ts:97 (origin/main 5f2aaba172) (the stop pairing dedupes only against its own announced set, so it writes a second, aborted tool_end for the same call); ai-sdk/src/transformer/stream-transformer.ts:1063 (origin/main 5f2aaba172) (a second tool_end for a call is dropped, so the client keeps the first frame); runtime-js/src/execution/tool-executor.ts:880-881 (origin/main 5f2aaba172) (a regular tool's failed tool_end takes extractErrorMessage of the thrown value, here the abort reason); related: reported by an app (peer-reported, 2026-11-06, on @helix-agents 9.1.0 / runtime-cloudflare 19.0.1): a remote sub-agent call cut by a stop from another tab, or by an interrupting keyed send, streams tool-output-error with errorText #<Object>, then data-run-interrupted; a reload shows the typed run_stopped pairing. Their workerd probe is on their own branch; related: WE-2's workerd probe v3 (peer-reported, 2026-11-06, Miniflare 5.20260925.0-alpha): a service-binding fetch, a fetched body's read and scheduler.wait each rethrow an Error whose message is #<Object> for a plain StopRequest reason, and the reason's fields are lost; an Error reason reads as its message. The root of FU-181-REFUSED-STEP-DOUBLE-TOOL-END's DO remote-call sighting; related: no step marker for the cut step is correct under #181: an interrupt keeps the cut reply as an assistant row; severity: the live view is unreadable and disagrees with the stored row; a reload shows the right state and nothing is lost; core/src/run/stop-request.ts:38 (fix: CP-277, WE-2; as on branch reliability-we2-cp277 481647df18) (StopAbortError: a stop aborts with an Error named StopAbortError carrying the stop) and :48 (as on branch reliability-we2-cp277 481647df18) (stopOfAbortReason reads it back), and runtime-js/src/execution/remote-subagent-executor.ts:1469 (as on branch reliability-we2-cp277 481647df18) (a remote call whose wait the stop cut writes no frame and no row of its own: the stop pairing is the one writer; a child whose end arrived first keeps its real result); conformance: WE-2's slot w-cp277b (red a15eb2e228, head 481647df18 on origin/main 5f2aaba172), read by the owner from its result files, 3 of 3 each phase. Red: subagent.stop-cut-call-live-text-host on cf-do fails .live-matches-stored and .live-text-readable, subagent.stop-cut-call-one-terminal-frame-host on cf-do fails .one-terminal-frame, and tool.stop-cut-native-wait-host fails .live-text-readable on cf-do (#<Object>) and js-chat-host (the raw StopRequest). Head: all of those pass. With only the executor change reverted, the remote cf-do cells fail .live-matches-stored and .one-terminal-frame and the native-wait cell passes; with only the StopAbortError reason reverted, the native-wait cell fails .live-text-readable on both hosts and the remote cells pass; related: fixed on runtime-js (js-chat-host's native-wait cell) and the CF DO by cells. CF Workflows is fixed in code and its regular tools' live text has no witness: no host lane, and the executor tier reads stored chunks, which an interrupted step's cleanup removes (FU-CONF-EXECUTOR-LIVE-TOOL-TEXT); runtime-cloudflare's steps.test.ts pins the reason's shape at unit level. A stopped in-process sub-agent call on runtime-js streams no tool frame at all: CP-280 | source, live-repro, peer-reported | verified | open |
| CP-278 | S2 | A run-start that was REFUSED, or failed transiently, is treated as a lost claim (another entry won), so the entry is dropped or answered busy. Core isLostClaimError counts every RunStartRejectedError but one as lost, which disagrees with core's own entry-failure classes for six causes (idempotency_mismatch, consumed_pending_changed, request_key_taken, sub_session_ref_conflict, stale_parent_ref_generation, child_binding_mismatch). runtime-js's client-tool deadline continuation drops the turn at info, its companion self-wake answers deferred while nothing continues, its persistent-child resume and core's companion start outcome answer child_busy instead of child_entry_failed, and DBOS's companion delivery reason answers child_busy for every lost-claim record. | core/src/errors/agent-errors.ts:1493 (origin/main 5f2aaba172) (isLostClaimError: every RunStartRejectedError except resumed_companion_ref_missing) against core/src/run/classify-entry-failure.ts:91 (origin/main 5f2aaba172) (classifyEntryFailure, whose cause table classes those six causes refused or transient); runtime-js/src/js-agent-executor.ts:1413-1416 (origin/main 5f2aaba172) (the deadline continuation's catch: any RunStartRejectedError reads as another entry winning; it drops the owed continuation, logged at info); runtime-js/src/js-agent-executor.ts:6241 (origin/main 5f2aaba172) (the companion self-wake treats any lost-claim error as stored-and-drained) and runtime-js/src/run-loop.ts:9380 (origin/main 5f2aaba172) (the persistent-child resume answers child_busy); core/src/orchestration/companion-start-outcome.ts:122 (origin/main 5f2aaba172) (child_busy for any lost-claim error but idempotency_mismatch) and runtime-dbos/src/steps/execute-companion-tool.ts:1649 (origin/main 5f2aaba172) (lostClaimDeliveryReason: child_busy for every lost-claim record but resumed_companion_ref_terminated); related: found by W1 (peer-reported, 2026-11-06) tracing CP-252: child_binding_mismatch reaches the deadline catch by run (CP-252's probe), the other causes by reading. CF Workflows' and DBOS's lostClaimRecordOf record any rejection with its cause and surface it typed (by reading). FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW's one-definition bullet predicted this drift; core/src/run/reject-cause-class.ts:15 and :64 (fix: CP-278, W1; as on branch reliability-w1-cp278-lost-claim-class 8bc64eef9f) (one cause table, REJECT_CAUSE_CLASS, which classifyEntryFailure now reads too) with core/src/errors/agent-errors.ts:1498 (as on branch reliability-w1-cp278-lost-claim-class 8bc64eef9f) (isLostClaimError: a rejection is a lost claim only when its cause is classed lost); runtime-js/src/js-agent-executor.ts:1435-1470 (as on branch reliability-w1-cp278-lost-claim-class 8bc64eef9f) (the deadline continuation acts on the class: a lost entry is dropped; a refused or fatal one ends the owed run failed, typed, at once; a transient one retries in its budget, then ends it failed) and core/src/orchestration/companion-start-outcome.ts:126 with runtime-dbos/src/steps/execute-companion-tool.ts:1651 (as on branch reliability-w1-cp278-lost-claim-class 8bc64eef9f) (a refused, transient or fatal cause is child_entry_failed with its detail, never child_busy); runtime-js/src/run-loop.ts:9388 and :10484 (same branch and pin) (a persistent child resume answers the same); conformance: unit-level only (no conformance control injects a run-start rejection with a given cause on the js lanes or DBOS: FU-CONF-EXECUTOR-RUN-START-REJECT-CONTROL). W1's slot w1-cp278-slot-20261106-080841, read by the owner, MISMATCHES 0: at the red head 00f5976599 the seven files of provenByException fail as predicted (core 9 and 4, runtime-js 3, 1, 2 and 1, DBOS 3) and pass at the fix 8bc64eef9f; core 6112, runtime-js 1963 and DBOS 1205 units pass in full, and eleven runtime-js integ files on Redis and Postgres. The companion cells keep their ledgers on js-memory, cf-do, cfw-workflows-d1, temporal-memory and dbos-postgres. Mutations: the old isLostClaimError restored fails only the core table test (runtime-js loads core's built dist, so its sites cannot see a core-source mutation); the deadline catch's class switch removed fails the refused and fatal site A cases; related: two cases remain, named by the orchestrator, not scheduled. (1) An EXPIRY whose wake retries run out (not the continuation) still leaves the run paused, with the client_tool.deadline_gave_up error log; the next resume drains it (runtime-js/src/js-agent-executor.ts:1299-1301 at the fix, same branch and pin, says so). (2) A refused companion self-wake now rejects typed to the submitter, but the answer stays stored on the companion's pending call and nothing continues the companion: no later wake, and a re-submit answers already_completed; related: FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW's one-definition bullet is narrowed: core holds the one table; CF Workflows' and DBOS's lostClaimRecordOf lists are still written by hand; severity: a silent stall, or a wrong child_busy the model acts on; nothing committed is lost | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 7 tests) |
| CP-279 | S1 | runtime-js and CF Workflows: a BLOCKING persistent companion whose remote child pauses on an approval outlives its root's turn with no way back. Once the companion's in-process parent and the root's turn complete, the root's terminal commit clears its remotePendingCalls, so the approval the user may already see cannot be answered through the root, and nothing ever continues the companion; a later root turn does not recover it. The companion's ref under its parent reads completed (runtime-js) or failed (CF Workflows) while its session is paused, against the blocking-spawn contract (paused_awaiting_client, never terminal). | conformance: companion.paused-outlives-root-turn (executor tier; a root, an ephemeral in-process mid, a blocking companion buddy, and buddy's remote child paused on an approval; mid's next model call held, then released): on js-memory, js-redis, js-postgres and cfw-workflows-d1 it fails exactly .ref-not-terminal, .answerable-after-root-turn and .next-turn-recovers, 3 of 3 identical, at a5919fc8e0 (CP-245's fix plus the cells; W1's slot w1-cp279-slot-20261106-063944, read by the owner from its result files). The answers fail judged, with the pending ids read on root, mid and buddy all empty. companion.paused-answerable-while-mid-runs (the same tree, answered while mid is held) passes 3 of 3 on every lane: that half is CP-245's; related: a companion suspended_awaiting_children on a remote child is deliberately not classed as paused on either runtime: runtime-js defers it to FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN and CF Workflows to its Q3; this cell is the first to reach it. W1's fix design (orchestrator-approved) classes a remote-parked companion paused, keeps a terminal's remote entries whose parked parent is still suspended (narrowing CP-157's premise), and continues the parked parent on a later answer; related: the CF DO, by reading (W1): a companion whose tree is declared local is refused at spawn by C2 (materialize step 2c walks the tree before the remote rewrite), a known refusal; a companion declaring a genuinely remote sub-agent tool (createRemoteSubAgentTool) is not refused, since the walk treats remote tools as opaque, so the DO's exposure through that shape is unverified: no host-tier producer for a genuinely remote child (FU-CONF-CF-DO-GENUINELY-REMOTE-CHILD-PRODUCER). Temporal and DBOS: declared gaps; related: CP-245 (fixed: the call is answerable through the root while the middle child runs); CP-187 and CP-233 (other paused-child surfaces); core/src/orchestration/remote-pause.ts:149-160 and :186-190 (fix: CP-279, W1; as on branch reliability-w1-cp279-companion-outlives-root a80d04453c) (F2: remotePendingCallsAfterTerminal, the one rule every terminal site calls: a terminal keeps the parked calls of a descendant whose run is still suspended, and drops its own; F1: remoteParkedCallSummaries, the calls a remote child parked for a parent, read from the park root), called by core/src/run/fail-committed-run.ts:271, core/src/run/fail-session-without-run.ts:98 and core/src/run/commit-stop-terminal.ts:215 (as on branch reliability-w1-cp279-companion-outlives-root a80d04453c); runtime-js/src/run-loop.ts:9237 and :9349 (as on branch reliability-w1-cp279-companion-outlives-root a80d04453c) (F1: a companion parked on a remote child's calls is paused on them, answered suspended, its ref paused_awaiting_client), :6524 and :6697 (F2, the terminal sites) and :6807-6817 (as on branch reliability-w1-cp279-companion-outlives-root a80d04453c) (F3: a companion's self-wake under an ended root consumes the call off the root's parked calls, fenced idle), with core/src/client-tool/route-submit.ts:930 and :1065 and core/src/orchestration/parked-entry-current.ts:98 (as on branch reliability-w1-cp279-companion-outlives-root a80d04453c) (F3: an answer through an ended root continues the parked companion); runtime-cloudflare/src/companion-ref-status.ts:85-90, runtime-cloudflare/src/steps.ts:1121, :7028, :8400 and :8507, runtime-cloudflare/src/executor.ts:4762 and runtime-cloudflare/src/workflow.ts:1117 (as on branch reliability-w1-cp279-companion-outlives-root a80d04453c) (the same three parts on CF Workflows: the suspended_awaiting_children event of a remote-parked companion is a pause on its calls, the terminal keeps a suspended descendant's park, and the companion's continuation is owed and started); conformance: fixed. The split cells: companion.paused-outlives-root-turn (.mid-held, .root-turn-completes, .ref-not-terminal, .answerable-after-root-turn, .tree-clean-after-answer) and its sibling companion.paused-recovered-by-next-turn (.mid-held, .root-turn-completes, .second-turn-settles, .park-on-root, .answerable-after-next-turn, .companion-continues). Red at rev 6's product dcbb603097 (W1's slot w1-cp279r6-slot-20261106-110522, read by the owner): on js-memory, js-redis, js-postgres and cfw-workflows-d1, outlives fails exactly .answerable-after-root-turn, .ref-not-terminal and .tree-clean-after-answer and recovered fails exactly .answerable-after-next-turn, .companion-continues and .park-on-root, 3 of 3 identical. At rev 8's fix bc046fe488 (slot w1-cp279r8-slot-20261106-172225, read by the owner, MISMATCHES 0) both, and companion.paused-answerable-while-mid-runs (CP-245's), pass every id 3 of 3 on the four lanes, .tree-clean-after-answer included (rev 6's race is gone: the consumed park leaves the ended root, and no plain save races a live root). Revs 7 and 8 were red first as unit reds (a consumed park on the ended root, a companion moving a live root, a remote child that ended before the companion re-enters); the twelve runtime-js integ files pass; related: mutations at the fix (same slot): the CF Workflows half of F2 reverted reds cfw-workflows-d1's outlives .answerable-after-root-turn and .tree-clean-after-answer; CF Workflows' owed gate for a companion reverted reds its recovered .companion-continues; runtime-js's live-root skip removed reds its unit. Each reds exactly its own target; related: still open, by reading: the stores' status CAS off live clears remotePendingCalls whole, per CP-157's contract, so a status-only writer (agent-server cleanup, runtime-js's active-to-failed CAS, DBOS's interrupt settle, core's checkpoint ops) would drop a still-paused descendant's park; not shown (FU-STORE-TERMINAL-CAS-DROPS-DESCENDANT-PARK; a run that shows it reopens CP-279 or takes its own S1 id). CP-282 (the route-submit plain save racing a live root's commit) is a separate record, fixed with the field-scoped op; severity S1: an approval the user can see has no route to an answer and the companion's work never continues; nothing committed is lost, and the root's turn answered | source, live-repro, peer-reported | verified | fixed (8 cells) |
| CP-280 | S2 | runtime-js, Temporal and DBOS: an in-process sub-agent call reaches the chat client with no tool part. The runtime wrote no tool_start for the call (only subagent_start, subagent_end and its tool_end), so ai-sdk's stream transformer, which owns no input part for it, sends the call's tool_end as data-tool-output-elsewhere: every completed local sub-agent call gives the client a spurious re-anchor trigger with a false reason and no tool-output part, while a reload renders the stored call; a call cut by a stop is left unfinished live, and on the CF DO a remote sub-agent call was announced after the step's server calls, out of call order. | conformance: subagent.stop-cut-call-live-text-host on js-chat-host (runtime-js's chat host) fails .cut ('the call's live frames: []'), and with it .live-matches-stored and .live-text-readable, in every phase of WE-2's slot w-cp277b (red a15eb2e228 and head 481647df18 on origin/main 5f2aaba172, and both mutation trees): 12 of 12 identical, read by the owner from its result files. The same cell passes on cf-do at the head (CP-277's evidence). Ledgered below; related: found running CP-277's cells (orchestrator ruling, 2026-11-06: a product behaviour, not a missing capability, so not a declared gap). Whether runtime-js streams any tool frame for an in-process sub-agent call that is not stopped decides whether the gap is only the missing end on a stop; WE-2 probes it; related: CP-187 (a local sub-agent's subagent_end written while it is still suspended: the reverse surface) and CP-277 (the remote call on the DO); severity S2 (owner, 2026-11-06, raised from S3): every completed local sub-agent call gives a stock client a spurious re-anchor trigger and no tool part, and the live view disagrees with a reload; nothing stored is lost; ai-sdk/src/transformer/stream-transformer.ts:1087 (origin/main 3dda2d7f2e) (a tool_end for a call the transformer owns no input part for goes out as data-tool-output-elsewhere), with core/src/orchestration/step-iterator.ts at origin/main 3dda2d7f2e starting only server and child-phase calls in its ordered start loop, so a sub-agent call got no tool_start there; related: rescoped (owner, 2026-11-06) from WE-2's probes w-pjs2 and w-pjs3 (peer-reported, read by the owner): the cause is the missing tool_start, not frames lost before the wire; a NORMAL completion has the same shape (data-subagent-start, data-subagent-end, then data-tool-output-elsewhere), so the record moves from S3 to S2 and is no longer stop-only. DI-48 owns the foreign-output routing, which stays as it is for a call announced in an earlier response; core/src/orchestration/sub-agent-announcement.ts:21 and core/src/orchestration/step-iterator.ts:2348-2357 (fix: CP-280, WE-2; as on branch reliability-we2-cp280 3e24192ad7) (a sub-agent call is announced like any server call, its tool_start written in the ordered start loop at its slot in call order), runtime-js/src/execution/remote-subagent-executor.ts:229 (as on branch reliability-we2-cp280 3e24192ad7) (the remote executor skips its own announcement once the iterator made it) and runtime-js/src/run-loop.ts:5520-5529 (as on branch reliability-we2-cp280 3e24192ad7) (an in-process call cut by a stop writes no terminal frame of its own: the stop pairing is the one writer, CP-277's rule); conformance: fixed on runtime-js and the CF DO. WE-2's slot w-cp280 (red 216f306ca9, head 3e24192ad7 on origin/main 3dda2d7f2e), read by the owner from its result files, 3 of 3 identical each phase. Red: subagent.local-call-tool-part-host on js-chat-host fails .tool-input, .tool-output, .no-elsewhere, .live-equals-reload and .call-order, and on cf-do .call-order; subagent.local-call-tool-part-client on js-chat-host fails .no-resync; subagent.local-call-announced on js-memory fails .one-tool-start and .start-before-end; subagent.stop-cut-call-live-text-host on js-chat-host fails as ledgered. Head: all of them pass every id, subagent.local-call-one-start-host on cf-do and subagent.local-call-announced on cfw-workflows-d1 included. Mutations: the remote executor ignoring the iterator's announcement fails only subagent.local-call-one-start-host on cf-do (two tool_start); the stop-cut rule dropped fails only the js-chat-host stop-cut cell (.live-matches-stored). The remote and sub-agent host, parked-tree client and js-memory sub-agent cells are identical at red and head; related: owed on Temporal and DBOS (a sibling MR, orchestrator ruling): subagent.local-call-announced fails .one-tool-start and .start-before-end at the head on the Temporal lanes and dbos-postgres (one tool_end, no tool_start); ledgered below. Temporal's workflow.ts:623-626 comment claims a dispatch tool_start that no code writes. DBOS's remote path is not yet read; runtime-temporal/src/workflow-dispatch.ts:262 and :1374 (fix: CP-280b, WE-2; as on branch reliability-we2-cp280b 74df351c93) (dispatchSubAgents and dispatchRemoteSubAgents write the call's tool_start when ctx.announce), with runtime-temporal/src/workflow.ts:1477 and :1505 (as on branch reliability-we2-cp280b 74df351c93) (announce: wf.patched(SUBAGENT_TOOL_START_PATCH), runtime-temporal/src/subagent-tool-start-patch.ts:8, the patch helix-cp280-subagent-tool-start-v1, so a pre-patch history replays unchanged); workflow.ts:623-627 (as on branch reliability-we2-cp280b 74df351c93) now states the dispatch tool_start for a patched execution; runtime-dbos/src/steps/execute-sub-agent.ts:644 (as on branch reliability-we2-cp280b 74df351c93) (the local begin step writes the call's tool_start before its subagent_start) and runtime-dbos/src/steps/execute-remote-subagent.ts:432-443 and :274-312 (as on branch reliability-we2-cp280b 74df351c93) (DBOS's remote path, which wrote no frame of its own for the parent's call: it now announces the call once, a re-attach announcing nothing again, and ends it with subagent_end and one tool_end, none for a call the run's stop cut, CP-277's rule); conformance: fixed on Temporal and DBOS. WE-2's slot w-cp280b (red 75d04629ee, head 74df351c93 on reliability-we2-cp280 39a6d275f2), read by the owner from its result files: subagent.local-call-announced on temporal-memory, temporal-redis, temporal-postgres and dbos-postgres reads expected-fail on exactly .one-tool-start and .start-before-end 3 of 3 at the red head, and unexpected-pass 3 of 3 at the head (the flip); the 36 regression cells are identical at red and head; the Temporal and DBOS unit suites (977, 1206) and integ files pass, the replay integ (subagent-tool-start-replay, the pre-patch local-subagent history recorded at 75d04629ee) included. Mutations: the Temporal announce off (notmp) reds the three Temporal lanes again 3 of 3 and the Temporal local and remote units, dbos-postgres staying green; DBOS's announce dropped (nodbos, slot w-cp280c) reds dbos-postgres 3 of 3 and the DBOS local unit, the Temporal lanes green; the remote stop-cut rule dropped (nocut) reds exactly the DBOS remote unit's stop-cut case; related: the remote paths are unit-level: no Temporal or DBOS lane claims remote-child (FU-CONF-TEMPORAL-REMOTE-PRODUCER, FU-CONF-DBOS-REMOTE-PRODUCER). DBOS remote: runtime-dbos/src/__tests__/subagent-call-announced.test.ts:150, :164, :180 and :190 (as on branch reliability-we2-cp280b 74df351c93) (a completed call's own frames exactly tool_start, subagent_start, subagent_end, tool_end; a re-attach announces nothing again; a failed call ends with one tool_end carrying the error; a stop-cut call writes none) fail with the local case at :52, 5 of 5, at the red commit 75d04629ee's product (WE-2's lock-free run w-lftest-1106-145753-24978, peer-reported commit, read by the owner: its failures show no tool_start written) and pass at the fix 8617ae01a7 (w-lftest-1106-145507-97010) and in the head's DBOS unit suite. Temporal remote: the notmp mutation reds its remote unit; related: a remaining case, named by WE-2: a DBOS remote step that throws past its retries (runtime-dbos/src/steps/execute-remote-subagent.ts:278, as on branch reliability-we2-cp280b 74df351c93: the throw leaves runRemoteSubAgentCall before any frame) becomes a failed call in the workflow's catch, outside the step, so the call gets no subagent_end or tool_end live, as before; the stored failed row is right on reload (FU-DBOS-REMOTE-STEP-THROW-NO-TOOL-END); related: Save-First Tools MR 1 (!754) after merging this record's fix: an in-process call cut by a stop now carries the same typed cutByStop as the remote executor's, so a stopped saved step drops its row and the stop pairing writes its one tool_end (runtime-js/src/run-loop.ts:5666, runSubAgentCallAs, as on branch reliability-cf-wa-sft-mr1 f46938af8b); without it subagent.stop-cut-call-live-text-host on js-chat-host failed .cut at the merge head. It passes 3 of 3 at f46938af8b on cf-do and js-chat-host (WA's slot slot-m2). An internal fix of the MR, so no id | source, live-repro, peer-reported | verified | fixed (9 cells) |
| CP-281 | S1 | runtime-js: a stop made from another process never reached the run's in-process owner while a tool or a model call ran. The stop was recorded and answered 202 stopping, but the owner read stop records only at loop boundaries, and its lease renewal read none, so the tool's or the call's abort signal never fired, even for a tool that listens, and the session stayed active until the work returned. agent-server documents Stop as working from any process with no sticky routing, so behind a load balancer most stops land on another instance. | runtime-js/src/execution/in-process-runs.ts:11-16 (origin/main 3dda2d7f2e) (a foreign owner is answered stopping, and the signal is a no-op for it), with runtime-js/src/run-loop.ts:1897, :1988 and :2469 (origin/main 3dda2d7f2e) (pollStop runs only at loop boundaries) and core/src/run/run-lease.ts:182 (origin/main 3dda2d7f2e) (the lease renews without reading the stop); related: the docs overstated it: docs/guide/interrupt-resume.md said an executing owner is signalled, and that Stop works from any process with no sticky routing (around lines 305 and 313 at origin/main 3dda2d7f2e); the fix corrects the first; related: found by WE-2's probe w-pjs3 (peer-reported, 2026-11-06): a second JSAgentExecutor's stop() answered stopping, the running tool's abort signal never fired, the session stayed active for 120 s, and the owner made only renewRunLease calls (24), no stop read. Scope by W1's reading: runtime-js and agent-server only; CF Workflows already polls its own stop (CP-16's watchOwnStop); Temporal and DBOS stop through their engine (a workflow signal, cancelWorkflow), so a record-only stop is no product route there, and their aborts are CP-62's (a running tool) and CP-16's (a model call); runtime-js/src/js-agent-executor.ts:1497 and :1513-1519 (fix: CP-281, W1; as on branch reliability-w1-cp281-cross-process-stop 064e4315e9) (every lease renewal the store accepted reads the run's stop record and aborts the run's signal with it) and runtime-js/src/run-start.ts:971 and :982 (as on branch reliability-w1-cp281-cross-process-stop 064e4315e9) (startRunLease calls onRenewed after an accepted renewal, never as part of its outcome); conformance: stop.record-only-mid-tool (a tool listening to its abort signal, held by a tool gate; a stop recorded by another process through the driver's request-stop) and stop.record-only-mid-llm-call (the same, during a held model call), executor tier, read by the owner from W1's result files. Red at 68d5a21782 (product identical to origin/main 3dda2d7f2e), 3 of 3 identical in slot w1-cp281-red3-20261106-125215 and once more in each of w1-cp281-slot-20261106-104434 (at de11f4267a, a one-line test-pin difference) and -111229: on js-memory, js-redis and js-postgres the tool cell fails exactly .run-interrupted and .tool-signal-aborted and the call cell exactly .call-aborted and .run-interrupted; .tool-held / .call-held and .stop-recorded hold. At the fix 064e4315e9 both pass every id 3 of 3 on the three js lanes and cfw-workflows-d1. cfw-workflows-d1 passes at the red head too: the contrast; related: mutations at the fix (w1-cp281-phase3-20261106-121212, read by the owner): the renewal's stop check made inert reds the unit and the js-memory tool cell again; the callback never invoked, and only an abort signalled, each red the unit as predicted. Temporal and DBOS are declared gaps on the capability record-only-stop-reaches-owner, citing CP-62 and CP-16; related: FU-STOP-DEADLINE-NODE (no stop deadline off the DO, for a call that IGNORES its signal) is a separate, narrower gap and stays open; severity S1: a user's Stop had no effect for the length of a tool or a model call when it reached another instance, which a documented any-process Stop makes the normal case behind a load balancer | source, live-repro, peer-reported | verified | fixed (6 cells) |
| CP-282 | S2 | runtime-js (and the CF Durable Object that runs it) and CF Workflows: an approval answered through the root while the root's own turn runs can fail that turn. The router stamps the root's parked entry with a plain saveState that moves the root's version; when it lands after the root's last version read and before its version-pinned commit, the commit is refused and the user's turn fails state_concurrency_conflict, though the answer itself was stored and forwarded. Temporal and DBOS never park a remote call on a root. | core/src/client-tool/route-submit.ts:830-850 (origin/main 234725409d) (markParkedCallAnswered: loadState of the root, then a plain saveState of remotePendingCalls with answeredAt, under the live root run fence, which moves the root version) with runtime-js/src/run-loop.ts:5891 (origin/main 234725409d) (the step commit pins expectedVersion: state.version) and core/src/orchestration/step-iterator.ts:2949-2965 (origin/main 234725409d) (after a step that ran sub-agents the iterator refreshes the version, so the window is from that read to the commit); related: W1's unit reds, slot w1-cp282red-slot-20261106-231133, read by the owner: the runtime-js A unit (the answer lands after the refresh and before the step commit) fails the turn 3 of 3, with the mark kept; the store contract's RPC-B2 (a pinned commit after an out-of-band write is refused) fails by behaviour 3 of 3 on memory, Redis, Postgres, D1 (Miniflare and the workerd pool) and DO-SQLite, beside the new op's ids red by absence. The two in-memory and d1-state files that showed every id not run name the contract only in comments (a file-selection error); related: a lost-mark half was predicted and is disproved: no runtime-js entry holds the parked map in memory. resumeSession's state comes from loadAgentState (runtime-js/src/js-agent-executor.ts:3119, origin/main 234725409d), a field-by-field projection that never carries remotePendingCalls, and execute() builds its state fresh, so the step commit re-reads the stored, marked map (runtime-js/src/run-loop.ts:5721, keepsRemote, origin/main 234725409d). W1's slot w1-cp282c2-slot-20261107-021036, read by the owner: the continue, step-partial and with_message plus abandonPending shapes all keep the mark 3 of 3 past their preconditions (the root paused, the answer accepted). They stay as guard units, and the contract's RPC-B1 stays as a pin of that behaviour, not as evidence of a defect; related: CF Workflows is in scope, against this record's first reading. Its commitStep re-reads and retries a stale version (runtime-cloudflare/src/steps.ts:9500, origin/main 234725409d, MAX_STALE_ATTEMPTS = 5), but the A cell fails its turn too ('State version mismatch … expected version 10, but store has version 11', state_concurrency_conflict, below), so the race is not absorbed. The follow-up planned for a retried model call (FU-CFW-RUN-AGENT-STEP-STALE-SAVE-RETRIES-MODEL-CALL) was folded into this record and never filed on main; related: Temporal and DBOS: not applicable. They never park a remote call on a root (their only writes of remotePendingCalls are failed-run clears, runtime-temporal/src/activities.ts:1011 and runtime-dbos/src/steps/run-commit.ts:1955 at 5ee15c8c24); a paused remote child fails the call there (CP-230), and a CP-230 fix brings them into scope; core/src/store/state-store.ts:1006 (fix: CP-282, W1; as on branch reliability-w1-cp282-on-main 52ef0bba6c) (updateRemotePendingCalls, the field-scoped op: no version move, no checkpoint, fenced in the same atomic step, and every full-state write keeps the stored map) with core/src/client-tool/route-submit.ts:839 (as on branch reliability-w1-cp282-on-main 52ef0bba6c) (markParkedCallAnswered writes through it), runtime-js/src/run-loop.ts:3821 and runtime-cloudflare/src/steps.ts:7066 and :9712 (as on branch reliability-w1-cp282-on-main 52ef0bba6c) (a nested parent, and the root's own suspension, recording parked calls onto the root), and the five stores: store-memory/src/in-memory-state.ts:1949, store-redis/src/redis-state.ts:5409, store-postgres/src/state/postgres-state-store.ts:3566, store-cloudflare/src/d1-state.ts:6532 and runtime-cloudflare/src/do/do-state-store.ts:5521 (as on branch reliability-w1-cp282-on-main 52ef0bba6c); conformance: fixed. companion.parked-answer-vs-root-commit (executor tier: CP-279's tree, the root's turn-2 model call held, the approval answered through the root, then released). Red at 9513ec119b (main 234725409d plus the red units, with the head's conformance files) on js-memory, js-redis, js-postgres and cfw-workflows-d1: exactly .turn-completes, 3 of 3 identical, the turn failed state_concurrency_conflict (non-retryable) with the mark stored. Green at 5a3e4f77cb, every id 3 of 3 on the four lanes (W1's slot w1-cp282cell7-slot-20261107-125616, read by the owner, MISMATCHES 0). 5a3e4f77cb..52ef0bba6c changes only packages/e2e conformance files (the slots ran on this branch before two rewrites of its history that changed no product code, only test-support files and comments); conformance: the guards. companion.parked-answer-vs-root-commit-mark-kept (js-memory, js-redis, js-postgres; buddy's continuation held at its run-start commit, the root's parked calls read after its turn) passes 3 of 3 at both heads (slot cell7). companion.parked-answer-while-mid-runs-kept (the four lanes) and companion.parked-answer-while-mid-runs-mark-kept (the js lanes) pass 3 of 3 at both heads, judged at 52ef0bba6c with .buddy-resolved (W1's slot w1-cp282cell8-slot-20261107-173309, read by the owner, MISMATCHES 0). The two -mark-kept scenarios' .mark-kept was judged with buddy's hold armed but never reached (all 18 runs of the second log 'armed true, continuation inside false'): with the root live nothing continues buddy (CP-287), so the read races nothing. CP-287's fix re-runs both with the hold engaged. The CF Workflows -mark-kept cells and the host tier are declared gaps (gaps.ts); related: the CF DO, by reading: it runs runtime-js, whose cells prove the step commit, and DO-SQLite runs the op's store contract (core/src/testing/remote-pending-calls-operations.ts, RPC-*); neither host lane has a remote-child producer (FU-CONF-CF-DO-GENUINELY-REMOTE-CHILD-PRODUCER); related: CP-157 (the parked map and its contract), CP-279 (a still-paused descendant keeps its park through terminal commits), CP-128 (the same pattern for the ownership map, fixed with a field-scoped writer), CP-287 (an answer through a live root never continues the companion, split out of these scenarios); conformance: the runtime-js proof moves to companion.parked-answer-vs-root-commit-owned (owner and orchestrator ruling, 2026-11-07, with CP-287's fix): once CP-287's fix continues buddy under a live root, buddy's continuation consumes the root's parked entry shortly after its run starts, so the A scenario's mark read races that consume on the js lanes (at a92bf589f8 it failed .answer-marked in 8 of 9 js runs; it is gapped there, never ledgered). The -owned sibling judges the answer owned (marked on the root, or consumed by buddy's committed run) instead. W1's slot w1-cp287b-slot-20261107-231718, read by the owner: at CP-282's red 9513ec119b it fails exactly .turn-completes 3 of 3 on js-memory, js-redis and js-postgres (owned as 'marked'), as A did; at CP-282's fix ee46c4575f and at CP-287's fix a92bf589f8 it passes 3 of 3. The CF Workflows cell stays on A, which passes there; severity S2: a user's turn fails because an approval for a descendant was answered at the wrong moment; the failure is typed (state_concurrency_conflict), nothing committed is lost, and the answer was delivered, but the user sees a failed turn | source, probe, live-repro, peer-reported | verified | fixed (10 cells) |
| CP-283 | S3 | D1: a late client-tool answer delivered by a plain append records the wrong row. The delivery's row index came from the append's pre-read message count, and the append's guard does not pin the checkpoint pointer, so another writer's rows landing between the pre-read and the batch leave the recorded index short by their count. A later truncate then fails to re-open the record (the late answer is lost after a rewind), or re-opens it when the truncate is above the notice. The other stores count inside their own atomic step. | store-cloudflare/src/d1-state.ts:1847 and :3236-3255 (origin/main 3dda2d7f2e) (guardedLateAnswerDeliveries binds lateAnswerRowIndex(view.messageCount, rowOffset) from the pre-read view) with :1798 (origin/main 3dda2d7f2e) (the append guard leaves the pointer unpinned) and :1612-1660 (origin/main 3dda2d7f2e) (a plain append or promote moves only the pointer, never version); related: not a defect, by the same reading: the append checkpoint's message_count is computed in SQL at execution time (guardedCheckpointInsert), so two appends from one pre-read record correct counts; contrast: store-redis/src/commit-script.ts:562 (origin/main 3dda2d7f2e) (Redis counts with LLEN inside its script); Postgres reads liveMessageCountInTx inside its transaction (store-postgres/src/state/postgres-state-store.ts:886-897, origin/main 3dda2d7f2e); the DO in its sync transaction (runtime-cloudflare/src/do/do-state-store.ts:2224, origin/main 3dda2d7f2e); memory synchronously; related: flagged unverified by WE-2's Additive Sessions MR 2 D1 work (peer-reported, 2026-11-07), traced by the owner; store-cloudflare/src/d1-state.ts:3236-3256 (fix: CP-283, WE-2; as on branch reliability-we2-cp283 bd7def2abb) (the delivered row index is computed IN the batch, after the commit rows are inserted: COUNT(*) of the session rows minus the commit row count plus the offset); conformance: unit-level only (the store contract, run by each store's own suite; no conformance store-tier scenario runs late-answer-operations). WE-2's slot w-cp283-1107-001101, read by the owner: at the red faef397c36 (tests only, on origin/main 3dda2d7f2e) LA-14a and LA-14b fail exactly, 2 failed of 587, 3 of 3 on both D1 runs (the Miniflare integ and the workerd pool); memory, Redis, Postgres and DO-SQLite pass. At the fix bd7def2abb all six contract files pass. The mutation (main's d1-state.ts restored) fails exactly LA-14a and LA-14b on the D1 integ run; the workerd-pool mutation run was cut by the slot cap (orchestrator: not repeated); severity S3: a narrow window (another writer beside a late-answer delivery) and a consequence only at a later rewind; nothing is lost without one | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-284 | S1 | A stock useChat app that auto-sends on ai's lastAssistantMessageIsCompleteWithApprovalResponses never sends a sub-agent's approval. LIVE, on every ai 6.x: the stream transformer marks the delegating call providerExecuted only through the opt-in findTool, which nothing shipped passes, and with no step boundary reported (llm-vercel, RM-48) the delegating call's unanswered part sits in the approval's step, so the predicate never holds. After a RELOAD, from ai 6.0.158: the snapshot puts the parent's delegating call in the same step as the child's approval with no step-start, and that version stopped skipping provider-executed parts. Either way the user's answer is never sent and the run stays paused, with nothing reported. | ai@6.0.170 with @ai-sdk/react@3.0.172 (the client pair Additive Sessions MR 1, !752, pinned in packages/e2e and packages/ai-sdk) against ai@6.0.49 with @ai-sdk/react@3.0.51 (origin/main e4ba7cd498's client pins): subagent.remote-child-approval-parks-parent-reload-client fails .approval-posted ('the first chat POST carrying the approval: null') and .turn-completes ('paused') on the newer pair; conformance: found by batch 14's pipeline 3039601298 (test:conformance 1/7, job 17829050206: the cell unexpected-fail on cf-do and js-chat-host, client tier, each at its 90 s bound) and isolated by WE-2's slot w-b14pin-1107-035515, read by the owner: at main e4ba7cd498 (ai 6.0.49 / @ai-sdk/react 3.0.51) and at batch 14's head with only the client devDeps set back to those versions, the cell passes on both hosts; at main + !752 alone and at batch 14's head (ai 6.0.170 / @ai-sdk/react 3.0.172) it fails on both, the same two ids, once each; contrast: client.stock-approval-predicate-after-reload (CP-182's provenBy: an approval of the root's own tool given after a reload) passes on ai 6.0.170 in the same batch, and subagent.remote-child-approval-parks-parent-client (no reload) passes on cf-do; so the server parks and answers the call, and what breaks is the client's send of the rehydrated remote child's approval; related: CP-182 (fixed: ai 6.0.0 to 6.0.91's own approval predicate dropping provider-executed parts, a different shape and version range); CP-157 and CP-186 (fixed, cite the failing cell, proven on the client tier's ai 6.0.49 only); related: not yet probed: the latest ai 6.x (6.0.324 at 2026-11-07); by WE-2's read the first affected version is 6.0.158. The client tier of the conformance suite runs only ai 6.0.49, older than any current app (FU-CONF-CLIENT-TIER-CURRENT-AI). Batch 14 keeps the client devDeps at 6.0.49 / 3.0.51 so the fixed records stay proven, and the fix moves them up; related: the examples ship this client pair (orchestrator ruling, 2026-11-07: examples show the current ai, so they are not reverted): approval-gates, client-tools, nextjs-redis, opennext-cloudflare-do and resumable-streams-nextjs pin @ai-sdk/react 3.0.172 and ai 6.0.170 in their package.json (as on branch reliability-we2-additive-mr1 2a06d55fa6, !752). The fix MR covers them; its verification runs those that use a remote child with an approval; related: the cause, traced by WE-2 (peer-reported, 2026-11-07, read by the owner from its cites): ai 6.0.158 dropped the predicate's .filter((part) => !part.providerExecuted) (ai@6.0.49 dist/index.mjs:12373 against ai@6.0.170 dist/index.mjs:13343), and ai-sdk's snapshot places the parked parent's delegating call part beside the child's approval part with no step-start between them (placeDelegatedCallPart, ai-sdk snapshot.ts:519 at origin/main e4ba7cd498). The failing cell mounts the stock useChat ('stock-autosend-approvals', the remote-park scenario), so whether useHelixChat apps are affected is not shown either way; the fix's cells settle it, and a live stream with includeStepEvents false (StreamTransformer's default) may meet the same placement without a reload (unconfirmed); related: the live case, by WE-2's unit at the red commit 7c64340f2b (branch reliability-we2-cp284; its lock-free run w-lftest-1107-051351-47869, read by the owner: 14 failed, 6 passed of 20, each case as below): ai-sdk/src/__tests__/handler/delegated-approval-autosend.test.ts:281-360 (as on branch reliability-we2-cp284 7c64340f2b) drives each aliased ai version's own AbstractChat with its own predicate, then answers a sub-agent's approval. After a reload (the remote child of CP-157's render and the local child of CP-186's) it sends on 6.0.0, 6.0.49 and 6.0.91 and never on 6.0.230 and 6.0.281; live through StreamTransformer with no step boundary reported, includeStepEvents false and true, it never sends on all five. The cause of the live case: ai-sdk/src/transformer/stream-transformer.ts:1022-1044 (origin/main e4ba7cd498) sets providerExecuted only when findTool resolves the tool, and nothing shipped passes findTool. The conformance non-reload cell passes only because the ScriptedModel reports steps and both hosts set includeStepEvents true; related: WE-2's second lock-free run at the red a71da9220b (branch reliability-we2-cp284; w-lftest-1107-051829-89124, read by the owner from WE-2's report: 16 failed, 34 passed of 50) proves two things the first run cannot: useHelixChat's own predicate (createHelixSendAutomaticallyWhen) sends in all 25 of its cases, on every ai version and shape, so it is not hit; and the live case with findTool wired sends on ai 6.0.0, 6.0.49 and 6.0.91 and never on 6.0.230 or 6.0.281, so findTool alone fixes it only below 6.0.158. The fix needs both the providerExecuted mark and the step boundary; ai-sdk/src/transformer/stream-transformer.ts:613 and :1185 (as on branch reliability-we2-cp284 564e709e34) (fix: CP-284, WE-2: a child's first part after an unanswered delegating call starts a step, and a reserved-prefix delegating call is providerExecuted without findTool), with ai-sdk/src/transformer/stream-transformer.ts:917 (as on branch reliability-we2-cp284 564e709e34) (a forwarded child's step chunks give the root's message no step events, decided by core's own/guest rule, core/src/stream/run-markers.ts:84, as on branch reliability-we2-cp284 564e709e34) and ai-sdk/src/handler/snapshot.ts:588 (as on branch reliability-we2-cp284 564e709e34) (the reload places delegated parts after one step-start, as live); related: the proof is unit-level (provenByException below): no conformance client lane runs ai 6.0.158 or later yet, and on the client tier's ai 6.0.49 the new cells pass at the red too (both approvals already share the last step there). WE-2's slot w-cp284s-1107-210757, read by the owner: at RED 18c57b3029 and FIX 564e709e34, 3 of 3 each, subagent.delegated-approvals-{parallel, parallel-reload, sequential, sequential-helix}-client on cf-do and js-chat-host and {parallel-helix, parallel-reload-helix}-client on cf-do pass every id (guards, with .root-step-marked); the CP-157 / CP-186 / CP-180 / CP-182 cells pass at FIX, and CP-157's cfw-workflows-d1 executor cell (slot w-cp284g-1107-225715). The client-tier red is owed by FU-CONF-CLIENT-TIER-CURRENT-AI's newest-pair lanes (Joe, 2026-11-07: the floor ai 6.0.170 and the newest pair), where these cells join provenBy; related: subagent.delegated-approvals-parallel-helix-client and -parallel-reload-helix-client on js-chat-host fail at RED and at FIX alike: CP-288, not this record; related: FU-STOCK-MIXED-STEP-DELEGATED-APPROVAL (a step that parks the root's own approval beside a child's keeps main's shape, by design) and FU-AISDK-NESTED-DELEGATION-RELOAD-PARTS (a reload omits an intermediate child's delegating part); severity S1 (owner, 2026-11-07, raised from S2 once the live case was shown): every live sub-agent approval in a stock auto-sending useChat app over llm-vercel, on every ai 6.x, does nothing when the user answers it: the run stays paused with nothing reported; useHelixChat apps are not shown either way | live-repro, peer-reported | verified | fixed (no lane can reach it: 8 tests) |
| CP-285 | S3 | CF Workflows: a continuation heal pairs a delegated remote call whose child may have run as never executed. G1's delegation commit saves the reply before the child starts; when that run is lost and a new turn, a with_message resume or a retry with a message heals the trailing call, the row says unpaired_on_continuation (the call was not executed) where runtime-js and the CF DO now say outcome_unknown for a saved reply's calls, so the model is told a call that may have taken effect never ran. | runtime-cloudflare/src/steps.ts:1740-1745 (origin/main fc9ecacc32) (a new turn settles the session's remote children first, 'orphan': a finished child's row is placed and a running one is orphaned, its call paired by the C5 heal) and :2025-2037 (origin/main fc9ecacc32) (a with_message resume over a lost delegation pairs its trailing calls with buildTrailingHealMessages and no reason, so unpaired_on_continuation); contrast: runtime-js/src/saved-row-reason.ts:1-33 (origin/main fc9ecacc32) (savedRowPairingReasonOf: runtime-js and the CF DO pair a saved reply's calls outcome_unknown at every heal, Save-First Tools MR 1, RM-91); inferred: scope is by reading only. Which calls reach the heal depends on the settle placing a finished child's row and on a continue's re-attach answering first (G1 / CP-216 D3); the first cell (a lost delegation, then a message entry, on cfw-workflows-d1) decides it. If every such call is answered before the heal, this record is withdrawn; runtime-cloudflare/src/steps.ts:1772-1775 and :2037-2038 (as on branch reliability-w1-cp285 cccb64745b) (fix: CP-285, W1: the new-turn heal (planNewTurnPendingSettlement's healReason) and the with_message / interrupted heal pass failurePairingReason(history)) with runtime-cloudflare/src/run-entry.ts:2224-2225 (as on branch reliability-w1-cp285 cccb64745b) (the retry heal, failurePairingReason(prefix)): every call of a trailing saved / delegation row reads outcome_unknown and any other row keeps unpaired_on_continuation, the whole-row rule this runtime's own failure pairing uses (:4586); CF Workflows records no finishWith attempts, so the rule is the whole row's (owner ruling, 2026-11-08); conformance: fixed. W1's slots w1-cp285-slot-20261108-100830 and w1-cp285-slot2-20261108-103514 (read by the owner, MISMATCHES 0): transcript.lost-delegation-new-turn-reason (a delegated child held, the run lost, a new turn while the child still runs) and transcript.lost-delegation-retry-message-reason (a regular sibling in the lost row, the run settled failed, then retry with a message) on cfw-workflows-d1 fail exactly .call-outcome-unknown and .sibling-outcome-unknown 3 of 3 at the red 36f125c7f3 and pass 3 of 3 at the fix 5e74dd20b1 (restacked on main as 1ac6b82eac / cccb64745b, the same +/- lines). One-site mutations each red only their own cell (m1 the new-turn heal, m3 the retry heal); the with_message heal (m2) is reached by no cell (a failed session refuses with_message, an active one races the closed-owner settle, an interrupted one is already paired run_stopped) and is proven by the unit W3e alone, with W4 (new turn) and W3c (retry) in cfw-g1-delegation-commit.test.ts. The lanes that lack terminate-workflow-instance are declared gaps; related: replaces FU-CFW-HEAL-SAVED-ROW-REASON (owner ruling, 2026-11-07; the Save-First Tools spec's §4.3 names CP-285), CP-294 (the same heals on Temporal and DBOS, open), CP-286 (the branch heal's same reason on every runtime), RM-91 and RM-117; severity S3: the model may repeat a call whose side effect already happened, and the transcript says it never ran; nothing committed is lost. S2 if a runtime re-runs the call itself | source, live-repro | verified | fixed (2 cells) |
| CP-286 | S3 | runtime-js and CF Workflows: a branch's seed heal paired a call that may have run in its source as never executed. A branch from a saved or delegation row's checkpoint copies a reply whose call may have taken effect in the world; the branch paired it unpaired_on_continuation, so the branch's model was told the call never ran and might repeat it. Now it reads outcome_unknown (Temporal and DBOS: CP-294) | runtime-js/src/js-agent-executor.ts:9484-9488 (origin/main fc9ecacc32) (pre-fix: the branch heal keeps the default reason; its comment names this record) and runtime-cloudflare/src/steps.ts:1831-1836 (origin/main fc9ecacc32) (pre-fix: seedHeal = buildTrailingHealMessages(seed), no reason); the Temporal and DBOS branch heals are CP-294; runtime-js/src/js-agent-executor.ts:9499-9501 and :9509 (as on branch reliability-w1-cp286 a08811899c) (fix: CP-286a, W1: both runtime-js branch heals, continuationHeal and cloneRowsHeal, pass savedRowPairingReasonOf over the seed, the per-call rule of every other runtime-js heal, its finishWith attempts read from the SOURCE session, since a clone carries none) with runtime-cloudflare/src/steps.ts:1838-1840 (as on branch reliability-w1-cp286 a08811899c) (seedHeal passes failurePairingReason(seed), CF Workflows' whole-row rule, as CP-285); conformance: fixed. W1's slot w1-cp286-slot-20261108-104919 (rev 2, read by the owner, MISMATCHES 0): history.branch-at-delegation-checkpoint-reason (a remote sub-agent call, the source completes, a branch from the delegation commit's checkpoint) on js-memory, js-redis, js-postgres and cfw-workflows-d1 fails exactly .call-outcome-unknown 3 of 3 at the red 9ef30395f8 and passes 3 of 3 at the fix 4b2219dc68 (restacked as b385bbd983 / a08811899c, the same patches); one-site mutations red only their own lane (m1 runtime-js continuationHeal, m3 CF Workflows seedHeal); m2 (runtime-js cloneRowsHeal) and m4 (attempts read from the target, not the source) are pinned by branch-trailing-heal.test.ts; the -paired, -suspended and -from-stopped-source shapes stay green. Temporal and DBOS are declared gaps on the -reason cell (no remote-child producer), and their heals are CP-294; related: RM-117 (fixed: a branch pairs its seed's trailing calls at all), whose .call-not-executed this record re-rules for a call that may have run. In the branch's own timeline the call never ran, but its side effect may have; Save-First Tools MR 1 left the branch sites at the default to keep RM-117's cells standing (WA's slot slot-b14c); related: CP-285 (CF Workflows' continuation heal, the same reason for a lost delegation), RM-91 (runtime-js and the CF DO pair a saved reply's calls outcome_unknown at every other heal); severity S3: a branch may repeat a call whose effect already happened; nothing committed is lost, and the source is untouched | source, live-repro | verified | fixed (4 cells) |
| CP-287 | S1 | runtime-js and CF Workflows: an approval a persistent companion's remote child parked on the root, answered through the root while the root's run is live, is accepted, marked and forwarded, but nothing continues the companion. The router makes the companion the continuation's owner only once the root is no longer live (CP-279's ended-root half); under a live root it starts the root's continuation instead, and the root's run is not the one parked on that call, so the companion stays paused and its work never resumes. | core/src/client-tool/route-submit.ts:930-938 (origin/main e4ba7cd498) (companionOwner is chosen only when the root's status is neither active nor paused) with :1055-1076 (origin/main e4ba7cd498) (otherwise the continuation is started for the root, ownerRole 'root'); conformance: companion.parked-answer-live-root-continues (A's shape: the root's turn-2 model call held) and companion.parked-answer-live-root-continues-while-mid-runs (B's shape: mid's model call held), executor tier, split out of CP-282's scenarios. On js-memory, js-redis, js-postgres and cfw-workflows-d1 both fail exactly .companion-continues, 3 of 3 identical, at the pre-fix product 9513ec119b and at CP-282's fix (5a3e4f77cb and 52ef0bba6c): the answer is accepted and marked, the root's turn completes, and buddy's session stays paused (did not settle within 60000ms). W1's slots w1-cp282cell7-slot-20261107-125616 and w1-cp282cell8-slot-20261107-173309, read by the owner; at 52ef0bba6c the B-shape cell finds buddy through the root's parked calls (.buddy-resolved passes in every run) (the slots ran on this branch before two rewrites of its history that changed no product code, only test-support files and comments); related: CP-282's -mark-kept guards (companion.parked-answer-vs-root-commit-mark-kept, companion.parked-answer-while-mid-runs-mark-kept) were judged with buddy's hold armed but never reached, because of this record. The fix's MR re-runs them with the hold engaged (the observe line reading 'continuation inside true') beside the flip of these cells; related: the CF DO, by reading: it runs runtime-js's router; no host lane has a remote-child producer (FU-CONF-CF-DO-GENUINELY-REMOTE-CHILD-PRODUCER), a declared gap. Temporal and DBOS never park a remote call on a root (CP-230), declared gaps; related: W1's fix design (peer-reported, 2026-11-07; built as below): core only, in the parked-call submit route: under a live root it applies CP-250's submit-owner rule over the parked parent, so an unawaited companion continues itself, the mark's fence following the root's state; related: CP-279 (the ended-root half, fixed), CP-250 (a submit for a paused companion's call wakes it), CP-282 (the same scenarios' commit race, fixed); core/src/client-tool/route-submit.ts:925-928 (as on branch reliability-w1-cp287 a92bf589f8) (fix: CP-287, W1: under a live root an answer to a call a companion's remote child parked continues the companion itself when its parent does not await it, core parkedAnswerContinuesCompanion, core/src/orchestration/parked-entry-current.ts:122, as on branch reliability-w1-cp287 a92bf589f8; an awaited child keeps the root's continuation) with core/src/client-tool/route-submit.ts:805 and :979 (as on branch reliability-w1-cp287 a92bf589f8) (the answered mark's fence follows the root's current run, cleared once if the root moved since the mark) and runtime-cloudflare/src/executor.ts:4765 (as on branch reliability-w1-cp287 a92bf589f8) (CF Workflows' companion owed gate applies the same rule); runtime-js/src/run-loop.ts:1377, :2033 and :7028 (as on branch reliability-w1-cp287 a92bf589f8) (fix: record first: a re-attached remote child's result row is committed before its entry comes off the root, and the deferred removal re-reads the root's run at the write; CF Workflows already appended first, pinned by a guard unit); conformance: fixed. companion.parked-answer-live-root-continues-owned and companion.parked-answer-live-root-continues-while-mid-runs-owned (CP-282's tree, the answer owned before the release, then the companion continues once) fail exactly .companion-continues 3 of 3 on js-memory, js-redis and js-postgres at CP-282's red 9513ec119b and fix ee46c4575f, and pass 3 of 3 at a92bf589f8; companion.parked-answer-live-root-continues and -while-mid-runs on cfw-workflows-d1 do the same (W1's slot w1-cp287b-slot-20261107-231718, read by the owner). The old C / CM on the js lanes race their mark read against the consume once the companion continues (at a92bf589f8 A and C fail .answer-marked, their other ids as its preconditions, in 8 of 9 js runs each; B and CM read the same mark and passed), so they are gapped there; the CP-282 -mark-kept guards were re-run with buddy's hold engaged ('continuation inside true') and pass; severity S1, as CP-279: an approval the user answered is accepted, yet the companion's work never continues; nothing committed is lost, and the root's turn answers | source, live-repro, peer-reported | verified | fixed (8 cells) |
| CP-288 | S2 | runtime-js behind handleChatStream: a chat request that answers only one of the approvals a root's parallel in-process children parked is refused 409 state_not_resumable, though the answer was stored. The submit answers caller_resumes for a call whose batch, spread over the children, is still incomplete (it should be awaiting_batch), so the handler resumes the root, which refuses while its other child still waits. useHelixChat sends each answer as it is given, so a second answer given while the first request is in flight is dropped by the errored chat and the turn stalls until the user retries. A request carrying every answer completes; the CF DO, whose children are separate DOs, completes each answer. | core/src/client-tool/submit-continuation.ts:102 (origin/main fc9ecacc32) (resolveSubmitContinuation, the batch rule that answers caller_resumes here) with runtime-js/src/js-agent-executor.ts:1814 (origin/main fc9ecacc32) (the root resume refused: "Awaiting client tool call(s) … of a sub-agent: submit every result of its batch"); conformance: subagent.delegated-approvals-parallel-helix-client and subagent.delegated-approvals-parallel-reload-helix-client on js-chat-host (client tier, useHelixChat's per-answer sends), added by CP-284's MR, fail exactly .turn-completes (the root stays paused) and .children-ran (precondition), 3 of 3 identical at CP-284's red 18c57b3029 and at its fix 564e709e34 (WE-2's slot w-cp284s-1107-210757, read by the owner); the same two cells on cf-do and the stock-useChat parallel cells (both answers in one request) pass on both hosts; live: WE-2's diagnostic (slot w-cp284g-1107-225715, read by the owner): in both answer orders a request carrying one answer is answered 409 with no events, the root paused and both children waiting; the next request, carrying both answers, is answered 200 and the turn completes. A cell's second answer, given while the first request is in flight, never goes out; live: WE-2's runtime-js repro (lock-free run w-lftest-1107-231714-45157, the real JSAgentExecutor behind handleChatStream, two parallel local children each parked on an approval, peer-reported and read by the owner): submitToolResult for one call answers { accepted, caller_resumes }; the handler's resume of the root throws ResumeRefusedError state_not_resumable; the response is 409 { code: 'state_not_resumable' }; the answered call reads stored afterwards. A later request carrying both answers gets already_completed and accepted, the resume succeeds and the root completes. Answer order makes no difference; core/src/client-tool/submit-continuation.ts:148 (as on branch reliability-we2-cp288 21367591535f) (fix: CP-288, WE-2: with a runtime's treeUnanswered, an owner other than the root whose batch is ready while the root's awaited in-process tree still has an unanswered call answers awaiting_batch, not caller_resumes) with core/src/client-tool/route-submit.ts:511 (as on branch reliability-we2-cp288 21367591535f) (the router passes it) and runtime-js/src/js-agent-executor.ts:8735 (as on branch reliability-we2-cp288 21367591535f) (runtime-js, and the CF DO that runs it, supply the tree's unanswered calls, core awaitingChildrenUnansweredIds); CF Workflows and DBOS pass none, since their root continuation settles its own children; conformance: fixed. subagent.parallel-children-one-answer (executor tier: two parallel in-process children each parked on an approval, answered one at a time through the root, each answer's continuation acted on) on js-memory, js-redis and js-postgres fails .first-answer-continues-cleanly (the first answer's caller_resumes, then the root's resume refused) at the red e8d224559b and passes every id at the fix 21367591535f, 3 of 3 each (WE-2's slot w-cp288f-1108-004519, read by the owner); DBOS passes at both (each answer a new_run child continuation). The two js-chat-host helix client cells, ledgered here before the fix, pass 3 of 3 at the fix (unexpected-pass) and fail as ledgered at the red (slot w-cp288s-1107-234210, at 650b4b7567, whose product equals the fix's: the diff to 21367591535f touches only the scenario and an ai-sdk fixture test); related: the scenario's cfw-workflows-d1 cell fails .children-ran at both heads: CP-289, a separate CF Workflows defect (the root completes without the second child's result); Temporal answers the second answer caller_resumes and refuses the resume state_already_running while the turn still completes: CP-290 (open); conformance: on the newest supported ai pair too (FU-CONF-CLIENT-TIER-CURRENT-AI, the client-pairs MR stacked on this fix): subagent.delegated-approvals-parallel-helix-client and -reload-helix-client pass on js-chat-host+ai-newest|client (ai 6.0.314, @ai-sdk/react 3.0.317) 3 of 3, and on js-chat-host|client (the floor pair, ai 6.0.170, @ai-sdk/react 3.0.172) 1 of 1, the run's pair self-test naming the pair it loaded (owner slot clientcur-restack run-20261108-025939 at b71fca3f35, the client-pairs branch over 507f2a332d). Before the fix both failed [turn-completes, children-ran] 3 of 3 on the newest pair (owner slot clientcur-slot run-20261108-012146, before the restack); severity S2 (owner, 2026-11-07, re-ruled from a provisional S1 once the repro showed the answer stored): nothing is lost and a route exists, retry() re-anchors to the snapshot (root paused, one answer stored) and the user's next answer posts the rest and completes; but the default client stalls the turn with an error for an answer that was accepted, as CP-282's class | live-repro, peer-reported | verified | fixed (7 cells) |
| CP-289 | S1 | CF Workflows: when a root's two parallel sub-agents each park on an approval and the approvals are answered one at a time through the root, the root completes without the second child's result. The first answer starts a continuation of the root while the second child still waits; that continuation re-spawns both children, the second answer starts a second continuation, and the second child's re-spawned run parks again on the call its answer was given for. The root's drain then skips a child that is not paused, so the root completes with that child's call unpaired, the child stays paused, and its answer and output are lost, with nothing reported. The same lost wake-up strands a root whose one answer lands while the run that will park on it is still running: the submit is answered new_run naming that running run, which had already read its pending entries, so nothing continues the answered child and the root stays paused. | runtime-cloudflare/src/executor.ts:4737 (origin/main fc9ecacc32) (holdsReadyBatch: a ready batch on the root OR on any owner in its clientToolCallOwnership, so one child's answer is enough to owe the root's continuation) with runtime-cloudflare/src/steps.ts:10275 and :10300 (origin/main fc9ecacc32) (the drain's scanChildrenForRespawn, and a root with no own pending call returns hasMoreSuspended false whatever its children are) and runtime-cloudflare/src/steps.ts:10951-10954 (origin/main fc9ecacc32) (only a child failed 'parent_suspended' or paused is re-spawned; any other awaited child is skipped with no row); conformance: subagent.parallel-children-one-answer (executor tier, added by CP-288's MR; .parent-parked waits until the root's run reads suspended_awaiting_children, nothing runs and the session is paused) on cfw-workflows-d1 fails exactly .children-ran ("the root's row for child 2: expected '' to contain 'PCOA-CHILD-OUTPUT-2'") and passes .turn-completes (the root completes), 3 of 3 identical at CP-288's red e8d224559b and fix 21367591535f (WE-2's slot w-cp288f-1108-004519, read by the owner); js, Temporal and DBOS complete with both children's results; live: the same slot's records (peer-reported, read by the owner): each answer is accepted with new_run, on two different __submit__ instances of the root; child 1 completed through a __respawn__ run; child 2, re-spawned by the same __respawn__, reads paused with its run suspended_client_tool; the completed root holds no row for child 2's call (the unpaired call RM-29 forbids). An earlier version of the scenario answered before the root's suspension commit and ended either way (completed, or paused), so its runs are not the evidence; runtime-cloudflare/src/steps.ts:1127-1138 (as on branch reliability-w1-cp289 c69db938de) (fix part 1: CP-289, W1: the fold save keeps the suspension context it wiped, CP-296, so a re-spawned child's awaited entry survives the re-spawn) with runtime-cloudflare/src/workflow.ts:1723-1753 (as on branch reliability-w1-cp289 eb7adb6e73) (fix part 2: after a run's suspension commits, one durable step re-decides the owed continuation through the executor's own continueOwedSession, the router's and the deadline watcher's rule, so an answer stored after the run's last read gets its continuation; the step's executor import is static, CP-298); conformance: fixed. subagent.parallel-children-one-answer on cfw-workflows-d1 fails exactly .children-ran at the RED 4f0c7204d9 and passes every id at the fix e5d7592f1e, 3 of 3 each; subagent.parallel-children-answer-during-respawn (added by this MR: the second answer lands while child 1's continuation still runs) on cfw-workflows-d1 fails exactly .turn-completes and .children-ran at the RED and passes every id at the fix, 3 of 3 each, its observe line 'first continued: root' in every run; js, Temporal and DBOS pass both at both heads (W1's driver289b, a local run of 2026-11-08; the slot's heads 4f0c7204d9 / e5d7592f1e carry the same patch, by patch-id, as the branch's 6f3683a619 / eb7adb6e73 after it was stacked on !781). Mutations at the fix: m1 (the save's echo removed) turns one-answer [children-ran] and state.turn-fields-kept-across-saves [turn-id-kept, turn-start-kept] red; m2 (the owed re-check removed) turns during-respawn [turn-completes, children-ran] red; live: W1's delta slot (a local run of 2026-11-08, at 998202f531, the branch's 4095f53420 by patch-id): commit-checkpoint-truth's R47 write-window race, re-expressed for the working re-check, passes 3 of 3; the racing resume was adopted 1 of 3 and refused state_not_resumable 2 of 3, the turn already complete; the answer consumed once in every run; with the re-check removed (m2) exactly that case fails ('the re-check created the root's continuation: expected [] to have a length of 1'); related: CP-288 (runtime-js's one-answer refusal of the same shape, fixed), CP-290 (Temporal's second-answer refusal, open), CP-250 (a paused companion's continuation owner), RM-29 (the pairing invariant), CP-82 / CP-160 (the children re-spawn rule), CP-296 (the save wipe, fixed in this MR), CP-297 (the same lost wake-up on runtime-js and the CF DO, open), CP-298 (the lazy import the re-check step first copied, fixed in this MR); severity S1: an answer the user gave and a child's output are lost, the completed transcript carries an unpaired call, and nothing reports it | source, live-repro, peer-reported | verified | fixed (2 cells) |
| CP-290 | S3 | Temporal: when a root's two parallel sub-agents each park on an approval and the approvals are answered one at a time through the root, the second answer is accepted caller_resumes, and the caller's resume() is then refused state_already_running, because the run the first answer's resume started is live. The live root takes the stored answer and the turn completes, but a client acting on the second answer's continuation gets a typed error for an answer that was stored | runtime-temporal/src/executor.ts:2883-2887 (as on branch reliability-we2-cp288 507f2a332d) (Temporal's submit passes startContinuation caller_resumes unconditionally: 'Temporal stores only; the caller resumes. Its resume can still be refused while the previous workflow closes'; it passes no treeUnanswered, which CP-288 wires on runtime-js only) with core/src/client-tool/route-submit.ts:779 (as on branch reliability-we2-cp288 507f2a332d) (passed as start to resolveSubmitContinuation, core/src/client-tool/submit-continuation.ts:105); runtime-temporal/src/executor.ts:2354 and :1098-1104 (as on branch reliability-we2-cp288 507f2a332d) (resume() calls assertNoLiveOwner, which throws AgentAlreadyRunningError when the current run is running and its owner workflow is open: here the run the first answer's resume started); live: WE-2's slot w-cp288f-1108-004519 (read by the owner): subagent.parallel-children-one-answer's unjudged observe (actedSecond.refused) logged 'second continuation refused: state_already_running' on temporal-memory, temporal-redis and temporal-postgres in every run, 18 of 18 at CP-288's red and fix, and on no other lane; the Temporal cells' .turn-completes passed 3 of 3 at the fix; inferred: 'nothing is lost' holds only when the second answer's write lands before child 2's re-spawned drain, or when the app resumes again after the root's first continuation suspends; otherwise the answer is stored but stranded until that retry: runtime-temporal/src/workflow.ts:502 (origin/main 3c6496ce00) (the drain applies the stored results once, applyResultsAndReload) then :914 and :897 (origin/main 3c6496ce00) (the run suspends again through suspendRun) with runtime-temporal/src/activities.ts:8679-8690 (origin/main 3c6496ce00) (suspendRun writes the run status and fires the hook, and re-decides no owed answer): by reading; W1's driver289b (a local run of 2026-11-08) logged the second answer's resume refused state_already_running in all 18 temporal-* during-respawn runs (3 lanes, 3 of 3 at the RED 4f0c7204d9 and at the fix e5d7592f1e), unjudged, and the cells' .turn-completes passed in each; related: CP-288 (runtime-js's one-answer refusal of the same tree, fixed), CP-289 (CF Workflows loses the second child's result), CP-104 (the submit continuation contract); severity S3: nothing is lost (the live root takes the stored answer and the turn completes); a client that acts on the second answer's continuation gets a typed refusal for an answer that was stored, which a chat client shows as an error | source, live-repro, peer-reported | verified | open |
| CP-291 | S2 | handleChatStream: a send that repeats a failed or stopped turn's user message (a stock regenerate() after an error or after Stop, or a useChat that keeps the failed message and sends the next one) stored that message a second time, and the model saw it twice: the turn's run-start commit had already stored it, and the handler forwarded the request's whole trailing user run. Now a failed turn is retried and a stopped turn continued, and the message is stored once and seen once | ai-sdk/src/handler/handle-chat-stream.ts:1660-1661 (as on branch reliability-we2-cp291 daf6ab4e72) (pre-fix: the fresh-turn path forwards the request's whole trailing user run, extractTrailingUserMessages(messages), whatever the store already holds); ai-sdk/src/handler/stored-turn.ts:101 (as on branch reliability-we2-cp291 5e221e56db) (fix: CP-291, WE-2: planStoredTurn matches the trailing run against the stored tail, the visible user rows after the store's last assistant row, by position and content, and only when the client sent its history) with ai-sdk/src/handler/handle-chat-stream.ts:1679-1686 and :1886 (as on branch reliability-we2-cp291 5e221e56db) (only a failed or interrupted session is planned; a run whose every row is stored tries that turn again: a failed session retry(), an interrupted one resume({ mode: 'continue' }); Joe's default, 2026-11-08, named in the changeset); conformance: fixed. WE-2's slot w-cp291c-1108-095257 (read by the owner, the 3 runs of each head identical): at the red daf6ab4e72 (the fix's 3 e2e scenario files swapped in; a :(glob) diff shows red and fix differ under packages/e2e by those 3 files only) chat.resend-after-failed-turn and chat.resend-after-stopped-turn on cf-do and js-chat-host (host) and chat.send-after-failed-turn-client on js-chat-host (client) fail [stored-once, model-sees-once] 3 of 3; at the fix 9d7c4acdc6 all pass 3 of 3 (its two later commits change a changeset and a comment only). The cf-do client cell passes at both heads: the cf-do lane's route forwards only the last user row (CP-293), so it is no proof; related: CP-292 (the same duplicate after a turn that committed a step, open), CP-293 (the opennext example route, which the cf-do lane mirrors, forwards only the last user row); severity S2: a user's message is stored twice and the model answers it twice, silently; nothing is lost and the turn completes | source, live-repro, peer-reported | verified | fixed (5 cells) |
| CP-292 | S2 | handleChatStream: after a turn that failed once it had committed a step, a stock regenerate() (which drops the partial assistant message and re-sends [..., first]) stores first a second time and the model sees it twice: CP-291's stored tail starts after the store's last assistant row, which is the failed turn's own step reply, so it holds no user row and the whole trailing run is forwarded | ai-sdk/src/handler/stored-turn.ts:10-15 and :142-145 (as on branch reliability-we2-cp291 5e221e56db) (the stored tail is the visible user rows after the store's last assistant row; a turn that committed a step has its reply there, so the tail is empty and no row of the re-sent run matches); inferred: WE-2's reading (2026-11-08), checked by the owner against the planner: the re-send is planned as a new turn, not a retry of the failed one, so its user message is stored again; no cell yet; related: CP-291 (the same duplicate for a turn that failed or stopped before its first step commit, fixed); severity S2, as CP-291: a user's message is stored twice and answered twice, silently | source, peer-reported | partly-inferred | open |
| CP-293 | S3 | The example chat routes (opennext-cloudflare-do: chat, brief-flow, coordinator, reasoning; nextjs-redis: chat, brief-flow, reasoning), and the cf-do conformance lane that mirrors the opennext chat route line for line, forwarded only the text of the request's last user row as one new message: the earlier rows of a trailing user run (an earlier refused or unsent message, a host's hidden-context row) and every file part never reached handleChatStream, and nothing reported it. Now each route hands the handler the client's conversation | conformance: the cf-do lane's chat route, e2e/src/harness/cf-do/worker/conformance-worker.ts:1822-1850 (as on branch reliability-we2-cp291 5e221e56db) (the last user row's text parts, joined, sent as one new message with a fresh time-based id; 'a line-for-line mirror of the opennext example's route', :1650): chat.send-after-refused-send on cf-do (host) fails [both-stored, model-sees-both] 3 of 3 at CP-291's red daf6ab4e72 and fix 9d7c4acdc6 (WE-2's slots w-cp291 and w-cp291c, read by the owner): after a refused send only the second message reaches the run, and the first is lost. js-chat-host, which forwards the conversation, passes; related: the shipped example, examples/opennext-cloudflare-do/src/app/api/chat/[sessionId]/route.ts:268-281 and src/lib/agent-client.ts:70-89 (as on branch reliability-we2-cp291 5e221e56db) (the last user row's text only, then startAgentExecution(sessionId, message), which builds one text-only message); conformance: the fix, e2e/src/harness/cf-do/worker/conformance-worker.ts:1899-1910 (as on branch reliability-we2-cp293 e32c9b95d6) (the lane mirror hands handleChat the request's conversation; a body with only a string message keeps the one-row form) with, in the shipped examples, examples/opennext-cloudflare-do/src/app/api/chat/[sessionId]/route.ts:203-218 and the six sibling routes (as on branch reliability-we2-cp293 e32c9b95d6) (fix: CP-293, WE-2); conformance: fixed. WE-2's slot w-cp293-1108-110750 (read by the owner, each head's 3 runs identical): chat.trailing-user-run-reaches-run on cf-do (host) fails [file-stored, hidden-row-stored, model-sees-run] 3 of 3 at the red c375e91863 and passes 3 of 3 at the fix 3652290ab3; chat.send-after-refused-send on cf-do (host), the cell this record tracked before the fix, fails [both-stored, model-sees-both] 3 of 3 at the red and passes 3 of 3 at the fix (restacked as efb374ceda / e32c9b95d6, the same +/- lines). js-chat-host passes both at both heads (the contrast). A :(glob) proof in the slot shows red and fix differ under packages/e2e by the 5 lane-mirror files only; contrast: the DO itself takes a whole run, runtime-cloudflare/src/do/protocol-schemas.ts:310 and :326 (as on branch reliability-we2-cp291 5e221e56db) (message: string | UserInputMessage[]), and the runtime-js entry behind it stores every row; WE-2's trace (2026-11-08), checked by the owner, found no cut in the DO product path; severity S3: example code, not the SDK; an app built from the example silently loses the user's earlier rows and files (no error), and an app that forwards the conversation is unaffected | source, live-repro, peer-reported | verified | fixed (2 cells) |
| CP-294 | S3 | Temporal and DBOS: a continuation heal (a new turn, an interrupted or with_message resume, a retry) or a branch's seed heal over a saved or delegation row pairs its calls unpaired_on_continuation, though an orphaned child had started and the row's other calls may have run (CP-285's defect on CF Workflows, CP-286's branch heal) | runtime-temporal/src/activities.ts:10915, :1598 and :1713 (origin/main 7eb2a4d679) (the new-turn, interrupted / with_message and retry heals call buildTrailingHealMessages with no reason) with runtime-temporal/src/activities.ts:4524 (origin/main 7eb2a4d679) (the delegation row is stamped withSavedCallsMarker) and :10928-10935 (origin/main fc9ecacc32) (the branch heal, no reason); runtime-dbos/src/steps/run-commit.ts:696, :804 and :871 (origin/main 7eb2a4d679) (the new-turn heal after settleBeforeInput orphans a running child, the with_message / interrupted resume heal and the retry heal: no reason) with runtime-dbos/src/workflows/shared.ts:3634 (origin/main 7eb2a4d679) (the delegation row is stamped); a DBOS branch heals through the turn heal over its clone; inferred: W1's reading (2026-11-08), checked by the owner: neither runtime references core failurePairingReason; no cell yet, since neither lane claims 'terminate-workflow-instance' (and Temporal / DBOS no 'remote-child' producer), the controls CP-285's cells need; related: CP-285 (the same heals on CF Workflows, fixed), CP-286 (the branch seed heal on runtime-js and CF Workflows), RM-91, RM-117, DI-06; severity S3, as CP-285: the model may repeat a call whose side effect already happened, and the transcript says it never ran; nothing committed is lost | source, peer-reported | partly-inferred | open |
| CP-295 | S2 | The nextjs-redis example's chat route had no branch for an auto-send (a request whose trailing row is the assistant's, as useHelixChat sends a client tool's answer), so it rebuilt a new turn from the last user row's text: the user's prompt was stored twice, a new run started, and the client tool's answer on that request was dropped, so the paused call never got it. Now the route hands the handler the client's conversation | related: the shipped route, examples/nextjs-redis/src/app/api/chat/[sessionId]/route.ts:78-97 (as on branch reliability-we2-cp291 b99f7f8d0c) (the last user row's text handed to startAgentExecution; no trailing-assistant branch), reached in the example as shipped: its page uses useHelixChat (src/lib/use-snapshot-chat.ts:80), whose predicate auto-sends the answer of colorPicker, an execute: 'client' tool (src/agent/agent.ts:49); related: the fix, examples/nextjs-redis/src/app/api/chat/[sessionId]/route.ts:81-93 (as on branch reliability-we2-cp293 e32c9b95d6) (fix: CP-295, WE-2: the route hands handleChat the request's conversation, which routes a trailing assistant row as the client's answer), with the stale useResumeClientTools comments removed; live: WE-2's slot w-cp293-1108-110750 (read by the owner): examples/nextjs-redis/src/__tests__/chat-routes.test.ts's case '/api/chat POST: an auto-send (CP-295) > forwards the conversation with the client tool's output, never a new turn from the last user text' fails at the red c375e91863 (the route hands startAgentExecution the string 'pick a color') and passes at the fix 3652290ab3, 1 of 1 each; the test is deterministic (every client module mocked, no timers, no I/O); related: CP-293 (the same routes dropped the earlier rows of a user run; fixed in the same MR), CP-291 (a repeated send stored twice, fixed); severity S2: reached in the example as shipped; the user's input is duplicated and their tool answer lost, silently; nothing committed is lost | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 1 tests) |
| CP-296 | S2 | CF Workflows: saveAgentState rebuilds the session from the agent state and drops every session field the agent state does not carry and the store does not keep. Each model step's fold save wiped the suspension context (suspendedAwaitingChildren, suspendedStepId, failureReason, tracingContext, expiresAt; and mode on the memory store), and it never copied the turn fields (turnId, turnStart) or the workspace ref, so every step nulled the turn. The wiped awaited-children map is what lets CP-289's root complete without a child's result | runtime-cloudflare/src/steps.ts:1089-1124 (origin/main 3c6496ce00) (saveAgentState's session literal: the agent state's fields plus the stored pendingClientToolCalls, and nothing else of the stored session, so a full-state saveState replaces them with undefined); runtime-cloudflare/src/__tests__/cfw-save-agent-state-keeps-fields.test.ts and its D1 twin cfw-save-agent-state-keeps-fields.wf-noiso.test.ts (as on branch reliability-w1-cp289 eb7adb6e73) (a table, one case per field, each 'survives the fold save'): at the RED 4f0c7204d9, the memory table fails exactly 9 (suspendedAwaitingChildren, suspendedStepId, failureReason, tracingContext, expiresAt, workspaceRef, turnId, turnStart, mode) and the D1 table exactly 8 (the same without mode, which D1's UPDATE does not name), 3 of 3 each; memoryExtractedThrough survives on both (store-owned, MA-40); green at the fix (W1's driver289b, a local run of 2026-11-08); conformance: state.turn-fields-kept-across-saves (executor tier, added by CP-289's MR) on cfw-workflows-d1 fails exactly .turn-id-kept and .turn-start-kept at the RED 4f0c7204d9, 3 of 3 identical, and passes every id at the fix e5d7592f1e, 3 of 3 (driver289b); js-* pass at both; temporal-* and dbos-postgres are declared gaps (no 'per-turn-entry-state'); runtime-cloudflare/src/steps.ts:1127-1138 (as on branch reliability-w1-cp289 c69db938de) (fix: CP-296, W1: the fields the agent state does not carry are echoed from the stored copy, as the pending map is; the turn fields come from the agent state and the workspace ref falls back to the stored one; the memory watermark is store-owned and no save writes it); related: CP-289 (the wiped suspendedAwaitingChildren lets a re-spawned child's root complete without its result; fixed with this in the same MR), #177's turn fields (the per-turn entry state CF Workflows writes at a turn's entry and this save then nulled); severity S2 (owner): no answer is lost by this alone, but every model step silently erases session state that other readers act on (the awaited children a drain re-spawns, a run's failure reason, its trace context and expiry, the turn's identity) | source, live-repro | verified | fixed (1 cells) |
| CP-297 | S1 | runtime-js and the CF DO: the same lost wake-up as CP-289, by reading. A submit that lands while the run that will park on it is still running is answered deferred (or new_run) naming that running run, but nothing re-decides the owed continuation after that run's suspension commits, so an answer stored after the run's last read is never continued. Three shapes: a paused companion's self-wake (runtime-js answers deferred and nothing continues it, silently); the CF DO's remote child, whose answer the DO's owed gate never reads, so the chat route resumes the root and the submitting client is answered 409 state_already_running, not retryable ('The assistant is still working on the previous message.'), while nothing will continue it unless the client retries anyway; and a runtime-js in-process child answered through the root while the root's own continuation sits between its drain and its re-park (the router answers caller_resumes, the caller's resume is refused busy, and once the root re-parks nothing continues it). The root, child or companion stays paused with its answer stored | runtime-js/src/js-agent-executor.ts:6387-6455 (origin/main 3c6496ce00) (startCompanionSelfWake: a companion whose current run is live and not suspended answers deferred naming that run, :6397-6405, and so does a self-wake that lost its claim, :6441-6448; nothing re-checks isSubmittedBatchReadyAtLastSubmit after that run's suspension commits); core/src/client-tool/deadline.ts:31-47 (origin/main 3c6496ce00) (isSubmitContinuationOwed, the CF DO's one owed gate, reads only the session's pendingClientToolCalls: a remote child's stored answer is never a ready batch there); ai-sdk/src/handler/handle-chat-stream.ts:1559-1578 (origin/main 3c6496ce00) (the chat route acts on the stored answer's caller_resumes by resuming the root, and catches the busy refusal as a typed entry error) with ai-sdk/src/handler/handle-chat-stream.ts:3689-3705 and :3797 (origin/main 3c6496ce00) (alreadyRunningResponse: 409, code state_already_running, retryable false once the input committed); live: WE-2's probe w-cp297p-1108-133728 (at accce3418f on b587102274, peer-reported, read by the owner): subagent.remote-answer-during-reattach-host on cf-do, 3 of 3: the second answer's POST is answered HTTP 409 with body error 'The assistant is still working on the previous message.', code state_already_running, retryable false, and the root stays paused; in each run one retry of that approval POST anyway continued the root to completed, child 2's result folded in, its own stream ending stream:error 'truncated past step 1 while a reader was attached'; inferred: on the CF DO the busy refusal is a drive rejection (AgentAlreadyRunningError, status running), most likely runtime-cloudflare/src/do/drive/run-drive.ts:2202 (the chat route's resume carries no request key while the first continuation's drive holds the producer claim); not traced, and why the remote-forwarded answer comes back caller_resumes rather than deferred is not traced either; runtime-js/src/js-agent-executor.ts:8731-8734 (origin/main 3c6496ce00) (startContinuation: any owner but a companion, an in-process child included, answers caller_resumes) with runtime-js/src/js-agent-executor.ts:2786-2793 (origin/main 3c6496ce00) (assertNoLiveOwner refuses the caller's resume with AgentAlreadyRunningError while the root's continuation runs; that run re-parks without re-deciding the child's stored answer); inferred: by reading, not run: CF Workflows had the same window (CP-289's during-respawn cell, red on cfw-workflows-d1 before its fix: the second answer lands while child 1's continuation still runs, and the root stays paused); runtime-js and the DO share the step loop, and neither runs an owed re-check after a suspension commits, so the same order strands the answer there; related: CP-289 (CF Workflows' lost wake-up, fixed with a post-suspension owed re-check), CP-250 (the companion self-wake this shape enters), CP-104 (the submit continuation contract), CP-300 (the retried approval POST's stream ending in a truncation error while the turn completes, the live line's caveat); severity S1 provisional (owner, 2026-11-08): an answer the user gave is stored and never acted on; on runtime-js nothing reports it, and on the CF DO the client is told a non-retryable 'still working' for work that never resumes unless it retries anyway; by reading, revisited if the cell shows a read or wake heals it; runtime-js/src/js-agent-executor.ts:6481-6583 and :1965 (as on branch reliability-we2-cp297 84737aac5f) (fix: CP-297, WE-2: after a suspension commits, recheckOwedAfterSuspension re-reads the owed continuation, readOwedContinuation, and starts it through resolveSubmitContinuation; absent at main 07d932612e) with :6958 and :7065-7069 (as on branch reliability-we2-cp297 84737aac5f) (a caller's resume adopts the owed continuation's run, owedContinuationRunId, instead of being refused busy); runtime-js/src/js-agent-executor.ts:6604-6630 and :6788-6804 (as on branch reliability-we2-cp297 84737aac5f) (fix: a companion self-wake waits for its own pending outcome write before it reads the parent's ref, and registers its outcome before the run starts, so the re-check and the submit route never read a stale running ref; pre-fix :6387-6394 at main 07d932612e read the ref at once); runtime-cloudflare/src/do/durable-object-agent-base.ts:2644-2697 and :2975-2976 (as on branch reliability-we2-cp297 84737aac5f) (fix: on the CF DO the release re-checks an owed answer on the session's own parked call and the wake continues it, trigger owed_answer; runtime-js's own re-check is off in the DO, build-executor.ts:120); conformance: fixed. WE-2's slot w-cp297c2 (2026-11-09) at FIX ca373a7051 (rebased onto main 07d932612e as 84737aac5f), RED c343b882a2 and RED2 df7a381d10 (the held cells with the race still present), every row as predicted: companion.self-wake-answer-during-suspension and subagent.child-answer-during-root-continuation fail at RED and pass at FIX on js-memory, js-redis and js-postgres; subagent.remote-answer-during-reattach-host fails at RED and passes at FIX on cf-do; companion.self-wake-answer-during-suspension-held-write and companion.self-wake-submit-during-outcome-write-held fail exactly .answer-used and .companion-continued at RED2 and pass at FIX on the three runtime-js lanes, the self-wake deciding inside the hold at RED2 in every run (its timeline printed); 3 of 3 at each head; conformance: guards, passing at both heads: companion.self-wake-answer-during-suspension on cfw-workflows-d1 and subagent.child-answer-during-root-continuation on temporal x3 and cfw-workflows-d1; the sweep of every companion, subagent, submit, client, approval and clienttool scenario on runtime-js and both hosts matched the ledger (537 rows). The dbos-postgres row of companion.self-wake-answer-during-suspension is ledgered under CP-299 (the companion's own hooks never run on DBOS) | source, live-repro, peer-reported | verified | fixed (13 cells) |
| CP-298 | S2 | CF Workflows: the client-tool deadline watcher's continuation step loads the executor with a lazy import('./executor.js') inside a workflow step. Where the module was first loaded in another Durable Object or request context of the isolate (a deploy that ships unbundled modules; the workerd test pool), workerd rejects the import's module-load fulfiller into the workflow's context, which fails the step attempt after its callback returned. The engine retries it for about 31 s and then the step rejects 'Promise will never complete.', so a lapsed client-tool deadline never continues the turn and nothing reports it | runtime-cloudflare/src/workflow.ts:856-858 (origin/main 3c6496ce00) (the deadline watcher's continueOwed: 'const { CloudflareAgentExecutor } = await import("./executor.js")' inside runClientToolDeadlineWatch's deadline-continue step) with runtime-cloudflare/src/client-tool-deadline.ts:235-242 (origin/main 3c6496ce00) (the step.do('deadline-continue') that awaits it; its rejection is not caught, so the __deadline__ instance ends errored); live: W1's diag 2 (a local run of 2026-11-08, at the CP-289 branch's 0132586696, three runtime-cloudflare wf-noiso files on the workerd pool, an isolate-wide unhandledrejection listener): a step body that does nothing but that lazy import fails every attempt exactly as the full body (33 attempts, 33 unhandled rejections 'Cannot perform I/O on behalf of a different Durable Object … (I/O type: RefcountedFulfiller)', no JS frames, then 'Promise will never complete.'); the same store reads with no import complete every step on attempt 1; live: W1's driver289b (a local run of 2026-11-08, phase 6: origin/main 3c6496ce00 plus the pool file cfw-step-executor-import.wf-noiso.test.ts): the deadline case fails 3 of 3 (a cross-context rejection; the lapsed call's continuation never completes the turn) and the re-check case passes (main has no re-check step): the watcher's lazy import predates CP-289; inferred: the watcher's end state (read from the code, not traced live): the lapsed call's timeout result is written by the deadline-expire-N step before deadline-continue fails, so the session stays paused with a complete batch and no continuation until a reader (a snapshot's reconcileSession, a submit, getHandle) finds it owed and starts it from a request context; with no reader it stays stranded with no typed end; inferred: default-bundled deploys are likely unaffected: wrangler's esbuild bundle inlines a dynamic import of a bundled local module, so no runtime module load happens; not verified on a deployed account; runtime-cloudflare/src/workflow.ts:170 (as on branch reliability-w1-cp289 eb7adb6e73) (fix: CP-298, W1: one static import of CloudflareAgentExecutor replaces the lazy import in the deadline watcher's continueOwed and in CP-289's owed re-check step; there is no runtime cycle: run-entry.ts imports only a type from workflow.ts and executor.ts's value imports never reach it, cfw-workflow-executor-import-order.test.ts loads both orders); related: CP-138 (introduced with its watcher, 1965fd8934); CP-289 (its owed re-check step copied the same lazy-import shape and failed the same way, which held suspended instances open past their callers' bounds); severity S2 provisional (owner, 2026-11-08): a silent loss of a contract (a lapsed client-tool deadline never acts, and the owed re-check never decides) limited to a deploy shape that ships unbundled modules | source, live-repro | partly-inferred | fixed (no lane can reach it: 2 tests) |
| CP-299 | S2 | DBOS runs none of an agent's OWN hooks (lifecycle, tool or sub-agent hooks) in a workflow started inside a run: an ephemeral sub-agent child, a persistent companion's spawn and its continue or respawn get no HookManagerRegistry entry, which only the executor's own entries (execute, resume, retry) set, so the child's hooks are silently skipped. | runtime-dbos/src/workflows/standard-workflow.ts:343-345 (origin/main 638aaed36c) (an ephemeral child: DBOS.startWorkflow(StandardWorkflows, { workflowID: childWorkflowId }).standardAgentWorkflow(childInput), started inside the parent workflow, with no hook registration); runtime-dbos/src/dbos-agent-executor.ts:241-256 (origin/main 638aaed36c) (the startPersistentWorkflow op: builds the PersistentWorkflowInput, :245-252, and starts persistentAgentWorkflow under its workflowID, :253-255; a companion workflow started inside a run, with no hook registration); runtime-dbos/src/lifecycle/execute.ts:326-329 (origin/main 638aaed36c) (contrast: an executor entry registers its merged hook manager under its workflow id, resolved in the workflow by HookManagerRegistry.get(DBOS.workflowID)); the only other registrations are execute.ts:594, resume.ts:303 and :570, and retry.ts:124 (all executor entry paths; no registration at either in-run start site, by a grep of runtime-dbos/src); conformance: WE-2's diagnosis slot w-cp297d confirmed the loss on a self-woken DBOS companion (its own onAgentSuspended never ran); conformance: companion.self-wake-answer-during-suspension (executor, CP-297's scenario) on dbos-postgres fails .answer-unused, .c1-reparked-held, .c1-settles, .companion-still-parked and .x-answer-stored, 3 of 3 identical in WE-2's slot w-cp297c2 at ca373a7051 (MR !796); the scenario's re-park hold is the companion's own onAgentSuspended hook, so .c1-reparked-held fails first and every later id follows from it (earlier records at older heads, a crash in w-cp297p and five ids incl. one unevaluated in w-cp297d, informed but did not decide the ledger); when CP-299 is fixed this cell tests CP-297's DBOS half; related: found by WE-2 while tracing CP-297 on DBOS (2026-11-08); scope confirmed by WE-2's source read, lines re-checked by the owner at origin/main 638aaed36c; DBOS is parked under the Cloudflare-first program; conformance: W1's CP-316 cells (slots c316-r1 and c316-r2, 2026-11-10; c316-r2 re-ran the fix twice, rows byte-identical to c316-r1's) on dbos-postgres: hooks.noop-manager-own-hooks-subagent .sub-agent-own-hook-ran and hooks.noop-manager-own-hooks-companion .companion-own-hook-ran fail at main ae4c3d8040 and at the fix (816ff95498, on R4 f1aeb92dbc), the child's own onAgentStart never witnessed, while its tool body ran (.child-ran, .companion-ran pass) and the root's own hook ran at the fix (.root-own-hook-ran, .parent-own-hook-ran pass): the child workflow gets no hook manager (standard-workflow.ts:343-345 (origin/main ae4c3d8040); its steps fall back to the executor config manager, steps/hooks.ts:261-277 (origin/main ae4c3d8040)); severity S2: an app's own hooks on a child agent (tracing, auditing, a lifecycle side effect) silently do not run on DBOS; S1 if a correctness hook is found to rely on it; related: its direct witness, hooks.child-own-hooks-run and hooks.companion-own-hooks-run (one id per hook kind, every executor lane, the hooks reporting through the tool-witness port), is WE-2's next probe; a respawn's start site is unwitnessed (by reading) | source, probe, peer-reported | verified | open |
| CP-300 | S2 | runtime-js and the CF DO: a continuation's orphan cleanup removes stream chunks of COMMITTED appends: an append-only continuation (a root parked on two children, child 1's answer folded by append, re-parked with no step commit) leaves its committed chunks stamped with a step above the latest checkpoint's stepCount, and the next continuation's cleanupToStep, which decides 'uncommitted' by step number, deletes them; a reader mid-way through them fails with a truncation error while the turn completes, and a later replay from the stream lacks committed content (inferred from source; probe owed). | runtime-js/src/js-agent-executor.ts:7429 and :7866-7871 (origin/main 07d932612e) (every runtime-js continuation, the CF DO's included, calls cleanupToStep with the latest checkpoint's stepCount and the previous run's startSequence after its run start); runtime-cloudflare/src/do/do-stream-manager.ts:1540-1587 (origin/main 07d932612e) (cleanupToStep removes chunks whose step is above the given step past the sequence and records the cut; a reader past it gets the cleanup's rewound); live: WE-2's CP-297 pre-fix probe (w-cp297p, cf-do, 3 of 3 identical): the retried approval POST's stream ended with 'stream truncated past step 1 while a reader was attached' while the root completed; the original reattach shape may no longer reproduce after CP-297, and the probe (branch reliability-we2-cp300-probe) now targets the plain shape; inferred: that the removed chunks belong to committed append checkpoints (a no-op cleanup's rewound is harmless on both hosts, WE-2's corrected trace); the probe records the removed range against the continuation's checkpoint cursors; related: CP-297 (the reattach shape this was first seen in), stream wire contract v2 (a rewound only for chunks a cleanup removed); severity S2 provisional: committed content is deleted from the stream and a live reader fails mid-turn; the snapshot, built from rows, is unaffected; revisited when the probe runs | source, live-repro, peer-reported | partly-inferred | open |
| CP-301 | S1 | runtime-js: the ROOT agent's own hooks run for every descendant's events, in the descendant's context: the user hook manager carries the root agent's hooks and every child and companion inherits it at every depth, so a root hook fires again for each child's start, tool and model call, and a root hook's state write lands in the child's persisted customState. | runtime-js/src/run-loop.ts:3409-3425 and :5620-5641 (origin/main 07d932612e) (a companion's and a sub-agent's hook manager is the parent's user manager's createChild plus the child's own hooks, passed down unchanged, so a grandchild gets the root's hooks too); runtime-js/src/js-agent-executor.ts:9154 and :9163 (origin/main 07d932612e) (buildUserHookManager registers the ROOT agent's own hooks with the execution-level ones); conformance: WE-2's records w-cp301r and w-cp301r2 (record-only, branch reliability-we2-cp299-witness): hooks.child-own-hooks-run-once fails .child-state-untouched and .no-root-hooks-in-child on js-memory, js-redis and js-postgres, and hooks.companion-own-hooks-run-once .companion-state-untouched and .no-root-hooks-in-companion (the root's onAgentComplete wrote rootHookWrites into the child's and the companion's customState, both companion runs); the child's own hooks fire exactly once; Temporal, CF Workflows and the CF DO keep the child's state clean; live: W1's slot c316-r1 (2026-11-10, the CP-316 cells hooks.noop-manager-own-hooks-subagent and -companion at the SDK-6 fix, MISMATCHES 0): on js-memory, js-redis and js-postgres the root's (or parent's) own onAgentStart fires under each child's session with the child's run id, beside the child's own onAgentStart, for a sub-agent and for a companion; on dbos-postgres the children hold no hook at all (CP-299); contrast: the CF DO runs each child and companion in its own DO (materialize steps 2 and 2b refuse in-process children, steps 3 and 4 rewrite them to DO stubs, runtime-cloudflare/src/do/drive/materialize.ts and durable-object-agent-base.ts:4254), so it is not affected; its host cells pass; related: the contract, docs/guide/sub-agents.md:2010 (a sub-agent's hooks fire independently of the parent's, the same on every runtime); the ruling (Joe, 2026-11-09) has two halves: an agent's own hooks (agent.hooks) fire only for its own run, on every runtime, a behaviour change in the release notes, no opt-in; execution-level hooks (passed to execute() or resume()) still span the whole run tree, and a resume carries them to children too, so the fix must not strip them from children; severity S1 (owner, from the measured state write): a root hook's state write lands in another session's state, and a root hook's side effects run once per descendant | source, live-repro, peer-reported | verified | open |
| CP-302 | S1 | runtime-js: a continued companion turn fires no start signal and writes the previous turn's fields back over its own: the re-consult's loop runs on the companion session read BEFORE its run-start commit, which keeps turn A's turnId and turnStart, so onAgentStart never fires for the new turn, and every later save writes turn A's turnId and committed turnStart back over the turn the run-start commit stored. | runtime-js/src/run-loop.ts:10473-10490 and :10637-10650 (origin/main 07d932612e) (the continued turn is planned as a new turn, turnId childRunId, but the loop runs on childPersisted, which keeps turn A's turn fields); runtime-js/src/run-loop.ts:1264 (origin/main 07d932612e) (owesStartWork is false for turn A's committed turnStart, so neither onAgentStart nor the start commit runs); store-postgres/src/state/postgres-state-store.ts:1225, :1281 and :1314-1315 (origin/main 07d932612e) (every save binds turn_id and turn_start from the state it is given: saveState, commitState and saveStateAndPromoteStaging); conformance: WE-2's records (record-only): hooks.companion-own-hooks-run .continue-hooks-ran fails on js-memory, js-redis and js-postgres (no onAgentStart in the continue run), and companion.continued-turn-fields-persisted fails .turn-id-is-consult and .turn-start-is-consult on the same lanes (after turn B the persisted turnId and turnStart are turn A's); CF Workflows passes; Temporal and DBOS store no turn fields (declared gaps, per-turn-entry-state); inferred: a retry of the continued turn is not harmed (companion.continued-turn-retry-uses-consult-input passes: retry plans from the run's own checkpoints); the corrupted fields reach owesStartWork on a recovery and turnInputMessages; contrast: the CF DO's companion continue is the child DO's own entry, which stamps its turn fields at its run-start commit; the in-process re-consult never runs on the DO; related: the contract, docs/guide/hooks.md:71-79 (every run fires exactly one start signal; a child's new turn is onAgentStart); CP-296 (the turn-field invariant); severity S1: a start hook is silently skipped and persisted turn fields are overwritten with another turn's | source, live-repro, peer-reported | verified | open |
| CP-303 | S2 | runtime-js and the CF DO record a throwing beforeTool hook as a FAILED tool row and let the run continue, on the regular step and in the approve drain, against the documented contract that a hook error fails the run. | runtime-js/src/run-loop.ts:4803-4836 (origin/main fc97182526) (the regular step: runPreparedBody's catch turns a beforeTool throw into a failed row and a tool_end, and the run goes on; its comment, :4807, says hook errors fail the tool, the one dissent from the documented contract); runtime-js/src/execution/tool-executor.ts:246-255 (origin/main fc97182526) (the approve drain's helper invokes beforeTool outside its only try, :201-203, so the throw reaches the drain's catch, js-agent-executor.ts:5069 onward, which records a failed row); conformance: approval.approved-tool-before-hook-throw-fails-typed on js-memory, js-redis and js-postgres fails .run-fails-typed (the run completes with a failed row carrying the hook's message) in WA's RM-88 slots slot-rm88r and slot-rm88f, 3 of 3; DBOS fails the run with the hook's error (CP-304's retry aside) and Temporal does after RM-88; related: the contract, docs/guide/hooks.md:416-420 and :1091, docs/internals/hooks-system.md:211 and :791-797 (hook errors fail the run, by design; catching and logging was considered and rejected); the ruling (orchestrator, 2026-11-09): the documented contract stands, a throwing beforeTool fails the run typed, never retried, on every runtime and path; the comment at run-loop.ts:4807 is corrected with this fix; paused under features-first, and the runtime-js change goes to Joe as a behaviour change when it resumes; contrast: the CF DO runs runtime-js's executor and is affected by inheritance; severity S2: the model is told the call failed and the run goes on, so a guard hook meant to stop the run does not | source, live-repro, peer-reported | verified | open |
| CP-304 | S2 | Temporal, DBOS and CF Workflows retry a throwing beforeTool hook as if it were a transient failure, on the regular step and (DBOS, CF Workflows) in the approve drain, so the hook's side effects repeat and the run's failure is delayed, against the documented contract that a hook error fails the run. | runtime-temporal/src/activities.ts:2873-2879 and runtime-temporal/src/workflow.ts:339-344 (origin/main fc97182526) (the regular step's beforeTool sits outside the activity's inner try and is not non-retryable, so the activity policy retries it, maximumAttempts 3, before the step fails); runtime-dbos/src/steps/execute-tool.ts:341-348 and runtime-dbos/src/config.ts:15-21 (origin/main fc97182526) (beforeTool outside the step's try, so the tool retry policy, maxAttempts 3, re-runs it); runtime-cloudflare/src/steps.ts:5552-5553 and runtime-cloudflare/src/workflow.ts:5961 (origin/main fc97182526) (runToolCall invokes beforeTool with no catch inside a step.do with no config, so the engine's default retries re-run it, up to 6 fires, about 32 s under Miniflare and about 310 s at production's 10 s base by reading); conformance: approval.approved-tool-before-hook-throw-fails-typed on cfw-workflows-d1 and dbos-postgres fails .before-tool-fires-once (6 and 3 fires) in WA's slots slot-rm88p and slot-rm88f, 3 of 3; both then fail the run with the hook's error; the CF Workflows approve drain also leaves the approved call pending; related: the contract, docs/guide/hooks.md:416-420 (the run fails with the hook's error, with no retry clause); the direction (orchestrator, 2026-11-09): a throwing beforeTool is never retried on any runtime, and DBOS's tool retry policy keeps covering the tool body (runtime-dbos.md:127-137 says so in the fix); widened to every runtime's regular path by the owner's table (2026-11-09); paused under features-first; severity S2: a hook's side effects repeat and the failure is delayed; no result is lost | source, live-repro, peer-reported | verified | open |
| CP-305 | S1 | A tool whose body SUCCEEDED was recorded as FAILED when its success tool_end stream write failed, on runtime-js (so the CF DO), DBOS and Temporal (a one-shot fault): the real result was discarded and the model told the tool failed, inviting a repeat of a done side effect. | runtime-js/src/execution/tool-executor.ts:833 and :871 (origin/main 07d932612e) (the success tool_end write sat inside the body's try, so a failed write reached the catch and recorded the call failed); runtime-dbos/src/steps/execute-tool.ts:463-475, :511 and :562-568 (origin/main 07d932612e) (the same in the DBOS tool step: a failed success write returned ok false with the write's error); core/src/store/best-effort-frames.ts:42 and :63 (as on branch reliability-cf-wa-cp305 82fc44d022) (fix: CP-305, WA: writeFrameBestEffort and bestEffortFrameWriter write a frame best-effort, a failed write logged with its chunk type and call id, never its content, while a run fence still throws); runtime-js/src/execution/tool-executor.ts:155, runtime-js/src/run-loop.ts:1691, :1832 and :4339, runtime-js/src/js-agent-executor.ts:2956, runtime-dbos/src/steps/execute-tool.ts:250 and runtime-cloudflare/src/steps.ts:6143 (as on branch reliability-cf-wa-cp305 82fc44d022) (fix: every frame written after a call's result is decided is best-effort on runtime-js, its entry drains and re-attach frames, DBOS and CF Workflows; Temporal's approve drain by RM-88's bestEffortFrameWriter, its regular activity already catching); conformance: fixed. tools.tool-end-write-keeps-result and approval.approved-tool-end-write-keeps-result fail .real-result-kept and .run-continues on js-memory, js-redis, js-postgres and dbos-postgres at the RED 8605187196 and pass at the fix, 3 of 3, in WA's slot-cp305g (132 checks, 1 failed: a unit count predicted 6 of 7 passing tests, all green; FIX f136e8cf7f, rebased onto main fc97182526 as 82fc44d022); the Temporal lanes pass at both heads, guards; cfw-workflows-d1 is a declared gap (no chunk-write fault seam, FU-CONF-CFW-CHUNK-WRITE-FAULT), CF Workflows proven by its units; the CF DO by inheritance (FU-CONF-DO-CHUNK-WRITE-FAULT); runtime-js's unit 'a failing tool_end write' of new-turn-pending-settlement.test.ts was flipped on purpose (the run completes); related: the ruling (orchestrator, 2026-11-09): a successful body's result is recorded as its real result whatever happens to its stream frames, the failure logged, the stream re-anchoring on the next read (the save-first spec, a run's own stream never decides its outcome); FU-CFW-STOPPED-CALL-FRAME (CF Workflows' stopped-call tool_end can still fail its step, a residual of this rule); severity S1 (orchestrator, 2026-11-09, outcome first): the model is told a completed side effect failed, which invites a repeat | source, live-repro, peer-reported | verified | fixed (8 cells) |
| CP-306 | S1 | A CP-297 regression on the CF DO, fixed before it shipped: an owed-answer wake whose run-start entry was rejected re-triggered itself through the release tail's owed re-check, an in-process loop that never spent the WakeRecoveryRecord ladder, so a session whose continuation could not start never ended and its DO could not be disposed. | runtime-cloudflare/src/do/drive/run-drive.ts:2062-2071 and :1960 (origin/main fc97182526) (a wake whose entry fails still runs the release tail, afterRelease); runtime-cloudflare/src/do/durable-object-agent-base.ts:2644-2697 (as on branch reliability-we2-cp297 84737aac5f) (the regression: CP-297's commit 1e12abf476, within !796's range a6a37e4dc9..84737aac5f, merged via 219c8cf969) (afterRelease runs continueOwedAfterRelease, which starts another owed entry for the still-owed answer); runtime-cloudflare/src/do/durable-object-agent-base.ts:673, :2700, :2708 and :2717 (origin/main fc97182526) (fix: the owed re-check runs at most once per suspended run, the run marked before its continuation starts and a repeat skipped, continuation.owed_recheck_skipped, so a failing owed entry falls back to the alarm ladder); conformance: fixed. wake.paused-submit-lost-bounded on cf-do (host) timed out at 255 s 3 of 3 at the batch-17 head 17f946dc85, passed 3 of 3 with CP-297's DO hunks reversed and 3 of 3 at main 07d932612e (the control), in WE-2's w-b17wake2; it passes 3 of 3 at the fix (w-cp306, e181c2e8ab, product equal to 4ba8373316); the unit fails 2 of 4 with the pre-fix DO source (the re-check ran 6 times, expected once) and passes 4 of 4; the cell prints the DO's warn and error lines on every exit; inferred: a residual, by reading: if a DIFFERENT wake for the run fails first, its release still runs the re-check once, one extra rejected entry per suspended run per DO lifetime, bounded, the alarm ladder owning the rest; related: CP-297 (the owed re-check this guards), the regression rule (a cell green before an MR and red after is that MR's regression, its own id); severity S1: a session that could never end and a DO that could not be disposed | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-307 | S2 | A sub-agent child's terminal error chunk reaches its parent's chat response as an error event, so a stock useChat (which stops reading at the first error event) reads the parent's turn as failed although the parent records the child's failure as its tool result and its session ends completed (no errorDetail). Two writers put it there: the remote dispatch proxies every chunk of a REMOTE child onto the parent's stream (on the CF Durable Object every sub-agent is a remote DO stub), and on runtime-js a LOCAL ephemeral sub-agent runs on the parent's stream and writes its own error chunk there as a guest chunk (its session, its run). The ai-sdk transformer forwards an error chunk without asking whether it is a child's. CP-215 is the companion form of the same symptom. | runtime-js/src/execution/remote-subagent-executor.ts:740-756 (origin/main 7b92366025) (runtime-js and the CF Durable Object: every chunk event of a remote child is written to the parent stream with remoteSource, the error chunk included; no chunk type is held back); core/src/orchestration/remote-sub-agent-dispatch.ts:707-725 (origin/main 7b92366025) (the same proxy write in the core dispatch, which runtime-temporal imports for its remote children); ai-sdk/src/transformer/stream-transformer.ts:717 and :1530-1556 (pre-fix, origin/main 7b92366025) (an error chunk became data-error-metadata plus an error event; isChildChunk at :971 was not consulted); runtime-js/src/run-loop.ts:5434-5440 (origin/main 7b92366025) (a local sub-agent child state takes streamId: parentState.streamId, so its chunks, its terminal error chunk included, are written to the parent stream stamped with the child session and run); runtime-cloudflare/src/do/drive/materialize.ts:161-166 (origin/main 7b92366025) (the DO rewrites every sub-agent tool to a DO-stub remote tool); conformance: WE-2's record run w-mr4ac3 (record-only, under mutation M1, which lets a disabled sub-agent run): tools.disabled-call-refused-host .subagent-tool-call and .subagent-call on cf-do read the parent's chat response as { terminal: failed, source: stream:error, error: 'Something went wrong.' } with the parent's persisted errorDetail null, while the parent's log holds the child's failure as its tool row (Remote agent failed: …). The same cell's companion form on cf-do ends stream:finish (a DO companion runs in its own DO and its chunks are not proxied); conformance: WE-2's record-only M1 phase of slot w-mr4aj (10d9f73993, 2026-11-09, one run): the same two cf-do ids read the parent's response as { failed, stream:error, 'Something went wrong.' } while the parent's session is persisted completed with errorDetail null, and its row holds the child's failure (helixToolFailed). The child's own failure there is a mutation artefact (its model had no script); the parent-side mismatch is the finding; conformance: WE-2's red-first slot w-cpr at 2eae8504b9 (main 7b92366025 plus the new tests, no fix), two runs, identical: subagent.child-error-not-parent-error-host fails .no-child-error-event and .turn-completes on cf-do and js-chat-host (host), with .child-failed and .parent-session-completed passing; subagent.child-error-not-parent-error-client and -client-stock fail .finish-clean and .no-error on cf-do and js-chat-host, each also on ai-newest; live: w-cpr's capture (runtime-js, a failing ephemeral LOCAL sub-agent, the parent stream read with getAllChunks): the error chunk carries the child's agentId and its own runId with no remoteSource (a guest chunk), and the transformer at main emitted data-error-metadata, error and then finish with finishReason error while the parent's session ended completed with errorDetail null; ai-sdk/src/transformer/stream-transformer.ts:732-745 (!820, as on origin/main 41ea410715) (the fix: in case 'error', a child's chunk (fromChild: a remote child's remoteSource chunk or a guest's) is dropped from the parent's response with a debug log at :722-728, before transformError at :729; the stored chunk is unchanged, and the stream's own error chunks still become an error event); conformance: fixed. WE-2's judged slot w-cpj (2026-11-10) at 741e01ce8b, 3 of 3 runs identical to the fix record w-cpf: subagent.child-error-not-parent-error-host passes on cf-do and js-chat-host (host); subagent.child-error-not-parent-error-client and subagent.child-error-not-parent-error-client-stock pass on cf-do, js-chat-host, cf-do+ai-newest and js-chat-host+ai-newest (client); subagent.child-error-chunk-kept-host passes on cf-do (the child's error chunk is still stored). Under MF (the guard removed) every row equals the red-first record w-cpr; under MW' (the guard removed and runtime-js's remote proxy dropping error chunks instead, the tempting wrong fix) the cf-do rows pass but .child-error-stored fails and every js-chat-host row stays red, as recorded first in w-cpm. The transformer unit child-error-chunk.test.ts passes 6 of 6 and fails exactly its 3 red-first cases under MF; related: CP-215 (the same symptom for a runtime-js persistent companion, which writes into the parent's stream as its own); its disposition applies: a child's terminal failure reaches the parent as the call's result, never as an error event on the parent's chat response. Ruling (orchestrator, 2026-11-09): CP-307 and CP-215 take one fix in the transformer, a child's error chunk never becomes an error event on the parent's response; first in the post-features reliability queue | source, live-repro, peer-reported | verified | fixed (11 cells) |
| CP-308 | S2 | CF Workflows: the client-tool deadline watcher can sleep until a time already past, and the engine's refusal ends the watcher uncaught, so a lapsed call's continuation never runs and the run stays paused past its deadline. Each round records now inside its deadline-target step, then sleeps until min(deadlineAt, now + maxSleepMs) outside any step. When that target has passed by the time the sleep is called (a short deadline on a loaded runner), Miniflare's sleepUntil throws 'You can't sleep until a time in the past', nothing catches it, and the deadline instance errors with nothing logged by Helix. | runtime-cloudflare/src/client-tool-deadline.ts:248-260 and :265-270 (origin/main 7b92366025) (deadline-target records now() inside its step; the chunk loop sleeps until min(target.deadlineAt, at + maxSleepMs), advancing from that recorded now, never from the clock at the sleep); runtime-cloudflare/src/workflow.ts:849-870 (origin/main 7b92366025) (the __deadline__ body awaits runClientToolDeadlineWatch with no catch; its rejection ends the instance); live: !812's MR pipeline 3047755811, job 17883359734 (test:workflows, refs/merge-requests/812/merge at ef8de80c4e): runtime-cloudflare's cfw-step-executor-import.wf-noiso.test.ts (clientToolTimeoutMs 1000) failed 'CP-298, the deadline watcher: a lapsed call's continuation completes the turn' with the session 'paused', not 'completed' (:170), after the job log's 'Uncaught (in promise) Error: You can't sleep until a time in the past' at client-tool-deadline.ts:268 via workflow.ts:849. The case passed on batch 18's final pipeline and on both of !810's, so the trigger is timing; inferred: whether production Cloudflare Workflows also refuses a past sleepUntil is unmeasured; the same window opens there whenever the gap between the target step and the sleep outlasts the remaining deadline. A later reader still expires the call (the reader backstop) and starts the owed continuation, so the run recovers only when something reads the session; related: CP-298 (the same watcher's continuation step; this failure is before it); CP-138 (the watcher's introduction). Fix direction (WA, via the orchestrator, 2026-11-09): skip the sleep when its target is not in the future, decided inside a step so a replay takes the same branch | source, live-repro, peer-reported | partly-inferred | open |
| CP-309 | S2 | runtime-js in a plain Cloudflare Worker (not a Durable Object): every run fails at its first lease start, before any step, with workerd's 'Illegal invocation'. The run lease stores the global setTimeout / clearTimeout as its own fields when no timer is passed in, then calls them as methods of the lease, so the native timer runs with the lease as its receiver. Node tolerates that and workerd refuses it. The CF Durable Object takes no JS lease, so it is not affected; no lane runs runtime-js in a plain Worker, which is why nothing caught it. | core/src/run/run-lease.ts:78-79 (origin/main 4e26f6cb0b) (setTimer = opts.setTimer ?? setTimeout; clearTimer = opts.clearTimer ?? clearTimeout, stored as fields); core/src/run/run-lease.ts:141, :152-153, :205, :211 and :215 (origin/main 4e26f6cb0b) (each call is this.setTimer(...) / this.clearTimer(...), so the native timer gets the lease as this); runtime-js/src/run-start.ts:970 (origin/main 4e26f6cb0b) (startRunLease, reached from JSAgentExecutor.startLease on every entry: executeEntry, enterNewTurn); live: WA's Save-First section 13 bench (2026-11-10), a plain Worker running JSAgentExecutor over D1StateStore under wrangler dev --local (workerd 1.20260925.1): the first execute() failed before any step with 'TypeError: Illegal invocation: function called with incorrect this reference' at RunLease.schedule, RunLease.start, startRunLease (run-start.ts:970), JSAgentExecutor.startLease, executeEntry, enterNewTurn; reproduced on every first request; related: workaround until the fix: runLease: { setTimer: (f, ms) => setTimeout(f, ms), clearTimer: (h) => clearTimeout(h) }. WA's scope scan of packages/*/src found no other native stored on an object and called as a method (the ai-sdk opts.fetch ?? fetch sites call through a local const). The same class as an earlier detached service-binding fetch in the remote-agents transport, fixed with a call-time wrapper. Verified at main 4e26f6cb0b by the orchestrator (2026-11-10); severity S2 (owner, provisional): a complete failure of one configuration (runtime-js in a plain Worker), with a one-line workaround; the configuration is not documented as supported, and no shipped example uses it; core/src/run/run-lease.ts:88-89 (!821, as on origin/main 41ea410715) (the fix: the default timers are bind-free arrow wrappers, (fn, ms) => setTimeout(fn, ms) and (handle) => clearTimeout(handle), so the native timer is never called with the lease as its receiver; an opts.setTimer / opts.clearTimer still wins); live: fixed. WA's real-workerd smoke at the fix (wrangler 4.141.0, workerd 2025-10-11, wrangler dev --local over the D1 bench Worker): one execute() over D1StateStore in a plain Worker completed past the run lease (smoke d1 1 MB: completed, 2 samples); without the fix the same leg answered 500 Illegal invocation at RunLease.schedule (WA slots sftb2 and sftb3) | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-310 | S1 | DBOS: a persistent companion's result can be lost after its parent's session is deleted and re-created under the same id. The companion's first-spawn run id is derived from its session id alone, so the re-created conversation reuses the deleted conversation's run id; a status write still pending from the deleted conversation's companion workflow (failed 'Run not found' while the run was gone, then retried by DBOS) lands on the NEW run and marks it terminal before that run commits its step. The new turn's verdict is never committed, the companion session reads completed with no output, and the parent is told the companion completed with no result. | conformance: session.delete-cascade on dbos-postgres (executor) failed exactly .parent-gets-new-verdict ('the companion's turn-2 verdict in the parent's transcript: expected [] to not deeply equal []') twice in CI, with .companion-input-fresh passing and the cell taking about 2 s: job 17887475508 (pipeline 3048376896, !819 at 27351ee208, 345 of 346 passed) and job 17882623187 (pipeline 3047618338, the !810 merge ref at 40dc1a2cd7; its retry 17883070349 passed). Neither MR touches runtime-dbos, core or store-postgres. The same cell passed on the other pipelines of those days; related: runtime-dbos/src/steps/execute-companion-tool.ts:1142-1195 (origin/main 4e26f6cb0b) (pollChildSessionTerminal then turnEnded, the blocking spawn's read: it reports the run's word and the session's output as they are, so it passes the stale completed-without-output state to the parent; the first-sighting suspect, now the reader, not the cause) and FU-DBOS-COMPANION-READS-RUN-ANCHORED; related: RM-75 (fixed; the same cell is its proof). Not RM-75's cause: its pre-fix signature is .companion-input-fresh and .parent-gets-new-verdict both red in 5 of 5 runs with turn 2 waiting about 31 s on the re-attached workflow; here the salted ids work (the companion ran fresh, fast) and only the verdict is missing. Owner ruling, orchestrator agreed (2026-11-10): a new record, RM-75 not reopened; conformance: self-reporting since packages/e2e/src/conformance/scenarios/session-delete.ts (!823, as on origin/main 41ea410715) (owner, batch 19): a failing .parent-gets-new-verdict now prints a CP-310 report. Proven by the owner's slot cp310 (run-20261110-033813, forced fail on the check's lookup, local only): on a healthy run the report shows the parent's companion__spawnAgent row as status completed with output verdict VERDICT-ROUND-TWO, and the companion completed with that output, its checkpoints input, step and state (completed) under its run r1. How to read a real red: LOST (the top severity) when the companion holds the verdict in its output but the parent's row lacks it, or reads completed with no output; NEVER PRODUCED when the companion's own output lacks it; severity S1 (owner, 2026-11-10, from the self-reported sighting): a sub-agent result is lost silently and the parent is told the companion completed; reached by deleting a session and re-creating it under the same id while its persistent companion is still finishing; conformance: third sighting, CI job 17905246024 (!823, batch 19, merge pipeline 3050652825, test:conformance:dbos), the first with the self-report: the parent's companion__spawnAgent row is {"name":"critic","status":"completed"} with NO output; the companion's ref and session read completed with no output, its only checkpoint is its run r1's input commit (status active), and its run record r1 reads completed; .companion-input-fresh passes (its model ran on the turn-2 input). The same job logs, during the cell, a DBOS step retried twice with 'Run not found: …-s-agent-critic-r1'; runtime-dbos/src/steps/execute-companion-tool.ts:875-877 (origin/main 4e26f6cb0b) (a first spawn's run id is the child session id plus -r1: not salted, unlike its workflow id companionSpawnWorkflowId at :874, RM-75's fix); store-postgres/src/state/postgres-state-store.ts:3107-3131 (origin/main 4e26f6cb0b) (updateRunStatus throws 'Run not found' for a run that does not exist; DBOS retries the failing step); related: the mechanism, traced by WA (2026-11-10) and reproduced by the forced-window witness below: the deleted conversation's companion workflow finalized its run after deleteSession had removed it; the write threw Run not found and DBOS retried the step; once the re-created conversation's companion run existed under the SAME id, the retry found it live and ended it completed before its turn committed, a terminal run status is sticky, so the new turn's step and K5 commits were refused, and the parent's blocking-spawn poll (turnEnded) read completed with no output. Intermittent because the retry had to land after the re-creation and before the new run's step commit; runtime-dbos/src/persistent/child-session-id.ts:63-90 (!823, as on origin/main 41ea410715) (the fix, first layer: companion turn run ids keyed on the parent RUN, like their workflow ids since RM-75: companionSpawnRunId at :68 is the session id plus r1 plus the parent run id, with companionRespawnRunId and companionMessageRunId), used at runtime-dbos/src/steps/execute-companion-tool.ts:874-878 (!823, as on origin/main 41ea410715); runtime-dbos/src/steps/run-lifecycle.ts:170-200 (!823, as on origin/main 41ea410715) (the fix, second layer: finalizeRun never finalizes a run its writer did not create: a missing run is a logged no-op, not a retried throw, and a run whose ownerToken is another execution's is left untouched with a warn; its read-then-write window is FU-DBOS-ATOMIC-EXPECT-OWNER-TOKEN); conformance: fixed. WA's slot-cp310 (2026-11-10, 41 checks, 0 failed) at the fix af276bd0d7: session.delete-cascade passes on dbos-postgres 5 of 5; the runtime-dbos unit suite 1280 of 1280 and the full DBOS integ suite 290 passed. The fourth sighting before the fix was the retried job 17907273417 on !823, which failed the same id again | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-311 | S2 | A chat stop pressed right after a send, before the send's run exists, is lost: the stop answers not_stoppable (nothing to stop), the send's entry then commits its run, and the whole turn runs and is billed although the user stopped it. The client stops reading, and the reply appears after its next re-anchor. On runtime-js nothing cancels an entry that has not committed its run; on the CF Durable Object only an entry already registered with its run id is cancelled, so a stop that reaches the DO before that point is answered as if no turn were coming. | ai-sdk/src/react/use-helix-chat.ts:746-776 (origin/main 4e26f6cb0b) (useHelixChat stop() with serverStop posts the server stop through requestServerStop, ai-sdk/src/client/server-stop.ts:50-66, its body only { kind, reason } to the chat id URL: nothing names the send in flight); core/src/run/stop-run.ts:145-151 (origin/main 4e26f6cb0b) (no live run: not_stoppable with the session status, nothing recorded; the owner probe cancelPendingEntry hook at :32 runs first, where a runtime implements it); runtime-js/src/execution/in-process-runs.ts:125-175 (origin/main 4e26f6cb0b) (runtime-js: the stop owner probe has executing and signal only, no cancelPendingEntry, so a stop before the run-start commit cancels nothing); runtime-cloudflare/src/do/durable-object-agent-base.ts:1930-1942 (origin/main 4e26f6cb0b) (the CF DO cancels a pending entry only when a drive is registered, unsettled and holds a run id that is not yet the current run; a stop that arrives before that registration, or before the entry has its run id, returns without cancelling); live: an app's report (peer-reported via the orchestrator, 2026-11-10): on its deployed Durable Object (runtime-cloudflare 23.1, also seen on batch 17), useHelixChat stop() pressed about 0.3 s after send is answered nothing to stop, the send's entry lands and the whole turn runs; the client detaches and the reply streams back after a re-anchor. Repro: send, then stop() within about 300 ms of the POST; related: CP-150 / FU-REMOTE-STOP-BEFORE-START-TOMBSTONE (the same race for a remote child's stop before its /start); CP-79 (DBOS: a stop before the in-body append drops the input instead, the opposite outcome); FU-CLIENT-STOP-RUN-TARGET and K73 (a stop that names its run; here no run exists yet). Fix direction (orchestrator, 2026-11-10, SDK-owned, runtime-js and the DO first): a keyless stop ends any entry the host has already received for the session but not yet run-started (no session-wide tombstone for future entries); a keyed stop (the product client) does the same and leaves a short-lived tombstone on the named request key, so a send that reaches the host after the stop also ends as a stop (run_stopped, with SDK-7's no-reply notice); a stock useChat stop never reaches the server (orchestrator ruling, 2026-11-10); severity S2 (owner, provisional): a user stop is lost and the stopped turn runs to completion and is billed; reachable by any quick Stop after Send; conformance: WE-2's red-at-main record w-c311r v3 and its re-run (cells head 325b31b36a on main 4e26f6cb0b, rows byte-identical; rebased unchanged onto 41ea410715 as 7f3b366956): on cf-do and js-chat-host (host) stop.entry-before-run-start-host fails .input-kept, .next-turn-notice, .no-model-call, .send-ends-interrupted and .stop-answers-stopping (cf-do 409 state_not_stoppable), stop.entry-before-run-start-first-turn-host the same without .next-turn-notice (404 on a fresh session), stop.keyed-overtakes-send-host .adopts-reserved-run, .input-kept, .no-model-call, .send-ends-interrupted and .stop-answers-stopping (cf-do 409; js-chat-host 400, handleChatStop's strict schema refusing requestKey); client.stop-before-run-start-client fails .input-kept, .no-model-call and .ui-marked-no-reply and client.stop-overtakes-send-client .input-kept and .no-model-call on cf-do, js-chat-host, cf-do+ai-newest and js-chat-host+ai-newest; stop.keyed-tombstone-own-key-host fails .stop-answers-stopping on both hosts (w-c311f 0b, two runs, at 6f108e64d5). The guards pass: stop.keyless-nothing-pending-host, the other-key half of stop.keyed-tombstone-own-key-host and client.stock-stop-stays-local-client-stock; core/src/run/pending-entries.ts:150-153, :193, :212-228 (!828, as on origin/main f4f4028489) (the fix: an entry the host has received is pending until its run-start commit; a keyless stop marks it (markPending), a keyed stop resolves its request key (resolveKeyed) and leaves a short-lived tombstone for a send that arrives after it, which the send's receipt consumes (receive)); runtime-js/src/js-agent-executor.ts:6067, :6090-6093, :6356, :6551-6552 and :2668-2694 (!828, as on origin/main f4f4028489) (runtime-js: the host reservation from receipt, execute() receiving under its pre-minted run id, the run adopting the reserved id, the entry ending as a stop when its pending entry was marked; stop() answering from the pending entries, 202 stopping); runtime-cloudflare/src/do/durable-object-agent-base.ts:4532-4533 and runtime-cloudflare/src/do/drive/stop.ts:183-205 (!828, as on origin/main f4f4028489) (the CF DO: a chat /start is received on arrival; a fresh DO's stop answers from the pending entries and resolves a keyed stop); ai-sdk/src/react/use-helix-chat.ts:775 and :810, ai-sdk/src/handler/handle-chat-stop.ts:73 and ai-sdk/src/handler/handle-chat-stream.ts:387 and :424 (!828, as on origin/main f4f4028489) (the client names its in-flight request key on stop and probes the snapshot for the stopped send; handleChatStop accepts requestKey; handleChatStream reserves the entry on receipt); conformance: fixed. WE-2's judged slots at the fix (75b32a6930, rebased with no product change as dc1eb9e466): w-c311j and the cells-only w-c311x, every cell all-pass, the cf-do / js-chat-host rows byte-identical across 5 runs and the ai-newest rows across 4. Mutations, each judged exact in w-c311j: M-APPLY, M-DO-RECEIPT, M-TOMB, M-TOMB-SCOPE, M-KEYLESS-TOMB, M-CLIENT-KEY, M-FRESH-JS, M-FRESH-DO, M-JS-ANSWER, M-KEYED-ANSWER, M-SCHEMA and M-PROBE; client.stop-before-run-start-client.ui-marked-no-reply is not judged under M-APPLY / M-DO-RECEIPT (CP-313's client-mark race, retired by its fix) nor under M-PROBE on js-chat-host (a race: M-PROBE's proof is its cf-do row, and on js-chat-host the probe's necessity is unproven). The negatives stop.keyless-nothing-pending-host and the other-key half of stop.keyed-tombstone-own-key-host pass, so a keyless stop leaves no session-wide tombstone and a tombstone ends only its own key; client.stock-stop-stays-local-client-stock passes as a guard | source, live-repro, peer-reported | verified | fixed (18 cells) |
| CP-312 | S3 | useHelixChat: retry() after an explicit stop() re-sends the stopped turn under the SAME request key, so the server treats it as a duplicate of the stopped run and replays that run's ended stream: the user's Retry shows the stopped turn again and the model is not called. The hook keeps one key per send and only a deliberate regenerate mints a new one; nothing rotates the key when the user stops. | ai-sdk/src/react/use-helix-chat.ts:796-806 (origin/main 4e26f6cb0b) (one request key per send, kept in lastSendKeyRef); ai-sdk/src/react/use-helix-chat.ts:822-829 (origin/main 4e26f6cb0b) (regenerateSameKey re-sends under lastSendKeyRef.current), with :811-819 (regenerateWithNewKey: a deliberate regenerate mints a new key) as the contrast; ai-sdk/src/react/use-helix-chat.ts:902 (origin/main 4e26f6cb0b) (retry() with no resync configured: return regenerateSameKey()); the explicit stop at :746-776 never touches lastSendKeyRef; ai-sdk/src/handler/handle-chat-stream.ts:534 (origin/main 4e26f6cb0b) (a request key equal to the current run key goes to keyedDuplicateResponse) and :2122 (origin/main 4e26f6cb0b) (an ended run is replayed, replayTerminalStream); ai-sdk/src/react/use-helix-chat.ts:908-912 (origin/main 4e26f6cb0b) (with resync configured, retry() after a stop reads the snapshot, finds the stopped turn held (serverHoldsTurn, :908-912) and re-anchors, so it does not re-run the turn either; so it does not re-run the turn either. Ruling (orchestrator, 2026-11-10): after an explicit stop, retry() re-runs the turn on both branches as a new request under a new key, so the resync branch changes too; related: peer-reported by WE-2 (2026-11-10), source only, no live probe; the owner re-read every cited line at main 4e26f6cb0b. CP-311 (a stop before the send's run exists; its keyed tombstone would also end a same-key re-send within its window). CP-232 (a different retry() defect: a stale resync error). Fix direction (orchestrator, CP-311 ruling 4 and the ruling above): useHelixChat rotates the request key after an explicit stop and re-sends on both branches; fixed in the CP-311 work; severity S3 (owner, provisional): a user Retry after Stop silently does nothing useful (the stopped turn is shown again); no data is lost and nothing is billed; conformance: WE-2's red-at-main record w-c311r v3 and its re-run (rows byte-identical; rebased unchanged as 7f3b366956): client.retry-after-stop-reruns-client and client.retry-after-stop-reruns-resync-client fail .retry-reruns on cf-do, js-chat-host, cf-do+ai-newest and js-chat-host+ai-newest; ai-sdk/src/react/use-helix-chat.ts:780 and :958-959 (!828, as on origin/main f4f4028489) (the fix: an explicit stop is remembered, and retry() after it re-sends as a new request under a new key, regenerateWithNewKey, on both the resync and the no-resync path); conformance: fixed. WE-2's judged slots at the fix (75b32a6930, rebased with no product change as dc1eb9e466): w-c311j and the cells-only w-c311x, every cell all-pass, the cf-do / js-chat-host rows byte-identical across 5 runs and the ai-newest rows across 4. M-RETRY (the branch removed) reds .retry-reruns on both forms and both hosts, the one mutation of the one branch | source, peer-reported | verified | fixed (8 cells) |
| CP-313 | S2 | useHelixChat: an explicit stop() pressed before the page received any of the reply marks the turn stopped with no reply (SDK-7's client mark) even when the server kept a partial reply, so the live page and a reload disagree. With resync, the mark is written once the re-anchor resolves, but the re-anchor starts the resumed stream without awaiting it; the stream then delivers the kept partial under a turn already marked no-reply, and nothing clears the mark. Without resync, the page marks no-reply at once and shows nothing, while a reload shows the partial reply with no mark. | ai-sdk/src/react/use-helix-chat.ts:761-771 (origin/main 4e26f6cb0b) (explicitStop: an interrupt the server took marks the last user message no-reply at once without resync, :762, or after the re-anchor resolves with resync, :764-770); ai-sdk/src/client/resync.ts:367-376 (origin/main 4e26f6cb0b) (reanchorNow: when the snapshot reads the run still live it starts control.resume and does not await it, so the re-anchor resolves before the resumed stream delivers anything); ai-sdk/src/client/no-reply-stop.ts:24 (origin/main 4e26f6cb0b) (markNoReplyStop: the only writer of the client mark; no path clears it when content arrives); related: #181 keeps the interrupted partial reply (an interrupt only), so a stop after the model began writing leaves an assistant row; the snapshot then marks the reply, not the user message, which is what a reload shows. Traced in code by WE-2 (2026-11-10), routed by the orchestrator; live-proven since by the cells below. Not CP-311: that record's window (a stop before the run exists) neither opens nor widens this one; related: fix direction (orchestrator, 2026-11-10; fixer WE-2, after CP-311): mark only once the resumed stream has ended with no assistant content after the user message, or clear the client mark when content arrives; severity S2 (orchestrator and owner): a wrong UI state live, while the server holds the truth and a reload corrects it; ai-sdk/src/react/use-helix-chat.ts:788-835 (fix evidence, !837, as on origin/main f4f4028489) (an explicit stop marks the turn only once it has settled on the page: with resync, after the re-anchor and, when it resumed the live run, after that resumed stream ran to its end while the re-anchor is still current; without resync, no live mark; the mark goes through the Chat live messages, never the last render); ai-sdk/src/client/no-reply-stop.ts:52-67 (fix evidence, !837, as on origin/main f4f4028489) (markNoReplyStopForTurn: only the stopped turn, the user message at the stop count, and nothing when a reply with content follows it or a newer user message exists); ai-sdk/src/client/resync.ts and ai-sdk/src/react/resync-hooks.ts (fix evidence, !837, as on origin/main f4f4028489) (the re-anchor reports whether the resume it started ended while it was current, onResume); conformance: client.stop-mark-above-partial-client (resync) and client.stop-mark-above-partial-no-resync-client (client tier, the real useHelixChat) on cf-do, js-chat-host and both ai-newest lanes: red at main plus the cells (WE-2's slot w-c313f5, two record runs rc1 and rc2 that agree; the resync form fails .live-no-false-mark and .live-matches-reload, the no-resync form .no-live-mark and .live-matches-reload), green at the fix in three judged runs per lane with identical rows (w-c313f5 cf1, cf2 and ainew1; w-c313x3 cf3, ainew2 and ainew3; again in w-b20h2 after the rebase onto batch 20, rows identical), and red as predicted under each mutation of the fix (M-RESUME-WAIT, M-ONRESUME, M-NO-RESYNC-MARK) | source, live-repro, peer-reported | verified | fixed (8 cells) |
| CP-314 | S2 | runtime-js and the CF Durable Object: a run that committed a successful finishWith step but lost its K5 calls the model again when it recovers, and the new answer replaces the output the run had already committed. The recovery that completes a run from a committed terminal step recognises a committed finish only, so a committed finishWith result is ignored and the run re-enters its loop. | core/src/orchestration/transcript.ts:471-488 (pre-fix, origin/main 41ea410715) (findCommittedFinish matches FINISH_TOOL_NAME only), used by the recovery at runtime-js/src/run-loop.ts:2294 (pre-fix, origin/main 41ea410715); conformance: W1's slot fwr2 (2026-11-10) recorded finish-with.recovery-mid-step on cf-do (host) with core and runtime-js at main 41ea410715 (e2e at 748da1995a): it fails .no-model-call-after-recovery and .output-is-committed; the model was called twice and the run ended with the woken deploy's v2:MODEL output instead of the committed v1:x; core/src/orchestration/transcript.ts:508 (!831, as on origin/main f4f4028489) (the fix: findCommittedTerminal wraps findCommittedFinish and adds the finishWith branch, the first call in emission order whose row carries a Zod-valid output marker and is neither failed nor not executed), with core/src/orchestration/finish-with-output.ts:18, :34 and :49 (same branch) (the marker schema, markFinishWithOutputRow, finishWithOutputOf); runtime-js/src/run-loop.ts:5439, :5450, :2330 and :11613 (!831, as on origin/main f4f4028489) (the live row is marked; the marker is dropped at the row cap with a warn core.finish_with.output_marker_dropped; the recovery checks the committed terminal; a pending forced phase over its own tool committed output is recorded succeeded); conformance: fixed. At the fix 748da1995a the cell passes on cf-do 3 of 3 in W1's judged slot fwr3 (one model call, output v1:x, each run), js-chat-host a declared gap. Mutations, each judged exact in fwr2: the recognizer returning nothing after the __finish__ check (m1), the marker never written (m2) and the first result row deciding (m3) each fail .no-model-call-after-recovery and .output-is-committed; re-deriving through the current agent transform on recovery (m4) fails .output-is-committed alone (output v2:x); finishWithOutputOf accepting an unparsed object (m5) fails the core unit finish-with-terminal.test.ts case 'no marker, or a malformed one, is no output (Zod, never a guess)'; related: runtime-js has no process loss at the executor tier: its recovery is the CF DO wake, so the executor tier has no cell (ruling); js-chat-host is a declared gap (no wake). CF Workflows, Temporal and DBOS are not claimed here: their recovery paths differ and are judged at FS-3, FS-4 and FS-5. Closes FU-FINISHWITH-COMMITTED-RECOVERY (named by the Final Step spec FS-1 section 7); severity S2 (owner, provisional): a committed result is silently replaced by a second model call (and that call is billed); it needs a lost K5 after the finishWith step commits | source, live-repro, peer-reported | verified | fixed (1 cells) |
| CP-315 | S2 | runtime-js: a submit for a non-blocking companion that has just paused on a client call, after its root's turn ended, can be answered caller_resumes instead of CP-250's self-wake (new_run). The companion's suspension commit writes its pending call, its paused status and the root's call ownership first; the parent's sub-session ref turns paused_awaiting_client only later, when the companion's run is recorded as suspended. A submit in between reads the ref still running, so the router refuses the self-wake and tells the caller to resume, although no caller resume continues a companion. And because the answer is already stored, the paused ref is never written and no re-check follows, so the companion stays paused on an answered call and nothing continues it. | runtime-js/src/js-agent-executor.ts:6762-6774 (origin/main 41ea410715) (startCompanionSelfWake: the self-wake needs the parent ref to read paused_awaiting_client and the companion run suspended; otherwise a suspended run answers caller_resumes); runtime-js/src/run-loop.ts:9710-9740 (origin/main 41ea410715) (recordSuspendedChild: the parent ref is written paused_awaiting_client after the companion run committed its suspension, called at :10121, :10386 and :10944); live: runtime-js integ companion-self-wake.integ.test.ts (shared case in runtime-js/src/__tests__/helpers/companion-self-wake.ts:462, '(b) a NON-blocking companion that pauses AFTER its root's turn ended') failed once on postgres in CI job 17911316241 (!824 merge pipeline 3051389731, a store-contract-only MR): the submit answered { status: 'accepted', continuation: { kind: 'caller_resumes' } } where new_run with relation self_wake was expected; the same file passed 24 of 24 in jobs 17902733930, 17905685178 and 17908772579 (batch 19's pipelines). The case waits for the companion's paused status and the root's ownership, but not for the parent ref; runtime-js/src/run-loop.ts:9690-9700 (origin/main 41ea410715) (childAwaitedCalls: the companion's awaited calls through core pendingCallSummariesOf, core/src/types/companion-pending-calls.ts:47-48 (origin/main 41ea410715), which drops a call whose answer is already stored, so recordSuspendedChild finds none, writes nothing and the parent ref stays running); runtime-js/src/js-agent-executor.ts:1977 and :10752 (origin/main 41ea410715) (CP-297's post-suspension owed re-check is started only from the executor's own run paths; an in-process companion runs through runChildLoop into core's run loop, so no re-check follows its suspension). Traced by WA (2026-11-10, read-only at 41ea410715); the owner re-read every cited line; contrast: the CF Durable Object runs each companion in its own child DO, which decides a submit from its own session (durable-object-agent-base.ts:3009 (origin/main 41ea410715)), so this window is runtime-js's alone; a cf-do control twin judges that; related: CP-250 (the self-wake contract), CP-297 (the owed re-check). Fix direction (WA, option C prime, batch 20): in the window the submit waits, bounded, for the in-process pause record and answers new_run with relation self_wake, and the owed re-check continues a stored answer, so the companion runs on; the test-helper wait ships with it. The residual, a submit from another process in the window, still answers caller_resumes but the re-check continues the companion (no strand): FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER, at most S3. The failing job was retried as 17912546292; severity S2 (owner, from the trace): an answered client call strands its companion with no error, after a fast but ordinary answer; runtime-js only; runtime-js/src/js-agent-executor.ts:478 and :7207-7209 (!833, as on origin/main f4f4028489) (the fix: the self-wake waits, bounded by COMPANION_PAUSE_RECORD_WAIT_MS, for the in-process pause record, and the companion driver's hooks run CP-297's owed re-check after the companion suspends) and runtime-js/src/run-loop.ts:9834 (!833, as on origin/main f4f4028489) (recordSuspendedChild records the pause even when every call was answered while the same run is still suspended, then hands the run to the re-check); conformance: fixed. WA's slot-cp315c (2026-11-10, 22 checks, 0 failed) at 9c54febb17 (bed01e209e changes only the changeset): the forced-window witnesses W1 (the pause record held while the submit lands: new_run with self_wake, the companion completes), W1b (held past the wait bound: caller_resumes, and the re-check still continues the answer) and W2 (the answer lands before the run is recorded suspended: the pause is recorded and the re-check continues it), and run-loop-persistent's amended CP-40 (C2) cases B and C, fail at main and pass at the fix; no re-check (M1) reds W1b and W2, the old early return (M2) reds W2, B and C, no window wait (M3) reds W1; W3 (the companion's own call through a live root) and A are green at main and at the fix, a guard. The runtime-js unit suite 2147 of 2147 and the integ suite on Redis and Postgres 32 of 32 at the fix; related: the residual is FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER (a submit from another process in the window answers caller_resumes, the companion still continued by its driver's re-check, by trace); FU-JS-COMPANION-SUSPENDED-CONTINUATION is closed by this fix with CP-250 (its live-ancestor case pinned by W3) | source, live-repro, peer-reported | verified | fixed (no lane can reach it: 2 tests) |
| CP-316 | S2 | runtime-js (and DBOS, for an app-passed no-op manager): a sub-agent's or a companion's own agent.hooks never fire when its parent has no hooks and the call has no usage store. The parent's hook manager is then the shared no-op manager, whose createChild returns itself, so registering the child's hooks on it does nothing: no hook runs and nothing reports it. The same no-op createChild also drops every hook registered over an app-passed no-op manager (ExecuteOptions.hookManager on runtime-js and DBOS, and the DBOS executor config hookManager): the agent's own hooks, the call's hooks and, on runtime-js, the usage hooks. | runtime-js/src/js-agent-executor.ts:9778-9786 (origin/main 41ea410715) (buildHookManager returns the shared noopHookManager when the agent, the call and the usage context bring no hooks; buildUserHookManager, :9746-9751, returns undefined then); runtime-js/src/js-agent-executor.ts:10616 (origin/main 41ea410715) (the run loop input carries that manager as agentHooks); runtime-js/src/run-loop.ts:5829-5847 (origin/main 41ea410715) (a sub-agent: with no userHookManager and no usageContext the parent hooks are input.agentHooks, the no-op manager; its createChild is taken, so the fresh createHookManager fallback is skipped, and register of the child agent hooks lands on the no-op); runtime-js/src/run-loop.ts:3547-3563 (origin/main 41ea410715) (a companion: the same shape over childAgent.hooks); core/src/hooks/hook-manager.ts:243-262 (origin/main 41ea410715) (NoopHookManager: register does nothing and createChild returns this); runtime-js/src/js-agent-executor.ts:3748 and :6135 (origin/main 41ea410715) (the usage context exists only when the call passes a usageStore; with one, parentUserHooks is undefined and the child gets a fresh manager, so its hooks fire); runtime-js/src/js-agent-executor.ts:9755 and :9792 (origin/main 41ea410715) (an app-passed options.hookManager is taken through createChild before agent.hooks, options.hooks and, in buildHookManager, the usage hooks are registered on it; a no-op one keeps none); runtime-dbos/src/observability/merge-hook-manager.ts:91-99 and :124 (origin/main 41ea410715) (the same over an app-passed options.hookManager, and over the executor config hookManager, runtime-dbos/src/config.ts:40, when the agent or the call brings hooks: both are taken through createChild before agent.hooks and options.hooks are registered); contrast: traced at origin/main 41ea410715, Temporal (runtime-temporal/src/activities.ts:2176-2199) and CF Workflows (runtime-cloudflare/src/steps.ts:1347-1371, client-tool-workflow-helper.ts:319-336) gate their worker-level manager on hasHooks() and build an agent-hooks-only manager from the agent's own hooks with createHookManager; DBOS bases its manager on a fresh createHookManager when there is no executor config manager (runtime-dbos/src/observability/merge-hook-manager.ts:124); the CF DO runs each sub-agent and companion as its own DO entry, a root there; contrast: never reached with an app-passed no-op manager (traced at origin/main 41ea410715): Temporal takes an app manager only as GenericActivitiesDeps.hookManager (runtime-temporal/src/activities.ts:328, kept at :1277) and reads it through hasHooks() first (:2179), so a no-op one takes the agent-hooks-only path and its createChild is never called; its executor does not read ExecuteOptions.hookManager. CF Workflows takes one only as an AgentSteps argument (runtime-cloudflare/src/steps.ts:994, gated by hasHooks() at :1350) and ClientToolWorkflowDeps.hookManager (client-tool-workflow-helper.ts:412, gated at :319); its own workflow passes none (workflow.ts:957 passes undefined, and the helper call at :5985 none), and its executor does not read ExecuteOptions.hookManager. The CF DO passes no hook manager to the runtime-js executor it builds, and always gives it a usage store (runtime-cloudflare/src/do/drive/materialize.ts:80, origin/main 41ea410715: the DO internal usage store when the app has no factory), so a hook-less parent child there gets a fresh manager; related: peer-reported by W1, in SDK-6's slot s6-r2 (2026-11-10); W1 fixes it in the SDK-6 MR; related: CP-301 (adjacent, open: the opposite leak on the same lines, the ROOT agent's own hooks run for every descendant through a real manager's createChild inheritance); this record's fix (W1, the SDK-6 MR: core NoopHookManager.createChild returns a new, empty, parentless manager, which also covers app-passed no-op managers) changes only the no-op child case and leaves that inheritance, CP-301's half, untouched; the contract both break is docs/guide/sub-agents.md:2010 (a sub-agent's hooks fire independently of the parent's); severity S2 (owner, provisional): user hooks silently never run, with no error; a hook that gates or records work (an approval, an audit, a state write) is skipped without a trace; core/src/hooks/hook-manager.ts:264 (fix: the SDK-6 MR !834, W1; as on origin/main 6c157d8193) (NoopHookManager.createChild returns a new, empty, parentless DefaultHookManager, so the hooks registered on a no-op manager child are kept; a real manager createChild is unchanged); conformance: fixed. W1's slot c316-r1 (main ae4c3d8040 plus the cells, the fix at 816ff95498, the pre-rebase head of this branch): hooks.noop-manager-own-hooks-subagent fails .root-own-hook-ran and .sub-agent-own-hook-ran, and hooks.noop-manager-own-hooks-companion fails .parent-own-hook-ran and .companion-own-hook-ran, on js-memory, js-redis and js-postgres at main, and every id passes on those lanes at the fix; the old createChild restored (m7) fails the same ids as main. dbos-postgres: the root ids fail at main and pass at the fix, and the two child ids fail at both (ledgered under CP-299). Slot c316-r2 re-ran the fix twice (fix-1, fix-2) with rows identical to c316-r1's (rows sha256 dac4856a0e8b6ddcb868d92607df46ab3c2171a14e54e2df569486d8aaeb8ca0), 3 of 3 | source, peer-reported | verified | fixed (6 cells) |
| CP-317 | S2 | runtime-js (the CF DO only until its next watchdog tick or entry): an entry that fails after its run-start commit and before its loop (a store error in the drain, the reconcile or the entry state write) ends its run failed but leaves its session active and its stream live. retry() is then refused ('Cannot retry: agent status is running'), a client tailing the stream waits out its live-wait bound, and the failure's detail never reaches the session; only a further continue moves the session on. | runtime-js/src/js-agent-executor.ts:3162-3174 (origin/main ae4c3d8040) (failStartedRun writes only the run status failed: no K5 session commit, no stream end or fail); runtime-js/src/js-agent-executor.ts:8346-8404 (origin/main ae4c3d8040) (resumeSession: a pre-loop failure that is not a stop, a refused state patch or a rewind over a finished source ends the started run with failStartedRun alone and rethrows); the drain inside that try (drainOrphanClientToolCalls, :8267) makes several store commits; runtime-js/src/js-agent-executor.ts:1970, :6366, :6387, :8375, :8402, :10545 and :11017 (origin/main ae4c3d8040) (every failStartedRun site); inferred: a real route, any store error a commit can raise there, e.g. core CommitNotAppliedError (transport_error, retryable) or CommitOutcomeUnknownError after a transport failure on Redis or Postgres, so it is not reached only by a test fault; live: W1's SDK-6 slot s6-r3 (unit P4, a one-shot store throw at the drain's first append after the run-start of a drain-first with_message resume of an approved call): resume rejected, the run read failed, the session stayed active with the message stashed on its carrier, and retry() threw ResumeRefusedError 'Cannot retry: agent status is running'; so retryEntry's rule-5 drain (a failed session holding a stash) is unreachable by a real failure on runtime-js; related: FU-FAIL-STARTED-RUN-LEAVES-STREAM-OPEN (the stream half and WE-2's 2026-11-05 session-half sighting by a test fault, which this record supersedes: a real store error takes the same path); contrast: the CF DO is partly affected (traced at origin/main ae4c3d8040): the same executor leaves the same state, but the DO heals it. Its execution-watchdog keeps ticking while the session reads active (durable-object-agent-base.ts:3141-3176: isExecutionActive, do-state-store.ts:4468-4474, is the stored running status of an active session), and the wake ladder's announced_terminal step (durable-object-agent-base.ts:3513, :3671-3760; do/drive/reconcile-wake.ts:645-660) commits the session failed with the run's failure and fails its stream, fenced on the run (do/drive/announced-terminal.ts:84-91, :108-150, :210-230: an ended failed current run over an active session whose latest commit is not its K5; no stream terminal is required). Any next request entry does the same before it plans (Ruling 62, do/drive/run-drive.ts:1213-1250), so a retry through the DO proceeds. What remains is the window until the next watchdog tick (at most EXECUTION_WATCHDOG_RETRY_MS, 30 s, :433), with the stream live and the session active; severity S2 (owner, provisional): nothing committed is lost and a continue recovers, but after one store blip the session, the stream and the run disagree: retry is refused, a reader waits out its bound, and the typed failure never reaches the session or getStatus | source, peer-reported | partly-inferred | open |
| CP-318 | S3 | CF Workflows, Temporal and DBOS: a hook's RunStoppedError (SDK-6) is not honoured. It leaves the hook's durable unit as an ordinary hook error, so that unit is retried or fails and the run then fails, instead of the run being stopped as an interrupt with the hook's reason, as on runtime-js and the CF DO. On CF Workflows an afterTool RunStoppedError also re-runs the tool on each step retry (CP-319). | runtime-cloudflare/src/steps.ts:5365, :5384, :5563-5573 and :5629-5640 (origin/main f4f4028489) (executeToolCall, then runToolCall: beforeTool through invokeHookWithTracker, outside any try, and afterTool on the validation-failure path; the manager is undecorated); runtime-cloudflare/src/steps.ts:1504-1534 (origin/main f4f4028489) (invokeHookWithTracker: the manager invoke inside a try with only a finally, so a hook throw propagates); runtime-cloudflare/src/workflow.ts:5958-5962 (origin/main f4f4028489) (the tool runs in step.do with no step config, so the engine default retries apply to a hook throw); runtime-temporal/src/activities.ts:2797 and :2898-2918 (origin/main f4f4028489) (executeToolActivity: beforeTool outside the tool body try; the comment at :2903 says a throwing beforeTool rejects the activity and fails the step); afterTool through fireDeferredToolHooks, :3328, invoked at :3410-3418; runtime-temporal/src/workflow.ts:335-343 (origin/main f4f4028489) (the activity retry policy, maximumAttempts 3); runtime-dbos/src/steps/execute-tool.ts:356-363 and :620 (origin/main f4f4028489) (beforeTool outside the tool try, inside a DBOS step with retriesAllowed and maxAttempts 3); runtime-dbos/src/workflows/shared.ts:4473-4481 (origin/main f4f4028489) (the phase-1 afterTool through dispatcher.runHook in the workflow body); live: W1's SDK-6 slot s6c-r2 (2026-11-11), hooks.run-stopped-error-before-tool and -after-tool, three green runs with byte-identical rows and no stop recorded: on temporal-memory, temporal-redis, temporal-postgres and dbos-postgres the run ends failed with the hook's message and the tool never ran (beforeTool) or ran once (afterTool); on cfw-workflows-d1 the run ends failed and the tool never ran (beforeTool) or ran 6 times (afterTool: the step's default retries re-run the tool, CP-319); conformance: W1's SDK-6 cells (slot s6c-r3, 2026-11-11: red at main f4f4028489, green x3 at the SDK-6 MR head with byte-identical durable rows, also identical to slot s6c-r2's, mutations M1-M3 as predicted): on cfw-workflows-d1, dbos-postgres and temporal-memory/-redis/-postgres, hooks.run-stopped-error-before-tool fails .call-paired-not-started, .interrupted and .stop-reason, and hooks.run-stopped-error-after-tool fails .interrupted, .real-result-kept and .stop-reason; on cfw-workflows-d1 the after-tool cell also fails its precondition .tool-ran-once (the tool body ran more than once) and with it .no-model-call-after and .transcript-paired; the runtime-js lanes and cf-do pass; contrast: runtime-js decorates every run hook manager with core withRunStopHooks (runtime-js/src/execution/run-stop-hooks.ts, used by run-loop.ts; core/src/hooks/run-stop.ts and core/src/errors/run-stopped-error.ts), and the CF DO runs that executor (!834, as on origin/main 6c157d8193); related: documented as such: docs/guide/hooks.md, the Runtimes bullet (!834, as on origin/main 6c157d8193), names runtime-js and the CF DO and calls the error an ordinary hook error elsewhere for now; related: CP-319 (the CF Workflows tool re-run on a throwing afterTool, any error), CP-299 (DBOS children and their own hooks), CP-316 (the no-op manager child hooks, the same SDK-6 MR), FU-SDK6-PRECOMMIT-ENTRY-HOOK-STOP; severity S3 (owner): a feature not yet wired on three runtimes and documented as such; an app that throws it there gets the pre-SDK-6 behaviour of a hook error (the duplicate tool run on CF Workflows is CP-319, S2) | source, peer-reported, live-repro | verified | open |
| CP-319 | S2 | CF Workflows: the tool body and its success afterTool share one try inside the tool's step.do, so an afterTool throw is taken as the tool's failure. A successful call becomes a failed row the model sees, and afterTool fires again with success:false outside any try; when that call throws too, the error leaves the step and the engine's default step retries re-run the tool (up to 6 runs in all), with no stop or typed signal, and the run then fails. Every other runtime fails the run on an afterTool throw and keeps the tool's result, run once. | runtime-cloudflare/src/steps.ts:5671-5673 and :5884-5898 (origin/main fef7348714) (one try holds tool.execute and the success afterTool through invokeHookWithTracker); runtime-cloudflare/src/steps.ts:5922 and :6044-6057 (origin/main fef7348714) (the catch builds the call's failed result from the hook's error, then fires afterTool with success:false outside any try); runtime-cloudflare/src/steps.ts:5371-5382 (origin/main fef7348714) (executeToolCall rethrows what runToolCall throws); runtime-cloudflare/src/workflow.ts:5958-5962 (origin/main fef7348714) (the tool runs in step.do with no step config); :345-347 (the engine default, 5 retries); live: W1's SDK-6 slot s6c-r2 (2026-11-11), cfw-workflows-d1, hooks.run-stopped-error-after-tool (an afterTool hook throwing on every call; on CF Workflows a RunStoppedError is an ordinary hook error, CP-318): the tool body witnessed 6 times and the run failed, in three green runs with byte-identical rows; inferred: from the trace, the same for an ordinary Error thrown from afterTool on its success:false call (a hook that throws whenever it is called, or a failed tool whose afterTool throws), and, for a hook that throws only on success:true, a successful call recorded as failed with no re-run; neither has its own cell yet; contrast: runtime-js (and the CF DO, which runs it) fires afterTool deferred, after the call row, and fails the run on a throw (core/src/orchestration/step-iterator.ts:3372-3392, origin/main fef7348714); Temporal fires it in its own activity, fireDeferredToolHooks (runtime-temporal/src/activities.ts:3328, from workflow.ts:1442), and DBOS in a hook step through dispatcher.runHook (runtime-dbos/src/workflows/shared.ts:4473, steps/hooks.ts:142-177, workflows/standard-workflow.ts:522-525), so their retries re-run only the hook; s6c-r2 witnessed the tool once on temporal and dbos-postgres; related: CP-318 (SDK-6 not honoured on CF Workflows, the case that witnessed this), DI-89 (the same CF Workflows tool catch), FU-CFW-DRAIN-APPROVE-ROW-COMMIT-RETRY (another tool re-run, a store fault in the approve drain); severity S2 (owner): a non-idempotent tool's side effect repeats up to six times on one throwing app hook, with no signal, and a successful tool is reported to the model as failed | source, live-repro | partly-inferred | open |
| CP-320 | S2 | useHelixChat: retry()'s two early branches read hook-level marks that are not scoped to the chat they were set on, so after the chat id changes they regenerate on the new chat. A stop() on chat A, then a switch to an empty chat B, then retry() on B calls chat.regenerate() on a chat with no messages, which throws (before ai-sdk 54.3.0 the same retry re-anchored). The explicit-stop mark (CP-312, new in 54.3.0) is the one witnessed; the last failure code (U9, #177, older) has the same shape. | ai-sdk/src/react/use-helix-chat.ts:866 (origin/main fef7348714) (explicitlyStoppedRef, a plain boolean for the whole hook); ai-sdk/src/react/use-helix-chat.ts:779-780 (origin/main fef7348714) (explicitStop takes the stopped chat id for its own re-anchor but sets the mark with no chat id); ai-sdk/src/react/use-helix-chat.ts:876 and :888 (origin/main fef7348714) (the only clears: a send, and regenerateWithNewKey); the chat id effects at :682 and :852-856 leave it; ai-sdk/src/react/use-helix-chat.ts:928 and :958-960 (origin/main fef7348714) (retry() takes the mark before any chat-scoped branch and calls regenerateWithNewKey, chat.regenerate on the current chat, with no check for a user message); contrast: the refused send carries its chat id and is dropped on a chat change (ai-sdk/src/react/use-helix-chat.ts:847-856, #173, origin/main fef7348714); retry() itself ends on a chat switch only after its own waits (:933-935); live: the app's repro, verified by the orchestrator on ai-sdk 54.3.0 (2026-11-11): stop() on chat A (with or without serverStop), the chat id changed to an empty chat B, retry() on B throws from ai's chat.regenerate ('message undefined not found'); inferred: the same for the last failure code: lastFailureRef (ai-sdk/src/react/use-helix-chat.ts:506) is cleared only by clearFailure (:511-514, a send or a resume), not on a chat change, and retry() reads it first (:948-954: a state_entry_patch_refused failure re-sends under a new key through regenerateWithNewKey); this half predates 54.3.0 (U9, #177, since 264bc55a35) and had no repro; ai-sdk/src/react/use-helix-chat.ts:507, :802 and :894 (fix evidence, !846, as on origin/main 6c157d8193) (the failure code and the explicit-stop mark each carry the chat id they were set on); ai-sdk/src/react/use-helix-chat.ts:975-979 and :983-998 (fix evidence, !846, as on origin/main 6c157d8193) (retry() on a chat with no user message and no held refused send re-anchors with resync, or does nothing without it; each mark applies only while its chat is current); ai-sdk/src/react/use-helix-chat.ts:772-789 and ai-sdk/src/client/stopped-send-probe.ts:36-58 (fix evidence, !846, as on origin/main 6c157d8193) (the stopped-send probe takes an abort signal that an unmount and a chat switch abort); ai-sdk/src/__tests__/react/use-helix-chat-chat-scope.test.tsx:182, :194, :203, :212, :221, :239, :247, :307 and :318 (fix evidence, !846, as on origin/main 6c157d8193) (the hook units T1 to T9: :182 (stop on A, switch to an empty B, retry re-anchors and resolves where the server has B), :194 (B with history: A's mark does not apply), :203 (an empty chat's own retry), :212 (without resync: retry on an empty B resolves and sends nothing), :221 (the same-chat CP-312 retry, a new key, unchanged), :239 and :247 (the failure-code half, empty B and B with history), :307 and :318 (an unmount and a chat switch end the probe); ai-sdk/src/client/stopped-send-probe.test.ts:90, an aborted probe reads nothing more); conformance: fixed. WE-2's cell client.stop-mark-stays-on-its-chat-client (!847, as on origin/main 6c157d8193): a page stops chat A, switches to a fresh chat B (the client tier's switch gives an empty seed, and B has no server session) and calls retry(). At main (slot w-cp320e) it fails exactly .retry-reads-b and .retry-outcome on cf-do, js-chat-host and both +ai-newest lanes: retry() throws before any read of B. At the fix it passes three times per lane set with byte-identical rows: retry() reads B's snapshot and rejects with the re-anchor's typed state_session_not_found, B having no session, and nothing is sent for B. Both guards removed (M-BOTH) fail the cell exactly as at main. The units (slot w-cp320d, at the fix's own code): 16 pass at the fix, 9 fail at main, and each mark's chat check, the empty-chat guard and the probe abort removed one at a time fail their own units; related: CP-312 (the explicit-stop mark this record scopes), FU-AISDK-HELIX-CHAT-REATTACH-ERROR-PER-CHAT (the same class for the re-attach error, open), FU-AISDK-STOPPED-SEND-PROBE-NO-ABORT (the stop re-anchor probe outlived a chat change; closed by this fix); severity S2 (owner): a shipped client API threw in an ordinary flow (stop, switch conversations, retry); client only, nothing was written to the server | source, consumer-documented, live-repro | verified | fixed (4 cells) |
| CP-321 | S3 | runtime-js and the CF Durable Object: a new turn sent between a completed run's K5 terminal commit and its own completed status write is refused state_already_running. The session already reads completed while the run record still reads running. On runtime-js the new entry plans to finish that run in its run-start commit (finishCurrent), but when the run's status write lands between the entry's current-run read and its commit, CP-96's run-status check refuses the commit, no retry applies, and the caller gets already running (status: superseded_entry). On the CF Durable Object the drive's stage-1 gate counts the run as executing while it reads running, so a /start in that window answers 409 (under interrupt_previous it stops a run that already completed). The terminal-status twin of CP-259, which accepts only the suspension step. | runtime-js/src/run-loop.ts:6687-6690 and :6735-6739 (origin/main fef7348714) (onCompleted: the K5 terminal commit, after which the session reads completed, then the run's own completed status write); runtime-js/src/run-start.ts:415, :420-427, :435-438, :476-478 and :499 (origin/main fef7348714) (the entry reads the current run, checks it against the run it saw, plans finishCurrent for a live run whose K5 landed, and commits); core/src/store/run-start.ts:305-311 (origin/main fef7348714) (CP-96's run-status check: the run now reads completed, not the running the entry planned on, so the commit is refused expected_run_status_changed before the finish verify at :318-328); runtime-js/src/run-start.ts:385-413 and :520-528 (origin/main fef7348714) (CP-259's advance over the run's own step, tried on that refusal) with core/src/store/run-start.ts:81-94 (origin/main fef7348714) (isOwnSuspensionStatusStep accepts only a suspended current run over a paused session, :90 and :92: a terminal step is not accepted); core/src/errors/agent-errors.ts:882 and :903-911 (origin/main fef7348714) (no retry: one attempt only, and the fresh run no longer matches the running the entry saw) and :825-833 (asRunStartLoserError: the refusal becomes RunStartConflictError with the status superseded_entry, its only producer); conformance: CI, batch 21 MR !848 train pipeline 3056097612, job 17946417572 (test:conformance 5/7, 16ae617e): companion.paused-recovered-by-next-turn|js-postgres|executor failed .second-turn-settles, the second root turn threw state_already_running, already running (status: superseded_entry); its other three ids failed as preconditions. The cell polls the persisted session status for completed (e2e/src/conformance/scenarios/companion-outlives-root.ts:439-446, origin/main fef7348714) and then starts the next turn (:466), so a loaded runner can put that start inside the window. The cell is one of CP-279 provenBy keys; runtime-cloudflare/src/do/drive/stop.ts:571-579 and :597-605 (origin/main fef7348714) (executingRunOf and drivePhaseOf: a registered drive that has not settled, whose run is current and running, is executing) with runtime-cloudflare/src/do/durable-object-agent-base.ts:1830-1844 (origin/main fef7348714) (the phase from a synchronous current-run read) and runtime-cloudflare/src/do/drive/host.ts:361-365 (origin/main fef7348714) (a drive settles once its handle result settles, after the run's status write); runtime-cloudflare/src/do/drive/run-drive.ts:461-472 and :1419-1444 (origin/main fef7348714) (stage 1: an executing drive is lost, reason executing, under the default reject policy, a 409 state_already_running before any run-start commit; under interrupt_previous it is stop_needed); runtime-cloudflare/src/do/drive/run-drive.ts:685-722 with core/src/run/stop-run.ts:188-201 (origin/main fef7348714) (interrupt_previous in the window: the stop is recorded, the store deciding on the running status alone, the run is signalled, and the answer stopping names the run as interrupted, run-drive.ts:703); runtime-js/src/run-loop.ts:7603-7632 (origin/main fef7348714) (interrupt_previous leaves the completed turn unchanged: the run's status write is not stop-guarded, and the DO store's updateRunStatus at runtime-cloudflare/src/do/do-state-store.ts:5445 (origin/main fef7348714) reads no stop; the settle's stop cascade and wake reset act only on interrupted or failed, runtime-cloudflare/src/do/drive/settle.ts:497 and :596 (origin/main fef7348714); the SDK-7 turn mark needs an interrupted run, core/src/store/turn-stops.ts:95 (origin/main fef7348714), and the stop notice an interrupted or failed one, core/src/orchestration/stopped-turn-notice.ts:88-90 (origin/main fef7348714); the DO stream reads a stop only once it is no longer active, runtime-cloudflare/src/do/do-stream-manager.ts:508-510 (origin/main fef7348714)); related: reach under interrupt_previous: ai-sdk/src/handler/handle-chat-stream.ts:5066-5079 (origin/main fef7348714) (the new send's X-Helix-Interrupted-Run header, like the afterEntry hook's interruptedRunId, names the completed run as interrupted. The SDK client does not read the header; one known consumer, an app's chat overlap, reads it to resync another tab, so a completed run named interrupted costs at most one extra resync there. The stop row left on the completed run is inert: every stop reader gates on interrupted, failed or a live run. While the status write is held, the DO's /status reports stopping beside a completed session, runtime-cloudflare/src/do/durable-object-agent-base.ts:5662-5670 (origin/main fef7348714)); runtime-js/src/execution/handle-factory.ts:171-204 (origin/main fef7348714) (reach: an active handle's result() resolves with the run loop's return, after the status write at runtime-js/src/run-loop.ts:6735 and the stream end at :2984 (origin/main fef7348714), so a caller that awaits it is past the window; an attached handle polls the run record, runtime-js/src/js-agent-executor.ts:2415-2436 (origin/main fef7348714); the DO frontend handle has no result()); runtime-js/src/js-agent-executor.ts:6690-6735 and runtime-js/src/execution/handle-factory.ts:351-359 (origin/main fef7348714) (reach: a getHandle() taken in the window builds its result() from the loaded session, completed, so its caller can send into the window, as can any caller that polls the session status; a chat client driven by the stream end cannot, the end following the status write, nor a snapshot reader, whose status comes from the run); related: CP-259 (the suspension twin, fixed by !647: the run-start accepts the run's own suspension status step), CP-84 (Temporal refuses in the same terminal window; its note on a runtime-js status-write window concerned the lease window that finishCurrent closed), CP-85 (the stream half of the window), CP-96 (owns the check that fires here; a fix must keep control.stale-entry-loses-after-winner losing); related: fix direction (owner, agreed by the orchestrator 2026-11-11): on runtime-js, accept the seen run's own terminal step as CP-259 accepts its suspension step (the run seen running is the current run, now terminal; the session reads that status; its latest checkpoint is that run's own state commit) and plan as over an idle run; on the CF Durable Object, stage 1 treats a registered run whose K5 landed as tail and waits for its settle; core/src/store/run-start.ts:128 (fix evidence, !858, as on origin/main 18f9d7509e) (isOwnTerminalStatusStep, rule T: the run-start accepts the seen run's own terminal status step only when the entry's seen read already found that run's K5 the session's latest commit, the run is current with that status, the session reads it and nothing has committed since; decided at the seen read, so CP-96's stale entry keeps losing); runtime-js/src/js-agent-executor.ts:3194-3198 and runtime-js/src/run-start.ts:405-430 (fix evidence, !858, as on origin/main 18f9d7509e) (the entry's seen read records the terminal status its K5 already gave the session, committedTerminalStatus; on the CP-96 refusal the entry advances once over that run's own terminal step, as over CP-259's suspension step, and re-plans from fresh reads); runtime-cloudflare/src/do/drive/stop.ts:566-583 and runtime-cloudflare/src/do/durable-object-agent-base.ts:1859-1869 with runtime-cloudflare/src/do/do-state-store.ts:4435 (fix evidence, !858, as on origin/main 18f9d7509e) (a registered run whose K5 landed while it still reads running is in its tail, not executing: stage 1 waits for its settle instead of answering 409, interrupt_previous stops nothing, and a stop in the window answers not stoppable; DOStateStore.committedTerminalStatusSync is the synchronous read); conformance: lifecycle.new-turn-across-terminal-status-write (executor; js-memory, js-redis, js-postgres) holds the run's completed status write after its K5 and releases it at the new entry's seen read (hold-run-status, releaseAtNextRead, with hold-run-status-at-seen-read); red at BASE 6f36e48c1a on all three lanes on .turn-admitted, .turn-completes and .ended-turn-intact, green on the fix (W1 slot cp321-r4). On cf-do (host), with the new hold-do-run-status on the DO store's own updateRunStatus: lifecycle.new-turn-across-terminal-status-write-host red at BASE on .released-at-tail, .send-admitted, .turn-completes and .ended-turn-intact; lifecycle.new-turn-across-terminal-status-write-interrupt-host (interrupt_previous) red on .released-at-tail, .start-admitted and .no-stop-recorded (BASE took the stop-probe branch); lifecycle.stop-across-terminal-status-write-host red on .stop-not-stoppable; all three green on the fix, each naming the branch its hold took (W1 slot cp321-r6, 0 mismatches). Mutations, each red on exactly its predicted tests: in core run-start.ts (M1, M3) the core and runtime-js units; in runtime-js run-start.ts M6 the runtime-js units, and M2 a runtime-js unit and CP-96's control.stale-entry-loses-after-winner|js-memory on exactly .resume.b-rejected-typed and .retry.b-rejected-typed; in stop.ts (M4) the stop and drive units and the -host and -interrupt-host cells; in durable-object-agent-base.ts (M5) the stop cell (cp321-r4, cp321-r5). The product fix is byte-identical across the three slots; related: Temporal, DBOS and CF Workflows are not traced for this window (FU-CP321-DURABLE-TERMINAL-WINDOW; their executor cells are declared gaps, as are the three host cells on js-chat-host); severity S3 (orchestrator, 2026-11-11): a typed, retryable refusal with nothing written, and a resend after the status write starts the turn; reachable only by a caller that keys on the session status (a polled status, or a reconnect handle taken in the window), never by one that awaits the entry handle or the stream end; the completed turn is unchanged on every path traced | source, live-repro, peer-reported | verified | fixed (6 cells) |
