Follow-up items and deferred work
This doc tracks deferred work and known incomplete items surfaced during the sub-project sequence on branch omnara/stateless-suspension-redesign. Items here are NOT blocking the current sub-project's acceptance gate — they are intentionally scoped out and tracked for follow-up.
For test-infrastructure-specific deferred items (atomicity stress, harness backlog, etc.), see test-infrastructure-roadmap.md.
How this doc works
- Each item has a stable ID (e.g.
FU-A2-01) so commit messages can reference it. - Items are organized by surfacing sub-project (A.1, A.2, etc.).
- Status:
open,in-progress,done. - Once closed, items move to the bottom of the doc with a closing-PR reference.
Known CI flakes
A running log of intermittent CI failures with root cause + mitigation, so a red pipeline on one of these is recognized rather than re-investigated from scratch. Recognizing one is not permission to retry it: since MR !272 no CI job retries on script_failure, and a manual retry of a genuine failure follows the disclosed-retry rule in development-workflow.md.
FU-CF-EXAMPLES-POOL-SERIAL-FILES: the CF example suites need a serial-files setting before they grow a second .cf file — open
Owner: WE-2 (CF example suites). Filed by: W1 MR3, from FU-FLAKE-D1-USAGE-STORE-CONTRACT-ROW-COUNT.
- Why:
@cloudflare/vitest-pool-workers(0.9.14) ignoresfileParallelism: falsewhenisolatedStorage: false: it starts one runner worker per test file over one Miniflare and runs them all at once over shared storage (dist/pool/index.mjs). OnlypoolOptions.workers.singleWorker: trueruns the files one at a time. store-cloudflare's contract files deleted each other's rows that way (!338); store-cloudflare and memory-cloudflare now setsingleWorker: true. - Where:
examples/research-assistant-cloudflare-do/vitest.config.tsandexamples/workspaces-showcase/vitest.cf.config.tssetisolatedStorage: falseand no serial setting. Each has one.cf.test.tsfile today, so nothing can overlap yet. - Fix: add
singleWorker: true(or give each file its own storage) when either suite gains a second.cffile, or now. store-cloudflare'shelpers/cf-file-serial-guard.tsis a setup file that fails a file starting while another runs, if a guard is wanted.
FU-FLAKE-EXAMPLES-NEXTJS-REDIS-TIMEOUTS: test:examples:nextjs-redis times out on different tests — open
Filed: 2026-11-01 (orchestrator).
The job failed on 3 of the last 4 pipelines read, each time on timeouts in different tests: brief-flow:460, then api-consistency:156 and :181, and an earlier brief-flow. It went green on retry on !634, which has no example or product change. Cause unknown: nothing here shows it is load. To investigate.
FU-FLAKE-EXAMPLES-NEXTJS-REDIS-LIVE-MARKER-TEXT: brief-flow.spec.ts asserts a live model's exact marker words — open (post-features; test only)
Filed: 2026-11-09 (orchestrator, via the conformance owner). Next to FU-FLAKE-EXAMPLES-NEXTJS-REDIS-TIMEOUTS, a different cause.
- The tests:
examples/nextjs-redis/tests/brief-flow.spec.ts::183"deterministic interleaved pattern appears correctly",:335"final message has correct interleaved structure",:391"text content survives multiple refreshes after stream completes". Each expects the LIVE model to write the prompt's exact[A…]/[B…]/[C…]markers (src/lib/brief-prompt.ts) in the text around its two tool calls. - What fails: the model often paraphrases the middle block ("The research returned…") instead of writing the
[B…]markers. The text between the tools is PRESENT, just not as markers. Seen on batch 17, twice on batch 18's first head and on both of !810's pipelines; it passes on retry, so it costs a retry per pipeline. - Fix direction: keep what the tests guard (the text between the tools is rendered live and survives every refresh), drop the model's exact words:
- assert the structure: text, tool, text, tool, text, with non-empty text between the tools;
- for the refresh test, capture each text segment as rendered live and assert the same segments, byte-exact, after each refresh (the regression it guards is lost or duplicated interleaved text, whatever the words are);
- or move the marker form to the recorded (VCR) mock, where the words are fixed.
- Sighting, batch 21 (MR !848 pipeline 3055911329, job 17945122494):
:391failed at:415("Baseline marker [B1] missing") and:183, which passed first, failed on the serial describe's retry at:217("Missing marker [B1]"). Both times the model wrote prose betweenresearch_topicandgenerate_brief(log lines 1029 and 1059) where[B1]…[B10]were scripted; the A and C markers were there. The job's retry, job 17946150743, passed all 32 tests. The same job's other two failures belong elsewhere:api-consistency.spec.ts:156timed out (FU-FLAKE-EXAMPLES-NEXTJS-REDIS-TIMEOUTS), andreasoning-display.spec.ts:42timed out once waiting for the chat input and passed on retry. - Same cause, filed twice before: FU-LIVE-BRIEF-FLOW-MARKER-ASSERTION (2026-10-20,
:183) and the brief-flow bullets of FU-EXAMPLE-REASONING-TEST-LIVE-MODEL-CHOICE (:391,:335; batch 20's:183). One fix closes all three.
FU-FLAKE-EXAMPLE-NEXTJS-REDIS-CHAT-ROUTES-TIMEOUT: the chat-routes unit's first case timed out, and its call leaked into the next — fixed in batch 34 (test only; both examples)
Filed: 2026-11-17 by the conformance owner, at the orchestrator's request. Related: CP-293, FU-FLAKE-COLD-IMPORT-IN-TEST-BODY. Next to FU-FLAKE-EXAMPLES-NEXTJS-REDIS-LIVE-MARKER-TEXT: the same example, a different job and cause.
- The test:
examples/nextjs-redis/src/__tests__/chat-routes.test.ts(origin/main 07819d59eb). - Sighting: !921 pipeline 3081663421, job 18126396793 (
test:unit, not allowed to fail, so it fails any MR's pipeline). "'/api/chat' POST > hands the handler the client's whole conversation (CP-293 …)" timed out in 5000 ms (5011 ms). Then "a body with onlymessage(a string) still sends that text" failed at:104:startAgentExecutionwas called 2 times, not 1. 8 of 10 passed; the file's collect phase was 129 ms. !921 does not touch the example. - Candidate cause (by reading, unverified): each case runs
await load(), a dynamic import of the Next route module, in its body (:46-62,:94), so the first/api/chatcase pays the cold transform inside vitest's 5 s default. That is the FU-FLAKE-COLD-IMPORT-IN-TEST-BODY class, whose sweep did not cover examples. The timed-out case's POST still ran after the timeout, so its call landed after the next case'sbeforeEachmockClear(:88-90): the count of 2. - Fix direction: import the three routes statically at the top of the file (the file uses only hoisted
vi.mock, so a static import still follows the mocks), or load them in abeforeAllwith its own timeout. - Second sighting (2026-11-19), in both examples: !937's MR pipeline 3091666527,
test:unitjob 18202614753. Inexamples/nextjs-redisthe same two cases failed (the first/api/chatcase "Test timed out in 5000ms", then "a body with onlymessage" counted 2 calls). Inexamples/opennext-cloudflare-do's copy of the test, the same pattern: its first/api/chatcase timed out, then its auto-send case read an extra forwarded conversation. !937 does not touch either example, and both suites passed in local slots at the same head. - Fixed in batch 34 (test only), in both examples:
chat-routes.test.tsimports every route module once, in abeforeAllwith a 60 s hook timeout, and each case takes its route'sPOSTfrom that map. A case body no longer pays the cold import, so it awaits only its ownPOST, and no abandoned body can post after the next case'smockClear. A timing flake cannot be forced red on demand, so the fix is shown by the code (noawait load()left in a case body) and by repeated single-file runs, not by a red-first run. - Not seen again (2026-11-18): the file is unchanged through origin/main fae53b26ac, and the 22 other
test:unitjobs of pipelines updated since 2026-11-17T21:00Z passed. - CI: until this lands, a recurrence is a disclosed retry under this entry.
FU-LOCAL-WORKTREE-PATH-LENGTH-POOL-WORKERS: a deep worktree path breaks the research-assistant example's pool-workers suite locally — open (low priority, dev ergonomics)
Owner: unassigned (dev ergonomics). Filed by: W1 MR3's fix-round gate (2026-10-16). CI is unaffected.
- Symptom:
examples/research-assistant-cloudflare-do'spnpm run testfails before any test withkj/filesystem-disk-unix.c++: failed: mkdirat(fd, path): File name too longandMiniflareCoreError [ERR_RUNTIME_FAILURE]: The Workers runtime failed to start. - Cause:
@cloudflare/vitest-pool-workersnames one storage directory per Durable Object class and test file as a single path component:vitest-pool-workers-runner--<40 hex>-<the test file's absolute path, '/' and '.' as '_'>- __VITEST_POOL_WORKERS_USER_OBJECT<class>. macOS caps a component at 255 characters. Foradmission.cf.test.ts(classMyAgentServer) everything but the checkout root takes 193, so the checkout root path must be at most 62 characters.<home>/.superset/worktrees/helix-agents/<dir>, where<home>is a 4-character user's macOS home directory (11 characters), leaves 17 characters for<dir>:reliability-cf-w1-mr3(21) broke it (259).- CI's
/tmp/builds/helix-ai/open-source/agentsmakes 232; a~/code/helix/open-source/helix-agentscheckout makes 240. - Other suites have shorter paths or class names, so they break at deeper roots.
- Fix direction: shorten the example's test file names or its DO class name, or ask pool-workers (upstream) to hash long components. Until then, run it from a checkout whose root path is 62 characters or fewer.
FU-TEST-RUN-TMPDIR-TEARDOWN-BEFORE-POOL-CLOSE: the run-scoped temp dir was removed before vitest closed the pool — fixed in code (lands with its MR)
Owner: W1. A regression from !362 (FU-LOCAL-INTEG-EPHEMERAL-PORT-EXHAUSTION's temp-dir half).
- Symptom: train pipeline 2952742945, job 17215305616 (
test:cfw-engine): 11/11 files and 129/129 tests passed, then a Startup Error failed the job:ENOTEMPTY: directory not empty, rmdir '/dev/shm/…/helix-test-tmp/202-…/miniflare-…/d1/miniflare-D1DatabaseObject'. It evicted !361 from the train. - Cause: vitest 2.1.9's
Vitest.close()runs every globalSetup teardown before it closes the pool.scripts/test-run-tmpdir.mjsremoved the run dir in its teardown, so a pool that still owned files there (@cloudflare/vitest-pool-workers' single Miniflare/workerd runtime, still writing D1's SQLite files; a forks pool's workers) had its dir deleted under it. The script's own comment assumed every worker had exited by then. - Fix: the run dir is removed on the vitest process's
'exit', which runs after the pool's close, so the workers and their workerd children are gone. The globalSetup teardown only restoresTMPDIR. The removal (removeRunDir) isrmSyncwithrecursive,force,maxRetries: 5andretryDelay: 100. If it still fails, it warns once, naming the dir and its reaper (the next run's sweep), and never throws: leaking a temp dir is hygiene, failing a green run is a false red. The setup's sweep removes stale dirs the same never-throwing way. - Guards (
runtime-cloudflare/src/__tests__/integration/test-run-tmpdir.integ.test.ts):- a child process in vitest's close order (setup, the runtime writes, globalSetup teardown, the runtime writes again, exit): the late write lands, the process exits 0, and the dir is gone after exit (red before: the late write got ENOENT);
- the teardown leaves the dir and registers one exit removal;
- a removal that keeps failing warns once with the dir, the error code and the sweep, returns false and leaves the dir; the removal passes
rmSynca bounded retry. - End to end: memory-cloudflare's pool-workers suite exits 0 and leaves no run dir.
FU-FLAKE-OPENNEXT-PROXYWORKER-DROPPED-POST: wrangler dev's ProxyWorker answers a dropped chat POST with its own plain-text 500 — open (dev tooling, not product)
Under runner load the ProxyWorker loses its connection to the UserWorker and answers a POST in flight with Error: Network connection lost. (500, plain text); the request never reached the chat route, and production has no ProxyWorker. The opennext specs re-send such a send once, only when the turn is not committed (!456, example-app-testing.md, "wrangler dev's dropped POST").
Occurrences (before !456's re-send): 2026-10-19 on !446, !449 and !452 (job 17300790896).
FU-LOCAL-INTEG-EPHEMERAL-PORT-EXHAUSTION: Miniflare-heavy suites exhaust the host's ephemeral ports — open (keep-alive half backed out; temp-dir half fixed by !362)
Owner: W1. Seen locally (macOS), not in CI.
Symptom: unrelated integration tests fail with
connect EADDRNOTAVAIL 127.0.0.1:…(agent-server) orTypeError: fetch failed(runtime-cloudflare, D1 store reads). This happens while Miniflare-heavy suites run in parallel, such as a fullturbo run test:integration.Measured (W1 MR3 matrix, 2026-10-14):
- With no other test job on the host,
turbo run test:integration --concurrency=2over agent-server, runtime-cloudflare, runtime-dbos and e2e took localhost TIME_WAIT from 203 to 13,412 sockets within 40 s. The macOS ephemeral range is 49152–65535 (16,383 ports), and TIME_WAIT lasts 2 × MSL = 30 s. do-state-store-contract.integ.test.tsalone made about 3,060 connections to the workerd entry port in 36 s.
- With no other test job on the host,
Measured again (WA's RM-121 / RM-122 slots, 2026-11-08): runtime-cloudflare
test:integrationALONE now exhausts the range on macOS, so a local run of that suite proves nothing; use CI'stest:integ:rest. A single file still runs fine locally.The suite's connections scale with its speed: a quiet host runs it fastest and overruns the 30 s TIME_WAIT window. Main's batch 15 (3c6496ce00) also added 25 tests to it (1,250 → 1,275).
Run Base Load at start Duration Tests Result RM-121 slot fc9ecacc32 + RM-121 31 85 s 1,250 green RM-122 slot fc9ecacc32 + RM-121/122 39 119 s 1,251 green restacked slot 3c6496ce00 + RM-121/122 18 79 s (overlapped) 1,275 70 fetch failedfailuresre-run, both locks 3c6496ce00 + RM-121/122 7 31 s 1,275 56 failures, 616 skipped RM-132 fix, both locks f4f4028489 + RM-132 fix not recorded 87 s 1,299 17 failures, all network In the re-run, the host held both test locks and nothing else ran. TIME_WAIT went from 71 to 16,549 within the 31 s run, more than the 16,383-port range. Every failure was a Node → Miniflare
fetch failed/EADDRNOTAVAIL; no product assertion failed.do-state-store-contract.integ.test.tsnever ran an assertion: its setup failed, so all 582 of its tests were skipped.In the RM-132 fix slot (2026-11-11), the host held both test locks and nothing else ran; TIME_WAIT was 1,342 at the start (the load was not recorded). 14 of the 17 failures were
fetch failed/EADDRNOTAVAILin their own error; the other 3 failed only on a run status, and their 3 files (25 tests) passed run alone, twice at the fix and once at main, so all 17 were the network.
Cause: Miniflare 5's
DispatchFetchDispatcher(src/http/fetch.ts) marks every runtime requestreset: true. Upstream's comment says this is "to avoid keep-alive race conditions". So every Node → workerd call opens and closes its own TCP connection:mf.dispatchFetch, a Durable Object stub'sfetch, and a binding-proxy call such as D1. Each close leaves a client-side TIME_WAIT.- Miniflare 4 never set it, and reused connections.
- workerd itself keeps connections alive: a raw keep-alive client reuses one socket for 50 requests.
- Linux reuses loopback TIME_WAIT sockets, which is why CI does not see it.
Backed out (2026-10-16): the keep-alive patch made trains fail. !362's
patches/miniflare@5.20260925.0-alpha.patch(noreset: true, a 1 skeepAliveTimeout) is removed, along with its guardminiflare-connection-reuse.integ.test.ts. Two train runs failed withTypeError: fetch failed/SocketError: other side closedon a REUSED socket (jobs 17217741005 and 17220638568: the samedo-stop.integ.test.tsstep, the GET right after a POST to a test route that never reads its body).- Cause: workerd closes a keep-alive connection after a Durable Object answered a request whose body it did not read. Repro, patched Miniflare: 300 POST-then-GET pairs opened 301 connections when the DO ignored the body, and 2 when it read it. undici reuses an idle socket after checking it for a close for one timer tick (
scheduleIdleSocketValidation, undici 7.29client-h1.js). On a loaded runner workerd's close lands after that tick, and undici never retries an in-flight request, GET or POST (it fails the head of the pipeline on a close). The keep-alive "race" upstream'sreset: truecomment names is this one. The idle timeout plays no part: the test polled every 50 ms. - Guard:
runtime-cloudflare/src/__tests__/integration/workerd-keepalive-unsafe.integ.test.ts. It pins workerd's close after an unread body and DO hop (and its absence when the body is read), and that Miniflare still opens one connection perdispatchFetch/ DO stub call. Those two cases are red with the keep-alive patch. - The guard's first version raced its own close (train job 17221753532,
read ECONNRESET, which evicted !358). It sent a GET right after the unread-body POST over Node's keep-alivefetch. It now sends exactly one POST per connection and only watches the sockets: it waits for the unread-body socket'sclose, then checks that the read-body socket is still open. It held under induced event-loop stalls of 0, 1, 3 and 6 s. - Until a guarded keep-alive returns (FU-MINIFLARE-KEEPALIVE-REINTRODUCE), the local relief is the macOS TIME_WAIT setting in
development-workflow.md. CI is unaffected (Linux reuses loopback TIME_WAIT).
- Cause: workerd closes a keep-alive connection after a Durable Object answered a request whose body it did not read. Repro, patched Miniflare: 300 POST-then-GET pairs opened 301 connections when the DO ignored the body, and 2 when it read it. undici reuses an idle socket after checking it for a close for one timer tick (
The original fix (!362, backed out):
patches/miniflare@5.20260925.0-alpha.patchdroppedreset: trueand gave the runtime Pool a 1 skeepAliveTimeout.- Its guard was
miniflare-connection-reuse.integ.test.ts(removed with it): at most 2 connects for 50 calls. - With the patch, the contract file above made 150 localhost TIME_WAIT in 36 s.
- Its guard was
Trial result (2026-10-14/15; the orchestrator closed the trial on 2026-10-15):
- Errors. Base (stock Miniflare) and patched trees ran interleaved, 3 iterations each, of the runtime-cloudflare and store-cloudflare integration suites and the cf-do and cfw conformance lanes (24 runs; the bar written here asked for at least 5 each). Every run passed, with zero ECONNRESET / "other side closed" / port-exhaustion errors on either tree. The runtime-cloudflare integration suite logs 3
fetch failedlines per run on BOTH trees, so they are not the patch's. Localhost TIME_WAIT, sampled every 10 s during the last patched cfw run, stayed under about 250. - Time. The patched cfw lane looked slower in that proof (945–2,028 s against 711–901 s). Two controlled runs on one tree showed it was host load. Keep-alive alone: the slow cfw cell (
session.create-observed-whole) took 289 s patched against 336 s stock (14% faster; the platform-proxy POST median fell from 200 to 148 ms). The TMPDIR globalSetup alone, off / on / off: 447 s (load 124) / 306 s (load 44) / 304 s (load 41), so no cost under matched load.
- Errors. Base (stock Miniflare) and patched trees ran interleaved, 3 iterations each, of the runtime-cloudflare and store-cloudflare integration suites and the cf-do and cfw conformance lanes (24 runs; the bar written here asked for at least 5 each). Every run passed, with zero ECONNRESET / "other side closed" / port-exhaustion errors on either tree. The runtime-cloudflare integration suite logs 3
A second leak, temp dirs (fixed in code, same MR): 5,426 stale
miniflare-*dirs (7.6 GB) had piled up in the shared temp dir, and one integration iteration added 63.- Cause: Miniflare 5's
dispose()deletes<tmpdir>/miniflare-<hex>fire-and-forget, so a process that exits right after it leaves the dir behind (vitestforceExit, a worker teardown, a kill). - Fix:
scripts/test-run-tmpdir.mjs, a vitest globalSetup wired into every config that runs real Miniflare/workerd. It pointsTMPDIRat<tmp>/helix-test-tmp/<run>/, removes that dir when the vitest process exits (after the pool's close; FU-TEST-RUN-TMPDIR-TEARDOWN-BEFORE-POOL-CLOSE), and sweeps run dirs older than 6 h that no process holds open. It never touches anything outsidehelix-test-tmp/. - Guard:
runtime-cloudflare/src/__tests__/integration/test-run-tmpdir.integ.test.ts. - The dirs already piled up are left for the user to clear.
- Cause: Miniflare 5's
Upstream: a cloudflare/workers-sdk issue is drafted with this MR, covering both
reset: trueand the fire-and-forget temp-dir delete. Filing it is the user's call.Relief without the patch: see
development-workflow.md, an optional macOS setting.
FU-INTEG-DEFAULT-PORT-FALLBACKS: integration tests fall back to the default Redis and Postgres ports — fixed in code (test only; lands with its MR)
!727 makes the shared e2e and DBOS helpers refuse a default port: a helper that can skip uses an address that never connects, and one that cannot skip refuses to start. The rest still fall back. At 4f0e1d1807, 62 integration test files default to a local Redis or Postgres URL when their env is unset (process.env.X ?? '…localhost…' or || '…'): store-redis 38, runtime-js 13, store-postgres 4 (plus its shared integration helpers.ts), runtime-temporal 3, e2e 2, agent-server 1, runtime-dbos 1. On a shared host the default port may be another project's service, so a test can pass or fail against data it does not own, and it writes there.
Sighting (2026-11-05): runtime-dbos onmessage-persistent.integ.test.ts ("fires onMessage exactly once…") failed 3 of 3 against another project's Redis on 6379 (onMessage fired exactly twice across both persistent-mode turns, length 1). On a private Redis it passed 3 of 3, with both turns' rows present at the status wait and 8 s later, and the full DBOS integ at that head passed (99 files). Whether that Redis or the host's load caused the failures was not isolated; no product bug is shown, so no id.
Sighting (2026-11-19): agent-server run-start-status-window.integ.test.ts (CP-333; REDIS_URL / POSTGRES_URL defaults at :27-29) ran in a local slot without its env and connected to another project's Redis on this host. It wrote only under its own random prefix with a 120 s TTL and cleaned up, but it must never connect there.
Fix (test only): @helix-agents/core/testing gains integEnv, describeWithIntegEnv and the per-store redisIntegUrl / postgresIntegUrl / temporalIntegAddress. An unset (or empty) address skips the suite, names the variable, and hands the body a placeholder that never connects (vitest collects a skipped suite's body); in GitLab CI (CI=true) it throws instead, unless HELIX_INTEG_ALLOW_SKIP=1. Every integration test that defaulted to a host port now reads its address through them (store-redis, store-postgres, runtime-js, memory-redis, agent-server, runtime-dbos, runtime-temporal and e2e). scripts/check-integ-env.mjs, run by core's lint, fails on a direct read of REDIS_URL / POSTGRES_URL / DATABASE_URL / TEMPORAL_ADDRESS in test code outside its allow-list, so a default cannot come back. The allow-list names each file with its reason: e2e's load-time setups (their default never connects), the DBOS helpers that refuse to start when an address is unset, and the published Temporal helper (below). A locally cached lint can skip the check, so run pnpm run check:integ-env itself.
Kept (user ruling 2026-11-19): the published runtime-temporal/testing helper keeps its documented default; in-repo runs are guarded by the driver sentinel. The static check allow-lists that one published file by name.
FU-CORE-TESTS-UNTYPECHECKED: core's test files are never typechecked — open
packages/core/tsconfig.json excludes **/__tests__/**, so neither core's pnpm run typecheck nor CI's typecheck job reads core's test files (every other package's tsconfig includes its tests). A tsc run over core with its tests included (a scratch config extending core's, with only node_modules and dist excluded) reports 528 errors in 91 test files at RM-119's d607603542, all of them under __tests__: workspace/__tests__/registry.test.ts 57, store/__tests__/snapshot-view.test.ts 41, workspace/__tests__/tool-injection.test.ts 40, __tests__/lock/noop-lock.test.ts 26, run/__tests__/fail-session-without-run.test.ts 19, __tests__/client-tool/route-submit.test.ts 18. A test can therefore drift from a changed core type and still pass at runtime: RM-119 found two typed step_discarded literals missing its new required field that the unit run could not catch.
Until it is fixed, a slot that touches core checks "no new error": no error in a core file the MR changes that the base lacks, compared by file, error code and message with line numbers stripped (an MR that only moves a pre-existing error, as WA's RM-123 insertions moved orchestration/__tests__/step-iterator-delegation.test.ts's TS2322, adds none), and core's total with its tests at or below the base's, measured in the same run. Fixing it (the errors, then including the tests in a typechecked config) needs its own plan; any CI change needs the user's go.
FU-MINIFLARE-KEEPALIVE-REINTRODUCE: bring Miniflare's Node → workerd keep-alive back only with a proven guard — open (owner: W1)
Filed by: the keep-alive backout (2026-10-16, FU-LOCAL-INTEG-EPHEMERAL-PORT-EXHAUSTION).
Keep-alive on Miniflare's runtime dispatcher is worth having locally: thousands fewer TIME_WAIT sockets, and the cfw cell measured 14% faster. But it is unsafe while workerd closes a connection after a Durable Object leaves a request body unread and undici reuses that socket. It comes back only with one of these guards, each proven by a repro under an induced server-side delay (the close landing after undici's one-tick idle check):
- (i) the
DispatchFetchDispatcherkeepsreset: truefor every request that carries a body, and keeps GET / HEAD alive, so no reused socket follows a body. Measure how much of the TIME_WAIT win is left: binding-proxy calls are POSTs. - (ii) every Durable Object route a Node-driven test reaches drains its body, product routes and test helpers alike, with a test that fails on any route that does not. This does not cover binding-proxy hops, so (i) is preferred.
workerd-keepalive-unsafe.integ.test.ts's "one connection per call" cases must be changed deliberately when it comes back.
FU-CF-D1-LOADSTATE-FETCH-FAILED-CAUSE: D1 store errors drop the transport cause from the message CI prints — open (owner: W1)
Seen: train job 17217741005 logged 13 D1 state loadState failed for sessionId: …: TypeError - fetch failed lines (runtime-cloudflare concurrent-subagent-fold, absorbed by its mock step.do retries). The atomic-suspend-write flake's assertion message was truncated the same way. D1StateError keeps the cause (error.cause) and prints TypeError - fetch failed, but not the cause's own cause: undici's SocketError: other side closed, ECONNRESET, and so on. So CI cannot tell a keep-alive reuse failure from any other transport failure. Fix direction: D1StateError's message includes the innermost cause's name / code / message, with a unit test on a nested cause. Check whether these lines still occur now that keep-alive is backed out.
FU-FLAKE-CHECKPOINT-MATRIX-INTERRUPT-RELEASED-BEFORE-RECORDED: the checkpoint matrix released a held tool before the interrupt was recorded — fixed in code (lands with its MR)
Seen: !358's train pipeline 2953563680, job 17221170430 (test:integ:e2e 1/4): checkpoint-truth-matrix.integ.test.ts, 'js-postgres' > run.new-on-entry.resume-with-message: "the resumed run settled failed (MockLLMAdapter: No more responses configured. Call count: 30)". It is not a loop: the mock's count was cumulative across the file's cells, and the resumed run made one call into an empty queue. First sighting in the failed test:integ:e2e jobs since 2026-10-11.
- Cause (a test race, reproduced on main):
interruptHeldscripts[gate tool, finish('f1')], firesinterrupt()and released the held tool at once. Only cf-do-d1 waited for the stop to reach the run (interruptDelivered).- Core
stopRunwrites the stop (requestStop) before it signals the run. When that write was slower than the release and the step commit, the run's boundarypollStopfound no stop, made its next model call (consumingf1), and only then met the stop, which refused its K5 (interrupted). The resume found no reply left. - Delaying
requestStopby 10–40 ms reproduced the CI message exactly; 0–5 ms passed. The product behaviour is right: the recorded stop wins.
- Fix:
interruptHeldalways waits, before the release, for the fired stop to be recorded (readStopon the session's current run in the backend's own store, which every runtime's step boundary polls). This is the defaultinterruptDelivered; cf-do-d1 keeps its own, since the test process cannot read the DO's store directly.- With the fix the same delay injection passes at 10, 20, 40 and 300 ms.
- Also:
MockLLMAdapter's exhaustion message names the calls since the lastclearResponses()("Call count: 30 (1 since the last clearResponses())"). An adapter never cleared keeps the old message, andgetCallCount()keeps its lifetime meaning.
FU-FLAKE-E2E-D1-NONSTREAM-COMPLETED-ONLY-POLL: the AI SDK non-streaming e2e waits read a failed run as a timeout — fixed (test only; lands with #180's MR)
Seen: !558's train car 2996141719, job 17517465008 (test:integ:e2e 3/4): ai-sdk-d1-with-js.integ.test.ts > Non-streaming mode > should preserve message history in D1 without streaming failed with waitForState: predicate did not become true within 15000ms, and !558 was dropped from the train.
- Classification: a starved runner, not the MR.
- The file ran 167.5 s there against 82.7 s on !558's green MR shard (17513302097), with its non-streaming neighbours 2–4× slower.
- Re-run ×3 on !558 merged with main (e19f77d50a, slot B), it passed 27/27 every time (that test: 1.2–1.3 s).
- Why it was undiagnosable:
waitForState(the D1 file andai-sdk-redis-temporal.integ.test.ts) polledloadStatefor the caller's predicate and threw a bare timeout. A run that endedfailedand a run stillactiveprinted the same line. - Fix (test only): both helpers now fail at once on a
failedsession (the session ended failed: …) and, on a timeout, print the last state they read (status,stepCount,error,errorDetailcode and message). The budgets are unchanged. Mutation: a wait forcompletedon the scripted-error test throwsended failedwith the run's error.
FU-FLAKE-RCF-SANDBOX-CODE-BACKPRESSURE-TIMER-ROUNDING: the sandbox backpressure test asserts a wall-clock floor that timer rounding can miss by 1 ms — open
Owner: WA (filed 2026-10-27; a test-only fix).
- Sighting: !555 pipeline 2996243283, job 17518208434 (
test:unit), runtime-cloudflaresrc/workspaces/__tests__/sandbox-code.test.ts, "CloudflareSandboxCodeInterpreter › 9. runCode streaming awaits the onOutput callback (backpressure)":expected 44 to be greater than or equal to 45. It was the job's only failure (4307 passed). !555 changes nothing underworkspaces/, and the file passed on the previous pipeline and in a full local runtime-cloudflare unit run. It passed on a retry. - Cause: the test sums three
setTimeout(r, DELAY_MS)waits (DELAY_MS = 15) and assertsDate.now() - start >= 3 * DELAY_MS(:209-222). Node may fire a timer about 1 ms early againstDate.now()(millisecond rounding), so a correct, awaited run can measure 44 ms. - Fix direction: anchor the assertion on what the test means, that each
onOutputcall starts only after the previous one's wait has resolved. Assert from the callback's own observed timestamps (each start at least the previous start plusDELAY_MS, allowing about 2 ms of rounding per wait), or record the resolution order of the waits instead of elapsed time. A larger constant is not the fix. - Why it matters: a failure in
test:unitdrops a merge-train car.
FU-SANDBOX-EXEC-ERROR-EVENT-DROPPED: a sandbox exec stream that ends with error reads as "no terminal event" — retired (premise refuted; see DI-101)
Retired (2026-11-19): WE-2's read of the @cloudflare/sandbox 0.10.3 container (not the owner's; the npm package ships no container source) finds that its streaming route never sends error: a command that fails there ends with complete and no exitCode, which the shell reads as exit 0. That defect is DI-101. Parsing error stays a defensive item of its fix (the type still declares it); the text below is kept as written.
Surfaced by: the exec-stream terminal-event fix (DI-99). Related: DI-99.
@cloudflare/sandbox's ExecEvent can end a command with error (carrying error, a message). The exec SSE parser drops error on purpose (DI-99's type contract lists it as dropped), so CloudflareSandboxShell reports such a run as exitCode -1 with "stream ended without terminal event (timeout or upstream disconnect)" instead of the sandbox's own message. Fix direction: parse error as a terminal event, end the run with a non-zero exit code and put the sandbox's message on stderr (or reject typed); move it from "dropped" to "parsed" in the contract test.
FU-POSIX-SHELL-TIMEOUT-UNTYPED: a local or docker workspace command that times out reads as an exit code, not a typed timeout — fixed in code (batch 37)
Surfaced by: the CF sandbox timeout work (b36), which adds core WorkspaceCommandTimeoutError (workspace_command_timeout) for the sandbox shell. Related: DI-100, DI-101 and CP-342.
- local-bash and local-sandbox (
SubprocessShell,workspace-posix-core/src/subprocess-shell.ts:355-373, origin/main d6dd0022d7): the timer kills the process group, and the close handler resolvesexitCode: code ?? -1. A timed-out command reads as -1 with no marker, the same as any signal-killed process. - docker (
workspace-docker/src/shell.ts:95-112,engine/dockerode-engine.ts:207-223): its own path. The engine folds a timeout into exit 124 (guessed from 124 / 137 / 143 whentimeoutMsis set), and the shell dropstimedOut. A command that exits 124 / 137 / 143 by itself under a timeout reads as a timeout, and a real one reads as an exit. - An abort reads the same way: SubprocessShell's abort kills the process group and resolves
-1(subprocess-shell.ts:376-380); docker's abort sets the engine's deadline flag (dockerode-engine.ts:173-176), so a stopped command resolves exit 124 and reads as a timeout (its command is not killed: FU-DOCKER-ABORT-NO-KILL). - The docker suites run in no CI job:
workspace-docker'sdocker-real.integand e2edocker-js.integare gated onHELIX_DOCKER_TESTS=1(docker-available.ts), which no CI job sets, so neither the timeout nor the abort path is exercised against a real daemon in CI. Whether CI gets a docker job is pending a user ruling. - Fix direction:
- Each shell rejects with core
WorkspaceCommandTimeoutErrorwhen its own deadline fires (one error across providers, as the sandbox's), and docker keeps the engine'stimedOutinstead of an exit-code guess. - On a run's abort each shell rejects with
signal.reason, so the call is a stopped result under the stop rule (user ruling 2026-11-07; FU-TOOL-STOPPED-RESULT-TYPED), never an exit code. That's the target the CF sandbox shell takes in b36 (orchestrator ruling 2026-11-19), where an abort today resolves exit 0 with the partial stdout. - The provider suites and their timeout and abort cases move with it.
- Each shell rejects with core
Fixed in code (batch 37, origin/main 60cb09f994):
SubprocessShell(local-bash, local-sandbox) records what ended a command, the deadline or a stop, before it kills the process group, and rejects once the process is gone and every chunk reached the callbacks: past its limit with coreWorkspaceCommandTimeoutError, stopped with the signal's ownreason; a call already stopped starts nothing (workspace-posix-core/src/subprocess-shell.ts:278,:302-305,:399,:407and:378-385, origin/main 60cb09f994).- docker: the engine decides
timedOutorabortedby its own clock and returns the exec's real exit code, never a guess from 124 / 137 / 143 (workspace-docker/src/engine/dockerode-engine.ts:191-204and:238-242, origin/main 60cb09f994);DockerShellrejects a stopped call with the signal'sreasonand a timed-out one withWorkspaceCommandTimeoutError(workspace-docker/src/shell.ts:98and:114-115, origin/main 60cb09f994). - Abort still does not kill a docker command in the container: FU-DOCKER-ABORT-NO-KILL. The docker suites stay outside CI (above).
FU-CF-SANDBOX-STREAMED-COMMAND-NOT-KILLED: a timed-out or aborted streamed sandbox command keeps running in the container — open (LOW)
Surfaced by: the CF sandbox timeout work (b36). Related: DI-100 (same path), CP-342 (the stop on the same path), FU-POSIX-SHELL-TIMEOUT-UNTYPED.
CloudflareSandboxShellstreams everyworkspace_runthroughsandbox.execStream. On a timeout or an abort it can only cancel the stream (runtime-cloudflare/src/workspaces/sandbox/shell.ts:121-137, origin/main d6dd0022d7, the "UPSTREAM SDK GAP" note).- On
@cloudflare/sandbox0.10.3 (WE-2's read of the container, not the owner's) the streaming exec route returns no process id (execute-handler.ts:147-152) and itscancel()only removes listeners (:255-257). So the command runs on until it exits or the container is idle-evicted. - With b36's deadline the tool call answers a typed timeout while the command keeps running and writing. A retried command can then run twice at once.
- docker has the same gap on an abort: FU-DOCKER-ABORT-NO-KILL.
- Fix direction:
- Run a streamed command through the process routes the client already exposes (
startProcess+streamProcessLogs), so a deadline or an abort cankillProcessit. - Or keep
execStreamand document the gap wheretimeoutMsis documented. - The shell.ts note's "the configured timeout fires" is corrected either way: the streaming route drops the timeout.
- Run a streamed command through the process routes the client already exposes (
FU-DOCKER-ABORT-NO-KILL: an aborted docker workspace command keeps running in its container — open (LOW)
Surfaced by: the POSIX shell timeout work (2026-11-20). Related: FU-POSIX-SHELL-TIMEOUT-UNTYPED (the same abort's result, typed by its fix), FU-CF-SANDBOX-STREAMED-COMMAND-NOT-KILLED (the sandbox's twin).
- On a run's abort,
DockerodeEngine.execdestroys the host exec stream and marks the resultaborted, and the shell rejects with the signal'sreason(workspace-docker/src/engine/dockerode-engine.ts:194-196, origin/main 60cb09f994). Nothing signals the command in the container, and killing adocker execfrom the host does not reach it (PID namespaces). - The in-container
timeout -s KILLwrapper (:149-158, origin/main 60cb09f994) bounds the command only when the call has atimeoutMs(ormaxDurationMs); with none, an aborted command runs until it exits or the container stops. A call whose signal had already aborted starts no exec (:138-146). docs/workspaces/docker.md:120(origin/main 60cb09f994) now says an abort only releases the call. Before, it said the wrapper and the backstop bound "an abortedAbortSignal". The workspace-docker design (2026-06-01, :102, :225) planned anexec.inspect()PID kill, which was not built.- Fix direction: on abort, kill the command inside the container: a pid file the wrapper writes, killed by a second exec, or the
timeoutwrapper's signal forwarding (coreutils and busybox differ). Proving either needs a live daemon, and the docker suites run in no CI job (FU-POSIX-SHELL-TIMEOUT-UNTYPED).
FU-LOCAL-SANDBOX-ISOLATION-NOT-IN-CI: no CI job runs workspace-local-sandbox's real-isolation suite — open (LOW)
Seen: !959 (FU-POSIX-SHELL-TIMEOUT-UNTYPED), MR pipeline 3096525586, test:unit job 18240216084: src/__tests__/isolation-real.test.ts (11 tests | 10 skipped); workspace-local-sandbox reads 47 passed + 10 skipped there, against 56 + 1 on a macOS host.
- Known, by design today: the unit job installs bubblewrap, but the shared runner cannot create user namespaces, so backend detection resolves to
noneand the suite skips; theHELIX_REQUIRE_SANDBOX_BACKENDtripwire is left unset for that reason (ci/test-matrix.yml:58-62, origin/main 60cb09f994, thetest:unitjob'sbefore_scriptcomment). The suite isdescribe.skipIf(BACKEND === 'none')(workspace-local-sandbox/src/__tests__/isolation-real.test.ts:48, origin/main 60cb09f994;isolation/detect.tsresolves seatbelt on darwin, bwrap on linux only when it can actually create a sandbox). Its one passing test in CI is the opt-in guard (:293), which returns early unless the tripwire is set. This entry records what that leaves out. - Untested in CI: every kernel-isolation pin of the provider: in-workspace commands through the wrapper, writes outside the workspace denied,
network: 'off'egress blocked, the seatbelt profile's tamper guard,readWritePaths/readOnlyPathsat the kernel,resolve()rebuilding the isolation config, and (since !959) a timeout and a stop through the wrapper. They run only on a developer host with a backend. - The bwrap-only case runs nowhere routinely: "bwrap confines reads: an out-of-bind host file is NOT readable" (
it.skipIf(BACKEND !== 'bwrap'),:253) skips on macOS hosts (seatbelt) and in CI (none). - Direction: a runner that can create user namespaces for a non-root user (a dedicated runner, or a job image and runner configuration that allow it), with
HELIX_REQUIRE_SANDBOX_BACKEND=1on that job so a missing backend fails instead of skipping. Runner and CI configuration are the user's call.
Test only: no product change.
FU-FLAKE-RCF-WITH-MESSAGE-OWED-SIBLING-DEADLINE-ENTRIES: a with_message owed-continuation unit counted two entry instances once — fixed in code (lands with its MR; a test-timing artefact, no product defect)
- Sighting: !861 pipeline 3065551041, job 18012472802 (
test:unit, merge ref c278ff1ffe), runtime-cloudflaresrc/__tests__/executor-with-message-owed-continuation.test.ts, "Task 7 review ruling: an EXHAUSTED submit continuation discharges the owed status to the next entry › fix round 5: a sibling deadline that lapses only AFTER the last submit is NOT owed …":expect(entries).toHaveLength(1)at:824got 2 entries. It was the package's only failure (4852 passed, 5 skipped). The runtime-cloudflare code is identical to main 9795bfeb38, the same job on the MR's previous pipeline (3065108926, job 18009065098) passed, and W1 ran the file alone 20 times on macOS with no failure. - Second sighting: a local slot (2026-11-21), runtime-cloudflare's whole
testsuite at branch reliability-cp336-cfw-reconnect-own-output 20b2329d00 (main 84b10abd73 plus a CF Workflows reconnect-handle change this case does not reach: its handles areexecute()/resume({ mode: 'with_message' })active handles). The same case and line:expect(entries).toHaveLength(1)at:824got 2. It was the suite's only failure (4945 passed, 5 skipped); the same suite at the same head passed an hour earlier (4946 passed). The entry ids were not captured (vitest truncates the array, and the run wrote no JSON report). - Cause (traced at origin/main 84b10abd73; the CF Workflows code is unchanged at 04b758ea69): turn 1's CP-138 deadline watcher. The test's inline engine runs every instance with
passThroughStep(), whosesleepUntilreturns at once (the test's:60-70), and the watcher runs on that step (runtime-cloudflare/src/workflow.ts:855-878), so its rounds (client-tool-deadline.ts:247-268, inrunClientToolDeadlineWatch) spin back to back. A round still pending when the case seeded c2's already-past deadline expired c2, found the batch ready and continued the owed root (workflow.ts:872-877): a__submit__instance, the second entry. Which entry came first was up to scheduling. The real engine sleeps until c2's original deadline, so this was the test's timing, not a product defect. - Fix (test only, as on branch reliability-flake-owed-sibling-deadline 1e56bbe1a8): "fix round 5" waits for the engine to drain (
e.drained()) before its seed (:823-827), and its assertion namese.created(:858). Two order pins hold the race with a gated watcher, "watcher first" (:932) and "entry first" (:977): in either order each entry starts one run and makes one model call, and the history stays paired; watcher first gives one__submit__continuation, then the message as the next__turn__. A slot at that head: the package'stestsuite 4945 passed, 5 skipped; the file 26 of 26, and 26 of 26 in five repeats; with the watcher's expiry disabled in the product, 1 of 26 fails, exactly "watcher first".
FU-FLAKE-CFW-WF-NOISO-WAIT-BUDGET-UNDER-LOAD: CF Workflows real-engine tests miss fixed wall-clock budgets on a starved runner — open
Owner: backlog (next CI-health session) / WB. Each case passes alone and on main.
- Sightings (
packages/runtime-cloudflare/src/__tests__/,test:workflows, runner load 24-34):- !325 pipeline 2931531317, job 17067827203:
cfw-stop-conditions.wf-noiso.test.ts, the C5 case "retry.max-steps-continues-without-message": itswaitForTurnpoll timed out withruns=3(:128) and the session stillactive, so the run was progressing slowly, not stuck;cfw-terminal-truth.wf-noiso.test.ts, "T4 (CP-89): 20 fresh executes": itswaitForTerminalpoll timed out (:149) at the sixth execute, with the sessionactive. The same job logged an uncaughtWorkflowTimeoutError(30000 ms) in workerd.
- !331 pipeline 2931527520, job 17067794608:
rm29.wf-noiso.test.ts, the "a-finishwith-siblings" and "a2-duplicate-finishwith" cases:Test timed out in 60000ms(vitest's per-test budget; 60.0 s and 70.8 s).
- !325 pipeline 2931531317, job 17067827203:
- Cause (a §F1 timing assumption): each test bounds real-engine work with a fixed wall clock: a 30 s polling budget, or vitest's 60 s test timeout. Under starvation the engine's turns run many times slower than usual, so a correct run misses the wall.
- Fix direction: budgets derived from observed progress (a run that is still advancing gets more time; a stalled one fails fast and names its last state), not larger constants.
FU-FLAKE-SCRIPTED-GATE-FIXED-DEADLINE: ScriptedModel.whenHeld failed a gate at a fixed deadline from the wait on a starved runner — done (test harness only; lands with its MR)
Owner: W1. Seen: batch 1 (!567) pipeline 3000815273, test:integ:e2e 1/4 job 17552299950.
Signature:
refresh-c1.integ.test.ts, "C-1 refresh mid a multi-step / HITL response (RM-08) — cfw-d1", all three variants of "refresh mid hitl-continuation" failed on aScriptedModelError: "gate '…' not held within 30000ms", fromwhenHeldinscripted-model.ts.Not a regression: W1's bisect ran exactly those 3 tests on main, +!562, +!563 and the batch, and every run was 3/3 green, at about 4 s each.
The runner was starved: in that job the whole file ran 727 s, against 157 / 186 s on two green car jobs (!560 train job 17525214175, !555 train job 17547756463). Unrelated cfw-d1 tests in the same file were 4–5× slower, and the hitl control without a refresh passed at 32–36 s (6–11 s on the cars).
Cause:
whenHeld's timeout was a FIXED deadline from the wait. So it had to cover the run's start and every step before the held call (here: a client-tool suspension, the submit, and the continuation's start), each of which stretches with the runner's load.PollingHoldModel's gate in the same scenario already used an idle budget for exactly this reason.Fix (no timeout raised):
whenHeld'stimeoutMsis an IDLE budget on the gate's session. It fails only after that session has made no model progress fortimeoutMs:- progress is a call starting, a text delta forwarded by a
holdAfterDeltascall, or a call ending, counted only for calls of the current epoch; - the gate's session comes from
scriptFor, or from a child's script once that child adopted it; - a gate whose session is unknown (agent-type or global scripts, a child not adopted yet) keeps the fixed deadline from the wait, so another session's progress never keeps a gate alive;
- a
reset()ends a pending wait at its next timer.
- progress is a call starting, a text delta forwarded by a
Limit: the budget restarts on the session's own model progress only. Runtime work between two model calls (here: the client-tool suspension, the submit and the CF Workflows continuation's start before the held call) must still fit in
timeoutMs. That latency is FU-FLAKE-REFRESH-C1-CFW-SUBMIT-CONTINUATION-LATENCY. This fix removes the run's start and the earlier calls from the budget; it does not bound that gap.Proof: the core unit file's cases run on fake clocks:
- progress past the old deadline: not failed, then held;
- idle for the budget: still fails, counted from the last progress;
- another session's progress: fails;
- a held call's own streamed deltas: keep it alive (Greptile);
- an agent-type gate: a fixed deadline;
- an adopted child's own progress: keeps it alive;
- a wait pending at
reset(): ends at its next timer; - a call from before
reset(): not progress.
The first two are red on the old fixed deadline, and each later rule has a mutation that reds its case. The
refresh-c1"C-1 … — cfw-d1" describe passed 12/12 in each of 3 local runs (W1 slotw1-flakegate-20261028-171128, on the pre-rebase head 4da9df0df4).
FU-FLAKE-DO-CONTEXT-413-UPLOAD-RESET: an oversized context upload to the DO can end in a write ECONNRESET or EPIPE instead of the 413 — done (test fix in batch 20; owner: W1)
Filed: 2026-11-10 (orchestrator, from W1's trace).
- Seen:
test:integ:restjob 17905685178 on batch 19's MR !823 (write ECONNRESET), passing on its retry 17907592881. Again on 2026-11-10 in WE-2's local run w-c311k (the full runtime-cloudflare integ at main's test):write EPIPE(errno -32). - W1's trace: the DO refuses the request on its declared
Content-Lengthbefore it reads the body, so the client's write of the rest of the upload can fail withECONNRESETorEPIPE(which one depends on when the close lands) before the 413 answer is read. The refusal is correct; the test reads the wrong signal. - Closes with: W1's batch-20 test fix, which accepts the 413 or exactly one of those two socket codes.
- Closed: batch 20's MR !838 pipeline 3052793322 (
/merge), job 17922754393 (test:integ:rest):do-add-context.integpassed its 15 tests and its cap case answered HTTP 413 (one CI pass).
FU-FLAKE-CONFORMANCE-COMPANION-CONTINUE-CHILD-REF-READ: the companion-continue cell read the child's ref right after the parent settled — fix in owner batch 32 (test only), the eventual ref still to confirm
Owner: conformance owner. Surfaced by: !356 merged-results pipeline 2952121312, test:conformance 3/5 job 17210523837: history.companion-continue|js-postgres|executor failed .child-ref-interrupted, stop={"ok":true} with the child's ref still running; 308 other cells in the shard passed. !356 changes no runtime-js, core or store code.
- Cause (a §F1 timing assumption in the cell): the cell read the parent's sub-session refs once, right after the stopped turn's outcome settled. The child's ref is written by the child's own loop as it unwinds, which can land after the parent's outcome (a stop returns at once under D-1), so the read raced that write.
- Fix (owner batch 32,
scenarios/history-b2.ts): the read waits, bounded by the turn bound, for the ref to leaverunning; a ref stillrunningat the bound, or any other status thaninterrupted, still fails the id. - Still to confirm: that the ref does reach
interruptedon js-postgres (the CI store is gone). If the bounded read ever fails with the ref stillrunning, the ref write is missing: a product finding, not this flake.
FU-FLAKE-CORE-COMPANION-DISPATCH-REALTIME-POLL-COUNT: a waitForResult guard counts polls inside a real 50 ms window — fixed by !364 (owner: WA; lands with !364)
Test: packages/core/src/orchestration/__tests__/companion-tool-dispatch.test.ts, "CP-103/W1 guard: NO probe, RUNNING ref + INTERRUPTED session but a LIVE child run … → NOT awaiting a resume". Seen: locally, W1 MR3's fix-round gate (2026-10-16) at a 1-minute load average of 88: expected 3 to be greater than or equal to 4. It passed 5/5 at load 14 and in CI.
- Cause: a test defect. It waits with
timeout: 50real milliseconds andwaitForResultPollMs: 1, then asserts at least1 + CONFIRMATIONS(4) ref reads. How many polls fit in 50 ms of wall-clock time depends on host load. Its sibling "CP-103 guard: INTERRUPTED ref but a LIVE child run" has the same shape. - Not fixed by !364 (WA checked !364's head 832797b787: both cases still pass
timeout: 50real ms with no injected clock; !364 only changed their expected result to the typed{ name, status: 'timeout', currentStatus }). - Fix (in !364, orchestrator ruling 2026-10-16; test-only; WA's commit e2e83c31fd, rebased sha to follow): !364 adds optional
now/sleeptoCompanionToolDispatchDeps, andhandleWaitForResulttakes one deadline decision per poll, after the read. Both CP-103 live-run guard cases now run on that injected clock, which advances only onsleep. They assert exactly 7 reads (the initial resolve plus 6 polls) and the unchanged typed timeout.- Red by mutation: with
sleepadvancing 3× per call, both fail withexpected 4 to be 7; the old>=bound could not see that. - Green: 114/114, 3 of 3 runs.
- If !364 lands without it, this entry goes back to open.
- Red by mutation: with
FU-FLAKE-COMPANION-RESUME-HANG-HEADERS-BOUND: the stalled-body companion case races the child's headers against the 300 ms resume bound — fixed by !364 (owner: WA; lands with !364, sha to follow)
Test: packages/runtime-cloudflare/src/__tests__/companion-resume-hang.cf.test.ts, "headers, then a stalled body: the abort ends the body read → child_start_unconfirmed; the ref stays interrupted" (test:cloudflare, pool-workers on Miniflare 4). Seen: !367's pipeline 2953580946, job 17221277249 (retried by the orchestrator): expect(resume.headersAt - resume.calledAt).toBeLessThan(RESUME_BOUND_MS) (:153) got expected 322 to be less than 300.
- Cause: a test defect, a wall-clock race. The case proves that the resume timeout aborts a hung BODY read, so it needs the hung child's response headers to arrive before the 300 ms bound (
RESUME_BOUND_MS, also the executor'sresumeTimeoutMs). On a loaded runner the headers arrived at 322 ms, so the abort hit the headers wait instead, and the case's premise failed. The product's abort is not in question. - Fix direction (orchestrator ruling): anchor on an observed event, not a larger bound. The case waits until the child has observably sent its headers before it relies on the body stall. For example, the hung child records "headers sent", the test awaits that, and the bound then measures only the body read. Its sibling ("no answer") asserts the abort at the bound and has no headers race.
- Fix (in !364, WA; test-only; WA's commit 9a395565d6, rebased sha to follow): in the stall case, the test's namespace wrapper gives
stub.fetcha test-owned signal, and forwards the tool's abort only after the response headers resolved. So "headers, then the abort ends the body read" holds under any load.- The wall-clock
headersAt - calledAt < 300is replaced by event order:headersAtis defined,abortForwardedAt >= headersAt, and the tool's signal is aborted. - The "no answer" case still passes the tool's signal straight through.
- Green 3/3 under the lock.
- If !364 lands without it, this entry goes back to open.
- The wall-clock
FU-FLAKE-COMPANION-RESUME-HANG-NO-ANSWER-LOWER-BOUND: both companion resume-hang cases asserted a wall-clock floor that workerd's clock cannot honour — fixed (test only; lands with its MR)
Test: packages/runtime-cloudflare/src/__tests__/companion-resume-hang.cf.test.ts, "no answer: the /resume fetch is aborted at the bound → child_start_unconfirmed; the ref stays interrupted" (test:cloudflare, pool-workers). Seen: job 17258050148 (retried by the orchestrator): expect(elapsed).toBeGreaterThanOrEqual(RESUME_BOUND_MS - 50) (:163) got 236 against 250.
- Cause: a test defect.
elapsedis measured withDate.now()inside workerd, which advances only at I/O boundaries (the Spectre mitigation), so it under-measures, and a floor on it is flaky by construction. The stall case carried the same floor. The product's bound is not in question. - Fix (test only): "aborted at the bound" is proven by events, not elapsed time.
- The bound's own abort: the signal's reason is a typed
HelixErrorwithcode: 'transport_error', which onlyfetchChild's timer sets. - Not before the bound: a probe timer armed at the
/resumecall fires at half the bound and records that the signal was still live. Timers fire in deadline order on the frozen clock. - The upper check (
< HANGING_CHILD_HOLD_MS) stays: a frozen clock can only under-measure. - Proof: ×10 green; mutations of
fetchChildturn both cases red: the bound firing at 0 ms, and an untypedabort().
- The bound's own abort: the signal's reason is a typed
FU-FLAKE-RCF-SSE-STALE-EVICTION-WORKERD-CLOCK: the workerd stale-eviction case waits 30 ms by timer for a 20 ms Date.now() window — open (test only)
Test: packages/runtime-cloudflare/src/__tests__/sse-unread-budget.cf.test.ts, "an evicted connection whose write is blocked on a read releases everything at once" (test:cloudflare, pool-workers). Seen: W1's local slot s-cp338 (2026-11-19, at 34e1be5cbd, which changes neither file; load 26, memory level 1): expect(manager.hasConnection('stalled')).toBe(false) (:103) got true; the case took 122 ms, and the file's other 4 cases passed. Green in main's train pipeline 3086985937 (test:cloudflare job 18166995347, 265 passed).
- Cause (from the code, not measured):
heartbeat()evicts only whenDate.now() - lastSuccessfulWrite > inactivityTimeoutMs(core/src/stream/sse-connection-manager.ts:634).lastSuccessfulWriteis stamped atregister(:358) and the stalled write never resolves. The test sets the window to 20 ms and waits a 30 mssetTimeoutbeforeheartbeat(). So the red meansDate.now()moved at most 20 ms across that timer. Inside workerdDate.now()advances only at I/O boundaries and can under-measure a timer's delay, as in FU-FLAKE-COMPANION-RESUME-HANG-NO-ANSWER-LOWER-BOUND (236 ms across a 250 ms bound). - Not a product defect: the product measures both sides with
Date.now()at a 60 s default; an under-measure only moves an eviction to a later heartbeat. Not FU-CF-TIMER-CLOCK-PRODUCT-TIMERS: no product timer, and no binding-only stretch. - Fix direction (test only): wait with core/testing
elapseWallClock(30)(it re-arms untilDate.now()moved 30 ms; pinned byworkerd-timer-clock.cf.test.ts) instead of the baresetTimeout. Then prove the case red with the stale check removed and green ×10. - Re-run: W1's s-cp338 run 2 at the same head passed the case (155 ms;
test:cloudflare265 passed). - CI: until fixed, a recurrence is this entry's: one disclosed retry, its links on record.
FU-E2E-DEFAULT-ESLINT-UNCOVERED: packages/e2e has no lint script, so no CI lint job sees its errors — open (LOW)
Owner: conformance owner. Found: owner batch 33 (2026-10-15); widened 2026-11-19 by WA's slot.
- Why CI is green:
packages/e2ehas nolintscript. CI's lint job runsturbo run lint --only(ci/test-matrix.yml), which skips a package without one.lint:conformance(the floating-promise config, run byconformance:verify) coverssrc/conformance,src/harnessand the conformance integ files only, and does not enable the default rules. - What it hides: two prefer-const sites from 2026-10-15 (
executor-driver.test.ts'slet h,:2162at e88a92f6cb;conformance-worker.ts'slet doSql,:765), and the 6 errors (unused names) and 10 warnings WA's slot found insrc/__tests__(corrupted-staging-recovery,follow-up-tool-streaming,multi-run-streaming×3 andstream-sequence-integrity, all.integ.test.ts). - Fix direction: add a
lintscript topackages/e2eoversrc/**, the harness and conformance included; fix what it reports; confirm CI's lint job then covers e2e with no CI config change.
FU-ROOT-SCRIPTS-UNLINTED: no CI job lints the root scripts/*.mjs — open
The lint job runs each package's lint script (turbo run lint) plus check:turbo-env, so the root scripts/ (check-turbo-env.mjs, check-own-key.mjs, scripts/ci/*) are never linted; the root eslint config declares no Node globals for them. To do: lint scripts/ in CI, with the Node globals imported or declared.
FU-CFW-MINIFLARE-IDLE-STEP-EVICTION: a CF Workflows step idle on a subrequest for more than about 60 s stops progressing under Miniflare — open (confirm production)
Owner: conformance owner (to confirm), then CF runtime owner if production shares it. Surfaced by: owner batch 33's probe (2026-10-15), companion.wait-slice-timeout-keeps-child-live on cfw-workflows-d1 with its child's gate released at a chosen time: released 25 s after the child's tool entered it, the child completed about 1 s later; released at 65 s, the child's session stayed active with 1 message for the 85 s the probe watched (host load about 9).
- Reading (WA, code-read): the tool's
step.dohas no step config (runtime-cloudflare/src/workflow.ts), so it gets the engine default (5 retries, a 10-minute timeout). The engine DO idle on the step's outbound request is likely evicted, losing the in-flightrun(); the instance resumes only at the step's timeout alarm. - To do: confirm against Cloudflare's Workflows documentation whether production has an idle bound on a step's outbound subrequest (or on an idle engine). If it does, a long-running tool hits it in production: a finding (a step config sized for long tools, or a heartbeat), with a real-engine witness. If not, this stays a Miniflare constraint, recorded in
docs/dev/conformance-scenario-authoring.md(the CF Workflows lane: never hold a tool step idle past about 60 s).
FU-FLAKE-AISDK-REACT-WAITFOR-UNDER-LOAD: ai-sdk React hook tests miss fixed waitFor / test budgets on a starved runner — open
Owner: backlog (next CI-health session). Surfaced by: !325 pipeline 2931531317, test:unit job 17067827187, with the runner at load 24-34 (three helix pipelines plus a local conformance matrix). Each test passes alone.
- Signatures (
packages/ai-sdk/src/__tests__/react/):failed-resume-and-retry.test.tsx› "review I9: useHelixChat after a failed resume / send" › "retry() after a refused send (the turn is not in the snapshot) re-sends it": over its 10 s test timeout (16.6 s);use-helix-chat-resync.test.tsx› "a resync on a RESUMED stream: stop aborts it, snapshot set, resumed from the cursor — written once": over its 10 s test timeout (13.1 s);use-auto-resync.test.tsx› "client.error-after-exhaustion: an unavailable snapshot surfaces SnapshotUnavailableError, messages untouched" and "I5: a resync is safe by default › … typed ConfigurationError; nothing fetched or set":waitFor's default 1 s budget ran out withresyncErrorstill null.
- Cause (a §F1 timing assumption): fixed wall-clock budgets on work whose duration scales with host load.
- Fix direction: condition-based waits with budgets derived from the work each test does, not larger constants.
FU-FLAKE-E2E-RESUME-COUNT-CONCURRENT-ENTRY-NO-HOLD: resume-count's "two concurrent resumes" let both entries win one after the other — done (the 12b(b) MR, WE-2)
Seen: 2026-10-17, C1's merge-train pipeline 2958771303, test:integ:e2e 4/4 (job 17256911401): resume-count-parity.integ.test.ts › "resumeCount: counted by the run-start winner — 'js-postgres' › resume(): two concurrent resumes" failed exactly one winner: expected [2 entries] to have a length of 1. It dropped C1 from the train.
- Not a double run. The Postgres store decides a run-start commit under the session's
statesrowFOR UPDATE, and runtime-js CASes on the run each entry read at entry (seenRun);shouldRetryRunStartretries only a stale read of that run. Two fulfilled entries therefore mean the second read the session after the first run had ended: a later, legitimate resume. The scripted reply finishes a run in milliseconds, so a loaded shard could reorder them. A/B, interleaved: main880cdb8c1420/20 and C1's train tree 20/20, 0 races. - Fix (test-only): the
resume()andretry()races start the second entry 200 ms after the first and hold the first's run live at its model step (holdModel) until both settled (heldRace,e2e/src/helpers/resume-count-scenarios.ts). The delay makes the hold load-bearing: without it,resume()gets two winners on every run (js-memory and js-postgres, 2/2); with it, 3/3 green. (retry()of an already-completed session is refused by its status gate, so its hold only removes the dependence on timing.)
FU-FLAKE-E2E-CUSTOMSTATE-PARALLEL-FOLD-CF-DO: the CF DO parallel customState fold e2e fails under load — open
Owner: backlog (next CI-health session). Surfaced by: !325 pipeline 2931531317, test:cloudflare job 17067827202 (load 24-34). It passes alone.
- Signature:
packages/e2e/src/__tests__/customstate-parallel-fold-cf-do.cf.test.ts. "composes all 16 folds (modify)" threwAgent did not complete within 60000msfromwaitForCompletion(:72); "composes all 16 folds (append)" endedfailed(expected 'failed' to be 'completed', :115) after workerd loggedtest-worker.ts changed, invalidating this Durable Object. Please retry the DurableObjectStub#fetch() call.(broken.inputGateBroken) several times. - Lead: the invalidations are the vitest-pool-workers module-reload path (the worker file "changed" mid-run). Under load a reload lands inside the run, breaking the DO's input gate. The 60 s completion budget is fixed (§F1).
- Fix direction: find what triggers the reload mid-file, and derive the completion budget. Don't add retries.
FU-FLAKE-SETUP-OPTIONS-TEMPORAL-ENV-TIMEOUT: the setup-options unit test's first Temporal case times out on a starved host — open
Surfaced by: packages/e2e/src/conformance/__tests__/setup-options.test.ts "temporal-memory rejects workspaceProviders with a HarnessError" failing at its 30 s test bound on a host at load ~110–140 (the owner's !381 verify, 2026-10-16; WA the same day), 9 of 9 green when re-run alone.
The case never reaches a Temporal environment: the setup helper rejects the option before it touches any infrastructure. What it does pay is the first case's cold await import('../../harness/setup-helpers/temporal-memory.js'), which loads the Temporal client and testing packages, inside the test's own 30 s budget (the same shape as FU-FLAKE-CORE-INVALID-RESULT-DYNAMIC-IMPORT). Fix direction: load the four helpers in a beforeAll with its own generous bound (or warm the import once per file), so each case judges only the rejection. Test only; no product change.
FU-FLAKE-CORE-COMPANION-LIVENESS-PROBE-POLLS-TO-TIMEOUT: companion waitForResult had two wall-clock timeout exits — done (fixed by !364, FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES)
Seen: 2026-10-12, a full @helix-agents/core unit run on the Mac at load average ~13 (other sessions' runs), during WE MR C1. core/src/orchestration/__tests__/companion-tool-dispatch.test.ts › "executeCompanionToolDispatch — waitForResult orphan detection › without the liveness probe, behavior is unchanged (polls to timeout)" failed in 57 ms; the file then passed 3/3 alone. A wall-clock polling budget under CPU starvation is the likely class (as FU-FLAKE-CFW-WF-NOISO-WAIT-BUDGET-UNDER-LOAD).
Recurred: 2026-10-17, CI, WE MR C1's MR pipeline 2957807112, job 17251692051 (test:unit): the same describe's "does NOT orphan a paused_awaiting_client child (legit stateless HITL wait)" failed with expected 'timeout (current: paused_awaiting_cli…' to be 'timeout'. C1 does not touch the file or companion-tool-dispatch.ts. Locally the file passed 3/3 on both C1 and main.
Root cause: waitForResult's poll loop (core/src/orchestration/companion-tool-dispatch.ts, the while (true) loop) has two timeout exits, chosen by the wall clock. A deadline that passes while the loop sleeps returns status: 'timeout' (the check before the poll's I/O). A deadline that passes during the poll's I/O (resolveAndSyncSubSessionRefs) returns `timeout (current: ${status})` (the check after it, and the awaiting-resume branch's own). The tests pass a 50 ms timeout with a 1 ms poll and pin the first string, so a slow poll under load takes the second exit. The product contract is "a timeout"; the suffix is extra context.
Fix: in the product, not the tests: one timeout answer per condition. !364 (WA, "waitForResult: one typed answer", FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES) decides the deadline once per poll, after the read, and clamps the sleep to the time left, so a single exit can fire. Verified (WE-2, 2026-10-17): with !364 on main (4d355e1eca), the loop decides its deadline once per poll, after the read, so one timeout shape remains; companion-tool-dispatch.test.ts passed 3/3 (118 tests each) in a capped lock slot.
FU-FLAKE-CORE-INVALID-RESULT-DYNAMIC-IMPORT: core's INVALID_RESULT_MESSAGE (R64) test times out on a cold barrel import — fixed in WA MR2 (7ab0b0218c), lands with its merge
Owner: WA MR2. Sightings: three on main before the fix merges. The latest is !346 pipeline 2939923693, job 17122046391 (edge runner).
- Signature:
packages/core/src/__tests__/client-tool/route-submit.test.ts,INVALID_RESULT_MESSAGE (R64), exceeds its 10 s test timeout under host load. - Cause: an in-test
await import('../../index.js')transforms the whole core barrel cold, inside the test's timeout. - Fix (WA MR2 Task 7,
7ab0b0218c): a static top-level import, so the barrel is transformed at collection, outside any test's timeout. An audit of core's tests found no other in-test barrelawait import(; the two left inllm/scripted-model.test.tsimport a 72-line module. - Until WA MR2 merges, main keeps the in-test import, and a retry of the job is the workaround.
FU-FLAKE-LLM-VERCEL-SETTLE-WITH-TIME-COLD-IMPORT: llm-vercel's drop-retry test times out when the adapter's first import('ai') is slow — fixed (test only; lands with its MR)
Owner: W1. Sightings: one, in !509's train car (pipeline 2986067390, test:unit job 17445898281; its retry 17446148065).
- Signature:
packages/llm-vercel/src/__tests__/vercel-adapter-connection-drop-retry.test.ts, the file's first test (workerd: one drop, then the retry streams normally), "Test timed out in 10000ms"; the other 22 tests pass. - Cause: the tests'
settleWithTimeadvanced fake time a fixed number of times (60 slices), and a slice yields ONE real event-loop turn, so the whole budget is about 60 real turns. The adapter's first call awaits a coldimport('ai')(loadAi); when that took longer on a busy runner, the slices ran out before the first attempt's drop, and the 2 s backoff timer it then set was never advanced. Reproduced by delaying the first drop by 300 real turns: the exact CI signature. Not a product hang: the retry completes once fake time moves. - Fix (test only): in that file and in
vercel-adapter-provider-unreachable.test.ts(the samesettleWithTime, andsettleWithin, which backs the twoconnectTimeoutMsdeadline tests), fake time moves only while a fake timer is pending (vi.getTimerCount()); otherwise the helper waits a real turn. The fake-time budget stays, andsettleWithTimethrows past it instead of hanging; a call that sets no timer and does not settle within 5 s of real time is reported as hung (a turn count cannot bound that: an idle loop spins thousands of turns a millisecond). No timeout was raised.
FU-FLAKE-E2E-STRICT-STRUCTURED-D1-JS-MODULE-FETCH-RPC: a vitest worker module-fetch RPC timeout fails a whole CF e2e file — open
Owner: backlog (next CI-health session). Surfaced by: !325 pipeline 2931531317, test:cloudflare job 17067827202 (load 24-34). It passes alone.
- Signature:
packages/e2e/src/__tests__/strict-structured-output-d1-with-js.cf.test.tsfailed as a file (FAIL … [ src/__tests__/strict-structured-output-d1-with-js.cf.test.ts ]) onError: [vitest-worker]: Timeout calling "fetch" with "[…/strict-structured-output-d1-with-js.cf.test.ts","ssr"]", before any of its tests ran. - Lead: the same family as FU-FLAKE-CF-VITEST-ONTASKUPDATE-RPC-TIMEOUT: a vitest 2.1.9 worker-to-main birpc call (here the module
fetch) has a fixed timeout, and a starved host misses it. - Fix direction: as there: measure, then fix the starvation or split the work. Don't raise the timeout.
FU-CORE-TESTS-TYPECHECK-DEBT: core's __tests__ do not type-check (482 errors in 77 files at #179 M2) — open (LOW, test hygiene)
packages/core/tsconfig.json excludes **/__tests__/**, and no other config type-checks them, so test code drifts from the types it exercises. #179's pre-slot check type-checked them through a scratch tests tsconfig (core's config, tests included, vitest/globals types) and found 482 errors in 77 files, nearly all pre-existing (for example client-tool/route-submit.test.ts submits without kind). #179 gates only on errors in the files a member changes.
- Fix: add a
tsconfig.test.jsonand atypecheck:testsscript to core and run it in CI, after burning the existing errors down (or ratcheting: fail only on a count increase).
FU-OTEL-JS-DRAIN-TOOL-SPAN: an approved approval-gated call executed by runtime-js's resume drain has no span — open (LOW, #179 follow-up)
drainOrphanClientToolCalls (runtime-js/src/js-agent-executor.ts) executes an approved approval-gated call (executeSingleToolCall) BEFORE the resumed run starts, so there is no run span to parent it and its beforeTool / afterTool hooks carry no trace fields (#179 M2). Fix: give the drain its own span under the resume's parent, or move the approved execution into the resumed run's first step.
FU-OTEL-CONCURRENCY-GROUP-SPAN-WAIT: a grouped tool call's execute_tool span includes its wait behind the group — fixed in code (lands with its MR)
runtime-js (and so the CF DO) opens a call's execute_tool span when the call is prepared (toolSpanFor in the prepare path), before core's step iterator chains a concurrencyGroup call's body behind its group's previous call (groupTails). So a queued grouped call's span (and its Langfuse tool observation) also covers the time it waited, not only its body.
- Fix: the span still opens at prepare, and the wait is its own child span,
helix.tool.wait(helix.tool.concurrency_group), from the end of the call's prepare to its body's start. Only a call that actually queued has one: core's step iterator says so in the call'sprepareServerToolinput (queued: true), so runtime-js never re-derives the group order. runtime-js opens the span at the end of the prepare and ends it when the body starts, so the SDK stamps both ends on its own clock. - A queued call whose body never runs (a stop, or a phase-1 call that rejected meanwhile) is reported through the prepared call's
skipped(reason). Its wait ends at the skip, and itsexecute_toolspan endshelix.tool.outcome=not_executed, reasonaborted/run_ended(before, it ended unset at the step's end). - A queued call whose run is superseded while it waits throws before its body (D2): its wait span and its
execute_toolspan end unset at the step's end, as any open stage does.
FU-OTEL-AI-SDK-CHAT-PARENT: handleChatStream cannot parent a chat turn's run on the app's span — fixed in code (lands with its MR)
A chat send or resume reached the executor through MinimalExecutor.execute() / resume() with no EntryTraceOptions.telemetryParent (#179 M4). So the run parented on the executor-level parent, else the active context, never on a span the app's route started for that request.
- Fix:
HandleChatStreamParams.telemetryParentis forwarded on every entryhandleChatStreamsends:execute()(a send, and path 4's fallback),resume()(caller_resumes,with_message), andsubmitToolResult(params, { telemetryParent }). On the Cloudflare DO, a submit starts thenew_runcontinuation. Without it every call keeps its shape. - agent-server's
POST /chatpasses its SERVER span to the chat handler. DOFrontendExecutor'sexecute/resume/submitToolResulttake the per-call parent over the client-level one, for the call's hops and its handle's.
FU-OTEL-CF-CHAT-HANDLER-TELEMETRY: createCloudflareChatHandler takes no telemetry for its DO clients — done (lands with its MR)
createCloudflareChatHandler built its DOFrontendExecutor / DOStateStoreClient / DOStreamManagerClient without telemetry, so their CLIENT hops always used the global provider. An app that registers one for its own spans (a reporting app) saw them in its HTTP metrics. Fixed: CloudflareChatHandlerOptions.telemetry, passed to all three clients (absent: the global provider, as before; { tracerProvider: new ProxyTracerProvider() }: none; a provider of its own: there). CloudflareHandleChatParams also declares telemetryParent now (FU-OTEL-AI-SDK-CHAT-PARENT forwarded it, but the factory's param type did not name it). Spans are not propagation: with spans off the request's parent still rides the traceparent of the executor's entry hops and their handles (the state store and stream manager reads carry the active context).
FU-OTEL-CF-CHAT-READS-REQUEST-PARENT: the chat handler's state store / stream manager reads take no request parent — open (LOW, #179 follow-up)
createCloudflareChatHandler's DOStateStoreClient and DOStreamManagerClient take a telemetryParent only at construction (DOClientOptions.telemetryParent), and the factory builds them once per handler. So their reads for a request (GET /snapshot, a reconnect's GET /sse, /chunks) parent on the active context: the root context unless the app registered a context manager. They don't join the request's trace the way the executor's entry hops do (DOFrontendExecutor takes the call's telemetryParent, FU-OTEL-AI-SDK-CHAT-PARENT). Fix: a per-call parent on the two clients' read paths (as DOFrontendExecutor.getStub(sessionId, parent) does), passed by handleChatStream / buildSnapshot from the request's telemetryParent.
FU-OTEL-TEMPORAL: runtime-temporal does not report to OpenTelemetry — open (MEDIUM, #179 phase 2)
#179 phase 1 instruments runtime-js and the Cloudflare DO (design D9: the durable runtimes follow). Temporal needs its run, step, model and tool spans across the workflow / activity boundary, the run's ids carried as serialized TraceIds (never an in-memory Context).
Propagation (part (a)) is fixed in code and lands with its MR. Each entry's telemetryParent is the workflow input's traceParent. The remote activity pins its transport to it (parentFromTraceIds + transportFor), and every child workflow inherits it.
When Temporal (or another runtime a test drives) builds core telemetry (createHelixTelemetry reads trace.getTracerProvider()), the tests that mock @opentelemetry/api module-wide must drop the mock or stub getTracerProvider: e2e/src/__tests__/tracing-temporal.integ.test.ts:339, and tracing-langfuse's client-tool-wait-span, langfuse-hooks, langfuse-hooks-extended, langfuse-hooks-strategy, start-signals, stateless-emitter, suspend-resume-hooks and token-tracking tests. Their mock replaces the API for core too (tracing-js.integ.test.ts failed that way in batch !600, fixed by keeping the API real).
FU-OTEL-DBOS: runtime-dbos does not report to OpenTelemetry — open (MEDIUM, #179 phase 2)
As FU-OTEL-TEMPORAL, for DBOS: spans across its durable steps, the run's ids carried as serialized TraceIds.
Propagation (part (a)) is fixed in code and lands with its MR. Each entry's telemetryParent is the workflow input's traceParent (standard mode), or, in persistent mode, the loop's start input (its first turn) and each inbox message's own traceParent (that turn only). The remote step pins its transport to it (parentFromTraceIds + transportFor); local sub-agent child workflows and persistent companions (spawn, re-consult, restart) inherit the parent run's.
FU-OTEL-CFW: CF Workflows does not report to OpenTelemetry — (b) fixed in code (lands with its MR); the executor's stop-cascade parent open
As FU-OTEL-TEMPORAL, for CF Workflows: spans across its step.do boundaries and instances, the run's ids carried as serialized TraceIds in the instance params.
Propagation (part (a)) is fixed in code and lands with its MR. Each entry's telemetryParent is recorded as the params' traceParent, and every remote hop is pinned to it (parentFromTraceIds + transportFor), the workflow's stop cascade included. CFW-local child instances inherit it. Left for (b): the executor's own stop cascade (a stop of a run no instance executes, e.g. a suspended one) has no parent to pin to, since the run's traceParent lives only in its instance params; a stop is not an entry, so it carries no telemetryParent of its own either.
Part (b) is fixed in code and lands with its MR (native spans, CF Workflows): live chat / execute_tool spans in their step.do units, containers (invoke_agent, helix.step, a delegated call's execute_tool) emitted once by their own emission steps with ids derived from the run and a segment generation, the run's link, and a child's traceParent set to its call's container. The executor's own stop cascade above still has no parent.
FU-OTEL-CFW-ATTEMPT-REAL-ENGINE: CF Workflows' helix.attempt is proven only from Miniflare 5's engine source — open (LOW, native spans)
Owner: W1. Filed by: W1, on the orchestrator's ruling (the CF Workflows native-spans MR).
helix.attempt on a CF Workflows chat / execute_tool span is the engine's step-context attempt, which guardLostClaims forwards to the step.do callback. Miniflare 5's engine (5.20260925.0-alpha, workers/workflows/binding.worker.js) calls the callback with { step, attempt, config }. Miniflare 4 passes nothing, and @cloudflare/workers-types 4.20260206 declares no argument. The unit tests drive a fake step in Miniflare 5's shape.
To do:
- One step through the real Miniflare 5 Workflows binding (the cfw lane's harness) asserting
helix.attempt= 1 on a run'schat. - A production-engine check before the docs claim production behaviour.
FU-FLAKE-LINT-CONFINEMENT-RPC-TIMEOUT: conformance:verify exits 1 on a vitest worker RPC timeout inside lint-confinement.test.ts — done (test only; lands with its MR)
Owner: W1. Filed by: W1, on the orchestrator's ask (second sighting).
- Seen:
- !555 pipeline 2997026259,
conformance:verifyjob 17524068574 (2026-10-28, 01:40Z): Test Files 85 passed (86), Tests 1904 passed (1912), Errors 2. - !563 pipeline 3000666349,
conformance:verifyjob 17551129507 (2026-10-28, 18:47Z): Test Files 87 passed (88), Tests 1911 passed (1919), Errors 2. The same job had passed on !563's previous head (pipeline 3000431750). - main 3dfd151570 (after batch 1), downstream test-matrix pipeline 3001163968,
conformance:verifyjob 17555044518: Test Files 87 passed (88), Tests 1911 passed (1919), the same twoonTaskUpdatetimeouts (retried by the orchestrator).
- !555 pipeline 2997026259,
- Signature: two unhandled
Error: [vitest-worker]: Timeout calling "onTaskUpdate", each "originated insrc/conformance/__tests__/lint-confinement.test.ts". That file never reports: its 8 tests are the gap in both runs. No assertion fails anywhere, and the job's second phase (the harness integ files) passes, but vitest exits 1 and so does the job. - Lead: the file builds an
ESLintovereslint.conformance.config.js, which lints type-aware (projectService: true). The firstlintTextmakes typescript-eslint build the e2e package's TypeScript program, and that work is synchronous, so the worker's event loop is blocked for its whole duration. A blocked worker cannot answer vitest 2.1.9's worker → main birpc calls, which have a fixed timeout. On a starved runner the program build outlasts it. This is the same family as FU-FLAKE-CF-VITEST-ONTASKUPDATE-RPC-TIMEOUT (a long single file) and FU-FLAKE-E2E-STRICT-STRUCTURED-D1-JS-MODULE-FETCH-RPC. - Measured (local, load 12-18): the first
lintTextwith the real config took 7.1 s and 12.1 s (the program build). Without type information it took about 1.25 s (plugin loading). Later calls take under 10 ms either way, and the restriction fires the same in both modes. The original file's first case still took 2.2 s with warm caches. - Fix (test only, no timeout raise): the cases lint with the real config minus type information:
- an
overrideConfigsetsprojectService: false/project: nulland turns off the config's two type-aware rules (no-floating-promises,no-misused-promises). The rule under test,no-restricted-imports, needs no types. - A new case proves that the override instance resolves the same restriction as the real config on every path a case lints (
calculateConfigForFile), including the exempt harness test. isPathIgnoredstill runs against the real config.- A type-aware rule added to the config and not turned off makes every case throw ("requires type information"), never pass silently.
- an
- Proof:
- the file passes ×3, at 244-312 ms of test time for the whole file;
- mutation: the override also turns the restriction off → 6 red, including the new case;
- mutation:
no-floating-promisesleft on → every case throws.
FU-FLAKE-PG-LIST-RUNS-EXPLAIN-SET-LOCAL: the Postgres V8 index EXPLAIN test's planner settings never applied — done (W1 MR3)
Owner: W1 MR3. Surfaced by: !340 pipeline 2936215040, job 17096150626 (do-nyc3): postgres-list-query.integ.test.ts › "the V8 composite index is available for the filter+sort path" › expected 'Limit (cost=…' to match /idx_.+_runs_session_status_turn/.
- Cause: the test ran
SET LOCAL enable_seqscan = offandSET LOCAL enable_bitmapscan = offas separate pooledadapter.querycalls outside a transaction.SET LOCALoutside a transaction does nothing (Postgres only warns), and the EXPLAIN could run on another pooled connection anyway. So the planner was free to pick a Seq Scan, and the assertion depended on table statistics. Red proof: aSHOW enable_seqscanafter the old calls readon. - Fix (test only): the two
SET LOCALs, aSHOW enable_seqscanand the EXPLAIN run in oneadapter.transaction, on one client. The test asserts the setting readsoff, so it can't pass vacuously. No otherSET LOCALoutside a transaction exists in the repo (git grep "SET LOCAL"; the migrator's runs inside one).
FU-FLAKE-CFW-LANE-MEASUREMENT-TIMEOUT: the cfw lane's measurement self-test ran out of its fixed 240 s budget on a loaded runner — done (test design + derived budgets)
Owner: the conformance owner. Closed by: this MR. Surfaced by: main pipeline 2927244229 (2e626e7f6a, post-!316), conformance:verify job 17036441143 on the shared Mac runner, which at the time also ran the !327 and !319 pipelines (retried once, disclosed by the orchestrator). The step was harness-tests:1: src/harness/cfw/__tests__/lane.integ.test.ts › "measurements: 20 back-to-back fresh execute() calls … create / run-start / settle-wait timings" › Test timed out in 240000ms, then BackendLane.teardown (conformance/suite.ts:371) threw cfw cleanup: binding fault(s) after the last cell: … status instance=h_…cfw-t0-early-4-…__turn__… : Attempted to use poisoned stub. Stubs to runtime objects must be re-created after calling Miniflare#setOptions() or Miniflare#dispose() (×3). It passed on !316's MR pipeline.
- Cause: one fixed budget for the sum of three load-proportional phases (a §F1 timing assumption). The test ran 20 fresh executes, then 5 two-turn sessions, then 5 store-gated two-turn sessions, about 40 serial executes under one 240 s timeout. Every call was still within its own 30 s
withinbound (the log: fresh execute p50 3.0 s, max 6.7 s; two-turn turn-2 execute p50 9.0 s, max 10.5 s). The first two phases used up most of the budget, and the timeout fired 25 s into the third, withearly-4(the last session) in flight. Nothing hung. - The teardown fault is a consequence, not a product fault. vitest abandons a timed-out test body but does not stop it. The body went on polling
early-4's turn instancestatus()(handle.result()), whileafterAll'scfw-miniflare.tscleanup()disposed Miniflare and then drained the binding faults. The faulting op is the harness binding'sstatus, and the message is Miniflare's own post-dispose stub error. No terminate or abort is involved (the lane's product-terminate and abort-op self-tests passed in the same run). - Fix (test only): the measurements are three tests, one per phase. Each budget is derived from its own work: its count of
within-bounded calls × 30 s, plus a 60 s margin.execute()is bounded too, now, not onlyresult(). Fresh executes drop from 20 to 10. The timings are logged as evidence and never gated (the looseexecute < 30 schecks are implied bywithin). AnonTestFinishedflag ends each loop once vitest is done with the test, so an abandoned body starts no new traffic into the teardown.
FU-FLAKE-DBOS-CHECKPOINT-TRUTH-COLD-IMPORT-TIMEOUT: two runtime-dbos unit tests timed out on a cold module import — done (test only)
Closed by: this MR. Surfaced by: main pipeline 2924780531 (e718a32798), test:unit job 17018320753 on the shared Mac runner (load ~33–35, a local e2e matrix overlapping; that job's collect phase took 599 s). Both tests hit Test timed out in 10000ms:
commit-checkpoint-truth-fixes-2.test.ts› "persistent route: never sends into a loop that already finalized › a session its loop already finalized starts a new loop…";commit-checkpoint-truth-fixes-3.test.ts› "R43 (c)… › the old loop took the message (its run exists under the old owner) → execute succeeds".Cause: the same pattern as FU-FLAKE-CORE-WORKSPACE-REGISTRY-EVICTION-RETRY-TIMEOUT. Each failing test was the FIRST in its file to run
await import('../lifecycle/execute.js')in its body, so that module graph's cold transform ran inside its 10 s timeout. Locally that import is the whole cost: 117 ms → 3 ms and 56 ms → 2 ms once hoisted. Neither file usesvi.mock/vi.doMock, so the dynamic imports sequenced nothing.Fix (test only): every dynamic import in both files is a static import (fixes-2:
executeImpl,classifyDbosWorkflowStatus,decideRoute,runExecuteCompanionTool; fixes-3:executeImpl), so the transforms run during collection.Left open, a separate risk (owner: DBOS bucket (orchestrator)): fixes-2 › "a delivered send (the loop is still active) never starts a second loop" takes 5 s locally (before and after), a real wait inside
executeImpl's persistent route, against the 10 s timeout. A runner about 2x slower would time it out. Fake timers or an injected wait bound would remove it; not done here.The class: a sweep of the remaining
await import(inside test bodies (31 files outsidevi.mockusers, some of them deliberate: optional or environment-gated dependencies, non-literal specifiers) follows as its own MR, classified file by file.
FU-FLAKE-STREAM-DO-SEQUENCE-ATOMICITY-BODY-REUSED: a concurrent-write StreamServer test reads a response body that GC cancelled — done: the two cases fixed by WE MR C1, the class swept to zero and guarded by the 12b(b) MR (owner: WE-2)
Owner: WE-2 (cf-we-do-streams-2), moved from CI-health-3 by the orchestrator, 2026-10-12.
Surfaced by: main's post-!324 pipeline 2926274748, test:integ:restjob 17028796145 (DigitalOcean k8s runner), 2026-10-08, in store-cloudflare stream-durable-object.integ.test.ts › "StreamServer Integration › persistence › should maintain sequence atomicity under concurrent writes". Seen again 2026-10-12 on !346's pipeline 2939923693, test:integ:rest job 17122046397 (do-nyc3), in the second copy of the case, stream-server-real.integ.test.ts ("Real StreamServer Integration › persistence with partyserver › …"). Both times TypeError: Body is unusable: Body has already been read (undici 7.29.0 lib/web/fetch/body.js:430 consumeBody, via _Response.json), and both §A3-retried.
- Mechanism (proven by WE-2, 2026-10-12): a harness defect, not the DO.
- Miniflare 5.20260925.0-alpha's Node-side
fetch(distindex.js:79466) re-wraps the undici fetch response's body in a newResponseand drops the original. - undici 7.29.0 registers every fetch
ResponseinstreamRegistry(lib/web/fetch/response.js:566). Its finalizer (lib/web/fetch/body.js:29-34) CANCELS the body stream of a garbage-collected response that nobody started reading. - The two cases held 20 unread responses across
Promise.all. A GC in that gap cancels the stream the test's re-wrappedResponseshares, and itsjson()throws. GC timing made it a flake, more likely on a memory-pressured runner. - Measured: one forced
gc()before the reads cancels 190 / 200 bodies; reading each body in its fetch's own.thennever fails under a GC every 1 ms (0 / 2 000). Every exit ofStreamServer.handleWritereturns a freshResponse.json(...), so the product side is not involved.
- Miniflare 5.20260925.0-alpha's Node-side
- Fixed (WE MR C1): both cases read each body as its fetch resolves (
readJsonNowinstore-cloudflare/src/__tests__/helpers/miniflare-setup.ts), and force a GC (collectGarbage()) before their checks. That GC is the deterministic red: with the old held pattern, both cases fail on every run. - The class sweep ran (WE-2, 2026-10-17, C1 lock window at
7ec3fb284b).MINIFLARE_ASSERT_BODIES_CONSUMED=trueover both Node-side Miniflare integ suites. Every test still passes; each held body is an unhandled error.- runtime-cloudflare integ: 871 reports, nearly all from shared request helpers:
helpers/do-state-store-proxy.ts:622(301, the state-store contract);do-stop.integ.test.ts:112(145),do-drive-pipeline.integ.test.ts:135(36),do-eviction-recovery.integ.test.ts:115(22),do-submit-tool-result.integ.test.ts:126(12),do-request-key.integ.test.ts:184(11),helpers/do-contract-miniflare.ts:98(9);- inline sites in about 15 more files (
do-rm29-transcript,do-multiturn-*,do-run-guards,do-hibernation-wake,client-tool-concurrent-single-do, …); 11 with no test frame.
- store-cloudflare integ: 751 reports.
- 371 were product code:
DurableObjectStreamManager.createWriter's/init. Fixed in C1, together with the success answers ofwrite/end/fail/pause/resume/reactivate/reset(thedoFetchaudit). Proven red first bydurable-stream-consumes-bodies.test.ts. - The rest are test sites:
stream-server-real.integ.test.ts:584(100) and:629(50), plusdurable-stream.integ.test.ts,durable-stream-realtime.integ.test.tsandstream-durable-object.integ.test.ts(3–10 each).
- 371 were product code:
- Done (the 12b(b) MR, WE-2, 2026-10-17).
- The rule is a check now: both Node-side integ setups set
MINIFLARE_ASSERT_BODIES_CONSUMED=true(runtime-cloudflarevitest-setup.ts, store-cloudflareinteg-setup.ts), so an answer nobody starts reading one task after it arrives fails the run instead of flaking under a GC. Red at the base:do-stop145,do-run-guards167. - Helpers: runtime-cloudflare
__tests__/helpers/read-now.ts(readJsonNow/readTextNow/consumeNow, a structural answer type) and store-cloudflareconsumeNow/readTextNownext toreadJsonNow. - Sites: about 400 across 23 files (warm-ups, seeds, bare and deferred
post()/start()/fetch(),clone()reads,Promise.allbatches, the state-store proxy'sclearAll, which now also checks each clear). The guarded sweep of every runtime-cloudflare and store-cloudflare integ file after the static pass: 167 held, in 13 files; after the fixes, 0, with every test green. (C1's unguarded sweep had counted 871 + 751 reports.) - Product, found by the sweep (red first, 5 cases):
do-clients.tsgetFromDO's 404 and the threeDOFrontendExecutor/snapshotreads dropped an answer unread; adiscardBody(as store-cloudflare's) reads them. - The
STREAM_DO_SCRIPTdouble's direct tests are deleted (stream-durable-object.integ.test.ts): it claimed parity with the real StreamServer but had drifted (no CP-129streamIdrequirement). Everything it covered is instream-server-real.integ.test.tsexceptPOST /reactivate, whose four cases moved onto the real DO. The double itself stays for the manager tests that use it on purpose (durable-stream*.integ, the contract harness). - Not swept: e2e's Node-side Miniflare harnesses (FU-E2E-NODE-MINIFLARE-BODY-SWEEP, below).
- The rule is a check now: both Node-side integ setups set
- runtime-cloudflare integ: 871 reports, nearly all from shared request helpers:
- Upstream: an issue for cloudflare/workers-sdk is drafted (Miniflare drops the undici
Responsewhose finalizer cancels the re-wrapped stream). It is not filed: that needs the user's go-ahead.
FU-E2E-NODE-MINIFLARE-BODY-SWEEP: sweep e2e's Node-side Miniflare harnesses for answers read late — open (owner: unassigned)
The 12b(b) MR swept and guarded runtime-cloudflare's and store-cloudflare's Node-side Miniflare integ suites (FU-FLAKE-STREAM-DO-SEQUENCE-ATOMICITY-BODY-REUSED, above). e2e also drives Miniflare from Node: the cross-service harness (src/cross-service/harness.ts) and the conformance cf-do host. Next: run those suites once with MINIFLARE_ASSERT_BODIES_CONSUMED=true, fix what it names with the same read-now rule, then set it in their setup.
FU-FLAKE-COLD-IMPORT-IN-TEST-BODY: unit tests awaited a cold module import inside their timeout — done for unit and workerd tests (integration files listed below)
Closed by: this MR, the class sweep after FU-FLAKE-CORE-WORKSPACE-REGISTRY-EVICTION-RETRY-TIMEOUT (!323) and FU-FLAKE-DBOS-CHECKPOINT-TRUTH-COLD-IMPORT-TIMEOUT. Owner: CI-health-3.
- The class: a test that runs
await import('…')in its body pays that module graph's cold transform inside its own timeout (10 s in the unit pools). On the loaded shared Mac runner the transform stretches (collect phases of 123 s and 599 s were seen), and the first test in a file to make the import times out. - Swept: every
await import(in a test body in a unit or workerd test file is now a top-level static import (merged into an existing import from the same module, no name collisions). Each file passed before and after; only import lines changed:- ai-sdk:
react/send-automatically-when.test.ts; - core:
hooks/hook-manager.test.ts,state/immer-state-tracker.test.ts,store/__tests__/load-history-prefix.test.ts; - runtime-cloudflare:
executor.test.ts,steps.test.ts,workspaces/__tests__/sandbox-provider.test.ts,workspaces/__tests__/filestore-provider.cf.test.ts; - runtime-dbos:
checkpoint-truth-16d.test.ts(fixes-2 / fixes-3 in FU-FLAKE-DBOS-CHECKPOINT-TRUTH-COLD-IMPORT-TIMEOUT); - runtime-js:
client-tool-resolver.test.ts,run-loop-persistent.test.ts; - runtime-temporal:
executor.test.ts; - workspace-local-bash and workspace-memory:
provider.test.ts.
- ai-sdk:
- Left as they are, deliberately: files that use
vi.mock/vi.doMock/vi.resetModules(14), where the import must follow the mock. Also:- imports gated on infrastructure or optional dependencies (e2e
cross-runtime-resume-matrix,retry-parity,multi-message-execute-dbos-persistent,multi-run-streaming,thinking-llm); - non-literal or data-URL imports (
reporter-config, runtime-cloudflarewrapper-module, skill-clicodegen); - lazily loaded per-lane harness setups (
setup-options); - the DBOS test-context
createRequireescape hatch; workspace-local-sandboxisolation-real(anode:builtin, no transform).
- imports gated on infrastructure or optional dependencies (e2e
- Still open (integration files, mechanical): memory-redis
redis-memory-store.integ, runtime-cloudflaresteps.integandworkflow-submit.integ, store-cloudflared1-collapsed-schema-parity.integ, e2eworkspaces/local-bash-js.integandrecovery-js.integ. These run in the amd64 integ jobs with 60 s timeouts and need infra to verify; hoist them in a later pass.
FU-FLAKE-CORE-WORKSPACE-REGISTRY-EVICTION-RETRY-TIMEOUT: a core registry test imported a large module inside its 10 s timeout — done (test only)
Closed by: this MR. Surfaced by: main pipeline 2924464071 (4d5329d0ee), test:unit job 17016291776 on the shared Mac runner (load ~25–29): @helix-agents/core src/workspace/__tests__/registry.test.ts › withEvictionRetry against the real registry — markEvicted transitions peekState and the retry routes through resolve() › Test timed out in 10000ms (1 of 202 files).
- Cause: the test has no wait. Its mocks resolve at once, no transient failure triggers the registry's backoff, and the close mocks return immediately. But its body began with
await import('../tool-injection.js')(about 1 300 lines plus its imports), so the module's cold transform ran inside the test's 10 s timeout. Locally that import was the whole test, 331 ms of it (every other test in the file takes a few ms). On a contended runner the transform stretches: the same job'sworkspace-posix-corecollect phase took 123 s. - Fix (test only):
injectWorkspaceToolsis a static import at the top of the file, so the transform runs during collection, outside any test's timeout. The redundant dynamicimport('../errors.js')is dropped (the file already importsWorkspaceEvictedErrorstatically). The test now takes 4 ms locally; the file passes 92/92. Not a fake-timers case: there is no timer in the path.
FU-FLAKE-D1-GUARDED-COMMIT-RETRY-EXHAUSTION-TIMEOUT: the D1 retry-exhaustion cases time out at 60 s on a loaded runner — done (test design + a derived budget)
Closed by: this MR (CI-health-3, handed over by CF-D2). Surfaced by: test:integ:rest (amd64 DigitalOcean runner) on main job 17007806558 (pipeline 2923287913), !310 job 17007855817 and !318 job 17013408039, each retry exhaustion: a run-start whose final fresh view is a live conflict throws the typed RunStartConflictError › Test timed out in 60000ms.
- Cause: the test's cost, not the store. Both retry-exhaustion cases lose all
MAX_COMMIT_ATTEMPTS(25) guarded commits, about 250 D1 statements, and through Miniflare's Node-side D1 proxy every statement is several Node↔workerd round trips,prepareandbindamong them, synchronously. Measured on an idle host: 6–10 s per case (the D1 calls about 1.6 s, synchronousprepare/bindabout 2.9 s, the rest the gaps between statements, which stretch with CPU load; the commit backoff is at most about 1.35 s: base 4 ms, cap 64 ms, 24 waits). A runner about 10x slower crosses the 60 s timeout. The same scenario inside workerd (the real workerd-bound D1, in-process) takes about 0.9 s, most of it the backoff. - CP-96 is not the cause. CF-D2's hypothesis was that CP-96 (!295,
shouldRetryRunStart) lengthened the loop. Before CP-96 (4c3cc736b9^) the case made 238 statements in about 5.9 s; after, 265 in about 6.4 s (one more session-status read per attempt): +11% statements, +8% time, with the loop and the backoff unchanged. - Fix (test only, no timeout change): the two retry-exhaustion cases moved to
src/__tests__/d1-guarded-commit-exhaustion.cf.test.ts, the real-workerd pool (test:cloudflare), with the same assertions: the outcome after N lost attempts, never a statement count, on a D1 database of its own (GUARDED_COMMIT_DBinwrangler.cf.toml: the pool runs files in parallel over shared storage, and the state-store contract suite clears every table ofAGENT_DBbetween its cases, which a first run on the shared binding hit). The interleave and run-start helpers moved tosrc/__tests__/helpers/d1-commit-interleave.ts, shared by both runners (CF-D2's MR2 builds on them). - Red/green, with injected latency (no load generators): +60 ms on every Node-proxy call (
prepare,bindand each execute; about 10x the measured 6.5 ms per call, the stretch CI shows) makes the Node-proxy case time out at 60 s, as in CI (control at +0 ms: 5.6 s and 7.8 s). The workerd cases with +60 ms on every D1 execute pass in 17.7 s at worst. Unloaded, the workerd cases take 0.6–0.7 s each (3 runs). - Reopened after the move (!320), closed by this MR. On main's post-merge pipeline 2924377152 (e98b11a52e),
test:cloudflarejob 17015659686 on the shared Mac runner, both moved cases timed out at 60 s (the file took 120 s) at host load ~51 (a local matrix running beside CI). In !320's own pipeline they had taken 22.6 s and 25.8 s at load ~38, against about 0.7 s locally. The move removed the Node-proxy cost, but in workerd the cases are still CPU-bound (about 10 D1 statements per lost attempt), and the 25 attempts cannot be lowered without a product seam (MAX_COMMIT_ATTEMPTSis read inside the store's loops). So each case now runs under a timeout derived from the work:MAX_COMMIT_ATTEMPTS x 6 s + 30 s(180 s). The 6 s per attempt is 2.5x the worst observed lower bound (over 2.4 s per attempt at load ~51), and the arithmetic is in the test. A recurrence past 180 s would mean the runner was more than 2.5x more loaded than that, which is the CI-runner contention's territory (!314 and after), not the store. - The retry of that job (17015842217) also failed
state-store-contract.cf.test.tsat file level:Timeout calling "resolveId"ford1-migrations.ts, vitest's RPC timing out during a 32.7 s module collection while the exhaustion case ran beside it on a starved host. That's module resolution, not D1 storage or the bindings. The store-cloudflare workerd pool now runs its files one at a time (singleWorker: trueinvitest.cf.config.ts; thefileParallelism: falsefirst set here was ignored by the pool withisolatedStorage: false, FU-FLAKE-D1-USAGE-STORE-CONTRACT-ROW-COUNT), so the CPU-bound exhaustion file no longer overlaps another file's collection; locally the pool takes about 8 s sequential.
FU-FLAKE-D1-SESSION-ID-ISOLATION-TIMEOUT: a D1 session-id isolation case times out at 60 s on a loaded runner — open (test only)
Surfaced by: !929's MR pipeline 3085692378, test:integ:rest job 18156791048. Lead: load, not the product.
- Signature: store-cloudflare
state-store-contract.integ.test.ts, D1StateStore, Session id isolation (MA-34):session-id-isolation.deletefailedTest timed out in 60000msat 60 035 ms (the integ config'stestTimeout). store-cloudflare readTest Files 1 failed | 30 passed (31)andTests 1 failed | 1183 passed | 2 skipped (1186); no assertion failed. - The same product passed: both pipelines merged the MR onto the same target commit. The earlier one, 3085375830 (
test:integ:restjob 18154337342, the same runner), ran head adc40193f0; the later head, 6664943c90, changes only the e2estale-entry.tsscenario. There the three isolation cases took 15 582 ms (.writes), 16 236 ms (.delete) and 11 997 ms (.cleanup-staging); here 48 676, 60 035 (timed out) and 45 727 ms. - The whole run was slower: store-cloudflare's integ Duration was 881.78 s there and 2328.37 s here, and the job took 1397.3 s and 2590.7 s. It overlapped
test:cloudflarejob 18156791062, which failed on the load timeout recorded under FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT. - The MR's D1 change does not reach the case: CP-333's run-start SETs (the session status and the error fields) apply only when the run-start carries an entry state, and the case's seed (core
seedMessages,startSeedRun) passes none. - The retry passed:
test:integ:restjob 18160012906 (pipeline 3085692378) took 983.1 s:.writes13 068 ms,.delete13 643 ms,.cleanup-staging10 073 ms; store-cloudflareTest Files 31 passed (31),Tests 1184 passed | 2 skipped (1186), Duration 851.16 s. - Lead (test only): the case takes about 16 s unloaded against a 60 s bound, so a runner about 3.7x slower times it out. FU-FLAKE-D1-CAS-RACE-TIMEOUT is the same shape in another D1 integ file (2026-11-19). Both show that D1 integ cases under the fixed 60 s
testTimeoutfail on a loaded runner. So the fix direction is a derived budget across the D1 contract integ, as FU-FLAKE-D1-GUARDED-COMMIT-RETRY-EXHAUSTION-TIMEOUT gave its cases, not a per-case raise. A smaller seed stays a candidate for this case. No fix is chosen yet.
FU-FLAKE-D1-CAS-RACE-TIMEOUT: the D1 concurrent CAS-race cases time out at 60 s on a loaded runner — open (test only)
Surfaced by: !931's MR pipeline 3087168720, test:integ:rest job 18168293060. Lead: load, not the product. Related: FU-FLAKE-D1-SESSION-ID-ISOLATION-TIMEOUT (the same shape: a D1 integ case past the integ config's fixed 60 s testTimeout on a loaded runner), FU-FLAKE-D1-CAS-RACE-LOSER-D1STATEERROR (this file's earlier, fixed, wrong-error-class flake).
- Signature: store-cloudflare
atomic-suspend-write.integ.test.ts, D1StateStore.saveStateAndPromoteStaging, "concurrent saveStateAndPromoteStaging — CAS race" (20 concurrent calls): "exactly one wins when 20 concurrent calls share the same expectedVersion" took 61 064 ms and the "all losers throw StaleStateError …" case 64 543 ms, bothTest timed out in 60000ms. store-cloudflare readTest Files 1 failed | 30 passed (31)andTests 2 failed | 1186 passed | 4 skipped (1192); no assertion failed. - Normally: the same two cases took 8 293 / 7 047 ms in
test:integ:restjob 18154337342 and 8 266 / 7 697 ms in job 18160012906, both passing. - The whole run was slower: store-cloudflare's integ Duration was 1633.88 s, against 881.78 s and 851.16 s in those two jobs, and the MA-34 isolation cases took 29.5 / 37.2 / 24.8 s against 10-16 s. The job took 2118.2 s. !931 is records only, so the cases ran main's code.
- The retry passed the CAS cases:
test:integ:restjob 18170344706 (pipeline 3087168720): 9 825 ms and 8 984 ms, store-cloudflareTest Files 31 passed (31),Tests 1188 passed | 4 skipped (1192), Duration 634.05 s. That job failed elsewhere, on FU-JS-CP315-W1-PAUSE-WINDOW-FLAKE. - Lead (test only): the cases take about 8 s unloaded against a 60 s bound, so a runner about 7x slower times them out. With FU-FLAKE-D1-SESSION-ID-ISOLATION-TIMEOUT this is the second D1 integ family to fail only on the fixed bound; the fix direction is in that entry.
FU-FLAKE-LANGFUSE-EVICTION-TEST-STARVES: tracing-langfuse stateless-emitter.test.ts fails 4 tests on a 0.5-CPU runner — done (test bug)
Closed by: this MR (CI-health-3 flakes-6). Surfaced by: the CI runner contract migration's proving runs on the new DOKS runner (do-nyc3, 0.5 CPU requested, limit 6): test:unit jobs 16985800322 and 16986057377, three runs in a row. Introduced by !309's audit round (the eviction-at-cap test).
- Signature:
emitRootSpan > keeps the identity of the last MAX_CLOSED_RUNS closed runs only (oldest evicted):Test timed out in 10000ms; thenshould emit root with ERROR level on failure(nolevel: 'ERROR'),should include name and output in trace update even without pending data(agent-785vsunknown-agent) anddistributed scenario > all spans share the same traceId(traceId mismatch). - Cause (a test bug, not the emitter): the eviction test registered and emitted 1 001 root spans to exercise the real 1 000-entry cap: about 100 ms on a developer machine, over 10 s on the starved runner. A timed-out test's async loop keeps running, so it kept registering runs and emitting spans into the file's shared mock span array, and the next three tests read them. The current runners have more CPU, so main's pipelines stayed green.
- Forced repro: the old file with
--testTimeout=40fails exactly those four tests (the leak shows asagent-516). - Fix: the emitter takes an internal
maxClosedRunsconstructor option (defaultMAX_CLOSED_RUNS, 1 000); the test exercises eviction with a cap of 3 (4 emissions) and pins theMAX_CLOSED_RUNSconstant (1 000) the default reads. With--testTimeout=40the file passes 19 of 19; the package passes 210 of 210. No timeout raise.
FU-FLAKE-REDIS-GUARDED-COMMIT-RETRY-EXHAUSTION: store-redis retry-exhaustion test's commit resolved instead of exhausting — done (test race)
Closed by: this MR (CI-health-3 flakes-6). Surfaced by: main pipeline 2919039735 (090a62cd73), test:integ:rest job 16978170218, first sighting.
- Signature:
redis-guarded-commit.integ.test.ts› "retry exhaustion: a view that moves before every script throws RedisCommitContentionError (a StaleStateError)":must reject with {"name":"StaleStateError"}: expected undefined to be defined. The run-start commit went through. - Verdict: a test race, not a product bug. The test moves the commit's view before every
COMMIT_SCRIPTby renewing r1's lease with a random ttl (30_000 + random(0..999)ms). The renew script setsleaseUntil = store now (ms) + ttl, and the commit script re-checks each live run'sstatus/leaseUntil/ownerTokenstrings. When two consecutive renewals land on the sameleaseUntil(the ttl difference cancels the elapsed milliseconds, about 1 in 1000 per attempt; attempts 2–5 can collide), the view did not move, and the guard correctly lets the commit through on that attempt. - Forced interleave (scratch): attempts 1–2 renew randomly, attempt 3 re-renews r1 to exactly its current
leaseUntil(…7790, …7790). The commit lands on attempt 3, the CI signature. A guard that passes an unchanged view is correct, so no integrity bug. - Not CP-96: !295 made the script's session-status re-check unconditional; a lease renewal never changes the session status, and the test's lines are unchanged since S5.
- Fix (test only): the ttl grows by 1 ms per renewal (
30_000 + n); the store clock never goes back, so eachleaseUntilis strictly later than the last and the view always moves. The test now also asserts that every injected renewal moved it. The file passed 10 runs out of 10 (36/36 each). At about 0.4% per run, ten passes cannot prove the fix; the proof is by construction (a strictly increasingleaseUntil) plus the in-test assertion. - D1 sibling (FU-FLAKE-D1-GUARDED-COMMIT-RETRY-EXHAUSTION-TIMEOUT, above): a different cause. Its injection is a
saveStateversion bump, which always moves the view, and its signature is a 60 s timeout, not a commit that went through.
FU-FLAKE-CFW-LANE-ABORT-OP-TERMINATE-RACE: the cfw lane's abort checks assumed the product's terminate wins — fixed
Seen: train pipeline 2963518906, job 17286942229 (2026-10-19), which ejected !442 (CP-45) from the train: harness/cfw/__tests__/lane.integ.test.ts's "abort op" case asserted terminatedBy === 'product', but the instance's ledger entry read closedStatus: "complete" with no terminatedBy. Then !444's CI, job 17291098628 (2026-10-19): the conformance cell control.abort-terminates-owner|cfw-workflows-d1|executor failed .owner-terminated (the owner instance read complete, the cell expected terminated). Then !447's own merged-result pipeline 2964584703, job 17295008848 (2026-10-19): the self-test's "product terminate" case (a held model call, handle.abort()) read no terminatedBy, the same race.
- Cause (W1's trace):
abortRunrecords the abort, then wakes the held body (runtime-cloudflare/src/executor.tsabortRun:stopRunById, thenterminateBestEffort). Since CP-16 the body aborts its in-flight model call at once and ends itself with its own abort terminal, often before the executor's terminate lands, and the engine then refuses the terminate. Both endings are the product ending the run: a race in the checks, not in the product. - It also cost CP-15 its proof. CP-15's recorded red mutation (the terminate throws, as the pre-fix
instance.abort()did) produced the samecompletereading, so with CP-16 in place.owner-terminatedcould not tell CP-15's defect from the product working: an owner left running is no longer reachable (the body ends itself). - Fix: the lane self-tests force each order instead of accepting a race (no race in a self-test): the product-terminate case and the "terminate wins" abort-op case run in a tool that ignores its abort signal, so the body cannot end itself and the terminate always lands (
terminatedBy: 'product', the session fencedproduct-terminated); the "body ends first" abort-op case holds the product's terminate until the instance has closed on its own (ObservedWorkflowBinding.terminateAfterSelfClose, a self-test knob), so the instance closescompletewith the run's typedframework_cancelledand nothing fences it (a closed instance sends no late traffic). Never an owner left open or reaped. The conformance cell accepts both endings (its verdict is the same either way): it takes the owner closedterminatedby the product orcompleteby its own abort terminal, and.persisted-cancelled(failed withframework_cancelled) still excludes a normal completion. CP-15 is now told apart by.abort-call-succeeds, re-proven with a mutation restoring its pre-fix rule (the handle's abort throws after the session is persisted failed). Proven on cfw-workflows-d1 (owner slots s57 / s57b, 2026-10-19): the new cell 5/5 green (main's old cell 5/5 in the same build: the race did not show; the two CI ejects are its evidence) and the CP-15 mutation red 2/2 on.abort-call-succeedsalone. The s57 run of the self-test's product-terminate case (3/3, still on a held model call) did not show the race either; !447's pipeline did. The forced orders (owner slot s60, 2026-10-19, tree 9b801966b9): the product-terminate case and both abort-op cases 5/5 each, the knob's unit file 38/38.
FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD: runtime-cloudflare workerd integ tests fail with fetch failed on Miniflare 5 — open
Owner: the conformance owner (from 2026-11-05; was CI-health-3). Surfaced by: !307 pipeline 2919324894, test:integ:rest job 16980311867 (amd64 k8s, 902 s). Related: FU-FLAKE-CF-D1-PROXY-FETCH-FAILED (closed by !272: Miniflare 3's Node proxy dropped loopback connections under concurrent load; 0 of 13,440 calls failed on Miniflare 5), FU-CI-K8S-QUOTA-SINGLE-PIPELINE-SATURATES.
Signature: three runtime-cloudflare integ tests failed with
TypeError: fetch failedfrom the Miniflare binding proxy (D1StateError: D1 state saveState failed … TypeError - fetch failed):client-tool-workflow.integ.test.ts› "multiple deferred finishWith calls [c1, finish_a, finish_b]: finish_a wins, finish_b is never executed and superseded" (54.7 s);do-multiturn-client-tool-resume.integ.test.ts› "submit + racing explicit /resume (helixSAW follow-up): no corruption, no happy-path resync" (48.3 s);concurrent-subagent-fold.integ.test.ts› "composes all 16 parallel afterSubAgent folds (append)" (79.4 s), whose mockstep.doretried 16loadState … fetch failedat once.
One window across three files: 21
fetch failedlines, all at 20:03:59 (1), 20:04:03 (17) and 20:07:59 (3), so a burst of dropped connections, not three test bugs. The packages are on Miniflare5.20260925.0-alpha, so this is a recurrence after !272's fix. The job's pod first waited onInsufficient memory(k8s scheduling notes at 19:57), so memory or CPU pressure on the runner is the lead.Recurred as a timeout (!310 pipeline 2923294110,
test:integ:restretry job 17008701771, DigitalOcean runner, 2026-10-07):runtime-cloudflareconcurrent-subagent-fold.integ.test.ts› "sub-agent wave D1 cost … recording a child's result costs the same D1 calls…" timed out at its 240 s cap after 13fetch failedstep retries (the first: the parent's…-llm-0step,D1 state getSubSessionRefs failed … TypeError - fetch failed, attempt 1). The same file passed 7/7 (418 s) in the first run of the same commit. A forced-interleave test (that exact failure on attempt 1, then the retry) completes on main and with !310's per-attempt stale guard alike, so the guard is cleared; the timeout is the retries' cumulative cost in a test that runs four full workflows near its cap.Seen in the conformance cfw lane on !422 (2026-10-18). Both are harness crashes (the cell's observation is discarded), never verdicts, but only the first is this follow-up's open question:
- pipeline 2960816844,
test:conformance 4/5job 17268393859:stream.paused-consumer-every-chunk|cfw-workflows-d1|executor(a 10,500-word reply) crashed at its cell end (endCell(epoch 80):cfw: binding fault (status instance=h_…__turn__57c28f994398 …): fetch failed). The same cell passed 3 of 3 in owner slots; the orchestrator retried the job. - train pipeline 2961125323,
test:conformance 4/5job 17270305546, and pipeline 2961372980 (job 17271731951):state.oversized-reply-fails-typed|cfw-workflows-d1|executorcrashed the same way (fetch failed, causeother side closed). Correction (orchestrator, 2026-10-18): this one is not load. It is that cell's own crash mode on today's product: its 17.6 MB reply sometimes reaches the cell's typed check and sometimes kills the transport (on the CF DO, the same pipeline's job 17271731948 loggedSQLITE_TOOBIGand the cell's next store read failed). The cell is unregistered until MA-31's fix makes the failure typed and deterministic (MA-31, DI-60). - Keep-alive is backed out (FU-LOCAL-INTEG-EPHEMERAL-PORT-EXHAUSTION): every Node → workerd call has its own connection, so the first sighting is not the reused-socket race either.
- A third sighting of the paused-consumer cell, on !422's pipeline 2961486608 (job 17272345391,
test:conformance 4/5): the samebinding fault (status instance=…): fetch failedat its cell end, after the cell had run for 58 s. - Local A/B (owner slot s46 part 1, 2026-10-19, !422 590689f2d6 merged with main a83323af26): not reproduced. The cell alone passed 5 of 5 in about 2.7 s each. The full cfw lane, in CI's order (after !422's heavy cells), passed 135 of 135 in 15.7 min, the cell in 2.5 s. So CI ran the same cell about 20 times slower before the binding call died.
- workerd memory, sampled every 10 s across that lane: one workerd process grew from 360 MB to 1.0 GB RSS over 8 min (01:04:52 to 01:12:54), then a fresh one took over (238 → 419 MB, then 474 → 767 MB). In total, workerd held 0.7 to 1.1 GB for most of the lane. Together with the 20× slowdown, this points at the CI runner's memory or CPU limit rather than the cell (the lead from !307's pod, which waited on
Insufficient memory). It is not established: nothing here reads the CI pod's limit or its OOM events. - The orchestrator retried the shard as an informed retry (job 17273537132). It passed once, then the cell crashed the same way on train pipeline 2961761793 (job 17273937439, after 34 s). Unregistered (orchestrator ruling, 2026-10-19), as the oversized-reply cell was; CP-137 is open again until it is re-registered.
- A second worker in the lane (owner slot s50, 2026-10-19, trial on main 397a2a5fa0): with the
remote-childproducer worker beside the cfw worker in the same Miniflare, workerd held a median 1.14 GB in total over the full cfw lane (123 samples, 10 s apart), at most 1.37 GB, its largest single process 935 MB: about 0.1 to 0.3 GB above the one-worker lane above. Kept as one Miniflare (orchestrator ruling: not material); if the lane's CI binding faults rise onceremote-childlands, split the producer into its own Miniflare for the cells that need it.
- pipeline 2960816844,
To do: the lane restart and the re-registered cell are done (!693, CP-137). Still to do: reproduce under memory and CPU pressure locally (the !272 transport A/B harness) and find whether the drop is in workerd, the proxy or undici.
Recurred with the lane rebuild in place (2026-11-05, owner).
stream.paused-consumer-every-chunk|cfw-workflows-d1crashed atendCellon !712's train pipeline 3031996667,test:conformance 7/7, twice: jobs 17771786644 (56.9 s) and 17773061477 (44.0 s). Each logged the rebuild before the cell, so the fault comes up on a fresh Miniflare: the rebuild is disproved as a cure. Nothing about the pipeline explains it: 7/7 passed on the G1, !708 and !713 trains (48 to 71 s), and only a version bump and records landed between. Passing at 71 s and failing at 44 s rules out duration.The cause was never in the log.
ObservedWorkflowBinding.observekept onlyerr.message, so every CFW sighting readfetch failedwith undici'scausegone. It now records the cause chain (describeTransportError), and astatusread re-sends up toSTATUS_READ_ATTEMPTSwhenisWorkerSocketClosedholds, logging each re-send, as the cf-do reads have since !486. Any other cause is one attempt and a fault, so a dead workerd is never retried away. That the sightings are this socket race is inferred until one logs its cause. Still open: why the socket closes, read from the next logged cause.
FU-CLIENT-EOF-WITHOUT-FINISH: a UI stream that ends with no finish chunk reads as ended — open (LOW)
Surfaced by: an app under a local wrangler restart (2026-11-05); the cause is inferred. The browser's UI stream ends cleanly with no finish chunk, nothing in packages/ai-sdk/src/react or client looks for one, so no typed error reaches the re-anchor and the client does not re-attach. This is one layer above stream-wire's rule (Helix's own reader treats an EOF with no terminal frame as a drop) and beside CP-25, whose open residue is the re-attach after a typed drop or an eviction. Becomes a finding if seen without a restart, or on a deployed worker.
FU-CI-NEXTJS-EXAMPLE-JOBS-NEVER-RUN: the nextjs-redis and resumable-streams-nextjs example jobs never run in CI — open: (c) done, (b) open (owner: CI-health-3)
Surfaced by: CF-D2's triage, verified by CI-health-3 (2026-10-07). User ruling (2026-10-07): (c) now, then (b).
- What happens:
test:examples:nextjs-redisandtest:examples:resumable-streams-nextjs(.gitlab-ci.yml) run onlyif: $OPENAI_API_KEY && …, and declare noenvironment:. The project variableOPENAI_API_KEYis scoped to thee2e-testsenvironment (masked, not protected), which only the fourtest:examples:*Cloudflare jobs declare. So the rule never matches and the jobs are never created. Neither appears in any of the last 8 main pipelines (2919725930 … 2924191318); the other four example jobs appear in all of them. The key's scoping dates from 241d774fd6 (2026-02-03) and these jobs' key gate from d7fab7fb2d (2026-05-16), so they have most likely not run since. No other job gates its rules on a variable. - Why declaring the environment alone is not enough: GitLab warns against environment-scoped variables in
rules:, which are evaluated at pipeline creation, when such a variable may not be defined (gitlab-org/gitlab#587006). - LLM spend: both suites hit the REAL OpenAI API (no mock): nextjs-redis has 7 specs / 32 tests on gpt-4o, gpt-5.2 (Responses API) and gpt-4o-mini; resumable-streams-nextjs has 1 test on gpt-5.2 (CI retries: 2). Unmeasured estimate: about $2–5 per nextjs-redis run.
- Plan (the ruling):
- (c), now: make both jobs
when: manual, declareenvironment: {name: e2e-tests, action: access}, drop$OPENAI_API_KEYfromrules:, and fail fast in the script when the key is empty. It's a small.gitlab-ci.ymlchange, landing after !314 (the CI-runner MR). Done by the provider-config MR (WM, 2026-10-19): both jobs and the newtest:live-llmjob extend the.live_llmtemplate (a manual, non-blocking button on same-repo MRs and nothing else, user ruling 2026-10-19;environment: e2e-tests, abefore_scriptthat fails namingOPENAI_API_KEYwhen it is empty). The four Cloudflare replay jobs no longer declare the environment: they never needed a key. - (b), then: port both suites to the VCR OpenAI mock the Cloudflare examples use (
test-utils/openai-mock-server.ts, replay fixtures, the CC57 discipline indocs/dev/example-app-testing.md), so they run on every pipeline with no spend. Recording the fixtures is a one-time spend: measure or estimate it first, and the user approves it explicitly. Queued behind CI-health-3's connection-drop and CP-102 MRs.
- (c), now: make both jobs
FU-LIVE-BRIEF-FLOW-MARKER-ASSERTION: nextjs-redis brief-flow asserts the live model's wording — open (test bug; owner: parked (manual live-job only))
Surfaced by: !468's live test:examples:nextjs-redis click (job 17306666085, 2026-10-20). brief-flow.spec.ts › "deterministic interleaved pattern appears correctly @slow" failed on both attempts (Missing marker [B1], then [A1]). The saved page snapshots show the model did not write the markers: on the first attempt it wrote [A1]…[A10], then prose (not [B1]…[B10]) between research_topic and generate_brief, then [C1]…[C10]; on the retry it wrote no A markers and the same kind of prose. Every text segment the model produced rendered, in order, between the tool parts, so the SDK dropped nothing: the assertion checks the model's compliance with the prompt, the "tests LLM output" anti-pattern in docs/dev/example-app-testing.md. The serial describe then skipped the file's other tests. To close: assert what the SDK owns (the assistant message holds a text segment before, between and after the two tool parts, in that order, and the same segments survive a reload), not the marker strings; or move the marker check onto replay fixtures once the suite is on the VCR mock (FU-CI-NEXTJS-EXAMPLE-JOBS-NEVER-RUN plan (b)). Not in !468.
FU-REDIS-STATE-TEST-HOST-CLOCK: a Redis store test compares the store's clock with the host's — open (test bug; owner: reliability orchestrator (route))
Surfaced by: WE-2's local DoD matrix for !351 (2026-10-14). redis-state.integ.test.ts › "Staging Operations — advanced promoteStaging › should update checkpointId and checkpointedAt on session" failed with expected 1791951054257 to be greater than or equal to 1791951054260, again when the file ran alone (… 1791958368168 … 1791958368169), and passed in a later whole-repo run: it is intermittent. !351 does not touch store-redis; the test is the same on main. Also surfaced by DO companion MR B's Task 12 sweep (2026-10-16, cited there as FU-REDIS-CHECKPOINTEDAT-CLOCK-SOURCE — the same item, not a second one): the same case failed once with expected 1792190787375 to be greater than or equal to 1792190787376, then passed 3 of 3. MR B does not touch checkpointedAt. The mechanism below is confirmed in code: the store's commit script takes now from Redis's own clock (redis.call('TIME') → nowMs, commit-script.ts), and commitCheckpoint stamps checkpointedAt with it (setMain(main, dirty, 'checkpointedAt', now)), while the test's lower bound is the host's Date.now(): two clocks, so the stamp can read 1 ms behind a host reading taken before the write.
- Mechanism: the commit script stamps
checkpointedAtfrom the store clock,redis.call('TIME')(packages/store-redis/src/commit-script.ts:145), and the test compares it with the host'sDate.now()read just before the commit (packages/store-redis/src/__tests__/redis-state.integ.test.ts:501,:515). The two clocks are different machines' clocks when Redis runs in a container: on the local Docker VM,TIMEread over one connection was 1 ms behind aDate.now()taken before the call. Any lag fails the>=. - Scope: the only tight host-vs-store-clock assertion in the store-redis tests (the lease check at
redis-guarded-commit.integ.test.ts:308has a 20 s margin). - Fix direction: read the lower bound from the same Redis server (
TIMEbefore the commit), or assert a bounded window around the host time. Never compare a store-clock stamp with an unbounded hostDate.now().
FU-CONF-DBOS-COMPANION-STAYS-ACTIVE-AFTER-FINISH: a DBOS non-blocking companion released after its parent's turn never completes — open (PARKED: DBOS)
Seen: owner slot s47 (2026-10-19, owner batch b4c3975207 on main 6ef75cb6c9), 3 of 3, in companion.oversized-completion-notice-delivered|dbos-postgres|executor. The parent spawns a non-blocking companion and completes its turn while the companion is held in its gated tool. Released, the companion calls __finish__ with an output of 512,000 double quotes (~1 MB stored). It never reaches a terminal status in 60 s: its session stays active with 5 messages and no errorDetail, and the run logs no error. The same cell completes on the js lanes and CF Workflows (the earlier owner slot also saw it with 480,000 quotes).
- Not the output's size (owner slot s48, 2026-10-19, trial 076bc0d8a9 = batch + main a971537989; a diagnostic run on an uncommitted edit, not ledger evidence): with the cell's store-row-cap need and DBOS gap dropped, the real size failed the same way once, and an output of 1,000 quotes failed the same way 3 of 3 (
active, 5 messages, noerrorDetail, 60 s). - Not established: whether the non-blocking companion finishing after its parent's turn ended is the cause, or something in how the DBOS lane releases a tool gate. Next step when DBOS is unparked: the same companion released BEFORE its parent's turn ends, as the contrast.
- Meanwhile: the cell runs only on lanes that cap a row (
store-row-cap), its subject; dbos-postgres is a declared gap that names this follow-up.
FU-FLAKE-DBOS-E2E-SHARED-FORK-POOL-TEARDOWN: a DBOS e2e integ file fails after another file's pool teardown in the shared fork — open (PARKED: DBOS)
Seen: !442's pipeline (2026-10-19, job 17284637277; the orchestrator retried it once). packages/e2e/src/__tests__/interrupt-resume-dbos-persistent.integ.test.ts failed with "session not interrupted/failed after 15000ms" right after Cannot use a pool after calling end on the pool.
- Not this MR, by A/B: W1 ran the file alone 5 of 5 on CP-45 and 5 of 5 on main.
- Likely mechanism (not established):
packages/e2e/vitest.dbos.integ.config.ts:50-54runs every DBOS file in ONE fork withisolate: false(singleFork: true), so module state such as a Postgres pool outlives its file. One file's teardown that ends a pool another file still uses fails the second file's DBOS work. - Next step when DBOS is unparked: find which file ends the shared pool (or what holds one across files), and give each file its own pool or its own teardown scope; then rerun the shard in its CI order.
FU-FLAKE-DBOS-PERSISTENT-COMPANION-TERMINAL-ERROR-10S-POLL: a DBOS persistent-companion integ file waits on a 10 s wall-clock poll — open (PARKED: DBOS)
Also (2026-10-19): packages/e2e/src/__tests__/interrupt-resume-dbos-persistent.integ.test.ts › "interrupt then second execute delivers message and completes" failed in CI (!457's pipeline 2965213058, test:integ:e2e 1/4) with waitForStoppedSession: session not interrupted/failed after 15000ms: the same mechanism, a fixed wall-clock poll of the session status. The test runs scripted model answers, and !457 changed no runtime-dbos behaviour (two doc comments), so the shape is the file's. Fix it the same way when DBOS is unparked: wait on the run's terminal record.
Surfaced by: the WA controller's local DBOS integ run for !364 (2026-10-15; CI runs no DBOS integ tests, its RUNTIME_DBOS_INTEG gate is unset there). In a full RUNTIME_DBOS_INTEG=1 pnpm --filter @helix-agents/runtime-dbos run test:integration at host load ~94, packages/runtime-dbos/src/__tests__/integration/persistent-companion-client-tool.integ.test.ts › "persistent child terminal LLM error: failure is silently swallowed by recv-loop finalizeLoop (regression guard)" failed with waitForCondition timed out after 10000ms waiting for: session sess-pccl-… status in [failed, completed, interrupted].
Not this MR: the file alone at load < 14 passed 2/2 three times on the !364 head (
832797b787), and once on main (dab67a452b) the file's OTHER test ("… submit at root routes through to child workflow") failed with the samewaitForCondition timed out after 10000msafter 13.4 s. The shape is the file's, on main.Mechanism: both tests poll the session status with a fixed 10 s wall-clock budget while a DBOS workflow (and its child) run in the same process; under CPU contention the run finishes after the budget.
Fix direction (when DBOS is unparked): wait on an observed event (the run's terminal record or the workflow's completion) rather than a fixed wall-clock budget, or derive the budget from the measured cost as the other FU-FLAKE entries did.
Recurrence at LOW load (WE-2's RM-86 gate, 2026-10-19): it is not only CPU contention. In the RM-86 gate the file failed once in the full DBOS integ run, then 1 of 3 alone. An A/B with the file alone at 1-min load ~4.5, alternating main (
cfed7b8ffa) and RM-86, 5 runs each: main failed 2/5 (once the terminal-error case, once both cases), RM-86 1/5 (the submit case). Every failure is the same 10 swaitForSessionStatusafter the test aborts the parent's persistent loop (or the child's terminal error). So the abort → terminal settle of DBOS's recv loop can take over 10 s with no load at all.Recurrence (WE-2's RM-90 gates, 2026-10-19):
- the "submit at root routes through to child workflow" case failed 2 of 4 on the RM-90 head: once in the full DBOS integ run (host load ~4), then once in an alone ×3 (the other two passed);
- every failure is at
test.ts:371, the cleanupwaitForSessionStatusafterparentHandle.abort(), after all of the test's own assertions have passed; - the same line failed on main in the RM-86 A/B above.
RM-90 touches no DBOS abort or recv-loop code.
Recurred 2026-10-30 in the !585 slot4 (load ~27); cleanup poll only: "… submit at root routes through to child workflow" failed at
test.ts:371after all of its assertions passed. The same file passed 2/2 on the previous !585 head, and the delta was one unit-test fixture.A second face, closed (2026-11-15): "persistent child terminal LLM error …" failed at
test.ts:586,expected 'running' to be 'failed', in WE-2's interleaved repeat w-c2d (the file alone: 1 of 3 at origin/main 3fee8e37bf, 1 of 3 at a batch-26 candidate head) and once in WE-2's full DBOS integ run at a batch-26 candidate head (287 tests passed and 3 failed, the other two known faces). Not the poll: the test waited until the child's SESSION readfailed, then read the child's RUN once. DBOS ends a turn K5 first (workflows/shared.ts:2199at 3fee8e37bf, which sets the sessionfailed), then runs the terminal hook steps (:2221 / :2251), and only then writes the runfailed(finalizeRun, :2266), so a read in that gap sawrunning. The terminal order is by design (K5 → run status → stream end), so this is not a product finding. Closed by !874 (batch 25, as on origin/main e93ffcc545): the case holds the child run'sfailedwrite (holdChildRunFailedWrite, :536 and :692), asserts the window, then waits on the run's own terminal record (:607), releasing the held write on every exit path (:679). The first face above stays open.
FU-FLAKE-DBOS-PERSISTENT-MODE-ABORT-5S-POLL: persistent-mode-recv-loop-basic's abort wait is a 5 s wall-clock poll — done (the CP-270 fix MR, !881; owner: WA)
Observed (2026-11-03, the w-lc1 slot of SDK-2 MR 1c's DBOS half): in a full RUNTIME_DBOS_INTEG=1 runtime-dbos integ run, packages/runtime-dbos/src/__tests__/integration/persistent-mode-recv-loop-basic.integ.test.ts › "three-message conversation: subsequent execute() calls route into existing workflow" failed at test.ts:72 with waitForCondition timed out after 5000ms waiting for: session … status in [failed] (the wait after h1.abort()). It failed once.
Not MR 1c's regression (the w-lc2 A/B): at the pre-1c base and at the 1c head the test passed 3/3 each. An untracked probe with the same steps, which logged the session and current-run status around the abort, passed 3/3 each too. It saw two interleavings, both on both sides:
- turn 3 already ended (session
completed):abort()returns in ~11-13 ms and the session isfailedat once (the run record stayscompleted); - turn 3 still running (session
active, runrunning):abort()takes ~540-555 ms, then the session and the run are bothfailed.
The slow path costs ~0.55 s on an idle host, so the 5 s bound leaves ~9× headroom. The w-lc1 failure needs a host stall of that order, or a third shape the probe did not see.
Watch: if it recurs, record the host load and whether turn 3 was still running at the abort. A wait keyed to the run's terminal record (as FU-FLAKE-DBOS-PERSISTENT-COMPANION-TERMINAL-ERROR-10S-POLL proposes) would remove the wall-clock bound.
More hits (2026-11-07, W1's CP-282 slots, read by the owner): main 234725409d 2 of 10 and CP-282's head 1 of 10 (interleaved, same services, gated; slot w1-dbosab-slot-20261107-095232), and once inside the full suite at slot 3b (load 12.9). It is not rare under load: the event-anchored wait is worth doing.
Cause (2026-11-14, WA's slot dbos-base2: ten full DBOS integ runs at origin/main 3fee8e37bf plus a test-only change that makes the waits name their last read): not the 5 s bound. Run 10's failure read last read: completed; current run: <the turn's run> completed: the session never reaches failed, so no longer bound would pass. WA's slot dbos-probe (2026-11-14, 60 iterations with every status write traced) puts every such outcome in CP-270's window: the abort lands after the turn's K5 but before its run status, reads the run live and never stops the loop as a loop. This entry closes with the fix MR that flips CP-270 (and CP-328), once this test passes there; the event-anchored wait stays worth doing.
Closed (2026-11-15, the CP-270 fix MR !881): the cause was CP-270, fixed there. The failures that name what they last read are all CP-270 outcomes: dbos-base2's run 10 read last read: completed; current run: <the turn's run> completed, the abort meeting the turn's tail, and every wrong outcome in WA's slot dbos-probe lay in CP-270's window. The base repeats in WA's slot f12c ran main's wait helpers, which name no last read, so their failures (this test 2 of 20, with the same timeout) are counted, not attributed one by one. The event-anchored wait is settled by the fix MR: the test no longer polls for 5 s but reads the session straight after the awaited abort (persistent-mode-recv-loop-basic.integ.test.ts:71, at origin/main bd30fa746b, !881 in batch 26), since an abort resolves once its stop is durable. At the fix the test passed 20 of 20 in WA's slot f12c's repeats (the four persistent stop and onMessage tests 0 failed in 120, against 7 failed in 100 at main), and it failed in none of the 10 counted full runtime-dbos integ runs (WA's slots f3 and f3x, 297 passed / 0 failed each).
More hits (2026-11-13, batch 24, W1's slots, read by the owner): in s-b24's full runtime-dbos integ run at batch 24's 3252a8218d (RUNTIME_DBOS_INTEG=1), the test failed at test.ts:72 after 6083 ms (1 of 292 failed), with the same waitForCondition timed out after 5000ms … status in [failed]. The control slot then ran the file alone 5 times at batch 24 and 5 times at origin/main 1eed01def5, alternating: batch 24 0 of 5, main 1 of 5 (6302 ms, the same wait). The passing runs took 0.62 to 1.75 s, and a second full suite at batch 24 passed it in 552 ms. The failing run's log shows no store or DBOS error. The timeout does not say which status the session held, so whether turn 3 was still running is unknown (FU-TEST-WAITFOR-STATUS-LAST-READ would show it). The suite stays paused in CI (FU-CI-DBOS-INTEG-PAUSED).
FU-FLAKE-DBOS-PERSISTENT-MODE-INTERRUPT: persistent-mode-interrupt's interrupt wait times out, and in one face the workflow is never cancelled — done (the CP-270 fix MR, !881; owner: WA)
Filed: 2026-11-07, from W1's A/B (main 234725409d vs CP-282's head; slots w1-dbosab-slot-20261107-095232 and w1-dbosab4b-slot-20261107-100600, read by the owner). packages/runtime-dbos/src/__tests__/integration/persistent-mode-interrupt.integ.test.ts › "interrupt sets session to interrupted; follow-up execute starts fresh workflow" fails waitForCondition timed out after 5000ms waiting for: session … status in [interrupted] after h.interrupt(). Pooled, main 3 of 30 and the head 6 of 30, within chance, both faces on main, every failure at load 15 to 101. Not CP-282's. Two faces:
- Late: the workflow is gone (about 6 s, no pending wait) but the session never reads
interrupted. It was filed as a load artefact (the status landing after the 5 s poll). WA's slot dbos-base2 (2026-11-14, at origin/main 3fee8e37bf) shows the status never lands (last read: completed; current run: <the turn's run> completed), and WA's slot dbos-probe puts it in CP-270's window. This face moves to CP-270 and closes with its fix MR. - Non-cancel: the interrupted persistent workflow is not cancelled. The test takes about 60.5 s, and DBOS prints "Waiting for pending workflows to finish" while it waits out the 60 s
persistentIdleTimeoutMsat cleanup. One main run passed while still waiting on a pending workflow at cleanup, which suggests the status write and the workflow cancel are separate and the cancel can be lost. Recorded as possibly a product defect (a DBOS stop that leaves the workflow running), not only a flake. First step when DBOS is unparked: a probe at idle load that checks, afterh.interrupt(), whether the workflow is cancelled (its DBOS status) independently of the session status. If it reproduces at idle load it is a finding, deduplicated against CP-62, CP-16 and the stop records first. Seen again in dbos-base2 run 2 (62.6 s, the pending wait at cleanup). Explained by WA's slot dbos-probe (2026-11-14): it is CP-270's interrupt sub-path. The stop reads the run live, the run status lands meanwhile, the stop answers that the run already ended and returns without stopping the loop, which stays PENDING (3 of 20 interrupt iterations). This face moves to CP-270 too, so the whole entry closes with CP-270's fix MR.
The DBOS integ suite runs outside CI (GL #108).
Closed (2026-11-15, the CP-270 fix MR !881): both faces were CP-270, fixed there. In dbos-base2 the test failed in runs 2 and 5, and both read last read: completed; current run: <the turn's run> completed: the session never reached interrupted. Run 5 (6.4 s, the loop gone) is the late face only. Run 2 shows both faces: the same last read, and 62.6 s with the loop pending at cleanup, which is the non-cancel face (the stop answers that the run already ended and returns with the loop still running; 3 of 20 interrupt iterations in WA's slot dbos-probe). The "if it reproduces at idle load it is a finding" step is answered: it was CP-270. At the fix the test reads the session straight after the awaited interrupt (persistent-mode-interrupt.integ.test.ts:75, at origin/main bd30fa746b, !881 in batch 26), passed 20 of 20 in WA's slot f12c's repeats, and failed in none of the 10 counted full runtime-dbos integ runs (WA's slots f3 and f3x).
FU-CONF-DBOS-IDLE-STOP-FINALIZE-CELL: CP-328 has no conformance cell; its fix is proven by runtime-dbos integ cases only — open (LOW, parked: DBOS; owner: the conformance owner)
Filed: 2026-11-14, with CP-328 (owner ruling on WA's question). CP-328's order cannot be forced through the product handle with store holds alone. The stop sends shutdown, the loop reads it at its control peek and starts its finalize, the stop's cancel lands without stopping that step, the stop's compare-and-set lands, and only then the finalize's session write. A hold on the loop's run-status write sends the stop down CP-270's live path instead, and a loop idle in recv is ended by the cancel with no finalize.
What a cell needs (two DBOS-only harness members): a run-status hold mode that lands the write and holds only its return, and a hold on the stop's DBOS.cancelWorkflow in the lane's process (the default control ops bind it per call, so a wrapper takes effect). Then stop.persistent-idle-stop-vs-finalize on dbos-postgres replaces CP-328's provenByException as its proof. Never seen live (0 of 60 in WA's slot dbos-probe), so LOW.
FU-TURBO-RUNTIME-CLOUDFLARE-TYPECHECK-GRAPH: on a cold build, runtime-cloudflare's typecheck can run before the e2e harness's dependencies are built — open (build graph)
Surfaced by: the WA controller's fast gate for the wait-timeout MR (2026-10-15), on a fresh worktree after a rebase. pnpm turbo run build typecheck lint --concurrency=4 failed @helix-agents/runtime-cloudflare#typecheck twice in a row:
- first run:
../e2e/src/harness/setup-helpers/js-chat-host.ts(52,68): error TS2307: Cannot find module '@helix-agents/ai-sdk' or its corresponding type declarations.(also@helix-agents/agent-server,@helix-agents/ai-sdk/cloudflare, and three follow-onTS18046); - second run:
../e2e/src/harness/setup-helpers/js-chat-host.ts(53,48): error TS7016: Could not find a declaration file for module '@helix-agents/agent-server'.
A third run, with every package already built, was green (122/122). CI's job order hides it today.
- Mechanism: runtime-cloudflare's
tsconfig.jsonincludessrc/**/*, andsrc/__tests__/checkpoint-truth-matrix.wf-noiso.test.ts:24-27imports@helix-agents/e2e/harness/…and@helix-agents/e2e/helpers/…. So its typecheck compiles e2e harness files (js-chat-host.ts), which import@helix-agents/ai-sdkand@helix-agents/agent-server. runtime-cloudflare does not declare@helix-agents/e2e(or those two) in itspackage.json, so turbo'stypecheck(dependsOn: ["build", "^build"]) never waits for their builds; theirdist/*.d.tsmay not exist yet. - Also a pnpm-rule breach: an import that the package does not declare (CLAUDE.md "pnpm rules").
- Fix direction: declare the dependency (a
devDependencyon@helix-agents/e2e,workspace:^, so^buildcovers e2e and its deps), or move that cross-package test out of runtime-cloudflare's typecheck scope (into e2e, where the harness lives). Check for a cycle (e2e depends on runtime-cloudflare) before choosing the first.
FU-SCRIPT-WORKSPACE-NARROW-ENTRY: CloudflareDynamicWorkerWorkspaceProvider and WorkerLoaderScriptRunner have no narrow entry — open (LOW, SDK startup cost)
Both classes are reachable only through the @helix-agents/runtime-cloudflare main entry. A Worker that wants the script workspace therefore loads the whole runtime entry. Add a subpath (like /do) for them and a startup-guard entry for it.
FU-WS-COMPANION-TERMINATED-SUSPENDED-LEFTOVER: a companion terminated while suspended leaves its workspace to reap — open (LOW, RM-20 phase 1 residue)
RM-20 phase 1 ends a companion consult's workspace at its run's terminal commit (the K5 clears the ref; the run's finally destroys). A companion terminated (companion__terminateChild) while SUSPENDED has no live loop to write that K5, and a later respawn deletes the child session (deleteSession in core companion dispatch) with the ref still in it. The workspace (a local-bash / local-sandbox / docker tmpdir) is then left until the app's reap removes it. Fix direction: the session-scoped workspaces spec's §5.1 trigger "companion terminate / respawn → destroy" (d3), destroying before the wipe.
FU-WS-DOCKER-CRASH-CONTAINER-LEFTOVER: a docker container left by a crashed process is never removed — open (LOW, pre-existing)
DockerWorkspaceProvider.destroy(ref) deletes the host tmpdir, and a run's release removes its own container, but a container left by a process that crashed mid-run is not found: the engine interface has no list-by-label call, and resolve() stamps helix.session.id with the workspace id. Fix direction: the spec's d2 docker reap over labelled containers, with the label fix.
FU-WS-INHERITED-FIRST-OPEN-NO-OWNER-RUN: an inheriting child's first open after its parent's run ended is not persisted — open (LOW, RM-20 phase 1 residue)
With SessionWorkspaces, a registry's persistRef is fenced on the session's latest owning run. If an inheriting non-blocking companion is the FIRST to open the parent's workspace after the parent's run ended, the write is refused by the fence (logged), so the parent's next run does not see that ref: it opens its own fresh workspace if no other ref is stored, and the companion's tmpdir is left to reap. Fix direction: the spec's store op (§6, updateWorkspaceBinding with refuseIfLiveRun-style fencing for a holder that is not the owning run).
FU-RUNTIME-CF-TYPECHECK-UNDECLARED-AI-SDK: runtime-cloudflare's typecheck compiles e2e harness files that import an undeclared @helix-agents/ai-sdk — open (build graph, pnpm rule)
Surfaced by: the DO companion MR B merge of MR A's head (2026-10-17): a capped turbo run build typecheck lint failed @helix-agents/runtime-cloudflare#typecheck with TS2305 (module has no exported member) and TS7016 (no declaration file) on @helix-agents/ai-sdk imports in ../e2e/src/harness/setup-helpers/js-chat-host.ts, while ai-sdk's dist was being rebuilt by the same turbo run. The same root as FU-TURBO-RUNTIME-CLOUDFLARE-TYPECHECK-GRAPH (above, a cold build), seen here on a warm tree, mid-rebuild.
- Mechanism: runtime-cloudflare's
typecheckcompiles e2e harness files (e.g.js-chat-host.ts, reached through the@helix-agents/e2e/…imports of its*.wf-noiso.test.ts/finishwith.test.tsfiles undersrc/**/*), and they import@helix-agents/ai-sdk, whichruntime-cloudflare/package.jsondoes not declare. That breaks the pnpm rule (declare every import; CLAUDE.md "pnpm rules"), and turbo has no edge from runtime-cloudflare'stypecheckto ai-sdk'sbuild, so it can read ai-sdk'sdist/*.d.tswhile they are being rewritten. - Fix direction: declare
@helix-agents/ai-sdkas aworkspace:^devDependency of runtime-cloudflare (so^buildorders ai-sdk's build first), or take those harness files out of runtime-cloudflare's typecheck scope (exclude the test files that pull them in from its tsconfiginclude, or move them to e2e). Fix together with FU-TURBO-RUNTIME-CLOUDFLARE-TYPECHECK-GRAPH (which also names@helix-agents/agent-serverand@helix-agents/e2e).
FU-CORE-UNDECLARED-TEST-DEPS: core's tests import vitest and yaml, which core's package.json does not declare — open (pnpm rule)
Raised by Greptile on !533. packages/core/package.json declares neither vitest nor yaml:
- 328 files under
packages/core/srcimportvitest; - the repo tests (
src/__tests__/repo/*.test.ts: train-verdict, ci-heavy-once, ci-image-and-retry) importyaml.
Both resolve today through the root's devDependencies, by parent-directory lookup. The CLAUDE.md pnpm rule says to declare every import, test-only ones included, in the package's own package.json. It is not caused by any one MR.
Fix: add vitest (the root's range) and yaml to core's devDependencies, update the lockfile, then check whether other packages lean on root devDependencies the same way (a grep of each package's imports against its own package.json).
FU-EXAMPLES-E2E-SHARED-MOCK-PORT: a local turbo run test:e2e starts four examples' mock OpenAI servers on the same port — open (harness isolation; owner: CI-health-3, routed by the orchestrator)
Surfaced by: WE-2's local DoD matrix for !352 (2026-10-13). The root pnpm run test:e2e failed @helix-agents/example-opennext-cloudflare-do#test:e2e in 2 s with listen EADDRINUSE: address already in use :::3030 from startOpenAIMockServer (examples/opennext-cloudflare-do/test-utils/openai-mock-server.ts:603). The same suite run alone passed (65 tests).
- Mechanism: opennext-cloudflare-do (
playwright.config.ts:5,OPENAI_MOCK_PORT || '3030'), approval-gates (playwright.config.ts:24-29,test-utils/openai-mock-server.ts:330), research-assistant-cloudflare-do (playwright.config.ts:40-46) and remote-agents-cloudflare-do (playwright.config.ts:34) all serve their mock OpenAI on 3030, and turbo runs theirtest:e2etasks concurrently. - Worse than a crash: each config sets
reuseExistingServer: true, so an example whose health check finds another example's mock already on 3030 reuses it. Its tests then run against a different app's scripted responses, a silent cross-wiring that can pass or fail for the wrong reason. - CI is not exposed: each example has its own
test:examples:*job (its own pod). The defect hits localturbo run test:e2eand the DoD's local matrix. - Fix direction: a distinct default mock port per example (or an ephemeral port passed to both the mock and the app), and a mock health endpoint that names its example, so
reuseExistingServercan never adopt a foreign mock. Alternatively, make the examples'test:e2eserial in turbo.
FU-CI-DBOS-INTEG-PAUSED: CI runs zero DBOS integration tests, so a DBOS integ regression is never seen on main — done (batch 26, !884; owner: WA)
Surfaced by: the CP-161 gate (its DBOS integ step, run locally with RUNTIME_DBOS_INTEG=1). .gitlab-ci.yml's test:integ:dbos job is a paused placeholder (its comment block above the job, ~lines 290-301: "this shard runs 0 tests today"), and packages/runtime-dbos/vitest.integ.config.ts sets include: [] unless RUNTIME_DBOS_INTEG=1 (GL #108: cold-start timing flakes under the shared single fork). So companion-continuation.integ.test.ts's negative control has been red on main since CP-155 (d919c9b584: its run-first notifier heals a forced-failed ref over a completed child, so the sendMessage continued it) and nothing noticed until a local gate ran it; CP-161 rewrote the control so the child genuinely fails. To close: run the DBOS integ suite in CI, or in the release gate (the un-pause steps in the job comment: the flake fixes of GL #108, then parallel + --shard), so a DBOS integ red is seen on the MR that causes it. CP-161 does not change CI.
Also (2026-10-29, the CP-153 DBOS gate): two more DBOS integ files are red at main e34b96e2c2 ×2 (local, RUNTIME_DBOS_INTEG=1), with the same failure as on the CP-153 branch, so neither is CP-153's:
approval-registration-window.integ.test.ts› "a forged approval submitted while the registration is held is refused; the gated tool never runs on it":Error: Awaited agent__aw-child__…-run-1 was cancelled, thenAssertionError: expected undefined to be definedat:193:31(the child's pending approval entry is gone when the test reads it to send the real answer).remote-subagents-step-retry.integ.test.ts› "D1 regression: forced mid-stream step retry reconnects via fromSequence with no duplicate proxied chunks": awaitForConditiontimeout after 10000 ms, waiting forfirst stream() batch cadence-persisted at chunk 25, thenHook timed out in 60000msat:491:5.
Triage (the conformance owner, from source): both are stale tests, not product bugs. The two DBOS integ files derive child session ids (pre-7503eb1de5) and are red at main: approval-registration-window (:163 / :168 / :192) and remote-subagents-step-retry D1 (:482 / :494 / :549). Fix: read the minted id from the parent's ref by parentToolCallId, and forge with the entry's real approvalId. Both product properties (!518's forged-approval refusal; D1's fromSequence reconnect) are unproven on DBOS until then. Fixed by !587 (both tests read the minted id from the parent's ref; red at main, green ×3, and a mutation per property fails them).
Also (batch 22, 2026-11-12, MR !859 merge pipeline 3060552140): job 17976466376 test:integ:dbos finished success with "No test files found, exiting with code 0" ("Run with RUNTIME_DBOS_INTEG=1 to exercise the 48 integration tests"). The job (ci/test-matrix.yml, pnpm exec turbo run test:integration --only --filter=@helix-agents/runtime-dbos) still sets no RUNTIME_DBOS_INTEG, and !259 (the un-pause) is still open. So a green test:integ:dbos is not coverage: a coverage claim names the job's test count from its trace. Batch 22's DBOS change (RM-133: findStepCommit and the retry plan read run checkpoint metadata) was covered by the runtime-dbos unit suite (1284 passed) and find-step-commit.test.ts in WA's slot rm133f, and by 15 chosen runtime-dbos integration files (40 passed, RUNTIME_DBOS_INTEG=1) in WA's slot rm133d at d9ff7ba60e, whose runtime-dbos, core and store-postgres code is byte-identical to the 4e74a4e850 that !854 landed (the rest of the suite did not run there). Then W1's slot sdbos ran the whole runtime-dbos integration suite (vitest.integ.config.ts, RUNTIME_DBOS_INTEG=1, local Postgres and Redis containers) at the batch head e5fa959f8a, whose tree is the one batch 22 landed as 18f9d7509e: 101 files passed and 1 skipped, 290 tests passed, 2 skipped, 0 failed. The options, for the user:
- set
RUNTIME_DBOS_INTEG=1on the job, with the Postgres service it needs (after GL #108's flake fixes, as above); - make the job fail when it runs zero tests (vitest's
--passWithNoTests=false, or a count gate), so a paused suite reads red instead of green; - drop the job and record that DBOS integration is local-only.
Closed (2026-11-15, batch 26, !884): test:integ:dbos runs the suite on every MR and train pipeline (ci/test-matrix.yml: RUNTIME_DBOS_INTEG=1, parallel: 2 with --shard, and RUNTIME_DBOS_INTEG_REQUIRE_TESTS=1, whose guard in packages/runtime-dbos/scripts/run-integ-tests.mjs fails a paused suite or a shard that ran zero tests). GL #108's flakes were fixed first (!874 in batch 25; CP-270 and CP-328, !881 in batch 26). First real runs, batch 26's !887: merge-result pipeline 3070902232 and train pipeline 3070968920, each shard 1/2 167 passed / 1 skipped and shard 2/2 130 passed / 1 skipped, 297 / 2 over 103 files, every job green. WA keeps a per-shard ledger of the batch pipelines. Release need (user ruling 2026-11-16): stays as is, no flip. test:integ:dbos has gated the release since batch 26 through the train gate and the child matrix; optional: true covers only a main push that has no test jobs. Green batches on record: 26, 27 and 28. Batch 29 (!908) added the CI comment and the repo pin test (core/src/__tests__/repo/ci-heavy-once.test.ts:287, origin/main 0aed8d1250).
FU-CI-K8S-QUOTA-SINGLE-PIPELINE-SATURATES: amd64 jobs are rejected at pod setup by the runner namespace's CPU quota — open
Owner: CI-health-3. Surfaced by: !308 pipeline 2918679433 (2026-10-06): test:examples:approval-gates and test:examples:remote-agents-cloudflare-do failed 3 attempts each in 1–18 s, runner_system_failure, no test run. Related: the CI runner contract migration (.gitlab-ci.yml, another session): a CI change here must fit around it.
Signature:
prepare environment: setting up build pod: pods "runner-…" is forbidden: exceeded quota: gitlab-runners-quota, requested: limits.cpu=6500m, used: limits.cpu=189700m, limited: limits.cpu=192. The runner is thehelix-aigroup runner "Intel based runners on digital ocean" (amd64), so every project in the group shares the 192-CPU quota. GitLab's automaticrunner_system_failureretries fire within seconds, so all three attempts hit the same full quota.Frequency (CI-health-3, from job traces): 94 quota rejections in helix-agents between 2026-09-30 and 2026-10-06 (19, 9, 17, 27, 16 and 6 on the six days that had any).
CPU limit per pod, from the rejection messages:
.gitlab-ci.ymlsets no CPU values, so these are runner defaults. The split below is inferred from the two values:Jobs Pod limits.cpuInferred split test:integ:e2e×4,test:integ:rest,test:integ:dbos,test:conformance×4,test:conformance:dbos9.5 build 6 + helper 0.5 + 3 services (redis, postgres, temporal) × 1 test:examples:*×4 (approval-gates,remote-agents-cloudflare-do,research-assistant-cloudflare-do,opennext-cloudflare-do)6.5 build 6 + helper 0.5 Stage overlap: all 15 amd64 jobs
needs: [build]only, so they all start the momentbuildfinishes (16:52:13–16:52:25 in 2918679433). One pipeline asks for 11 × 9.5 + 4 × 6.5 = 130.5 CPU at once, 68% of the quota. Two overlapping helix-agents pipelines (261) cannot fit, and neither can one pipeline beside other group projects' jobs.At the 16:54 rejection: 189.7 CPU in use. This pipeline's live pods account for about 108 (10 × 9.5 + 2 × 6.5, after
test:integ:dbosfinished), and the only other group pipeline live then (video-gen/agents2918797303) had 2 jobs, about 15. About 67 CPU are unaccounted for (189.7 − 108 − 15; an earlier "about 57" still counted the finishedtest:integ:dbos). Pods outliving their jobs (termination lag, or a pod whose cleanup failed: the rejected job's trace logsError cleaning up pod) are a lead. Confirming it needs a pod listing of the runner namespace.Fix options, in priority order:
- Lower the per-pod limits where the job doesn't need them:
- drop services a job does not use (each costs 1 CPU; e.g.
test:integ:dboslikely needs only postgres; unverified); - set job-level
KUBERNETES_CPU_LIMIT/KUBERNETES_SERVICE_CPU_LIMIT(needs the runner's*_overwrite_max_allowed). First measure each job's real CPU use so a limit doesn't slow it.
- drop services a job does not use (each costs 1 CPU; e.g.
- Restage: bound one pipeline's peak by running the examples after integration/conformance, or by giving each heavy job a
resource_group, so two pipelines queue on a job instead of overlapping. Whetherresource_groupaccepts$CI_NODE_INDEXfor the parallel jobs is unverified. - Runner side (the user's infrastructure; a request, not an action): set the runner's
limit/concurrentso it never starts more pods than the quota fits (jobs then wait aspendinginstead of failing), or raise the quota, and clear any leaked pods. The CI runner contract migration may change these numbers: re-measure after its flip.
- Lower the per-pod limits where the job doesn't need them:
FU-CI-RUNNER-GET-SOURCES-FAILURE: CI jobs fail in get_sources, before any script runs, on the do-nyc3 runner — open (CI infrastructure; owner: unassigned)
Surfaced by: !931's MR pipeline 3087168720 (2026-11-19). Related: FU-CI-K8S-QUOTA-SINGLE-PIPELINE-SATURATES (the same runner pool, a different stage).
- Sighting 1, DNS:
test:conformance 3/7job 18168293063, do-nyc3 (Kubernetes executor, namespace ci-jobs). The pod started (the trace loggedContainersNotReadyfor build, helper and the services while they came up), thenget_sourcesranFetching changes with git depth set to 20...and failed after 52 s:fatal: unable to access 'https://gitlab.com/helix-ai/open-source/agents.git/': Could not resolve host: gitlab.com (Timeout while contacting DNS servers), thenERROR: Job failed: command terminated with exit code 128. The job record readsscript_failure, 67.8 s. Its one disclosed retry, job 18168514443, passed. - Sighting 2, an interrupted fetch: !932's MR pipeline 3087479794,
test:examples:opennext-cloudflare-dojob 18170434391, do-nyc3:get_sourcesfailed 10 s into the fetch witherror: 23827 bytes of body are still expected,fetch-pack: unexpected disconnect while reading sideband packet,fatal: early EOFandfatal: fetch-pack: invalid index-pack output, then exit code 128;script_failure, 66.7 s. Its one disclosed retry, job 18170900991, passed (554.4 s, do-nyc3). - Not a test result: in both, no script line ran, so no test ran and nothing was uploaded; the job's tests are unrun, not red.
- Why GitLab did not retry them:
default.retryinci/common.yml(:42-56at origin/main fae53b26ac) retriesrunner_system_failure,unknown_failure,stuck_or_timeout_failure,api_failure,scheduler_failureandrunner_configuration_error(max 2). A failedget_sourcesis reported asscript_failure, which the block deliberately excludes so a real test failure never re-runs silently. - Candidates (not chosen; a CI config change needs the user's go): GitLab Runner's
GET_SOURCES_ATTEMPTSvariable, which retries only the source fetch and leavesscript_failureof the tests alone; or a look at the runner cluster's DNS (its CoreDNS / node resolver) and its egress to gitlab.com. - Not established: whether the two shapes share a cause (the pod's network, the node's, or gitlab.com's side).
- CI: until this is settled, a recurrence is infrastructure. It gets a disclosed retry under this entry, and its links go on record.
FU-FLAKE-CONF-TEMPORAL-CELL-REAP-WALK-TIMEOUT: the Temporal cell-end reap timed out on session.create-observed-whole with nothing left running, and the rebuild's dispose poisoned the lane — fixed
Closed by: !318 (test(conformance): the Temporal cell-end reap scales with session count and never races itself). Owner: conformance owner. Seen on: main job 17007806563 (test:conformance 5/5, temporal-redis) and !310 job 17009383753 (temporal-memory, the cell crashed at 98.7 s).
- Signature: cell
session.create-observed-whole|temporal-*|executorcrashes withcell end: could not end the backend work of [<48 sessions>]: ending the backend work of the runs timed out after 5000ms. The next cell (often a declared gap) then takes about 5 s and the lane is poisoned:rebuild after crash failed: dispose: could not end the backend work of [<every session the instance ran>]: … timed out after 5000ms. Every later cell of the lane is GATED. The dispose message lists every session the instance ran, so the cell after a rebuild (e.g.stop.mid-tool-ignores-abort) can look implicated. It is not. - Not a leak: all 48 runs had finished; the reap had nothing to terminate.
- Cause (harness design, exposed by CI load):
StartedWorkflows.terminateTreewalked every execution sequentially, 5 backend calls each even when already closed (240 calls for 48 sessions), under a fixed 5 s bound for the whole call (REAP_TERMINATE_BOUND_MS). At about 21 ms per call (a ~3x loaded node) that exceeds 5 s. The rebuild's dispose then repeated the same walk under the same bound while the timed-out walk was still running (480 overlapping calls), so the retry was all but certain to fail too. RCA: the conformance owner'stemporal-reap-rca.md. - Fix: an execution seen closed is never read again (2 calls each); up to
TEMPORAL_REAP_CONCURRENCY(8) trees are walked at once; each backend call is bounded byREAP_CALL_BOUND_MS(3 s), the whole cell-end call byreapCellWorkBoundMs(runs)(5 s + 10 ms per run), dispose byDISPOSE_CELL_WORK_BOUND_MS(10 s); a timed-out walk is aborted and a later call waits for it, then walks only what is left; the error names only the executions still unchecked or unended, the calls made and the elapsed time. A workflow still RUNNING after terminate still fails the cell. Red-first unit tests intemporal-client-adapter.test.tsandexecutor-driver.test.ts(the latter reproduces the poisoned rebuild over a slow fake), plus a 48-session slowed-reap probe intemporal-cell-reap.integ.test.ts.
FU-CF-CLIENT-DISCONNECT-QUIET-RUN-EDGE: does Cloudflare's edge report a client disconnect on a quiet response promptly? — done (edge-measured)
Owner: cf-we-do-streams. Surfaced by: CP-102 (local Miniflare and wrangler dev did not report a quiet client disconnect to a Worker within 5 s, so a quiet run was noticed at its next chunk; W1's 2026-11-13 probes measured local workerd per platform, CP-102). Closed by: the CP-102 MR, with an edge probe on 2026-10-08 (the user approved two throwaway workers.dev Workers; both deleted afterwards, confirmed by the API's "does not exist" and HTTP 404).
- Measured (5 runs each): with
enable_request_signal, a client that leaves a quiet stream is reported throughrequest.signalwithin about 0.2–0.3 s, and the Durable Object's/sserequest (the Worker passes its signal tostub.fetch) learns of it within about 0.1 s of the disconnect (upper bounds, observed from the client). Without the flag nothing fires within 30 s, not even on a stream writing every 50 ms. The response body'scancelnever fired on the edge in any configuration. - Outcome: the "learns at the next write" limit is local only; on the edge the prompt close works with the flag. The flag is REQUIRED (the docs say so), and
handleChatStreamnow warns once per isolate when the runtime reports it off (corerequestSignalSupport()). The table is in the CP-102 spec's As-built section; the raw results are archived with the reliability orchestrator.
FU-FLAKE-AI-SDK-DISCONNECT-PATH6-TIMEOUT: ai-sdk client-disconnect.test.ts terminal-replay case timed out once under heavy load — open (watch)
Owner: cf-we-do-streams (from CI-health-3). Surfaced by: local runs on the client-disconnect branch (CP-102), 2026-10-07; not yet seen in CI.
- Signature:
handleChatStream: client disconnect > terminal replay (path 6): an abort closes the store reader and ends the bodytimed out at 10 s: the response body never ended after the client's abort. - Seen: once in about 27 full-file runs, in the first run right after the tests switched to a held upstream (one chunk, then a read that never resolves, so the abort is the only way the body can end), at a load average of about 138 on 14 cores. That load included 14
yesburners CI-health-3 had left running from its own loaded opennext runs (07:28Z to about 15:00Z, killed by the orchestrator; about 14 of the load), not only other sessions' work. - Not reproduced: 0 failures in 33 further runs: 21 full-file (6 with the SSE builder's abort path instrumented) and 12 of the path-6 case alone, at load 115–125. Starvation of the test process at about 10× oversubscription is plausible but not proven.
- If it recurs (in CI or locally): capture the abort path (does
onAbortrun, doescontroller.close()throw, is a pull still in flight) and the replay reader'sclose(), before changing anything; never raise the timeout.
FU-FLAKE-RUNTIME-JS-K5-COMPLETED-STATUS-WRITE: runtime-js commit-checkpoint-truth-fixes-2.test.ts K5 completed-status-write case resolves instead of rejecting — done
Closed by: this MR (CI-health-3 CI-flakes bundle). Owner: CI-health-3 (routed by the orchestrator). Surfaced by: MR !297 pipeline 2912057160, job 16928257935. Seen once before on !288's branch, job 16900882183 (2026-10-02).
- Test:
runtime-jssrc/__tests__/commit-checkpoint-truth-fixes-2.test.ts› "Task 11 fix round 2 — runtime-js › a failed completed-status write after K5 reports completed, and the next entry finishes (never supersedes) that run". - Signature:
AssertionError: promise resolved "{ sessionId: 'k5-status', …(11) }" instead of rejecting. The otherELIFECYCLElines in the job are turbo cascade kills, not failures. - Not load: nothing else was stressing the host at the time (00:4x, 2026-10-05), so this is treated as a genuine intermittent test or product race, not CI starvation. !297 changes no runtime-js code.
- Cause (test design, not a product bug): the test gave the run a real 300 ms lease, judged on the store's
Date.now().writeRunStatusstops renewing it right before the terminal status write, which the test fails 3× (25 + 50 ms backoff). The'early'entry then had to reach its run-start commit inside what was left of 300 ms of wall time to be refused. When it arrived after the lapse it legitimately ended the run and won. That is the documented JS lease contract (the FU-FLAKE-JS-LEASE-BLOCKED-RENEWAL family). - Red-first: a 350 ms stall injected before the
'early'entry reproduces the CI assertion on the old test. - Fix:
Date(the store's lease clock and the lease's local clock) is frozen for the test and advanced past the ttl instead of sleeping. The same stall passes. - Siblings (same latent race):
commit-checkpoint-truth-fixes.test.tsI3 and I4 held a real 300 ms lease through 500 + 200 ms hook sleeps and asserted the intruder is refused. A 400 ms busy-wait in the hook failed both (expected 'superseded' to be 'completed'). They now hold 5 + 2 periods of lease time on the frozen clock, advancing by ttl/3 only after a renewal has landed, so a run whose renewals stopped times out instead of passing on luck.
FU-FLAKE-AGENT-SERVER-STATUS-LIVENESS-SLEEP: agent-server tests slept a fixed window for a run to finish — done
Closed by: this MR (CI-health-3 CI-flakes bundle). Surfaced by: !299 pipeline 2914044694, test:unit job 16942478423 (retried once under the disclosed-retry rule as 16944739774).
- Signature:
agent-server-status-liveness.test.ts› "reports isExecuting:false after a run completes and the handle is released":expected 'running' to be 'completed'. - Cause (test design): it waited a bare
setTimeout(r, 250)for the run to complete. On a loaded runner the run had not, sogetStatushonestly reportedrunning. - Red-first: a 400 ms model step (a loaded host) fails the old test with the CI signature, and the fixed one passes.
- Fix: an event wait, the existing
waitForStatushelper, plus a newwaitForNotExecutingfor a released handle. Both bounds are failure signals, never performance budgets. - The family (every bare sleep in the agent-server tests, classified):
- Fixed, sleep before a positive assumption:
agent-server.test.ts×2 andchat-routes.integ.test.ts×2 (interrupt and abort of a "terminal" session),http-handler.test.ts(replica A's released handle),agent-server-end-frame.test.tsandintegration/lifecycle.integ.test.ts(interrupt, abort andisExecuting"while running"), andintegration/workspace-route.integ.test.ts×2 (the registry cleared after completion). The two slow tools now signal when they start and run until aborted (each test ends by aborting), instead of completing on their own after 2 s of wall time. - Kept, with a comment:
express-adapter.test.ts×3 (each delay precedes a negative "not called" check, which a slow host can only pass vacuously, never fail falsely) andruntime-parity.integ.test.tscleanup (a teardown pause with no assertion after it). - Fine as is: poll intervals inside polling loops (
agent-server-stateless,runtime-parity,helpers/test-serverwaitForStatus,workspace-routepolling), and theworkspace-routemock provider's deliberate window delay.
- Fixed, sleep before a positive assumption:
FU-CI-UNIT-TURBO-STOPS-AT-FIRST-FAILURE: a turbo CI job stopped at its first failing package — done
Closed by: this MR (CI-health-3 CI-flakes bundle). Surfaced by: !299 pipeline 2914044694, test:unit job 16942478423: Tasks: 6 successful, 36 total, so runtime-cloudflare's tests never ran.
- turbo's default
--continue=nevercancels every other task once one fails, so a red job hid whether the other packages passed (and the MR's own tests could have no CI result at all). - Fix:
--continue=alwayson the multi-package turbo jobs (test:unit,test:integ:rest,test:cloudflare,test:workflows,typecheck,lint). Build and single-package jobs are unchanged. - Demo (6 packages, one with a deliberately failing test,
--concurrency=1, ×2): withnever, 0 and then 4 of 6 tasks ran to success; withalways, 5 of 6 both times. Both modes exit 1.
FU-FLAKE-CFW-STOP-CONDITIONS-RUN-STATUS-READ: CFW stop-conditions tests read the run record before its terminal status — done
Closed by: !299 (CI-health-3). Surfaced by: !298 pipeline 2913874706, test:workflows job 16941132802 (retried once under the disclosed-retry rule as job 16941808210, green).
- Signature:
runtime-cloudflarecfw-stop-conditions.wf-noiso.test.ts› "retry.max-steps-continues-without-message (C5, no step boundary)":expected undefined to be 'max_steps'at the run record'scompletionReason, while the result'serrorDetail.codewas alreadyframework_max_steps_exhausted. - Cause (test, not product): the file's
waitForTurnreturned once the session was terminal, which the K5statecommit writes. The terminal order on every runtime writes the run's terminal status, carryingcompletionReason, after K5. An assertion on the run record could read that gap. - Red-first: a store wrapper (
testWorkflowInjection.setStateStoreWrapper) that delays the workflow's terminalupdateRunStatusby 2 s fails the CI case with its exact signature, and a sibling (.fresh-budget,expected +0 to be 2). With the fix, both pass under the same delay. The demo wrapper is not committed. - Fix:
waitForTurnalso waits until the current run has left the live statuses (isLiveRunStatus), so every run-record assertion in the file reads after the run's own terminal write.
FU-FLAKE-CLIENT-TOOL-RECOVERY-REDIS-JS-SUSPEND-RACE: client-tool recovery tests resumed before the original run finished suspending — done
Closed by: !306.
Surfaced by: !305 job 16956379754 (a text-only MR).
- Signature: e2e
client-tool-recovery-redis-js.integ.test.ts› "orphan drains with runtime_restarted on fresh-executor resume":AgentAlreadyRunningError … (status: suspended_client_tool)from runtime-jsrun-start.ts(the CP-96expected_mismatchloser), called fromexecutorB.resume(). - Cause (test): the test waits only for
pendingClientToolCalls['orphan-tc']to persist, then flips the session tointerruptedand resumes from a fresh executor B. On runtime-js the suspension order is frames → suspension commit (the pending entry persists here) → the run'ssuspended_client_toolstatus →pauseStream, and executor A is still alive finishing it. B planned against A'srunningrun; A's status write landed before B's commit, so B correctly lost the run-start CAS. - Not a product bug: in every variant A is live and still writing, so refusing B is the CP-96 gate doing its job. A genuinely crashed A holds its run only until its JS lease lapses on the store clock (documented).
- Fix: wait for the stream's
pausedstatus (it comes last, and it is the documented "suspension committed" signal) as well as the pending entry, in this test and its two siblings with the same wait (client-tool-recovery-js,client-tool-error-code-invariant-js). - Red-first: a store wrapper on executor A that delays its
suspended_client_toolrun-status write by 300 ms fails the old test 1/1 (AgentAlreadyRunningError) and passes the new one. The wrapper is not committed.
FU-FLAKE-SKILL-FS-PROVIDER-HOOK-TIMEOUT: skill-fs filesystem-provider.test.ts setup hook timed out at 10 s — open (diagnostics in)
Owner: CI-health-3. Surfaced by: !304 pipeline 2915625900, test:unit job 16953822082 (2026-10-05).
- Signature: the file fails at file level with
Hook timed out in 10000ms, thenTypeError: The "path" argument must be of type string … Received undefinedatafterAll(rm(root, …)). - What the trace shows:
rootwas still undefined inafterAll, so thebeforeAll's first await,mkdtemp(join(tmpdir(), 'helix-skills-')), had not resolved after 10 s. The hook timer fired on time (10 014 ms), so the event loop was running; the filesystem call was what stalled. - Context:
- The job ran on the shared local runner (arm64, Docker executor): this Mac's Docker VM. Its disk was at 83% (339/410 GB), its executor prepare took 63 s to pull
node:24, and in the same job store-memory'scollecttook 42 s. - At about the same time, CI-health-3 was running single-file store-postgres integ and unit runs against its private Postgres container on that same VM disk (within the shared-host rules, but extra I/O).
- vitest's default
forkspool gives each file its own process, so nothing else in that process was using its libuv threadpool. - The working hypothesis is an I/O stall on the VM's disk, not this test's work. Unproven.
- The job ran on the shared local runner (arm64, Docker executor): this Mac's Docker VM. Its disk was at 83% (339/410 GB), its executor prepare took 63 s to pull
- Frequency: 1 of the 19 failed
test:unitjobs since 2026-09-26 (the first sighting). - Done (!306): the setup is self-diagnosing and the teardown null-safe, with no timeout raise. Each fs step is timed; if the hook never finishes,
afterAllfails with the step still pending and how long (fixture setup did not finish: step "mkdtemp" still pending after 10005ms …), not with aTypeErrorthat masks it. Demo with a never-resolvingmkdtemp: the old file reproduces the CI signature exactly; the new one names the step. - Open: the root cause, from the next sighting's named step. On a recurrence, an unrelated-flake retry with disclosure (§A3).
FU-FLAKE-CF-MULTIRUN-150MS: fixed sleep before interrupt() lets mock responses leak into the resumed run — open (partly fixed by !306)
Partly fixed by: !306 (the two files below); the siblings are open.
Origin: !264 (cf-multirun-flake-fix, closed as superseded; carried forward).
- Cause: the multi-run tests queued run 1's mock responses (text, then a slow tool call), called
execute(), slept a fixed 50–150 ms and interrupted. On a slow runner run 1 had not reached the tool call yet, so the scripted responses stayed queued and the resumed run 2 consumed them. Setting the sleep to 0 reproduces it every time. - Fixed:
multi-run-streaming.integ.test.ts(4 sites): a localwaitForToolStartwaits for the slow tool'stool_startchunk and assertsgetRemainingResponseCount() === 0, so a leak cannot pass silently.ai-sdk-d1-with-js.cf.test.ts"interrupt and resume with state preservation via DO stream": it waited only for the increment commit, which lands before step 2's LLM call, sotc-abortcould leak into the resume. It now also waits fortc-abort'stool_startand asserts no response remains.
- Open: the same pattern elsewhere. A heuristic grep (a
setTimeout(r|resolve, N)within 4 lines beforeinterrupt(/stop(/abort() inpackages/e2e/src/__tests__:interrupt-resume-js(15),interrupt-resume-d1-with-js(13),interrupt-resume-redis(12),redis-cas-races(11),interrupt-resume-dbos(11),interrupt-resume-subagent(9),deep-subagent-abort(7),concurrent-coordination(7),stream-ordering(6),error-recovery(6), and fewer elsewhere (!264's counts). Some are poll loops or deliberate "interrupt at an arbitrary point" races, so audit each before converting it.
FU-FLAKE-WAITFOR-NO-FINAL-RECHECK: poll helpers that give up without checking once more after the deadline — open (partly fixed by !306)
Partly fixed by: !306 (the shared helpers); the local copies are open.
Origin: !266 (unit-timing-flakes-fix, closed as superseded; carried forward).
- Defect: a
while (Date.now() - start < timeout) { check; sleep }loop followed bythrow. On a loaded host the last sleep can overrun the deadline, and a condition that became true during it fails the wait anyway. - Fixed:
- the shared helpers, one change each covering many tests: runtime-dbos
integration/helpers/poll.ts(waitForCondition, also behindwaitForSessionStatus), agent-serverhelpers/test-server.ts(waitForStatus) andhelpers/wait-for-status.ts(waitForStatus,waitForNotExecuting), e2ehelpers/stream-collector.ts(waitFor),helpers/timing-helpers.ts(waitForCondition) andhelpers/do-subagent-bridge.ts(waitForCompletion). Each now checks, and only then breaks on the deadline, so its last check lands at or after it. - red-first:
agent-server/src/__tests__/wait-for-status-final-check.test.ts(fake clock; the condition holds only after the deadline): 2 of 3 fail on the old helpers, 3 of 3 pass now. - !266's local fixes in
client-tool-real-crash-recovery,companion-usage-scopingand theresearch-assistant-cloudflare-dosmoke test. - Where a real causal signal exists, no poll at all:
completed-marker-reaper.integ.test.tsawaitshandle.result()(the suspension commit precedes it), andresearch-assistantconcurrency.test.tsawaitsexecute()/retry()(their run-start commit has landed; the mock LLM answers only after asetTimeout(10)macrotask, so the concurrent call always meets the live run).
- the shared helpers, one change each covering many tests: runtime-dbos
- Open: local copies of the same loop. A structural scan (each
waitFor*function's body; the statement after itswhileloop is a barethrow) finds, after this fix (b57ac80864), 86 helpers in 80 files: e2e 57, runtime-cloudflare 15, runtime-temporal 1, runtime-js 1, agent-server 1, and 11 in examples (opennext-cloudflare-do 5, research-assistant-cloudflare-do 2, workspaces-showcase, research-assistant-dbos, remote-agents-cloudflare-do, nextjs-redis 1 each). Outsidetest:unit, the dominant flake driver there is service latency, not this one-interval window, so they are left for a per-job diagnosis rather than an 80-file sweep; switching a file to a shared helper fixes it for free.
FU-TEST-WAITFOR-STATUS-LAST-READ: runtime-dbos waitForSessionStatus times out without the status it last read — open (test only)
Filed: 2026-11-13 (W1, from batch 24's s-b24 slot; orchestrator-approved). The runtime-dbos integration helper waitForSessionStatus (packages/runtime-dbos/src/__tests__/integration/helpers/poll.ts:35-49) names only the expected statuses in its timeout (… status in [failed]), never the status it read last. In s-b24 and its control slot (FU-FLAKE-DBOS-PERSISTENT-MODE-ABORT-5S-POLL) a timeout could not tell a session that ended in another status from one still live. Fix: keep the last loadState(...) status (or null) in the predicate and put it in the timeout message, for example … status in [failed]; last read: active.
FU-FLAKE-FIXED-SLEEP-THEN-ASSERT: a fixed sleep, then an assertion on detached background work — open (partly fixed by !306)
Partly fixed by: !306 (run-loop-persistent); the js-agent-executor* sites are open.
Origin: !266 (carried forward).
- Defect: a test starts fire-and-forget work (a non-blocking persistent child, a background resume), sleeps a fixed 50–80 ms and asserts on its effect. Under load the work has not landed; unlike a poll there is no timeout error, the assertion just fails (or passes on a stale value).
- Fixed,
runtime-jsrun-loop-persistent.test.ts: every positive site waits for the real signal (a ref registered or terminal, the child's LLM called, its abort observed, the follow-up message appended).- Two of !266's conversions were vacuous: the 'recur' and 'r1' re-spawn tests waited for a ref that is seeded before the run, so the wait was true at once and the
completionDeliveredassertion still raced the reset. Both now wait for the child's new user message, which the continue / re-spawn path appends only after the ref reset. - Two sites stay as documented negative windows (a regression would start a second loop or call the child's LLM later; absence needs a window): the failed-history-read resume (the failure itself is already in, inside the awaited
resumeChildAgent) and the CAS-loser continuation.
- Two of !266's conversions were vacuous: the 'recur' and 'r1' re-spawn tests waited for a ref that is seeded before the run, so the wait was true at once and the
- Open:
js-agent-executor-interrupt.test.ts(138–329, 377, 414, 440, 469, 503, 525, 594, 626),js-agent-executor.test.ts(973, 1976, 2071),js-agent-executor-resume.test.ts(327, 1078, 1143): a fixed 30–500 ms sleep after starting a slow tool, thenabort()/interrupt()/canResume(). A tool's stagedupdateStateis not visible in the store until its step commits, so a store poll does not work; the tool'stool_startstream chunk does (as in FU-FLAKE-CF-MULTIRUN-150MS), or an in-test gate the tool signals.
FU-FLAKE-CF-DO-PERSISTENT-AGENTS-SHARED-SCRIPT-QUEUE: a stray child from one CF DO persistent-agent test steals the next test's scripted responses — done (test only)
Owner: cf-we-do-streams-2. Closed by: this MR, together with its duplicate FU-FLAKE-CF-DO-CASCADING-SUBAGENTS-SPAWN-RESULT (the fix and its evidence are recorded there). Surfaced by: the CF-D2 MR1 DoD matrix (test:cloudflare, 2026-10-07); diagnosed by an instrumented trace on origin/main (the counts and refs below).
- The test:
packages/e2ecloudflare-do-persistent-agents.cf.test.ts› "cascading nested sub-agents (3 levels)" fails intermittently at:954(expected [] to have a length of 1 but got +0: no child spawn), or in the:922shape (expected 'failed' to be 'completed', about 120 ms). It needs the full.cf.test.tssuite's timing; the file alone never failed (6 runs).- origin/main e2f6bdddf3: uninstrumented full suite 1 failure in 7 runs (the
:922shape); with an in-memory trace 1 in 5 (the:954shape). - the CF-D2 MR1 branch (0e386bcb08): uninstrumented full suite 2 failures in 7 runs (one of each shape; 3 in 8 with the DoD run); with the in-memory trace 0 in 5. The rate difference is within noise. Not caused by MR1.
- origin/main e2f6bdddf3: uninstrumented full suite 1 failure in 7 runs (the
- Root cause (trace captured on main):
test-worker.ts:394gives every persistent-agent test DO one shared FIFOMockLLMAdapter(persistentAgentTestLLM). The:750test ("interrupting the parent while a non-blocking child runs") leaves its backgroundwriterchild running (the CF DO does not cascade the interrupt; the test accepts the child asrunningornot_found), andbeforeEach(:143-144) only clears the queue. When the leftover writer's next steps land in the first milliseconds of the cascade test, it takes that test'sl2-spawn/l2-finresponses, so the mid-level child finishes with no spawn (the:954shape), or it takesl1-spawnand the root gets the wrong script (the:922shape). - Red herring: the workerd "
test-worker.ts changed, invalidating this Durable Object" lines appear in every full-suite run, passing or failing, on both commits (vitest-pool-workers invalidates the previous file's DOs at each file boundary). - Fix (preferred; removes the whole class): key the scripted responses by each test's root session id (child agent ids are
${parent}-agent-${name}, so a prefix covers the tree), so a stray child from an earlier test gets an empty queue instead of stealing. Minimal alternative: the:750test stops (or waits out) its child before it ends.- Done (this MR): both. Responses are keyed by each session's own id, and every test waits for every session it started; see FU-FLAKE-CF-DO-CASCADING-SUBAGENTS-SPAWN-RESULT.
- Inferred related case (not traced; tracked as FU-FLAKE-USAGE-REDIS-TEMPORAL-SHARED-SCRIPT-QUEUE): the
usage-redis-temporal.integ.test.ts"1 call instead of 2" failure (:609) seen under a host load of about 100 is likely the same queue stealing after the previous test's timeout: that file also shares one adapter queue, andbeforeEachonly clears it. That test normally takes about 850 ms of its 30 s budget, and the file passed 3/3 on the branch and 2/2 on main at normal load.
FU-FLAKE-DBOS-CLIENT-TOOL-PARITY-TIMING: DBOS client-tool submit / timeout races in the e2e parity tests — open
Owner: the DBOS bucket (was B8). Origin: !266 (carried forward; referenced in the CI-health table above with no entry until now).
client-tool-happy-parity.integ.test.ts›dbos-postgres: "exactly one submit wins" fails withAgentAlreadyRunningError … (status: active); "emits tool_end with client_tool_timeout error and continues" fails withexpected 'completed' to be 'suspended_client_tool'. CI job 16758483177 (!268, passed on its then auto-retry).- Same family:
client-tool-timeout-dbos.integ.test.tsreturnedsuspended_client_toolinstead ofcompleted(1 of 2 isolated runs on59d47f1e0f). - FU-A2-43 marked the parity test fixed, so this is a recurrence. Suspected: fixed waits around the DBOS client-tool timeout and submit. Not root-caused; it belongs with the DBOS integ work (!259's revival).
FU-FLAKE-LOCAL-LOAD-TEMPORAL-DURABLE-STREAM: local-only flakes under heavy host load — open (no owner)
Origin: !266 (carried forward). Seen locally only, never in CI, at load average 90–138 on 14 cores; both passed standalone:
- Temporal gRPC
Failed to connect before the deadline, only inside a full concurrentturbo run test:integration. - A durable-stream-contract concurrency-timing miss.
Recorded so a future local repro is not re-diagnosed from scratch.
FU-FLAKE-DO-SUPERSEDED-ALSO-COMPLETE: DO hooks.superseded-no-terminal-hook saw ['complete', 'superseded'] — done (test race fixed; product window split out)
Closed by: CI-health-3's flakes-5 MR. Surfaced by: main pipeline 2918408193 (6b9f649c0d), test:integ:rest, job 16973198710 (the first sighting).
- Signature: runtime-cloudflare
do-run-guards.integ.test.ts› "hooks.superseded-no-terminal-hook (DO): a superseded run fires onAgentSuperseded only":expected [ 'complete', 'superseded' ] to deeply equal [ 'superseded' ]. - Test race (fixed): the test worker's
hookLog(andtoolGate) were module-level, shared by every session's DO in the workerd process, and/test/inspectreturned the whole log. A background execution of an earlier test (a wake alarm, an R26 reset, or any run waiting on the shared gate) that completes during this test is counted as the superseded run's own. Forced interleave (scratch): a foreign session held at its model call, released into its gate tool after this test's reset, completes on this test'srelease-gate; the unscoped log then reads["superseded","complete"]. Fix: each hook entry records{ event, sessionId, runId };/test/inspectreturns only the inspected session's events plushookRuns, and the test asserts its single entry belongs to the superseded run (never the intruder). With the fix, the same interleave gives["superseded"]. - Product window (split out): a supersede landing between
onAgentCompleteand the K5 commit makes one run firecompletethensuperseded(runtime-js repro: the agent's ownonAgentCompletemakes an intruder run-start commit). The user ruled "allow both, last wins", like R40's['complete', 'fail']; documented and contract-tested on every runtime by FU-HOOKS-COMPLETE-THEN-SUPERSEDED-WINDOW. - Which one CI hit cannot be told from its log (no run ids). In this test the supersede lands before the gate is released, so the fenced step commit after the gate tool should stop the run before completion; with run ids in the log now, a recurrence names the run.
FU-RESUME-EXECUTION-HOOKS: resume() and retry() take no execution-level hooks, so a resumed tree's children fire only their own — open (for the user)
Raised by: the CP-301 fix (W1). Ruling (orchestrator, 2026-11-14): no new API in CP-301; file this for the user.
- Since CP-301, an agent's own
agent.hooksfire only for its own run. Execution-level hooks (ExecuteOptions.hooks/ExecuteOptions.hookManager) span the whole run tree, children included. ResumeOptionsandRetryOptionscarry no hooks, so an app that observes a tree through execution-level hooks loses them after aresume()/retry(). The resumed or retried run itself, and its children, fire only their ownagent.hooks. Before CP-301 the root's own hooks reached every child, which hid this gap.- The CP-301 direction (user ruling 2026-11-09) expected "a resume carries them to children too". With no field to carry them, the decision is whether to add
ResumeOptions.hooks/hookManager(andRetryOptions), on every runtime, with the same tree-wide rule. - The docs say so plainly: the hooks guide's "Hooks in a child's run" section.
FU-HOOKS-COMPLETE-THEN-SUPERSEDED-WINDOW: a supersede between onAgentComplete and K5 fires both terminal hooks — done
Closed by: CI-health-3's hooks-window MR. Ruling (user): allow both, last wins.
- Terminal hooks precede K5 by design (their customState writes land in it). A supersede in that window ends the run
supersededafteronAgentCompletealready fired:completethensuperseded, the same shape as R40's['complete', 'fail']. The last terminal hook is authoritative. - Verified on every runtime with a contract test that supersedes the run from inside
onAgentComplete(exactly[complete, superseded], both hooks carrying the losing run's id; that run's statussuperseded; the intruder current; and, where a hook can write state, a markeronAgentCompletewrites is not persisted. DBOS terminal hook contexts have noupdateState, so that check does not apply there): runtime-jsk5-failure-hooks.test.ts; the CF DOdo-run-guards.integ.test.ts(a per-session armed side effect in the test worker); CF Workflowscommit-checkpoint-truth.wf-noiso.test.ts(abeforeStoreCallseam before the run's own K5); DBOS and Temporalcommit-checkpoint-truth.integ.test.ts. None produced a baresupersededor the reverse order. - Tracing-langfuse fix: the second closing hook re-emitted the root span as
unknown-agentwith no input (the run's pending data was dropped by the first emission), overwriting the trace's name; this also affected R40'scomplete → fail. Emitted runs now keep their identity in a bounded map for a later terminal hook. Best effort, same process only: when the second hook runs on another worker (Temporal, DBOS, CF Workflows) it still emitsunknown-agent. - Docs: the hook-order table and
onAgentSupersededin the hooks guide and reference, concepts.md §Superseded runs, execution-flow, the tracing guide, the runtime guides and references, hook-type TSDoc and CLAUDE.md (user-approved).
FU-STOPPED-STEP-HOOK-WRITES: a stopped step's hook state writes are kept on some stop paths and dropped on others — done (#181)
Filed by: W1 with CP-242's CF Workflows fix. Owner: WE-2's #181, which adopted the rule below (orchestrator, 2026-10-28) and closes this entry. A model call that a stop cuts short still fires afterLLMCall, and the stopped step itself is not committed. What happens to that hook's updateState today depends on the runtime and on where the stop is noticed:
| Runtime / stop path | The stopped step's afterLLMCall state writes |
|---|---|
| runtime-js / the CF DO, HARD abort | dropped: staged writes dropped and customState rolled back to the last checkpoint (runtime-js/src/run-loop.ts:2190-2212) |
| runtime-js / the CF DO, SOFT interrupt | kept: orphan cleanup only, no rollback; the interrupted K5 saves the in-memory customState (run-loop.ts:2152-2185) |
| CF Workflows, the stop aborts the in-flight call (the step's stop branch; CP-242) | dropped, both kinds: staged under the step, discarded with it |
| CF Workflows, the stop is noticed only at the step commit (CP-59's window: the call returned normally) | kept, both kinds: the completed path's afterLLMCall saves its writes directly, unstaged (invokeHookWithStateTracking without stage), and the refused step does not undo them |
- The adopted rule (#181): a stopped step never commits its state writes, hook or tool, on any runtime or stop kind. Only an interrupted step's partial ROWS persist (an
appendcommit), over the pre-stepcustomState; an abort discards as today. - What #181 changes: CF Workflows stages the completed path's
afterLLMCallwrites under the step (the commit window), and runtime-js / the DO roll a soft interrupt'scustomStateback too. CP-242's stop branch already follows the rule and is left as is. - Closed by #181, with the rule refined by the orchestrator (2026-10-28, option a) so it reads as one rule with O-73: a state write persists iff the row that records its work persists.
- Dropped: the stopped step's
afterLLMCallwrites, and the writes of a call pairedrun_stoppedor in a refused step commit. - Kept: a recorded call's writes.
- Per runtime:
- runtime-js / the DO restore the last checkpoint's
customStateafter an interrupt; - CF Workflows stages the completed path's
afterLLMCallunder the step and neutralises the unrecorded keys of a step committed under a stop (RM-69); - Temporal's stop K5 never carried them (they ride the refused commit's state);
- DBOS hooks write no state.
- runtime-js / the DO restore the last checkpoint's
- Each runtime has a red per stop kind.
- Dropped: the stopped step's
FU-FLAKE-CF-CONCURRENT-SUBAGENT-FOLD-TIMEOUT: CF D1 concurrent sub-agent fold test timed out at 60 s — open
Owner: CI-health-3 (root cause after the DBOS measurement). Surfaced by: main pipeline 2913857752 (17ebc280ce), test:integ:rest job 16940995440 (amd64 k8s), not retried.
- Signature:
runtime-cloudflareconcurrent-subagent-fold.integ.test.ts› "concurrent sub-agent customState fold (D1 integration) › composes all 16 parallel afterSubAgent folds (append)":Test timed out in 60000ms. - The first such failure in about 12 failed
test:integ:restjobs. A 16-way parallel fold on D1 running out of 60 s may be a real serialisation or contention problem in the fold path, not just a slow runner: root-cause before calling it load. - CI-health-3 analysis (2026-10-06), from job history: not a hang. In the failing job (amd64 k8s, whose trace carries "Insufficient memory" scheduling notes) the file took 334 s;
(modify)passed at 56 979 ms and(append)timed out at 60 316 ms. In 8 passingtest:integ:restjobs (Intel DigitalOcean runners, 2026-10-05/06) the 6-test file took 106–257 s:(modify)17.8–46.6 s,(append)15.8–42.5 s, the half-failing wave 14–38 s. The tests are intrinsically slow, vary about 3×, and run with a thin margin. - A second defect: the timed-out test's workflow kept running. Its mock
step.doretries threwD1 state getCurrentRun failed … TypeError - fetch failedwhile the next describe ("removal …", with its own runtime context) ran, so a timed-out run keeps driving D1 into later tests. - Profile (local, idle host; a counting proxy on the D1 binding, scratch, not committed): each test makes 405–420 D1 calls whose latencies sum to 57–83 s inside a 6–8 s wall clock: the 16-way wave's calls queue behind one another through Miniflare's proxy, about 150 ms each. Under CI load every round trip slows and the wave stretches linearly. Per child, the PARENT's full state and history were read twice:
steps.tsrecordSubSessionMessage→loadAgentState(state, every message, refs, interrupt flag) and corerecordSubSessionResult(state, refs, and every message again, to look for onetoolCallId). The D1 store's append commit, first counted here as a third read, reads the state row and runsCOUNT(*)s but no message payloads (corrected by FU-SUBAGENT-RESULT-RECORDING-RELOADS-PARENT, which fixed the two reads). That was O(children × history) per wave in production too. - Leak fixed (!306): every test bumps a module-level counter, and a mock step (
do,waitForEvent) created for an earlier test throws aNonRetryableError, so a timed-out run stops at its next step. Demo (scratch): test A starts a slow 4-child wave and returns without awaiting it; while test B runs, D1 calls bound to A's session went from 106 to 7. A later batch made the guard re-check before every retry attempt too (a step in flight kept retrying against a cleared database: in a demo, 5 attempts after its test ended before, 0 after). The 7 that remain are unexplained (they are not through the mock'sdoorwaitForEvent; likely the stale run's own failure path, not verified). - Load cascade (main pipeline 2918595891, job 16974699413, on the k8s runner with 3 pipelines in flight): 4 of 6 tests timed out (60.3–64.2 s; the half-failing wave passed in 55.6 s, 14–38 s on the Intel runners). The first test ran past 60 s; the next test's
beforeEachcleared the tables while its run was mid-step, and that run's retries (parent session not found,loadState failed×5,fetch failed) loaded D1 during the next test. Not the guard: a guard refusal fails a run at once, never at the 60 s timeout, and a local A/B of the file before / after the guard ran 26.1 / 26.0 s. - Product cost (done): the per-child re-reads (FU-SUBAGENT-RESULT-RECORDING-RELOADS-PARENT): a 16-child wave makes 324 D1 calls (was 404), and a long parent history no longer adds calls per child. The timeout itself stays at 60 s.
- Another occurrence (2026-10-08, !319 pipeline 2927780203, head f65868ffa5,
test:integ:restjob 17040664161, DigitalOcean runner, 1 060 s, ~19:09Z): "composes all 16 parallel afterSubAgent folds (modify)" timed out at 60 451 ms (1 failed / 33 passed files, 1 / 494 tests). Mock adapters; !319 does not touch this path. Retried by the orchestrator under DoD §A3. - Recurrence (2026-10-10, !337 pipeline 2933981207, head 041db0a68d,
test:integ:restjob 17083529197, Intel DigitalOcean runner, 1 385 s against main's ~545 s): the file took 654 s, and 4 of its 9 tests timed out. The 60 s timeout hit "folds every child once — failures included — when half the wave fails" (60 180 ms), "a transient failure on a failing child's record step (attempt 1) …" (60 395 ms), and the mixed-step "preserves BOTH the committed regular-tool write and the sub-agent folds" (60 761 ms). The 240 s timeout hit the wave-cost test "recording a child's result costs the same D1 calls with a short or a long parent history" (242 496 ms). Both 16-way(modify)/(append)folds passed (39.5 s / 31.4 s). Same job as FU-FLAKE-TEMPORAL-CONCURRENT-SUBAGENT-FOLD-TIMEOUT, under hypervisor CPU steal on the shared droplets (see that entry). Retried by !337 under DoD §A3. - Recurred 2026-10-10 in !334 (W1 MR2a) pipeline 2933538936,
test:integ:restjob 17080790035, at 13:32Z.- The job ran on the DOKS amd64 k8s runner (
gitlab-runner-7d7f8779c7-4sndc,ynNez3uig). Its trace has 33Unschedulable: "0/9 nodes are available: 1 Insufficient memory …"notes. - Three tests timed out against the 60 s limit:
(modify)at 60 485 ms, failed-child at 60 235 ms, mixed-step at 62 113 ms. - Main's last three pipelines (2932802793, 2932599552, 2932088347) ran this job green on the Intel DigitalOcean runners. Locally on the MR2a head the file passed 9/9 in 67.6 s.
- Retry: 17081308216.
- The job ran on the DOKS amd64 k8s runner (
FU-FLAKE-C3-HITL-INTERRUPT-BARRIER: concurrency-invariants-stress C-3 resumed before the suspension committed and lost typed under CP-96 — done (W1 MR2b)
Owner: WD (the test's scenarios; fixed in W1 MR2b Task 9, orchestrator ruling). Surfaced by: main job 17083046907 (main 5ac3ac53b8), js-postgres, retried as 17083490504. Second sighting: main pipeline 2936204106 (after the batch 19 merge), job 17095324029, js-postgres, same signature (AgentAlreadyRunningError, suspended_client_tool) — main does not carry the fix until W1 MR2b lands.
- Signature:
packages/e2e/src/__tests__/concurrency-invariants-stress.integ.test.ts› C-3 (HITL interrupt from another process): theresume({ mode: 'continue' })rejected withAgentAlreadyRunningError(suspended_client_tool) instead of observing the interrupt. - Cause (the test's barrier, not a runtime defect): C-3 waited only for the pending entry. runtime-js registers the pending call BEFORE its suspension commit (the JS order: frames → commit → status → pause), so a fast
resume()entered while the suspending run was still live, and under CP-96 the store refused that entry (runtime-js/src/run-start.ts:263-265, the seen-run loser). Not caused by W1 MR2a (no runtime-js or store-postgres product change there). - Fix: C-3 and C-5 now wait for the committed suspension, pending AND stream
paused, the barrier C-1 already used (R46: the pause follows the commit on every runtime). C-2 and C-4 were checked: noresume()follows their barrier. Not red-first: a timing race (the pending-only barrier passes most runs); the evidence is the reasoning above plus the main job. Proof: MR2b's final matrix ran the file at--retry=0on every lane it runs, green (concurrency-invariants-stress.integ.test.ts, 19 passed, 0 failed).
FU-FLAKE-C5-JS-REDIS-CONTINUE-FAILED: concurrency-invariants-stress C-5's continue ended failed on js-redis: C-4 left a CP-297 continuation on the shared adapter — done (fixed in batch 20; owner)
Owner: conformance owner. Cause class: harness (a test that did not settle a continuation it caused), exposed by CP-297's post-suspension re-check (6c3d27878d, batch 17).
- Signature:
packages/e2e/src/__tests__/concurrency-invariants-stress.integ.test.ts› v7 concurrency invariants — 'js-redis' › "C-5: CAS-on-version absorbs parallel submissions — no lost updates". Both parallelsubmitToolResultcalls were accepted and landed durably, thenresume({ mode: 'continue' })endedfailed, notcompleted. - Seen: four times. !810's MR pipeline 3047618338, job 17882570650 (
test:integ:e2e 3/4), at 4a 27902c18a0 merged with main 7b92366025 (before the self-report printed a cause; unprinted); !825's pipeline 3051412625, job 17911506965; !829's pipeline 3051613726, job 17913083753; !831's pipeline 3051714620, job 17913832536 (test:integ:e2e 3/4, 1 failed, 823 passed). The last three ran main's stress file (before this fix) and printed the same cause:MockLLMAdapter: No more responses configured. Call count: 125 (3 since the last clearResponses()). - Arithmetic: js-redis expects C-1 20 + C-2 100 + C-3 1 + C-4 1 + C-5 2 = 124 model calls; 125 is one extra. It fell inside C-5, after its first step: an earlier extra would have taken C-5's suspending step, and C-5 would have waited out its 90 s waitFor instead of failing in 104 / 129 / 111 ms.
- Cause: C-4's cleanup submitted its parked call's answer and returned, without continuing or awaiting the session. Its waitFor polls pending every 5 ms, so on Redis the answer often lands before the run has parked (its status and stream pause follow the suspension commit). CP-297's re-check then finds the answer owed, decides
caller_resumesand continues the C-4 session itself, as designed (pinned by runtime-jsowed-recheck-after-suspension.test.ts, "an answer stored inside the suspension"). That continuation's model call, on the shared adapter, took C-5's__finish__, and C-5's own resume found the script empty. - Ruled out (trace at 41ea410715): a duplicate continuation of C-5's own session. The re-check and the caller's resume are one continuation under every interleaving: the run-start CAS, the adopt of a live owed continuation, the owed-run gate and the completed refusal (
js-agent-executor.ts:7029-7041, :7105-7137, :7231, :7399-7405), pinned byowed-recheck-after-suspension.test.ts:395 / :414 / :454. - Witness (record-only, lock-free, at main 4e26f6cb0b, whose dists the tree had built; CP-297 present): a scratch runtime-js unit replaying C-4 then C-5 on one adapter. (A) The CI interleaving forced (C-4's pause held until its answer was stored; the C-4 session's later calls held until C-5's first was served): calls since C-5's clear were C-5 step 1, the C-4 session's continuation (an owed-marker run, completed, it took
__finish__), C-5 step 2 exhausted; C-5's resumefailedwithNo more responses configured. Call count: 4 (3 since the last clearResponses()), the CI message. (B) Control, C-4's cleanup settling its continuation: C-5 completed with its two calls. - Fix (test-only): C-4 queues a
__finish__for its winner and its cleanup settles the continuation its answer causes: it resumes (a winning resume, or the adopt of a live re-check continuation), or, refused, waits for the continuation the runtime started, and asserts itcompleted. C-5 self-reports every model call made while it runs (session id, step, message count, caller frame), in its status assertion's message only; it reads no store. - Same shape elsewhere: none on runtime-js. Every other e2e test that answers a paused session settles it in the same test, or answers a session nothing continues (
unknown_tool_call, late).stop-recorded-wins-cfw-engineshares one adapter across its cases (:115, lane inbeforeAll:119-120); its twonew_runcases (:218,:271) await the__submit__instance's close (closedOutput) and assert no model call before they end. - Status: fixed in batch 20 (the C-4 settle and the C-5 self-report); closes on the batch-20 pipeline's js-redis
test:integ:e2eshard. Should that shard fail on C-5 again, the self-report names the calling session and this entry reopens with that record. - Closed: batch 20's MR !838 pipeline 3052793322 (
/merge), job 17922754390 (test:integ:e2e 3/4):concurrency-invariants-stressran on js-memory, js-redis and js-postgres (C-5 skips only Temporal and DBOS) with no failure, and the C-5 self-report printed nothing; the job's e2e tests: 817 passed, 27 skipped (one CI pass).
FU-FLAKE-CONF-APPROVAL-DISABLED-STOCK-MIDFLIGHT-READ: tools.disabled-approval-refused-client-stock's completion poll reads cf-do persisted history mid-flight — done (!885, batch 26; owner: W1)
Filed: 2026-11-11 (owner, from WE-2's slots w-rm131f2 and w-cp320e, cf-all ledgers).
- Seen: three times,
tools.disabled-approval-refused-client-stock|cf-do|clientcrashed withshort read … paged 5 messages but getMessageCount=4 (consistent across a re-read), both in a full cf-do conformance ledger: WE-2's slot w-rm131f2 (the same cell passed oncf-do+ai-newestthere), and WE-2's slot w-cp320e (2026-11-11, at the CP-320 cell head; the same cell passed in slot w-cp320d at the same fix code). Every other row of both ledgers matched. Seen a third time: W1's slot s-ht (2026-11-15, the hidden-turn page-race fix 6eb19928d0 over origin/main e93ffcc545): the same cell crashed in.run-completeswith the identicalshort read … paged 5 messages but getMessageCount=4 (consistent across a re-read); the js-chat-host cell passed in the same run, and every other row matched. - Cause: the scenario's
.run-completespoll (packages/e2e/src/conformance/scenarios/add-context-app-client.ts, thesettlePollafterapproveTool) callspersistedOfwhile the approved run is still committing, with no catch. On cf-do a persisted read is one request per store call, so a step commit betweengetMessageCountand the pages tears it, andreadPersistedthrowsShortReadErrorwhen the re-read tears too. The authoring doc forbids this ("cf-do persisted reads are several requests, not one snapshot": observe after a namedsettle()sees the run finish, or while it is held). A test defect, not a product one. - Wanted: the poll treats a
ShortReadErroras "not yet" (as the scenario's pending poll already does), or polls the run's status instead of persisted history; then a lock-free check of the cell on cf-do. - Closed: the
.run-completespoll now reads the persisted observation itself and treats aShortReadErroras "not yet" (counted, and named in the poll's failure text), any other read failure still crashing the cell, until the run reads completed and the session reads completed (packages/e2e/src/conformance/scenarios/add-context-app-client.ts:422-443, at origin/main bd30fa746b, !885 in batch 26). The cell's persisted-history checks are unchanged and read once, after the run ends.
FU-TEST-MOCK-ADAPTER-INPUTS-UNBOUNDED: a long-lived MockLLMAdapter keeps every input, and each input keeps its step's state copies alive — open (test infra; owner: the conformance owner)
Filed: 2026-11-10 (owner, from WA's Save-First bench trace, slot-sfthn; orchestrator ruling: no product record).
- What:
MockLLMAdapterrecords everyLLMGenerateInputit receives (core/src/llm/mock-adapter.ts:555,this.allInputs.push(input)), and onlyclearInputs()empties the record.ScriptedModelandRoutingMockLLMAdapterinherit it. Each input'scallbackscloses over its step's scope, the per-call state copies included, so a mock that lives across many steps retains about 3·k·S + S bytes per step (k = calls per step, S = state size; WA measured it linearly, r² 1.00). - Seen: WA's bench Worker held one
ScriptedModelper isolate, so a deployed CF DO's isolate reset on its memory limit after ~30 steps at 1 MB state. The product holds no input past its call (traced at origin/main ae4c3d8040: llm-vercel's per-call cache is aWeakMapkeyed by tool,vercel-adapter.ts:309; tracing-langfuse keeps closed runs bounded,MAX_CLOSED_RUNS,stateless-span-emitter.ts:106,:845-850; runtime-js removes its abort listener in the loop'sfinally,run-loop.ts:3106; core'sholdLLMErrorReportholds only the call's own error in a per-call closure,report-llm-error.ts:114-127). Trace: WA's slot-sfthn heap snapshots at steps 10 and 30 (k1, 1 MB state), their diff and retainer paths, and its measurement table (kept locally). - Control (WA's slot-sfthc): the same seven shapes with the mock's
clearInputs()at every committed step retained −0.05 to +0.02 MB per step (it was 2.5 to 10), and the step 10 → 30 heap diff kept its large strings flat (25 → 25, against 57 → 137). The mock's input record was the only per-step holder. A residual of 3 to 8 MB above each shape's baseline after dropping every reference is neither per step nor state-proportional (the 100 KB, 200-step shape has the largest); WA's slot-sfths ran ten sessions back to back in one process (each on a new executor, model and streams): after two warm-up sessions about 2 MB higher, the dropped heap rose 0.129 and 0.140 MB per session over sessions 3 to 10 (r² 0.93), under the 0.5 MB/session bound for a one-time offset. Its session 2 → 9 diff (+0.5 MB) is existing array buffers growing (count unchanged; buffer pools, likely the Redis client's) and compiled code tiering up, with no object, map or large-string growth, so it is read as runtime warm-up, not run data. A plateau was not shown within ten sessions. - Wanted:
- the mock keeps a bounded record (for example the last N inputs), or a harness that re-scripts a shared mock also calls
clearInputs()(the e2e harness's per-backend adapter clears only its responses inbeforeEach); - done in this commit:
docs/llm/custom.mdsays, under "Best Practices", that an adapter keeping whole inputs after its call keeps each step's state alive throughcallbacks, and, under "MockLLMAdapter for Testing", that a mock shared across tests callsclearInputs(). The entry stays open for the mock change only.
- the mock keeps a bounded record (for example the last N inputs), or a harness that re-scripts a shared mock also calls
FU-TEST-REMOTE-PARITY-DUPLICATE-TITLES: runtime-js's remote parity file registers its 11 tests twice under identical titles — open (LOW, test only; owner: the conformance owner)
Surfaced by: W1's CP-301 fix slot (2026-11-14), checked by the owner at origin/main 3fee8e37bf. packages/runtime-js/src/__tests__/remote-parity.integ.test.ts calls core's remoteConsumerParityTests twice (:87, the JSAgentExecutor setup, and :187, the hand-written loop), and the battery opens one fixed describe('remote consumer parity') (core/src/testing/remote-consumer-parity.ts:311). The setup's name comes from an async factory, so it is not in the title. Every test therefore appears twice under the same full title (for example "remote consumer parity > D5: a paused end frame never folds as success-null"), and a run judge keyed by file and title collapses the twins: W1's judge counted 2237 where vitest counted 2248. A judge without a count check could hide a failing twin behind a passing one.
Fix: give each call a distinct label, either a label argument to remoteConsumerParityTests used in its describe, or a describe(<setup name>) around each call. Judges should also count from vitest's own list and fail on a duplicate title.
FU-TEST-STRINGIFY-CONTENT-SUBSTRING-ASSERTIONS: 29 test assertions match a substring of a row's JSON-serialised content instead of its decoded value — open (LOW, test only; owner: the conformance owner)
Surfaced by: WA's DBOS O-73 slot slot-b2 (2026-11-15). Its new integ test asserted an approved tool's result with expect(JSON.stringify(row.content)).toContain(<a needle holding quotes>). The row was right, but a string result is stored verbatim (helixContentEncoding: 'text') and JSON.stringify escapes its quotes, so the needle never matched and the test read red over a correct product. WA rewrote that test, and the new loop units, with core's decodeToolResultContent and toEqual before the slot was re-run.
Where: git grep -nP 'JSON\.stringify\([^)]*[cC]ontent[^)]*\)\)\.(not\.)?toContain' at origin/main 9ca43ae3ee finds 29 such assertions in 14 test files (e2e 6 over 4 files, runtime-js 17 over 7, runtime-cloudflare 2, runtime-dbos 1, runtime-temporal 3; three of them, interrupt-resume-subagent.integ.test.ts:764 and storage-efficiency.integ.test.ts:397-398, read a user message's content, not a tool result's). None is wrong today: no needle holds a character JSON.stringify escapes, so each passes for the reason it means to.
Why it matters: the assertion checks the serialised envelope, not the value. A needle that gains a quote, a backslash or a newline turns a .toContain red over a correct row (the slot-b2 failure), and turns a .not.toContain (three today, each 'runtime_restarted') into a check that can never fail. Short needles ('42', 'two', 'sent', 'name') also match key names and unrelated text in the serialised form.
Fix: decode with decodeToolResultContent (core) and assert with toEqual on the value, or on the field that carries the text (expect(decoded).toEqual({ error: … })); for a user message, assert its text part. Then a lint rule, or a guard test in the style of no-derived-child-ids.test.ts, that refuses JSON.stringify(<…content…>) inside an expect(…).
FU-CONF-LEFTOVER-SCRIPTS-EXPECTED-SENTINELS: verify:lanes lists every leftover scripted response, the intended sentinels included — open (LOW, test infra; owner: the conformance owner)
Filed: 2026-11-16, from batch 27's conformance:verify:lanes report (job 18056357964). Its informational section "Cells that left scripted responses unconsumed" ran to about 1000 lines. Many are intended: a cell that asserts the model is NOT called again scripts a sentinel response it must never reach (for example completionguard.output-contract-violation's 'NEVER CALLED', left once on every lane, its passing state), so its leftover count is part of the claim, not noise. A stray leftover, a real hint that a cell scripted a step its runtime never took, is lost in that list. Wanted: a scenario declares its expected leftover per cell (the sentinels it scripts), the cell records it beside leftoverScripts, and verify:lanes lists only cells whose leftover differs from the declared count. The section stays informational, never a gate.
FU-FLAKE-CORE-SCRIPTED-MODEL-BARRIER-WINDOW: the ScriptedModel delivery-barrier test assumed a 100 ms window — done (owner batch 19)
Owner: conformance owner. Surfaced by: !338 gating pipeline 2934478005, test:unit job 17086002971 (the shared local arm64 runner, 382 s), retried as 17086592283 under DoD §A3.
- Signature:
corescripted-model-hold-after-deltas.test.ts› "a delivery barrier is awaited (with the call's input) before the call counts as held":expected [] to have a length of 1 but got +0(theseencheck afterwhenHeld('g', 100)). - Cause: the test (owner commit 27523d5900) let
whenHeld('g', 100)reject and then assumedgenerateStephad already entered the delivery barrier. On a loaded runner the call had not reached the barrier within those 100 ms. A wall-clock assumption in the test, not a ScriptedModel defect. - Fix: the barrier resolves an "entered" promise; the test awaits it, then asserts
whenHeldstill rejects while the barrier is closed. Red first: with a 200 ms delay before the barrier loop inScriptedModel(working tree only, reverted), the old test failed with the CI signature and the new one passed (26/26).
FU-FLAKE-CF-DO-REFRESH-C1-HITL-CONTROL-LIVE-RENDER: a CP-53 sighting — the DO's live chat render lost the step after a client-tool answer (no refresh) — done (WA MR1)
Owner: WA. Surfaced by: !338 gating pipeline 2934478005, test:cloudflare job 17086002986 (the shared local arm64 runner), retried under DoD §A3. The first sighting: the four earlier failed test:cloudflare jobs failed on other tests. !338 touched no product code, so this ran main's DO code.
- Signature: e2e
refresh-c1-cf-do.cf.test.ts› "ai 6.0.49: control hitl-continuation (no refresh): the live render equals the final snapshot" (1 of 301). The live render of the assistant message ended at "I need your favourite colour before I can finish this up."; the final snapshot continues "…Thanks, teal it is; I am recording that and wrapping up now." Both have theask_colourclient tooloutput-available. - Root cause (WA, from source): CP-53's mechanism on main. The DO answers the client-tool submit before the continuation's entry starts; the handler's
/resumethen gets 409AgentAlreadyRunningErrorwhilestillPendingis still true, andrejectedOnlyResponseanswers 200 SSE with onlydata-resume-rejected. The client endsreadywith no error and no step-2 text while the server finishes the run. - Proof (WA): a test-only file holding the
afterToolhook for 3 s on the answered call inside runtime-jsdrainOrphanClientToolCalls(after the run-start commit and the result append, before the pending entry is cleared): red 3 of 3 on mainf12eef982d, green 3 of 3 on WA MR1a22ccc7893; an unforced ×10 alternating A/B was green on both (a rare race). - Closed by WA MR1 (R69 / CP-53: the DO awaits the continuation's entry before answering a submit). WA MR1 also makes the e2e C-1 settle wait fail on a
data-resume-rejectedpart (b3e59cba13). Recorded in CP-53 as arelated:line; no new record.
FU-FLAKE-D1-USAGE-STORE-CONTRACT-ROW-COUNT: the real-workerd D1 usage contract loses rows to a concurrent file's table clear — done (test isolation + a wall-clock wait; W1 MR3)
Owner: W1 MR3 (taken over from WF, unstaffed, 2026-10-12). Surfaced by: !338 gating pipeline 2934478005, test:cloudflare retry 17086630864 (the shared local arm64 runner, 295 s, host load ~21); the first attempt of the same job passed store-cloudflare (313/314). Filed by the conformance owner.
- Signature: store-cloudflare
usage-store-contract.cf.test.ts(D1UsageStore, real workerd) failed 9 of 30 tests, each reading too few rows:queryUsage› "cross-session query without sessionId filter returns all entries" (expected 1 to be 3), "filters by userId" (expected 1 to be 2), "filters by kinds", "supports pagination", "supports combined filters", "sorts by sessionId descending", andgetRollupremote rollup (b), (g), (h) (totals off). - Mechanism (from the job's timestamps): the files ran CONCURRENTLY although
vitest.cf.config.tssetsfileParallelism: false:usage-store-contract(5.9 s) finished at 20:07:15, insidestate-store-contract's run (41.4 s, finished 20:07:34) andd1-guarded-commit-exhaustion's (42.8 s, finished 20:07:34). WithisolatedStorage: false, every file uses the oneenv.AGENT_DB, andstate-store-contract.cf.test.ts'sclearD1Tables(itsbeforeEach) runsDELETE FROM __agents_usage. So a state-store test starting mid-way through a usage test wipes the rows the usage test just wrote. It is not a time-range filter (the failingqueryUsage({})has none) and not an id collision (recordEntryids are random UUIDs, plain INSERT). - Confirmed (W1 MR3):
@cloudflare/vitest-pool-workers0.9.14 ignoresfileParallelism: falsewhenisolatedStorage: false: it starts one runner worker per file over one Miniflare and runs them all at once (dist/pool/index.mjs). Fix:poolOptions.workers.singleWorker: true(store-cloudflare and memory-cloudflare), which runs the files one at a time in one worker. Red: the setup filehelpers/cf-file-serial-guard.tsholds a one-row lock in its own D1 database (TEST_FILE_LOCK_DB) for each file's run and fails a file that starts while another holds it; on the old config it failed "d1-run-start-owner-pin.cf.test.tsstarted whileworkerd-timer-clock.cf.test.tswas still running". The usage table stays in the state-store clear: the contract's cascade cases write usage rows. - A second signature with a different mechanism: !349 job 17124184867 failed
state-store-contract.cf.test.ts›listSessions› "should filter by dateRange on updatedAt field", two sessions in its ±1 ms window (expected [ …, … ] to have a length of 1 but got 2). No other file wrote toAGENT_DBthen. workerd measures a timer from the start of the event that set it, and a D1 call does not move that base: after the isolate stalls 10 ms or more past a timer (a loaded host),setTimeout(10)resolves withDate.now()advanced 0-1 ms (measured: stalls of 10 / 12 / 20 / 40 ms → 0 / 0 / 1 / 0 ms), so one session's insert and another's update share a millisecond. Red: aD1StateStorewhosecreateSessionfirst holds the isolate 12 ms fails thelistSessionscontract 3/3 runs (that case, andupdatedAtasc / desc). Fix: core/testingelapseWallClock(ms)re-arms untilDate.now()movedms; everylistSessionswait uses it (green 3/3 under the same stall).workerd-timer-clock.cf.test.tspins the helper (red with the helper as a baresetTimeout). Its canary on the workerd behaviour itself was withdrawn: it asserted a timing race (whether the clock moves 10 ms during a 10 ms timer depends on host scheduling), and the train runner saw exactly 10 (pipeline 2959489233, job 17260988687). - Not fixed here: the two CF example suites have no serial setting and one
.cffile each: FU-CF-EXAMPLES-POOL-SERIAL-FILES.
FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT: a .cf file after a long D1-heavy file fails to load on an RPC timeout, and the run silently dropped every later file — fixed in code (the silent drop by the every-file-ran reporter; the timeout by a patch of vitest's worker RPC timeout inside workerd, pending landing)
Owner: WE-2. Surfaced by: !527's MR pipeline (test:cloudflare job 17455337316) and !533's train (job 17455846143). Train cars were dropped.
Fixed by (pending landing): a pnpm patch of vitest 2.1.9's worker RPC timeout, patches/vitest@2.1.9.patch (listed in pnpm-workspace.yaml's patchedDependencies).
Cause (traced in the installed code): vitest 2.1.9's
createRuntimeRpc, the worker RPC that@cloudflare/vitest-pool-workers0.9.14 runs inside workerd, passes notimeout, so birpc's 60 s default applies, and no pool or vitest option reaches it. Its timer runs on workerd's clock, which stands still while a test only awaits D1 and jumps on the next incoming event, so a call armed in such a stretch times out early with its reply still on the way.The patch: inside workerd only (
navigator.userAgent === 'Cloudflare-Workers') the worker RPC times a call out after 10 min; every Node pool keeps vitest's 60 s. vitest 4.1 arms no worker RPC timeout at all (timeout: -1).Guard:
packages/runtime-cloudflare/src/__tests__/pool-workers-rpc-timeout.test.tsruns thecreateRuntimeRpcthat pool-workers resolves in a child Node process and reads the delay one call arms: 600 000 ms as workerd, 60 000 ms as Node. It pins vitest 2.1.9, and pnpm refuses a patch it cannot apply, so a vitest bump cannot keep the patch by accident. Every worker-to-Node call (onTaskUpdate,resolveId,fetch, …) goes through that one birpc instance and its one timeout.Witness (local slot, 2026-11-19): store-cloudflare's real
vitest.cf.config.ts, a scratch file with one 70 s D1-only test, then a second file; judged only from Node-side outcomes (exit code, the files that ran,Timeout calling, WALL-TIMELINE's wall-clock gaps). Two variants: the test writes to the console; or it writes nothing and ends with a cold dynamic import.- Unpatched: the console variant 8 of 8 red (
Timeout calling "onTaskUpdate"7,"snapshotSaved"1; the second file never ran). The other variant 8 of 8 red: 5Timeout calling "onTaskUpdate", before the import ran, and 3 of the AggregateError shape below. - Patched: 16 of 16 green (8 + 8) across message-free stretches of 71.7 to 81.8 s, with no RPC timeout and no AggregateError.
- The patched value set back to 60 s: 6 of 6 red (5
Timeout calling "onTaskUpdate", 1 of the AggregateError shape). - Inside the isolate, the console variant measured the timer clock 69 990 to 73 684 ms behind wall time.
- The AggregateError shape: the runner stops on an empty
AggregateErrorfrom pool-workers'waitForWaitUntil, with noTimeout callingprinted, and EVERY-FILE-RAN names both files. pool-workers registers every worker RPC promise and throws oneAggregateErrorfor two or more rejections without printing the errors inside, so the shape names no call. It appeared only under the 60 s timeout (4 of 11 such runs of that variant, 0 of 8 patched), and the mutation changes nothing but the timeout's value, so the controlled mutation attributes it to this mechanism; no message does. - With the patch,
test:cloudflare,test:cloudflare-request-signal,test:workflows, e2etest:cfw-engineand workspaces-showcase'stest:cfall passed (noTimeout calling, no EVERY-FILE-RAN failure). The Nodeturbo run testcounts were unchanged except runtime-cloudflare's +1 file and +1 test (the guard).
- Unpatched: the console variant 8 of 8 red (
What is shown and what is inferred: the shown victims are
onTaskUpdate(job 18170434388 below, and the local runs) andsnapshotSaved(local). The victims at a file's load (resolveId,fetch,resolveSnapshotPath: jobs 18081809031, 18154337356, 18156791062) go through the same worker RPC and are inferred to be the same mechanism; no local run has shown that class red.Upstream path: vitest 4.1 with
@cloudflare/vitest-pool-workers0.15 or later (which needs vitest ^4.1) has no worker RPC timeout. That is a repo-wide vitest 2 to 4 migration (the package configs, the two reporters' hooks, the pool config API), its own project; the patch is dropped with it.CI witness after landing: no
[vitest-worker]: Timeout callingand no EVERY-FILE-RAN failure intest:cloudflareon patched builds; a passing run with a WALL-TIMELINE stretch of 60 s or more is the positive sighting.Signature: store-cloudflare
d1-settle-fence-order.cf.test.tsfailed to LOAD:[vitest-worker]: Timeout calling "fetch"(test-worker.ts) or"resolveId"(cf-file-serial-guard.ts). It was the file that ran right afterstate-store-contract.cf.test.ts, which took 103 s and 213 s there (19 s and 58 s in green main runs; vitest's worker RPC times a call out after 60 s). Then came "Error running worker", and the summary readTest Files 1 failed | 2 passed (3). The run has 8 files; the other 5 never ran, and nothing named them.The silent drop (fixed):
scripts/vitest-every-file-ran.mjs, a reporter wired into all 11@cloudflare/vitest-pool-workersconfigs. A file vitest planned to run (onPathsCollected) with no result inonFinishedfails the run with exit code 1, and the reporter names it. A file that fails to load still counts as having run (it is reported as FAIL). Red:core/src/__tests__/repo/vitest-every-file-ran-run.test.tsruns a child vitest on a pool that never runs its specs; before the reporter that run exited 0 with "Test Files no tests".The timeout (fixed: see "Fixed by" above; the investigation as it stood): measured locally (vitest 2.1.9,
@cloudflare/vitest-pool-workers0.9.14, Miniflare; 2026-10-26):- It is not the file itself:
cf-file-serial-guard.tsis asetupFilesentry, so every.cffile loads it. - Inside workerd, the clock vitest's timers use (
performance.now()) does not move while a test only awaits D1: over a 70 s D1-only testDate.now()moved 70 000 ms andperformance.now()10 ms, the lag growing one-for-one with wall time. An incoming event (an RPC reply) brings it level; a 0 ms timer does not move it. - vitest's worker RPC (birpc) arms a 60 s timeout per call on that clock. Under the real
vitest.cf.config.ts, one 70 s D1-only test followed by a second file failed 8 of 8 times when the test wrote to the console (Timeout calling "onTaskUpdate", the runner died, the second file never ran) and passed 5 of 5 times when it wrote nothing. - Two in-isolate fixes failed their green 3 of 3 and are parked (WE-2's local branch
reliability-we2-cf-workerd-clock-catchup):- a catch-up timer in per-test hooks: the console flush arms its RPC inside the test body, before any hook;
- a 1 s heartbeat timer per file: timers do not fire during a D1-busy stretch at all (the clock they wait on is frozen).
- Not established: that CI's failure is this shape. Its long file was the state-store contract (477 short tests, 103 s and 213 s), and each test's task update should bring the clock level, so the ≥ 60 s stretch with no incoming event has not been located.
- Instrumentation:
scripts/vitest-wall-timeline.mjs, a reporter in all 11 pool-workers configs, stamps each worker message with Node's wall clock and prints the largest gaps (WALL-TIMELINE (<config>): …). The worker's own durations cannot show them (they run on the frozen clock: a 70 s test reports 15 ms). test:cloudflare, test:workflows and test:cfw-engine keep the full timelines (packages/*/.vitest-reports/,examples/*/.vitest-reports/) as artifacts. The next failing run locates the stretch, to be fixed at its source. - Also worth checking: a newer
@cloudflare/vitest-pool-workers/ vitest whose worker RPC timeout is configurable.
- It is not the file itself:
Recurred (2026-11-16, batch 28's train, pipeline 3075668637, test:cloudflare job 18081809031): a different victim file, same mechanism: d1-guarded-commit-exhaustion.cf.test.ts (not d1-settle-fence-order.cf.test.ts) failed to load (Timeout calling "resolveId", cf-file-serial-guard.ts) right after state-store-contract.cf.test.ts (103 s); Test Files 1 failed | 1 passed (2), and the train car was dropped. The store-cloudflare WALL-TIMELINE's largest gaps: 166.4 s from [init] to the contract file's start, then 45.0 s, 33.7 s and 31.4 s inside it.
Recurred twice on !929's MR pipelines (2026-11-18): the same victim file as on 2026-11-16, d1-guarded-commit-exhaustion.cf.test.ts, failed to load right after state-store-contract.cf.test.ts. Both runs read Test Files 1 failed | 1 passed (2) and Tests 654 passed | 6 skipped (660), EVERY-FILE-RAN named the 6 files that never ran, and no assertion failed.
- Pipeline 3085375830,
test:cloudflarejob 18154337356:Timeout calling "resolveSnapshotPath"and"fetch". store-cloudflare tests 103.77 s, wall 389.30 s. WALL-TIMELINE's largest gaps: 229.1 s from[init]to the contract file's start, then 58.4 s, 45.8 s and 41.2 s inside it. - Pipeline 3085692378,
test:cloudflarejob 18156791062:Timeout calling "fetch". store-cloudflare tests 92.97 s, wall 271.87 s. WALL-TIMELINE's largest gaps: 148.5 s from[init]to the contract file's start, then 48.2 s, 34.8 s and 25.4 s inside it. - The retry passed:
test:cloudflarejob 18160013864 (pipeline 3085692378) took 553.5 s: store-cloudflareTest Files 8 passed (8),Tests 722 passed | 6 skipped (728), tests 101.94 s, wall 199.52 s, the contract file 79 738 ms. WALL-TIMELINE's largest gaps: 83.8 s from[init]to the contract file's start, then 29.9 s. - Observation: the three failing jobs (18081809031, 18154337356, 18156791062) took 954.6 to 1096.9 s. The 13 green
test:cloudflarejobs of 2026-11-17/18 before these runs took 272.7 to 504.6 s, with store-cloudflare tests of 56.9 to 97.6 s. The failures' store-cloudflare tests (93.0 to 103.8 s) overlap the greens', and the 2026-11-16 failure ran the contract file before !929 lengthened it (648 cases, 103.7 s). The job time and the gap before the contract file separate them: 148.5 to 229.1 s in the failures, 32.3 to 89.2 s in the greens. So the failures go with slow, loaded jobs, not with a longer contract file.
Seen again (2026-11-19, the onTaskUpdate shape): !932's MR pipeline 3087479794, test:cloudflare job 18170434388 (729.6 s). store-cloudflare's state-store-contract.cf.test.ts passed (666 tests, 8 skipped, 99 221 ms), then the runner stopped on Error running worker: Error: [vitest-worker]: Timeout calling "onTaskUpdate": Test Files 1 passed (1), Tests 658 passed | 8 skipped (666), Errors 1 error, and EVERY-FILE-RAN failed the run with 7 of 8 planned files never run. No assertion failed. WALL-TIMELINE's largest gaps: 119.3 s from [init] to the contract file's start, then 36.5 s, 35.1 s and 25.2 s inside it. This is FU-FLAKE-CF-VITEST-ONTASKUPDATE-RPC-TIMEOUT's RPC this time, not a file's load, after the same long contract file. Its one disclosed retry, job 18171061692, passed: store-cloudflare Test Files 8 passed (8), Tests 726 passed | 8 skipped (734), the contract file 104 441 ms, Duration 246.04 s, with a 115.5 s gap from [init] to the contract file, about the failing run's.
FU-CF-TIMER-CLOCK-PRODUCT-TIMERS: CF product timers armed after a binding-only stretch may fire early — open (candidate finding with the conformance owner; production unverified)
Filed by: WE-2, from FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT (orchestrator question, 2026-10-26).
- Ruled out: no product code in runtime-cloudflare, store-cloudflare, memory-cloudflare, core or runtime-js measures elapsed time with
performance.now(). Deadlines and TTLs useDate.now()or the store's clock. - Not ruled out: in local workerd, a timer armed after a stretch of binding-only awaits with no incoming event is measured from a clock that is behind by the stretch. A 1 s timer armed at a lag of 1 490 ms fired after 1 ms of wall time. So a Durable Object that runs a long D1 or storage-only stretch with no incoming request, and then arms a bounded wait, can cut that wait short. Sites, as of main 5c74c8c57c:
runtime-cloudflare/src/do/persistent-companion-tools.ts:2404, the sharpest: a companion wait's wall-clock deadline armed assetTimeout(deadline - now()).do/companion-parent-ref-fence.ts:82anddo/companion-settle-notice.ts:230/671/702: fence and notice timeouts.do/do-stream-manager.ts:1944/2359: safety-net polls.do/drive/run-drive.ts:682/2048anddo/drive/submit-answer.ts:192: drive waits.steps.ts:6330: the CF Workflows remote-call timeout.store-cloudflare/src/durable-lock.ts:169: the lock heartbeat.
- Witness (conformance owner): a cf-do cell where the DO runs a D1-heavy loop with no request reaching it, then a bounded wait, asserted against a
Date.now()floor taken when the wait is armed. Plus a production probe (a deployed Worker that logs both clocks across D1 awaits). If early fires are real, a timeout should re-checkDate.now()against its deadline when it fires, and re-arm if it is early.
FU-FLAKE-CFW-TRUTH-MATRIX-TEXT-TERMINAL-TIMEOUT: CF Workflows truth-matrix text-terminal instance missed its 25 s completion wait — open
Owner: WB. Surfaced by: !338 gating pipeline 2934478005, test:workflows job 17086002987 (the shared local arm64 runner, 1 047 s, host load ~21 from non-helix work). Filed by the conformance owner.
- Signature: runtime-cloudflare
checkpoint-truth-matrix.wf-noiso.test.ts› "checkpoint / run-truth matrix — 'cfw-workflows-d1-pool' › text-terminal: a plain text reply ends the run" (29 380 ms). The instanceagent__ct-matrix-text__…did not reachcompletewithin 25 000 ms (the wait ine2e/src/harness/setup-helpers/cfw-workflows-d1.ts:469). - The other seven pool boundaries in the same file passed (0.9–11.7 s;
forced-success11.7 s).text-terminalis the file's first boundary, so it may pay the pool lane's cold start under load; not measured. The pool lane is the legacy workerd adapter (cfw-workflows-d1-pool) kept for this file (see the pool-consumer retirement note above). - Fix direction: measure the first boundary's start-to-complete time on an idle and a loaded host. Then size the wait from that (or warm the pool before the first boundary) instead of a fixed 25 s.
FU-FLAKE-CF-UNIT-VITEST-STARTUP-HANG: test:unit hangs on runtime-cloudflare's unit vitest run at startup — open
Owner: CI-health-3. Surfaced by: !310's pipeline 2923294110, test:unit (job 17007855811, the shared local arm64 runner), 2026-10-07. !310 changes no runtime-cloudflare unit test.
- Signature: the job's trace stops (its last line 17:08:47Z, after
core's summary) and turbo never prints itsTasks:summary. Every task that printed anything had finished. - Process evidence (read-only
docker exec … psin the job's own container on the runner, about 42 minutes in): turbo waits onpackages/runtime-cloudflare'svitest run(sh -c vitest run→node (vitest), started ~2 540 s earlier, 0.5% CPU), whoseesbuildchild is<defunct>. vitest hung during startup and printed nothing. - Next: capture the hang's stack (
node --cpu-profis useless at 0% CPU;kill -USR1+ inspector, or avitest --reporter=verboserun withDEBUG=vite:*in a loop) to see whether it waits on the esbuild service (a dead child it still expects), a worker pool, or a Miniflare start. - Cost of a hang:
test:unitsets notimeout:, so it takes the project default (3 600 s): a hang holds the Mac runner for up to an hour. This one was cancelled by hand about 45 minutes in. Options: a per-package startup watchdog (fail a vitest task that prints nothing for N minutes), or a job-leveltimeout:near the job's normal length plus a margin.
FU-FLAKE-REDIS-USAGE-TIME-RANGE-WALL-CLOCK: store-redis "should filter by time range" reads a fresh Date.now() against setup-stamped entries — open
Owner: backlog (next CI-health session). Surfaced by: !319's local matrix, 2026-10-08 (host load average 22–32 during a full turbo matrix on the shared Mac; CI's commands; private infra: redis-stack 6401, postgres 5473, temporal 7253). !319 does not touch the package.
- Signature:
packages/store-redis/src/__tests__/integration/redis-usage.integ.test.ts:275-282"should filter by time range":AssertionError: expected 0 to be greater than or equal to 2(the test took 4 128 ms). - Isolated:
pnpm --filter @helix-agents/store-redis run test:integrationat load ~4 on the same head: 33 files, 815 passed / 1 skipped. - Likely cause: the window is
now - 2500 .. nowwithnowread in the test, while the entries were stamped at setup (−2000 / −1000 ms then); under load the gap exceeds 500 ms and both fall out of the window. - Fix direction: stamp the window from the setup's own clock (or inject a clock), never a fresh
Date.now().
FU-FLAKE-AGENT-SERVER-RUNTIME-PARITY-RESUME-RUN-STATUS-RACE: agent-server "runtime parity › resume after interrupt" resumed before the old run was terminal — done (closed by this MR)
Owner: cf-we-do-streams-2. Surfaced by: !342's pipeline 2939506066, test:integ:rest job 17118984115 on do-nyc3 (992 s), 2026-10-12; §A3-retried green. Related: FU-FLAKE-AGENT-SERVER-RUNTIME-PARITY-PG-HOOK (the same file, a different signature).
- Signature:
packages/agent-server/src/__tests__/integration/runtime-parity.integ.test.ts, "AgentServer runtime parity: JS + Postgres" › "resume after interrupt":AgentAlreadyRunningError: Agent session '…' is already running (status: completed)from runtime-jscommitRunStartviaresumeSession. - Mechanism: the test's
waitForCompletionpolled only the SESSION status. The terminal order commits the K5 session state before the run's terminal status (run-loop.tswritesupdateRunStatusafter it), so the test could CAS the session tointerruptedand resume while the previous run still readrunning. The resume planned withexpectCurrentRunStatus: 'running', and the store's run-status CAS then rejected it once the run turnedcompleted. - Fix (test only):
waitForCompletionalso waits untilgetCurrentRunis no longer live (isLiveRunStatus). Red first: a case that holds the run's terminalupdateRunStatusshowed the old helper returning inside the window on all three stores (memory, Redis, Postgres); with the fix the file is 12 / 12, three runs in a row. The other waits in the file all go through the same helper.
FU-FLAKE-AGENT-SERVER-RUNTIME-PARITY-PG-HOOK: agent-server "runtime parity: JS + Postgres" hook timed out at 30 s — open
Owner: backlog (next CI-health session). Surfaced by: !319's local matrix, 2026-10-08 (host load average 22–32; the same conditions as FU-FLAKE-REDIS-USAGE-TIME-RANGE-WALL-CLOCK). !319 does not touch the package.
- Signature:
packages/agent-server/src/__tests__/integration/runtime-parity.integ.test.ts, describe "AgentServer runtime parity: JS + Postgres":Error: Hook timed out in 30000ms(the file took 46 s; 3 skipped). - Isolated: at load ~4, 9 / 9 with
pnpm --filter @helix-agents/agent-server run test:integration src/__tests__/integration/runtime-parity.integ.test.ts. - Recurrence (CP-102's local matrix for !332, 2026-10-09, head b815e58a9c): the same signature (the file took 51 s; 66 passed, 3 skipped). Run: the
integ-restphase, 19:22–19:25Z, i.e.turbo run test:integrationover every package except e2e and runtime-dbos, against one Postgres 16 container (postgres:16-alpine). The phase started at a load average of about 12 (not starved), while the store-postgres, runtime-js and Temporal Postgres integration suites ran against the same server, so load alone does not explain it. The timed-out hook is the suite'srunLifecycleTestssetup: onePostgresMigrations.run()over a fresh table prefix, every migration in series. CP-102 does not touch the package. Isolated straight after the matrix (22:12Z, the same container, no other suite running, a load average of 13–15, about the failing run's load), the JS + Postgres suite passed 3 / 3 (the file alone, the same command as the Isolated line above, with onlyPOSTGRES_URLset: 6 passed, the Redis suite's 3 skipped). So what triggers it is the concurrent Postgres suites, not host load. - Fix direction: time each migration in that setup under concurrent Postgres suites, and find whether it waits on a lock another suite's DDL holds (a catalog lock, or the migration lease) or is just serial work stretched past 30 s. Do not raise the timeout without the cause.
FU-FLAKE-CF-DO-CASCADING-SUBAGENTS-SPAWN-RESULT: CF DO "cascading nested sub-agents (3 levels)": a live child from the previous test took the cascade's scripted response — done (test only)
Owner: cf-we-do-streams-2. Closed by: this MR. Duplicate: FU-FLAKE-CF-DO-PERSISTENT-AGENTS-SHARED-SCRIPT-QUEUE (filed 2026-10-07 with an instrumented trace of the same mechanism; closed with this one). Surfaced by: !319's local matrix, 2026-10-08, the e2e test:cloudflare run (1 of 289 tests; host load average 22–32). !319 does not touch the path.
- Signature:
packages/e2e/src/__tests__/cloudflare-do-persistent-agents.cf.test.ts:954, "cascading nested sub-agents (3 levels) › root spawns child which spawns grandchild; all three reach terminal state":AssertionError: expected [] to have a length of 1 but got +0onchildSpawnResults(the child's${COMPANION_TOOL_PREFIX}spawnAgenttool result absent when read). - Isolated: 9 passed / 2 todo on each of 3 runs of
pnpm --filter @helix-agents/e2e run test:cloudflare src/__tests__/cloudflare-do-persistent-agents.cf.test.ts. - First guess (superseded by the mechanism below): the test reads the child's messages once the grandchild is terminal; under load the child's own tool-result commit may not have landed yet. Check whether it waits on the child's terminal state (a condition) before reading.
- Rate (measured 2026-10-10 for !332): 2 of 15 cold-cache runs of e2e
test:cloudflareon main a1c6655f56 and 2 of 10 on !332's head, interleaved on one host (load average 4.6–11.7), all at :954 (expected [] to have a length of 1). Earlier !332 runs: about 4 of 21 overall, including one concurrent run that failed at :922 instead (the root runfailed). So it predates !332, at roughly 1 run in 7. - Mechanism (confirmed by the deterministic red proof below, and by the duplicate's trace; line numbers pinned to origin/main a07e5aa1fb): the test before the cascade test, "parent interrupt during (non-blocking) child run" (
:750), accepted its non-blockingwriterchild stillrunningornot_foundwhen it ended (:834), as did the no-wait (:639) and waitForResult (:309) tests, and every test shared one FIFOpersistentAgentTestLLMqueue (test-worker.ts:394) thatbeforeEachonly cleared (:143-144). A child still alive then called the LLM during the cascade test and took one of its responses. Takingl2-spawnleft the mid-level child the grandchild's finish as its first step, so it completed without spawning and its spawn results were[](:954); takingl1-spawnran the root on the wrong script, which failed (:922). - Red herring (corrects an earlier note here): every run logs
src/test-worker.ts changed, invalidating this Durable Object(@cloudflare/vitest-pool-workers'createDurableObjectWrapper; withisolatedStorage: falseeach test file re-imports the worker module). The duplicate's trace found these lines in every full-suite run, passing or failing, so they are not a second leak path. The liveness guard below also means no agent run from this file outlives it. - Done (this MR): two rules, so the class cannot recur in this file.
- No session can take another session's response. The fixture's mock LLM (
test-worker.tspersistentAgentLLM,e2e/src/helpers/session-scripted-model.ts) is a coreScriptedModelbehind a session-only control, and the runtime gets only itsLLMAdapter. Each session is scripted on its own queue (scriptFor(sessionId, …); a child's id isgeneratePersistentSubSessionId(parent, name)); a global, agent-type orholdAfterDeltasscript is refused (UnsupportedScriptError). A call for a session with nothing scripted is answered with a scripted failure naming the session and recorded, and the file'safterEachthrowsUnscriptedSessionCallErrornaming it. The "schema-compatible finish" workaround for the non-blocking tests is gone. - No test leaves an agent running. Every test ends with all its sessions terminal: blocking children by construction, non-blocking ones by an explicit bounded wait. The three tests that accepted a
running/not_foundwriter at their end now require itcompleted. AnafterEachacross the file runs every check and reports all failures together: a tracked session not terminal, a call still held, an unscripted call, scripted responses never consumed (unless the test allowed it), a sticky harness error. It then resets the mock, which answers any still-held call as cancelled (framework_cancelled, like core'sScriptedModel.reset()), so a leftover call cannot act in the next test. - The interrupt test holds its writer. The writer's first LLM call is held, and the parent is interrupted only once it is: the interrupt provably lands while the child runs, and the child then completes on release. Without the hold, an interrupt that lands before the parent's spawn tool runs leaves no child at all (it stays
not_found, seen once in a full-file run), which no "wait for terminal" rule can satisfy. That pre-spawn path (4 of 5 unheld runs of the old test reached it) keeps its own deterministic test, "interrupting the parent before its spawn tool runs starts no child": the parent's spawn-emitting LLM call is held,/interruptis sent, and the call is released only once the interrupt's abort has reached it (abortedAton its call record), so the run's abort check before the tool batch always sees it. The parent endsinterruptedwith no spawn result, no sub-session ref and no child. (The/interruptresponse itself comes only once the run has stopped: the DO's interrupt waits for the run's result.) - Holds that work in workerd. A scripted
{ kind: 'hold', gateId, then }is resolved by the session-only control, which polls a plain flag in the held call's own context; the test pollsisHeld(gateId)in its own.ScriptedModel.whenHeld()/release()hang the file instead (FU-SCRIPTEDMODEL-HOLD-CROSS-DO-CONTEXT). - Regression test: "session isolation of the mock LLM › a live child's LLM call amid another session's queued script takes only its own". A writer is held in its first call; a cascade starts and is held at its mid-level child's first call, with the root, child and grandchild scripts still queued; the writer is released, and its second call dequeues in that window. It takes its own finish, the cascade's queues are untouched, and the cascade then completes; every session made exactly its own scripted calls. Against a shared FIFO (the control's queues collapsed into one, a temporary local check) it fails at its first attribution check.
- Red first (main a07e5aa1fb's fixture, deterministic): a one-off test (not committed) held a live writer's first LLM call before the FIFO dequeue and released it when the cascade's mid-level child made its first call. 3 of 3 runs: the recorded call order showed the writer's call taking the second response (
l2-spawn), the root endedfailed, and the cascade test's:954assertion failed (expected [] to have a length of 1 but got +0; the one-off checked that assertion, not the root's status, which:922checks first in the cascade test). - Runs after the fix (2026-10-10, host stress lock held, Mac CI runner idle throughout): 15 of 15 cold-cache runs of e2e
test:cloudflarepassed (37 of 37 files, 289 tests each; 52–67 s; host load average 3.8–19.6), against the baseline of 2 of 15 failing on main and 2 of 10 on !332; and 50 of 50 runs ofcloudflare-do-persistent-agents.cf.test.tsalone passed (11 passed + 2 todo each; host load 5.2–19.7).
- No session can take another session's response. The fixture's mock LLM (
FU-SCRIPTEDMODEL-HOLD-CROSS-DO-CONTEXT: core ScriptedModel holds hang a workerd test whose held call runs in a Durable Object — open
Owner: cf-we-do-streams-2 (CF test-infra backlog). Surfaced by: the cascading-subagents isolation MR (FU-FLAKE-CF-DO-CASCADING-SUBAGENTS-SPAWN-RESULT), 2026-10-10.
- Signature: in a
@cloudflare/vitest-pool-workerstest, aScriptedModelhold ({ kind: 'hold', gateId, then }) caught by an LLM call running inside a Durable Object, with the test awaitingmodel.whenHeld(gateId, …)and then callingmodel.release(gateId), makes the whole file hang. workerd logsError sending to pool inside runner: Cannot perform I/O on behalf of a different Durable Object … (I/O type: SpanParent)for vitest'sonTaskUpdate/snapshotSavedRPCs, then[vitest-worker]: Timeout calling "snapshotSaved"; the run only ends at its outer timeout. - Cause: every DO in the pool shares the test's isolate, but workerd refuses a promise settled in one I/O context and awaited in another.
whenHeld()'s promise is settled from inside the DO's call (ScriptedModel.hold) and awaited by the test, so the test's continuation runs in the DO's context and its next runner RPC is refused;release()settles the DO's gate promise from the test's context. - Repro: in
packages/e2ecloudflare-do-persistent-agents.cf.test.ts, give the session-scripted fixture a plainScriptedModel(no custom control), script a spawned child's session with ahold,await model.whenHeld(gateId, 30_000)in the test, thenmodel.release(gateId); run the file withpnpm exec vitest run --config vitest.cloudflare.config.ts <file>. The isolation MR's first draft did exactly this and hung on every run; the same file without holds passes. - Workaround in use:
e2e/src/helpers/session-scripted-model.tsresolvesholdin itsScriptedModelControl: the held call polls a plain flag in its own context, and the test pollsisHeld(gateId)in its own. Nothing awaitable crosses a context. - Fix direction: give
ScriptedModelthe same poll-a-flag gate (or an option for it), sowhenHeld()/release()are safe wherever the held call runs, and test it in the workerd pool. Check the other gates (holdAfterDeltas, delivery barriers,waitForAbort) for the same shape.
FU-FLAKE-PERSISTENT-SUBAGENT-D1-JS-FILE-TIMEOUT: e2e persistent-subagent-d1-with-js.integ.test.ts — every test in the file times out at 30 s — open
Owner: CI-health-3. Surfaced by: !310's pipeline 2923488532, test:integ:e2e 1/4 (job 17009383743, DigitalOcean runner), 2026-10-07. !310 changes no store or runtime code on that path.
- Signature: all 4 tests of
Persistent Sub-Agent - JS runtime + D1 storetime out at 30 s ("non-blocking spawn persists ref with spawned status to D1", "listChildren retrieves children from D1 correctly", "blocking child re-spawn continues the preserved session…", "re-spawning child with same name after completion continues…"): a whole-file stall, not one test. The job ran 2 357 s against about 1 363 s in the previous pipeline, so the runner was loaded. - Next: check whether the file's D1 (Miniflare) setup stalls rather than the tests (every test failing at the same cap suggests a shared resource: the D1 binding or its proxy); run the file in a loop locally, single worker.
- Second occurrence (2026-10-08, !319 pipeline 2927290260,
test:integ:e2e 1/4, job 17038972568, DigitalOcean runner, 1 542 s, ~17:39Z): 3 tests of the file failed at ~30 s each ("blocking spawn persists SubSessionRef", "non-blocking spawn persists ref with spawned status to D1", "listChildren retrieves children from D1 correctly"); 1 failed / 58 passed files, 3 failed / 1 067 passed tests. The same whole-file stall shape. The runner had just lost and regained its k8s API auth (~17:10Z: earlier attempts of 1/4 and 4/4 failedrunner_system_failure, "Getting job pod status Unauthorized"). !319 does not touch this path. The next attempt (17039603793) ran the file green (11 tests, 75 s; the shard 59/59 files, 1 070 tests) before failing at pod cleanup. - Fix direction (orchestrator, DoD §F1): the tests sit on fixed 30 s per-test timeouts that a loaded runner exceeds: replace them with condition waits under a named bound (and find what in the D1 / Miniflare setup is slow under load) rather than raising the timeout. The test uses
RoutingMockLLMAdapter.
FU-FLAKE-REFRESH-APPROVAL-CFW-PAUSED-BEFORE-COMMIT: e2e approval refresh on cfw-d1 read the snapshot before the suspension committed — done (W1; test only)
Owner: W1. Surfaced by: the finish batch !679's merge-train pipeline 3026016182, job 17727050639, and batch 10's MR pipeline 3023653714: refresh-approval.integ.test.ts › "refresh during a pending approval (review C2) — cfw-d1" › "ai 6.0.49: after a reload the approval is approval-requested; approving it completes the run" › the snapshot's tool part: expected undefined to be 'approval-requested' (the call's tool part was absent, not in another state). Green in non-train pipelines.
- Mechanism (from source): the case waited for the SNAPSHOT to read
paused, then read it again and asserted the call's part. A snapshot'spausedcomes from the run's status alone (corederiveSnapshotStatus: a live run whose status is notrunning), and its messages from the latest checkpoint. CF Workflows writes the run'ssuspended_*status, then the suspension frames, then the suspension commit, thenpauseStream(DBOS too; JS and Temporal write the status after the commit). Between the status and the commit the snapshot readspausedwith only the user's row. The spec already says it: "suspension committed" is the stream'spausedstatus, not the run's. The window widens under load; the case's own 25 ms snapshot poll (D1 reads on cfw-d1) added load. - Done: the case anchors on the stream's own
pausedstatus (the last write of every runtime's suspension sequence), then reads the snapshot once and assertsapproval-requestedwith the approval id, as before. The sibling refresh helpers already anchor on "paused and the snapshot holds the call". - Red proof (local slot
w1-rap-slot-20261104-040539): with a 3 s hold between CF Workflows'suspended_*status and its suspension frames (a local mutation), the pre-fix case failed exactly its 6 C2 rows (3aiprofiles on cfw-memory and cfw-d1, each "the snapshot's tool part: expected undefined"), the controls passed; the fixed case passed all 8 with the same hold; unmutated, the fixed file passed 32 of 32 on all eight lanes, 3 runs of 3. - Product question (with the conformance owner): a client that reads the snapshot inside that window sees
pausedwith no pending call. A resuming client tails the stream from the snapshot's cursor and receives the approval request; a snapshot-only reader does not. Whether that is a finding is the owner's call.
FU-FLAKE-REFRESH-C1-CFW-SUBMIT-CONTINUATION-LATENCY: e2e C-1 refresh on cfw-d1 ran its whole client-tool continuation inside one 30 s window — done (W1 MR3)
Owner: W1 MR3. Surfaced by: !342 pipeline 2939506066, job 17118984110, and main 2939922050, job 17122034986 (both do-nyc3): refresh-c1.integ.test.ts › "C-1 refresh mid a multi-step / HITL response (RM-08) — cfw-d1" › "ai 6.0.49 snapshot-seed: refresh while the client tool is pending …" › snapshot … not terminal within 30000ms (last: active).
- Mechanism (WE-2, 2026-10-12;
handoffs/cf-we-do-streams-data/cfw-submit-latency-mechanism.mdin the orchestrator dossier): latency, not a lost continuation. The answer was stored and the__submit__continuation was created and ran.- Cost: wall time grows linearly with the cost of one D1 statement. With 160 ms injected per statement, the run started 8.1 s after the submit and finalized 22.1 s after it; unloaded, the cell takes 8 s.
- The test added load: the whole continuation is 219 D1 statements. In the same window, the test's own 50 ms terminal-snapshot poll issued 210 more.
- One window over everything: the cell's single 30 s window covered client B's resume GET (14.4 s on CI), the tool, the POST, the handler's live wait and the whole continuation.
- Harness half (done):
pollSnapshotUntil(helpers/headless-chat.ts) backs off from 25 ms to 500 ms. It replaces the fixed 25 / 50 ms loops inwaitForTerminalSnapshotand the pending-call wait, for every scenario that uses them.runPendingClientToolRefreshtimes each phase on its own. The suspend phase and client B's submit phase gettimeoutMseach. The continuation is timed from B's POST: ai-sdk's newly exportedDEFAULT_LIVE_WAIT_TIMEOUT_MSplusCONTINUATION_AFTER_LIVE_BUDGET_MS(30 s; the 160 ms model needs 14 s after the run goes live).- The cell's test timeout derives from
pendingClientToolRefreshBudgetMs(), so the phase error, with its diagnostics, fires first.
- Diagnose (done):
pendingRefreshDiagnosticslists every__submit__generation of the suspended run with its instance status and its own run record, through the new supported readCloudflareAgentExecutor.listSubmitContinuations, so a sighting tells "never created" from "created but slow". - Proof (2026-10-12, under the stress lock): the cfw-d1 cell "ai 6.0.49 snapshot-seed: refresh while the client tool is pending", 20 sequential runs on the real refresh-c1 cfw-d1 lane with 160 ms injected before every D1 statement (WE-2 measured this cell FAIL at 160 ms on the old harness): 19 pass (58–63 s each). The one failure (run 18, 83 s) is NOT this mechanism: the diagnose showed the
__submit__continuation created, committed and running; its snapshot read failed typed (FU-SNAPSHOT-PASS-TIMEOUT-SLOW-STORE). The continuation latency held in 20 of 20 runs. Raw logs: the measuring session's local scratch directory (x20/:summary.txt,delay160-run18.log). The diagnose also lists the generations of every suspended or superseded run of the session (from its run list), so the original suspended run's continuations show after one commits. - The product half (redundant same-session reads on the submit path) is FU-CFW-SUBMIT-PATH-READ-ONCE (WB).
FU-SNAPSHOT-PASS-TIMEOUT-SLOW-STORE: a snapshot read times out every pass on a slow store while a continuation runs — ✅ fixed (!351, fa93248ae5)
Fixed by !351 (WE-2): readSnapshotView has no per-pass timeout any more. It has one whole-read deadline, deadlineMs (default 10 s; Infinity means none). The chat path sets it through snapshotReadDeadlineMs on BuildSnapshotDeps and HandleChatStreamDeps. A pass that is still reading is no longer failed as torn.
Correction to the record below: before !351, passTimeoutMs was an option of core readSnapshotView only. buildSnapshot called it with { includeThinking } alone, so no chat route could set it; the record's "BuildSnapshotDeps.snapshotOptions.passTimeoutMs exists" was wrong. !351 added the chat path's deadline option.
Surfaced by: the W1 MR3 ×20 proof of FU-FLAKE-REFRESH-C1-CFW-SUBMIT-CONTINUATION-LATENCY (run 18 of 20).
- Mechanism: core
readSnapshotView(core/src/store/snapshot-view.ts) gives each read pass a fixed wall-clock budget,DEFAULT_PASS_TIMEOUT_MS= 2000 ms, for 3 passes; then a typed, retryablestate_history_incomplete(snapshot_unstable). At 160 ms per D1 statement, with the__submit__continuation's own statements competing for the same D1, a pass runs past 2 s: run 18's read failed "no stable snapshot read in 3 passes (timeout@getCurrentRun ×3)" on all 4 fetches of the client rule (fetchSnapshotWithRetry), so the read surfaced asSnapshotUnavailableError, which a realuseHelixChatrefresh would get too. CI's regime looks like about 300 ms per statement (WE-2), so it is likely more frequent there. - Red recipe: the delay lane from the proof (the refresh-c1 cfw-d1 lane with a per-statement delay, as in the raw logs' run); 1 of 20 at 160 ms; a higher delay should make it reliable.
- The question (design first): the default and the CF Workflows chat path's value (
BuildSnapshotDeps.snapshotOptions.passTimeoutMsexists), and whether a fixed wall-clock per-pass budget is the right shape at all, rather than a progress-based one (a pass that is still reading is not a torn read). - Not a harness fix: the test's poll keeps the client's rule and surfaces the typed error.
FU-JS-RULE6-REDELIVERY-E2E-PIN: pin rule 6's stash-delivery marker end to end on runtime-js / the CF DO — open (LOW, test debt; owner: W1)
Surfaced by: the W1 MR3 F4 test review (FU-JS-RULE6-UNMARKED-NOTIFICATION-REDELIVERY is fixed and proven in core: retry-plan.test.ts, red first).
No runtime test drives the redelivery end to end, and none round-trips COMMON_METADATA_KEYS.STASH_DELIVERY through a SQL store. The precondition is a failed run whose session still holds ANSWERED, undrained trailing calls: runtime-js refuses a new turn over them (execute() answers "already running with unresolved client-tool calls"), so it arises from an answer that lands while the entry's run is live (a late submit), which the existing suites have no hold for. Fix direction: a hold between the entry's run-start commit and its first model call (a held MockLLM response), a submit of the second call inside it, then fail / retry() / fail / retry(), asserting the notification and the message each appear once; run it through describeWithMessageDurability so memory and the SQL stores are covered.
FU-CFW-SUBMIT-PATH-READ-ONCE: the CFW client-tool submit → continuation path re-reads the same session many times — open (LOW, perf; owner: WB)
Surfaced by: WE-2's measurement for FU-FLAKE-REFRESH-C1-CFW-SUBMIT-CONTINUATION-LATENCY (orchestrator ruling, 2026-10-12: not in W1 MR3; design it with FU-CFW-RECONCILE-SESSION-PER-REQUEST-PROBE, the same "too many D1 reads on the CFW chat path" theme).
Starting analysis (unloaded, every store call tagged by its caller):
- The submit call (
routeSubmitToolResult+ensureSubmitContinuation, up to the instance create): 11 store calls, 27 D1 statements.loadState×7, all of the same root:submitToolResult's retention read, the router's reads,decide's root, andholdsReadyBatch→isBatchReadyOn.getCurrentRun×3 andsaveState×1.
- The workflow body, up to
run_started: 14 store calls, 56 statements.getCurrentRun×4,loadState×3,getRun×3,getMessages,getMessageCount,getLatestCheckpoint;- the run-start commit, at 15 statements.
- Every
ensureSubmitContinuation(including each chat POST and snapshot throughreconcileSession) callscreateon an instance that already exists. That is a throw, thenget, thenstatus, thengetRun.
Fix direction: pass a value already read down the chain wherever no write lies between the two reads. Keep every deliberate fresh read (CP-96 seen-reads, fix-round re-reads) and document why each is kept. Measure the statement count before and after with the tagged lane.
FU-FLAKE-REFRESH-C1-CFW-PENDING-CLIENT-TOOL-NOT-TERMINAL: e2e C-1 refresh on cfw-d1 — the run never reaches a terminal snapshot after the refreshed client submits the pending tool's output — done (W1 MR2b)
Owner: cf-w1-cfw-stores-submit (the CP-104 fix, W1 MR2; CI-health-3 diagnosed it). Surfaced by: !310's pipeline 2923294110, test:integ:e2e 1/4 (job 17007855812, DigitalOcean Intel runner), 2026-10-07. !310 changes no CF Workflows or refresh code.
- Test:
refresh-c1.integ.test.ts› "C-1 refresh mid a multi-step / HITL response (RM-08) — cfw-d1 › ai 6.0.281: refresh while the client tool is pending: client B runs it once, sends the output, char-exact" (47 s). - Signature:
snapshot of c1-pd-… not terminal within 30000ms (last: active)(headless-chat.ts:681, called fromrunPendingClientToolRefresh). A liveness failure, not a content mismatch: after client B ran the client tool and submitted its output, the CF Workflows run never reached a terminal snapshot within 30 s. 1 of 1 075 tests failed in the job. - No diagnostics in the trace beyond the snapshot status (
active): the run record, the instance status and the stream status at the deadline are not logged. - Second occurrence (!310 pipeline 2923488532,
test:integ:e2e 1/4, job 17009383743, 2026-10-07): the same test at ai 6.0.49, and nownot terminal within 30000ms (last: paused).pausedafter client B submitted the tool's output means the run is still waiting on that client tool: the refreshed client's submit did not take effect. Two in two pipelines, at both ai versions: this now points at a product stall, not load. - Product path found (static read, 2026-10-07): when CF Workflows'
resume()refuses because the previous (suspended) instance is still live afterpriorInstanceSettleTimeoutMs(30 s, the test's own deadline), the chat handler's Phase B catch sees the submitted-but-undrained entry as still pending and returnsrejectedOnlyResponse: the answer is stored, the run stayspaused, and nothing resumes it. Catalogued by the conformance owner as CP-104 (owner batch 13); the user ruled the fix (server retry, then a typed retryable error, plus thestillPendingcorrection and a cross-runtime check), owned by CF-D2 after !316. Still open here: why the suspended instance stayed live. - Open question: load (the job ran beside the rest of the pipeline on the shared group runner) or a real stall: does the CF Workflows submit-tool-result after a refresh fail to resume or drain the run on cfw-d1? If it is a product stall it is T1-adjacent on the CF path. Next: reproduce the case in a loop (single worker), and make the timeout path log the run record, the workflow instance status and the stream status, so a CI recurrence names the stuck layer.
- Recurred 2026-10-10 in !334 (W1 MR2a) pipeline 2933538936,
test:integ:e2e 1/4job 17080790030, at 13:31Z, on the DOKS k8s runner (gitlab-runner-7d7f8779c7-xrxjs).- Two cfw-d1 cases failed: "ai 6.0.49 snapshot-seed" and "ai 6.0.281".
- The first was a snapshot fetch failure:
no stable snapshot read in 3 passes (timeout@loadState; timeout@getMessages; timeout@getLatestCheckpoint). At that point the server had the runrunning, the instancerunningand the sessionactive, so the slow part was the D1 reads. - The second has this FU's not-terminal signature.
- Main's last three pipelines ran this shard green on the Intel DigitalOcean runners. Locally on the MR2a head the file passed 96/96 in 59.6 s.
- Retry: 17081308904.
Done (W1 MR2b, CP-104). The stall was the chat handler inferring the continuation: CF Workflows' submit only stored the answer, and the handler's resume() could be refused while the suspended instance was still live, leaving the answer stranded. Now the submit that completes the batch starts a deterministic __submit__ continuation that waits in its body for the suspended instance to close (continuationPredecessorTimeoutMs), answers continuation: { kind: 'new_run' }, and the handler attaches without calling resume(). Tests: runtime-cloudflare/src/__tests__/executor-submit-continuation.test.ts, cfw-predecessor-wait.test.ts, executor-resume-adopts-continuation.test.ts, cfw-submit-continuation.wf-noiso.test.ts (real engine); ai-sdk/src/__tests__/handler/resume-continuation.test.ts. Proof: the witness, refresh-c1.integ.test.ts's cfw-d1 pending-client-tool case at --retry=0, is green in MR2b's final matrix on every ai version (6.0.49, 6.0.49 snapshot-seed, 6.0.281). The CI-load half of the 2026-10-10 recurrence (the D1 snapshot reads timing out on the memory-starved runner) is not a product stall and stays with FU-FLAKE-CFW-ENGINE-REWIND-SNAPSHOT-UNSTABLE's load analysis.
FU-FLAKE-CFW-ENGINE-REWIND-SNAPSHOT-UNSTABLE: rewind-resync-cfw-engine.wf.test.ts "retry same-turn" refresh read is snapshot_unstable — open (T1 path)
Owner: CI-health-3 (T1 priority, orchestrator). Surfaced by: MR !291 pipeline 2910439346, job 16919362444.
Test:
rewind stream resync (RM-60, R34) — cfw-engine-d1-do › retry same-turn: a page refreshed mid the rewinding run renders the rewound transcript (ai 6.0.281).Signature:
SnapshotUnavailableError: … no stable snapshot read in 3 passes— the fenced snapshot read (readSnapshotView) saw the session's status, current run or latest truncate change in every pass, or a pass timed out.A/B (!291, same machine, sequential,
--retry=0, no pipeline running): headd73f5e177d5/5 and main5bb1eaaf2b5/5 — not reproduced on either side; no product change between the 26/26-green876627b81cand the failing head. Likely CI-load timing in the 3-pass fence window; RM-60 rewind-refresh is a T1 path, so it is worth a root cause rather than a wider window.CI timing (CI-health-3): the failing case took 33.4 s.
fetchSnapshotWithRetrymakes 4 fetches with 1.75 s of total backoff, each over 3 passes capped at 2 s.- So every pass hit its 2 s cap. A torn fence fails a pass in milliseconds, and the run was parked at the model gate, so its fence could not move.
- That leaves two candidates: (a) slow D1 / DO reads under load, or (b) a read that hangs.
Not reproduced on macOS (CI-health-3):
- A lane-level wrapper timed every snapshot read (alert over 250 ms or still pending at 1.5 s). It is uncommitted; the patch is in the reliability dossier.
- Runs: the 12 "refreshed mid the rewinding run" cases × 5, and the full
test:cfw-enginesuite × 6 (79/79 each). Both ran at--retry=0beside a looping forced build of every example (Next.js included), at load average up to 111. - The wrapper fired zero times and nothing failed. That argues against (a), at least on macOS.
Diagnostics (!293):
snapshot_unstablenow records each pass's outcome on the error (snapshotPasses): atimeoutwith the read in flight, or afence_changedwith the field and its before/after values. The message carries a short summary, so the next CI occurrence names its cause.Recurrence (!294, job 16925112876):
refresh-char-exact-cfw-engine"step3-after-tool-step +reasoning (ai 6.0.281)". The model stalled before call 2 for 30 s, and the snapshot read failed with(timeout@getStreamInfo; timeout@getStreamInfo; timeout@getStreamInfo): thesnapshotPassesdiagnostics named the read. Both recorded failures are ai 6.0.281 cases.Linux container repro (CI-health-3): a linux/arm64
node:24container ran the fulltest:cfw-enginesuite × 10 at--retry=0(load average 5–21): 10/10 green, no stall.Caller side:
getStreamInfoand D1loadStatewere regularly slow in linux (up to 2.5 s; zero on macOS).DO side: per-request entry/exit timing inside the stream DO (
console.warn; a positive control logged all 103 requests) shows every handler takes 0–1 ms. The time is spent outside the DO's code: workerd's output gate holds an object's responses until its storage writes are durable.Disk-sync pressure (
dd oflag=syncbeside therewind-resyncandrefresh-char-exactfiles, 50 tests, × 5): 164 slow reads, the worst 3.2 s, and the only slow DO handler seen in any run was a/write(380 ms). Still 5/5 green: the latency mechanism is confirmed, but a ≥ 30 s stall was not reproduced.The two failing CI jobs (683 s, 760 s) sat within the green range (21 green jobs, 435–865 s).
The runner's Docker VM disk was 95% full at both CI failures. It was cleaned to 31% on 2026-10-04.
Before vs after that cleanup (full suite, main's lane config, container,
--retry=0; slow = a lane read over 250 ms). The sets ran at different host loads, so this is directional, not a controlled A/B:Set Runs Slow reads Per run Slowest Runs with a read ≥ 1.5 s snapshot_unstableBefore the cleanup 10 71 7.1 2460 ms 4 0 Disk-sync stress (2 refresh files, 50 tests) 5 164 32.8 3208 ms 5 0 After the cleanup (2026-10-05) 5 6 1.2 998 ms 0 0 All runs were green, and the DO's own handler timing logged no slow request after the cleanup.
Lane fix (CI-health-3):
vitest.cfw-engine.config.tsputs Miniflare's DO and D1 storage on a private tmpfs dir under/dev/shm(Linux, ≥ 256 MiB free; the CI runner's job containers have ~16 GiB). That takes disk fsync out of the lane.Proof (same container, one lock hold per run, Miniflare's storage seen on
/dev/shmin every sampled run):Set Runs Slow-read events Per run Slowest Run time per run snapshot_unstableDisk-sync stress, main lane (2 refresh files) 5 164 32.8 3208 ms 281–775 s 0 Disk-sync stress, tmpfs lane (2 refresh files) 5 2 0.4 1725 ms 23–70 s 0 Full suite, tmpfs lane, after the cleanup 15 7 0.47 3688 ms — 0 All 15 full-suite runs were 79/79. They had no
data-resume-rejected, and every live attempt-switch cell passed.The tmpfs lane makes the lane immune to the disk-sync stalls a filling runner disk causes. It does not remove CPU contention: the one ≥ 1.5 s read (3.7 s) came at host load 41 from other projects' containers, while a run at load 37 had none.
Production is not affected: Cloudflare's storage durability is not a local fsync, so the output gate does not wait on a host disk.
Recurred on !313's pipeline (job 17003004633, 556af137, a catalogue-only MR), 2026-10-07: now on RM-08 char-exact refresh,
refresh-char-exact-cfw-engine.wf.test.ts›step2-mid-text-before-tool, both ai versions (56 s and 52 s), not only the rewind test.- The !293 diagnostics name the read:
timeout@getStreamInfoin all 3 passes of every fetch (the stream DO'sgetStreamInfotook over 2 s each time); nofence_changed. workerd logged "A WritableStream is experiencing excessive backpressure" in the same minute (14:48–14:49Z). - The runner was this Mac's Docker VM (the shared local arm64 runner,
workerd-linux-arm64), at host load averages of about 120–176 across the job (14:33–14:55Z, sampled from a session on the host). Its disk was at 49% when checked at 15:00Z, not the 95% of the two earlier failures above. The first reading was CPU contention, which the tmpfs lane does not address; superseded by the reassessment below. - Part of that load was our own: 14
yesburners left running from CI-health-3's loaded opennext runs (E2/E3, below), started 03:28 local (07:28Z) and killed by the orchestrator at about 15:00Z, each at about 55% CPU. They add about 14 to the load average; the load was still 89–110 after they were gone, so they were a contributor, not the whole of it. A re-run without them is the next data point. SQLITE_CANTOPENis teardown, not the cause: workerd loggedsqlite.c++:515 … unable to open database file: SQLITE_CANTOPENtwice, uncaught, at 14:55:48.6, after vitest's summary and "Shutting down runtimes", six minutes after the two failures (14:49:53). A Durable Object request still in flight met its storage being removed at pool shutdown.
- The !293 diagnostics name the read:
Again on main with no burners (pipeline 2923013233, job 17005652371, da5f85ce):
refresh-attempt-retry-cfw-engine.wf.test.ts› "live attempt-switch re-anchor, no refresh (RM-08, I9)", ai 6.0.281, 85 s;timeout@getStreamInfoin every pass of every fetch again. The runner was this Mac's Docker VM (the shared local arm64 runner) at host load 24–41 (1/5/15 min, 12:00 local), disk at 54%. Two occurrences today whose traces CI-health-3 read, on two different RM-08 refresh tests (the orchestrator counts a third today).Reassessment (CI-health-3):
- Not CPU contention: a read that normally takes 0–1 ms timing out at 2 s in every pass is not explained by load 24–41.
- Not a tight budget for a slow-but-working read: every one of the 12 passes (4 fetches × 3) timed out; a budget increase would only lengthen the wait.
- Not the backpressure warnings: workerd's "WritableStream is experiencing excessive backpressure" lines come from SSE consumers that read slowly; the stream DO's fan-out (
SSEConnectionManager.broadcast) never awaits a writer (it evicts a consumer at its pending-write cap), so a slow reader cannot stall/info. - Best-supported mechanism (not yet proven in CI):
/infois a pure read (SELECTonmeta,COUNT(*)onchunks), but workerd's output gate holds its response until the object's concurrent writes (every/writeis anINSERTplus anUPDATE) are durable, and Miniflare keeps that SQLite on the runner's disk. !300's container evidence measured exactly this: handlers took 0–1 ms while callers waited seconds under disk-sync pressure, and a tmpfs store removed it. Disk I/O contention from other jobs on the same VM does not show in the load average. Production is unaffected (Cloudflare's durability is not a local fsync). - Fix: the tmpfs lane above (this MR), not a budget change. If the flake recurs with the store on tmpfs (the run's
[cfw-engine] Miniflare storage:line says which), this mechanism is wrong, and the next step is per-request instrumentation in the stream DO (handler time vs response-delivered time) in CI.
Status: a watch item. Close it after ≥ 10 green
test:cfw-engineCI runs after the lane fix with nosnapshot_unstable;snapshotPassesattributes any recurrence.Runner dependency: the tmpfs path needs ≥ 256 MiB free in the job's
/dev/shm. Today's runner (the shared local arm64 runner, a Docker executor withshm_size = 0) gives its job containers ~16 GiB, inspected on live jobs on 2026-10-07. A Kubernetes runner's/dev/shmdefaults to 64 MiB, so after the CI-runner platform flip the job needs a memory-backedemptyDirat/dev/shm(or a Docker runner), or the fix stops applying. Every run logs its choice ([cfw-engine] Miniflare storage: …), so check that line before reading a recurrence as "recurred on tmpfs".
FU-FLAKE-CONF-HARNESS-BRIDGE-NETWORK-LOST: conformance harness host-driver.integ.test.ts client-tool cell loses the Miniflare bridge — ✅ fixed (plan 0c MR-1b)
Fixed (plan 0c MR-1b): CI-health-3 found the race: workers reach the bridge through an external service binding, so workerd pools keep-alive sockets to it and expires idle ones at about 5 s, while Node's bridge server closed them at its keepAliveTimeout (5 s) plus a ~1 s buffer. The failing job ran 41.7 s on a starved runner, where workerd's timer fired late, Node closed first, and the non-idempotent hook.fired POST went out on the dead socket. The bridge now sets server.keepAliveTimeout = 0 (packages/e2e/src/harness/cf-do/bridge-server.ts), so it never closes an idle socket first; there are no client retries. Same mechanism as cloudflare/workers-sdk#14850. Regression test: packages/e2e/src/harness/cf-do/__tests__/bridge-keepalive.test.ts (one raw keep-alive socket, a POST, a 6.5 s idle, a second POST on the same socket); it fails with keepAliveTimeout = 5_000 and passes with 0. Loaded before/after (CI-health-3): a standalone Miniflare 5 external-binding repro made POSTs after 5.5–7.0 s idles, 15 iterations at a recorded host load of 4–44 (from concurrent activity; the light example rebuild run beside it added almost none). It lost 0 of 210 calls at keepAliveTimeout 5000 and 0 of 210 at 0. That is inconclusive: at that load workerd's ~5 s pool expiry reliably wins. The deterministic regression test above is the red-first evidence.
Owner: CI-health-3. Surfaced by: MR !291 pipeline 2910439346, conformance:verify job 16919362440.
- Test:
host driver live on 'cf-do' › a client tool is answered by the responder (the product auto-send POST); the run completes; the client rejection is only recorded. - Signature:
bridge hook.fired: fetch failed: Network connection lost.— the stream failed (['failed'], expected['paused','ended']) andendCellrecorded a harness failure. - A/B (!291,
vitest.conformance-harness.config.ts): headd73f5e177d5/5 and main5bb1eaaf2b5/5 — not reproduced; a harness-side Miniflare bridge connection drop under CI load.
FU-FLAKE-CF-WORKERD-FATAL-WRAPPABLE-AFTER-MODULE-RESET: e2e test:cloudflare workerd crashes (expected wrappable.wrapper != nullptr) after the module-reset invalidation storm — open
Owner: CI-health-3 (triage after the FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT work). Surfaced by: !312 pipeline 2920873266, test:cloudflare job 16989690093 (arm64, @cloudflare/workerd-linux-arm64@1.20251011.0, @cloudflare/vitest-pool-workers@0.9.14). Related: FU-FLAKE-CF-E2E-TEST-WORKER-INVALIDATED (closed by CI health 2 as a benign log; this is the first time it crashed workerd), FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD (possibly the same family).
- Signature (
@helix-agents/e2e:test:cloudflare): afterlifecycle-hooks-parity.cf.test.tspassed, the usualtest-worker.ts changed, invalidating this Durable Object/broken.inputGateBrokenstorm (24 in the job), thenkj/async.c++:220: disconnected: PumpToReader was destroyed, then*** Fatal uncaught kj::Exception: workerd/jsg/setup.c++:313: failed: expected wrappable.wrapper != nullptr: workerd died, and the rest of the run failed withfetch failed/ECONNREFUSED. Every test that ran had passed (runtime-cloudflare 160 passed, store-cloudflare 293 passed, e2e's files up to the crash green). - Known mechanism (from the closed entry): vitest-pool-workers calls
vi.resetModules()after each file, so the next file re-importstest-worker.tswith new class identities; a Durable Object still alive from an earlier file (isolatedStorage: false) fails the wrapper's identity check when touched again. The closed entry's guidance was "if a test ever fails on it, give that test DOs that no other file touches". Here no test failed: the RUNTIME aborted, so the crash takes down later files regardless of which DO they use. - To triage: which file touches a stale DO instance after the reset (the storm's timestamps against the file order); whether a workerd / vitest-pool-workers bump fixes the fatal (the assertion is in workerd's JS wrapper setup, so likely a runtime bug); and whether per-file DO isolation (distinct DO namespaces or ids per file) removes the stale-instance access entirely. Check whether FU-FLAKE-CF-WORKERD-FETCH-FAILED-UNDER-LOAD's bursts are the same crash on x86 (
fetch failedwithout the fatal line).
FU-FLAKE-CF-E2E-TEST-WORKER-INVALIDATED: e2e test:cloudflare Durable Object invalidated by "test-worker.ts changed" mid-run — reopened 2026-10-07 as a crash (FU-FLAKE-CF-WORKERD-FATAL-WRAPPABLE-AFTER-MODULE-RESET); was ✅ closed (CI health 2: benign log, never a failure)
Closed (CI health 2): the message is vitest-pool-workers' own: after each test file it calls vi.resetModules(), so the next file re-imports test-worker.ts with new class identities, and a Durable Object instance still alive from the earlier file (with isolatedStorage: false they outlive the file) fails the wrapper's constructor-identity check when anything touches it again. Nothing writes the file. In the 164-pipeline sweep every job that logged it failed on another, now-fixed cause (CP-52, the fixed-sleep interrupts, the D1 parity timeout). The 22 per-test { retry: 2 } options that cited it are removed, and the e2e test:cloudflare suite ran 15 of 15 green at --retry=0 under CI-like load while logging it 376 times. If a test ever fails on it, give that test DOs that no other file touches.
Reopened as a crash (2026-10-07): on !312's job 16989690093 the storm ended with a fatal workerd expected wrappable.wrapper != nullptr: see FU-FLAKE-CF-WORKERD-FATAL-WRAPPABLE-AFTER-MODULE-RESET.
Owner: B8 (CI health). Not caused by any one MR.
@helix-agents/e2e's test:cloudflare sometimes fails with workerd reporting packages/e2e/src/test-worker.ts changed, invalidating this Durable Object. Please retry the DurableObjectStub#fetch() call. (broken.inputGateBroken) partway through the run. Nothing in the failing MRs edits that file, so something in the job changes a file in the worker's module graph while it runs. The first suspect is a sibling test or build step writing under packages/e2e/src/: compare FU-E2E-CROSS-SERVICE-BUNDLES-IN-SRC, where esbuild bundles are left under src/. Seen on unrelated MRs, including catalogue/docs-only !273 (16758514869), !266 (16759542448), !268 (16757428811), !272 (16760323616) and !275 (16764206430). To close: find the writer (for example, by watching packages/e2e/src for writes during a CI-equivalent run), keep generated output out of the worker's module graph, and pin that with a check.
FU-FLAKE-CF-D1-DO-SEQUENCE-PRESERVE: ai-sdk-d1-with-js.cf.test.ts interrupt/resume stream never reaches ended in 5s — ✅ fixed (CI health 2)
Fixed (CI health 2): a wall-clock bound, not a stream that never ends. The test interrupted Run 1 after a fixed 150ms sleep, so on a loaded runner Run 1 was interrupted before it consumed its scripted text and Run 2 ran the slow 2s tool instead; waitForStreamStatus then gave up after 5s. The test now interrupts once the tc-abort tool_start chunk is on the stream (waitForStreamChunk), and waitForStreamStatus has no deadline (the test timeout catches a stream that never ends).
Owner: B8 (CI health). Not caused by any one MR.
"should preserve chunk sequence across interrupt/resume with DO" fails with Timeout waiting for stream 'do-sequence-preserve-…' to reach status 'ended' after 5000ms. Seen on unrelated MRs: !268 (16757428811), !269 (16758270746), !272 (16760323616) and !275 (16764371786). It is either a wall-clock bound too tight for a loaded runner (the same family as FU-FLAKE-CF-DO-STREAM-WALLCLOCK-BOUND) or a real resumed stream that is never ended. That must be decided before the bound is widened. If the stream truly never ends, it is a product finding for the conformance owner.
FU-FLAKE-CF-D1-GETCHUNKS-AFTER-RESUME: ai-sdk-d1-with-js.cf.test.ts "should filter chunks correctly with getChunksFromSequence after resume" — ✅ fixed (CI health 2)
Fixed (CI health 2): confirmed cause. With the sleep at 0ms (a loaded runner) the original test fails every time with the CI signature ('Filter Run 1 contentFilter Run 2 cont…' not to contain 'Run 1'): Run 2 consumed Run 1's scripted text. All four sleep-then-interrupt sites in the file now wait for the slow tool's tool_start chunk. 6 failures in the sweep (main 16850833009, !269, !279, !285, …).
Owner: orchestrator (re-brief). Not caused by any one MR.
The e2e test (packages/e2e/src/__tests__/ai-sdk-d1-with-js.cf.test.ts:544, assertion near :613) failed once in CI on the B2 MR (16764777038) and passed on retry (16764870705). Locally it passed 25 of 25 on both B2 and main, including under CPU load. Suspected cause: the test's real-clock setTimeout(150) before the interrupt (around :561-563) — under CI load the run can reach a different point by the time the interrupt lands, so the chunks after the resume sequence differ. To close: replace the sleep with a condition wait on the observed stream state (for example, wait until the first tool_start chunk is visible) before interrupting, and pin it with a repeated run.
FU-FLAKE-WAIT-STATUS-TRANSITION: wait-for-status-transition.test.ts real-clock race — ✅ fixed
Status: done · Closed by: branch chore/followups-flakes-and-cf-docs-demos
The suite drove status transitions via real setTimeout and polled against the real clock with a 500ms deadline. On a saturated CI worker the 100ms status-change timer can be delayed past the deadline, so a must-succeed scenario observed false ("expected false to be true") — hit on !223 test:unit. Fixed by converting to vi.useFakeTimers() + advanceTimersByTimeAsync so the outcome is a pure function of the controlled clock (the production helper's deadline behavior is correct and unchanged). Reproduced deterministically by blocking the event loop past the deadline; verified 0/30 failures under heavy event-loop load.
FU-FLAKE-REMOTE-DUP-START-RACE: duplicate-/start tests raced the child run to completion — ✅ fixed
Status: done · Closed by: branch fix-remote-dup-start-race (!271) · Jobs:test:integ:e2e 2/4
Symptom: remote-agent-flow.integ.test.ts › "duplicate start rejection › duplicate start throws ALREADY_RUNNING for active session" failed at its third /start with Remote agent start failed (409): {"error":"Session idem-session already completed","code":"ALREADY_COMPLETED"} instead of RemoteAgentAlreadyRunningError. 10 of the last 40 pipelines hit it, on main (jobs 16755525951, 16747461115) and on 7 MRs (16756192309, 16756128346, 16756105472, 16756403531, 16756029835, 16756279865, 16755705432, 16755700151, 16750110303). The earliest occurrence is job 16721832769 (!254, 2026-09-24), before any change on the path.
Root cause (test defect, not a product regression): the test assumed the child session was still active when the duplicate /start calls arrived, but its un-gated MockLLMAdapter queue (text, then __finish__, commented "Slow") completes in roughly 30 ms. The third /start, two HTTP round trips later, sometimes arrived after completion, and AgentServer.startAgent then correctly answered ALREADY_COMPLETED. Between the last green main (8a6114093c) and the first failures, nothing changed in agent-server, runtime-js, store-memory, MockLLMAdapter, the HTTP transport, the remote-agent helper or the test file, and vitest's shard-2/4 file set stayed the same. The race dates from f6cf57aa91 (2026-08-03), which added the third /start. The onset coincides with CI load changes (the conformance jobs added by !254 run on the same runner pool), not with a code change.
Fix: the test now holds the child's first model call on a ScriptedModel hold gate, waits for whenHeld, asserts that the session is active, and makes both duplicate calls. It then releases the gate, waits for completed with nothing left scripted, and asserts that a later duplicate gets ALREADY_COMPLETED. Two sibling tests made the same assumption and got the same kind of fix:
remote-agent-matrix.integ.test.ts› "ALREADY_RUNNING → attach" holds the JS, Temporal and DBOS producers' nextgenerateStep.agent-serverhttp-flow.integ.test.ts› "POST /start returns 409 for already-running session" usesScriptedModelhold.
A 300 ms (flow and agent-server) or 5 s (matrix) delay before the duplicate call reproduced the failure on the old tests (flow 10/10, agent-server 10/10, matrix 8/9 cells). With the same delay, the fixed tests passed every time. After the fix, all three tests passed 30/30 on a normal machine and 10/10 under CPU load (one yes worker per core).
FU-FLAKE-WORKERD-INTROSPECT-TEARDOWN: vitest-pool-workers workflow-introspection teardown race — open (benign log; no retry)
Status: open (no safe in-code fix) · Jobs: test:workflows (full-execution.workflow.test.ts), test:cloudflare / test:workflows (lifecycle-hooks-parity.wf-noiso.test.ts)
Symptom: intermittent job failure citing jsg.Error: Instance dispose (workerd/api/actor-state.c++: broken.outputGateBroken) or Network connection lost, emitted when the workflow Durable Object actor is disposed at test/isolate teardown.
Root cause (common): these tests start a REAL Workflow instance and read its terminal status via cloudflare:test's introspectWorkflowInstance(...). Disposing the workflow Durable Object actor races with the actor's output gate, emitting Instance dispose / broken.outputGateBroken from workerd C++ (io-context) — NOT a JS-catchable rejection, so it cannot be suppressed from test code. Locally the tests pass and the log is benign; on a loaded CI runner the race nondeterministically escalates to a job failure (Network connection lost). The two affected files dispose the actor on different schedules:
full-execution.workflow.test.tsruns underisolatedStorage: true(vitest.workflows.config.ts), where the introspection handle MUST be disposed per-test (await using) for the storage-isolation teardown assertion to pass — so the actor is disposed at the END OF EACH TEST. Removing theawait usingis NOT an option: it breaks that assertion (verified — 2 tests drop to failing).lifecycle-hooks-parity.wf-noiso.test.tsruns underisolatedStorage: false(vitest.workflows-subagent.config.ts; the.wf-noisosuffix exists specifically to sidestep the isolatedStorage teardown assertion) and does NOTawait usingthe handle — the actor is instead disposed at ISOLATE TEARDOWN.
Either way the disposal-vs-output-gate race is the same, and no test-code change closes it.
Investigated (branch chore/followups-flakes-and-cf-docs-demos): reproduced the teardown exception locally for full-execution.workflow.test.ts (fires on every run as benign stderr — tests still pass 7/7 at the vitest level); confirmed removing its await using breaks the isolatedStorage teardown assertion.
Update (CI health 2): CI no longer retries test:workflows or test:cloudflare on a test failure, and no failure in the 164-pipeline sweep came from this race: the Instance dispose lines are logged on every run and are harmless, and the Network connection lost failures in these jobs were the loopback keep-alive race (FU-FLAKE-POOL-WORKERS-STACKED-STORAGE-LOST, fixed). If this race ever fails a test again, it is a red job to investigate, not a retry. The original mitigation was:
Mitigation (historical): retry the failed job. Consistent with the documented precedent for pool-workers / CI-capacity flakes ("the root cause was CI infrastructure capacity, not an in-code defect — there is no code change that would close it", see the post-!199 merge-train note below). A real fix would require an upstream @cloudflare/vitest-pool-workers improvement to introspection-handle disposal (currently pinned ^0.9.0); revisit on the next pool-workers bump.
FU-FLAKE-DBOS-PER-CALL-HOOKS-TURN1: per-call-hooks.integ.test.ts persistent SEND test reads the hook log before turn 1's onAgentStart is recorded — fixed by MR !259 (not yet on main)
Status: fixed by MR !259 (B8, dbos-integ-ci-unpause, GL #108), which is open and not yet on main. !259 waits on the turn-1 assistant reply (waitForAssistantText(…, 'turn-1-reply')) instead of the persisted user message, which removes the race this entry names. Move this entry to Done items when !259 merges. · Class: test-side race, reproduced on origin/main @ 3560d6c72a (3 failures in 11 runs of the file, private Postgres, a heavily loaded laptop) and on the conformance drain branch (2 in 4), which does not touch runtime-dbos. · Test: packages/runtime-dbos/src/__tests__/integration/per-call-hooks.integ.test.ts, "persistent SEND path: only the first execute()-time hooks fire on subsequent turns".
Signature: AssertionError: turn-1 onAgentStart fires the first execute()-time hooks: expected [] to include 'first' (line ~517).
Cause (suspected): the test's waitForCondition gate waits only for the session to be active with the turn-1 user message persisted. Nothing orders the onAgentStart hook step's recording before that state becomes visible, so under load the snapshot of the hook log can be taken before the hook runs.
Fix direction: make the wait condition also require the firstonAgentStart record (or poll the log with a bound), rather than widening a timeout. If the hook genuinely can fire after the turn is persisted, confirm that ordering is intended before changing the test.
Owner: B8 (MR !259).
FU-FLAKE-MINIFLARE-SYNC-PROXY-DESYNC: Miniflare Node binding proxy desynchronises under CPU contention — ✅ fixed (MR !272)
Status: fixed by MR !272 (the Miniflare 5 bump; see "Resolution" at the end of this entry) · Class: test-harness race in a third-party dependency (Miniflare's Node-side binding proxy), triggered by CI CPU contention. Not a D1 store defect. · Jobs: test:integ:e2e 3/4 on MR !267 (failed 16755572339, passed on automatic retry 16755916593); same signature earlier in test:integ:rest jobs 16746282265 (MR !257, runtime-cloudflare steps.integ.test.ts), 14622985680 (MR !222) and 14429769579 (MR !172).
Symptom: partway through a Node-pool Miniflare suite (here packages/e2e/src/__tests__/ai-sdk-d1-with-js.integ.test.ts), every later D1 call fails with D1 state loadState/createSession/getMessages failed ... AssertionError ... (message?.id === id), thrown from miniflare/src/plugins/core/proxy/fetch-sync.ts:138 under D1StateStore → db.prepare(...). The first test that trips it can instead show sessionId: undefined: its POST's execute() threw inside handleFreshTurnPath, which returns a 200 rejectedOnlyResponse (ai-sdk/src/handler/handle-chat-stream.ts:1030, :2155-2163) with no X-Session-Id header. Async calls (d1.exec in clearD1Tables) keep working, so each beforeEach still passes.
Root cause: D1Database#prepare is synchronous, so Miniflare's Node proxy serves it through SynchronousFetcher (miniflare@3.20250718.3/dist/src/index.js:13047-13114): the caller posts {id} to a helper worker thread, Atomics.waits on a shared 0/1 flag, then takes one message with receiveMessageOnPort and asserts its id. The helper posts the reply, then Atomics.store(flag, 1), then Atomics.notify (:13040-13041) as two separate steps. If the caller reaches Atomics.wait only after the helper has stored 1 (the caller was paused, e.g. GC or descheduled, after posting), the wait returns at once. If the helper is then preempted between the store and the notify, the caller has time to post the next request and wait again, and the late notify for the old request wakes it. The port is empty, the assertion fires, and the next reply is left queued. From then on every call reads the previous call's reply, so the proxy is off by one for the rest of that Miniflare instance's life, which is why one flake fails the rest of the file. The same code is in every Miniflare we install (3.20250718.3, 4.20251011.0 and 4.20260205.0). Upstream fixed it in cloudflare/workers-sdk#15552 (merged 2026-09-16, which uses per-request generations instead of the 0/1 flag). It ships in miniflare 5.20260916.0-alpha and wrangler 4.134.0; we pin miniflare@^3 in e2e, runtime-cloudflare, store-cloudflare and memory-cloudflare. Production is unaffected: in a Worker, D1StateStore gets a real workerd binding, and SynchronousFetcher only exists in Miniflare's Node proxy (getD1Database / getBindings / getPlatformProxy).
Reproduced (MR !267 DoD pass): a --require preload that pauses the main thread for 300 ms between posting and Atomics.wait, and pauses the helper 600 ms between Atomics.store and Atomics.notify, once, against a real Miniflare 3 D1 binding. Result: call 1 ok, then calls 2-10 all fail with (message?.id === id). Without the preload, 10/10 pass. The suite itself passed 3/3 locally unmodified. Not a regression: it occurs in May 2026 MR pipelines (before the pnpm migration), and no recent main commit touches the proxy path.
Mitigation: the .integration_base automatic retry (retry: max 1, when: script_failure, .gitlab-ci.yml:165-171) absorbs it. The retry started 2 s after the failure. Fix options: move the Node-pool Miniflare suites to a Miniflare with the upstream fix once a stable release carries it (or a pnpm patch of SynchronousFetcher backporting the generation counter); or in the harness, rebuild the Miniflare instance when a call throws this assertion instead of letting the rest of the file cascade.
Resolution (MR !272): the Node-pool suites (e2e, runtime-cloudflare, store-cloudflare, memory-cloudflare) moved from miniflare@^3 to miniflare@5.20260925.0-alpha, the version wrangler@4.141.0 ships. Its SynchronousFetcher carries the upstream generation fix (Atomics.store publishes (id + 1) | 0 and the caller waits for its own generation), so a late notify can no longer wake the wrong call. The .integration_base script_failure retry that used to absorb this is gone (same MR). Evidence: ai-sdk-d1-with-js.integ.test.ts 10/10 green on the bump (7 on an idle machine, 3 under a 10-process yes CPU stress), no (message?.id === id) in any run. The deterministic --require preload repro was not re-run (it targets the Miniflare 3 code path, which is no longer installed).
FU-FLAKE-DO-STREAM-CONTRACT-TRIMMED-AFTER-ATTACH: CU.trimmed-after-attach reads [] on a loaded runner — ✅ fixed (CI health 2)
Fixed (CI health 2): the case reads the three retained chunks, then reads until the reader's own caughtUp resolves (no 300ms window). The whole contract lost its wall-clock windows; see FU-FLAKE-CF-DO-STREAM-WALLCLOCK-BOUND.
Owner: orchestrator (routing). Not caused by any one MR.
runtime-cloudflare's test:cloudflare fails do-stream-manager-contract.cf.test.ts > "CU.trimmed-after-attach: backlog chunks cleaned up after attach are not waited for" with AssertionError: expected [] to deeply equal [ 1, 2, 3 ] (the case is in core/src/testing/stream-manager-contract.ts, shared by every stream manager). The case gives each it.next() a 300 ms window (settledWithin(..., 300)) and stops reading at the first one that does not settle, so [] means the FIRST backlog read did not arrive in 300 ms. On the DO, every read is a runInDurableObject proxy call. Seen on !287 at 650491a381: job 16913998801 failed, and the job's own retry: when: script_failure passed it as 16914066439. That runner was loaded: the same run's G1.concurrent-writers took 4.6 s. It did not reproduce locally on main bd26dc9b57 or on !287's head: the file alone 30/30 on each, and the whole runtime-cloudflare test:cloudflare suite 10/10 on each (alternating). main's test:cloudflare also fails on its first attempt in several recent pipelines on other DOStreamManager contract cases (e.g. 16882767276). To close: replace the wall-clock window with a deterministic end of the backlog (the reader's catchUpHead / caughtUp, which the next assertion already uses), keeping the case's point: chunks cleaned up after attach are never waited for.
FU-E2E-DBOS-LOAD-SENSITIVITY: DBOS e2e integration cells fail under parallel turbo load — open
Update (CI health 2): reproducing it (the three named files' DBOS cells on a fresh Postgres, under CPU load) failed 3 of 4 runs, but on a different cause: concurrent first DBOS.launch()s racing DBOS's schema migration (FU-E2E-DBOS-LAUNCH-RACE, fixed: 15 of 15 fresh-database runs green after). The "Workflow … has been cancelled" cells did not reproduce in those runs; they were only ever seen under a full local turbo run test:integration (every package at once). Left open until reproduced that way.
Seen again (W1 MR3 matrix, 2026-10-13, product 6c21449ba7): a full local turbo run test:integration --continue --concurrency=2 failed three DBOS cells of cross-runtime-resume-matrix.integ.test.ts: "path 3: submit-tool-result then resume picks up the run" and "B2 C3 (input): resume(with_message) after a stop" (Workflow … has been cancelled, thrown from DBOS setEvent under publishRunStart, runtime-dbos/src/workflows/shared.tsrunAgentLoopOneTurnBody), and "path 6: replay terminal stream after run completes" (expected 'failed' to be 'suspended_client_tool'). The file's DBOS cells alone, same tree and infrastructure: 13/13, twice.
Owner: orchestrator (routing). Not caused by any one MR. Seen locally, not in CI.
Under turbo run test:integration (all packages at once, --concurrency=2), a few DBOS cells in @helix-agents/e2e fail, a different set on each run, and pass when the same files run alone. On !287's head (650491a381) the run failed five, all passing package-direct (the e2e DBOS files 272/272) and in CI:
cross-runtime-resume-matrix.integ.test.ts(dbos + postgres + redis): "B18/A2 C7 (terminal route)" and "B2 C7 (interrupted route, completion)" (expected 'failed' to be 'completed'), and "two concurrent resume(from_checkpoint) on a suspended session — exactly one wins" (Workflow … has been cancelled);session-model-branching.integ.test.ts('dbos-postgres'): "C11: branching with a checkpointId from ANOTHER session is rejected typed" (Workflow … has been cancelled);remote-agent-matrix.integ.test.ts(producer agent-server/dbos): "late-join snapshot + fromSequence patches == terminal loadState" (Matcher did not succeed in 30000ms).
Two failures report a DBOS workflow cancelled mid-test, which suggests one cell's cleanup (or the shared DBOS executor's shutdown) cancelling another cell's workflow when the machine is slow. To close: reproduce under load (for example the e2e DBOS files with a CPU hog alongside), find who cancels the workflow, and fix the isolation or the bound.
Conformance catalogue leads (not yet catalogued)
Leads the conformance owner has not verified yet; each becomes a finding (or is withdrawn) in a later catalogue batch.
- LEAD-CF-DO-REMOTE-CHILD-CLIENT-TOOL-RUNS-ON (owner batch 29): a remote sub-agent cannot use a client-executed tool by design, and each runtime's remote dispatch should fail fast with
RemoteSubAgentClientToolUnsupportedError(docs/internals/concepts.md, Remote Sub-Agents). Seen once on cf-do while writingchat.send-during-live-run-awaiting-children(CP-13): the parent's sub-agent child, run through the DO's remote rewrite, answered its first model call with a client-executed call and then made a second model call instead of stopping. The scripted model had no second response, so the parent's tool result wasRemote agent error: MockLLMAdapter: No more responses configured. Call count: 2(helixToolFailed), not the typed unsupported error. To verify: a host cell whose remote child calls a client tool, asserting the parent's tool result carriesfailureReason: 'client-tool-unsupported'and the child makes no second model call.
(FU-CONF-STOPWHEN-PARITY was catalogued as DI-29 in batch 6 and moved to Done items.)
Conformance follow-ups from !288 (owner-approved designs)
FU-CONF-CFDO-SUBAGENT-NAMESPACE: the cf-do conformance worker cannot host a persistent companion — done (owner batch 26)
Done in owner batch 26: the cf-do worker passes its own AGENTS namespace as subAgentNamespace (the documented wiring), so companions and sub-agents run in sibling DOs that resolve their agents through the same bundled registry. The full cf-do lane is unchanged with it (every verdict as ledgered). New host-tier cells: companion.spawn-discarded-on-stop-host (CP-106's DO half, ledgered red) and companion.child-stream-closed-host (MA-22, case b). The host driver may now observe and read the stream info of a sub-session of a session it ran. The history below is kept as written.
- What: the cf-do conformance worker binds only
AGENTS(harness/cf-do/miniflare-backend.ts) and passes nosubAgentNamespace, and DOAB refuses persistent companions without one (runtime-cloudflare do/durable-object-agent-base.ts). So no cf-do cell can spawn a persistent companion. - Why it matters: CP-106 (a spawn step discarded on interrupt leaves its ref and child) was reproduced live on the CF DO by WE-2, but its cells (
companion.spawn-discarded-*) run only at the executor tier, where the DO has no lane. The DO half stays unproven by a cell until the worker can host a companion. - Fix direction: bind a sub-agent namespace in the cf-do worker (the conformance DO class, or a sibling DO class for companions) and pass it as
subAgentNamespace; then add a host-tier sibling of the CP-106 cells on cf-do. - Owner: the conformance owner (harness work).
FU-CONTEXT-ORPHANED-TOOL-ITEMS: queued tool items whose call a rewind removed stay undelivered — open
Owner: Additive Sessions (WE-2). Filed: 2026-11-09 (MR 4a, ruling 2026-11-09).
- What: since MR 4a a tool's queued context item is delivered only while its call is in the log (spec §9.4 as amended, core
toolItemCall). A rewind that removes the call leaves the item undelivered for good: the pass skips it each time, at the cost of one more listed item per pass. - Done when: such items are cleared (marked void at the truncate that removes their call, or dropped by the pass), on every store, with a unit that a skipped item no longer lists.
FU-SUBAGENT-COMPLETION-NOTICE-CONTEXT-ROWS: sub-agent completion notices are still hidden user rows, not context rows — open (deferred by ruling R9)
Filed: 2026-11-10 (orchestrator; the Additive Sessions spec 2026-11-05 §23 ruling R9 says to open it).
- Today: a sub-agent or companion completion notice is written as a hidden
userrow, outside thecontextrow model Additive Sessions introduced (spec table at :543, ruling R9 at :1027). - Why it was deferred: their delivery sits outside the checkpoint, and
planRetry(which never treats a hidden notification as a trigger), CP-106'sfindCommittedCompanionNoticeand each runtime's delivery path depend on the current shape. Moving them is its own job. - Scope when taken: the row shape,
planRetry's trigger rule, the committed-notice readers and every runtime's delivery path, together.
FU-STOP-GUARD-OUTCOME-PERSISTED: the stop guard's error detail and continue count are on the live result only — done (RM-132)
Owner: the stop guard (WE-2). Filed: 2026-11-09 (the stop guard's MR, orchestrator ruling 2026-11-09).
- What: the guard's outcome is persisted as the run's
completionReason(stop_guard_exhausted/stop_guard_error), so a reconnect and the DO's/statussee it. The guard's own error detail (underonError: 'stop') andcontinuesUsedare on the liveAgentResult.stopGuardand theonAgentCompletepayload only, and in thecore.stop_guard.*log events and the span. A reconnect, and every caller of a Durable Object, cannot read them. - Done when: the run record carries them (a field on every store, with a migration and a contract id), a reconnect's
AgentResult.stopGuardis rebuilt from it, and the runtime-js unit pinning "a reconnect has no stopGuard" is restated. - Done (batch 21, WE-2's !844 cells and !845 fix; RM-132):
RunMetadata.stopGuardis written with the run's terminal status on every store (Postgres V33, D1 V35, DO-SQLite 31; Redis and memory with the run), also by recovery from the forced phase that ended the run; a reconnect'sAgentResult.stopGuardand the DO's/statusread it; the runtime-js unit is restated.
FU-STOP-GUARD-RUNTIMES: CF Workflows, Temporal and DBOS refuse a completion-guard agent — open
Owner: each runtime's Additive Sessions MR. Filed: 2026-11-09 (the stop guard's MR). Updated: finishing API v3 (user rulings 2026-11-14) replaced the stop guard with the completion guard; this item now tracks it.
- What: the completion guard (
onOutput,onReply) runs on runtime-js and the CF DO. CF Workflows (Additive Sessions MR 5), Temporal (MR 7) and DBOS (MR 8) refuse an agent tree that setscompletionGuardat every root entry (UNSUPPORTED_AGENT_FEATURES), and thecompletionguard.*cells are declared gaps on their lanes, exceptcompletionguard.refused-unwired, which judges that refusal there (lane flagcompletion-guard-refused). - Done when: each runtime calls
isReplyGuardOffered/runReplyGuardat its terminal-text site andrunOutputGuardat each finish site (CF Workflows inside the LLM step'sstep.doand its tool steps, the verdict recorded with the step result; Temporal inrunLLMStepand its finishWith phase, applied behind awf.patchedid, alongside FU-CFD1C-DI29-DI25-TEMPORAL; DBOS inrunCallLLMand its tool steps), commits the outcome with the judged step, writes the accepted-output marker and the completion reason, removes its phase from the table's row, and moves its lanes fromcompletion-guard-refusedtocompletion-guard(the refusal cell's gap and the positive cells' gaps flip together). - Note (FS-3 / FS-4 / FS-5): writing the accepted-output marker on CF Workflows, Temporal or DBOS changes that runtime's
completionKeyfor an output from its reply row (${runId}:reply:${index}) to${runId}:${acceptedCallId}. That is fine: the key is documented as opaque, and it stays the same on every re-fire within one version.
FU-SEVERAL-FINISHWITH-FORCED: forced completion and Final Step offer only the first finishWith tool — done
Owner: finishing v3 MR 3. Filed: 2026-11-15 (finishing API v3, decision 5 of user rulings 2026-11-14).
- What: forced completion and Final Step name the agent's first
finishWithtool only (coreforced-completion.tsresolveExpectedTool), so an agent with severalfinishWithtools can only finish through the first once forced. v3 documents it and logscore.completion.several_finish_with_toolsonce per agent (warnSeveralFinishWithTools; runtime-js at its completion refusal check, CF Workflows, Temporal and DBOS at entry). - Done when: on every runtime, forced completion offers every
finishWithtool (adapter-level forcing throughcapabilities.forcedToolChoice,native_outputas a union of their input schemas, Final Step naming them all), an agent whose union a provider profile cannot represent fails typed (framework_completion_schema_unsupported) at first use before any model call (a unit test per profile), and the warning is removed. - Done: finishing v3 MR 3. Every runtime offers every
finishWithtool (resolveExpectedTools;forced_toolnarrows to the set with'required'or the set-shaped choice,native_outputasks for thecompletionenvelope, Final Step names them all).checkCompletionSupportchecks the envelope at entry (framework_completion_schema_unsupportedbefore anything is written; llm-vercel units per Anthropic native profile) and the warning is gone. Cells:forced.several-finishwith-{tool,native},final-step.several-finishwith-named,forced.single-finishwith-unchanged.
FU-COMPLETION-POST-COMMIT-HOOK: no exactly-once point after a run's terminal commit — open (future work)
Owner: unassigned. Filed: 2026-11-15 (finishing API v3, decision 4 of user rulings 2026-11-14).
- What:
onAgentCompleteruns before the run's terminal commit and is at-least-once: anonAgentFailoronAgentSupersededcan follow it, and a recovery can call it again. v3 gives itcompletionKeyand documents a dedupe recipe; an app has no hook that runs once, after the commit. - Done when: a durable post-commit hook (outbox semantics: recorded in the K5 commit, delivered after it, redelivered after a crash, at-least-once with a stable key) exists on every runtime, or the item is closed by a ruling.
FU-COMPLETION-KEY-REFIRE-CONTROLS: no conformance control re-fires onAgentComplete on runtime-js's executor lanes, CF Workflows or DBOS — open
Owner: unassigned. Filed: 2026-11-15 (finishing API v3 MR 1, orchestrator ruling 2026-11-15).
- What:
completionguard.completion-key-refireproves decision 4 (the samecompletionKeyon a re-fire) only where a lane can force the hook to fire twice for one completion: Temporal ('fault-terminal-hook-stage': the record of the fired terminal hooks fails once and the retried activity fires them again) and the CF Durable Object (completionguard.completion-key-refire-host: evicted inside the hook, woken). The other lanes are declared gaps: runtime-js's executor lanes have no process loss (the DO wake is their re-fire), CF Workflows caches the hook's ownstep.do(only an engine crash inside that step re-runs it), and DBOS re-fires only on workflow recovery after a process loss. Each runtime's own units cover the key's read; no cell covers those re-fires. - Done when: the CF Workflows lane can crash the engine inside a step and the DBOS lane can recover a workflow from a killed process (or an equivalent control each), and the cell runs there; or a ruling closes the item.
FU-CONF-STEPCONTEXT-REFUSAL-CELL: no conformance cell judged the stepContext entry refusal — done (lands with its MR)
Owner: Additive Sessions (WE-2). Filed: 2026-11-09 (the stop guard's MR, orchestrator ruling 2026-11-09).
- What: CF Workflows, Temporal and DBOS refuse an agent tree that sets
stepContextat every root entry (coreUNSUPPORTED_AGENT_FEATURES, Additive Sessions MR 4), pinned by core units only. Final Step and the stop guard each judge their refusal on those lanes with a cell (final-step-refused/stop-guard-refused);stepContexthas none, so a runtime that dropped itsrefuseUnsupportedAgentFeaturescall would still pass thecontext.*gaps. - Done when: a
context-pass-refusedlane flag on the three lanes and acontext.refused-unwiredcell in the same shape (.entry-settles,.refused-typednamingstepContextand the phase,.nothing-written), recorded and judged, with the js lanes' gaps declared. - Done: lane flag
context-pass-refused(CF Workflows, Temporal, DBOS; the complement ofcontext-pass) andcontext.refused-unwiredonconf-cpx-hook-echo, in the same shape asstopguard.refused-unwired; the js lanes are declared gaps for it. When MR 5 / 7 / 8 wire the context pass, a lane moves fromcontext-pass-refusedtocontext-pass, and the refusal cell's gap and thecontext.*gaps flip together.
FU-CONF-CFDO-MEMORY-PORT: the cf-do conformance worker cannot run a memory agent — open (first in the post-MR-4 harness queue)
Owner: the conformance owner (harness work). Filed: 2026-11-09 (Additive Sessions MR 4).
- What:
conf-memory's store is the Node-sidememoryStoreport (agents/memory.ts). The CF Workflows bridge wires it (harness/cfw/worker/bridge-memory-store.ts,observe-memoryonEXECUTOR_CFW_CAPABILITIES); the cf-do worker builds its agents without it, so the memory agent's store throws there and HOST_CF_DO claims noobserve-memory. - Why it matters: a lost step is reachable only on the CF Durable Object (evict / wake), so
memory.survives-lost-step-hostis the real proof that a recalled memory's row (its own commit since MR 4) survives the loss and is not injected again. Until this lands the DO holds it by inheritance only (the runtime-js executor form, through a refused step commit and a retry, is secondary).memory.oversized-skippedneeds it too: an oversized recall is reachable only where the store caps a row (store-row-cap: cf-do and CF Workflows), so MA-39's runtime-js / DO half has a core unit (context-pipeline.test.ts) and no cell until then. - Done when: the cf-do worker passes the memory port to its agents (as the CF Workflows bridge does), HOST_CF_DO claims
observe-memory,memory.survives-lost-step-hostruns on cf-do, andmemory.oversized-skipped(-host)is written and runs on cf-do. Lands before Additive Sessions MR 5.
FU-CONF-RUNTIME-JS-PLAIN-WORKER-LANE: no conformance lane runs runtime-js inside a plain Cloudflare Worker — open (post-features; owner: the conformance owner)
Filed: 2026-11-10 (orchestrator, via the conformance owner).
- The gap: every runtime-js lane runs under Node (js-memory / js-redis / js-postgres, js-chat-host), and the CF Durable Object lane runs runtime-js inside a DO, where no JS run lease is taken. Nothing runs
JSAgentExecutorin a plain Worker under workerd, so a Node-only assumption on that path goes unseen. CP-309 (the run lease's timers called with the wrong receiver) was found by a benchmark, not by a cell. - What a lane would need: a workerd-hosted runtime-js executor driver (a plain Worker with a non-DO store, e.g. D1), driven from Node like
cf-do, claiming the executor capabilities it can serve. - Priority: post-features, unless an app reports runtime-js in a plain Worker. Until then CP-309's fix is proven by a core unit and the bench's workerd smoke.
FU-CONF-EXECUTOR-LIVE-TOOL-TEXT: the executor tier cannot see a stopped call's live error text — open
Seen: CP-277 (WE-2, 2026-11-06). A cell that stops a regular tool inside a native abortable op needs the call's LIVE tool_end text, but the executor tier reads only the session's stored chunks (readStoredChunks), and an interrupt cleans the stopped step's frames out of the stored stream. On js-memory the run ended interrupted with no tool_end left to read. The tool-witness port records only that a call ran, with no payload.
- Covered meanwhile:
tool.stop-cut-native-wait-hostreads the chat route's live SSE on cf-do (workerd) and js-chat-host (runtime-js on Node). CF Workflows' in-tool reason shape is pinned byruntime-cloudflare/src/__tests__/steps.test.ts(aStopAbortErrorcarrying.stop); its text is owed. - Done when: the executor tier can read a call's live terminal text (a live-writer capture, or a witness note the tool can attach), and a
tool.stop-cut-native-waitexecutor cell runs on js-memory and cfw-workflows-d1.
FU-CONF-WIRED-COMPANION-IDENTITY: the executor driver retries a companion with a copy of the object its parent registered — open
Seen: CP-302's companion.continued-turn-retry-uses-consult-input on dbos-postgres (WE-2, 2026-11-09). The executor driver's prepare returns env.wireAgent(rebuilt), a NEW agent (harness/recorded-hooks.tswireAgentHooks), and every run op requires that wired object (requireWired). A parent's rebuild takes its companion from the factory, unwired, so the parent registers one object and the cell's retry() of the companion's own session hands the runtime another. DBOS's one-config-per-name rule (MA-13, claimAgentNames) refuses the second (framework_agent_name_conflict); the other lanes do not compare objects.
- Meanwhile: the lane capability
retry-companion-sessionis claimed by every executor lane except dbos-postgres, so that cell reads declared-gap there (gaps.ts). - Done when: wiring a parent also wires its
persistentAgentschildren and the companion's ownpreparereturns that cached wired child, with a red-first harness test; then the flag is dropped and the DBOS row runs. A separate harness MR: it starts recording companion hooks in every companion cell on every lane.
FU-CONF-CP257-RESEND-CELL: CP-257's re-send has no cell of its own — open
Owner: the conformance owner. Filed: 2026-11-01.
CP-257's only ledgered row is its conflict, so its pending entry is gone and nothing else holds this place. The re-send itself is owed its own scenario on every executor lane: pause on a client call, submit, continue as the submit says, complete, re-send the same answer, expect already_completed and nothing stored. Ledger only from identical runs, 3 of 3.
FU-CONF-RM60-HOST-CLIENT-CELLS: host / client rewind-resync cells for RM-60 — open
!288 fixed RM-60 at the executor tier (one rollback stream_resync per from_checkpoint rewind on every runtime) and dropped its unwired host / client drafts. The owner-approved design:
retryandresume-from-checkpointare separate host capabilities;- the outcome is a bounded poll of the snapshot's terminal status;
- the js-chat-host retry is a declared gap;
- a client cell may run the host op and then reload.
RM-60 stays open until these cells exist.
FU-CFW-INTERRUPT-CATCH-PATH-MARKER: CF Workflows' outer catch ends an interrupted LLM step without its step_discarded — open (owner: WE-2; CF Workflows, after the features-first window)
Seen in the batch-17 pipeline (3045815608, job 17868808273): stream.interrupted-in-flight-call-truth|cfw-workflows-d1|executor failed .announced-calls-resolved with no step_discarded at all (live: tool_arg_stream_start, two deltas, then run_interrupted), though it passed 3 of 3 in WE-2's RM-119 slot. WE-2's read at e7af182e81: the cut-call marker is written only by steps.discardStagedChanges (runtime-cloudflare/src/steps.ts:11056, :11071), and when the interrupted LLM step's step.do throws out with no step in flight, the workflow's outer catch (runtime-cloudflare/src/workflow.ts:7812 onward) ends the run through finishCommittedOutcome and handleStop (:7841-7848) without discarding the step, so the cut call is announced and never resolved for a live reader. Meanwhile the row is withdrawn: RM-119's provenBy drops it and the cell's cfw row is a declared gap on the flag 'interrupt-marks-every-exit', which CF Workflows does not claim. Done when: a held-order cell forces the outer-catch path red, the fix discards the step with its marker there, and the flag is claimed on cfw again (the gap removed). To be minted as a record in batch 18 (RM-130), which this item closes into.
FU-CONF-POLL-ABSENT-BEFORE-SEEN: status polls that crash on a session not created yet — open
Owner: the conformance owner. Filed: 2026-11-04.
persistedStatus throws SessionAbsentError for a session the store does not hold yet, and on Temporal the workflow creates a session after start() returns. state.subagent-fold-kept-with-late-notice crashed there on the Temporal lanes (its fix: pollReadFailure in scenarios/subagent-fold.ts, an absent session reads "not yet" until a poll has seen it). Four more polls read a session the same way and have the same exposure if they ever run on a lane that creates it late: companion-child-error-stream.ts:105, companion-reconsult-evicted.ts:204, companion-respawn-fresh.ts:135 and finish-siblings-host.ts:487. None has crashed so far. Fix direction: move the seen tracker next to persistedStatus and use it in all five polls.
FU-CONF-STOCK-PAUSED-END-RESPONDER: the client driver's 'paused-end' responder never answers on a stock mount — open (harness; owner: W1)
W1's RM-124 probes (probe124, probe124c, probe124d; 2026-11-09): the client driver's 'paused-end' responder waits for the Chat's status to leave 'streaming' after a run pauses, then calls addToolOutput. On the stock mount it timed out at 60 s (client.hidden-turn-pending-call-found-client-stock, all 4 client lanes). The same page shape with nobody answering reaches 'ready' 3 of 3 (client.resumed-stream-pause-unanswered-client-stock): the resumed response ends with tool-input-available, data-step-committed, data-run-paused and finish. So the product ends a resumed pause correctly, and the responder most likely waits for a status transition the stock page has already made. Untraced. Trace or fix it when the responder is next touched; until then a stock cell answers at 'input'. The two resumed-stream-pause-unanswered cells guard the product side.
FU-TEST-CHAT-STATE-NO-CLONE: four test chat states store messages by reference, so write-dependent client behaviour passes in tests and fails in React — open (test only)
Surfaced by: CP-340's pin (WA, 2026-11-19). Related: CP-340.
- The gap:
@ai-sdk/react'sReactChatState.replaceMessagestores astructuredCloneof the message (this.snapshot(message)). A test state that stores the message object itself lets a later in-flight mutation thatainever writes (astep-startpushed without awrite()) show through. So a test can pass on behaviour a React app never sees: CP-340's stock-loop pin passed at the base onai6.0.170 and 6.0.230 with such a state, and only a cloning state reproduced the loop (WA's measurement). - Where (as on origin/main 320ebf80ac):
ai-sdk/src/__tests__/helpers/headless-chat.ts(MemoryChatState, shared by 29 ai-sdk test files: handler, transport, react, transformer); and inline states inai-sdk/src/__tests__/transport/stock-chat-state-size-413.test.ts,e2e/src/__tests__/sub-agent-approval-reload.integ.test.tsande2e/src/conformance/__tests__/host-driver.test.ts. e2e'shelpers/headless-chat.tsandhelpers/stock-ai-client.ts,ui-chunk-schema-contract.test.tsandai-sdk-do-reattach-real-chat.cf.test.tsalready clone. - Fix direction (test only): make
MemoryChatState.replaceMessageclone as React does (or share one React-faithful state), move the inline states onto it, and re-run every file on it. A case that turns red there was passing on behaviour a React app does not see: file it, never weaken the assertion. - The first one exists: CP-340's fix (!949) adds a
ReactFaithfulChatState(astructuredCloneonreplaceMessage) inai-sdk/src/__tests__/handler/stock-autosend-step-boundary.test.ts; the four by-reference states move onto it or clone the same way.
FU-FLAKE-CONF-CLIENT-HIDDEN-TURN-ENDS-BEFORE-PAGE: chat.hidden-turn-after-stop-finished-client's hidden turn can end before the page loads, so the page sends no request — open (harness; owner: W1)
Seen: 2026-11-15, batch 25's merge-train pipeline 3070281623, test:conformance 1/7 (job 18043201641): chat.hidden-turn-after-stop-finished-client|js-chat-host|client unexpected-fail in 53 ms, the hidden turn did not complete: {"terminal":"none","source":"chat:ready:no-request"} (its two other assertions failed on that precondition). The cf-do cell passed in the same job, and the js-chat-host cell passed on the batch's MR pipeline 3070220982 and in its local slot.
Mechanism (traced at the batch head 0568cf0ad4): the client driver's hiddenTurn (drivers/client-driver.ts:2170-2189) awaits the server entry, tears the page down, reads the snapshot, derives the page's resume options from it (pageResumeOptions(snapshot, 'page')) and mounts. The cell's hidden turn is the second script entry of stoppedThenHidden (scenarios/hidden-turn-input.ts:349), a plain text reply with no hold, so on the in-process js-chat-host it can finish before the snapshot read. The snapshot then reads the turn ended, the page mounts with no resume, sends nothing, and the handle's outcome is none / chat:ready:no-request (client-driver.ts:403). Not a product finding: a page opened after a turn ended correctly resumes nothing.
Same shape elsewhere: no. The other client cells that run hiddenTurn hold the hidden turn's reply until the page is up (hidden-turn-input.ts:501, :523, :526, :553: holdAfterDeltas, hiddenTurn, model.whenHeld, release), or end it paused on a client call, which the snapshot reads as live (the pending-call cell, :1140). The host driver's hiddenTurn attaches to the run's stream, which replays an ended run to its end frame, and the executor driver reads the handle.
Fix: hold the hidden turn's reply as the sibling cell does (holdAfterDeltas on the second reply, model.whenHeld after hiddenTurn returns, then release), so the snapshot always reads a live turn. Red first: release the hold before the page mounts and the cell fails chat:ready:no-request on every run. Also, hiddenTurn refuses a snapshot that no longer reads the turn live (status neither active nor paused) with a HarnessError naming the race, so a future cell of this shape fails as a harness error rather than a product mismatch; a turn paused on a client call (the pending-call cell) reads paused and passes.
FU-CONF-LEDGERED-CELL-DETAIL: a ledgered cell logs no failure messages — open
Owner: the conformance owner. Filed: 2026-11-01.
The suite prints a failing cell's messages only when its verdict is unexpected, and the results files hold ids only. So once a cell is ledgered, a run shows that the same ids failed and not why: CP-257's conflict error was read before its row was ledgered, and the next run could only match the ids. A different failure at the same step would look identical. Keep each expected-fail cell's failure messages in its results, or in the log, so a run can compare the reason as well as the ids.
FU-CONF-ORPHANED-BRANCH-CITATIONS: fix citations pinned to squashed-away branch commits — open (LOW; owner: the conformance owner)
Filed: 2026-11-04 (orchestrator ruling). About 80 fix citations of the form (fix: …; as on branch <branch> <sha>), over 32 commits, name a member commit that a squash landing left off main. The finish batch's were re-pinned to its squash; the older ones stay as history. The risk: those commits are reachable only through their deleted branches, so a garbage collection on the server can make a citation unresolvable, and a reader then cannot check the line. To keep them reachable without rewriting the records: push one archive ref per commit (refs/archive/<sha>), or re-pin a record's citations to its squash when the record is next edited. New records cite origin/main <sha> once their code has landed.
Batch 21 (2026-11-11, orchestrator ruling): batch 21's C6 re-pinned every batch-19 and batch-20 member citation to its landing squash (origin/main 41ea410715 and f4f4028489; each cited line checked byte-identical, or moved to its one exact match). The older ones remain: 93 shas on 74 branches, listed here by branch as read with git ls-remote that day. A branch still on the remote is not deleted until a later C6 re-pins its records.
- Branches gone from the remote (32): their shas are reachable only while a local clone or a merge request ref still holds them.
reliability-cf-w1-cp180-submit-contract(1089168988): CP-198reliability-cf-w1-cp186-local-child-approval-reload(316c3d252e): CP-186reliability-cf-w1-cp225-failed-snapshot-error(c6233db0f9): CP-225reliability-cf-w1-cp231-submit-kind-binding(ff2d2aacb3): CP-231reliability-cf-w1-ma26-llm-step-retry(94a90437a2, 9a30379cee): CP-235, DI-35, MA-26reliability-cf-w1-rm81-new-turn-pending(e677e56aad): RM-81reliability-cf-wa-cp161-spawn-entry-outcome(07012bb291, a2dac83f2f): CP-190, CP-191, CP-192, CP-196reliability-cf-wa-cp173-174-expiry-seam(1bee877bd3, 1c28613186, 2c7f6030c4, c819c211c7): CP-197, RM-81, RM-82reliability-cf-wa-cp245(423f79a66f): CP-245reliability-cf-wa-cp40-c2(3b864439b0, bc5a339945): CP-40, CP-127, CP-134, CP-233reliability-cf-wa-cp40-c3-other-consult(109b656d82): CP-40reliability-cf-wa-cp40-c3-wait-fence(fe704ebd04): CP-40reliability-cf-wa-di80-strict-bridge(b95febb96c): DI-80reliability-cf-wa-di81-finish-parse-miss(37faeb7553): DI-81reliability-cf-wa-di81-temporal-dbos(d9dbf68e48): DI-81reliability-cf-wa-di82-strict-tool-schema(ad50baf26e): DI-82reliability-cf-wa-g1-real-retry(b370a6889a): CP-275, CP-276, RM-91reliability-w1-cp252-deadline-binding(4ab8cf8257): CP-252reliability-w1-cp274-reentry-hooks(2d41b5e1ea): CP-274reliability-w1-ma49-null-custom-state(ff87c78e29): MA-49reliability-w1-ma50-companion-defaults(77f83d7876): MA-50reliability-w1-rm124(b4eb5810c5): RM-124reliability-we-cp138-cfw-deadline(e6881d4e2c): CP-197reliability-we-cp157-remote-park(d438ffb3a7, fec5d5c49e): CP-185, CP-218reliability-we2-cp106-child-phase-mr-a(b8b37e035e): CP-106, CP-117reliability-we2-cp138-js-deadline-timer(96805fc866): CP-138reliability-we2-cp267(f0cb451b8a): CP-267reliability-we2-cp267-cfw(4c188f9e16): CP-267reliability-we2-cp277(481647df18): CP-277reliability-we2-di83(a7546fc0b7): DI-83reliability-we2-rm116-main(5b8ad1f329): RM-116reliability-we2-rm117(8854e92413): RM-117
- Branches still on the remote (42): nobody deletes these until a later C6 re-pins their records.
reliability-cf-wa-cp228(f3b7f73bd9): CP-228reliability-cf-wa-cp259(ac6faf6133): CP-259, DI-59reliability-cf-wa-cp305(82fc44d022): CP-305reliability-cf-wa-cp60(6ff080c5ab): CP-60reliability-cf-wa-dbos-di19(6c03f6ab8b): DI-19reliability-cf-wa-dbos-rm126(b9522383f2): RM-126reliability-cf-wa-dbos-rm75(539db19460): RM-75reliability-cf-wa-dbos-rm83(895c342c53): RM-83, RM-129reliability-cf-wa-di85-error-stop-coded(9df469081a): DI-85reliability-cf-wa-finish-siblings-1-core-js(1062387aeb): RM-113reliability-cf-wa-forced-msg-tool(e89f9f02bb): DI-84reliability-cf-wa-late-answers-1c-temporal(3789d47333, 67485cadfb): CP-263reliability-cf-wa-rm121-flush(cffc53ec56): RM-121reliability-cf-wa-rm122-turn-stops(0e46353a10): RM-122reliability-cf-wa-rm123-queued-stop(74b4ddecc6): RM-123reliability-cf-wa-sft-cells(be93b00669): RM-121, RM-122, RM-123reliability-cf-wa-sft-mr1(8d06533c7b, a9ef6b573d, f46938af8b): CP-194, CP-216, CP-280, MA-31, RM-69, RM-91reliability-cf-wa-temporal-rm128(ae0167624a): RM-128reliability-cf-wa-temporal-rm88(9d27696637): RM-88reliability-cf-wa-tool-stopped-cfw(3648385f3c): DI-89reliability-w1-cp278-lost-claim-class(8bc64eef9f): CP-278reliability-w1-cp279-companion-outlives-root(a80d04453c): CP-157, CP-214, CP-279reliability-w1-cp281-cross-process-stop(064e4315e9): CP-281reliability-w1-cp282-on-main(52ef0bba6c): CP-282reliability-w1-cp285(cccb64745b): CP-285reliability-w1-cp286(a08811899c): CP-286reliability-w1-cp287(a92bf589f8): CP-287reliability-w1-cp289(c69db938de, eb7adb6e73): CP-289, CP-296, CP-298reliability-w1-truncation(2dbfba537e, 61f91f9fd3, f8570cfd61): DI-24, DI-26, DI-27, DI-88reliability-we2-additive-mr1(2a06d55fa6): CP-284reliability-we2-cp280(3e24192ad7): CP-280reliability-we2-cp280b(74df351c93): CP-280reliability-we2-cp283(869626e341, bd7def2abb): CP-283reliability-we2-cp284(564e709e34, 7c64340f2b): CP-284reliability-we2-cp288(21367591535f, 507f2a332d): CP-288, CP-290reliability-we2-cp291(5e221e56db, b99f7f8d0c, daf6ab4e72): CP-291, CP-292, CP-293, CP-295reliability-we2-cp293(e32c9b95d6): CP-293, CP-295reliability-we2-cp297(84737aac5f): CP-297, CP-306reliability-we2-late-answers-1b(cdd653d36a): CP-260, CP-263reliability-we2-late-answers-1c-dbos(13957ef575, 72d4cdefde): CP-264, DI-24reliability-we2-rm119(abd68aff41): RM-119reliability-we2-rm125(f035cb623b): RM-125
Batch 23 scan (2026-11-13, at origin/main f6378fcb05): batch 23's own member citations were re-pinned to its squash (DI-92, RM-133, DI-93). The scan still finds 417 citations pinned to a member head that a squash left off main, in 86 records over 86 shas:
- on a sha some remote ref still holds (61 shas, 330 citations, 68 records): 76 still byte-identical at main, 173 moved to one exact match, 62 whose text is no longer at main, 7 with more than one match, 12 whose file no longer resolves;
- on a sha no remote ref holds (25 shas, 87 citations, 23 records): 29 identical, 49 moved to one exact match, 7 gone, 2 unresolved.
Method (read-only): every file:line citation in a record whose pin parenthetical names as on !N <sha>, as on branch <name> <sha> or as on the same branch … <sha> (attached, after list glue, or enclosed after the pin, as pinned-refs.ts reads origin/main pins); its lines compared byte for byte between the pin sha and main. A sha counts as held when, after git fetch --prune, any refs/remotes/origin/* ref contains it. File-level citations (no line) are not counted. The pass: re-pin each record's identical and one-match citations to the squash that landed it (origin/main <sha>), keeping the numbers when identical and moving them only to the one exact match; the gone, ambiguous and unresolved ones stay pinned to their member sha (history), after an archive ref keeps that sha reachable.
FU-CONF-PINNED-REFS-CHAINED-CITES-UNFROZEN: cites a record pins in prose ("the same pin") are live to conformance:pinned-refs — open (LOW; owner: the conformance owner)
Surfaced by: the post-batch-32 re-pin (2026-11-18), as its control.
- What the tool freezes: a cite counts as pinned only when an
origin/main <sha>parenthetical ends its list or directly follows it, or when the cite sits inside the parenthetical right after the pin (pinned-refs.tsparseCitations/pinOf; the authoring guide's "Catalogue citations: pins are frozen"). Records also extend a pin in prose: "(the same pin)", "the same pin: a.ts:1, b.ts:5". The tool reads those cites as live, so it neither checks nor freezes them, and a C6 pass would treat them as live. - Evidence: on the re-pin commit,
--basethat commit, moving RM-138'sstore-redis/src/commit-script.ts:1272-1278(a cite after "the same pin:") to:1273-1279left the tool green. Moving CP-302'srun-loop.ts:11140, which its pin directly follows, to:11141failed it ("is pinned to fae53b26ac"). By the tool's own parser, 91 cites in 29 records sit in strings that say "same pin" and carry no pin (among them CP-333's 9 entry cites and RM-138's 8 store cites). - A second instance (2026-11-20, the post-b35 re-pin): the commit's control moved a cite that follows "the same pin" and the tool stayed green, so it proved nothing. The control was re-run on a directly pinned cite, which failed as it should. Until this is fixed, a re-pin's control must move a directly pinned cite.
- Not this gap: an
as on !N/as on branchcite is provisional by design (the guide says the tool neither checks nor freezes it); the records commit after each landing converts it by a manual re-derive. - Fix direction (not chosen): either
pinOftreats a "the same pin" phrase as the nearest preceding pin, with a test, or records repeat the pin parenthetical on each list. The first keeps the records' prose; the second needs no tool change. A fix starts with a list of the 91, each re-derived at its pin before it is frozen.
FU-CONF-JS-PROCESS-RESTART: a runtime-js process-restart control for #177's start-hook durability — open
#177 spec §6 proves hooks.start-writes-durable on cf-do (an eviction during the first model call) and on runtime-js by a process restart. The conformance harness has no runtime-js restart: its executor lanes run the executor in the test process, and nothing can kill it mid-turn and recover the session in a fresh executor over the same store. The cf-do cell hooks.start-writes-durable-host covers the DO, which runs runtime-js's executor; the executor lanes and js-chat-host are declared gaps. A restart control would build a second executor over the lane's store, abandon the first with its turn held, and recover the session in the second.
FU-CONF-CLIENT-TIER-PROVIDER-ROUTING: no client-tier cell runs the real model adapter — open (LOW; owner: the conformance owner)
Surfaced by: RM-48's cells in the b37 cells MR (2026-11-20).
- The provider mock (
'provider-connection-drop','provider-scripted-finish';harness/provider-routing-adapter.ts) is claimed at the executor and host tiers only, so RM-48's UI truncation cells (truncation.provider-ui-finish-length-hostand-default-host) read the chat frames at the host tier. - No cell checks what a real
useChatmakes of a reply the real adapter cut at the limit (its message, its finish reason). The client tier runs only on the scripted model, which reports steps itself and hid RM-48. - Fix direction: claim the provider mock at the client tier on both hosts (the routing already runs on the host's model path), then add client twins of the truncation UI cells.
FU-CONF-SANDBOX-CELLS-NODE: the cloudflare-sandbox cells run on the CF DO only — open (LOW; owner: the conformance owner)
Surfaced by: the sandbox cells of the b38 cells MR (2026-11-21). Related: DI-99, DI-100, DI-101, CP-342.
- The four
sandbox.*host cells (conformance/scenarios/sandbox-workspace.ts) run on cf-do, where the conformance worker opens thecloudflare-sandboxworkspace provider over oneFakeSandboxper session (harness/cf-do/worker/sandbox-fakes.ts, the'sandbox-fake'capability). js-chat-host declares all four as gaps (conformance/gaps.ts), so the shell's streaming path is pinned on one host only. - Hosting the provider in Node needs it to load and run there over the fake; that is not checked yet.
- Fix direction: a Node-side
'sandbox-fake'on js-chat-host that opensCloudflareSandboxWorkspaceProviderover aFakeSandboxfrom@helix-agents/runtime-cloudflare/testing, then drop the four gaps. The value is modest: the DO is the provider's shipped home.
FU-CONF-SHARD-NEAR-JOB-DEADLINE: a conformance shard came within minutes of its 1 h job limit on a slow runner — open
Owner: the conformance owner. Filed: 2026-11-01 (orchestrator ruling: track it).
test:conformance 2/7 on main ac42f447d5 ran 3238 s (job 17641492401, failed) and 2804 s on its retry (job 17642229519, 95 of 95), both on do-nyc3, against 1141 s on !624's green job 17640732425 on the same runner. At that speed a shard has about 6 minutes of margin, and a job killed at its limit reports no cell verdicts. Next time a shard passes 3000 s, compare the runner's load first; if it recurs on a quiet runner, rebalance the shards or raise the limit.
The shard is slow on main, not everywhere: on the same runner, test:conformance 2/7 took 3238, 2804 and 2698 s in main's pipelines (the last, job 17644937194, failed lifecycle.get-handle-live-turn on cfw-workflows-d1, a cell taking 6 s on !633, at its 30 s bound) against 1141, 1528 and 1176 s in MR pipelines (!624, !626, !633), while main's other shards in that last pipeline took 1043 to 1494 s. Cell by cell against !633, that main job ran a median 3.3 times slower. The cause isn't known. Finding why main's run of this shard differs comes before raising any cell's bound.
A related timing limit: the conf-unit test session.create-observed-whole > abandon(): a stuck attempt's observers stop reading once its bound fires runs fake timers on shouldAdvanceTime and has a 30 s real-time limit. It hit that limit once locally just after a heavy host load peak, and took 11.5 s in the next run. Its bound stays as it is unless it recurs on a quiet host.
FU-CONF-DBOS-REMOTE-PRODUCER: the DBOS lane has no remote-child producer, so DBOS's remote sub-agent path has no cell — open
Owner: the conformance owner. Filed: 2026-11-06 (CP-280).
CP-280's DBOS remote half (the parent's call announced and ended on DBOS's remote path) is proven by a unit only (runtime-dbos/src/__tests__/subagent-call-announced.test.ts, red at the pre-fix product and green at the fix), because dbos-postgres does not claim remote-child. The member: a remote-child producer on the DBOS lane (agent-server over a real transport, as harness/js-remote-producer.ts does on the js lanes), the capability claim, a self-test, the pins and test:conformance-harness. Then subagent.local-call-announced's remote sibling runs on dbos-postgres. The same producer would reach CP-216's lost-delegated-row cells on DBOS (see FU-CONF-TEMPORAL-REMOTE-PRODUCER for the Temporal lanes).
FU-CONF-TEMPORAL-REMOTE-PRODUCER: the Temporal lanes have no remote-child producer, so RM-114 has no cell — open
Owner: the conformance owner. Filed: 2026-11-04.
RM-114 (a Temporal step commits a remote result before the reply that holds its call) is open with no cell because no Temporal lane can run a real remote child. The member: a remote-child producer on the Temporal lanes (agent-server over a real transport, as harness/js-remote-producer.ts does on the js lanes), the remote-child capability claim, a self-test, the pins, and test:conformance-harness. Then the cell: the committed row order, and that the next model request is well paired.
The same producer makes a second cell reachable: a message-carrying entry over a lost delegated row with the session active (CP-216). Today no Temporal or DBOS lane claims remote-child, so each runtime's delegation-commit.integ.test.ts proves it through the real resume(); write the executor cell with this member.
FU-CONF-HOLD-K5-BEFORE-RUN-STATUS: a control that holds a child between its K5 and its run-status write, and CP-159's stop-order sibling cell — open
Owner: the conformance owner. Filed: 2026-10-19 (orchestrator ruling: file it, don't build it now).
Why. CP-159's fix (!452) records a signalled local child's parent ref from the terminal its cascade observes (recordChildTerminalRef in runtime-cloudflare stop-children.ts). Its completed arm, "a signalled child that finished before it handled the stop", is reachable in one window only. In that window the child's K5 is committed but its run status is still live (the terminal order is K5, then run status, then stream end), so the cascade's stopRun answers stopping and the poll then reads completed.
A stop recorded before the K5 is the run's terminal (CP-59), so the order "the stop reaches the child, then the child completes" always ends the child interrupted. The first sibling cell, companion.stop-after-child-completes, held exactly that order and was red at its preconditions on every lane (W1 gQ159s; the owner's s66 on temporal-*). It was withdrawn and never landed. It had a second defect: turn B's companion__sendMessage did not wait for the answer (it returned delivered: true at once), so the parent made a model call its script did not cover and ended failed on js-memory / js-redis before the stop (owner diagnostics s67 / s68). The rebuilt cell must make that call wait for the companion's answer. A stop of a parent whose run already ended answers not_stoppable and does not cascade to a still running companion: that is the documented stop contract (docs/internals/concepts.md, Stops), not a finding.
What's owed.
- A control that holds the child after its K5 commit and before its run-status write.
- The FIRST lane is CF Workflows: a worker-side bridge hold between the
commitTerminalStateandfinalizeRunsteps, like the DO'sref.write.enter. - The other lanes take a Node-side store wrapper on
updateRunStatus: runtime-js, and Temporal / DBOS via their activities / steps. - Each lane needs a harness self-test.
- The FIRST lane is CF Workflows: a worker-side bridge hold between the
- A sibling cell over that control. The child is held after its K5, the parent is stopped, and the cell asserts the parent's ref reads
completed. - Done for CP-321 (batch 22, !858). CP-321's cells hold the K5 → run-status window on runtime-js and on the CF Durable Object:
- runtime-js:
hold-run-statusgained the release modereleaseAtNextRead(the held write lands at the next entry's seen read of the current run, which returns the pre-write value), and the runtime-js executor lanes gainedhold-run-status-at-seen-read, with self-tests. - cf-do (host tier): the new capability
hold-do-run-status(SessionDriver.doRunStatusHolds,harness/cf-do/run-status-holds-do.ts) holds the DO store's ownupdateRunStatusinside the worker. Its release,releaseAtEntryDecision, lands the write by the next entry's stage-1 decision and records the branch that landed it (at once in the drive's tail, at the stop probe's read when stage 1 counted the run executing, else after a named 60 s bound). Self-test:harness/cf-do/__tests__/run-status-hold.integ.test.ts. - Still owed: the CF Workflows bridge hold, the Node-side store wrapper for Temporal and DBOS, and the CP-159 sibling cell. The new holds release at an entry's reads, not at a stop cascade's poll, so the sibling cell may still need its own release mode.
- runtime-js:
Nothing waits on it: CP-159 already has its CF Workflows cell evidence (gQ159c) and !452's unit case for the completed arm. Pick it up after the owner's owed CF cells, or earlier if a CF finding needs the K5 → run-status window.
FU-CP321-DURABLE-TERMINAL-WINDOW: CP-321's terminal window on Temporal, DBOS and CF Workflows — open
Filed: batch 22 (agreed by the orchestrator, 2026-11-12; text by W1).
CP-321's window, where a new entry arrives between a run's K5 terminal commit and its own terminal run-status write, is fixed on runtime-js and the CF Durable Object only (rule T: an entry that saw the run live with its K5 already the session's latest commit advances once over that run's own terminal status step; spec docs/superpowers/specs/2026-11-12-cp321-terminal-status-step-design.md). Temporal, DBOS and CF Workflows are not traced for this window. For each: trace the terminal order (K5 → run status → stream end) and the entry gate's seen/expected reads, decide whether a new turn, resume or retry sent once the session reads terminal is refused while the run record still reads live, and if so apply rule T at that runtime's run-start builder (seen-time K5, so CP-96's control.stale-entry-loses-after-winner keeps losing). Related: CP-84 (Temporal's own terminal window), CP-96, CP-259.
FU-CONF-CP74-DBOS-HOOKS: DBOS hook seams for the CP-74 dual-window cell — open
The owner approved the direction, on conditions:
- the seams are awaited only inside recorded step bodies, so replay never re-runs them;
- they are
@internaland not exported from the package index; - they must pass the W1 / W2 window proofs;
- they ship in their own MR with the full DoD.
CP-74 stays open until then.
FU-CONF-NOOP-MANAGER-HOOKS-DEDUPE: CP-316's own-hook witness agents duplicate CP-299's convention — open (owner: W1)
Filed: 2026-11-10 (orchestrator ruling on CP-316's cell). agents/noop-manager-hooks.ts (the hooks.noop-manager-own-hooks-* cells) witnesses an agent's own onAgentStart through the witness port as hook:onAgentStart:<agentBase>:<runId>, the convention of CP-299's own-hooks witness agents, which were not on main when it was written. Wanted: once CP-299's witness MR lands, build CP-316's agents from its shared helpers (one hook: prefix, one per-cell cache) and delete the duplicate.
Conformance follow-ups from MR-1a (agent bundling)
FU-CONF-SESSION-CREATE-ABANDON-SELFTEST-WALL-TIME: a harness self-test that timed out under host load — fixed (owner, batch 12)
Fixed: the abandon() case in packages/e2e/src/conformance/__tests__/session-create-scenario.test.ts ("a stuck attempt's observers stop reading once its bound fires") already ran on fake timers; its cost was the number of fake reads it drove. A read every 20 ms over the 190 s run bound made about 150,000, while it asserted only 1,600 (the first ~2 s of the bound). It now reads every 2 s and requires 90% of the reads the bound holds (SESSION_CREATE_OBSERVERS × floor(0.9 × run / readDelayMs)), and takes its "no more reads" snapshot after the observers' own final read has started. Owner slot s282 (2026-11-06): the old test x5 ran 3.6, 3.6, 30.0 (timed out, at a 1-minute load of 62), 3.8 and 11.6 s; the new one x5 ran 1.05 to 1.08 s. Observers that never stop fail its "no more reads" check, and observers that stop at half the bound fail its threshold (784 against 1,360), 3 of 3 each; under the old test the half-bound mutation passed. Earlier sightings: owner slots s210 (26.9 s) and s210b (30.0 s, timed out), W1's lock-free x5 at MA-49's head (8.3, 7.8, 30.0 at a load of 112, 9.6, 8.0 s) and one timeout in MA-49's post-stack slot. The 5-minute load gate this entry asked of conf-unit phases is no longer needed for this test.
FU-CONF-PARTIAL-BATCH-T0-WINDOWS: submit.partial-batch-resume (dbos-postgres, cfw-workflows-d1) fails under load because its windows are anchored on t0 — ✅ fixed (owner batch 17)
Fixed (owner batch 17): both partial-batch scenarios now judge every window against the answer instant the executor driver records (RunHandle.clientToolSubmits), through one shared check, judgeWindowBeforeAnswer (packages/e2e/src/conformance/scenarios/submit-partial-batch.ts). submit.partial-batch had the same t0 anchor in .window-open and inside .execute-refused-pending, so it is fixed too. WINDOW_MARGIN_MS is gone. The check reads the submit record after the window's work finished, and a call with no record yet passes: the driver pushes each record synchronously before the product's submit runs, so a later answer starts after the window. No id changed, and no ledgered id is touched (the only ledger entry for these scenarios is dbos submit.partial-batch: CP-64 and CP-93 ids, no window id). CP-65, CP-82, CP-86 and CP-92 (fixed) and CP-64 and CP-93 (open) keep their status (CP-64 was later fixed on DBOS, by !894, batch 27). Each window's real slack is now recorded as an informational timing.
- Before, on CI (pipelines since 2026-10-05, 105 of which ran
test:conformance*): 3 failures in 62 cfw-workflows-d1 runs (4.8%), all.nothing-submitted-window-open: job 17036638618 (branch ci-runner-proving-edge, 1163 ms late), job 17036802634 (!327, 3060 ms), job 17069728529 (!331, 225 ms). 0 in about 570 runs on dbos-postgres and thetemporal-*andjs-*lanes. Nosubmit.partial-batchwindow ever failed. - Forced-delay proof (red first): a working-tree mutation that sleeps 7500 ms between t0 and
driver.start(the backend start-to-suspension latency the old check charged to the window; past its 7000 ms budget, so the red is deterministic), on the cfw-workflows-d1 cell, then reverted:- old check (main a1c6655f56), no delay: pass;
- old check, 7500 ms: fails exactly
.nothing-submitted-window-open("finished 1705ms after its window closed") and.nothing-submitted-refused-typedas its precondition: the CI signature; - new check, 7500 ms: pass, with 7044 ms of real slack before c1's answer and 19994 ms before c2's;
- new check, no delay: pass, with 7056 ms and 20062 ms. The slack does not move with the start latency.
- After (×5, every executor lane, under the stress lock, 2026-10-10 00:18-00:44Z): before (main a1c6655f56) and after, 5 rounds each, all clean: 21 pass + the 1 ledgered dbos
submit.partial-batchcell every round on both trees (the old check also passed all 5, as expected at a 4.8% CI rate; the forced-delay proof is the red). The new windows' real slack on cfw-workflows-d1, worst of 5: 5990 ms before c1's answer (submit.partial-batch-resume), 14016 ms before c2's (submit.partial-batch) and 16347 ms (resume). Host load at each round's start: 9, 11, 40, 12 and 7 on 14 CPUs; no pipeline was running. The full matrix on the rebased head (main a07e5aa1fb) passedconformance:verify, both cells passing on cfw-workflows-d1.
The original entry follows.
The cell's settle windows are measured from the cell's start (t0), not from an observed event. On a loaded machine the first DBOS step can take longer than about 7 s, so the batch is not yet pending when a window closes and the cell fails with a timing assertion, not a product verdict. It surfaced as a load flake around the MR-1a work.
Seen on cfw-workflows-d1 too (2026-10-08): !327 pipeline 2927288553,
test:conformance 4/5job 17036802634 (DigitalOcean runner, during the DOKS starvation window):.nothing-submitted-window-open: the nothing-submitted mode finished 3060ms after its window closed, with.nothing-submitted-refused-typedfailing as its precondition. It is the same t0-anchored window, so it is not a CF Workflows product verdict. The shard was retried under §A3 (job 17038294157). The fix below covers both lanes.Fix: judge each window against the exact answer instant the executor driver now records:
RunHandle.clientToolSubmits(capabilityobserve-client-tool-submits, read throughrequireClientToolSubmits), theperformance.now()taken immediately before the responder submits each answer. Compare the window's read-doneperformance.now()with it, asstream.suspension-marker's.window-opendoes withjudgeReadsBeforeAnswer, instead oft0 + delay - margin. Keep the bounds as ceilings, not as the sequencing.Owner: the conformance owner. It lands after MR-1a (do not fold it into the pure-refactor MR).
Conformance follow-ups from plan 0c MR-1b (the CF Workflows harness)
FU-CONF-CFW-PRODUCT-WAITS-PRE-FLIP: decide the CF Workflows product waits before the lane runs — ✅ closed (decided 2026-10-05; plan 0c MR-2, !307)
Closed: the MR-2 dry run measured every product wait, and the owner decided them with the user on 2026-10-05, before the flip. session.create-observed-whole overran its single batch bound on cfw; the D1-over-Miniflare cost of its harness reads was the cause, not a product wait. Its final design is a user decision of 2026-10-06: each of the 48 attempts settles under SESSION_CREATE_ATTEMPT_BOUND_MS (60 s, its own constant, sized on CI evidence; superseded 2026-10-24 by per-stage guards, FU-FLAKE-CONF-CFW-SESSION-CREATE-ATTEMPT-BOUND), the same on every lane, and the 16 observers per session keep the full observe() read. The uncapped handle.result() poll backoff and the llmStepRetry backoff were catalogued as CP-99 and DI-35, with no bound changed. stop.mid-tool-honors-abort's cost is about 7.6 s of its 8.2 s: the 5 s tool abort-check tick plus 2.58 s of result() overshoot. The settle-wait never waited, and the run-start gate took 20-105 ms. The figures and decisions are in the plan 0c spec, §9 "MR-2 pre-flip decisions (2026-10-05)" and "As built: MR-2". The text below is the original entry.
The cfw-workflows-d1 executor lane is built and declared not-yet-run. Before MR-2 flips it to running, the plan 0c spec's §2 rule ("product waits versus named settles") must be applied to every product wait the lane measured (entry gate in spec §9, MR-2). The MR-1b self-tests (packages/e2e/src/harness/cfw/__tests__/lane.integ.test.ts) measured these on Miniflare:
stop.mid-tool-honors-abortcosts 8.4-8.8 s per cell, against 0.7-2.3 s for the other three live-smoke cells. Review attributed it tohandle.result()'s poll backoff, a product wait. That is not confirmed: it needs a long-run measurement of where the time goes.- No settle wait occurred (0 of 10 two-turn sessions in each of 3 runs), so the prior-instance settle-wait has no duration sample. Stop→resume and retry settle waits were not measured.
The decision is the owner's, taken with the user, and is never a per-cell bound or a per-backend tweak (spec §11, "Product waits against bounded settles"). Record the figures and the decision in the spec's MR-2 as-built section.
Owner: the conformance owner (plan 0c MR-2, before T7).
FU-CONF-CFW-BODY-TERMINATE-UNLEDGERED: body-side terminates are not ledgered on the CF Workflows lane — ✅ fixed (plan 0c MR-2, !307)
The worker's body-side binding wrapper (observedBinding, now packages/e2e/src/harness/cfw/worker/observed-body-binding.ts) reported each successful create (wf.created) but returned the inner get handles unwrapped. So the product's unknown-owner terminate (workflowBindingOwnerProbe's terminate, which does binding.get(id) then terminate(), called by the escape hatch in finishCfwEntryStream; both runtime-cloudflare/src/run-entry.ts) was never ledgered as terminatedBy: 'product' and its session was never fenced, so its late traffic failed the NEXT cell and the failure was blamed on the wrong cell.
Fixed: the wrapper's get handle now reports a successful terminate() to the bridge (wf.terminated, recorded per cell as BridgeServer.terminatedInstances). foldBodyCreated in cfw/cell-work.ts folds the wf.created reports first, then marks each terminated instance terminatedBy: 'product' (ObservedWorkflowBinding.markProductTerminated, a no-op for an unknown instance or one the reap terminated), so endCell fences its session 'product-terminated'. The report carries no session id (the wrapper knows the instance, not its session); the fence follows from the ledger entry. Limit: a terminate report for an instance whose wf.created report was lost is ignored.
Coverage: no lane cell reaches the body terminate today: the escape hatch fires only when a persisted owner's status reads unknown past the threshold, and Miniflare never produces that. The in-workerd path (real Miniflare handles from get) has no test; worker-deps.integ.test.ts proves the wrapper over the real bridge with a stand-in engine handle.
Owner: the conformance owner (plan 0c MR-2).
FU-CONF-DO-SUBMIT-ANSWER-DROP-CONTROL: a cf-do control that drops the DO's submit answer after the DO stored it — open (harness; owner: the conformance owner)
Owner: the conformance owner. Findings: CP-198 (its witness), and every future lost-answer path.
fault-do-submit-answer answers a client-tool submit in the worker BEFORE it reaches the Durable Object, so the DO never stores it. CP-198's window is the opposite: the DO stores the answer and its wake starts the continuation, then the answer back to the chat handler is lost (a stub fetch failure, a retryable transport_error), and the answer is submitted again. No cf-do control can lose that response today.
To close:
- A worker-side response fault, armed per session and one-shot (or
times: N), that lets the submit reach the DO and then fails the handler's read of the answer the way a dropped stub fetch does. The served count must be observable, it must be cleared and reported at cell end, and it needs a harness self-test. - A cf-do host cell for CP-198, in both
concurrentEntrymodes: the handler's re-submit answersdeferredand the client attaches, with no 409 underrejectand no rerun underinterrupt_previous. The cell also needs a hold on the continuation's drain read (the shape of W1's/test/hold-drain-readgate indo-drive-pipeline.integ.test.ts), so the duplicate lands while the first submit's continuation is still in flight, not only the dropped answer. - Once it's red on the pre-!453 DO and green after, add it to CP-198's
provenBy. CP-198 is flippedfixedbyprovenByExceptionon W1's integ tests until then (!453 merged as 16ae386528).
Scheduled after the CP-193 flip batch (orchestrator, 2026-10-20, Durable-Objects-first).
FU-CONF-DI92-CLIENT-GONE-CELL: a host cell for DI-92 needs a control that observes the chat route's own answer — done (!861; cf-do a declared gap)
Filed: batch 22 (orchestrator ruling; text by W1). DI-92 is pinned by units until this lands.
Closed by: W1's !861 (batch 23): observe-chat-answers on both hosts and the cell chat.send-client-gone-before-stream-host, which passes on js-chat-host and is red there with DI-92's streamHandle branch removed (DI-92 is now fixed, proven by that cell). The cf-do row is a declared gap, not the "no declared gap" planned below. On local workerd cf-do never gives DI-92's 499, and how it fails depends on the platform: on a macOS host request.signal aborts about 10 s after the cut and the departed request's in-flight DO /start never settles, so the route never answers; on the Linux CI runner request.signal does not abort within 30 s and the route answers 200 once released. Either way the DO completes the run (CLIENT_GONE_BEFORE_STREAM_CF_DO in gaps.ts; production Workers not measured). The same MR narrowed the older quiet-cut gap text (CLIENT_DISCONNECT_CF_DO and its sites) from "only at the next write" to "not within the cell's 5 s close bound". The entry below is the plan as filed.
The cell:
chat.send-client-gone-before-stream-host, on cf-do and js-chat-host (no declared gap). It aborts the send before its stream opens with existing controls only:- Arm
hold-stream-claim, which lands after the send's run-start commit. The send'sexecute()(and the DO's/startanswer) waits on the claim, sostreamHandlehas not started. - Settle on the claim being held.
- Cut the POST with the host driver's
disconnect(). - Release the claim.
execute()returns, andstreamHandlefinds the client signal already aborted, which is DI-92's path.
control-upstream-readcannot reach it. Its hold applies to the/ssebody after the fetch resolves, sohandle.stream()returns a stream. The DO handle answers an abort during its open with an empty iterable, not null (W1's trace).- Arm
The new control,
observe-chat-answers(both hosts): per session, the status of every chat-routeResponse, plus the handler logger's warn and error lines for that request.- cf-do: a worker-isolate recorder around the chat route's
handleChat, read through/__conformance/chat-answers/:sid. It needs its own logger that records warn and error lines: the worker's chat logger (readerBufferLogger,harness/cf-do/worker/reader-buffer-recorder.ts) drops them today. - js-chat-host: an in-process wrap of its route's
handleChatStreamcall. - Harness pieces: a driver read, the capability and its pinned lists, and
test:conformance-harnessself-tests.
- cf-do: a worker-isolate recorder around the chat route's
What the cell asserts:
answer-499: the recorded answer to the send is 499, with an empty body (the orchestrator's plan B: one shared helper withstreamLiveResponse's client-gone answer, and a debug line instead of a warning);- no warn or error line from
streamHandlefor the session; - the run reaches
completed(the input was committed, so the run is not lost); - an attach streams the reply.
Red at main: the recorded answer is 500
transport_stream_drop, with the warning.Done when: the cell passes on both hosts at the fix's head, and is red with the fix removed (main's
streamHandle). DI-92 then names it inprovenByand flips to fixed.
FU-CONF-SCRIPTED-FAIL-REPORTS-ONERROR: the scripted fail response never calls the adapter's onError — open (harness; owner: the conformance owner)
Owner: the conformance owner. Findings: CP-235 (where the blind spot was found), and any held-report double write.
ScriptedModel's { kind: 'fail' } returns an error step without calling callbacks.onError (core/src/llm/scripted-model.ts), while a real adapter (llm-vercel) reports the error through onError and then returns the step. So a cell that drives a model failure with fail never exercises a runtime's held LLM error report (core holdLLMErrorReport), and its error-chunk assertions cannot see a held-report double write. stream.one-error-chunk-classified uses the mock adapter's { type: 'error', errorInstance } response instead, which reports through reportLLMError; it is red on CF Workflows at main for exactly that reason (CP-235).
Cells whose error-chunk verdicts share the blind spot (host tier, cf-do and js-chat-host): do-handle-failed-stream's .one-error-chunk and failed-reconnect's error-frame checks. Neither runs on CF Workflows (no CFW chat-host lane), so they hide no CFW defect today.
To close:
- A sibling host cell,
stream.one-error-chunk-reported-host(cf-do and js-chat-host), driving the reporting mock error and asserting one error chunk. It's a sibling, not an in-place change, because the existing cells' ids may sit in fixed findings'provenBy. - Make
{ kind: 'fail' }report throughonErrorthe way a real adapter does (a core testing change). Land it only after a full-lane run shows no cell's verdict moves, or ledger what moves.
FU-CONF-449-CELLS: cells owed for the findings !449 fixed in code — open (owner: the conformance owner)
Owner: the conformance owner.
!449 (0889c0ee05, one result per remote call: remote settlement, Invariant S) names RM-98, RM-99, CP-176, CP-177, CP-178, RM-100 and RM-101. It added no conformance cell, so all seven are fixed in code but not witnessed. Each needs the cell its record's pending plan describes, red against main before !449 and green after:
- RM-98: Needs a host cell: reconcile's result append fails with an outcome-unknown store error after it landed (a store fault); assert exactly one tool_result for the call.
- RM-99: done (!691):
remote.settled-child-evicted-after-commit-host(cf-do host), slot-rmb v3, 3 of 3; red with !449's settle-before-ref rule restored (.one-result,.history-paired; the settle half alone.ref-terminal). RM-99 is fixed. - CP-176: Needs a host cell: a remote child interrupted producer-side while the parent's ref says running; assert a resume with a message is answered (not refused forever).
- CP-177: Needs a host cell: the parent reconciles while its child's /start is held (fault-child-do-route hold start); assert the child is not settled lost while its start may still land.
- CP-178: Needs a host cell: a remote child settled lost while it is still running; assert it is interrupted.
- RM-100: Needs a host cell: a new turn whose child's status read fails once while the child has completed; assert the child's real result is delivered.
- RM-101: Needs a host cell: a remote ref whose tool the agent no longer has; assert a resume pairs the call and the child is interrupted.
FU-CONF-444-CLIENT-CELLS: client cells owed for the findings !444 fixed in code — open (owner: the conformance owner)
Owner: the conformance owner.
!444 (5cf0bf0ae9, one re-attach controller for useHelixChat and useReanchorOnError) names CP-165..170, RM-93, RM-94, DI-61 and DI-62. CP-166, CP-169, CP-170 and RM-94 have client cells in the suite. The others are fixed in code but not witnessed. Each needs a client-tier cell on cf-do and js-chat-host (the real hooks), red against the pre-!444 client and green on main:
- CP-165: a send whose stream drops while the snapshot route answers 503. The failure reaches
onError/resyncError, and the run is re-attached once the route recovers. - CP-167: a send answered 2xx whose body fails before its first message chunk. The client re-attaches to the live run.
- CP-168: an
onToolCallthat throws aTypeError. The error reaches the app (onError) and is not re-anchored as a drop. - RM-93: a send whose stream drops after the run's last chunk. The turn renders, and the chat status is not
error. - DI-61: a re-anchor whose resume is refused (a typed 409).
onError(oronResumeError) fires with it. - DI-62: a re-anchor whose snapshot answers 404. The hook's documented behaviour holds: a 404 means the server does not hold the turn.
Also owed, a harness flake: src/harness/cf-do/__tests__/spike.integ.test.ts M5 ("a failed first start disposes Miniflare and removes the persist dir") diffs a global /tmp listing of hxc-* before and after (:536 / :551). Any concurrent test that creates an hxc-* dir in that window fails it (!452's conformance:verify job 17302751043 saw [ 'hxc-OINfva' ]). The race is inside one run too: the harness config runs two forks (maxForks: 2), and every cf-do backend mkdtemps /tmp/hxc- (miniflare-backend.ts), so it can flake locally as well. Assert on the persist dir that start created (or give each run its own persist root) instead. M5 fixed (the owner's next batch, 2026-10-22): it again flaked on !474's conformance:verify job 17400137117 ([ 'hxc-HMGHet' ]). The global listing is gone; M5 is judged only on the persist dir its own failed start named and removed (existsSync(dir) false).
FU-CONF-REMOTE-CHILD-STOP-RETRIED-BOUND: subagent.remote-child-stop-retried-host's .stop-retried bound raced the owed-stop schedule — fixed (test only; lands with its MR, !577)
Owner: the conformance owner. Filed: 2026-10-29.
The cell's doc and its RETRY_BOUND_MS assume the owed-stop retry fires 1 s after the parent's failed stop. Since CP-217's R1 schedule the first retry is 30 s, doubling to 1 h (core REMOTE_STOP_OWED_BACKOFF_BASE_MS, core/src/run/stop-remote-child.ts), and the conformance cf-do worker keeps the production schedule. The cell waits RUN_OUTCOME_BOUND_MS (30 s) counted from its disarm, which comes a few seconds after the failed stop, so it passes only while the Miniflare alarm's lateness stays under that gap: batch 3's build 3 (pipeline 3002836888, job 17566312627) missed with .stop-retried did not settle within 30000ms. Fixed (no race ledgers), in the same MR:
- (a) anchored on the debt: after the disarm the cell reads the parent ref's observed
stopOweddue time (nextAttemptAt, elseat) and bounds.stop-retriedfrom it, capped at 3 xREMOTE_STOP_OWED_BACKOFF_BASE_MS(imported from core) plusRUN_OUTCOME_BOUND_MS; an observe line prints the due time against the disarm on every run, so the fallback (a fixed bound imported from core) was not needed; - the cell's doc no longer claims a 1 s first retry;
- proven by owner slot s207 (3 of 3 on cf-do: the retry was due about 29.9 s after the disarm and settled within about 130 ms of it, against a bound of about 59.9 s) and owner slot s208 (the cell once more, and the full
test:conformance-unit, 88 of 88 files).
FU-CONF-DBOS-MIXED-STEP-SESSION-ACTIVE: attribute RM-112 (on DBOS a step that ran a server tool and left a client call pending keeps the session active with the server tool's row uncommitted) — open (owner: the conformance owner)
Owner: the conformance owner. Filed: 2026-10-28.
CP-115's sibling lifecycle.new-turn-over-step-partial-refused (one step calls noop, executed, and ask_user, pending) never reaches its anchor on dbos-postgres: owner slot s183 (3 of 3) and batch 2's !571 pipeline 3001450587 saw the session stay active for 30 s with ask_user pending and no noop row, while the run record read suspended_client_tool (that label is RM-111). The dbos-postgres row is ledgered under RM-112, which records the symptom with its cause unattributed. Owed: a probe that reads whether noop ran, when its row commits and what the session reads after ask_user is answered, then either the cause stated in RM-112 (for example a completed server-tool result not committed with the suspension, the C1 contract) or RM-112 withdrawn with the by-design reason (DBOS's in-workflow wait). Parked with DBOS under the CF-first direction.
FU-CONF-CP237-OWED-CELLS: CP-237's owed cells (the re-issue variant, the cf-do host cell) and the child-session sibling — open (owner: the conformance owner)
Owner: the conformance owner. Filed: 2026-10-28.
CP-237's executor cell subagent.with-message-over-children-park-delivers-child is ledgered on the JS lanes (owner slot s188). Still owed:
- The re-issue variant (JS executor): the new turn re-issues the parked call's id. Before child identity MR 1 the derived child id re-entered the old child and masked the orphan; the cell pins that it no longer can.
- A cf-do host cell: the same park and a
with_messagethrough the DO's native/resume. It decides the CF DO's CP-237 row (the DO's settle path cascades stops, so it may differ from runtime-js). subagent.awaited-child-session-paused(withdrawn, branchconformance-owner-child-session-followup): a sub-agent child waiting on a client call must readpaused. On Temporal it was not identical over owner slot s185's three runs (the child readfailedon temporal-redis twice, on temporal-redis and temporal-postgres once), so it is timing-dependent there. Re-anchor it on the child's own committed suspension (its run recordsuspended_*, then its session) before it gets an id.
FU-CONF-CLIENT-TIER-OOB-STOP: a client-tier control that records a server stop from outside the page — open (LOW)
Owner: WA (reliability-cf-wa-do-drive), after RM-121. Filed: 2026-11-08 (the conformance owner's ruling on RM-121's client cell).
Why. stop.finished-shown-on-reload-client runs on the useHelixChat mount only. Its stimulus is a recorded server stop. The page's own stop() produces one on that mount (it POSTs the host's stop route, CP-12). A stock useChat page's stop() only aborts its own fetch (client-driver.ts:1414-1422), so the page cannot stop the run. A stock app does stop runs, through its own server route, and its user then reloads: that path is real, and unjudged.
What's owed.
- A client-tier control that records a server stop from outside the page (the host's stop route, as
stop-answerdoes on the host tier), claimed on cf-do's client tier. - A stock-mount sibling
stop.finished-shown-on-reload-stock-clientwith the ids.stock.stop-point,.stock.finished-in-snapshot,.stock.finished-output-shownand.stock.held-call-no-output(the helix cell's ids carry the mount, so none of them changes meaning).
FU-STATUS-LEADS-SNAPSHOT-DOC: document that a session's status can read terminal a moment before its snapshot does — open (LOW)
Owner: WA (reliability-cf-wa-do-drive), with RM-122. Filed: 2026-11-08 (the conformance owner's ruling on RM-122's cells: by design, not a finding).
Why. Every terminal exit writes K5 (commitState, the session row) first, then the run's terminal status, then the stream end (the terminal order). A snapshot's status and its turnStops come from the RUN record (deriveSnapshotStatus, core deriveTurnStops), so between K5 and the run status a client that polls getStatus reads the session interrupted while the snapshot still reports a live run and no stopped turn. The window is bounded and nothing in it is lost or contradicted once it closes: a client that resumes on the snapshot's word attaches to a run that is ending and gets its end. A streaming client never sees it (the stream ends after the run record). stop.turn-marked-* waits for the snapshot's own ended for this reason.
What's owed. A sentence in the session-model reference (docs/internals/session-model.md) and the ai-sdk reference (docs/reference/ai-sdk.md, beside turnStops): the session status can read terminal a moment before the run-derived snapshot does, so a chat client reads the snapshot. If a real client is ever hurt by it, it becomes a finding then.
FU-PRE-BODY-FRAME-WRITES: a frame written before a call's body runs can still fail its run — open (owner: WA)
Filed: 2026-11-09 by WA, from CP-305 (the orchestrator's ruling 2: pre-body frames are out of CP-305, their own follow-up). CP-305 makes every frame written AFTER a result is known best-effort. The frames written BEFORE a body runs still throw on a failed write: a server tool's tool_start (runtime-js's step announcement, DBOS steps/execute-tool.ts:293, CF Workflows steps.ts runToolCall's tool_start, Temporal executeToolActivity's tool_start), and the frames that announce a suspension (tool_approval_request, a client call's tool_start). No result exists yet, so none is lost, but the run fails on a stream outage. The exception is runtime-js's saved-step re-attach loop: its tool_start follows other calls' known results, so CP-305 covers it.
The same open question covers the frames that carry no result at all (CP-305's audit): runtime-js's rewind writes an aborted tool_end per discarded call (js-agent-executor.ts, the rewind's discardWriter) after its commit, so a failed write fails the entry; the turn's entry state_patch (run-loop.ts commitStartWork); and the CF DO's legacy stranded-run demotion writes run_interrupted (durable-object-agent-base.ts demoteStrandedExecution) after its CAS, so a failure skips the stream end and the settle.
What's owed. A ruling per kind. An announcing frame of a suspension is how the client learns of the pending call, though the snapshot shows it too (a pending approval renders as approval-requested). Then the same best-effort rule, or a typed failure, at each site, with units and a 'fault-chunk-write' cell.
FU-CFW-STOPPED-CALL-FRAME: CF Workflows' stopped-call tool_end can still fail its step — open (owner: WA; post-features)
Filed: 2026-11-09 by WA, from CP-305's rebase onto batch 17 (the orchestrator's ruling: a follow-up, not a CP-305 fix round; CF Workflows is outside features-first). Batch 17's DI-89 / CP-277 added CF Workflows' stopped-call frame: runtime-cloudflare/src/steps.ts (finishStoppedInBodyCall, a call that stopped itself; the comment "The call's one terminal frame: the stop pairing's (CP-277)", line 6297 at CP-305's head) writes the call's tool_end with a raw writer.write(...) AFTER the call's not-executed run_stopped row is decided and BEFORE its afterTool hook. That is CP-305's class (a frame written after a result is known), not a pre-body or result-less frame: a failed write skips the hook and fails the step, which step.do then retries. Planned fix: writeFrameBestEffort(writer, …, this.frameWriteFailed(sessionId)), as CP-305's other CF Workflows sites; unit, red first, in steps.test.ts's CP-305 block: a self-stopped call whose tool_end write fails keeps its run_stopped row, fires afterTool and does not throw (no step.do re-run).
FU-DBOS-CHILD-WRITER-LIFECYCLE: a DBOS sub-agent's stream-writer setup or close can lose or replace its child's result — open (owner: WA)
Filed: 2026-11-09 by WA, from CP-305's child-site trace (the orchestrator: a follow-up, not CP-305; these are a writer's creation and close, not a frame write). Cites at the CP-305 head (reliability-cf-wa-cp305, off da8aa1e1af), packages/runtime-dbos/src, each verified read-only.
- Local sub-agent.
ExecuteSubAgentStep.endSubAgentStep(steps/execute-sub-agent.ts:678-679, a@DBOS.step()with no retry config of its own) opens its writer withcreateWriterat:709, before any try, and closes it in afinallyat:752. The workflow calls it atworkflows/standard-workflow.ts:386, after the child's result is known and outside the try that spans:324-372. A failingcreateWriterorclosethere fails the parent's turn with the child's result unrecorded. - Remote child.
runExecuteRemoteSubAgent(steps/execute-remote-subagent.ts) opens its writer at:289, outside the try at:296, and closes it in afinallyat:332(and:982on the other path). The step retries (@DBOS.step({ retriesAllowed: true, maxAttempts: 3 })at:1006), and after exhaustion the dispatch site's catch (workflows/shared.ts:4080-4085) degrades the call into a failed remote result. To confirm when fixing: whether each retry re-attaches to the running child or restarts it (the CP-305 trace agent reported a re-attach through the D-2 guard; not verified). - What's owed. Guard the writer's creation and close the way CP-305 guards a frame write (logged, never deciding the child's result), with a unit per site (a failing
createWriter, a failingclose, each keeping the child's real result), and a DBOS cell once'fault-chunk-write'can fail a writer's creation.
FU-DBOS-STOP-REFUSED-BETWEEN-STEPS: a DBOS stop between two steps was not accepted once — open (LOW)
Owner: WA (reliability-cf-wa-do-drive), with RM-122. Filed: 2026-11-08 (orchestrator and conformance owner: seen once, not RM-122's).
Seen. In RM-122's first cells slot, stop.turn-marked-between-steps on dbos-postgres (executor) failed .stop-point, .run-interrupted and .turn-stop-reported: the interrupt, sent while the run was held between its two steps, was not accepted, and the run did not end interrupted. The host was at load 30–35. The cell did not print the stop's result then. It now prints it on .stop-point.
Not reproduced. The stop was accepted in all 25 later runs: the 2 other runs of that first slot, then 23 at load 14 or below (at RM-122's head, 3 slot runs and 10 stress runs; at its red tree, where the derivation is stubbed and the stop path unchanged, 10 stress runs). RM-122 changes no stop or DBOS code, so the possible origins are main's DBOS runtime and RM-121's core change (the stopped step's completed results recorded at once).
What's owed. If it recurs anywhere, capture the printed stop result and the run's records, and run the cell ×10 on main to tell the two origins apart. Until then, nothing.
FU-CONF-CLIENT-TIER-CFW-HOST: a CF Workflows chat-host lane for the host and client tiers — open
Owner: the conformance owner. Filed: 2026-10-19 (orchestrator: a coverage hole under the Cloudflare-first direction; ranked after the CP-170 and CP-186 cell work).
Why. The host and client tiers run only on cf-do and js-chat-host. CF Workflows has an executor lane (cfw-workflows-d1) and no chat-host lane, so every bug in a CF Workflows app's chat route is invisible to the suite: reload, resume, the snapshot, the replay prefix and stock / Helix client behaviour over it. CP-186 shows the cost. A reload while a local sub-agent awaits an approval breaks a stock useChat; CF Workflows runs local children as workflow instances, so it can hit this, but no cell can reach it there, and its CF evidence is WE-2's CFW unit test instead.
What's owed.
- A CF Workflows chat-host lane at the host and client tiers. It serves the product's chat route (the opennext / CF Workflows example route shape) over the real
CloudflareAgentExecutoron the Miniflare Workflows binding thatcfw-workflows-d1already uses (setupCfwMiniflare), with the observation the other hosts give (client, stream, persisted, hooks where it has them). - Harness self-tests for its controls.
- Its first cells: CP-186's
client.delegated-approval-reload-helix/-stock. - Then a sweep of the host and client scenarios already gapped or untested on it.
FU-CONF-CFW-REAP-TRANSIENT-POISON: one transient Miniflare binding fault in a cfw cleanup poisons the whole lane — open (LOW, conformance harness; owner: the conformance owner)
Surfaced by: main pipeline 2966843082 (16ae386528), conformance:verify job 17310853774 (WE-2, 2026-10-20; the orchestrator accepted the failure itself as a flake).
The reap self-test's cleanup read one spawned child instance's status and got a transient binding fault (terminated). The harness then marked the backend poisoned: "backend poisoned in cfw-selftest.reap-companion|cfw-workflows-d1|executor: rebuild after left-running failed: cfw cleanup: binding fault(s) after the last cell: - cfw: binding fault (status instance=…-agent-childspawne81ec699-spawn-1 session=?): terminated". Every later cell on the lane failed with it (4 live-smoke and 3 measurement cases, lane.integ.test.ts:1100).
Fix direction: in the cfw reap / rebuild, retry a transient status read (terminated, a dropped binding call) once before poisoning the backend, and report the retry; keep a second failure poisoning as today. Add a harness self-test that injects one transient fault and asserts the lane survives it.
FU-FLAKE-CONF-CFW-RESPAWN-FRESH-TURN1-SETTLE: companion.respawn-after-failure-fresh on cfw-workflows-d1 missed its turn-1 settle bound once, and its cell end poisoned the lane — open (watch; owner: the conformance owner)
Surfaced by: batch 21's MR !848, train pipeline 3056097612, job 17946417569 (test:conformance 2/7, the train ref refs/merge-requests/848/train at 16ae617e). The same shard passed on the MR pipeline 3055911329 (job 17945122474) with identical content (main unchanged, so the train ref equals the merge ref). Batch 21 does not touch the CF Workflows companion respawn path.
- Signature:
companion.respawn-after-failure-fresh|cfw-workflows-d1|executorran 49,063 ms and failed.turn1-completes("did not settle within 30000ms"); its other five ids failed only as preconditions. Its cell end then crashed: "could not end the backend work of [cf-companion.respawn-after-failure-fresh-cfw-workflows-d1-…-s]: ending the backend work of the runs timed out after 5010ms". - Slow, not stuck: the harness's rebuild then read late traffic from the closed cell on the root session (
hook.fired:onAgentComplete) and on its-agent-helperchild (model.generateStep,afterLLMCall,afterTool,onMessage): turn 1 finished after the cell closed. Cells earlier in the shard also ran long under the load (approval.approved-tool-before-hook-throw-fails-typed39 s,companion.paused-outlives-root-turn32 s). - Blast radius: "backend poisoned in companion.respawn-after-failure-fresh|cfw-workflows-d1|executor: rebuild after unsettled failed: harness unhealthy (7 failure(s))" (the 7 are that late traffic). Every later cell in the shard part was GATED: 98 cells across every scenario family, not only
companion.*(conformance-cfw-2.integ.test.ts: 163 tests, 99 failed). - Retry: green. Job 17946989368 (
test:conformance 2/7, the same train pipeline 3056097612, 16ae617e): the cell passed in 10,355 ms, the shard part passed whole (no gate, no late traffic). - Closes with: a recorded green retry, and no second sighting within batch 22. On a second sighting: trace the turn-1 wait on the cell's own run (what the parent waits on after the child fails) before calling it load. The lane-wide poison from one cell's late traffic is the same blast radius as FU-CONF-CFW-REAP-TRANSIENT-POISON; a rebuild that waits out a cell's late completion (bounded) instead of poisoning the lane belongs with that entry.
FU-CONF-CFW-POOL-ROW-RETIREMENT: retire the cfw-workflows-d1-pool row and the legacy workerd helper — open (plan 0c MR-2: every consumer kept; blocked on scenario support)
Status (plan 0c MR-2): open. MR-2 compared each consumer with the cells of the running cfw-workflows-d1 lane, assertion by assertion. No consumer is fully covered, so each is kept whole and nothing was removed. The cfw-workflows-d1-pool row, CFW_POOL_CAPS, setup-helpers/cfw-workflows-d1.ts and its e2e/package.json export all stay. The entry is re-scoped to the scenario support the lane lacks:
HookObservationhook payloads: run ids,previousRunId,reasonand firing order (the executor tier carries only counts);- an approval-deny scenario (none drives an approval);
- executor-tier checkpoint
streamSequencetruth (no executor-tier cell reads checkpoint stream sequences).
Trigger: retire each pool consumer (lifecycle-hooks-parity.wf-noiso, checkpoint-truth-matrix.wf-noiso, cfw-d1-backend-smoke.wf-noiso, harness-smoke.cf) once a lane cell covers every assertion listed below for it. Retiring the last one removes the helper, its export and the descriptor (the "To close" rule below).
runtime-cloudflarelifecycle-hooks-parity.wf-noiso.test.ts: kept. Partly covered bysmoke.client-tool-completes|cfw-workflows-d1|executor(.outcome-settles,.completes): the client tool's suspend, submit and resume complete. Not covered:- Scenario 1's hook payloads:
onAgentSuspendedandonAgentResumedexactly once each,reason: 'client_tool',pendingToolCallIds: ['tc-1'],awaitingChildren: [], a fresh resumerunId,previousRunIdequal to the suspended run's id, the firing order['suspended', 'resumed'], and both run ids inlistRuns; - Scenario 3 (approval deny), which is not covered at all.
The executor tier's
HookObservationcarries only counts (no run ids, reason or order). No scenario assertssuspended/resumed, and none drives an approval.- Scenario 1's hook payloads:
runtime-cloudflarecheckpoint-truth-matrix.wf-noiso.test.ts: kept. No lane cell covers it. The pool runs 8 boundaries:text-terminal,finish,terminal-state,deferred-finish,forced-success,forced-failure,client-suspendandapproval-suspend. Each assertsboundary.row-present/.count/.seq-below/.seq-aboveplus S1 (P1 and P2) at every checkpoint. No executor-tier cell reads checkpoint stream sequences.stream.chunk-step-matches-commitandstream.suspension-markerassert adjacent claims: chunk step stamps against the committed step, and the suspension marker.runtime-cloudflarecfw-d1-backend-smoke.wf-noiso.test.ts: kept. Partly covered bysmoke.client-tool-completes|cfw-workflows-d1|executor(.completes,.llm-input-has-result). Not covered:- the closure
recordedHooksacross the suspend and resume legs: suspended 1 and resumed 0 after the first leg, then 1 and 1 withpreviousRunIdlinkage; - the agent's own hooks firing alongside the recorder;
- the structured
output; - the adapter's
options.branchrefusal, a self-test of the helper that stays.
- the closure
e2e
harness-smoke.cf.test.ts: kept. No lane cell can cover it. It asserts the workerd view ofCF_BACKENDS: the pool row is present and viable, and the Node-drivencf-doandcfw-workflows-d1rows are filtered out. It guards the kept row, so it is retired with it.
The text below is the original entry.
Plan 0c MR-1b repointed cfw-workflows-d1 to the Node-driven conformance lane and kept the legacy workerd adapter (packages/e2e/src/harness/setup-helpers/cfw-workflows-d1.ts, setupCfwWorkflowsD1) as a separate cfw-workflows-d1-pool row with CFW_POOL_CAPS (no branching), because live consumers still use it: runtime-cloudflare's lifecycle-hooks-parity.wf-noiso.test.ts, checkpoint-truth-matrix.wf-noiso.test.ts and cfw-d1-backend-smoke.wf-noiso.test.ts, and e2e's harness-smoke.cf.test.ts.
To close (MR-2 T9): for each consumer, check whether a cfw lane cell now covers the same contract; retire what is covered and keep the rest with a stated reason here. Retiring the last consumer removes the helper, its e2e/package.json export and the cfw-workflows-d1-pool descriptor. The pool-workers test:cfw-engine lane and runtime-cloudflare's own *.wf-noiso suites that do not use the helper are out of scope.
Owner: the conformance owner (plan 0c MR-2, T9).
FU-CONF-SHARD-BALANCE: the cfw conformance shard sets every pipeline's wall-clock — ✅ fixed (!491; owner: the conformance owner)
test:conformance shards with vitest's --shard, which splits by FILE, and vitest.conformance.config.ts includes exactly five files for parallel: 5 (pinned by ci-guard.ts), so each shard runs one lane file. The shards are now very uneven (job durations in s; orchestrator report 2026-10-23, read by the owner from the pipelines' jobs):
| pipeline | cfw | temporal | js | cf (host / client) | canaries |
|---|---|---|---|---|---|
| 2978937130 (!460, MR) | 2644 | 1385 | 732 | 498 | 127 |
| 2982622920 (train) | 2627+, still running at 106 cells | 1508 | 731 | 734 | 94 |
At !307 the cfw shard took 765-1120 s (FU-CONF-SOAK-BUDGET-AFTER-CFW), so it has grown about 2.5x as cfw cells were added (the stop-cascade cells alone take up to ~33 s each), and it now sets the wall-clock of every MR pipeline and every train car.
Options (pick by measurement, not by guess):
- Split the cfw lane (and, when needed, temporal) into several files by a deterministic scenario partition (e.g. a stable hash of the scenario id, or a balanced partition by recorded cell duration), and fold the near-empty canaries file into another shard; raise
paralleland theci-guard.tspin to match. Each file keeps its lane's own setup. - Shard below file level: a
CONFORMANCE_SHARD=i/nfilter applied by the lane file to its cell list, soparallelcan grow without new files (the conformance reporter must then prove no cell ran in two shards and none in zero). - Not: running a lane's cells concurrently within one file, since the cells share one Miniflare instance and the timing-sensitive controls assume one cell at a time.
Fix (this MR, the owner): seven files, one shard each (parallel: 7). The cfw lane runs in three files and the Temporal lanes in two (conformance-cfw-1 / -2 / -3, conformance-temporal-1 / -2; each calls scenarioPart(ALL_SCENARIOS, k, n), a deterministic round-robin over the id-sorted scenarios, src/conformance/scenario-parts.ts, with the lane setup in src/conformance/lane-parts.ts), the canaries moved into the js file (defineCanarySuite()), and conformance-cf is unchanged. conformance:verify already fails a pipeline on a cell with no result line or with two ("ran more than once across shards"), and scenario-parts.test.ts proves the split covers every scenario once. The MR's pipeline reports each shard's duration and the pipeline's peak concurrent pods (the ci-slice quota); if the quota trips, the fallback is six shards (cfw in two parts).
First measurement (!491's pipeline 2984208929, round-robin parts): shards 863 (cf) / 1,153 (cfw-2) / 886 (temporal-2) / 918 (temporal-1) / 700 (js + canaries) / 1,697 (cfw-1) / 837 (cfw-3) s; dbos 994, store 194; peak 22 concurrent jobs. cfw-1 was one cell: session.create-observed-whole took 1,039 s of its 1,539 s of cell time (757 s and 900 s in earlier pipelines: 48 sessions x 16 observers, about 10 s a session on cfw). So the cfw parts are now split by measured weight (scenarioPart's longest-first greedy, the table in src/conformance/lane-weights.ts): that cell gets a part of its own and the other two hold about 1,040 s each. That bounds the longest shard at about 1,230 s (the cell plus the job's setup), about 1.35x the median. Refresh the table from a recent pipeline's cfw cell times when the parts drift apart (scenario-parts.test.ts checks every key is a real executor scenario). The last step to 1.3x is to make that one cell cheaper on cfw: a lower per-lane session count could still catch RM-50 (pre-fix it was seen in 10-35 of 48 sessions on js-redis and 23-34 on temporal-redis), but no cfw pre-fix rate exists, and RM-50's proof rests on these cells, so measure cfw's pre-fix rate before choosing an N. The same pipeline's submit.partial-batch|cfw missed its 10 s .paused-with-two precondition; an order test (owner slot s104) showed no leftover state, so that bound is now a 60 s hang guard.
Closed (!491's pipeline 2984288307, the weighted cfw parts): shards 876 (cf) / 1,115 / 768 / 872 / 666 / 1,105 / 1,025 s; dbos 883, store 172; peak 14 concurrent jobs. The longest is 1,115 s against an 876 s median, 1.27x, inside the close criterion (it was about 2,650 s). The lane setup lives in src/conformance/lane-parts.ts, inside lint:conformance's scope (Greptile on !491; lint-confinement.test.ts checks it is not ignored). Open follow-on: the cfw session count of session.create-observed-whole (above), and refreshing lane-weights.ts when the cfw parts drift.
To close: a pipeline whose longest conformance shard is within about 1.3x of the median, with the shard-coverage guard proving every cell ran exactly once; record the before / after durations here. Re-measure the soak budget (FU-CONF-SOAK-BUDGET-AFTER-CFW) at the same time, since it runs the same cells unsharded.
FU-FLAKE-CONF-CFW-SESSION-CREATE-ATTEMPT-BOUND: session.create-observed-whole on cfw-workflows-d1 hit its 60 s per-attempt bound under CI load — open, decided (owner: conformance owner; user ruling 2026-10-24: per-stage guards)
- Seen: !504's MR pipeline 2985502933, job 17443284570 (2026-10-24): one attempt never settled within
SESSION_CREATE_ATTEMPT_BOUND_MS(60 s) and the cell failed, still in phaseobservers-done, with phases script 1 ms, observers-ready 9.7 s, run 38.8 s, observers-done 11.5 s (60.0 s in all). The same product passed in !504's previous pipeline, and the orchestrator's retry is the evidence of a flake, not a defect: the attempt was slow, not stuck. - Why it is tight: the bound (
scenarios/session-create.ts, one bound per attempt, the same on every lane: user decision 2026-10-06) was set at about 2.6x the slowest attempt in 5 CI runs then (max 22.9 s, p95 17.0 s on cfw). This attempt's run phase alone took 38.8 s: the cfw lane under the 7-shard matrix's load (FU-CONF-SHARD-BALANCE above). - Options (decision needed, not taken): (a) per-phase hang guards, each anchored on the observed phase transition (
enter(...)), so a slow but progressing attempt is never cut and a real stall is still named by its phase; (b) a larger single bound re-measured from recent cfw CI times; (c) fewer sessions on cfw (the open follow-on above), which also lowers the load. (a) keeps the user's "one stall named per session" intent and is the owner's recommendation. - Decided (user ruling, 2026-10-24): (a). Each attempt is three settles on
.runs-settle, one per stage, each bounded by its own guard from the observed start of that stage (SESSION_CREATE_STAGE_GUARDS_MS, the same on every lane): setup (script + observers-ready) 25 s, run 125 s, wrap-up (observers-done) 20 s. Each guard is at least 5x its stage's p95 on cfw-workflows-d1 in CI: 7 CI runs on 2026-10-24/25 (pipelines 2985451717, 2985419479, 2985388533, 2985350280, 2985344388, 2985321834, 2985556320; 336 attempts) gave setup p95 4.8 s (max 6.2 s), run p95 24.6 s (max 41.3 s), wrap-up p95 3.6 s (max 10.0 s), and per-attempt totals median 22.0 s, p95 31.7 s, max 48.0 s. A batch budget (SESSION_CREATE_CELL_BUDGET_MS, 50 min) caps each stage's wait at what is left of it, so a pathological lane still fails the cell, not the 90 m job (cell timeout 3 060 s, vitest timeout 3 300 s, plus 600 s for the shard's other cells:scenario-budgets.test.ts). It supersedes the single per-attempt bound (user decision 2026-10-06). The stage guards are pinned insession-create-scenario.test.ts, which also proves a slow but progressing attempt (58 s run, about 70 s in all) passes and a wrap-up stall is named in its phase. - To close: the change lands and the cell passes on cfw-workflows-d1 across 5 consecutive MR pipelines.
FU-CONF-SOAK-BUDGET-AFTER-CFW: measure conformance:soak once the cfw lane is in it — open
conformance:soak (.gitlab-ci.yml, scheduled only) runs the conformance matrix 3 times unsharded through scripts/conformance-soak-run.sh. Since plan 0c MR-2 (!307) that includes conformance-cfw.integ.test.ts, which takes about 360-420 s per run, so each soak gains about 6-7 minutes per pass. The estimate is about 1 h locally and 2.2-2.45 h in the pessimistic CI case, against the job's 3 h timeout. No scheduled soak has run yet, so there is no CI measurement. Decision (orchestrator, 2026-10-06): leave the 3 h timeout until a run is measured.
Update after the 5 CI measurement runs of !307 (jobs 16980311871, 16981126935, 16981528729, 16981899493 and 16982287641): the cfw shard took 765-1120 s in CI, above the 534-844 s (1.5-2x local) the pessimistic estimate used, so the estimate rises by up to about 280 s per iteration, about 14 minutes over 3 iterations: about 2.45-2.7 h, still under 3 h but at or above the 2.5 h trigger. session.create-observed-whole's per-attempt bound went from 30 s to 60 s (user decision 2026-10-06), which changes only the pathological worst case (its cell timeout was then 2 940 s; with the per-stage guards of 2026-10-24 it is 3 060 s, FU-FLAKE-CONF-CFW-SESSION-CREATE-ATTEMPT-BOUND), not a passing run's time; a soak iteration in which that cell really hits its worst case would overrun 3 h, but it is a failing run either way.
Trigger: the first scheduled conformance:soak pipeline after !307 merges. Record its wall-clock here. If it is over 2.5 h, raise the timeout or shard the soak (raising it needs the orchestrator's OK).
Owner: the conformance owner.
FU-CONF-PROVENBY-ON-GAPPED-CELL: a fixed finding's provenBy may name a cell that is a declared gap — done (owner batch 35)
Surfaced by: !372's review (orchestrator, 2026-10-16): dropping hold-ref-write from the Temporal lanes gapped companion.spawn-then-send-same-step there, and whether any fixed record's provenBy named one of those cells was checked by git grep only. checkDrift rejects a cell that is both a declared gap and ledgered (drift.ts), but nothing rejects a provenBy key whose cell is a declared gap on that lane, a fix "proven" by a cell that never runs. Fix: a drift rule failing a fixed finding whose provenBy names a declared-gap cell, with its unit test.
Done (owner batch 35): checkDrift (drift.ts) now reports <id>: provenBy key '<cell>' is a declared gap, with a unit test in drift.test.ts; the real catalogue has no such key.
FU-CONF-PROVENBY-CELL-LEDGERED: a fixed finding's provenBy may name a cell that is ledgered under another finding — open (LOW; owner: the conformance owner)
Surfaced by: RM-48's flip in the b37 cells MR (2026-11-20). stream.step-end-real-adapter proves RM-48's two step-end ids on every executor lane, but its .tool-frames-before-step-end id is ledgered under RM-141.
checkDrift(drift.ts) checks a fixed finding'sprovenBykeys for parse, known scenario, backend and tier, a lane that has run, a declared gap and aconformance:evidence line. Nothing compares them with the ledger.- So a fixed finding can be "proven" by a cell whose verdict is an expected-fail: its verdict says the ledgered ids failed and nothing about the others. RM-48 cites that cell in an evidence line and keeps it out of
provenBy, by convention only (as CP-138 kept CP-341's cells out). - Measured: no fixed record's
provenBynames a ledgered cell today (0 of 203 ledger cells, b37 records commit). - Fix direction: a drift rule failing a fixed finding whose
provenBykey is a ledgered cell, with its unit test beside the declared-gap rule's (FU-CONF-PROVENBY-ON-GAPPED-CELL).
FU-CONF-TEMPORAL-LARGE-PAYLOAD-HISTORY-READ: the executor driver's cell-end history read times out on a Temporal workflow that carries a ~2 MB payload — open (harness; owner: the conformance owner)
Owner: the conformance owner. Finding: MA-31's cell on the Temporal lanes (a declared gap until this closes).
state.oversized-tool-write-not-silent writes 1.9 MB into the session state. On temporal-memory and temporal-postgres (stores with no staging limit) that payload lands in the workflow's history, and the executor driver's cell-end check (fetchHistory, bounded at 3 s) times out reading it: the cell crashes at its end ("could not end the backend work … fetchHistory timed out after 3000ms") and the lane's teardown reports the env left running, whatever the product did (owner slot 2a, 2026-10-17, 3 of 3; temporal-redis passes, its store refuses the write first). To close: bound the history read by the history's size (or page it), then give the Temporal driver 'reap-large-payload' (the oversized-state cell's three Temporal lanes are declared gaps until then) and run the cell there.
FU-CONF-LATE-NOTICE-TEMPORAL-TEST-SERVER: the late-notice cell fails .partial-present on Temporal's test server only — open (harness first; owner: the conformance owner)
transcript.stopped-partial-omitted-after-late-notice passes on the three Temporal lanes on the docker dev server (CI's backend; CP-268's s274 / s275 ledgered it there). On the time-skipping test server (HELIX_TEMPORAL_BACKEND=test-server), temporal-memory fails .partial-present 3 of 3: the stopped run's interrupted partial is not in the log. It fails the same way at main fc97182526 and at Additive MR 4a's head (WE-2 slot w-mr4au, 2026-11-09, attr.txt); the cell's other ids fail only as its preconditions.
- Until traced: run this cell's Temporal rows on docker only.
- Owed: a trace of Temporal's stop path under the test server's timing, before Temporal work resumes after the features window. The capabilities note's held-activity retry hazard on the test server is a candidate, not a finding. If the cause is the product's own (the partial's write losing to the activity's cancellation under a timing a production server can also give), this becomes a catalogue finding, not a harness note.
FU-CONF-REDIS-LANE-TTL: the conformance Redis lanes expired a cell's keys after 60 s — done (W1, the Redis lane TTL MR)
Owner: W1. Finding: none (a latent harness hole that changed no verdict; no catalogue record, owner ruling 2026-11-04).
The hole. helpers/redis-setup.ts built every test Redis context with defaultTTL: 60 / streamTTL: 60, and the three conformance Redis lanes (js-redis, temporal-redis, the store tier's store-redis) used it unchanged, so any cell or mutation run longer than 60 s could read its own session as gone: a crash or a false red, never a verdict. Found by W1's CP-41 probe slot: under redis-cli MONITOR the store's own writes set EXPIRE … 60 on every session key, and a 120 s cell lost its sessions 60 s in with no delete command.
The fix. createRedisTestContext(testId?, { defaultTTL?, streamTTL? }) takes optional TTLs (the plain integration tests keep 60 s / 60 s), and the three lanes pass CONFORMANCE_REDIS_TTL ({ defaultTTL: null, streamTTL: 86_400 }: state keys never expire, stream keys after 24 h). Two guards: conformance/__tests__/redis-lane-ttl.test.ts (both TTLs outlive the longest scenario timeoutMs, computed at test time, and a source pin that each lane's setup call passes CONFORMANCE_REDIS_TTL, since the stream manager has no TTL getter), and harness/__tests__/redis-lane-ttl.integ.test.ts (each lane, built, has a state store whose getTTLConfig() is { defaultTTL: null, stagingTTL: null }).
The audit. W1 slot w1-ttl-slot-20261104-023416: the full js-redis lane (222 cells) and the full temporal-redis lane (222 cells) once each at the fix, 0 cells moved from the ledger, so no current verdict depended on the TTL.
FU-MEMORY-CF-RUNNER-INDEX-NOT-CLEARED: memory-cloudflare's contract runners keep the fake index between cases — open (test; owner: W1)
clearStore in memory-store-contract.integ.test.ts deletes the D1 rows and leaves the fake Vectorize's vectors, so a later case runs beside an earlier case's vectors. The plain Vectorize runner never reads the index (only the filtered cases pass an embedding, and those are scored exactly in D1). The over-fetch runner does: the filtered semantic and hybrid cases query it, so its result could depend on case order, though a top K of 50 over a few vectors hides that today. To close: each runner calls the fake's clear() in clearStore, or builds the index per case.
FU-CONF-MEMORY-STORE-TIER: no conformance tier runs a memory store — fixed (harness; owner: the conformance owner)
Fixed by W1's memory-store tier MR: tier memory-store, five lanes (memory-in-memory, memory-redis, memory-d1, memory-d1-vectorize, memory-d1-vectorize-overfetch), nine scenarios, 45 cells (25 run, 20 declared gaps). RM-85 is flipped from it and MA-47 ledgered on it. MA-29 stays open for its executor half; MA-32 is unchanged (its witness needs a euclidean fake index, not written yet).
Owner: the conformance owner. Findings: MA-29 (fixed in code by W1's MA-29 MR, witnessed only by the memory store contract), MA-32 (euclidean threshold).
The store tier (conformance-store) runs session-state stores only. Memory stores (@helix-agents/memory's in-memory, memory-redis, memory-cloudflare over D1 with an in-test Vectorize index) are covered by core's memoryStoreContractTests, whose runners live in each package, so a memory finding can be witnessed red and green there but never ledgered or flipped. To close: memory-store lanes on the store tier (or a sibling tier) that run the contract's cases by title, as store.ts does for the session-state contract, then flip MA-29 from them.
FU-CONF-STORE-UNDER-TEST-NOT-GATED-ON-NEEDS: the cell driver forwards storeUnderTest whatever a scenario's needs says — open (harness; owner: the conformance owner)
cellDriver (drivers/cell-driver.ts) forwards memoryStoreUnderTest only when the scenario declares 'memory-direct', so a memory scenario that reads the store without declaring it fails. The session-state store tier's storeUnderTest has no such gate: it is forwarded whenever the driver has one, so a store scenario missing 'store-direct' still runs instead of being refused. To close: gate it on scenario.needs.includes('store-direct') as the memory member is, with a test beside cell-driver.test.ts's memory case.
FU-CONF-NEEDS-CHANNEL-TEXT-CHECK-PROPERTY-NAMES: the needs-channel text check reads any .llm property as an observation read — open (harness; owner: the conformance owner)
scenario-needs-channels.test.ts checks, for every scenario, that a read of an observation channel in its run source declares the channel's capability in needs. It judges llm, runRecords and stopRecords (UNAMBIGUOUS) with a plain text match, \.<channel>\b(?!\s*\(), on the grounds that their names appear in scenario code only as observation reads. That does not hold for a property NAME: an assertion-id map key such as C.llm in own-hooks.ts (W1's branch, ${CHILD_ID}.before-llm-call-ran) matches as obs.llm, so the test reported "hooks.child-own-hooks-run: reads 'llm' without 'observe-llm'" and W1's slot aborted on a test fault. W1 renamed the key (beforeLlmCall); main has no such key today, so nothing fails, but the next id map or result field named llm, runRecords or stopRecords will.
- To close: match observation reads only: a read on an identifier bound from
driver.observe(…)(or typed as the observation), or(await driver.observe(…)).<channel>, rather than any.<channel>property; or skipas constid maps. Add guard cases beside the existing ones: an id-map keyC.llmis not a read;obs.llmand(await driver.observe(sid)).llmstill are. Keep the test's "Limits" paragraph in step with the new rule.
FU-CONF-SB-MIXED-SUSPENSION-PRECONDITION: the S-B cell's mixed-suspension precondition never settles — open (harness; owner: the conformance owner)
Owner: the conformance owner. Finding: WA's S-B (!409; the send over a client-call + blocking-companion suspension refused busy with nothing written).
chat.with-message-during-mixed-suspension (packages/e2e/src/conformance/scenarios/client-deadline.ts) scripts one parent step that calls a client tool and spawns a blocking companion whose model call is held, and waits for that hold. In five owner slots (2026-10-17/18: sb, c3, c4, c5, c6), on cf-do and js-chat-host, on the base and on WA's S-B tree, the hold never came within 30 s and turn 1 did not end first. The cell's timeout report (a poll of the parent's status, pending calls, sub-sessions and the child's model calls, through settle's describeTimeout) never printed, which is unexplained: the recorder appends it on every timeout. The cell is unregistered (not in the batch's ledger) until the shape is confirmed: next, run it alone with a log line from the poller, and check whether the companion spawn co-emitted with a client call is dispatched in-step on runtime-js (the CF DO runs it).
FU-CONF-LARGE-REPLY-BRIDGE-LATENCY: a scripted reply of ~1100 deltas takes over 90 s through the cf-do model bridge — withdrawn (wrong cause; MA-31)
Withdrawn (conformance owner, 2026-10-18; orchestrator ruling): the cause was not the bridge. Owner slot c3's logs show the DO's step commit failing on the reply's single 17.6 MB assistant message (SQLITE_TOOBIG on the messages INSERT): the run failed (a retryable transport_error, slot c4), and the cells waited only for completed. js-chat-host streamed the same reply in 381 ms. The misclassified failure is MA-31's (witness state.oversized-reply-fails-typed), and the CP-136 cells now spread the reply over 11 steps of about 1.64 MB each (owner batch 43r), with a 180 s run bound. The original entry follows.
Owner: the conformance owner. Finding: CP-136 (its read-ahead cell needs more than 1024 chunks).
stream.unread-client-worker-read-ahead-host streams 1100 text deltas (16 KiB each) so a count-bounded reader holds more than the byte budget. Its run did not complete within 90 s, and neither did the no-hold control (stream.large-reply-completes-host, the same reply with no client held; owner slot c2, 2026-10-18, 3 of 3): the slowness is the harness's model bridge (each scripted delta crosses it to the worker), not a held reader stalling the run. The cells now allow 240 s (READ_AHEAD_RUN_BOUND_MS). To close: measure the control's time, and if the bridge stays slow, batch scripted deltas across the bridge (or script the reply worker-side) so a large reply streams at production speed.
FU-CONF-CFDO-SOCKET-CLOSE-CAUSE: workerd closes the cf-do harness's keep-alive socket — fix in review (HIGH; harness; owner: the conformance owner)
Cause. The lane's Node -> workerd requests all went through Node's global pooled fetch, so they shared keep-alive connections, and a request sent on a connection workerd then closed failed fetch failed (ECONNRESET locally, UND_ERR_SOCKET: other side closed in CI): on local runs of stream.large-reply-completes-host (owner slots s100, s105), in CI on store reads (!486's train, !483) and on a mutation (!489: POST …/subagent/conf-smoke-branch/abort). fb0538e286's re-send of idempotent reads only hid the read case. One trigger is proven: workerd closes a keep-alive connection right after a Durable Object answers a request whose body it never read (runtime-cloudflare's workerd-keepalive-unsafe.integ.test.ts). The one close s105's socket probe caught does not fit it: the reset request was the 7th on its connection, reset 1.9 s after it was sent, and the 6 before it were all POST /do/:id/__conformance/store reads, a harness-only route that reads its body. So the trigger of that close is not established; every request on the connection was a harness route, and no product route was involved.
Fix. src/harness/cf-do/worker-connection.ts: every request of the lane (dispatch, so the store reads and the host / client drivers' routes, plus the health check, the harness-error drain and the fingerprint check) goes through an undici Agent that sets reset: true on each request, so each connection carries one request and a close can only end the request that caused it. worker-connection.test.ts makes the hazard deterministic (a socket that served a body-unread reply is destroyed when the next request arrives on it): red on the pooled client, green on this one. Every socket close is logged once per request as a FU-CONF-CFDO-SOCKET-CLOSE-CAUSE recurrence (naming the request), so a recurrence stays visible: the lane client logs a close before the response, for every route including the chat routes, and fetchJsonAt one while it reads a body (Greptile on !495). An idempotent READ (a store read, a status, alarm, open-count or stream-info GET) is still re-sent after a close, up to 3 attempts, each re-send logged; a call that mutates the DO never is. The chat drivers' streamed bodies are not wrapped (a wrapper would sit in their cancel path); a close there fails with the driver's own transport error.
Recurrence with this client (2026-10-24): !497's train car pipeline 2984965902 (main + this MR + !497), job 17440315603 (test:conformance 1/7): stream.large-reply-completes-host|cf-do|host crashed after 156 s on a store read, twice in that cell (POST /do/…-s/__conformance/store: the worker closed the socket … SocketError UND_ERR_SOCKET: other side closed, then store read listRuns … request failed). So with one request per connection, workerd still closes sockets under CI load, and the earlier version of this MR, which made every close fatal, turned it into a crash. The read re-send is restored (red-first: remote-state-store.test.ts, the three idempotent-read cases); the cause stays open.
Evidence (owner slot s105, at 237f55ad79): stream.large-reply-completes-host on cf-do, 3 runs with this client: 0 socket closes, 101 connections each carrying 1 request; 2 runs with main's pooled client swapped back in: 1 close (above), 4 connections per run carrying up to 18 requests.
To close: the trains after this MR show no recurrence line on the cf-do shard. The recurrence above is a close of a connection that carried a single request, so a different mechanism from the keep-alive race: the next step is to read the logged requests' timing under CI load (which route, how long after it was sent) before changing the client again.
FU-CONF-CRASH-LATE-TRAFFIC-POISONS-REBUILD: a crashed client cell's torn-down page poisons the lane's rebuild — open (owner: conformance owner)
In batch 16's ai-newest retry (pipeline 3043030569, job 17848935155), stop.finished-shown-on-reload-client|cf-do+ai-newest|client crashed (a harness crash: its held snapshot read never came). The rebuild that follows a crash then failed its health check with "session … reused / late traffic from closed cell 24: host.log": a request the crashed cell's torn-down page made after endCell. The lane was marked poisoned, and 13 later cf-do+ai-newest client cells were GATED (job 17848519400, the first run, rebuilt cleanly). A crash should not take unrelated cells down with it. Options: endCell waits (bounded) for the crashed cell's pages to go quiet before the rebuild's health read, or the rebuild's health check tolerates late traffic from the crashed cell's own sessions while still failing on any other session's. The crash itself is fixed (the reload holds its own page-load read, batch 16); this FU is the cascade.
FU-CFW-HOLD-WORKFLOW-STEP: no control holds the CF Workflows lane's workflow body at a step — open (harness; owner: WE-2, promoted if a second cell needs it)
Filed by: the !676 review (a stop between two owed children, O-73), orchestrator ruling 2026-11-04.
The workflow body runs in workerd, and the lane has no control that holds it at a named step.do. So no cell can lose a run between the delegation commit and registration, or place a stop between two owed children: with both children registered they have already started, so "the next child never starts" cannot fail, and placing the stop would be a timing bet. That order is proven only by the unit pin W5b (cfw-g1-delegation-commit.test.ts), which holds a step.do in its in-process harness.
To close: a hold-workflow-step control (self-test, capability pins, test:conformance-harness), then a CFW cell of the stop between children. Half a day; ruled not worth it for one cell ahead of the S1s, so a later cell needing the same hold promotes it.
FU-CONF-READ-AHEAD-CI-TIMEOUT: CP-136's read-ahead cell timed out on its first CI run — open (HIGH; harness, possibly product; owner: the conformance owner)
Owner: the conformance owner. Finding: CP-136 (open again until this closes).
stream.unread-client-worker-read-ahead-host (cf-do) streams ~32.8 MB (2,000 custom events of 16 KiB) while the attached client is held after its first bytes, and judges the worker reader's own buffered-byte high-water. Locally a run takes about 35 s (owner slot s49, 3 of 3). On its first CI run (!464 job 17302397122, 2026-10-20) the run did not complete within .run-completes' 120 s bound (the cell ran 204.9 s). The same job's no-hold control stream.large-reply-completes-host took 77.2 s for ~18.4 MB (18 s locally; 25 to 39 s in an earlier CI run), about 0.24 MB/s, so the runner may simply be too slow for this payload. A stall is not ruled out: a held client's backpressure reaching the DO's producer, so the run never completes. That would be the incident's own "work stops" symptom.
The cell is a declared gap on cf-do, and CP-136 is open again (pending this item). The cell now samples the run's progress every 5 s while .run-completes waits: the DO stream head (observe-stream-info), the persisted status, the bytes the worker took in and the client read. It logs one [conformance] observe line with that timeline on every run.
Calibrated, 2026-11-04 (WE-2 slots w-a1 and w-c136, the gap lifted in scratch trees only): a slow runner, not a stall. Under taskpolicy -b the head kept advancing and the run finished late (116.6 s and 118.7 s passed, 3 of 5 missed 120 s with the head at 2010 of 2013; the no-hold control slowed 22.3 s to 100.8 s). READ_AHEAD_RUN_BOUND_MS is now 360 s, about 1.76x CI's 204.9 s cell, with the 32.8 MB payload unchanged; the cell's 600 s timeout holds its 540 s worst path, pinned in scenario-budgets.test.ts. The Node process running the cell has its event loop saturated for the whole transfer (utilization 100%, delay max 3.2 to 8.6 s across seven runs), which is what starves the sampler's timers; the function responsible is not identified. So each probe now runs at an absolute due time and carries its own timestamp: a missed sample reads GAP, and a probe whose timer fired late reads busy (timer late N ms), a harness artefact that is never ledgered. Sampling can't follow the consumer's progress instead, because the reader runs in workerd and the client is held.
To close:
- Reproduce locally under a CI-like constraint (fewer cores / background QoS, several runs) and read the timeline.
- Head keeps advancing and the run finishes late: calibrate a payload and bound to the measured CI throughput (stated in the cell's comment, still past the reader's 8 MiB budget so the red mutation discriminates). Re-prove it green ×3 and red with the budget removed, un-gap it, and flip CP-136.
- Head stops while the client is held: file a new finding (likely S1) for the DO-streams owner, and keep the cell gapped until its fix.
FU-CONF-STREAM-DO-UNREAD-READER-HOLDER: no conformance lane can hold an unread StreamDurableObject reader — open (harness; owner: the conformance owner)
Owner: the conformance owner. Finding: CP-26 (its StreamDurableObject site is unit-evidenced only).
CP-26's eviction fix (!397) covers two SSE sites: the agent DO's /sse and the standalone StreamDurableObject's. The host-tier cell stream.unread-client-evicted-host (cf-do) holds the agent DO's reader worker-side (the chat route's upstream body, control-upstream-read). The StreamDurableObject is read by CF Workflows' binding-side stream manager, which on the cfw-workflows-d1 lane runs in Node, and Miniflare does not pass a Node-side stall through to the DO, so no lane can hold a StreamDurableObject reader unread. That site is proven only by !397's .cf unit test. The cell also observes "the slot is released" only indirectly (the held body ended before the turn's last sequence); there is no connection-slot count in the observation. To close: a worker-side holder of a StreamDurableObject /sse reader for the cfw lane (a conformance-worker route that opens the reader in workerd and stops pulling on request), then a cfw sibling cell of the unread-client scenario, and a connection count in the observation if one is exposed.
FU-CONF-DO-CHUNK-WRITE-FAULT: a chunk-write fault control on the cf-do host lane — open (owner: the conformance harness; with FU-CONF-CFW-CHUNK-WRITE-FAULT)
Filed: 2026-11-09 by WA, with CP-305 (the conformance owner: the DO is the priority runtime, and "by inheritance" is weaker than a row).
Why. CP-305's cells run at the executor tier, where 'fault-chunk-write' wraps the env's own stream manager. The cf-do host lane has no control that fails a write to the Durable Object's own stream: its bridge faults cover the DO's store ('fault-do-store'), its routes and its agent factory, and the bridge's only stream fault is CF Workflows' /info read ('fault-stream-info'). So CP-305 is proven on the DO by inheritance (the DO runs runtime-js's run loop, its tool executor and its drains, all fixed and pinned by runtime-js units through the real executor), not by a row.
What's owed. A bridge control that fails a session's DO-stream writes of one chunk type (once, N times, until disarmed; served counts; an unserved arm fails endCell), granted as 'fault-chunk-write' on the cf-do host lane, with harness self-tests; then a -host twin of tools.tool-end-write-keeps-result (the regular path) on cf-do, judged x3. Build it with FU-CONF-CFW-CHUNK-WRITE-FAULT, the same bridge shape.
FU-CONF-CFW-CHUNK-WRITE-FAULT: a chunk-write fault control on the CF Workflows lane — open (owner: the conformance harness; next harness item)
Filed: 2026-11-09 by WA, with CP-305 (the orchestrator: the next harness item, so CF Workflows' CP-305 fix gets a cell soon; CF is the priority platform).
Why. 'fault-chunk-write' (CP-305) is an in-process wrapper on the env's own stream manager, so it exists only on the lanes whose executor writes through it in the test process (the js, temporal and dbos-postgres lanes). CF Workflows' frames are written inside the workflow, across the bridge, which has no such control. CP-305's two cells (tools.tool-end-write-keeps-result, approval.approved-tool-end-write-keeps-result) are declared gaps there, and its CF Workflows fix is proven by runtime-cloudflare steps.test.ts units only, including the case where the catch path's own tool_end fails persistently and the body must run exactly once (no step.do re-run).
What's owed. A bridge control that fails a session's writes of one chunk type (once, N times, until disarmed; served counts; an unserved arm fails endCell), granted as 'fault-chunk-write' on cfw-workflows-d1, with harness self-tests. Then the two cells' gaps are removed and the cells run there.
FU-CONF-CFW-LATE-HARNESS-ERROR-FENCE: a late worker harness error crashes the next cell — open
On the CF Workflows (and cf-do) lanes, a worker-side BridgeClient.call failure is recorded as a harness error (harness/cf-do/worker/bridge-client.ts, recordHarnessError) and ingested by the bridge's pushError (harness/cf-do/bridge-server.ts). pushError applies no per-cell fence: the reap and leak fences live only in noteIfLate, for traffic. So a failure the worker records AFTER its cell's endCell (for example a runstart.enter / runstart.settled call of an abandoned run-start attempt whose rejection lands late) is swept by the NEXT cell's endCell and crashes that cell, which did not cause it. The message is labelled late error from previous cell <n>, session <sid>, so the misattribution is visible in the crash. Never observed (!325's Task 16A part 2: no runstart bridge failure in any cfw run).
Fix direction: apply the same fence noteIfLate applies (a session closed by a failing endCell, or reaped) to a late worker harness error, keeping it in bridge.late as a diagnostic.
Owner: the conformance owner.
Conformance follow-ups from owner batch 29
FU-EXAMPLE-REASONING-TEST-LIVE-MODEL-CHOICE: three examples/nextjs-redis tests pass only when a live model reasons or follows a script — open (LOW, test reliability; owner: WE-2)
Filed by: WE-2 (2026-11-07), from batch 14's example jobs. Each failed with the product behaving correctly:
tests/reasoning-display.spec.ts:77("reasoning content survives page refresh", job 17830025545, both attempts) fails at:96, before any refresh: the turn's SSE carried text only (noreasoning-*chunk), so the live model did not think on that prompt. Its sibling at:42(another prompt) did, the same job; the test's own comment admits the prompt was picked to trigger reasoning. It passed in the batch's first pipeline (job 17829050226).tests/brief-flow.spec.ts:391("text content survives multiple refreshes…", job 17830199709) fails before any refresh, "Baseline marker [B1] missing": the model wrote prose where[B1]…[B10]were scripted.tests/brief-flow.spec.ts:335("final message has correct interleaved structure", the same job's retry) timed out waiting for[C1]and[C10]: the final text carried no[C*]markers. Both page snapshots show the right interleaved structure.
Sightings: batch 20 (job 17914624370, 2026-11-10): reasoning-display.spec.ts:77 again failed at :96 on both attempts while :42 passed in the same job, and brief-flow.spec.ts:183 ("deterministic interleaved pattern…") failed once on "Missing marker [B1]" (prose where the markers were scripted) and passed on retry.
Fix direction: mock the model for structure tests, or judge structure by part kinds and order rather than marker text; for the reasoning test, force thinking for its agent, or fail with a clear "model did not reason" when the first turn's SSE has no reasoning chunk.
FU-EXAMPLE-OPENNEXT-UNTYPED-ERROR-RESPONSE: the opennext example's chat route answers every non-handler HelixError as an untyped 500 — open (owner: WE-2)
- What: the catch blocks of
examples/opennext-cloudflare-do/src/app/api/chat/[sessionId]/route.tstype only anAiSdkHandlerError(itsstatusCodeandcode); any other error, aHelixErrorincluded, is answered500 { error: message }with nocode. The submit action already relays typed refusals throughsubmitToolResultErrorResponse(W1's DI-51 branch), and the snapshot route answers throughsnapshotErrorResponse, so this covers the rest of the chat route: the send and the resume GET. - Seen live: the cf-do conformance worker's chat route mirrors this one (
errorResponse,harness/cf-do/worker/conformance-worker.ts). Insnapshot.read-failure-typed-host(RM-07, owner batch 29) the handler throws the typedDOPeerErrorfor an unreadable/snapshotand the client receives500 { error }without its code; the RM-07 record notes it as related. - Why it matters: examples are what users copy. A client cannot tell a retryable failure from a refused one, and
parseHelixChatErrorgets no code. - Fix direction: answer through core
typedErrorResponse(error, { inputCommitted })(the one builder of a typed HTTP error, withhttpStatusForHelixError's status), keeping theAiSdkHandlerErrormapping first; then mirror the change in the conformance worker'serrorResponseso the host cells see what the example answers. (The worker's snapshot mirror, which had drifted to a plain 404 / 500, follows the example's builders since owner batch 29.) - Owner: WE-2 (the CF examples). Cross-reference: DI-51.
CF Workflows per-step cost
FU-CFW-PER-STEP-COST: !288 makes a CF Workflows LLM step about 1.25× slower than before — open
Measured on the T9 workload (cfw-terminal-truth.wf-noiso, 50 steps), on the same machine and harness:
main 5bd30e20da | !288 | |
|---|---|---|
| wall time, idle | 117 ms/step | 144 ms/step (1.22×) |
| wall time, under load | 216 ms/step | 276 ms/step (1.28×) |
step.do per LLM step | 6 | 7 |
| D1 round-trips per LLM step | ~55 | ~44 |
| stream-DO calls per LLM step | ~6 | ~9 |
D1 got cheaper. The extra time is the committed-count step.do, plus three stream-DO calls: an /info head read for the commit cursor, an extra /init, and the step_committed marker /write. Real CF Workflows users pay this.
Consolidation to do:
- fold
committed-countinto the step commit'sstep.do; - piggyback the head read and the
step_committedwrite on calls the step already makes; - drop the redundant
/init.
Guard it with a step / round-trip count test.
Related: FU-SUBAGENT-RECORDING-PER-CHILD-CALLS (the per-child D1 calls of a sub-agent wave). The 9 full-history reads a turn with one sub-agent wave makes (its LLM steps and the wave's fold step) are per turn, so they belong here.
FU-SUBAGENT-RECORDING-PER-CHILD-CALLS: a CF Workflows sub-agent wave still makes 12 D1 calls per child — open
Owner: unassigned (perf; the next increment after FU-SUBAGENT-RESULT-RECORDING-RELOADS-PARENT). Related: FU-CFW-PER-STEP-COST.
- Measured (statement counts at 4 vs 16 children,
concurrent-subagent-fold.integ.test.ts"sub-agent wave D1 cost"): a 16-child wave with a 1-message history is 324 calls = 132 per turn + 12 per child. The 12:- the parent state row twice (the record step, and the append commit's own read);
- the run record twice and
current_run_idonce (fences); - the refs once;
- one idempotency scan page plus the
COUNT(*)D1getMessagesruns with every page (2); - the append commit's
COUNT(*)and its batch (2); - the ref update's two writes (
UPDATE refs,UPDATE states.updated_at).
- Next increment (about 12 → 8 per child):
- pass the child's agent type (CF Workflows already has it) so the helper skips the refs read;
- drop the per-page
COUNT(*)from D1getMessages(a store change); - fold the ref update into the result's append commit (a store change).
- Guard: the test's budget (≤ 330) and per-child equality; lower the budget with the cut.
UI-chunk schema conformance (DI-20) follow-ups
Surfaced by the DI-20 fix (docs/superpowers/specs/2026-09-26-ui-chunk-schema-conformance-design.md). Finding IDs are catalogued by the conformance owner.
FU-DI20-01: @helix-agents/llm-vercel never reports LLM step boundaries (RM-48) — fixed in code (lands with its MR)
Fixed: the adapter's fullStream loop reports each step's finish-step through onStepEnd (usage and ai's unified finish reason), once per streamText call, and the dead onChunk step cases are gone. onStepStart stays unwired: every runtime writes its own step_start (I2). MockLLMAdapter now reports steps the same way by default (stepCallbacks: 'end'). The original description:
vercel-adapter.ts maps chunks inside streamText({ onChunk }), and ai only calls onChunk for text/reasoning/source/tool/raw parts, never start-step / finish-step (the adapter iterates fullStream but ignores those parts). So the adapter never fires onStepStart / onStepEnd: no core step_start / step_end chunks, no per-step usage / cachedTokens / finishReason on the Helix stream, and includeStepEvents emits nothing with the production adapter. The opennext-cloudflare-do example sets includeStepEvents: true but gets no step events (its comment now says so).
FU-DI20-02: DBOS emits step_start but never step_end (RM-49) — fixed in code (lands with RM-48's MR)
Fixed in code with RM-48's fix (2026-11-19): runtime-dbos already wires onStepEnd (steps/call-llm.ts:224-237), and the fixed VercelAIAdapter now calls it, so DBOS writes a step_end per LLM step; the fix branch flips the it.fails pin below to a passing case. The stepId gap (RM-49's related line) is separate.
runtime-dbos/src/steps/call-llm.ts writes step_start (without stepId) directly; there is no step_end anywhere in runtime-dbos. packages/e2e/src/__tests__/ai-sdk-dbos.integ.test.ts pins the gap with an it.fails finish-step non-vacuity test that turns red once DBOS emits step_end.
FU-RJS-HELD-STEP-END-LOST-ON-CRASH: a runtime-js / DO process that dies between the model call and the step's commit loses the step's step_end — open (LOW)
Surfaced by: RM-141's fix (2026-11-21). A step's step_end is now written after its call frames, before the commit that records the step (ai's step order). runtime-js and the CF DO hold it in memory (core holdStepEndReport, released by the step iterator), so a process that dies after the model call and before that commit loses it: the attempt keeps its step_start with no step_end, so that failed run's UI stream has a start-step with no finish-step. A crash during the model call already leaves the same shape. The durable runtimes record the held chunk with the model call's result, so their replays still write it. A fix: carry the held report on the saved reply's metadata at commitSavedCalls, so saved-call recovery writes it.
FU-DI20-03: live vs reload shape of a denied approval differs — open (pre-existing)
A live denial settles the tool part via a synthetic tool-output-available (deny string), while the replay path (replay-events.ts, output-denied state) emits tool-output-denied, so the part state differs between the live stream and a reload. The replay output-denied state has no production producer today.
FU-DI20-04: snapshot converter emits a top-level filename on file parts — open (pre-existing)
helix-to-aisdk-converter.ts puts filename on reloaded file parts, while the live file chunk now carries it in providerMetadata.helix.filename (DI-20). No core file chunk producer exists today.
FU-AISDK-FOREIGN-TOOL-OUTPUT: a tool result for a call an earlier message holds failed the whole response — done
Surfaced by: W1 (MR2b Task 5b-r23), with a probe on the real StreamTransformer and ai 6.0.49's readUIMessageStream. Owner: cf-we-do-streams-2. Design: option 2 (orchestrator ruling, 2026-10-10). Catalogue: DI-48 (open until W1 MR2b's path-4 cell, chat.send-over-pending-client-calls, turns green with this fix in).
ai's processUIMessageStream applies a tool-output-* chunk only to a tool part of the message the response renders into, and throws No tool invocation found for tool call ID for any other call (6.0.49 dist/index.mjs:4845-4855, 6.0.281 :5874); the response fails and Chat drops the rest of it, so the reply is lost, on an uninterrupted client too. A run's stream legitimately carries results for calls an earlier message holds (a with_message entry's consumed or abandoned client-tool calls, an approval answered earlier, a refresh tail re-delivering an entry's frames), and the transformer forwarded every one (stream-transformer.ts:944-1010 on f12eef982d). The host tests read raw SSE, so none saw ai's throw.
Fix (ai-sdk only; the runtime stream is unchanged): a result is forwarded only for a call the response message owns (its input emitted in this response, live, replayed or in the C-1 prefix; or listed in the new StreamTransformerOptions.continuedToolCallIds, which handleChatStream sets for a client-tool / approval continuation and for a resume of a client-held message without the prefix). Any other result becomes a data-tool-output-elsewhere resync trigger (a third member of AISDKResyncTriggerEvent), and the client re-anchors on the snapshot. Plain useChat re-anchors only through useAutoResync(collectResyncTriggers(messages)); without it the reply renders but the earlier message keeps its last-seen state until the next load (documented in docs/guide/resumable-streams.md). The path-4 end-to-end proof (a send over pending client-tool calls) ships with W1's MR2b, which fixes that path and rebases onto this fix.
LLM attempt retries and the error chunk (E5) follow-ups
E5 made every runtime write an LLM step's error chunk only for a failure that ends the run (core holdLLMErrorReport). The Temporal attempt fence and the DBOS exhausted-step settle came with it. These two items are what is left.
FU-E5-LLM-ERROR-STEP-NOT-RETRIED: llm-vercel error steps are never retried on Temporal / DBOS — open
@helix-agents/llm-vercel never throws: it returns every provider failure as an error step (shouldStop: true), including a transient, retryable: true one (a 429, a 5xx, a dropped connection). Temporal's runLLMStep and DBOS's runCallLLM retry only a THROWN generateStep (the activity retry / the step retry), so on those runtimes a transient provider error fails the run at once. CF Workflows: done (CF-D1 MR 1b, !287). Its LLM step rethrows a retryable error step so step.do retries it under llmStepRetry, and its onError goes through holdLLMErrorReport (MA-26 MR 1 moved it onto the shared rule, core isRetryableStepErrorDetail: an unclassified error is no longer retried, DI-35). runtime-js / the CF DO: done (MA-26 MR 1). The core step iterator retries a retryable: true error step inside runStepIteration under the executor's llmStepRetry, each retry a new attempt of the same step. The item stays open for Temporal and DBOS (MA-26 MR 2). Temporal, forced calls only: done by completion strategies MR 2 (a forced call's retryable or unclassified error step is rethrown for the activity retry, with its detail; MA-26 MR 2 moves it to the shared rule, so an unclassified one stops being retried); an ordinary call's error step still fails the run at once. A fix there rethrows a retryable error step inside the activity / step, so the durable retry runs it again with no error chunk (E5 already holds the report). It must also keep the exhausted retry's last classified detail (DI-24).
FU-E5-EXHAUSTED-DRAFT-NOT-DISCARDED: a failed attempt's draft is not marked discarded when the retry runs out — open
When every attempt of an LLM step fails (Temporal's activity retry, DBOS's step retry, and since MA-26 MR 1 the runtime-js / CF DO in-step retry), the run fails with the last attempt's error. The attempts' streamed text is never marked discarded (step_discarded, as discardInterruptedStep does for an interrupt). A connected client keeps the dead draft on screen next to the error, and a refresh does not show it. The terminal failure path should emit step_discarded for the in-flight step before the error chunk on Temporal (finalizeTerminalRun after a runLLMStep failure), DBOS (the turn settle in runAgentLoopOneTurn) and runtime-js / the CF DO (the run loop's failed-step exit).
FU-CP225-THROWN-LLM-ERROR-CHUNK-NO-CODE: a thrown model call's error chunk carries no code on runtime-js / the CF DO — open
CP-225's second half. When an adapter's generateStep THROWS (instead of returning an error step), core runStepIteration's catch writes an error chunk with only error (the message): no code, no errorDetail. Every onError chunk (runtime-js / the DO's buildLLMCallbacks, CF Workflows, Temporal, DBOS) carries both, and so do the other runtimes' terminal paths for a thrown error. So on runtime-js and the DO a live client gets no data-error-metadata code for such a failure (the snapshot's errorDetail has it after a reload, CP-225 MR 1). Fix: the catch writes code (when classified) and errorDetail = resolveErrorDetail(err), recoverable: false, with a core unit red. Sequenced after MA-26 MR 1 (!520), which restructures the same catch (a thrown retryable error is retried before any chunk); owner: W1.
FU-LLM-WORKERD-CONNECTION-LOST-NOT-RETRIED: on Workers, a dropped connection to the LLM provider fails the run instead of retrying — fixed
Closed by: !319 (the connection-drop retry MR; spec docs/superpowers/specs/2026-10-07-llm-connection-drop-retry-design.md). llm-vercel's mapVercelError classifies a drop in either phase by its structured properties (workerd's plain Error with own retryable: true, not overloaded; undici's TypeError whose cause.code is UND_ERR_SOCKET or ECONNRESET) as provider_error, retryable: true, and the adapter retries a drop before any output (maxRetries ?? 3, 2 s doubling backoff, abortable), on every runtime. A drop after output started is a retryable error step; only CF Workflows retries it today (FU-LLM-STEP-RETRY-PARITY, MA-26). Proven by the conformance provider.drop-before-output (cf-do, cfw), provider.drop-after-headers, provider.drop-reset-after-headers, provider.drop-mid-output-classified and provider.drop-reset-mid-output-classified (all ten lanes) cells (DI-37). Surfaced by: CI-health-3, from main's opennext job 17002761997 (50252a54), 2026-10-07.
The record below is the pre-fix state, as filed.
- What happens: workerd's
fetchto the provider rejects withError: Network connection lost., carrying workerd's own hintretryable: true(andremote: true). The AI SDK wraps only Node'sTypeError: fetch failedand Bun's network codes into a retryableAPICallError(@ai-sdk/provider-utils4.0.27handleFetchError), so it neither wraps nor retries this one. Helix'smapVercelError(llm-vercel/src/error-mapper.ts:77, pre-fix, origin/main a67fc68801) then falls through toclassifyError, whose fallback is a non-retryableframework_internal_error. The step ends{ type: 'error', retryable: false }and the run fails. - Evidence: in that job 3 runs failed, all on
Network connection lost.; the logged step error is{"code":"framework_internal_error","retryable":false,"cause":{"remote":true,"retryable":true}}. They account forclient-tool-multiturn:26(second turnfailed, no reply) andbrief-flow:231. - Why it matters beyond CI: a production Worker whose connection to the provider drops mid-request fails the user's turn, where Node (Undici's
fetch failed) would have been retried. - Fix direction (as filed, before the fix): classify a workerd network error (an
Errorcarryingretryable: truefrom the runtime) as a retryable provider/transport error inmapVercelError(or coreclassifyKnownError), so each runtime's LLM retry policy applies. Red-first with a thrownObject.assign(new Error('Network connection lost.'), { retryable: true, remote: true }), plus a workerd integ test whose provider fetch drops once. Check each runtime's retry path (CF DO step iterator, CFWllmStepRetry, Temporal, DBOS) and that the held error report (holdLLMErrorReport) writes noerrorchunk for the retried attempt.
FU-LLM-ABORTED-CALL-NO-USAGE (#180): a model call cut short by Stop / interrupt reported no usage — fixed in code (lands with its MR)
Issue: #180 (from the Broadcast app's live validation: every turn interrupted mid-text reached afterLLMCall with usage: undefined, so the provider-billed input was never booked).
- Cause: Anthropic sends its input and cache usage in
message_startand its output count once, inmessage_delta, at the end.@ai-sdk/anthropicemits usage only in its finalfinishpart,streamText'sonAbortcarries completed steps only. A Stop mid-text ends the stream with no finish step: the adapter returned the partial text withusageundefined (result.usagerejects with the abort reason), and an error aftermessage_starthad none either. - Fix:
@helix-agents/llm-vercelalways requests the provider's raw parts in its per-attempt model wrap (providerCallBoundary) and drops every one there, so none reachesstreamText. It reads Anthropic'smessage_startusage (readRawUsage, Zod) and puts it on the cut call's result (the partial text, or an error result) when its finish reported none: a LOWER BOUND, input and cache only.completionTokens/totalTokensstay absent, and the provisionaloutput_tokensis ignored.- No step end is reported for the cut attempt.
- Every runtime already forwards the result's
usagetoafterLLMCalland the usage store. - Covered: an
@ai-sdk/anthropicmodel object, direct or through an Anthropic-format proxy (LiteLLM/v1/messages). - Not covered: a gateway string id or a V2 model (no wrap), and an OpenAI-compatible route (no usage on the wire).
- Docs: core
TokenUsage/ErrorStepResult.usageJSDoc, the usage guide (a callout: compute cost from the per-kind fields, nevertotal),docs/upgrade-guides/aborted-call-usage.md. - User rulings (2026-10-27):
TokenUsage.partialis docs only (no new field; an absentcompletionTokensreads "output not reported"); dropped-attempt usage deferred (FU-LLM-VERCEL-DROPPED-ATTEMPT-USAGE); no new upstream issue. - Upstream: tracked by vercel/ai#7628 (open, a feature request: providers do not report usage on abort), draft PRs #21792 (main) / #21793 (release-v6.0: provisional usage on
response-metadata), and #21778 (totalUsagesums the finished steps after an abort). The raw-chunk tap is theai6 workaround (FU-LLM-VERCEL-V7-MESSAGE-START-PART). - Witness: the conformance owner's cell (an Anthropic-shaped provider stream behind the real adapter, stopped mid-text; cf-do + js first). It also settles whether an interrupt that cancels a Temporal activity / DBOS step before
generateStepreturns skipsafterLLMCallthere.
FU-USAGE-BOOKING-INDEPENDENT-OF-HOOKS: a throwing afterLLMCall hook can skip a model call's usage booking — open
Filed by: W1 with CP-242's CF Workflows fix (orchestrator ruling, 2026-10-28: file it, do not fix it there).
- The rule: a model call's usage is booked whatever its
afterLLMCallhook does. The provider billed the call, and a hook failure is not a billing event. - Holds:
- CF Workflows (CP-242):
reportModelCallbooks the usage in afinallyaround the hook, on the completed path and on the stop branch (runtime-cloudflare/src/steps.ts). - Temporal:
runLLMStepbooks the usage (recordUsageBestEffort,runtime-temporal/src/activities.ts:3779) before it firesafterLLMCall(:3844). - runtime-js / the CF DO without
options.hookManager:buildHookManagerregisterscreateUsageHooksfirst (runtime-js/src/js-agent-executor.ts:8095-8101), so an agent hook or anoptions.hookshook that throws runs after it. The DO passes onlyhooks(runtime-cloudflare/src/do/drive/run-drive.ts:1158).
- CF Workflows (CP-242):
- Gaps:
- runtime-js with
options.hookManager: the executor registers the usage hooks onuserManager.createChild()(js-agent-executor.ts:8079-8092), andDefaultHookManager.invokeruns the PARENT's handlers first, with no catch (core/src/hooks/hook-manager.ts:35-37). So the caller's ownafterLLMCallthat throws ends the invoke before the child's usage hook: the step fails (core step iteratorfireAfterLLMCall) and the call books nothing. - DBOS:
fireAfterLLMCallruns beforerecordStepUsage, on the stop path and the normal path (runtime-dbos/src/workflows/shared.ts:3037-3038,:3043-3044), and a hook throw propagates out ofrunHookStep(steps/hooks.ts,runRunHook). So a throwing hook skips the booking.
- runtime-js with
- Fix shape: book the usage outside the hook chain, after it, in a
finally(as CF Workflows does now): on runtime-js, a step-iterator booking (or the usage hooks on a manager that runs before the caller's parent); on DBOS,recordStepUsagein afinallyaroundfireAfterLLMCall. Red-first per runtime: a throwingafterLLMCall(on runtime-js throughoptions.hookManager) and one usage entry for the call.
FU-LLM-VERCEL-DROPPED-ATTEMPT-USAGE: a connection drop after message_start but before any output books nothing for the dropped attempt — open (deferred by user 2026-10-27)
Filed by: W1 with #180's fix. Decision: deferred by the user, 2026-10-27: a follow-up later, out of #180.
- What happens: a drop before any model output is retried inside
generateStep(FU-LLM-WORKERD-CONNECTION-LOST-NOT-RETRIED). The dropped attempt never surfaces as a result. If the provider had already sentmessage_start, that attempt's billed input reaches neitherafterLLMCallnor the usage store; only the final attempt's result carries usage. - How rare: it needs a drop between the response headers (with
message_start) and the first content block. - Open question: how an internal retry reports a dropped attempt's usage (a per-attempt report, or a sum on the final result), which is new API surface.
FU-LLM-VERCEL-V7-MESSAGE-START-PART: on the move to ai 7, read Anthropic's message_start stream part and drop the raw-chunk tap — open (with the ai 7 upgrade)
Filed by: W1 with #180's fix.
- Why:
@ai-sdk/anthropic4 (withai7; checked 4.0.90 /ai7.0.155) emits acustomstream part withkind: 'anthropic.message_start'carryingproviderMetadata.anthropic.usage, on by default, andai7 forwardscustomparts.@ai-sdk/anthropic3 (withai6; checked up to 3.0.135 /ai6.0.317) has no such part, which is why #180 reads the rawmessage_startthroughincludeRawChunks. - Do: when llm-vercel moves to
ai7, read the usage from that part (same lower-bound mapping asreadRawUsage), dropincludeRawChunksandwithRawUsageTapfromproviderCallBoundary, and keep #180's tests (the Stop / LiteLLM / error / finish-usage-wins cases) green. If vercel/ai#21792 lands first (provisional usage onresponse-metadata), prefer that provider-neutral field.
FU-CFW-EXPORT-LLM-STEP-RETRY-DEFAULT: export CF Workflows' default llmStepRetry and use it in the conformance budget pin — open (owner: CI-health-3; after CF-D2 MR1)
Surfaced by: the conformance owner's review of the provider-connection-drop control (2026-10-07).
- CF Workflows' default LLM-step retry (
{ limit: 3, delay: '1 second', backoff: 'exponential' }) is an inline literal inruntime-cloudflare/src/workflow.ts, not an exported constant. The conformance budget pin forprovider.drop-*(packages/e2e/src/conformance/__tests__/scenario-budgets.test.ts) therefore repeats its first delay as a literal (1_000) with a source comment. - Fix: export the default from
runtime-cloudflare(e.g.DEFAULT_LLM_STEP_RETRY) and use it in both places. Waits for CF-D2 MR1, which ownsworkflow.ts.
FU-LLM-STEP-RETRY-PARITY: a provider drop after output started fails the run on runtime-js, the CF DO, DBOS and Temporal ordinary calls — open (owner: orchestrator to assign)
Surfaced by: CI-health-3 (spec 2026-10-07-llm-connection-drop-retry-design.md §4). Pre-existing on Node too.
- What happens:
VercelAIAdapter.generateStepreturns{ type: 'error' }; only a runtime that readsretryableoff that step and throws retries it. CF Workflows does (steps.tsisRetryableStepError→step.dounderllmStepRetry); Temporal does for forced calls only (activities.tsretryForcedError). runtime-js and the CF DO (core step iterator) and DBOS (call-llm.ts's@DBOS.stepretries only a throw) have no LLM retry, so a mid-output drop (provider_error,retryable: truesince the connection-drop MR) fails the run. - Why not in the connection-drop MR: the adapter retries only before output (nothing to supersede). A retry after output needs attempt supersede on the JS / DO stream (as CFW and Temporal have) and its own spec.
- Conformance:
provider.drop-mid-outputledgers the failing lanes under MA-26.
FU-ERROR-CHUNK-RECOVERABLE-PARITY: Temporal, CF Workflows and DBOS write recoverable: false on every error chunk — open (owner: orchestrator to assign)
Surfaced by: CI-health-3, 2026-10-07 (the consumer grep for the connection-drop MR).
The contract: an
errorchunk'srecoverableis "whether the client can retry", mapped from the error'sretryable(docs/guide/error-handling.md, "ErrorChunk");docs/frontend/react.md("Typed Stream Errors") shows retry UI on it throughHelixStreamError.retryable(data-error-metadata). runtime-js (and the CF DO, which writes its error chunks through runtime-js) honours it (run-loop.tsonError:recoverable: error.retryable), pinned bye2e/src/__tests__/error-handling-pipeline.integ.test.ts.The deviation: Temporal (
activities.tsonError), CF Workflows (steps.tsheld report andfailRunStream) and DBOS (call-llm.ts) hard-codefalse, so a retryable failure (a rate limit, a provider drop) never offers retry UI on those runtimes.Fix direction: write
recoverable: error.retryableon each; cross-runtime test on thedata-error-metadataevent.Check first: the remote-sub-agent consumers (conformance owner, DI-38). core
remote-sub-agent-dispatch.ts, runtime-jsremote-subagent-executor.ts, CFWsteps.tsand DBOSexecute-remote-subagent.tstreat arecoverable: truetransporterrorevent as "the stream dropped: re-check status or retry". As traced on 2026-10-07 (CI-health-3), those events never carry a child's in-banderrorchunk'srecoverable:- agent-server emits its own (
recoverable: trueonly when its reconnect retries are exhausted;falsefor a failed session) and passes the child's chunks through aschunkevents; - the HTTP and DO stub transports build
errorevents from those frames, from the DO'sfailframe (which carries{ error, errorDetail }and norecoverable, sofalse), or from a malformed frame (true).
If that still holds when the fix is made, the chunk-level change does not reach the parents' drop-retry path. Re-verify it (a test: a remote child failing with a retryable error must surface to its parent as
failed, not enter drop recovery) before changing any runtime, and change the consumers in step if any path lifts the chunk's flag.- agent-server emits its own (
FU-DBOS-LLMCONFIG-SERIALIZE-DROPS-FIELDS: DBOS passes the adapter only temperature of an agent's llmConfig — done (the DBOS live-agent MR, batch 28; owner: WA)
Surfaced by: CI-health-3, 2026-10-07 (tracing maxRetries for the connection-drop MR).
- What happens:
serializeAgent(runtime-dbos/src/serialize.ts) keepsllmConfigas{ model: 'serialized', temperature, maxTokens: agent.llmConfig?.maxOutputTokens }, and the call config is{ ...agentSerialized.llmConfig, model }(workflows/shared.ts).maxTokensis not anLLMConfigfield, so on DBOS an agent'smaxOutputTokens,topP,topK,presencePenalty,frequencyPenalty,stopSequences,seed,headers, agent-levelproviderOptionsandmaxRetriesnever reach the adapter. Silent; DBOS only. - Fix direction: serialize every JSON-safe
LLMConfigfield under its own name (the model stays the registry placeholder); a cross-runtime test that every field reachesgenerateStep'sconfig.
Closed (2026-11-16, the DBOS live-agent MR): instead of serializing fields, the workflow body resolves the registered agent's live systemPrompt and llmConfig per process (resolveLiveAgent, runtime-dbos/src/live-agent.ts) and every call's config is the agent's whole llmConfig with the live model (workflows/shared.ts), so every field reaches the adapter; serializeAgent keeps only the model placeholder. The cross-runtime test is the conformance cell materialize.llm-config-reaches-adapter (MA-27, fixed): one id per setting, every executor lane.
CF-D1 MR 1b (!287) follow-ups
!287 made runtime-js, the CF DO and CF Workflows end a failed run with its real, classified error. These items are the Temporal and DBOS halves, which the CF-D1 MR 2 (Temporal) and MR 3 (DBOS) rounds own.
FU-CFD1-DI03-TEMPORAL-DBOS: a hard abort on Temporal / DBOS carries no framework_cancelled — open
Owners: CF-D1 MR 2 (Temporal), CF-D1 MR 3 (DBOS).
DI-03's record names runtime-js, the CF DO and CF Workflows, which now record framework_cancelled (category framework, retryable: false) for a hard abort. Neither runtime-temporal nor runtime-dbos writes that code anywhere (grep framework_cancelled finds nothing outside tests), so an abort there is indistinguishable from any other uncoded failure. To close: record the typed cancellation at each runtime's abort seam (the handle's abort() and the workflow side that observes it), and prove it with an executor-tier cell. That cell needs a hard-abort driver primitive, which the conformance harness does not have yet (it has only the soft stop()); ask the conformance owner for one.
FU-CFD1-DI22-TEMPORAL-DBOS: a classified cause on Temporal / DBOS terminal seams — open
Owners: CF-D1 MR 2 (Temporal), CF-D1 MR 3 (DBOS).
Temporal LLM-error half closed by completion strategies MR 2: runLLMStep returns the classified detail (terminal.errorDetail), which the workflow persists and passes to onAgentFail. DI-04 (Temporal's other activity failures) and DBOS remain.
resolveErrorDetail() now walks the cause chain (DI-22) and both runtimes call it in their handle's catch, but their workflow-side terminal seams persist only the detail that reaches them. Temporal's top-level workflow catch falls back to the wrapping ActivityFailure (DI-04), and an ordinary LLM error reaches both as { message } (DI-24). To close: carry the classified detail across the activity / step boundary so the walk has something to find, then prove it with the discipline.error-step-detail cells (ledgered under DI-24 today) plus a wrapped-cause case.
FU-DBOS-SIZE-ERROR-DETAIL-CODE: a DBOS commit refused as too large may persist { message } without the code — open (parked; owner: the parked DBOS bucket)
Filed by: MA-31. DBOS's step-result round-trip strips a HelixError's code (DI-24), so a state_size_exceeded at promoteStaging / the terminal commit is expected to fail the run with errorDetail: { message } only (not exercised by a test; DBOS's usual stores, Postgres and Redis, refuse a commit only at Postgres's jsonb cap). The staging path is unaffected: the typed row's marker is plain data and survives. Fixed with DI-24.
FU-CFD1-CP71-DURABLE-RUNTIMES: a throw inside the agent loop on DBOS / CF Workflows — open
Owners: CF-D1 MR 3 (DBOS). CF Workflows: done (plan 0c MR-2, !307; see below).
CP-71 (a throw inside the loop, for example a throwing systemPrompt function, strands the session) is fixed on runtime-js and the CF DO and proven by lifecycle.loop-throw-fails-run on js-*. The record never checked the durable runtimes; the cell now does:
- Temporal passes it on
temporal-*, on main as well: it never stranded the session. - DBOS fails it on
dbos-postgres:execute()/result()throws the prompt's error instead of resolving a failed result, the persisted session never turns terminal, and a follow-upexecute()throws the same way. Catalogued as CP-94 (the prompt is resolved inexecute(), before any run). - CF Workflows passes it:
lifecycle.loop-throw-fails-run|cfw-workflows-d1|executorpassed in all 5 of plan 0c MR-2's runs (16.1-16.2 s per cell; the engine's retries of the throwing step are DI-35). The CF Workflows half of this item is closed; the DBOS half (CP-94) stays open.
CF-D1 MR 1c (!291) follow-ups
!291 gave runtime-js, the CF DO and CF Workflows one stop decision (core resolveStopDecision), a typed framework_max_steps_exhausted failure for a maxSteps cut-off, retry() that continues such a run, and RunMetadata.completionReason for stop_when / max_steps stops. Verbatim string tool results (RM-32) reached all five runtimes. These items are the Temporal and DBOS halves, a store finding, a JS / CFW divergence and an open product question.
FU-CFD1C-DI29-DI25-TEMPORAL: Temporal ignores stopWhen and completes a maxSteps cut-off — open
Owner: CF-D1 MR 2 (Temporal).
The Temporal workflow still calls shouldStopExecution with maxSteps alone (DI-29) and ends a cut-off run of an agent without outputSchema completed with no output (DI-25). To close: decide stop / continue through core resolveStopDecision with the agent's stopWhen (C1, C2: a throwing predicate fails the run once, never retried; the decision must be replay-deterministic, so evaluate it in an activity or record it); fail a max_steps stop with createMaxStepsExhaustedError through the terminal path so the detail reaches the result, the session, the stream and onAgentFail (C3); write completionReason: 'stop_when' | 'max_steps' on the run record (C4) so planRetry's C5 rule continues a cut-off run with a fresh budget (the retry entry must accept an empty plan.input). Note that Temporal child workflows are also uncapped (FU-TEMPORAL-CHILD-MAX-STEPS). Proof: flip the pinned divergence in e2e/src/__tests__/stop-conditions-parity.integ.test.ts (assertADivergent / assertBDivergent → assertA / assertB) and clear the ledgered discipline.stop-when-honoured, discipline.max-steps-exhausted and lifecycle.retry-after-max-steps cells on temporal-*.
FU-CFD1C-DI29-DI25-DBOS: DBOS ignores stopWhen and completes a maxSteps cut-off — open
Owner: CF-D1 MR 3 (DBOS).
The same two gaps on DBOS (workflows/shared.ts passes maxSteps alone). To close: wire resolveStopDecision (C1, C2) inside a recorded step, fail a max_steps stop typed (C3), and write completionReason (C4) so planRetry's C5 rule applies to DBOS retry(). DBOS's own completionReason mapping (DI-27, 'failed' for every text answer) must not write 'max_steps' on a completed run any more. Proof: flip the pinned divergence in e2e/src/__tests__/stop-conditions-parity-dbos.integ.test.ts and clear the ledgered cells on dbos-postgres.
FU-DELETE-SESSION-ORPHANS-CHILDREN: deleteSession(parent) leaves its child sessions behind — done
Owner: cf-w1-cfw-stores-submit (W1 MR3; CF-D2 handed it over). Closed by: W1 MR3 (cf-w1-cfw-stores-submit; the MR number is added when it opens). Finding: RM-64 (reserved; catalogued in owner batch 10). Surfaced by: CF-D1 MR 1c, Task 8 (C7).
deleteSession(parentSessionId) deletes only the parent's own rows on memory, Redis, Postgres and D1 (store-cloudflare/src/d1-state.ts: WHERE session_id = ?); DO-SQLite does not support deleting sessions. A child session id is derived from the parent session id and the tool-call id (core generateSubSessionId, <parent>-sub-<callId>, on runtime-js and the DO; <parent>__sub__<callId> on CF Workflows), so a reused parent session id (execute → deleteSession → execute) whose model repeats a tool-call id makes the new child reuse the old child's session, history included, and the child's model sees the deleted conversation's turns. Seen on the CF Workflows real engine (the C7 case: the second child ran a new turn on the preserved <sid>__sub__dup-call session); in principle on every runtime, since each derives the child session id from the parent session id and the call id. Real providers mint unique tool-call ids, so it needs a deterministic model or a provider that numbers calls per conversation. To close: cascade deleteSession to the session's sub-sessions (its sub_session_refs, recursively) on every store, or key child session ids on something a delete retires. Prove it with a store contract case and a cross-runtime e2e case.
Resolution (W1 MR3, RM-64; spec 2026-10-09-cf-w1-mr3-delete-session-cascade-design.md). deleteSession(S) now deletes S and every descendant on memory, Redis, Postgres and D1: the session's LOCAL sub-session refs plus the sessions whose parentSessionId names it, minus the sessions a REMOTE ref names (the new required SubSessionRef.location). The walk is cycle-safe and starts from S even if it is gone, so a retry finishes an interrupted delete. Redis's parent index now follows parentSessionId on every state write; D1 also deletes usage rows. DO-SQLite refuses with a typed framework_not_supported. Proof: the store contract cascade cases on every store, the cross-runtime e2e (cross-runtime-delete-session-cascade*.integ.test.ts, delete-session-cascade-cfw-engine.wf.test.ts) and the conformance scenario session.delete-cascade. DBOS was divergent until RM-75: FU-DBOS-CHILD-WORKFLOW-ID-REUSE. Upgrade guide: docs/upgrade-guides/delete-session-cascade.md.
FU-USAGE-STORE-DELETE-SESSION: memory and Redis deleteSession leave usage rows behind — open
Owner: the store maintainers. Surfaced by: W1 MR3 planning (RM-64).
Postgres and D1 own their usage rows, so their deleteSession cascade deletes the usage of every deleted session. The memory and Redis state stores keep usage in a separate UsageStore the state store cannot reach, so deleting a session leaves its usage rows. To close: give the state store a way to reach the usage store (or add a deleteSession-time hook that the executor wires), then assert usage deletion on those stores through the contract's rawHooks.usage hooks.
FU-DBOS-CHILD-WORKFLOW-ID-REUSE: a deleted-then-re-executed parent reuses its companion's DBOS workflow result — done (RM-75; the sub-agent half by child identity MR 1)
Finding: RM-75 (S1), the DBOS divergence from the RM-64 cascade. Surfaced by: W1 MR3 (RM-64), its cross-runtime e2e and conformance scenario.
DBOS treats a workflow id as an idempotency key. Child identity MR 1 minted a sub-agent's session id per dispatch, so its workflow id was already new after deleteSession(parent) and a re-execute. A companion's session id is still derived from the parent id and its name, and its workflow id was derived from that session id alone, so the re-executed parent got the OLD recorded companion workflow and the companion's model did not run.
Resolution (RM-75, interim until child identity MR 3). A companion's workflows are keyed on the parent RUN that starts them (runtime-dbos/src/persistent/child-session-id.ts: companionSpawnWorkflowId for a first spawn, companionContinuationIds for every continuation site), as the CP-149 respawn already was: a run id is minted per entry and fixed for the run, so a replay derives the same id and a re-executed conversation a new one. Session and run ids are unchanged; later deliveries reach the workflow through the routing marker, and a ref whose child recorded none (its loop never launched) is answered child_not_running, never sent to a derived id. Proof: the unit cases in execute-companion-tool.test.ts ("RM-75"), cross-runtime-delete-session-cascade-dbos.integ.test.ts (its it.fails flipped) and the session.delete-cascade cells on dbos-postgres. MR 3's DBOS port must keep or replace the salt (FU-DBOS-CHILD-DISPATCH).
FU-REDIS-STAGING-INDEX-SET: Redis staging delete scans the whole keyspace — open (performance; owner: reliability orchestrator (route))
deleteOneSession and cleanupOrphanedStaging find a session's staged steps with SCAN MATCH session:staging:{keyPart(id)}:*, which walks every key in the database. Since MA-34 the match is exact (an encoded id has no : or glob character), so this is a cost, not a correctness, problem. Fix: a per-session SET of staged step ids (SADD in the stage script, SREM on promote / discard), read by both deletes. Ruled an FU, out of MA-34.
FU-REDIS-INDEX-ZSET-TTL: the idx:sessions:* ZSETs get no TTL while session keys expire — open
Owner: the store maintainers. Surfaced by: W1 MR3 planning (RM-64). Pre-existing.
The Redis store sets defaultTTL on a session's own keys, but the idx:sessions:* index ZSETs (all sessions, by status, by parent, ...) get no TTL, so an entry for an expired session stays until something removes it. With the delete cascade, a TTL-expired child can stay listed under a deleted parent's index. To close: expire index members with their session (a periodic prune of members whose hash is gone, or a score-based TTL sweep), and cover it with an integration case.
FU-DELETE-SESSION-REMOTE-CHILD-AFTER-PARENT-GONE: a delete repeated after the parent is gone deletes a store-sharing remote child — open
Owner: the store maintainers / the cf-w1 backlog. Finding: RM-64 (a limit of its remote exclusion). Surfaced by: the W1 MR3 self-review (fix round 3).
On every store that cascades (memory, Redis, Postgres, D1), a remote child is excluded from deleteSession(parent) only by the parent's location: 'remote' ref. Once the parent is fully deleted its refs are gone too. A deleteSession(parent) repeated after that (for example a retry after a reply lost on a dropped connection) still starts its walk from the missing parent, finds the remote child through the parent index / parent_session_id, and deletes it when it shares the store and names the parent. A delete that did not finish is safe: the parent and its refs are deleted last. To close: make a remote child's exclusion independent of the parent's refs surviving. For example, store a remote marker on the child session itself (set when the remote host creates it under a forwarded parentSessionId), or skip the parent-index / parent_session_id children of a session that no longer exists unless they were reached as local. Prove it with a contract case: delete the parent, then delete it again, and the store-sharing remote child survives.
A second entry point has the same root cause: Temporal's claim release. TemporalAgentExecutor.execute() deletes a never-run session it just claimed when startWorkflow fails (executor.ts, the claimedNewSession release) and when the run-start entry is abandoned (releaseAbandonedFreshSession). With the cascade, that delete also walks the session's parent index / parent_session_id children. If the id is reused (an earlier session of that id was deleted while a store-sharing remote child that names it outlived it), the claim release deletes that remote child. The fix above covers it; its contract case should include a fresh claim of the reused id followed by its delete.
FU-CP157-REMOTE-STUB-FIDELITY: cfw-cp157-remote-park's hand-built remote tool carries both _isSubAgent and _isRemoteSubAgent — open (LOW, test only)
Owner: W1. Filed by: W1, on the orchestrator's ask (!657's Greptile review). The stub in packages/runtime-cloudflare/src/__tests__/cfw-cp157-remote-park.test.ts is not what createRemoteSubAgentTool makes (only _isRemoteSubAgent, core types/tool.ts), so a check that tells the two kinds apart cannot fail there. To do: build its remote tool with the real factory (or set only _isRemoteSubAgent) and re-run every CP-157 case.
FU-CP157-TEMPORAL-DBOS-REMOTE-PARK: Temporal and DBOS do not park a parent on a paused remote child (CP-157 MR 1 is CF-first) — open
Record: CP-230 (split from CP-157, which !504 fixed for runtime-js, the CF DO and CF Workflows; owner, 2026-10-25).
Ruled: 2026-10-20 (orchestrator, CF-first). CP-157 MR 1 parks a parent on a paused remote child and routes a root submit to the producer on runtime-js, the CF DO and CF Workflows only. On Temporal and DBOS a paused remote child keeps the CP-45 typed, retryable failure: the call fails rather than waiting for the child's client tool or approval.
- Not silent: neither runtime writes
remotePendingCalls, so the submit router's remote branch is unreachable there; if it were reached, their router calls pass noremoteresolver and it answers the typedframework_not_supported(spec2026-10-19-cp157-remote-child-suspension-design.md§9, the runtime matrix, with file:line). - Wanted: the CP-157 park classification in Temporal's activity (
executeRemoteSubAgentDispatch) and DBOS'sexecute-remote-subagentstep, their root'sremotePendingCallswrite, and the router'sremoteresolver (remoteTransportForParkedCall+transportForRefViaToolLookup), with the same tests as MR 1.
FU-G1-CFW-REATTACH-PARKED: a CF Workflows re-attach of a lost delegation fails a child parked on root-answerable calls — open (known limit of G1 / CP-216 MR 2; owner: WE-2)
Ruled: 2026-11-03 (orchestrator, accepted as a known limit of G1 MR 2). A run that re-enters with no input over a delegation its ended run lost (retry(), a continue resume) re-attaches each delegated remote child before its next LLM step (runtime-cloudflare workflow.ts, the owed-delegation block; steps.getOwedDelegation). That re-attach runs outside any model step, so it has no step suspension to park the parent with. A re-attached child that is PAUSED on calls the root can answer (CP-157's park) is therefore failed typed, as a pause the root cannot answer: the child is interrupted and the call paired with its typed failure, and the model sees it. The live step path and CP-157's own re-attach (a parent parked by a committed suspension) still park.
- Why it is rare: it needs the parent's instance to die while its remote child waits, AND that child to pause on a root-answerable client tool or approval before the re-entry.
- Wanted: the re-attach parks the re-entered run on the child's calls, as the step's cascade suspension does (a
suspended_awaiting_childrencommit with aremotewait, the root'sremotePendingCalls), then CP-157's resume path re-attaches as usual.
FU-CP157-MR2-DURABLE-POLL: a remote child answered on the producer's own side does not wake its parked root — open (MR 2, owner: WE-2; closes CP-185)
CP-157 MR 1 wakes a parked root only through its own submit router (the root's answer is forwarded, then the root continues). A call answered somewhere else — the producer's own UI, another consumer — leaves the child running or finished while the root still waits; a reload re-presents the call, and a re-answer is already_completed (CP-185: a DIFFERENT re-answer is discarded silently). Wanted (spec 2026-10-19-cp157-remote-child-suspension-design.md §6): while parked, the consumer polls the child's /status on a durable timer with backoff (5 s doubling to 5 min) and re-enters the parent when the child leaves paused — the DO an alarm, CF Workflows step.sleep + a status step, Temporal a timer + an activity. With it MR 2 may park on EVERY remote pause (MR 1 parks only answerable ones).
FU-RJS-REMOTE-EXECUTOR-TESTS-MODEL-REATTACH: runtime-js remote-executor tests seed a ref for the id the call just minted — open (LOW, test realism; owner: W1)
Surfaced by: child identity MR 1 (!535), its test migration. About seven cases in runtime-js/src/__tests__/remote-subagent-executor.test.ts model "a previous attempt left a ref for this child" (seed lastSequence, skip addSubSessionRefs, the ref-without-identity group, crash-recovery emit-once, the D-2 streamId re-attach) with a mock store that returns a ref for the id the call itself just minted. Since MR 1 a non-reattach call always mints a new child id, so in production that state is reachable only through reattach: true, childSessionId. The code path and assertions are the same. To close: re-model those cases on a reattach call that names the seeded child.
FU-REMOTE-REF-STREAM-IDENTITY-BEST-EFFORT: a failed stream-identity write makes a parent resume replay the remote child's stream from 0 — open
Owner: unassigned (reliability orchestrator to route). Finding: RM-64 (a consequence of the ref-first split). Surfaced by: the W1 MR3 final review (m-4). Pre-existing exposure.
The remote call writes its location: 'remote' ref before it starts the child, then records the child's stream identity (remote: { streamId, lastSequence }) on the ref once transport.start returns or a status pre-check knows it (core remote-sub-agent-dispatch.ts and runtime-js remote-subagent-executor.ts, recordStreamIdentity). That second write is best-effort: a failure is logged at warn with parentSessionId, parentRunId and remoteSessionId, and the call goes on. The ref is then running without remote, and a parent resume reconnects the child's stream from sequence 0, replaying everything the child already streamed. Every chunk is re-proxied. The parent stream drops one it already holds (RM-86: a high-water per child run, inside the write), but a parent resume is a NEW run, whose run-start clears that high-water (its clients read past the earlier copies). So the whole replay is written again after the new run's commit, and a client sees the child's output from its start once more. Before MR3 the single post-start write was best-effort too. The DO store persists the identity too since S-C (CP-141: its sub_session_refs.remote_json column). To close: decide whether the identity write should fail the call like the ref write (a retried call records it again), or be retried in place; prove it with a case where the first identity write fails and a resume does not replay from 0. A ref without the identity is still remote by its location (core isRemoteSubSessionRef): runtime-js's entry settle (S-C) and stop cascade (CP-150). (CF Workflows' entry reconcile, RM-80, was deleted with RM-69: no CF Workflows commit leaves a started remote child's call unanswered.)
FU-REMOTE-PARKED-CALL-DEADLINE: a remote child's parked client call renders with no deadline in the root snapshot — open
Filed by: late-answers MR 2 (SDK-3, WA). Owner: tree client calls MR T1 (where it should land). SDK-3 puts a client call's deadline on its part: the live tool-input-available (providerMetadata.helix{ deadlineAt, serverNow }) and a reloaded snapshot's pending part (markPendingCalls, from the pending entry), so the browser skips a call that lapsed. A REMOTE child's call parked at the root renders from SessionState.remotePendingCalls (markRemotePendingCalls, ai-sdk/src/handler/snapshot.ts), whose entries store no deadline, so after a reload that part has none and never lapses on the client (the handler runs; the server answers a lapsed call as it does today). The live part is unaffected when the child's tool_start is proxied with its fields.
- Where it lands: tree client calls MR T1 (
docs/superpowers/plans/2026-10-26-tree-calls-t1.md, branchdesign-tree-calls-t1-plan) foldsremotePendingCallsinto the inbox, keyed withownerSessionId. The inbox entry should carry the call'sdeadlineAt, and the snapshot's tree-call rendering should stamp{ deadlineAt, serverNow }asmarkPendingCallsdoes. - Wanted: a snapshot case with a parked remote call past its deadline: its part carries the deadline, and
useResumeClientToolsskips it.
FU-DBOS-REMOTE-STEP-THROW-NO-TOOL-END: a DBOS remote sub-agent step that throws past its retries writes no terminal frame for its call — open (owner: the parked DBOS bucket)
Filed by: CP-280 (2026-11-06, the conformance owner, from WE-2's CP-280b MR). Since CP-280b, DBOS's remote path announces the parent's call (tool_start, subagent_start) and ends it (subagent_end, one tool_end) in runExecuteRemoteSubAgent (runtime-dbos/src/steps/execute-remote-subagent.ts). A step that THROWS past its retries (for example a StreamDropError) leaves runRemoteSubAgentCall before any frame is written, and the workflow's catch, outside the step, turns it into a failed call. So that call's live tool part is left unfinished (no tool_end), while the stored failed row is right on a reload. The fix: the workflow's catch writes the call's terminal pair (once, and not for a call a stop cut, CP-277's rule), or the step catches its own exhausted failure. A unit pins one tool_end with the error for a step that throws.
FU-DBOS-REMOTE-SETTLE-ATOMIC: DBOS writes a remote child's terminal ref before the workflow appends its result — open (owner: the parked DBOS bucket)
Filed by: the remote settlement MR (WE-2; Invariant S, RM-98 / RM-99). runtime-js, the CF DO and CF Workflows settle a remote ref inside the commit that writes its call's result (MessageCommit.settle). DBOS does not: its remote step writes the ref terminal through a best-effort persistProgress (runtime-dbos/src/steps/execute-remote-subagent.ts :362-375, called at :409-465 and :643), and the workflow appends the result later, in its step commit. A crash between the two leaves the ref terminal and the call unpaired (the CP-146 shape; a later entry heals it not_executed, never with the child's real outcome on DBOS).
- Wanted: the step returns the outcome without writing the ref; the workflow's step commit carries
SaveAndPromoteOptions.settle(coreremoteSettlementsFor), as the other runtimes do. Plus the DBOS integ witness.
FU-REMOTE-STOP-BEFORE-START-TOMBSTONE: a stop can reach a remote child before its in-flight /start creates it — open
Owner: unassigned (reliability orchestrator to route). Finding: CP-150 (a residual window of its class; the catalogue record points here). Surfaced by: WE-2's CP-150 design.
When a parent run that is not executing is stopped, the stop cascade stops each live remote child through its transport by session id (runtime-js stopChildren; the CF DO's do/drive/stop-children.ts). If an earlier execution (a crashed process, a reset Durable Object) sent the child's /start and that request is still in flight, the stop can reach the child first. The child does not exist yet, so it answers not found or state_not_stoppable, the /start then lands, and the child runs to its end. The next parent entry settles it (S-C / RM-80: a running child is orphaned and stopped), but a stop with no later entry leaves it running. The window is a dead executor plus one request's latency. CF Workflows has the same cascade since CP-152 (stop-remote-children.ts), with the same window. Temporal and DBOS are not checked. To close: a child-side stop tombstone keyed on the D-2 start request key, so a /start arriving after its stop is refused.
FU-CFW-COMPANION-COMMENT-DRIFT: two CF Workflows comments describe companion behaviour the code no longer has — open (LOW, docs; owner: the conformance owner)
Surfaced by: the companion-lifetime trace (owner, 2026-10-20).
runtime-cloudflare/src/workflow.ts:2443(main 16ae386528) says a parent's live persistent children "persist their own terminal SubSessionRef". On CF Workflows the child never writes its parent's ref: everyupdateSubSessionRefinworkflow.ts/steps.tstargets the instance's own session. The parent's next turn syncs the stalerunningref from the child's own read (checkCompletedChildren,steps.ts:9778-9784).workflow.ts:3218-3222says a detached continuation proxying to the parent's ended stream "throws 'Stream is ended, cannot write' and fails the child". A child's chunk to an inactive stream is now a guest drop logged at debug (core/src/stream/run-markers.ts:117-160,store-cloudflare/src/stream-durable-object.ts:536-540).
Fix: correct both comments. No behaviour changes.
FU-CFW-REMOTE-CHILD-STOP-OWED-RETRY: an unconfirmed CF Workflows stop of a remote child is never retried on its own — ✅ fixed by CP-203 (WE-2)
Fixed: CP-203 generalizes it across runtimes. A live owed ref is re-stopped by the next entry's lost-step sweep (D3) or a later cascade; a stop still owed on a SETTLED ref is retried by core retryOwedRemoteStops at every entry and in the stop cascades, with backoff (30 s doubling, capped at 1 h) and a 24 h give-up (remote_stop_owed_abandoned, abandonedAt kept). The CF DO's own cascade reads live refs only; its settled debts are retried at its next entry.
Owner: unassigned (reliability orchestrator to route). Finding: CP-152 (its named limit). Surfaced by: W1's CP-152 fix.
When a stop ends a CF Workflows parent run that nothing executes, stopChildrenOfRun stops each live remote child through its transport, bounded at 10 s. A stop the child does not answer (the bound, a failed transport, no transport for the child) stamps the parent's ref stopOwed: { kind, at } and is not sent again on its own: CF Workflows has no alarm (the CF DO retries its owed child stops from its alarm, MR C1). Since RM-69 a step lost with its instance commits no call, so no committed call waits on the child; the next entry reaches it through D3 instead (!458: once its run-start commit won, it interrupts the child and settles the ref failed with no row). An interrupt that is NOT confirmed keeps its owed stop on that settle (SubSessionSettle.stopOwed), so the ref reads failed with stopOwed set: the child may still run and nothing resends the stop. A later stop of the session's run never reaches it either (the cascade stops live refs only). To close: a durable retry, e.g. a reader (getHandle(), result(), buildSnapshot's reconcileSession) that resends an owed stop older than a backoff, or a short-lived Workflows instance that retries it with step.sleep.
FU-CFW-PHASE1-INDEX-BEFORE-SUSPENDED-SIBLING: a CF Workflows phase-1 tool's onMessage index can count an earlier sub-agent call that later suspends — open (LOW)
Owner: unassigned (reliability orchestrator to route). Surfaced by: !458's Greptile fix (d).
A step's rows commit once, in call order, and a call with no row in that commit occupies no position. rowIndexFor counts only the earlier calls that do (or will) commit a row: a sub-agent fold's index is taken once its wave settled (a suspended sibling, a pending client or approval call and a finishWith the suspension defers are skipped). A phase-1 regular or approved tool's index is taken when it runs, before the step's sub-agent calls: an earlier sub-agent call that later suspends is still counted there, so that tool's onMessage index is one past its row. To close: fire the phase-1 tools' onMessage once the step's rowless calls are known (with the step's other folds), and prove it with a sub-agent call that suspends before a regular tool.
FU-CFW-TEST-MOCK-STEP-COMMIT-DROPS-ROWS: runtime-cloudflare test mocks of saveStateAndPromoteStaging drop the rows the step commit carries — open (test integrity)
Owner: unassigned (reliability orchestrator to route). Surfaced by: !458 (RM-69), whose workflow.test.ts mock had the same gap and failed 93 cases once a step's later results moved into its one commit.
Since RM-69 a CF Workflows step's later results (companion, sub-agent, remote, phase-2 finishWith) ride the step's ONE commit, saveStateAndPromoteStaging(sessionId, state, rows, …). These test-local mock stores answer that call without appending rows to the session's messages: agent-hooks-invocation.test.ts, concurrent-subagent-fold.test.ts, executor.test.ts, executor-resume.test.ts, executor-retry-genesis.test.ts, executor-suspended-result.test.ts, hooks-updatestate.test.ts, skills-catalog.test.ts, skills-dispatch.test.ts, steps.test.ts, steps-forced-completion.test.ts (all under packages/runtime-cloudflare/src/__tests__/), plus the second-level overrides spread from createMockStateStore() in workflow.test.ts (fixed for the base mock in !458). No current case reads a step's later row back, so they pass; a future regression on the RM-69 path (a row lost from the step commit) could pass there unseen. To close: one shared mock-store helper whose step commit appends its rows as every store does, used by these files.
FU-CFW-STOP-CASCADE-CHILD-REF-CRASH-WINDOW: a cascade that dies between a child's terminal and its parent-ref write leaves the ref running — ✅ fixed (CP-184)
Owner: unassigned (reliability orchestrator to route). Finding: CP-159 (ruling C's residual). Surfaced by: W1's CP-159 parent-ref fix (!452).
stopChildrenOfRun writes a local child's parent ref from the child's terminal: at once for a child it ENDS, from the terminal it OBSERVES (within 10 s) for a child it SIGNALS, else stopOwed. The write is not atomic with the child's terminal. If the cascade's process dies in between (an executor-site stop, or the body's stop-children step lost with its instance after the child ended), the ref keeps running with no stopOwed stamp. Narrowed by !452's Greptile fixes: the cascade now writes the ref only after the child's own children were visited, and a cascade that runs again (the body's stop-children step re-run) treats a live ref over an ended run as ended (it reaches the children, then writes the ref). Still open where no cascade runs again (an executor-site stop that dies): the companion ref sync leaves an interrupted child's ref running, and a companion__sendMessage then finds a running ref over an ended child. The cascade's ref writes are now generation-scoped (CP-161: core writeChildOutcome at the generation it checked, so a newer consult's reset is never overwritten and terminated is never replaced), which bounds what a late write can do but does not close this window. To close: a reader-side reconcile of a local ref whose child run ended (the same rule the cascade writes), with a test that kills the cascade between the two writes.
Fixed by CP-184 (design C). (2a) Core's lazy sync (resolveAndSyncSubSessionRefs, behind getChildStatus / listChildren / waitForResult on every runtime that uses core's companion dispatch) folds a LIVE ref over a child whose current run ended interrupted with its own K5 to interrupted (no completedAt, no notice), scoped to the status and outcome generation it read; a companion resume landing between that read and the write moves the generation (every resumedCompanionRef run-start now bumps it, live ref or not), so the write is dropped stale. (2b) CF Workflows' cascade retries its ref write (3 attempts, CASCADE_REF_WRITE_ATTEMPTS) and records a write it lost (StopChildrenOutcome.local.refWriteLost, logged at error); runtime-js's companion ref write (recordPersistentChildNonCompletion) retries the same way instead of swallowing its failure. Witnessed by the conformance cells companion.stop-cascade-ref-write-lost (2b) and companion.stop-cascade-ref-write-retries-spent (2a) on cfw-workflows-d1.
FU-CFW-STOP-CASCADE-OWED-STAMP-STATUS-REWRITE: the cascade's owed-stop stamp rewrites the status it read, at the same generation — ✅ fixed (CP-195, !458)
Owner: done. Finding: CP-159 (its residual after CP-161's scoping). Surfaced by: CP-161's cascade-scoping review.
stampLocalStopOwed re-states the status it read (status: ref.status). It is now scoped twice: to the generation stopLocal checked (CP-161, core writeChildOutcome), and, by !458's CP-195, a compare-and-set on that status (UpdateSubSessionRef.expectStatus, op 'stamp'): an outcome written at the same generation during the wait refuses the stamp (SubSessionRefConflictError, logged stop.child_ref_settled_meanwhile, no stop owed) instead of being put back to running. The remote path's stopOwed writes carry the same status CAS. Pinned by stop-children.test.ts ("a settle that lands meanwhile is never reverted (CP-195)") and the five-store stamp.* contract cases.
FU-CFW-STOP-CASCADE-SPAWN-IN-FLIGHT: a CF Workflows stop cascade skips a local child whose run has not committed yet — open
Owner: unassigned (reliability orchestrator to route). Finding: CP-159 (its named limit). Surfaced by: W1's CP-159 design.
stopChildrenOfRun stops a live local child through its current run (core stopRun). A child whose spawn is in flight (its ref registered, its instance created, its run-start not committed yet) has no run: the cascade logs stop.child_not_started and skips it, because at the executor's sites (a stop or abort of a run nothing executes, a reader's settle) it does not know the child's instance id to terminate it (the body's own wave does, and terminates a child not yet created, propagateStopToChild). The child then starts and runs under a parent that has ended. To close: record the child's instance id on its ref at spawn (or derive it from the parent run and call id, as the body does) so the cascade can terminate an unstarted child, and prove it with a stop issued between the spawn step and the child's run-start commit.
FU-TEMPORAL-CP159 / FU-DBOS-CP159: a stop of a parent parked on a suspended child does not reach the child — open
Owner: unassigned (parked with Temporal / DBOS). Finding: CP-159 (out of scope there). Surfaced by: W1's CP-159 design.
CF Workflows stops a suspended local child with its parent (stopChildrenOfRun, CP-159). Temporal and DBOS were not checked: a parent suspended_awaiting_children stopped from outside may leave its child suspended. To close: the same cascade at each runtime's stop of a run nothing executes, with a cell for a parent parked on a child suspended on a client tool.
FU-TEMPORAL-CP201 / FU-DBOS-CP201: a failed parent's ephemeral children are not checked to stop — open
Owner: unassigned (parked with Temporal / DBOS). Finding: CP-201 (out of scope there). Surfaced by: W1's CP-201 fix.
runtime-js, the CF Durable Object (CP-201) and CF Workflows stop a FAILED parent's ephemeral children (cause 'failure': companions keep running, C3). Temporal starts a blocking child with PARENT_CLOSE_POLICY_TERMINATE (runtime-temporal/src/workflow-dispatch.ts), so the engine may end it with its parent's workflow, but a parent suspended_awaiting_children whose resume fails was not checked; DBOS was not checked at all. To close: a cell on each lane for a parent that fails while its ephemeral child is suspended on a client tool (the child's run ends, the parent's ref is terminal, a live companion keeps running).
FU-DO-CONTRACT-HARNESS-OVERRIDES-STORE-METHODS: the DO store contract runner replaces createSession and deleteSession with its own code — open (test integrity; owner: a CF slot)
Owner: unassigned (a CF slot, orchestrator to route). Finding: none (test integrity: a false green). Surfaced by: the owner's store-contract skip audit (2026-10-17, source only).
The DO session-state contract runner dispatches every contract call to a test worker (runtime-cloudflare/src/__tests__/helpers/do-contract-worker.ts), which answers two methods itself instead of calling DOStateStore (:121-130): createSession writes its own SQL row (handleCreateSession, :193-242: status 'running', parent_session_id NULL, mode ignored) and deleteSession wipes the whole object (handleDeleteSession, :249-251; the real DOStateStore.deleteSession always throws framework_not_supported). So these contract cases are green on the DO without touching the store (core testing/session-lifecycle.ts): "should create a session and verify it exists" (:66), "a loadState racing createSession sees no session or the whole initial session" (:72), "should create session with custom options and preserve them" (:117), "should initialize session with correct defaults" (:139), "should reject duplicate session creation" (:153), "should delete session and all related data" (:159), "should delete session idempotently" (:171) and "should clean up checkpoints on delete" (:175). Every other DO contract case also seeds its session through the override, so none of them starts from a row the real DOStateStore.createSession (do-state-store.ts:3315-3344) wrote. To close: route both methods through the real store, and mark the delete cases not applicable on the DO explicitly (a contract skip with its reason, like skip.deleteSession), so no case is green by substitution; check whether anything else in the worker (clear) stands in for a store method.
FU-STORE-CORRUPT-REF-TYPED-ERROR: a corrupt persisted sub-session ref makes its tree undeletable — open
Owner: the store maintainers. Surfaced by: the W1 MR3 store review (RM-64). Pre-existing strictness.
Every store now parses a stored ref's location (and Redis the whole ref) strictly. A corrupt persisted ref throws a raw ZodError / SyntaxError from the cascade's walk before anything is deleted, so the whole tree cannot be deleted until the row is repaired by hand. To close: wrap the parse failure in a typed state_* corruption error (as CorruptMessageRowError does for messages) that names the session and ref, and decide whether the cascade should skip the bad ref and delete the rest.
Two more cases belong here. Redis's getSubSessionRefs silently drops an id that is in the session's ref set but has no ref hash (a lost or expired hash), so the cascade loses that ref's location marker; it should be reported as corruption, not read as absence. And the pre-existing empty catches around getSubSessionRefs in runtime-js run-loop.ts now also swallow the strict-location failure, each falling back to a wrong default; once the typed error exists they should rethrow it and keep their fallback only for the case each was written for. (The remote call's seed read in core dispatch, runtime-js remote-subagent-executor.ts and DBOS execute-remote-subagent.ts no longer catches: it fails the call through core readRemoteSubSessionRef.)
FU-STALE-HANDLE-CANRESUME-AFTER-DELETE: a handle kept across deleteSession answers canResume() from captured state — open (LOW; runtime-js reconnect handle done in W1 MR3)
Surfaced by: the W1 MR3 §8 survey (a handle kept across a delete; spec 2026-10-09-cf-w1-mr3-delete-session-cascade-design.md §8, out of scope there).
After deleteSession removed a handle's session, result() reports the run's own outcome or a typed state_session_not_found (§8), but canResume() on these handles still answers from the state captured when the handle was built, as if the session still existed:
- runtime-js active handle,
runtime-js/src/execution/handle-factory.ts:255-262: reads no store at all; answers from the capturedstate.status. - runtime-js reconnect handle,
handle-factory.ts:427-439: re-reads the session but falls back to the captured status when it is missing. - Temporal reconnect handle,
runtime-temporal/src/executor.ts:1431-1436: the same fallback. - CF Workflows reconnect handle,
runtime-cloudflare/src/executor.ts:3537:(await this.loadAgentState(sessionId)) ?? state.
So a deleted session can answer canResume: true (or "already completed" / "has failed"), and the resume() it invites then fails on the missing session. Fix direction: re-read the session and, when it is missing, answer { canResume: false, reason: 'State not found' } as the DBOS handles' canResume() already do; prove it with a unit case per handle (execute, deleteSession, then canResume()).
Done in W1 MR3 (D2): the runtime-js reconnect handle. Its resume() also started the loop and then threw a TypeError on the null fresh state; now canResume() answers from a fresh read only and resume() reads once before starting anything and throws core sessionNotFoundError. Red first in js-agent-executor-resume.test.ts; integ on memory, Redis and Postgres in e2e cross-runtime-delete-session-cascade.integ.test.ts. Still open: the runtime-js ACTIVE handle (reads no store), Temporal's and CF Workflows' reconnect handles. (Their missing-session refusals: CF Workflows' are typed state_session_not_found since W1 MR3's F4 round; Temporal's handle and resume reads still throw a plain Error('State not found for sessionId: …'), runtime-temporal/src/executor.ts, to type with the rest of this FU.)
FU-RESUME-AT-CAP-JS-CFW-DIVERGENCE: a resume at or past the step cap runs one step on JS, none on CFW — open
Owner: CF-D1 MR 2 / the reliability orchestrator. Surfaced by: the MR 1c final review (C5). Pre-existing.
A run that suspends on its cap step (for example a client tool called on step maxSteps) records the cap as the resume's entryStepCount. On runtime-js (and the CF DO, which runs its executor) the resume runs ONE more step: the step iterator decides stop / continue only after a step (resolveStopDecision(stepResult, stepCount, …)), so the model is called once and the run is cut off after it. On CF Workflows the loop guard stepCount < maxSteps (workflow.ts) is false at entry, so the resume calls the model zero times and fails max_steps with no step commit. Both end failed with framework_max_steps_exhausted, and retry() continues either with the full budget (C5), but the number of model calls a resume at the cap gets differs. Seen in runtime-js/src/__tests__/stop-conditions.test.ts (the resumed run's extra step) and runtime-cloudflare/src/__tests__/cfw-stop-conditions.wf-noiso.test.ts ("no step boundary": zero model calls). To close: pick one rule (a resume is a continuation of the same turn's budget, so most likely "no step past the cap", or "a resume always gets at least one step") and apply it on every runtime, Temporal and DBOS included, with a conformance cell.
FU-WITH-MESSAGE-RESUME-STEP-BUDGET: does a with_message resume start a fresh maxSteps budget? — open (user decision)
Owner: user / reliability orchestrator decision. Surfaced by: CP-338's fix trace (2026-11-19). Related: FU-RESUME-AT-CAP-JS-CFW-DIVERGENCE, FU-MAX-STEPS-COMPANION-CONTINUATION, CP-338.
Since #177 a with_message resume is a new turn (its own turnId, onAgentStart owed), but on runtime-js it keeps the step count of the run it resumes: entryStepCount: rewindTarget ? rewindTarget.stepCount : state.stepCount, "a resume continues its turn" (runtime-js/src/js-agent-executor.ts:8568, origin/main 4f0e1d1807). The CF DO runs that executor, so it does the same, and CP-338's fix keeps it for a runtime-js companion re-consult. A message sent to a session interrupted partway through a turn therefore gets only what is left of that turn's maxSteps. CLAUDE.md says "maxSteps is per-turn — a continued turn resets stepCount → 0 on all runtimes" (written for a completed companion's continued turn, CP-155).
The runtimes already disagree on the same re-consult of an interrupted companion (origin/main 4f0e1d1807):
- CF Workflows runs it as its turn plan, which restarts the count: "
maxStepsis per-turn: a new turn restarts the step count",stepCount: 0andentryStepCount: 0(runtime-cloudflare/src/steps.ts:1912-1918,:1944-1948); - runtime-js (CP-338's fix included) and the CF DO keep it, as their
with_messagedoes; - Temporal keeps it too: its interrupted-companion resume (
companionResume,workflow-dispatch.ts:1038) restarts the count only for a completed session or a companion continuation (activities.ts:11977-11980). - DBOS was not traced (it does not resume an interrupted companion from
sendMessage).
- CF Workflows runs it as its turn plan, which restarts the count: "
The question (a cross-runtime default): should a
with_messageresume start a new step budget, as a new turn, or keep the resumed turn's budget, as a continuation?To close: the user rules; then apply the rule on every runtime (the companion re-consult paths included) with a conformance cell, and state it in the docs where
maxStepsis called per-turn.
FU-MAX-STEPS-COMPANION-CONTINUATION: should a max_steps-failed companion be continuable like a completed one? — open
Owner: user / reliability orchestrator decision. Surfaced by: the MR 1c final review.
MR 1c (U1) made a maxSteps cut-off of an agent without outputSchema end failed. For a persistent companion that means a re-consult no longer continues it: companion__sendMessage throws not active (status: failed), and companion__spawnAgent re-spawns it fresh, losing its memory (core companion-tool-dispatch.ts continues only a completed child; the CF DO's persistent-companion-tools.ts mirrors it). Before MR 1c the companion completed and was continuable. The behaviour is documented (upgrade guide §1, the sub-agents critic-loop section, the JS / DO / CFW runtime pages: use stopWhen or a larger maxSteps). The open question: should a companion whose run record says completionReason: 'max_steps' be continued (a new turn with a fresh budget, as retry() does for a root run) rather than re-spawned? If yes, change both continuation sites (core dispatcher and the DO's tools) and every runtime's continuation primitive, with a cross-runtime e2e case.
FU-RUNSTART-FINISH-COMPLETION-REASON: a run start that ends a stranded max_steps run writes no completionReason — open
Owner: CF-D1 MR 2 / the reliability orchestrator. Surfaced by: the MR 1c final review (C5). Narrow crash window.
When a run's K5 terminal commit landed but its run-status write did not (a crash between the two), the next entry ends that stranded run inside its own run-start commit (StartRun.finishCurrent, core RunStartFinish in core/src/store/run-start.ts, R27/R35). RunStartFinish carries status and error only, so the run record gets no completionReason. finishCommittedRun derives 'max_steps' from the session's errorDetail code on recovery (!291); this path does not. So if a run failed framework_max_steps_exhausted and crashed in that window, a later retry() reads run.completionReason as unset and C5 does not apply: with a step boundary the retry needs a message and restores the boundary's step count; with no step boundary a resume's retry throws validation_error.
Amended (batch 21, owner, from WE-2's trace; the orchestrator ruled WE-2's gap a duplicate of this entry; lines at origin/main f4f4028489): the path loses every completion reason, not only max_steps. RunStartFinish is { runId, status, error? } (core/src/store/run-start.ts:111-116), built at :329-335 from the finish facts (the session's status and error and the latest checkpoint), and StartRun.finishCurrent is { runId, status } (core/src/store/message-commit.ts:190). Every store's finish write sets only the status, the lease, completedAt, endSequence and error: memory (store-memory/src/in-memory-state.ts:1003-1011), Postgres (store-postgres/src/state/postgres-state-store.ts:2345-2358), D1 (store-cloudflare/src/d1-state.ts:2520-2538), DO-SQLite (runtime-cloudflare/src/do/do-state-store.ts:1987-1993) and Redis (store-redis/src/redis-state.ts:2943-2945, store-redis/src/commit-script.ts:1012-1015). So a stranded run's stop_guard_exhausted / stop_guard_error reason, which a reconnect and the DO's /status read, is lost too, and so is the stop guard's report that RM-132's fix records on the run (RunMetadata.stopGuard). Recovery's finishCommittedRun (core/src/run/finish-committed-run.ts:121-125) writes max_steps only; with RM-132's fix it also writes the report and its reason from a forced phase that ended the run. The C5 consequence above reads run.completionReason === 'max_steps' (core/src/store/retry-plan.ts:772).
Done when: RunStartFinish, and the facts the store re-reads inside its commit, carry what finishCommittedRun derives (max_steps from the session's errorDetail code, and a forced phase's stop guard report and reason); every store writes them in the same commit; and a contract case pins a run settled by finishCurrent after a max_steps cut-off keeping its reason, with a stop-guard case beside it.
Related, low risk: a max_steps retry (C5) of a resumed run with no trigger and no drain continues with input: [], so the retried run's first model call sees exactly the kept prefix. CF Workflows (run-entry.ts, buildTrailingHealMessages over the kept prefix) and DBOS heal trailing unpaired tool calls in that prefix before the call; runtime-js and Temporal do not heal it on this path. The kept prefix ends at a step or drain commit, so it is normally paired (RM-29); only a prefix that somehow ends in an unpaired call would reach the model unhealed. To close: run the C5 continuation heal on the empty-input retry path on runtime-js and Temporal too, with a unit case each.
T1 refresh path: stream readers lose a chunk after a resume cleanup (T1-A)
FU-T1A-REDIS-REBUILD-DROPS-LIVE-CHUNK: a Redis resumable reader drops a live chunk during its truncation rebuild — done
Closed by: this MR (CI-health-3 T1-A). Found by CP-96 on !295's CI.
- Symptom: e2e
refresh-c1js-redis › "refresh while the client tool is pending" rendered"Thanks, it is"for the snapshot's"Thanks, teal it is"(pipeline 2911894663, job 16927336679): client B lost one continuationtext_delta. It is about 1 in 10 to 20 under load on main. - Cause (
RedisStreamManager.createResumableReader):- The resume of runtime-js (and the CF DO, which runs it), DBOS and CF Workflows calls a run-scoped
cleanupToStep(Temporal never calls it), which sets__truncated_at_stepeven when it removes nothing (G4). The continuation run then streams with the marker set. - With the marker set, each drained
next()rebuilds its buffer fromLRANGEon the main connection. - It then dropped every pending pub/sub chunk missing from that snapshot as "cleaned up". A chunk written just after the
LRANGEreachespendingChunksover the subscriber connection, so it was dropped. The next chunk went straight to the waiting consumer, and the dropped one was never re-read.
- The resume of runtime-js (and the CF DO, which runs it), DBOS and CF Workflows calls a run-scoped
- Fix: the rebuild reads the writer's counter and the chunk list in one
MULTI. A pending chunk above that counter was written after the snapshot, so it is kept. - Proof: new contract case
G4.rebuild-keeps-live-chunks, with an optionalholdNextRebuildReadforce.- Redis holds the rebuild read's reply until a newer chunk reaches the subscriber.
- On main it fails with the exact signature:
expected 'c5' to be 'c4'. - It passes on every manager after the fix.
- Audit of the other managers:
- Memory and the in-process DO read one authoritative source with no separate pending queue, so they are not affected.
- runtime-cloudflare's
DOStreamManagerClient(the reader behindcreateCloudflareChatHandler, e.g. the opennext example) reads the DO agent's own/sse, served by the in-process DO manager. It has no truncation handling and no client-side prune, so it is not affected either. - The binding-side DO reader had a different loss in the same family: FU-T1A-BINDING-PRUNE-IGNORES-FLOOR.
FU-T1A-BINDING-PRUNE-IGNORES-FLOOR: a binding-side DO reader drops buffered chunks a floored cleanup kept — done
Closed by: this MR (CI-health-3 T1-A audit).
- Cause: on a
truncatedevent,DurableObjectStreamManager's readers pruned every unread buffered chunk withstep > truncatedAtStep. The DO'scleanupToSteponly removes such chunks above itsminSequencefloor. The resume of runtime-js (and the CF DO, which runs it), DBOS and CF Workflows passes the previous run'sstartSequenceas that floor, and their interrupt the interrupted run's when its record reads. Unread chunks of earlier turns with a higher per-turn step number were dropped, although the DO kept them. - Fix:
StreamTruncatedEventcarriesminSequence. The DO broadcasts it on WebSocket and SSE, and the reader prunes only chunks above it. - Proof: new contract case
G4.floor-kept-backlog-survives. On main it fails asexpected [ 't2s1' ] to deeply equal [ 't1s2', 't1s3', 't2s1' ], and it passes on every manager after the fix.
FU-REDIS-REBUILD-LTRIM-PENDING: a Redis rebuild silently drops a pending chunk that maxChunks trimmed — open
Owner: unassigned (reliability orchestrator to route). Pre-existing; out of T1-A's scope.
- The G4 rebuild drops a pending chunk with a sequence at or below the snapshot's counter that the snapshot does not hold, as "cleaned up". A chunk that
WRITE_CHUNK_SCRIPT'sLTRIMremoved (retention,maxChunks) meets the same test, so a slow reader loses it silently instead of getting the retention-floor signal (TRUNCATED, the floor infirstSequence). - Fix direction: in the rebuild's
MULTI, also readfirstSequence. A pending chunk at or below it was trimmed, not cleaned up, and should surface the retention gap the way the attach path does.
FU-G4-NOOP-CLEANUP-MARKER: a resume cleanup that removes nothing still sets the truncation marker — open
Owner: unassigned (G4 contract; reliability orchestrator to route).
- Every resume's cleanup sets the marker, including no-op ones. On Redis, each drained reader
next()for the rest of the continuation run then pays an O(n)MULTI/LRANGErebuild. The in-process DO pays aSELECT, and the DO binding broadcasts atruncatedevent. - That is a cost, not a correctness issue, after T1-A.
- Options:
- Record the marker only when the cleanup removed a chunk.
CLEANUP_TO_STEP_SCRIPTknows that atomically. - Clear the marker when the resumed run claims the stream.
- Record the marker only when the cleanup removed a chunk.
- Either one changes the G4 contract ("a no-op cleanup is still observable"), so it needs a contract decision and matching changes in every manager and in the contract suite.
T1 refresh path: a CFW SEND answered before its run claimed the stream (T1-B)
FU-T1B-CFW-ENTRY-BEFORE-CLAIM: a CFW execute() returned before its run claimed its stream — done
Closed by: this MR (CI-health-3 T1-B). Found by CP-96 on !295's CI.
- Symptom: cfw-engine
refresh-attempt-retry› "live attempt-switch re-anchor" (ai 6.0.49):no resync trigger re-anchored within 30000ms of the hold (data parts: ["data-resume-rejected"])(pipeline 2911894663, job 16927336694). A load flake on main since S5. - Cause:
CloudflareAgentExecutor.entryStartedreturned on the run row alone for a new oractivesession, right after the run-start commit. The instance claims and makes the stream live after that commit. With a claim lag of about 1 s on a loaded runner,handleChatStream'sstreamHandlegothandle.stream()→nullon all four attempts. It answered the SEND withdata-resume-rejected/stream_unavailable, and the run went on with no listener. - Fix: for a run that owns its stream, the entry has started once the stream holds a chunk stamped with this run at or after its start sequence (its
run_started; Temporal's rule), or the run leftrunning. The stream's owner andactivestatus are not enough: the binding-side DO's/inforeads a never-created stream asactive, a claim does not create it (only the entry's writer/initbeforerun_starteddoes), and a manager may implement noclaimStream. A child writing into its parent's stream (isForeignStream, R41) keeps the run-record check. The existing fail-safe still bounds the wait. - Proof:
- Unit (
runtime-cloudflarecfw-entry-stream-live.test.ts):- red on main for new and
activesessions; - a DO-like stream manager whose claim does not create the stream: red on a claim-and-
activerule; - a manager with no
claimStream: that rule waited for the fail-safe; - failed-before-claim and foreign-stream controls.
- red on main for new and
- e2e (
send-claim-lag-cfw-engine.wf.test.ts): the harness'sbeforeClaim/beforeCreateWriterhooks hold the entry at its claim, or after it and before the writer's/init, until the SEND has answered.- On main both holds failed for all three client profiles, with the CI's
data-resume-rejected. - With a claim-and-
activerule, the claim hold passed but the writer hold still failed. - With the
run_startedrule, all six pass.
- On main both holds failed for all three client profiles, with the CI's
- Unit (
FU-DBOS-ENTRY-FALLBACK-BEFORE-CLAIM: a DBOS entry wait that times out can return before the claim — open
Owner: DBOS bucket (CF-D1 MR 3). Found by the T1-B cross-runtime audit.
- On its normal path DBOS's
execute()/resume()/retry()wait for the workflow'sRUN_START_EVENT. The workflow publishes it only after the entry step committed, claimed the stream and made it live, so the handle's stream is live. - The edge: when no event arrives within one
getEventwait (RUN_START_EVENT_TIMEOUT_SEC), or the workflow ends first,awaitEntryRunStart'scommittedRun()(lifecycle/run-start-event.ts) returns a committed, stillrunningrun without checking its stream. An entry step stalled that long between its commit and its claim would handhandleChatStreama stream that is not live yet, answering the SEND withstream_unavailable. - Fix direction: CFW's T1-B rule. For a run that owns its stream,
committedRun()returns only once the stream holds a chunk stamped with the run (itsrun_started), or the run leftrunning.
FU-CFW-ENTRY-FAILSAFE-SILENT-HANDLE: the CFW entry fail-safe returns a handle for a run that never started its stream — done
Closed by: CF-D2 MR 1 (!316). At the bound, a run that committed but never started (no run_started in its own stream, or no entry state) no longer gets a handle: the executor terminates the instance; stopped → the run is settled failed (K5 → run → stream) with the error's own detail and the call throws core RunStartStalledError (transport_error); not stopped → nothing is settled (ownerStopped: false). The stall is never retryable (user ruling 2026-10-07: its input is committed; recover with executor.retry(agent, sessionId)). A committed run first gets the same bound again, from when the call first saw the commit, until its instance closes or is paused. Both the 'committed' verdict and the 'unknown' verdict's re-read re-check entryStarted first. Proven by runtime-cloudflare/src/__tests__/cfw-settle-closed-owner.test.ts, cp85-c3r4-cfw.test.ts, cp85-executor-entry-outcome.test.ts and, on the real engine, e2e/src/__tests__/cfw-terminal-settle-cfw-engine.wf.test.ts (owner stopped / not stopped). Temporal and DBOS keep the old shape: FU-TEMPORAL-ENTRY-FAILSAFE-SILENT-HANDLE.
Owner: unassigned (reliability orchestrator to route). Pre-existing, now more reachable after T1-B.
- At
entryOutcomeFailSafeMsthe executor runsabandonRunStart. A'committed'verdict returns the handle, even when the run's entry never wrote itsrun_started, for example when it stalled after its commit. - The caller then reads a stream that does not exist (
stream_unavailable), with no typed signal. - Fix direction: on
'committed'withoutrun_started, return a typed outcome (or let the handle'sstream()wait for the run), instead of a handle indistinguishable from a started one.
CF-D2 MR 1 (CF Workflows terminal truth) follow-ups
CF-D2 MR 1 (2026-10-05-cf-d2-cfw-terminal-and-cf-store-integrity-design.md §0 / §2) fixed DI-33, DI-34, the CFW entry fail-safe, RM-70 (stream info errorDetail), and applied rule M1 (a pre-commit entry failure never touches a session that already has a run) to CF Workflows and runtime-js / the CF DO. These items are what it left open.
FU-JS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED: runtime-js / the CF DO marked a session with a run failed on a pre-commit entry failure — done
Closed by: CF-D2 MR 1 (!316; the user's decision of 2026-10-06: leave the session untouched everywhere). runtime-js execute() (its first history read, C9, and its run-start plan) and retry() (its plan before the run-start) went through persistRunFailure on a state_history_incomplete read failure: the session became failed with the code, its ended stream was reactivated and failed, and onAgentFail fired, although no run had started and the session had finished earlier runs. A new persistPreCommitFailure seam (runtime-js/src/js-agent-executor.ts) writes nothing when the session has any current run (or the read fails); only a never-run session is recorded. The CF DO inherits it. Proven by runtime-js/src/__tests__/pre-commit-entry-failure-m1.test.ts (12 cases), runtime-cloudflare/src/__tests__/pre-commit-entry-failure-do.cf.test.ts (real DOStateStore) and runtime-cloudflare/src/__tests__/integration/do-multiturn-execute-continuation.integ.test.ts (real DOAB). This reverses the earlier C9 run-level rule for a session that already has a run.
FU-CFW-CHAT-READER-SETTLE-CLOSED-OWNER: a chat-only host never settles a closed CFW owner until its next send — done (W1 MR2b)
The closed-owner reconciliation (DI-34's settle, and since W1 MR2a the suspended report and the stream repair) runs in the executor readers (result(), getHandle(), canResume(), abort()) and the parent's consult. handleChatStream (ai-sdk/src/handler/handle-chat-stream.ts) and the snapshot route call none of them, so a host that only serves the chat route shows a session (and stream) active behind an externally terminated instance until the next send, whose entry supersedes the stranded run. Fix direction: settle (or report) a closed owner on the snapshot / resume read path, or have the CFW snapshot source call the executor's settle.
Done (W1 MR2b §4). The optional MinimalExecutor.reconcileSession(sessionId) is called by handleChatStream (once, after its first reads, for a live run; it re-reads on changed) and by buildSnapshot (BuildSnapshotDeps.executor). CloudflareAgentExecutor.reconcileSession runs MR2a's settleForReader, then ensures the __submit__ continuation a stored answer is still owed. A continuation that cannot start is logged and never rejects the read; exhausted generations discharge the owed status to the next entry. Tests: runtime-cloudflare/src/__tests__/executor-reconcile-session.test.ts, cfw-reader-reconcile.wf-noiso.test.ts (f)(g)(h); ai-sdk/src/__tests__/handler/reader-reconcile-hook.test.ts. runtime-js, Temporal and DBOS do not implement the hook yet; they will with their closed-owner adoption (the three *-RECONCILE-CLOSED-OWNER-ADOPT items).
FU-TEMPORAL-ENTRY-FAILSAFE-SILENT-HANDLE: Temporal and DBOS fail-safes hand out a committed run that never started — open (MED, owner: the next Temporal bucket, assigned by the reliability orchestrator; the DBOS half goes with it)
The CFW half is fixed (FU-CFW-ENTRY-FAILSAFE-SILENT-HANDLE). At their fail-safe bounds:
- Temporal
runtime-temporal/src/executor.tsawaitEntryOutcome: anabandonRunStartverdict of'committed'returns (if (verdict === 'committed') return;) without checking that the run started its stream. - DBOS
runtime-dbos/src/lifecycle/run-start-event.tsfailSafe:'committed'returns the run id (from the late event,committedRun(), oropts.runId) without a started check, and never cancels the workflow (I-2).
Either way the caller gets a handle whose stream may not exist yet (stream_unavailable on a SEND). Fix direction: CFW's rule — re-check the started condition; stop the owner; settle the run failed and throw core RunStartStalledError when it stopped, else throw it non-retryable.
FU-DBOS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED: DBOS records a finished session failed on a pre-commit entry failure — open (MED, owner: DBOS bucket)
DBOS is the one runtime without rule M1. persistPreCommitFailure (runtime-dbos/src/steps/run-commit.ts, the export async function persistPreCommitFailure) returns early only for validation_error or a LIVE current run, so a completed / failed / interrupted session is overwritten failed with the entry's errorDetail; and awaitExecutorEntry (runtime-dbos/src/lifecycle/run-start-event.ts) then calls restoreStreamAfterFailedEntry. Fix direction: return when the session has ANY current run (Temporal / CFW / runtime-js), and for a never-run session write only the session through core failSessionWithoutRun (currentRunId: null, no stream manager; user ruling R43, as CF Workflows does), dropping the restoreStreamAfterFailedEntry stream write.
FU-TEMPORAL-RECONNECT-ERRORDETAIL-DROPPED: a Temporal reconnect handle reports a failed run's errorDetail without its code — open (LOW, owner: the parked Temporal bucket)
Surfaced by: WA MR2 R2d triage of conformance discipline.abort-result-detail (temporal-*, reconnect-detail-matches-persisted, CP-22). This is already present on main: the file is unchanged since main. Tracks: catalogue record DI-98 (2026-11-17: the rows moved from CP-22, whose terminal claim is fixed).
- Signature: after an abort, the persisted
SessionState.errorDetailcarriescode: 'framework_cancelled', but the reconnect handle'sresult().errorDetailcarries only{ message }. - Cause:
loadAgentStateFromSession(packages/runtime-temporal/src/state-helpers.ts:37) does not copyerrorDetailfrom the session, so the reconnect handle derives a message-only detail (the DI-33 class). - Fix direction: carry
errorDetailthroughloadAgentStateFromSession, and let the handle report the persisted detail (coreresolveErrorDetail). R2d showed that this one field turns the cell into an unexpected pass.
FU-TEMPORAL-NEVER-RUN-PRE-COMMIT-STREAM-WRITE: Temporal's never-run pre-commit failure still writes the stream (R43) — open (LOW, owner: Temporal bucket)
User ruling R43: a session that fails before its first run-start commit writes only the session; streams are written only by their runs. CF Workflows does this through core failSessionWithoutRun (W1 MR2a). Temporal's persistPreCommitFailure (runtime-temporal/src/activities.ts) saves the session unfenced and then fails an active / paused stream (fenced on the never-committed entry run id, so an ownerless stream still fails), and executor.ts (ruling P18) fails the stream the call made live. A failure that lands after the next turn's winner made the stream live can leave the stream failed or an error chunk after its cursor (the race R43 removes on CFW, proven by cfw-terminal-truth.wf-noiso.test.ts "R43"). Worse, that save is unfenced: a winner that commits between its getCurrentRun read and its saveState gets its session overwritten to failed. Fix direction: replace the never-run write with failSessionWithoutRun({ currentRunId: null }) (idle-fenced, so a winner's session is never overwritten; no stream manager) and drop the executor's P18 stream fail. DBOS is covered by FU-DBOS-PRE-COMMIT-ENTRY-FAILURE-MARKS-SESSION-FAILED.
FU-TEMPORAL-COMMITSTEP-LANDED-AWARE: Temporal's step commit cannot tell its own landed write from a lost race — open (parked; owner: the parked Temporal bucket)
Surfaced by the guarded-commit outcome MR (spec 2026-10-16-guarded-commit-outcome-design.md §6.2 B2). After CommitOutcomeUnknownError (the store could not read back whether the step commit applied), a Temporal activity retry would repeat commitStep (activities.ts) with the same expectedVersion and get StaleStateError for a step that landed. Ruling B2 (i): toActivityFailure hands the unknown outcome to the workflow non-retryable, so the run fails typed (truthful, but it can fail a run whose step landed). Fix: make commitStep landed-aware, as CF Workflows' findLandedStepCommit is (look for the step's own step checkpoint before re-committing), then map the unknown outcome to retryable.
FU-JS-PRE-COMMIT-NEVER-RUN-FENCE-AND-HOOK: runtime-js's never-run pre-commit failure is unfenced and fires onAgentFail — done
Closed by: WA MR1 (the CF DO drive pipeline, spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, branch reliability-cf-wa-do-drive). runtime-js persistPreCommitFailure uses core failSessionWithoutRun({ currentRunId: null }): an idle-fenced session write only, with no stream write and no hook (rulings R40 / R43). Proven by runtime-js/src/__tests__/pre-commit-never-run-session.test.ts (including both hold points of the cross-entry race).
For a session that never had a run, persistPreCommitFailure delegates to persistRunFailure:
- its session write is a status CAS, not idle-fenced (CF Workflows uses
fenceIdle: { runId: null }, so a run that commits meanwhile wins); - its
failStreamis unfenced; - it fires
onAgentFailalthough no start hook fired, against R49 (Temporal / CFW fire none).
Fix direction: idle-fence the write, fence or skip the stream fail, and drop the hook for this path.
FU-DOAB-HANDLERETRY-REFUSAL-STATUS-WRITE: DOAB /retry refusals write failed before any commit — done
Closed by: WA MR1 (the CF DO drive pipeline, spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, branch reliability-cf-wa-do-drive). handleRetry is deleted: /retry is a request drive of runDrive, and a request drive that fails before its run-start commit writes nothing (I-D6); its refusals answer typed (wire-answer.ts). Proven by do-drive-pipeline.integ.test.ts retry-refusal-writes-nothing (the store write log, version, updatedAt and stream status are unchanged).
DurableObjectAgentBase.handleRetry (runtime-cloudflare/src/do/durable-object-agent-base.ts, the agent-resolution failure and the "persistentAgents without subAgentNamespace" refusal) calls stateStore.updateStatus(sessionId, 'failed') before returning 400 / 500. The session's precheck saw it failed, so the status is idempotent, but it is still a pre-commit write (it bumps updated_at / version), against M1's "untouched". Fix direction: refuse without writing.
FU-JS-CP10-RESUSPEND-REFUSE-IF-STOPPED: CP-10's re-suspension save is not stop-guarded — done (WA's CP-173 / CP-174 MR)
Closed by: WA's CP-173 / CP-174 MR (branch reliability-cf-wa-cp173-174-expiry-seam, design docs/superpowers/specs/2026-10-20-cp173-174-deadline-expiry-seam-design.md §7.9). reSuspendOnAwaitedChildren saves paused with { fenceRunId, refuseIfStopped: true }. A stop recorded after the loop's stop poll and before the save's atomic step refuses the save (CommitStoppedError, nothing written, no onAgentSuspended), and the loop ends the run by that stop (stopBoundaryOutcome, then the post-loop stop resolution). Proven by subagent-cascade-resume-stops.test.ts "FU-JS-CP10-RESUSPEND-REFUSE-IF-STOPPED (CP-173)": red at main (the paused save landed over the recorded stop and onAgentSuspended fired), then green. The window this entry named after the save (a stop recorded between the post-save poll and the run's suspended_awaiting_children status write) is every suspension's post-commit window, read by the next continuation (Ruling 41); it is not changed.
Owner: WA (CP-173 / CP-174 add refuseIfStopped to saveState in all five stores). Surfaced by: !421's Greptile note 7ef245b5.
A runtime-js parent whose resumed awaited child re-suspends saves itself paused (reSuspendOnAwaitedChildren, saveState fenced on the run only). It polls the run's stop and its signal before that save and again after it, and ends by a stop it finds (its K5 supersedes the paused save). A stop recorded between that last poll and the run's suspended_awaiting_children status write is still read only by the next continuation (Ruling 41), as for any suspension's post-commit window. To close: the save passes refuseIfStopped once CP-173 lands, and the refusal ends the run by the stop.
FU-JS-NEW-TURN-STALE-FIELDS: runtime-js keeps stale per-turn fields on a new turn — open (LOW, owner: runtime-js bucket)
Found by the DI-36 parity audit (CFW resets these on a new turn; runtime-js / the CF DO do not):
- the session row keeps a stale
forcedCompletionandfailureReason(toSessionStatestarts from{ ...existing }and never writes them; the in-memory run ignores the staleforcedCompletion, but readers of the row see it); - a continuation overwrites
parentSessionId,rootSessionId,userId,tagsandmetadatawith THIS call's options, so a continuation that omits them drops the session's context (CFW keeps the session's).
Fix direction: clear the per-turn fields in the new-turn entry state; keep the session context unless the call passes it.
FU-CFW-STREAM-ACTIVE-AFTER-ENDED-RUN: a stream left active behind an already-ended CFW run is repaired only by the next entry — done
Closed by: W1 MR2a (!334, merged as 137b07d57a): the CF Workflows readers plan through core planClosedOwnerReconciliation, whose repair_stream rung ends or fails an active or paused own stream of an ended run from its committed outcome once the owner is proven closed (reconcileClosedOwnerRun, fenced on the run); the parent's consult and the owner's own R27 path repair it the same way. Tests: cfw-settle-closed-owner.test.ts (completed and failed runs, an active and a paused stream), cfw-reader-reconcile.wf-noiso.test.ts (c) (real engine), cfw-consult-run-died.test.ts (the stream ended on the second consult).
DI-34's settle acts on a running run. A stream left active behind a run that already ENDED (its terminal stream write lost, owner closed) is not repaired by any reader; since CF-D2 MR 1 a failed pre-commit entry no longer touches it either. Only the next entry's claim repairs it. Fix direction: a reader settle for "run ended, stream active, owner closed" (end / fail the stream from the committed outcome, fenced on the run).
FU-FAIL-STARTED-RUN-LEAVES-STREAM-OPEN: runtime-js's failStartedRun ends the run but not its stream or its session — open (owner: WA)
Filed by: MA-31 (Task 3 review; the approve drain's retryable state-commit fault now reaches it). An entry that fails after its run-start commit but before its loop runs (an orphan drain that throws; a retryable fault on the approve drain's own state commit) ends the run with JSAgentExecutor.failStartedRun (runtime-js/src/js-agent-executor.ts), which only writes the run's status (failed). It never ends or fails the run's own stream, which the run-start made live, so a client tailing that stream waits until its live-wait timeout. The CF DO inherits it. Wanted: fail the run's own stream (fenced on the run) with the entry's typed detail, with a test that a tailing reader sees the failure.
Sighting, the session half (WE-2, 2026-11-05, b3cbbd3dd2, memory store, an injected throw): a throw at the reconcile's settling append, after the entry's run-start commit, ends the run failed and leaves the session active. retry() is then refused (agent status running), and a further continue succeeds. Not a finding: it was reached only by an injected throw, and the session is not trapped. A real route to it, or a session no entry can leave, makes it one. Wanted, with the stream half: the started run's failure also reaches the session, fenced on the run.
The session half is now CP-317 (owner, batch 20): a real store error in an entry's post-run-start, pre-loop writes (core CommitNotAppliedError / CommitOutcomeUnknownError after a transport failure) takes the same failStartedRun path, so it is no longer reached only by a test fault (W1's SDK-6 slot sighting, a drain-first with_message resume whose retry() was then refused). The stream half stays here.
Restore path (CP-333's MR): resume() over an interrupted or paused source now restores the session's status and leaves the run's stream live. Before the restore path fails that stream, a trace must show whether the continue that follows reuses it.
FU-CFW-SUSPENDED-RUN-TERMINATED-EXTERNALLY: a CF Workflows handle reports failed / "Unknown error" for a suspended run whose instance closed without its suspended output — done
Closed by: W1 MR2a (!334, merged as 137b07d57a): a run whose suspension is committed (core isSuspensionCommitted({ session })) and whose instance is proven closed is planned complete_suspension: the run reads suspended_* (written when it still read running), its own active stream is paused, and every reader's result() answers { status: 'suspended_*', runId, suspended } from the session; canResume() is true. A run killed before its suspension commit is fail_uncommitted (settled failed with closedOwnerFailure). A reader whose owner probe failed still reports its own run's committed suspension without writing. The parent's consult reports such a child suspended (never a failure). Tests: cfw-settle-closed-owner.test.ts, cfw-consult-run-died.test.ts, cfw-reader-reconcile.wf-noiso.test.ts (a), (b), (d) (real engine), cfw-terminal-settle-cfw-engine.wf.test.ts (e2e).
Surfaced by: CF-D2 MR 1, Task 11 review (a deferred minor; never filed by !316). Verified: in source at 2e626e7f6a; not reproduced.
A CF Workflows step suspension writes its suspended_* status before its suspension frames, commit and stream pause (a cascade suspension, e.g. a sub-agent wait, commits first: the fenced paused save lands, THEN the run status), and its instance then stays live for the parent notification and the onAgentSuspended hook step before it returns the suspended_* output (runtime-cloudflare/src/workflow.ts suspendRun, :1003-1050). If the instance is terminated from outside the product (or errors) anywhere after that status write and before it returns:
- DI-34's settle does nothing:
settleClosedOwneracts only on arunningrun (runtime-cloudflare/src/executor.ts:1334); - the active handle's
mapWorkflowStatusToAgentResult(:1666) finds no suspended output, andrecorded(:1708) is the run'ssuspended_*status, which no branch adopts, soresult()answersfailedwith the engine's message or "Unknown error" (:1754); the reconnect handle (getHandle(), :2938-2975) does the same; - the stream is left
activewhen the kill lands before the pause.
When the suspension commit had landed, the session is in fact resumable: the handle reports a failure for a run that is waiting on its client. When it had not (killed between the status write and the commit), the run is suspended_* with no committed pending calls, and nothing settles it. Fix direction: the closed-owner reader settle covers a suspended_* run too. Suspension committed (the latest checkpoint is the run's suspension commit) → report the suspended status, and pause the stream if it is still active. Not committed → settle it failed as DI-34 does (closedOwnerFailure). Same reader-settle family as FU-CFW-STREAM-ACTIVE-AFTER-ENDED-RUN.
Correction (W1 MR2a). The suspension is committed when the SESSION is paused and awaits calls or children (core isSuspensionCommitted({ session })), not when the latest checkpoint is the run's suspension commit: the CFW cascade is an append commit followed by a fenced paused save, so its cascade order is commit (fenced paused save), then status. The fix lands in Task 8 of this MR through core planClosedOwnerReconciliation / reconcileClosedOwnerRun; Tasks 6 and 7 landed the planner and the executor.
FU-CFW-ENGINE-LANE-MINIFLARE4-NO-TERMINATE: the e2e cfw-engine lane cannot terminate an instance — open (LOW, owner: conformance owner / plan 0c-cfw)
test:cfw-engine runs @cloudflare/vitest-pool-workers 0.9.14 on Miniflare 4, whose WorkflowHandle.terminate() throws "Not implemented yet". CF-D2's real-engine file (e2e/src/__tests__/cfw-terminal-settle-cfw-engine.wf.test.ts) therefore emulates a kill (cfwEngineInjection.killInstance in e2e/src/cfw-engine/worker.ts; the engine reports errored) and an emulated terminate on the executor's binding. DI-34's terminated → framework_cancelled row is unit-proven only, and the killed bodies log uncaught "Stream is failed/ended" noise. The real-terminate proof belongs to the Miniflare 5 cfw-workflows-d1 lane (plan 0c-cfw); DI-34 flips there. Also: on Miniflare 4 abort() cannot stop an instance, so result() waits for it to end on its own (local dev only; documented).
FU-CFW-FAILSAFE-UNKNOWN-VERDICT-TERMINATE-ORDER: a failed abandon terminates the instance on an absent or unreadable run (no tombstone), so a run that commits just before the terminate is killed — done (the kill is owned; W1 MR3)
In awaitEntryRunStart (runtime-cloudflare/src/executor.ts), when abandonRunStart throws (verdict 'unknown') the call re-reads the run (readRunWithRetries, 3 attempts) and goes to the start grace when it finds it committed. Only when that read finds no run, or keeps failing, does it terminate the instance (with no tombstone, the only fence against a late commit), then re-read the run (readRunWithRetries again) and check its start once (confirmEntryStart). The window left: a run that commits between that pre-terminate read and the terminate is killed by this call. If it had started, the call returns a normal handle (the caller gets no entry-time signal) and the run settles typed framework_cancelled through the next reader (DI-34) instead of running on, with a message that blames a terminate from outside the product although this executor terminated it; if it had not, the call throws RunStartStalledError and settles it as its settle says (usually failed); if its start cannot be read, the same stall carries startUnknown: true, says its start could not be confirmed, and tells the caller that retry() needs options.message if the run committed a step or its entry carried no user message of its own. Narrowed (W1 MR2a): abandonWithRetries now makes START_CHECK_ATTEMPTS abandon attempts (START_CHECK_BACKOFF_MS * attempt backoff) before giving up, so a transient store error lands the tombstone instead of reaching the terminate. The residual is a store that is down for the whole budget: the unfenced terminate path above still applies.
Resolution (W1 MR3 D3.4, Option A: own the kill). The terminate stays (with no tombstone it is the only fence against a late commit), but the executor never hands back a run it killed: a run that started before a terminate that landed gets RunStartStalledError with start: 'started' (new core RunStartStalledStart), settled through failStalledEntry's closed-owner settle with that message, so the run's persisted detail names this fail-safe instead of a terminate from outside the product. A terminate that failed still returns the handle (the run goes on). Red first in cfw-settle-closed-owner.test.ts (a handle came back).
FU-CFW-FAILSAFE-TERMINATE-JUST-CLOSED: a fail-safe terminate that fails because the instance just closed is reported "not stopped" — done (W1 MR2a)
failStalledEntry (and the abandon path) in runtime-cloudflare/src/executor.ts treat any instance.terminate() throw as "the owner could not be stopped": ownerStopped: false, nothing settled, RunStartStalledError / RunStartAbandonedError non-retryable. An instance that closed on its own between the fail-safe's last status read and the terminate also makes the terminate throw, although nothing runs on. Fix direction: after a failed terminate, probe the instance status (inspectCfwOwner); a closed instance counts as stopped (settle as usual).
Done: terminateOrConfirmClosed replaces both raw terminate sites; a failed terminate followed by an inspectCfwOwner answer of closed counts as stopped, any other answer or a failed probe is "not stopped" (logged).
FU-CFW-PRECOMMIT-RUN-READ-THROW: CF Workflows' pre-commit failure swallows a failed current-run read on a fresh session — done
Closed by: W1 MR2a (!334, merged as 137b07d57a): persistPreCommitFailure lets a failed current-run read, and any save failure other than RunSupersededError, reject, so its error-pre-commit step retries it (PRE_COMMIT_FAILURE_STEP_CONFIG: 3 retries, about 47 s at most; the error-run-exists step before it uses the same config, about 94 s for both: inside the 120 s entry fail-safe when the entry's own steps fail fast, though a hang in check-existing / run-start can outlast it, and the caller then gets the typed abandon / outcome-unknown error). Once those retries are exhausted the workflow logs the failed write at error level, notifies the parent errored once and returns the entry's own classified failure, so the caller's preCommitFailure is never replaced by the write's store error. Residual: a store that stays down for the whole retry budget still leaves a never-run session active with no run (only the caller and the parent learn of the failure); the never-run write is core failSessionWithoutRun (user ruling R43: it writes only the session; no stream is created, read, failed or written). Tests: i5-pre-commit-failure-fence.test.ts, cp85-c3r4-cfw.test.ts (retries exhausted, with a parent), cfw-terminal-truth.wf-noiso.test.ts (real engine: a transient and an exhausted write).
AgentSteps.persistPreCommitFailure (runtime-cloudflare/src/steps.ts) catches a getCurrentRun throw, logs cfw.entry.pre_commit_failure_run_read_failed and returns, never throwing, so its step.do is never retried. For a fresh session (no run ever) the session stays active with no run and no recorded failure. Temporal's persistPreCommitFailure (runtime-temporal/src/activities.ts) lets the read throw, so the activity retries it. Fix direction: rethrow the read failure (let step.do retry it); keep "write nothing" only for a session that has a run.
FU-CFW-OWNERLESS-RUN-SETTLE-MISMATCH: the executor settles an ownerless running run, the workflow's consult probe does not — done
Closed by: W1 MR2a (!334, merged as 137b07d57a): the executor readers and the parent's consult (consultClosedChildRun, which replaces consultRunDied) share one owner probe, cfwOwnerClosure (runtime-cloudflare/src/run-entry.ts): a run with no ownerToken is closed (kind gone) for both, and both then plan and execute the same core reconciliation. Test: cfw-consult-run-died.test.ts (the ownerless case).
FU-DO-RESUME-PRE-COMMIT-SILENT: the DO's /resume still answers resumed for an entry that failed before its run-start commit — done
Closed by: WA MR1 (the CF DO drive pipeline, spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, branch reliability-cf-wa-do-drive). /resume is a request drive: it awaits the entry-outcome gate like /start and /retry, and a pre-commit failure answers typed and writes nothing. Proven by do-drive-pipeline.integ.test.ts resume-pre-commit-typed.
B1 (CF-D2 MR 1, user ruling S13) made the DO's /start and /retry wait for their entry's outcome and answer a pre-commit failure typed ({ error, code }, core typedErrorResponse). /resume (DurableObjectAgentBase.handleResume, runtime-cloudflare/src/do/durable-object-agent-base.ts) still detaches its executor call and answers { status: 'resumed' } at once, so a resume whose entry fails before its commit (e.g. a short history read, state_history_incomplete) looks started, and the caller reads the stream's previous end. The ruling covered only /start and /retry. Fix direction: the same entryOutcomeGate / entryFailureResponse on /resume.
FU-CFW-CLOSED-NO-RUN-NO-PRE-COMMIT-FAILURE: a CF Workflows instance that closed errored / terminated with no run reports no preCommitFailure — done
Closed by: W1 MR2a (!334, merged as 137b07d57a): the entry wait's closed-instance / no-run / no-failed-output branch returns closedEntryFailure(...) for the engine status, mapped by the one cfwClosedKindFromStatus() that inspectCfwOwner also uses (errored → the engine's message, no code; terminated → framework_cancelled; complete with an output that is not the workflow's failure → a message naming what was observed, no code; each says the entry committed no run), and that handle's result() reports the same detail. Tests: cp85-executor-entry-outcome.test.ts (all three, for execute(), resume() and retry(), preCommitFailure and result() agreeing), cfw-settle-closed-owner.test.ts.
awaitEntryRunStart (runtime-cloudflare/src/executor.ts) returns a preCommitFailure only when the closed instance's output is { status: 'failed', … }. An instance that closed errored or terminated before its run-start commit has no such output, so the handle carries no preCommitFailure and a chat SEND (handleChatStream) still attaches to the session's ended stream and streams an empty turn. Not reproduced; found by reading the code. Fix direction: for a closed instance with no run, derive a detail from the engine status (errored → its message, terminated → framework_cancelled, as DI-34's settle does) and return it as preCommitFailure.
FU-AGENT-SERVER-START-PRE-COMMIT-HANDLE: agent-server startAgent answers a runtime-js pre-commit failed handle as started — done
Closed by: the completion-strategies test-hardening MR (!325), Task 15: AgentServer.startAgent rejects a handle carrying preCommitFailure with helixErrorFromDetail(handle.preCommitFailure), and POST /start answers any thrown HelixError with core typedErrorResponse, as the DO's /start does. The final-review fix round extended it: 503 only while the failed entry left no session behind (a re-sent /start on an existing session is refused by the pre-check), and resumeAgent / POST /resume follow the same rule (503 only while the current run did not move). Tests: agent-server/src/__tests__/http-start-entry-failure.test.ts. Not done: the fix direction's "extend the remote client to type it", now tracked as FU-HTTP-REMOTE-TRANSPORT-TYPED-START-CODE.
AgentServer.startAgent (agent-server/src/agent-server.ts, after executor.execute()) never reads handle.preCommitFailure. runtime-js's C9 path (execute()'s first history read fails before the run-start commit) returns a failed handle instead of throwing, so the Node remote-agent host answers /start with { sessionId, streamId, runId } and the caller attaches to the session's previous stream: the B1 defect on the Node host. Fix direction: when the handle carries preCommitFailure, answer the typed error (core helixErrorFromDetail → typedErrorResponse) instead of started; extend the remote client to type it.
FU-HTTP-REMOTE-TRANSPORT-TYPED-START-CODE: HttpRemoteAgentTransport.start turns agent-server's typed /start answer into a plain Error — open (partly done) (LOW, owner: the next agent-server / remote-agents MR)
Done in WA MR1 (the typed-code half): core remoteProtocolError parses a non-envelope body with PeerErrorBodySchema and throws helixErrorFromPeerBody (the body's code and cause, retryable from the body, else exactly on a 503), on start(), resume() (which used to throw a generic Error for every answer) and the other routes (R54). Still open: the fetchWithRetry half below.
Since !325 (Task 15) agent-server's POST /start answers a typed entry failure (a thrown HelixError, or a pre-commit failed handle) with core typedErrorResponse: { error, code } with the core status (400 / 409 / 503 / 500). HttpRemoteAgentTransport.start (core transport/http-transport.ts, toProtocolError) recognises only the ALREADY_RUNNING, NOT_FOUND and FEATURE_UNAVAILABLE envelopes, so that typed answer becomes new Error('Remote agent start failed (<status>): {…}'), with the code only in the message text. A Node consumer of a remote agent cannot tell a typed /start refusal (e.g. framework_completion_strategy_unavailable, state_history_incomplete) from a transport failure, nor read retryability. Fix direction: parse the body with a Zod schema shaped like runtime-cloudflare's PeerErrorBodySchema ({ error?, code?, cause? }) and, for a known core ErrorCode, throw a typed HelixError carrying the code (retryable only on 503), mirroring FU-DO-STUB-TRANSPORT-TYPED-CODE-LOST and DOFrontendExecutor's typedPeerError; with a test per status row.
Also (!325 final review): HttpRemoteAgentTransport.fetchWithRetry retries every 5xx on its own, including a typed 500 (the non-retryable row) and a typed 503 whose re-send agent-server's pre-check then refuses. A remote consumer therefore never sees the typed answer: it sees the retry's 409 (ALREADY_COMPLETED / ALREADY_RUNNING), which masks the root cause. The fix must also stop retrying a /start (and /resume) 5xx that carries a typed { error, code } body: return it to the caller, who decides from the status (503 only is retryable).
FU-DO-STUB-TRANSPORT-TYPED-CODE-LOST: DOStubTransport turns a child DO's typed { error, code } into a plain Error — done
Closed by: WA MR1 (the CF DO drive pipeline, spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, branch reliability-cf-wa-do-drive). DOStubTransport and core HttpRemoteAgentTransport share core remoteProtocolError: an envelope with errorCode (the DO's /subagent routes now send it, plus cause) or any 503 becomes a HelixError with that code (retryable exactly on a 503), and a non-envelope typed body falls back to helixErrorFromPeerBody (R54). Tests: do-stub-transport.test.ts.
DOStubTransport.toProtocolError (runtime-cloudflare/src/do/do-stub-transport.ts) recognises only the ALREADY_RUNNING, NOT_FOUND and FEATURE_UNAVAILABLE envelopes. A child DO's typed B1 answer to /start (e.g. 409 state_history_incomplete, 400 framework_completion_strategy_unavailable) becomes new Error('Sub-agent DO start failed (409): {…}'), with the code only in the message text, so the parent cannot tell a typed refusal from a transport failure. Fix direction: parse the body with PeerErrorBodySchema (as DOFrontendExecutor does since B1) and throw a typed DOPeerError (retryable only on 503).
FU-DO-DUPLICATE-SUBMIT-DURING-REQUEST-DRIVE: a duplicate DO submit whose stored answer a REQUEST drive is about to drain, with no wake in flight, still answers caller_resumes — open (LOW, owner: W1)
Surfaced by CP-198's fix (!453). A local duplicate submit (already_completed) whose answer is still pending now waits for the DO's in-flight wakes (_inflightWakes: every requestWake, from the submit route, the CP-52 pending wake, onStart and the alarm) and answers what the continuing wake did. The case it does not cover is a REQUEST drive that holds the stored answer between its run-start commit and its drain while no wake is in flight on the DO: /resume continue (its drain follows the commit), /retry with answered trailing calls (rule 6 drains before the loop), and a with_message whose stash needs a drain (an approved approval's tool runs first; a consumed answer rides the run-start commit, so it has no window). A duplicate there still answers the executor's caller_resumes, and handleChatStream's resume() meets that drive: 409 state_already_running.
Why no red is reachable today: every way a ready, stored answer stays pending also runs a wake on that DO (the submit that completed the batch, the onStart wake after an eviction, the unconsumed-submit alarm), and the wake is tracked from before its producer-claim wait. A wake that meets a registered request drive records the CP-52 pending wake, which stays in flight until that drive settles, so a duplicate during the request drive's window awaits it. The window needs a wake that SETTLED without consuming the answer while the request drive has not drained yet (e.g. ended lost or skipped before the request drive's run-start commit, with nothing re-requesting a wake). No code path was found that does that.
What would make it reachable: an entry that drains a stored answer without any wake being requested (a new route, or a wake path that resolves skipped while leaving the answer to a request drive), or a test hold that ends a wake lost / skipped before a request drive's commit. Fix direction: register a request drive's entry outcome in the same in-flight set (resolved once its run-start commit and drain landed), so the duplicate answers that drive's run (deferred / new_run) through answerFromContinuation.
FU-DO-RPC-COMMIT-LOST-RESPONSE: a DO store commit made over RPC cannot resolve a reply lost on the way back — open (LOW, owner: CF backlog)
Surfaced by the guarded-commit outcome MR (spec §3.4 / §7). Inside the Durable Object a commit runs in one synchronous transactionSync, so its reply cannot be lost, and DOStateStore answers a failure CommitNotAppliedError. A caller that reaches the store over RPC (DOStateStoreTestProxy, a stub transport) can still lose the reply after the commit ran, and today gets the transport error raw. Fix: resolve it at the RPC layer as the remote stores do (read back the commit's own checkpoint by id), or document the proxy as test-only.
FU-DO-PEER-ERROR-REJECT-CAUSE-LOST: the DO path drops a rejected run start's runStartRejectCause — done
Closed by: WA MR1 (the CF DO drive pipeline, spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, branch reliability-cf-wa-do-drive). The DO's native answer is 409 { error, code: 'state_run_start_conflict', cause } (core typedErrorResponse({ cause })); the DO clients rebuild RunStartRejectedError(sessionId, cause) when the cause parses with core RunStartRejectCauseSchema (an unknown cause is dropped, never echoed); handleChatStream reads it through its shared classification. Tests: do-frontend-executor-typed-errors.test.ts, handle-chat-stream-resume-typed-errors.test.ts.
A rejected run start on the DO (RunStartRejectedError, e.g. entry_status_changed) is answered 409 { error, code: 'state_run_start_conflict' } with no cause, and DOFrontendExecutor builds a DOPeerError that has no rejectCause. handleChatStream therefore cannot read the cause: its 409 has no cause, unlike the same rejection on a runtime-js host ({ error, code: 'state_run_start_conflict', cause }), so a client cannot tell which gate refused the send. Fix direction: carry the cause in the DO's typed body and on DOPeerError (as rejectCause), and let runStartRejectCause read it.
FU-AISDK-ENTRY-RUN-ID-COMMITTED-CHECK: handleChatStream's "did the entry commit?" check depends on a consistent getCurrentRun read — done for runtime-js and the CF DO (b22 A1/A2); Temporal, DBOS and CF Workflows: FU-ENTRY-RUN-PARITY-TEMPORAL-DBOS-CFW
Closed by (runtime-js; the DO in A2): b22 A1 (spec docs/superpowers/specs/2026-11-11-entry-input-answer-design.md §5.2). entryRunCommitted is replaced by resolveEntryInput: the error's own entry (core HelixError.entryAnswer, set by runtime-js's answerEntry seam and by core's RunStart* errors) decides first; only an executor that reports nothing falls back to the current-run comparison, whose failed read now answers unknown (never a guessed "committed").
When execute() throws a retryable HelixError, handleChatStream (ai-sdk/src/handler/handle-chat-stream.ts, entryRunCommitted) decides whether the entry's run committed by re-reading getCurrentRun and comparing it with the run read before execute(). That assumes a store read after a commit is consistent. On a lagging read (a Postgres or Redis read replica, D1 read replication) the re-read can still return the prior run for an entry that DID commit: the SEND is answered 503 and a re-send appends the message twice. The stores as wired today read their own writes, so this is theoretical; the CF Durable Object is immune, since it tests its own minted run id (getRun(runId)) locally. It is also conservative the other way: a run another entry committed meanwhile (e.g. a path-4 abandon continuation) makes a genuinely pre-commit failure answer 409 / 500 instead of 503. Fix direction: pass the entry's runId back to handleChatStream (let MinimalExecutor.execute take it, or return it on the thrown error; runtime-js already has entryRunId), so it can use the DO's exact getRun(runId) test instead of comparing current runs.
FU-CHAT-HANDLE-RESPONSE-MESSAGE-ID: a committed chat send still re-reads its run to learn the response message id — done for runtime-js (b22 A1) and the CF DO (A2); the other runtimes: FU-ENTRY-RUN-PARITY-TEMPORAL-DBOS-CFW
Closed by: b22 A1: a handle with runId + startUIMessageCount (reportsEntryRun) is streamed with no run read (streamHandle); a handle without them keeps the read.
Surfaced by: FU-EXAMPLE-OPENNEXT-UNTYPED-ERROR-RESPONSE (WE-2, 2026-10-14), the MR in which handleChatStream answers every HelixError itself.
- What: after a send's
execute()returns its handle (the run-start commit landed),streamHandle(ai-sdk/src/handler/handle-chat-stream.ts) re-readsstateStore.getCurrentRun(handle.sessionId)only to derive the response message id,msg-${startUIMessageCount}. A failure of that read after the commit is now answered typed and never 503: the per-request entry ledger marks the input committed, so a client is never told to re-send the message. But the send still fails on a read it does not need. - Fix direction: carry the run's
startUIMessageCount(or itsrunIdand the count) back on the execution handle (AgentExecutionHandle/MinimalExecutionHandle) from every runtime's run-start commit, and drop the re-read.- That is a cross-runtime handle change: runtime-js, the CF Durable Object (
DOFrontendExecutorand the DO's/startanswer), Temporal, DBOS and CF Workflows. - It pairs with FU-AISDK-ENTRY-RUN-ID-COMMITTED-CHECK, which wants the entry's
runIdback for the same reason. - The ledger stays as the backstop.
- That is a cross-runtime handle change: runtime-js, the CF Durable Object (
FU-AISDK-RESUME-SUBMIT-PRE-COMMIT-UNTYPED: handleChatStream's resume and submit paths answer a typed pre-commit failure as a generic rejection — done
Closed by: the completion-strategies test-hardening MR (!325), Task 15: handleResumePath answers a typed HelixError from a submit or the resume (after the auto-resume-race reattach), and a resume handle's preCommitFailure, with core typedErrorResponse; inputCommitted once the request stored a submit. Tests: ai-sdk/src/__tests__/handler/handle-chat-stream-resume-typed-errors.test.ts; conformance DI-39 (fixed).
B1 made a SEND (handleFreshTurnPath) answer a typed entry failure with core typedErrorResponse ({ error, code } and the status table). handleResumePath (ai-sdk/src/handler/handle-chat-stream.ts) does not: its resume catch (after the auto-resume-race reattach check) and its submit loop (submitToolResult throws) both turn a typed HelixError, e.g. a pre-commit state_history_incomplete or a completion-support refusal, into a generic rejection (rejectedOnlyResponse(… makeErrorRejection(message))), so the code reaches the client only as message text. Fix direction: after the auto-resume-race reattach check, map every typed HelixError there with typedErrorResponse; pass inputCommitted: true when this request already stored a submitted tool result (a re-send would submit it again); add a pre-commit-failure test case for each path (the resume catch and the submit loop).
FU-CFW-STALL-RETRY-SAFETY-WORDING: the entry stall message says "retrying is safe" for every caller — done
Closed by: CF-D2 MR 1 (!316), user ruling 2026-10-07: RunStartStalledError is retryable: false for every outcome, and its message names the recovery (executor.retry(agent, sessionId); for not_committed, a reader such as getHandle() settles the run first). not_found now reads "could not be read".
RunStartStalledError's outcome wording (STALLED_OUTCOME_WORDING in core/src/errors/agent-errors.ts) ends the failed, not_committed and not_found outcomes with "retrying is safe". That holds for retry(): it rewinds the failed run to its entry (planRetry), so the run's input row is not duplicated. A plain re-execute() of the same input after a stall does not rewind: the stalled run's run-start commit already appended the input row, so the new turn appends it a second time. not_found can also be a lagging read (the session or run exists but the read did not see it), not proof that nothing was written. Fix direction: say which call is safe ("call retry()"), and word not_found as "could not be read" rather than "no longer exists".
W1 MR2a (CF store and reader truth) follow-ups
W1 MR2a (2026-10-08-cf-w1-mr2-store-reader-truth-and-delivery-design.md Part A) moved D1 and memory-cloudflare migrations onto one core runner, fixed RM-65 and added the core closed-owner planner and executor. These items are what it left open.
FU-RESUSPEND-INTERMEDIATE-ACTIVE-WINDOW: a re-suspension writes the session active before setting it back to paused — open for CF Workflows only (JS / DO half done in WA MR1, 5e4d3792c1; CFW half: WB, the CFW-loop session)
Surfaced by: W1 MR2a review of core isSuspensionCommitted.
The planner decides a suspension from the session (paused and awaiting calls or children). Two re-suspension paths write the session active first and set it back to paused only afterwards, while the calls / children are still recorded:
- runtime-js R47
reSuspendOnRestoredBatch/writeEntryState(js-agent-executor.ts~798-858); - the CF Workflows resume / respawn path (
workflow.ts~1896-1990).
An owner killed in that window reads as not suspended, so the planner chooses fail_uncommitted and the run is failed although the turn's answers are stored.
- JS / DO half: done in WA MR1 (Task 6b,
5e4d3792c1). The resume entry state carriespausedand the restamped calls (planRestoredBatchEntry), written in the DO's run-start commit and in runtime-js's one fenced entry write;reSuspendOnRestoredBatchwrites no session state, only thesuspended_client_toolrun status (nocompletedAt). Proven byruntime-js/src/__tests__/r47-atomic-resuspend.test.ts. The first bullet's write order is gone. - CFW half: WB (the CFW-loop session).
FU-RECONCILE-SUSPENDED-KIND-MIXED-STEP: a mixed server+client step killed in its status window derives suspended_client_tool — open (owner: the reliability orchestrator to route; it touches every runtime's suspension writes)
suspendedStatusOf(session) reads only the session's maps, which do not record a completed server tool beside a pending client call. A running run killed between its suspension commit and its run-status write therefore gets the label suspended_client_tool where the runtime would have written suspended_step_partial. Resume is unaffected (it is driven by suspendedStepId and the gating predicate); only the run-status label differs. Fix: persist the decided suspension kind in the suspension commit, so a reader never derives it.
FU-RM111-RESUSPEND-LABEL-PARITY: Temporal and DBOS label a mixed step's re-suspension suspended_client_tool — open (LOW, parked with the Temporal / DBOS buckets)
RM-111 makes Temporal and DBOS write suspended_step_partial for a step that runs a server tool beside a pending client call (core classifyPhase1Settle). A resume that re-suspends that step still writes suspended_client_tool: Temporal's resume early exit derives the kind from the maps (deriveSuspensionFromState, workflow.ts), and DBOS's R47 re-suspend hard-codes it (shared.ts, the second suspendRun call). CF Workflows keeps the label from the previous run's status (continuesPureClientTool). Fix: carry the suspended run's status into the re-suspension, as CFW does. No conformance cell witnesses it. Two more labelling notes, both cosmetic:
- DBOS decides the kind before the step's tools run, from the call kinds. A client call whose arguments then fail validation still counts as awaited, where CF Workflows counts its error row as a completed call.
suspendedRunRefusalHint('suspended_step_partial')says the run "continues on its own when its partial step resumes", which reads wrong for a server+client mix (it continues once the client results are submitted). C-4 refuses a step with pending client calls before that hint, so it does not show for a mixed step today.
FU-DBOS-HANDLE-SUSPENDED-KIND: a DBOS handle reports suspended_client_tool for every parked workflow — open (LOW, parked with the DBOS bucket)
The active handle synthesizes a suspended_client_tool result for any workflow parked on DBOS.recv (handles/active-handle.ts), so a mixed step's AgentResult.status disagrees with its run record (suspended_step_partial since RM-111). Fix: read the current run's status for the result.
FU-JS-RECONCILE-CLOSED-OWNER-ADOPT: runtime-js adopts the core closed-owner planner / executor in its recover path — open (owner: WA where the DO's embedded runtime-js path is involved, else the Node backlog)
js-agent-executor.ts ~:1003 still carries its own finish-or-fail copy. Replace it with planClosedOwnerReconciliation and reconcileClosedOwnerRun, supplying runtime-js's owner probe. And implement MinimalExecutor.reconcileSession (W1 MR2b §4.2), so a chat-only host reconciles a closed owner.
FU-TEMPORAL-RECONCILE-CLOSED-OWNER-ADOPT: Temporal adopts the core closed-owner planner / executor — open (parked; owner: the parked Temporal bucket, DoD §E)
Temporal's finish-or-fail copy (executor.ts ~:942) adopts the planner and executor. It also adds the proof that Temporal's real suspension writes satisfy isSuspensionCommitted (the CFW and runtime-js proofs are suspension-committed-predicate tests). And implement MinimalExecutor.reconcileSession (W1 MR2b §4.2), so a chat-only host reconciles a closed owner.
FU-DBOS-RECONCILE-CLOSED-OWNER-ADOPT: DBOS adopts the core closed-owner planner / executor — open (parked; owner: the parked DBOS bucket, DoD §E)
DBOS's finish-or-fail copy (interrupt.ts ~:366) adopts the planner and executor, with the same suspension-predicate proof. And implement MinimalExecutor.reconcileSession (W1 MR2b §4.2), so a chat-only host reconciles a closed owner.
FU-CFW-COMPANION-WAIT-SUSPENDED-CHILD-AS-FAILED: a CF Workflows companion wait records a suspended companion as a failure — fixed for suspended_client_tool (CP-40 (C2) T5); open for the other two suspensions (LOW, unowned, parked)
Fixed (CP-40 (C2) T5, ruling 3): a companion whose event (delivered, or from a consult of a closed child) is a committed suspension on client / approval calls (suspension.childStatus: 'suspended_client_tool') is answered suspended with its awaited calls, and its ref records paused_awaiting_client (never delivered); a later-turn waitForResult also reads that suspension from the store (readCommittedChildSuspension). Still open (Q3): a suspended_awaiting_children / suspended_step_partial event keeps the failed mapping. The "cascade" direction below is dropped (ruling 3: the companion answers its own suspension); the remaining fix is to keep a park on those two (they are live on the DO, Q-B1). The original record follows.
Surfaced by: W1 MR2a review (a deferred minor).
The two companion waits in runtime-cloudflare/src/workflow.ts (the blocking companion__spawnAgent wait and companion__waitForResult) treat any completion event other than complete as a failure. Since W1 MR2a the parent's consult of a closed child (consultClosedChildRun) can answer { status: 'suspended' }, exactly as a delivered notification of a suspended companion already did, so a companion that suspended on a client tool or its own children is recorded as a failed consult instead of cascading the parent's suspension. The outcome predates the MR (a delivered suspended event was already a failure there); the consulted path's message is now the weaker generic one. Fix direction: give the companion waits the sub-agent wait's suspension cascade (push the child to the collected suspended children and suspend the parent suspended_awaiting_children), for a delivered and a consulted event alike.
FU-CFW-LATER-TURN-WAIT-LIVE-ROUTING: a later-turn companion__waitForResult learns a child that finishes mid-wait only at the next slice end — open (S3, deferred; owner: W1 if picked up)
Surfaced by CP-131's design (W1; orchestrator ruling 2026-10-16: deferred). A later-turn wait started no consult of its own, so CP-126 drops every event, and the store consult answers. With consult-first, a child that had ALREADY finished answers at once. A child that finishes DURING the wait is learned at the next slice end, up to CFW_WAIT_SLICE_MS (30 s), because its event goes to the closed spawning instance. Pick this up only if the cell or a user shows the latency matters.
Design (both parts together; either alone does nothing):
- (2) Accept the child's live consult. At park start, the wait reads the child's current run (a step) and accepts an event whose
parentToolCallIdis that run'sparent.toolCallId. Stale events carry other ids and stay dropped, so CP-126 stays fail-closed. - (3) Route. At notify time, the child sends to the parent session's CURRENT run's instance (CFW
ownerToken, viagetCurrentRun(parentSessionId)when live), else to the spawner. The status-aware gate applies to the RESOLVED instance.
Conditions (WA):
- Prove that a CFW child run records
RunMetadata.parent.toolCallId, citing the run-start commit for a spawn, a__continue__and a resume. A child with no parent would make (2) silently accept nothing. - Tests:
- a later-turn wait accepts the child's live-consult event before the slice ends;
- another call id is still dropped;
- a later-turn wait answers, and the next turn's notifier does NOT re-inject it (MR B's delivered mark).
- A failed current-run read falls back to the spawner and never throws the child's tail.
- An event to a current-run instance that is not waiting is dropped fail-closed (debug log).
(1) Consult first (built, then dropped from the notify-gate MR by ruling, 2026-10-16): a park with no consultToolCallId (a later-turn wait) runs its store consult once, as a step (<prefix>-consult-first), before its first slice. Value is narrow, because turn 2's checkCompletedChildren syncs a stale ref from the child's session and handleWaitForResult already answers a terminal ref before parking. So it only helps a child that finishes after that sync and before the park, and (1) alone does nothing for a child finishing mid-wait. Parked on origin branch reliability-cf-w1-parked-later-turn-wait-consult-first (head 5aa78b880d), two commits on cea4b87780:
- the change:
parkForChildCompletion, beforeonRejectedSlice; - its unit test
cfw-park-consult-first.test.ts, 3 cases: a finished child answers before any listen; a running child's first read answers nothing, then the park listens; a park with its own consult still listens first. It was red 2 of 3 on cea4b87780.
FU-FLAKE-E2E-TEMPORAL-COMPANION-RESUME-15S-WAIT: a Temporal persistent-companion parity case waits 15 s of wall clock for a detached resume — done (W1; test only)
Seen: 2026-10-17, CI, WE MR C1's MR pipeline 2958660983 (head 5e05c41342), test:integ:e2e 3/4 job 17256285013 (retried as 17256555410 by the orchestrator). persistent-subagent-parity-harness.integ.test.ts › "Persistent sub-agent parity harness — Temporal › resumed-companion-outcome-deliverable" failed with child …-agent-helper never reached 'completed' (1 failed | 787 passed; the shard ran 718 s under a runner backlog).
- Not the MR's: C1 touches no Temporal code and not this test. The same shard was green in C1's previous pipeline 2958565094, and main was green on it at
da336553a6,fc7b3bc622,8361a4d41d,7e49f42fd1and4d355e1eca. - Mechanism (from the test): the case's
assertcallswaitForChildStatus(stateStore, childSessionId, 'completed')(the helper atpersistent-subagent-parity-harness.integ.test.ts:343-355), which polls the child session every 50 ms against a fixed 15 000 ms wall-clock deadline. The child's resume is detached (a Temporalwf.startChildcontinuation), so under CPU or runner pressure it can finish after the budget. The same class as FU-FLAKE-CFW-WF-NOISO-WAIT-BUDGET-UNDER-LOAD.
Next (when Temporal un-parks): anchor the wait to an observed event, not a bigger number. Await the child workflow's own result through the Temporal client handle (or the run's terminal record), then assert the status. Do not ledger a bigger timeout as the fix.
Seen again: 2026-11-04, merge-train pipeline 3025869575 (the shard ran 738 s), costing the finish batch a train cycle.
A second cause (found while fixing it). The Temporal driver ran the parent through TemporalTestRunner.executeWorkflow, whose per-execution worker runs worker.runUntil(parent.result()). The resume child is started with ParentClosePolicy.PARENT_CLOSE_POLICY_ABANDON on the parent's task queue (workflow-dispatch.ts, the __resume__ start), so once the parent completes, a child that still needs workflow tasks has no worker, and no budget, however large, lets it finish.
Done (W1, test only): the driver runs the parent through executeWorkflowWithHandle and stops that worker only after the case's assert; the case awaits the child's own completion through a driver-supplied awaitDetachedChild (Temporal: the resume child's workflow, found by its StartChildWorkflowExecutionInitiated event in the parent's history, awaited to its result; in process: the resumed run's terminal status), with the case's test timeout as the only bound. Local red proof (slot w1-tpar-slot-20261104-031809, the resumed child's model call held 16 s): the pre-fix file failed on both drivers (Temporal by the test timeout: executeWorkflow returned only after its drain, 16462 ms); the fix's wait with the worker stopped as the parent completed still timed out on Temporal (the child had no worker); the fix passed all 20 tests, then 3 more runs unmutated.
Latent, same pattern, no fix needed today: persistent-subagent-temporal-routing.integ.test.ts ("live non-blocking spawn …") also leaves a detached child with no worker after executeWorkflow returns, but asserts only what the parent writes, with no wait on the child. A new Temporal e2e case that asserts a detached child's progress must keep its worker (executeWorkflowWithHandle) and await the child itself.
FU-E2E-PARITY-HARNESS-WAIT-CANCEL-ON-TIMEOUT: the parity harness's detached-child waits are not cancelled when a case times out — open (LOW, test hygiene; owner: W1)
Surfaced by: Greptile's review of !687 (persistent-subagent-parity-harness.integ.test.ts, the awaitDetachedChild / done driver hooks of FU-FLAKE-E2E-TEMPORAL-COMPANION-RESUME-15S-WAIT).
- The gap: the case's 30 s Vitest timeout fails the test but cancels nothing it started. In process (JS),
awaitDetachedChildkeeps polling the child's current run every 50 ms; once a later case clears the shared store it readsnullforever. On Temporal, a pending childresult()never settles, so the case never reaches thefinallythat callsdone(), and the parent's worker is not shut down. - Scope: the failure path only. A passing case awaits the child's own completion and then shuts its worker down (the red proof in the entry above). A timed-out case is already a red test; what stays behind is a poll loop or a running worker for the rest of that test file, not a wrong verdict.
- To close: a cancellation signal from test cleanup (an
AbortControlleraborted byonTestFinished/afterEach) that ends both waits (the JS poll checks it; the Temporal wait races the child'sresult()against it) and lets thefinallyrundone(), with no larger budget. Test only; needs its own red proof (a case that times out with a never-finishing child leaves no poll or worker behind).
FU-FLAKE-TEMPORAL-CONCURRENT-SUBAGENT-FOLD-TIMEOUT: Temporal 16-way parallel sub-agent fold test timed out at 60 s — open (parked; owner: the parked Temporal bucket)
Surfaced by: !337 pipeline 2933981207 (head 041db0a68d), test:integ:rest job 17083529197, Intel DigitalOcean runner, 1 385 s against main's ~545 s; retried by !337 under DoD §A3. Filed by the conformance owner at the orchestrator's request.
- Signature:
runtime-temporalconcurrent-subagent-fold.integ.test.ts› "Temporal runtime — parallel sub-agent customState fold › composes all 16 parallel afterSubAgent folds (append)":Test timed out in 60000ms(60 642 ms; the file took 152 s, 1 of 2 tests failed). - Cause context (measured by the CI-runner session in SigNoz): hypervisor CPU steal on the shared DigitalOcean droplets, up to about 3 of 8 vCPUs stolen. In the same job the CF sibling (FU-FLAKE-CF-CONCURRENT-SUBAGENT-FOLD-TIMEOUT) timed out in 4 of its 9 tests.
- Fix direction: a dedicated-CPU runner pool (the user's decision), and test margins derived from the measured per-fold cost instead of a fixed 60 s. FU-FLAKE-CF-CONCURRENT-SUBAGENT-FOLD-TIMEOUT holds the CF profile of the same 16-way wave.
FU-FLAKE-TEMPORAL-C1-CONCURRENT-SUSPENDS-TIMEOUT: concurrency-invariants-stress C-1 on the Temporal lanes misses its 90 s barrier about 1 run in 4 — open (parked; owner: the parked Temporal bucket)
Surfaced by: the W1 MR2a DoD matrix, local test:integration on 2026-10-10 (Temporal HELIX_TEMPORAL_BACKEND=test-server, private Redis Stack and Postgres). Reproduced on origin/main 7dc07aa1ec at the same rate, so it does not come from MR2a.
The test:
packages/e2e/src/__tests__/concurrency-invariants-stress.integ.test.ts› "C-1: 20 concurrent suspends each write atomically". It throwswaitFor timeout after 90000ms(:132, called from:282) ontemporal-memory,temporal-postgresortemporal-redis. The barrier waits for all 20 sessions to show a pending client-tool entry and apausedstream.Rates (
-t "C-1", alternating branch and main, host load 2-16):- origin/main
7dc07aa1ec: 2 timeouts in 9 runs; - W1 MR2a
54b020b48a: 3 in 10, including one fulltest:integrationrun at load 60-78. - Fisher one-sided p = 0.56, so the branch is not worse.
- Every timeout run also logged the crash in FU-FLAKE-TEMPORAL-TEST-SERVER-WORKER-EXITED.
- origin/main
Leads, unconfirmed:
- the 20 concurrent
execute()calls pop one shared FIFOMockLLMAdapterqueue, so a stalled or retried LLM activity can take another session's response; - the test-server crash below can drop a workflow mid-run.
Check which sessions miss the barrier (pending entry vs stream status) before changing the 90 s budget.
- the 20 concurrent
FU-FLAKE-TEMPORAL-TEST-SERVER-WORKER-EXITED: e2e Temporal integ runs log "Worker exited unexpectedly" / "Channel has been shut down" — open (parked; owner: the parked Temporal bucket)
Surfaced by: the same W1 MR2a matrix runs (2026-10-10).
- What is logged: vitest reports an unhandled
Error: Worker exited unexpectedly(tinypoolonUnexpectedExit). The Temporal test-server's gRPC logsNettyServerHandler onStreamError/Channel has been shut down. - Rates: origin/main
7dc07aa1eccrashed in 3 of 9-t "C-1"runs, MR2a in 4 of 10 (Fisher p = 0.57). In one main run the file died after 4 s with no test result. - Effect: it fails the file even when every test passes, through vitest's unhandled-error exit. It also shows up alongside every C-1 timeout above.
- Not yet known: whether the worker exits on its own or the test-server drops the channel first. Capture the worker's stderr and the test-server log for one crash before choosing a fix.
FU-FLAKE-USAGE-REDIS-TEMPORAL-SHARED-SCRIPT-QUEUE: usage-redis-temporal.integ.test.ts shares one adapter queue across tests — open (parked; owner: the parked Temporal bucket; not confirmed flaking)
The file shares one ctx.llmAdapter across its tests and beforeEach only clears it. A lingering workflow could drain the next test's scripted responses, the same shape as FU-FLAKE-CF-DO-PERSISTENT-AGENTS-SHARED-SCRIPT-QUEUE. It has not been seen to flake for this reason.
FU-RM65-COMMIT-TO-CLAIM-GAP-UNTESTED: no e2e pins the CFW commit-to-makeLive gap — open (LOW, owner: the conformance owner / WC)
The RM-65 e2e (send-claim-lag-cfw-engine.wf.test.ts) holds the instance's first claim, after makeLive has /inited the stream. The earlier gap, between the run-start commit and the claim step's makeLive (the path-2b waitForLive attach on the binding-side manager), is covered by waitForLive but not pinned at e2e. It is reachable with a beforeInitStream hook on the cfw-engine lane.
FU-AISDK-STREAM-INFO-READ-FAILURE-MASKED: handle-chat-stream.ts reads a failed getStreamInfo as "no stream" — open for headBeforeExecute (owner: WC, cf-wc-chat-contract); the headBeforeSubmit half done (W1 MR2b)
handleChatStream calls getStreamInfo(...).catch(() => null) at ~:708 (the stream head before the submits, headBeforeSubmit) and ~:1100 (headBeforeExecute, on the SEND path, before execute(); ~:1092-1101). Those are the only two swallowing sites in packages/ai-sdk/src and packages/agent-server/src (the other calls, ~:327, ~:392, ~:1451, ~:1626, let a throw propagate). Since RM-65 and the F4 fix wave, null from getStreamInfo means only "no such stream" and a failed read throws StreamConnectionError, so the .catch turns an outage into a missing head: the continuation then streams from the wrong cursor instead of failing with a typed error. Replace the swallow with a typed failure (retryable, as resolveCommitSequence does with state_checkpoint_sequence_unavailable) or let it propagate.
headBeforeSubmit half: done (W1 MR2b Task 5, Q7). The resume path reads the pre-submit head without the mask (a read / unread union): a failed read is logged, never blocks the submits, and is answered after them as a typed transport_error with inputCommitted: true (500), never an attach from sequence 0 (on caller_resumes after resume() ran). Tests: ai-sdk/src/__tests__/handler/resume-continuation.test.ts (the unreadable-head cases). Still open: headBeforeExecute on the SEND path (WC).
FU-RECONCILE-SUSPENDED-K5-RUNTIME-TESTS: no runtime-level test covers finishing a suspended run whose own K5 landed — open (owners: WA for the DO; the parked DBOS bucket for DBOS)
Since the F4 fix wave, core finishCommittedRun and committedTerminalStatus accept any live run. Two runtime callers now finish a suspended_* run whose own K5 landed (correct under I-R, where they used to leave it live or take the fallback status): the CF DO wake path (durable-object-agent-base.ts ~:1416-1440) and DBOS settleDeadOwnerRun (runtime-dbos/src/lifecycle/interrupt.ts ~:335 / :366). Core's executor and finish tests cover the shape, but neither runtime has a test that drives it end to end.
FU-SQL-SPLITTER-CASE-END-WRAPPED: splitSqlStatements closes a trigger block at a CASE's END; wrapped onto its own line — open (LOW, owner: core / store maintainers)
The block closes only at a ; preceded by END starting a body statement (at a line start or right after a ;); a bare end identifier or t.end before a ;, and a mid-line CASE … END, no longer close it, and a block still open at the end of the input throws. A CASE's END wrapped onto its own line (or right after a ;) and followed by ; still closes the block early, and the trigger then fails to prepare (loudly). No shipped SQL is affected. A full fix needs a tokenizer that tracks CASE depth.
FU-DO-STREAM-READERS-READ-FAILURE-AS-ABSENT: the DO stream readers still read a failed /info or /chunks as absent — done (WE MR C1)
DurableObjectStreamManager (store-cloudflare/src/durable-stream.ts) follows the reader rule agreed with W1: null / [] only when /info answers exists: false; a non-2xx /info (createReader, createResumableReader), /chunks (getAllChunks, getChunksFromStep, getChunksFromSequence) or /sse answer throws StreamConnectionError (recoverable on a 5xx), and a /sse body that ends with no terminal frame is a typed retryable StreamDropError. The CF DO's DOStreamManagerClient follows the same rule (CP-28).
W1 MR2b (CF delivery: submit continuation, CP-104 / CP-105) follow-ups
W1 MR2b (2026-10-10-cf-w1-mr2b-delivery-design.md, upgrade guide submit-continuation.md) added the required submit continuation, CF Workflows' self-waking __submit__ continuation, the one-entry abandonment (abandonPending), the with_message drain-first order with planRetry rules 4–6, the core companion delivery union and the chat-reader hook. These items are what it left open. Each names its owner; parked Temporal / DBOS items are also listed in the orchestrator's backlog (PARKED).
FU-STORE-COMMIT-FIELDS-SILENTLY-IGNORED: a store written before a MessageCommit field silently ignores it — open (LOW; owner: WE-2, Additive Sessions MR 4)
Filed by: WE-2 (2026-11-07), at the orchestrator's request. Additive Sessions MR 2 (!756) adds two optional fields to appendMessages' commit: expectMessageCount (refuse the append, MessageCountConflictError, unless the session holds exactly that many rows) and deliverContextItems (mark queued context items delivered by this commit's rows). A third-party store that predates them still implements appendMessages, which every runtime guard requires, but ignores both: no TypeError, no refusal. It appends unguarded and never marks an item delivered, so the item is delivered again. The runtimes' construction guards (assertStoreImplements over the required run and late-answer methods) check methods, never fields. It already applies to deliverLateAnswerNotices, whose expectMessageCount (core late-answer-submit.ts:340) fails open on such a store.
- Related, pre-existing:
deliverLateAnswers(the SDK-2 late-answer delivery field) has the same gap: its methods are guarded, but a store that has them and ignores the field is not caught. - Fix direction (MR 4): tie the fields to guarded methods, so a store implementing the context-item queue (
recordContextItem,listUndeliveredContextItems, added to the required methods and to a coreCONTEXT_ITEM_STORE_METHODSlist spread into every runtime guard, red units first) must honour both fields; a capability marker is the alternative. The store contract suites (CI-, EMC-) already prove the five first-party stores honour them. Recorded despite the no-back-compat rule because the ignore is silent.
FU-STORE-STALE-STATE-ERROR-UNTYPED: StaleStateError carries no code on any store — open (owner: WF, the migration-free store-change order)
Surfaced by: W1 MR2b design §1.3.7 (the owner's correction; catalogue CP-110).
StaleStateError (core/src/errors/agent-errors.ts, export class StaleStateError extends Error) is a plain Error with no code and no retryable, on every store (memory, Redis, Postgres, D1, DO-SQLite). A caller can only match it by class; a host classifies it as an uncoded, transient failure. W1 MR2b only made CF Workflows' submit re-read and re-apply on it (persistDurableSubmit, 5 attempts shared with the superseded re-read, Task 2a; CP-110). Fix direction: give it a typed code (e.g. state_stale_version, retryable) through classifyKnownError, across all five stores and every runtime that catches it, with the store contract asserting the code.
FU-DO-SIZE-EVIDENCE-UNRECORDED-BINDS: a few large DO / D1 binds are not recorded for the size backstop — open (LOW, owner: CF backlog)
Filed by: MA-31 (Task 2b review, amendment A8). The by-evidence backstop (BoundSizeWitness.mapDriverError against CF_SQL_MAX_VALUE_BYTES / D1_MAX_VALUE_BYTES) maps a driver throw to state_size_exceeded only when a value it RECORDED in that operation was over the platform cap. MA-31 measures or records every column of the state row, every message row, the staged writes and every commit's checkpoint copy. Still unrecorded, outside those paths and small in practice:
- DO-SQLite (
runtime-cloudflare/src/do/do-state-store.ts):interrupt_contextinupdateStatusandcompareAndSetStatus;createCheckpoint's rawINSERT INTO checkpoints, which stringifiescheckpoint.stateinline (the commit checkpoint,writeCommitCheckpointSync, already records its copy); the child consult'soutcomeJSON onsub_session_refsinrecordChildOutcomeandrestoreChildConsult. - D1 parity (
store-cloudflare/src/d1-state.ts):interrupt_contextinupdateStatusandcompareAndSetStatus; a sub-session ref'sremoteJSON (the ref insert and the ref update).
A production SQLITE_TOOBIG / D1 value error on one of them stays an untyped DOStateError / D1StateError. Wanted: record (or assert, against maxStateBytes) each such bind and bind the measured string, with a sqlite unit case per column.
FU-CFW-RESPAWN-SUFFIX-FNV-COLLISION: CF Workflows child instance ids use a 32-bit FNV-1a run suffix — open (owner: WB; Q9)
Surfaced by: W1 MR2b design §1.4 alternative 6.
respawnSuffix(runId) (runtime-cloudflare/src/workflow.ts, export function respawnSuffix) hashes the parent run id with 32-bit FNV-1a. Two runs of one session whose suffixes collide make createChildInstance adopt the OTHER run's child instance (already_exists → get()), so a respawn or spawn could attach to a stale child. The id is computed in the workflow BODY and matched by later poll steps, so changing the hash is not replay-safe: a parent replayed after the deploy must keep deriving the id its recorded create used. Fix direction: a replay-safe migration (e.g. a wf.patched-style version marker in the instance params, the new SHA-256 key deterministicInstanceKey for new parents only), with a replay test of a pre-change history.
FU-CONCURRENCY-GROUP-TEMPORAL: Temporal refuses agents that use a tool concurrencyGroup — open (MEDIUM)
runtime-js and the CF DO run a group's calls one at a time on the folded state; Temporal runs a step's regular tools as per-tool executeToolActivity instances (Promise.all), each seeded from the committed pre-step base. Until it sequences a group's activities, each seeded from the previous one's staged writes, TemporalAgentExecutor refuses a grouped agent (assertRuntimeSupportsConcurrencyGroups, framework_not_supported, at execute / resume / retry, recursing into sub-agents and persistent companions). Fix: per-group activity chains in the workflow's phase 1, passing the predecessor's StepWrites into the next activity's tracker baseline (replay-deterministic: the order is the model's toolCalls order).
FU-CONCURRENCY-GROUP-DBOS: DBOS refuses agents that use a tool concurrencyGroup — open (MEDIUM)
As FU-CONCURRENCY-GROUP-TEMPORAL, for DBOS's per-tool durable steps (each loads committed state): sequence a group's steps and seed each from the predecessor's staged writes. Refused at DBOSAgentExecutor execute / resume / retry until then.
FU-CONCURRENCY-GROUP-CFW: CF Workflows refuses agents that use a tool concurrencyGroup — open (MEDIUM)
As FU-CONCURRENCY-GROUP-TEMPORAL, for CF Workflows' own step loop (announceToolCalls, one step.do per tool): chain a group's step.do bodies and seed each from the predecessor's staged writes. Refused at CloudflareAgentExecutor execute / resume / retry (refuseUnsupportedAgent) until then.
FU-FINAL-STEP-DYNAMIC-CHILD-REFUSAL: CF Workflows and Temporal check only a child's declared definition for unwired features — open (MEDIUM; closed by FS-3 / FS-4)
Surfaced by: FS-2 (orchestrator ruling 2026-11-09: refuse, never warn, at every start; scoped to the root walk and refuseChild). CF Workflows, Temporal and DBOS refuse an agent whose DECLARED tree sets a feature they do not run yet (core refuseUnsupportedAgentFeatures: finalStep until FS-3 / FS-4 / FS-5, stepContext until Additive Sessions MR 5 / 7 / 8) at every root entry, before anything is written. The tree is the root, its sub-agent tools' _agentConfig and its persistentAgents, recursively. On DBOS that is exact: registerAgent registers that tree and the workflows run only registered definitions. CF Workflows and Temporal also supply core's companion refuseChild (refuseChildWithUnsupportedFeatures), so a companion spawn, respawn or continue through handleSpawnAgent is answered not_spawned / child_not_supported with the typed detail. Not checked yet, because the child's definition there comes from the runtime's registry or resolver at run time, which can differ from the declared one:
- CF Workflows: the local sub-agent started in the workflow body (
registry.get(agentType)beforesteps.registerSubSessions,workflow.ts); theparent_suspendedrespawn in the workflow body; the companionsendMessagecontinue / resume deps inbuildCfwDispatchDeps(steps.ts), outsidehandleSpawnAgent'srefuseChild. - Temporal: the local sub-agent merged in
runLLMStep(activities.ts; refuse it as a synthetic failed row, as the unregistered-tool branch does, never a throw, which would re-run the LLM call); theparent_suspendedrespawn (loadChildStateinworkflow.ts, routed to a failed sub-agent result); the companion continue / resume (reopenCompletedChildand thecompanionResumeRefusalcheck inactivities.ts). Each answer: a failed row for that call with the typedframework_not_supporteddetail; the parent's run goes on. FS-3 / FS-4 wire Final Step on those runtimes and close this forfinalStep. The guide's runtime table says "children resolved dynamically aren't checked until FS-3 / FS-4".
FU-CONF-FINAL-STEP-REPAIR-HOLD: no conformance cell for Final Step's crash-window repair on the CF DO — fixed in code (lands with its MR)
Surfaced by: FS-2's cells (conformance owner, 2026-11-09). Final Step's D11 repair (appendFinalStepRows: the rows a pending attempt owes a log that lacks them, written in ONE fenced append before the forced call) runs only after a process loss between runtime-js commitForcedPhase's state commit and its rows' append. No harness control holds there: 'fault-do-store''s appendFailedOnce makes that append THROW (a failure, which ends the run), not the crash window. Today the repair is pinned by core units only (core/src/orchestration/__tests__/final-step-iterator.test.ts: one fenced append of exactly the owed rows; nothing when the log holds them; a moved log refused, re-read and recomputed; no hook fails framework_not_supported). Fix: a DO-worker hold on commitForcedPhase's rows append (released on an observed event), the DO evicted while held, then 'wake'; the cell final-step.repair-after-crash-window-host judges .rows-once (the attempt's rows written exactly once before the forced call) and .completes.
Closed by: the cf-do control 'hold-final-step-append' (driverFinalStepAppendHolds). The conformance worker's store wrapper holds every appendMessages whose rows carry a Final Step row on the bridge (final.step.append.enter) and counts them. Its harness self-test is harness/cf-do/__tests__/final-step-controls.integ.test.ts. The window exists only where the phase begins after a COMMITTED tool step: the maxSteps cut-off of an outputSchema agent mid-tool-loop (the step iterator's C3 entry, phaseOnly). A phase begun on a prose reply commits the reply and its rows in one step commit, so it has no window. The cell final-step.repair-after-crash-window-host therefore runs conf-fs2 (maxSteps: 4) through four fs2_note steps. It holds the forced phase's rows append, evicts the DO, releases, then wakes, and judges:
.window-reached: the held append, the four work calls and one rows append before the crash;.rows-once: the instruction row and thetoolsrow, once each;.rows-before-forced-call: both rows before the forced answer, one forced call after the wake;.repair-one-append: exactly one rows append after the wake;.completes;.checkpoints-legal.
Mutations: the repair skipped (owed.length >= 0 in the step iterator) and the appendFinalStepRows hook made a no-op each red .rows-once, .rows-before-forced-call and .repair-one-append. js-chat-host is a declared gap (no host to lose).
FU-FINISHWITH-COMMITTED-RECOVERY: a recovery that finds a committed finishWith step with no K5 calls the model again — open (owner: W1, fix planned)
Filed: 2026-11-10 (orchestrator, from W1's survey; named by the Final Step spec FS-1 §7 and never filed).
- What happens: on runtime-js and the CF Durable Object, a recovery that finds no K5 for its run completes the run from a committed terminal step instead of calling the model again, but only for
__finish__. CorefindCommittedFinish(core/src/orchestration/transcript.ts:471-488at a220d5dfa8) matchesFINISH_TOOL_NAMEonly. A run that committed a successfulfinishWithstep and lost its K5 is re-entered, and the model is called again after its answer was already recorded. - Fix direction: recognise a committed, successful
finishWithcall (its real result, not a not-executed or continuation row) as the same honoured terminal, and finish the run from its output. - Proof owed: a red-first case on runtime-js and the DO (lose the K5 after a
finishWithstep commits, recover, and assert no further model call and acompletedrun with that output).
FU-TEMPORAL-DBOS-STEP-FRAME-ORDER: Temporal and DBOS announce a step's parallel calls in completion order — open (parked; owner: the parked Temporal / DBOS bucket)
Surfaced by: W1 MR2b matrix-2 triage F (orchestrator ruling, option a, for core / runtime-js / the CF DO and CF Workflows only).
A live client creates a step's tool parts in the order of their announcing frames (tool_start, tool_approval_request). Core runStepIteration now starts a step's parallel calls one at a time in the model's toolCalls order before running their bodies (RunStepIterationHooks.prepareServerTool). Temporal (per-tool executeToolActivity under Promise.all) and DBOS (per-tool durable steps) write each call's tool_start inside its own activity / step, so the frames follow completion order, and a refresh-free client renders the parts in a different order than the snapshot. Fix direction: write the step's announcing frames in toolCalls order before dispatching the activities / steps (keeping each kind's frame semantics), with a red test where the per-call reads complete in reverse.
FU-TEMPORAL-FINISHWITH-NO-TOOL-FRAMES: a Temporal finishWith call streams no tool_start / tool_end — open (LOW)
Surfaced by: the RM-141 fix's plan (2026-11-21). Related: FU-TEMPORAL-DBOS-STEP-FRAME-ORDER (the order of a step's frames), FU-DBOS-REJECTED-FINISH-FRAMES (DBOS's rejected __finish__ rows), RM-141.
- Temporal runs a step's
finishWithcalls in phase 2,runPhase2FinishWith(runtime-temporal/src/activities.ts:5012, origin/main 200ff4a1f1). Phase 2 runs after the step'scommitStep(workflow.ts:1908, then:2010) and writes onlycustomandstate_patchchunks (:5219,:5271), never the call'stool_startortool_end. A regular tool writes both inexecuteToolActivity(:2509). - Still so with RM-141's fix:
runPhase2FinishWith(activities.ts:5154-5655, origin/main 28e535ae00) writes neither frame. - So a live client sees the step's text and other calls, and the run's completion, but no tool part for the
finishWithcall; its result row reaches the client only through the snapshot. runtime-js, the CF DO and CF Workflows stream afinishWithcall like any tool. - Fix direction: write the call's
tool_startbefore its body and itstool_endfrom the result row (coretoolEndChunkForResultRow) inrunPhase2FinishWith, in call order. Pin it with a cell on the Temporal lanes (afinishWithstep's frames on the stored stream).
FU-TEMPORAL-DEFERRED-HOOKS-STAGE-SIZE: Temporal's deferred-hooks staging throws out of its activity on a size refusal — open (parked; owner: the parked Temporal bucket)
Filed by: MA-31. fireDeferredToolHooks (runtime-temporal/src/activities.ts) stages the onMessage / afterTool hooks' state writes under the synthetic __deferred_tool_hooks__ tool call id with a bare stageChanges. A core StateSizeExceededError there propagates out of the activity, which the workflow's activity retry policy repeats (same bytes, same refusal) until its attempts run out; the run is then expected to fail with the activity failure's detail (DI-04), not state_size_exceeded (not exercised by a test). Wanted: return the refusal as data, discard the hooks' writes (logged with keys and bytes) and continue, as the tool path does, with a test.
FU-TEMPORAL-DBOS-COMMIT-SIZE-REFUSAL: a commit the store refuses as too large is not typed on Temporal / DBOS — open (parked; owner: the parked Temporal / DBOS buckets)
Filed by: MA-31 (Task 8a docs review). runtime-js, the CF DO and CF Workflows fail a run typed and non-retryable when a step or terminal commit is refused with StateSizeExceededError. On Temporal the only size check is the tool path (executeToolActivity's catch, runtime-temporal/src/activities.ts, isStateSizeExceededError): commitStep / commitTerminalState have no size conversion, so the activity is likely retried up to the workflow's maximumAttempts: 3 (workflow.ts, the main proxyActivities retry) with the same bytes, and the run then ends with the activity failure's detail (DI-04), not state_size_exceeded. DBOS has the same single check (runExecuteTool) and drops a HelixError's code across its step boundary (FU-DBOS-SIZE-ERROR-DETAIL-CODE). Neither is exercised by a test: on their usual stores only Postgres's SQLSTATE 54000 (a jsonb value over 2^28 − 1 bytes) can raise a commit refusal. Wanted: convert a size refusal at the commit activities / steps to a non-retryable failure that carries the typed detail, fail the run state_size_exceeded over the stored state, and test it with a store whose limits are small.
FU-TEMPORAL-DRAIN-ROW-ORDER: Temporal's resume drain appends results in the pending map's key order — open (parked; owner: the parked Temporal bucket)
Surfaced by: W1 MR2b matrix-2 triage F (orchestrator ruling 4).
applyResultsAndReload (runtime-temporal/src/activities.ts, the first-pass for (const [toolCallId, entry] of Object.entries(pending))) classifies, and so appends, the drained rows in the stored map's key order; Postgres JSONB returns call_9 before call_10, so the results do not follow the assistant's toolCalls. Not a one-line change: the activity holds no log (it needs a count-checked history read), and the approve-path entries run in a second pass after the synth rows. Fix direction: core pendingEntriesInCallOrder(pending, log) (as runtime-js and CF Workflows do), plus the approve-path interleave below.
FU-DRAIN-FAILURE-ROW-BUILDER: the resume drains build their client-tool failure rows by hand — open (LOW; owner: orchestrator to route)
Surfaced by: CP-265 (two drains had left the clientToolErrorCode stamp off).
Temporal's and CF Workflows' applyResultsAndReload, runtime-js's drain and DBOS each build their timeout, submitted-error and abandonment rows with createToolResultMessage (core drainStoredAnswer builds its own), so the stamp can drift again. Fix direction: one core builder for a client-tool failure row (humanized text + the code), used by every drain.
FU-DRAIN-HOOK-STATE-AFTER-CONSUME: a drain's hook state is written after the commit that consumed its calls — open (LOW; owner: orchestrator to route)
Surfaced by: CP-263 (Greptile on !656). Target: Rule R (#181), "a state write persists iff the row that records its work persists".
A resume drain records the answered calls' result rows, then fires their onMessage / afterTool hooks, then saves the hook-driven customState in a later write: Temporal's applyResultsAndReload (its final saveState) and CF Workflows' (fireSettledAnswerHooks, "the caller persists it with its own write"). Once the rows' commit also CONSUMES the pending calls (Temporal since CP-263; CF Workflows with the late answers 1b drain), a failure of that later write (a non-stale error after the commit) loses the hook writes for good: the retry finds the calls consumed and fires no hooks. runtime-js's drain hooks get no state tracker at all (updateState is a no-op there, js-agent-executor.ts), so their writes are never persisted: a different gap with the same Rule R target. Before the consume, the same failure left the calls pending and the retry appended their rows a second time and fired the hooks twice. Fix direction: commit the hook state with the rows it records (a hook tracker evaluated before the consuming commit, its writes carried by the commit, as #177's drain-first entryState), on every runtime at once. That changes when the hooks fire relative to the gate, so it is its own design step.
FU-LATE-NOTICE-DELIVERY-COUNT-GUARD: a late-answer notice delivery guarded on the message count it read — open, narrowed (LOW; owner: Temporal, Additive Sessions MR 7)
Surfaced by: Greptile on !660 (late answers 1c, the Temporal half, P1). Batched with the next store-contract change (orchestrator ruling, 2026-11-03).
A model step delivers the session's late answers (deliverLateAnswerNotices) after it loads its history. Where two attempts of one step can overlap (a Temporal runLLMStep attempt that timed out keeps running in the worker; possibly a timed-out CF Workflows step.do and a live DBOS step on recovery, WE-2's call), the stale attempt's delivery can commit after the retry loaded its history, and the retry then finds nothing undelivered. Temporal (!660) narrows it two ways: a cancelled attempt (its fence aborted) delivers nothing, and after its delivery a step re-reads the message count and reloads its history when the log holds rows it did not load. The residual: a stale append landing after that re-check and before the stale attempt's cancellation reaches it. The retry's model then misses the notice for that step (it sees it at the next one; nothing is lost) and its baseMessageCount is short by the notice count (the workflow's hook message indexes for that step). Fix direction: an expected-count guard on the delivery commit (MessageCommit, refused typed when the log moved since the caller's read), on all five stores and in the contract, so a stale delivery cannot commit over a newer reader.
Narrowed (Additive Sessions MR 2). The guard landed: MessageCommit.expectMessageCount on all five stores (store contract EMC-1..EMC-4), and deliverLateAnswerNotices passes the length of the log it read. A refused delivery (MessageCountConflictError) delivers nothing and returns no rows, and its records wait for the next model call. So a stale attempt can no longer commit once a newer attempt has WRITTEN since the stale attempt's read. It still can when the newer attempt only READ (its own delivery found nothing, so the count did not move): that is the residual left open, after the runtimes' count re-checks (CF Workflows steps.ts, Temporal activities.ts, DBOS load-state.ts), which stay. Closing it needs a guard on the step commit itself, Temporal's to own with its Additive Sessions wiring (MR 7; orchestrator ruling 2026-11-06).
FU-CFW-STEP-FRAME-CROSS-KIND-ORDER: a CF Workflows step's regular-tool frames precede its suspension frames whatever the call order — open (owner: WB; accepted residual)
Surfaced by: W1 MR2b matrix-2 triage F (orchestrator ruling, CFW half).
CF Workflows writes a step's regular tools' tool_start frames in toolCalls order (one pre-dispatch step, announceToolCalls) and its suspension frames in toolCalls order across client and approval-gated calls, but every regular frame precedes every suspension frame: the suspension frames must follow the run's suspended_* status and precede the suspension commit (status → frames → commit), which outranks cross-kind call order. For toolCalls [approval A, server B] a live client renders B's part before A's; the snapshot renders [A, B]. Pinned as today's behaviour by cfw-step-frame-order.test.ts / .wf-noiso.test.ts. Fix direction, if wanted: announce the suspending calls' parts (a frame that is not a suspension frame) in the pre-dispatch step, keeping the suspension frames where the contract puts them.
FU-CFW-DRAIN-APPROVED-ROW-ORDER: an approved approval's re-executed result is appended after the drain's other rows — open (owner: WB)
Surfaced by: W1 MR2b matrix-2 triage F (orchestrator ruling 4).
CF Workflows' applyResultsAndReload (runtime-cloudflare/src/steps.ts) now classifies the pending entries in toolCalls order (core pendingEntriesInCallOrder), but it appends the simple rows (client results, denies, timeouts, abandonments) in one append commit and runs the approve-path entries (approved: true, which execute the tool) serially AFTER it. For toolCalls [approved A, client B] the log holds [B, A]. Fix direction: append in call order by segments (the simple rows before an approved call, its executed result, the rest), keeping each row's tool_end before its append (R23) and the per-row onMessage indices.
FU-CFW-DRAIN-APPROVE-ROW-COMMIT-RETRY: a store fault between an approved tool running and its row committing re-runs the tool on the drain's retry — open (owner: W1)
Surfaced by: RM-87 (the approve drain's rethrow and its retry skip).
CF Workflows' approve drain (applyResultsAndReload, runtime-cloudflare/src/steps.ts) is ONE step.do. RM-87 makes a framework fault fail it (the engine retries it), and the retry skips every approved call its freshly loaded log already pairs. A fault that lands AFTER an approved tool ran but BEFORE its row's appendCommitted lands leaves the call unpaired, so the retry runs the tool a second time. Phase 1 has no such window: each regular tool is its own step.do, whose cached result replays. Fix direction: run each approved execution in its own step.do from the workflow body (its row returned and committed like phase 1's), or record a durable per-call "ran" marker before the commit.
FU-TEMPORAL-CP154-APPROVAL-BY-ID: Temporal's resume drain classifies an approval by finding its tool — open (parked; owner: the parked Temporal bucket; CP-154)
Surfaced by: CP-154's class sweep (W1; a local probe, not committed).
Temporal persists approvalId on the pending entry (activities.ts, the approval registration) but its drain (GenericActivities.applyResultsAndReload, the isApprovalSubmission check) classifies an approval only by finding an approval-gated tool. With the tool gone, the probe persisted {"approved":true} as the call's SUCCESS (helixToolFailed: false). Fix direction: as CF Workflows and runtime-js: classify by entry.approvalId only, pair a gone tool's approved call as an error, refuse an approval-gated tool's answered entry without the id typed.
FU-DBOS-CP154-RECOVERY-APPROVAL: DBOS's approval answer after a redeploy that removed the tool is unproven — open (parked; owner: Temporal/DBOS bucket (parked); CP-154)
Surfaced by: CP-154's class sweep (W1).
DBOS has no resume drain: an approval's answer arrives by DBOS.recv inside the workflow and is checked against the live tool map (workflows/shared.ts, the phase-1 dispatch). A gone tool only arises when a recovery after a redeploy replays the workflow without it. What it records then is unproven. Fix direction: a recovery test with the tool removed; pair an approved call whose tool is gone as an error.
FU-TEMPORAL-ATTACH-COMPLETED-GUEST-MARKER-WINDOW: Temporal's attach-to-completed handle could project a companion's suspension marker — open (LOW, no cell: no hold point)
Filed by: the conformance owner, 2026-10-30, closing a lead from WE-2's CP-130 cross-runtime read. resume() over a completed / failed root returns attachToCompletedWorkflow (runtime-temporal/src/executor.ts:2433-2445, called at :2149-2162), which projects the first decisive chunk of the stream from fromSequence with no runId filter (handle.ts:125-142, :309-316). Not reachable on the default path: fromSequence defaults to the stream's latest sequence at the call (:2112-2126), and a later guest chunk to the ended stream is dropped (core decideStreamWrite, stream/run-markers.ts:117). Residual: an explicit caller fromSequence (an undeclared v7 option), and a window between the root's K5 and its stream end. Fix direction: filter the attach reader by the root's runId (as chunkFenceRunId does).
FU-TEMPORAL-SUBMIT-SELF-WAKE: Temporal's submit answers caller_resumes and its resume can still be refused while the previous workflow closes — open (parked; owner: the parked Temporal bucket)
Surfaced by: W1 MR2b design §1.3.2.
Temporal's startContinuation answers { kind: 'caller_resumes' } (runtime-temporal/src/executor.ts, the routeSubmitToolResult call), and the follow-up resume() can be refused while the suspended workflow is still closing (the CP-84 shape), so the turn depends on a second request surviving. Fix direction: adopt the CF Workflows shape: a deterministic continuation workflow started by the submit (keyed on the suspended run and a generation), with a body-level predecessor wait, new_run answered, and resume({ mode: 'continue' }) adopting it.
CP-250 (2026-10-30): until this lands, a submit for a paused companion's call whose parent's turn ended is refused typed before storing (unknown_tool_call / owner_not_continuable: Temporal declares no canSelfWake); the companion self-wake (CP-246) is the Temporal half of this item.
FU-OWED-RECHECK-FAILURE-POLICY: a failed post-suspension owed re-check leaves the run paused until another entry — open (LOW)
CP-297's re-check (runtime-js recheckOwedAfterSuspension; the CF DO's continueOwedAfterRelease) is opportunistic. When its continuation start is refused, fatal or still transient after 3 attempts, it logs continuation.owed_recheck_failed at error and leaves the run paused with its answer stored: the next entry (a caller's resume, a later submit, on the DO the durable alarm, whose wake reads the same rule) continues it. Undecided: whether a later entry or a DO wake should retry it on its own, and how often, and whether a refused or fatal start should end the run typed as the client-tool deadline path does (failOwedDeadlineRun).
FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER: a submit from another process inside a companion's pause window answers caller_resumes — open (S3 at most; runtime-js multi-process only)
Surfaced by: CP-315's fix (WA, batch 20). A runtime-js companion driven in process records its pause on the parent's ref only after its suspension commit. The fix makes the self-wake (startCompanionSelfWake) wait, bounded, for that record, then answer new_run (self_wake). It can only wait for a record pending in ITS OWN process (companionPauseRecords, registered by the driver).
The residual: a submit handled by a DIFFERENT process (a multi-process deployment on one shared store) during the window still reads the ref running and answers caller_resumes. The answer is stored and the companion is not stranded: the driver's post-suspension re-check (CP-297's, which CP-315 now runs after an in-process companion's pause record) continues it. Only the continuation the caller is given is wrong: it is told to resume, and no caller resume continues a companion.
Narrowed (CP-315 W1): once the companion's re-check continuation is the current run, a submit from any process reads it and answers new_run (self_wake), since a run that continues the answered call's run is the self-wake (js-agent-executor.ts:7279-7300, W1's branch). What remains is a submit from another process before that run starts: it still answers caller_resumes, and the re-check continues the companion.
Fix direction: decide the self-wake from a durable "pause pending" marker the companion's suspension commit itself writes (for example on the companion's run record, or the ref written inside the commit: CP-315's option A, five stores), so any process can wait on it. Only worth it if multi-process runtime-js deployments answer through other processes in practice.
FU-CFW-COMPANION-SELF-WAKE-REF-READ-ONCE: CF Workflows decides a companion self-wake's ref reopen on one read of the parent's ref — open (PARKED: CF Workflows; static, unmeasured)
Filed: 2026-11-17 with CP-315 W1's runtime-js fix, by the conformance owner. Related: CP-315, CP-250, FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER.
- The code:
companionSelfWakeParams(runtime-cloudflare/src/executor.ts:4677-4705, origin/main 07819d59eb) reads the parent's sub-session refs once and returnsundefined(nocompanionRef/companionRefExpect/companionSelfWakeinput) unless the companion's ref already readspaused_awaiting_client. The submit's continuation label does not depend on that read:startContinuationnames a companion's continuation{ sessionId, relation: 'self_wake' }from the owner role alone (executor.ts:5479-5485). - The question (not measured): if a CF Workflows companion's pause write to its parent's ref can lag its suspension commit, as runtime-js's in-process driver's did (CP-315), a submit inside that window starts the companion's continuation without the ref reopen, so the parent's ref may stay
paused_awaiting_clientor miss the continuation's outcome. CP-315's runtime-js fix waits for, and then re-reads, the pause record; CF Workflows has no such wait. - Next step: trace when a CF Workflows companion writes its paused ref relative to its suspension commit. If a window exists, pin it on the
cfw-workflows-d1lane before changing anything.
FU-JS-CP315-W1-PAUSE-WINDOW-FLAKE: CP-315's W1 cell answers caller_resumes instead of new_run, intermittently on runtime-js × redis — open (recurred after CP-315 W1; owner: W1)
Filed: 2026-11-17 by the conformance owner, at the orchestrator's request. Related: CP-315, FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER.
- The test:
runtime-js/src/__tests__/helpers/companion-pause-window.ts:297(origin/main a183c1b669), "W1: the pause record held while the submit lands", run per store byintegration/companion-self-wake.integ.test.ts. Its assertion (:322) wants{ status: 'accepted', continuation: { kind: 'new_run', relation: 'self_wake' } }. - Sightings: batch 28's train pipeline 3076232893, job 18086032357 (
test:integ:rest), runtime-js × redis: the submit answered{"status":"accepted","continuation":{"kind":"caller_resumes"}}and the case took 254 ms (the same job's W1b passed). And W1's local run on unrelated code, 2026-11-17, as W1 reported it. Then !920's pipeline 3081633337, job 18126147548 (test:integ:rest), runtime-js × redis:caller_resumesagain, 135 ms; and !921's pipeline 3081663421, job 18126396800, runtime-js × postgres: a second shape,{"kind":"deferred","blockingRunId":"owed:…"}instead ofnew_run, 641 ms. Neither MR reaches runtime-js. - Not a timeout: the self-wake waits for this process's pause record for at most
COMPANION_PAUSE_RECORD_WAIT_MS, 2 s (runtime-js/src/js-agent-executor.ts:499), and W1 runs with that default (only W1b shortens it). The submit returned well inside it, so it did not wait out the bound. - Impact: the continuation kind only: as FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER records, the answer is stored and the post-suspension re-check continues the companion.
Done (W1, CP-315 W1). W1's repro (12 of 40 on Redis) found two orderings, neither a timeout. A pause record that settled between the self-wake's first ref read and its wait check left the decision on the stale running read (caller_resumes). And when CP-297's re-check continuation took the companion first, the submit answered deferred (or caller_resumes once that run ended) for a run of the companion's session. The self-wake now reads the ref and run again after the check, and a run that continues the answered call's run is the companion's self-wake (new_run, self_wake). Pinned by W1c, W1c-norecheck, W1d, W1e, W1f (guard) and the e2e re-check on / off pair.
Recurred after the fix (2026-11-18): !927's pipeline 3084748012, job 18149353626 (test:integ:rest, allowed to fail; the merged result 57de27e65b, which changes neither the helper nor js-agent-executor.ts, so the product is CP-315 W1's), runtime-js × redis: the forced-window pin W1c (the combined fix, re-check on) answered {"status":"accepted","continuation":{"kind":"caller_resumes"}} instead of new_run (self_wake), at companion-pause-window.ts:502. The seam's own check just before it (:501, window.applied()) passed, so the window was held. Every other CP-315 case in that job passed, W1c on Postgres included, and W1c passed on Redis and Postgres in W1's slot at the same head.
Investigated (W1, 2026-11-18): Investigation of the post-fix CI sighting (job 18149353626, W1c (the combined fix, re-check on) × redis, answered caller_resumes): not reproduced, so the finding stays OPEN. The whole failing case took 271 ms, so neither the 2 s pause-record wait bound nor the test seam's 5 s cap can have elapsed: the "held past the wait bound" shape (W1b) does not explain it. A static trace of every caller_resumes return in runtime-js startCompanionSelfWake at the fix's base excludes, under W1c's interleaving, the no-owner return, the settled-by-a-stop return (W1c writes no stop), the same-run return, an answer read from the old suspended run after the parent's ref moved (Redis writes the ref reopen and the run start in one script call), and a stale answered-run id (the owner is read before the self-wake starts, and only the self-wake's own read releases the test's hold). Two paths remained open: a claim lost before the winner's run-start commit landed, and a winner whose run ended without carrying the answer before the loser read it. 50 instrumented local runs at the fix's base (40, then 10 recording every write to the companion's session) gave 0 reproductions: every loss was the store's run-start check against a committed, running winner, neither pre-commit refusal ever fired, and the only write to the companion's session before the first run start was the submit's own answer save, which keeps the session paused. The sighting is recorded as unexplained: one of the two open paths under a CI interleaving that was not produced locally, or a cause outside the traced set.
Seen again (2026-11-19, a new shape): !931's pipeline 3087168720, test:integ:rest job 18170344706 (a disclosed retry of the job's D1 timeout; !931 is records only, so this is main's code), runtime-js × postgres: the W1d pin failed its own precondition, the re-check's run started first: expected undefined to be defined (companion-pause-window.ts:549 at origin/main fae53b26ac), after race.applied() (:548) passed. W1d's seam holds the submit's own run start for at most 5 s, until another run start of the companion lands, and records that run (delaySubmitSelfWakeStart, :231-261). The case took 5374 ms, so the CP-297 re-check's continuation did not start the companion inside that cap (inferred from the timing). That shape is new: earlier sightings answered caller_resumes or deferred. Every other case in the job passed.
- Also seen before the fix (2026-11-18): !922's train pipeline 3082294426,
test:integ:restjob 18130972422 (not allowed to fail), runtime-js × redis: W1 answeredcaller_resumesin 166 ms. It predates the fix, as the entry's first three sightings do. - Count since the fix (owner, 2026-11-19): 2 of the 11
test:integ:restjobs whose code holds the fix (its W1d pin present in the log, 2026-11-18 08:32Z to 2026-11-19 05:11Z) failed a CP-315 case: 18149353626 (W1c on redis,caller_resumes) and 18170344706 (W1d on postgres, above). Before the fix, 3 of the 12 such jobs since 2026-11-17T21:00Z failed W1.
Pinned (batch 33, origin/main f0abcc475b): runtime-js logs which self-wake branch answered caller_resumes (companion.self_wake.caller_resumes, its own unit cp315-self-wake-caller-resumes-log.test.ts), and two order pins hold the re-check's run start landing before the submit's own start reaches the W1d seam, W1d-order and W1f-order (runtime-js/src/__tests__/helpers/companion-pause-window.ts:772 and the W1f twin, origin/main 04b758ea69). Since then runtime-js's integration suite runs 226 cases (222 before).
Seen again (2026-11-21, a third shape): !967's pipeline 3097232385, test:integ:rest job 18244471285 (allowed to fail; !967 is test-only CF Workflows, so this is main 04b758ea69's runtime-js code), runtime-js × postgres: W1d-order failed its precondition the submit's own run start was held before the seam (companion-pause-window.ts:787), in 427 ms, with one captured decision, companion.self_wake.lost at +419 ms (Agent session '…-agent-w' is already running (status: running)). Traced, not run: the order seam (holdSubmitSelfWakeStartEntry, :319-343) holds only a self-wake run start that reaches the store's run-start commit (appendMessages with startRun, :330), but here the submit's self-wake was refused before any store write, by the owed-run guard: the self-wake resumes with the run it is owed for, and its first read (readSeenRun, runtime-js/src/js-agent-executor.ts:8215, origin/main 04b758ea69) found the re-check's run current and running, not the owed suspended run, so the guard (:8225-8229) threw RunStartConflictError (expected_mismatch), a lost claim (core isLostClaimError), which the self-wake logs as companion.self_wake.lost. So the re-check won earlier than W1d-order models, and the case stopped at its precondition before reading the submit's answer: a gap in the pin's seam, the shape W1f's seam once had ("a start that landed first was missed"); the submit's answer in that order is not observed.
- Count (owner, 2026-11-21, every
test:integ:restjob from 2026-11-19 05:28Z to 2026-11-21 10:05Z, 50 jobs):- before batch 33's pins (runtime-js 222 integration cases), 4 of 10 jobs failed a CP-315 case, all on postgres: W1d
the re-check's run started first(18177038472, 18180827594, 18193601835) and W1f the same (18187261188); - with them (226 cases), 3 of 40: 18191217811 (!938 before it landed: W1c × redis,
caller_resumes), 18199056115 (!939's merge ref, whose helper was not yet main's: W1d-order and W1f-order on both stores, the seam's cap reached with no other start) and 18244471285 (above).
- before batch 33's pins (runtime-js 222 integration cases), 4 of 10 jobs failed a CP-315 case, all on postgres: W1d
Corrected (2026-11-21): an earlier version of the paragraph above named the live-owner pre-check (js-agent-executor.ts:8414, assertNoLiveOwner). That check runs after the owed-run guard, and its plain AgentAlreadyRunningError is not a lost claim, so it would have been rethrown, never logged as companion.self_wake.lost.
Fixed in the test (as on branch reliability-cp315-w1d-precheck d35863fe53): two cases hold that earlier order, W1d-precheck (start) and (completed) (runtime-js/src/__tests__/helpers/companion-pause-window.ts:918, their seam holdSubmitSelfWakeSeenRun at :379): the re-check's run starts, or starts and completes, before the submit's seen-run read, and the submit loses at the owed-run guard, before any run-start append, and answers new_run naming that run (self_wake). The completed case waits for the companion's session AND its run record to read completed (:405-411): K5 marks the session terminal before the run's own status write, so a session-only wait let the read see the run still running on a store with separate round trips. That is how the earlier seam failed the case on Postgres in the MR's first pipeline (job 18245985906, test:integ:rest: the case expected the guard's refusal to name status completed, and it did not), a test seam's timing, not a product defect. W1d-order and W1f-order gate the re-check's start until the submit's own start reaches their hold (holdSubmitSelfWakeStartEntry, :325, its doc at :312-324; bounded at 10 s, a crossed gate fails the case), so the earlier order no longer reaches them. A local slot at the earlier content (judged at 28cd2fbfb6): the pause-window file 13 of 13 in 20 runs on the memory store, and runtime-js's unit suite 2337 of 2337; with the self-wake's lost branch not consulting the current run, W1d, W1d-order and both W1d-precheck cases fail; without the gate, under a local stressor, W1d-order and W1f-order fail in 3 of 3 runs with job 18244471285's precondition, and with it all 13 pass in 3 of 3. A re-check slot at d35863fe53: the file 13 of 13 in 20 runs on the memory store, and with the seen-run seam a no-op both W1d-precheck cases fail. The memory store cannot show the K5-to-run-status gap, so the new wait's store proof is the MR pipeline's Redis and Postgres run.
Tracks: CP-315.
FU-JS-COMPANION-SELF-WAKE-DRAINED-ENTRY-LABEL: a companion self-wake whose answered entry is already drained names the companion's run deferred — open (owner: W1, after CP-315 W1 lands; static, unmeasured)
Filed: 2026-11-17 with CP-315 W1's runtime-js fix, by the conformance owner. Related: CP-315, CP-104, CP-250, FU-JS-CP315-W1-PAUSE-WINDOW-FLAKE.
- The case: CP-315 W1's fix names a continuation that carries the answer as the companion's self-wake (
new_run { runId, sessionId, relation: 'self_wake' }) by comparing the current run with the answered call's run, read from the owner state the router passes in (ownEntry(owner.pendingClientToolCalls, toolCallId)?.runId,js-agent-executor.ts:7279on W1's branch). When a live self-woken continuation has already drained that entry before the owner is read, the id is gone. The self-wake then answers as before:deferred { blockingRunId }naming the companion's own live run. CP-104 definesdeferredas a run of the addressed session, so this is the label question CP-315 W1 fixed, in the one case its fix cannot see. - Seen in:
runtime-js/src/__tests__/self-wake-outcome-write-order.test.ts("a route reaching the self-wake while the self-woken run is still LIVE never waits on it"), which callsstartCompanionSelfWakedirectly with an owner whose pending entries are empty; it expectsdeferrednaming the live run (:282-285), and it passed in W1's runtime-js unit runs at the fix (s-cp315, 2270 of 2270 at 910a1f29b0; s-cp315r, 2271 of 2271 at 61e7c39d4c (7f1d98884e after the rebase)). - Reachable from a submit (static, not measured): a live self-woken run that drains the entry between the router's write and its owner read.
- Client effect (static):
handleChatStreamtreats adeferredwithout asessionIdas the addressed session's run, and itsdecideAttachre-anchors at once (answered_calls_not_owned_by_run_before): a 200 step-ending response, no error, only a wrong label. - Fix direction: carry the answered call's run id from the router's own write into core's
SubmitContinuationStartInput, so a runtime'sstart()never re-derives it from a later owner read; then the self-wake names a carrying continuation as itsself_wakewhatever the owner read shows. A core contract change: every runtime'sstart()receives it. Pin it with the test above, expectingnew_run(self_wake).
FU-SPAWN-BY-NAME-UNSYNCED-REF: runtime-js: spawnAgent by an existing companion's name reads the parent's sub-session refs unsynced — open (post-features; owner: WA)
Filed: 2026-11-10 (orchestrator, from WA's CP-315 window audit; lines re-checked by the owner at origin/main ae4c3d8040).
- What: core's
handleSpawnAgentreads the parent's refs raw (core/src/orchestration/companion-tool-dispatch.ts:1342,getSubSessionRefs), while its siblings callresolveAndSyncSubSessionRefsfirst (handleListChildren:2648,handleGetChildStatus:2670,handleWaitForResult:2719 and :2761). In the window after a companion's run completes and before its self-wake outcome write lands on the parent's ref (CP-250's write order),resolveChildName(core/src/tools/companion/helpers.ts:95, its busy branch at :132) reads the stalerunningref and answerschild_busy, so a re-consult by name is refused. - Impact: transient (one store write); nothing is lost, and a retry continues the companion.
- Wanted:
handleSpawnAgentsyncs the refs first, as its siblings do, with a unit that holds the outcome write and spawns by name inside the window. - Related: CP-250 (the self-wake write order), CP-315 (the pause window).
FU-CP297-CROSS-PROCESS-ADOPT: a caller's resume() in another process is refused, not adopted, while its turn continues — open (LOW)
CP-297's adopt (runtime-js adoptOwedContinuation) returns the live handle of the owed continuation that the SAME executor started (owedHandles). An app with two processes over one shared store (e.g. a serverless host), where the caller's resume({ mode: 'continue' }) runs in a different process from the re-check, still gets AgentAlreadyRunningError (state_already_running, retryable: false) while its turn continues. Nothing is lost or silent: the caller attaches to the turn through the session stream (getHandle() / stream()) and must never re-send. Undecided: a store-backed handle whose result() waits for a run another process owns, so that process can adopt too.
FU-CP297-APPROVAL-PART-NEWEST-AI: restore the newest-ai half of the approval-part self-test once !770 lands — done
CP-297's harness self-test "POSTs an approval response at once as Chat.addToolApprovalResponse writes it…" (packages/e2e/src/conformance/__tests__/host-driver.test.ts) compares the host driver's approval part with what ai's own Chat.addToolApprovalResponse writes at the supported floor (6.0.170) only. Its second comparison, at ai 6.0.314 through the ai-newest alias, was dropped because that alias comes from !770, which CP-297 does not stack on. Restore it as a small follow-up commit once !770 has landed (in batch 16's assembly if both are in, else the next batch): the ai-newest AbstractChat subclass and one more viaChat(...) assertion.
Done (batch 17): !770 landed on main with batch 16 (the ai-newest alias), and the 6.0.314 comparison is back in the self-test: the ai-newest AbstractChat subclass and its viaChat(...) assertion.
FU-TEMPORAL-RESUME-WITH-MESSAGE-HOOKS: a Temporal with_message resume fires no onAgentStart / onMessage for its new turn — open (owner: Temporal bucket; a cell candidate for the conformance owner)
Filed by: CP-237 (orchestrator, 2026-10-28). Temporal invokes onAgentStart only in initializeAgentState, the fresh-turn entry (runtime-temporal/src/activities.ts:10330-10331); a resume({ mode: 'with_message' }) begins a new turn (its message is the turn's input) but fires neither onAgentStart nor the input's onMessage. runtime-js, the CF DO and CF Workflows fire both for a with_message turn (#177's start work). Wanted: fire the new turn's start work on a Temporal with_message resume (its writes in the turn's start commit), with a cell for the hook parity.
FU-TEMPORAL-WITH-MESSAGE-DRAIN-ORDER: Temporal commits a with_message message before the drained client-tool results — done (RM-74's Temporal half; owner: WA)
Surfaced by: W1 MR2b plan deviation 11, ruling 1 (by reading); confirmed live by the conformance owner (RM-74).
A resume({ mode: 'with_message' }) on a complete batch commits the message rows in startRunCommit (runtime-temporal/src/workflow.ts, the resume branch's activities.startRunCommit), then applyResultsAndReload appends the tool results after it. A user row then sits between an assistant tool_use and its tool_result (core/src/orchestration/transcript.ts; RM-29's pairing contract), and a provider rejects the transcript. The deferred-finishWith leg: a finishWith call left beside the answered client call is hidden by the trailing user row (core findTrailingUnpairedToolCalls walks back over tool rows only), so it is never executed or paired. Red cells (RM-74's ledger): transcript.with-message-after-answered-batch .a-llm-input-paired, .a-message-after-results, .a-persisted-paired, .b-llm-input-paired, .b-message-after-results, .b-persisted-paired on temporal-memory, temporal-redis and temporal-postgres. Fix direction: the CF Workflows / runtime-js rule from W1 MR2b Tasks 5a / 5b: drain first, record a deferred finishWith not executed (unpaired_on_continuation), then commit the message marked markEntryInputRow as the entry's own commit; core planRetry rule 4 then makes its retry need no message. Pre-existing; MR2b does not touch Temporal's step loop (CF-first).
Done (2026-11-16, !909, batch 29): the executor decides the order with core planWithMessageEntry (inputAfterDrain on the workflow input), so the run-start commit carries no message. After the drain, recordDeferredNotExecuted pairs the deferred companion / finishWith / __finish__ calls unpaired_on_continuation, then appendEntryInput appends the message marked markEntryInputRow, stop-guarded (CF Workflows' option A). The owner's slot-rm74m2 (3 of 3, temporal ×3): transcript.with-message-after-answered-batch and finish.with-message-over-deferred pass every id, red at base (slot-rm74m base1). There is no wf.patched marker: only a workflow started with the flag runs the new activities (drain in-flight workflows before upgrading). Tests (origin/main 0aed8d1250): with-message-drain-first.integ.test.ts:202 (leg a), :245 (leg b, each deferred kind) and :289 (planRetry rule 4), and the replay checks with-message-drain-first-replay.integ.test.ts:154 (flagged) and :172 (unflagged).
FU-CFW-RM81-SETTLED-HOOKS-RUNSTART-CRASH: a crash inside a CF Workflows turn's run-start step, after its commit, loses its settled calls' hooks — open (LOW, owner: W1)
Surfaced by the Greptile finding on !420 (fixed in !417): a CF Workflows new turn's settled calls (RM-81) fire the drain's hooks (onMessage, afterTool for a client result; a hook's updateState persisted) in the turn's own durable step, <agentType>-settled-answer-hooks, which receives them through the run-start step's cached result (StartRunCommitResult.settledAnswers). That step has the drain's guarantee (fired again until it completes).
The residual window: the run-start step's body crashes after its run-start commit and before the step's result is persisted. The engine re-executes it with alreadyCommitted and no plan memo (committedEntryPlans is in-process), and the commit already consumed the pending entries, so the plan cannot be rebuilt: settledAnswers is empty and the calls' hooks never fire (their state update is lost). Accepted by the orchestrator's ruling (a), 2026-10-20: one step body between a commit and its result's persist, far narrower than the finding.
Fix direction (b): on a no-memo re-execution, rebuild the records from durable data: the run's committed rows (its run-start prefix: the heal / settled rows before the input) plus the already_completed markers the commit writes for the consumed calls (PendingRestamp.consume, markCompleted), which tell a settled row from a plain C5 heal row; input from the assistant row's tool-call arguments, the kind from the row (not-executed / denial / client result). Child sessions have no markers (markCompleted only on the root), so (b) needs a child path, or (c): carry the records durably in the run-start commit (a run-record field, a 5-store change), not justified for this window now.
FU-TEMPORAL-FAILED-RUN-SUBMIT-SAYS-SUPERSEDED: a late submit for a FAILED Temporal run is refused as "superseded" — done (CP-256, !621)
Surfaced by: the owner's submit.answer-after-failed-suspension cell (owner slot s224 / s225, 2026-10-31; CP-256); its late-submit check is now submit.late-answer-after-failed-suspension-refused.
Temporal's applySubmitToOwner saves an answer fenced on the entry's own run (runtime-temporal/src/executor.ts:2872 at 47f0892469), so a submit for a call whose run ended failed throws RunSupersededError (state_run_superseded, retryable false): "Run '…' of session '…' was superseded". The refusal is typed and loud, and nothing is stored, but the run was not superseded, it failed. Name the refusal for what happened (a terminal run), or drop the failed run's unanswered calls (CP-256's fix) so the submit answers unknown_tool_call as the CF DO's deadline path does (CP-174).
Done by CP-256's fix (!621): the failed terminal drops the call, so the late submit answers unknown_tool_call on Temporal (submit.late-answer-after-failed-suspension-refused, owner slot s229, 3 of 3 on temporal x3).
FU-TEMPORAL-RM81-NEW-TURN-PENDING: a Temporal new turn over a terminal session leaves its pending client calls unpaired — done (RM-128)
RM-81 makes runtime-js, the CF DO and CF Workflows settle a terminal session's pending client-tool / approval calls in a new turn's run-start commit (core planNewTurnPendingSettlement). Temporal's new-turn plan (activities.ts, the root new-turn heal around :9761-9771 at origin/main 826d9665af) still passes the pending ids to buildTrailingHealMessages as excludeIds and sets no pendingRestamp, while its failure paths keep the entries. So a new message over a session that failed while awaiting a client call sends the model an unpaired tool_use. Fix: call the core planner there (approvalDenial: 'result', Temporal's shape), put its restamp on the turn's StartRun, and write the settled calls' tool_end after run_started.
Narrowed by CP-256: a failed terminal now drops its run's UNANSWERED pending calls (core withoutUnansweredPendingOf, at both of commitTerminalState's failed writes), so the new-turn heal no longer spares them and pairs them unpaired_on_continuation. What is left is an ANSWERED or denied entry that was never drained: the failed session keeps it (a retry drains it), and a new turn still excludes it from the heal, so its tool_use stays unpaired. An entry a completed or interrupted session holds is unchanged by CP-256.
Done (RM-128, 2026-11-09): Temporal's new-turn plan (initializeAgentState) calls core planNewTurnPendingSettlement (approvalDenial: 'result', as Temporal's own drain and runtime-js), and the turn's run-start commit consumes the settled entries (pendingRestamp, markCompleted on a root). After the commit, the first execution writes each settled call's tool_end, and the root's complete removes its route; onMessage per settled row and afterTool for an answered client call fire with the start hooks. Before, nothing consumed the entries: the turn's first step commit (commitStep, whose state projection omits the pending map) wiped them with no row and no marker, so the model saw the tool_use unpaired and a late submit answered unknown_tool_call.
FU-TEMPORAL-COMMITSTEP-PENDING-REPLACE: a Temporal step commit replaces the session's pending map from its own projection — open (latent: no writer reaches it today; owner: Temporal bucket)
Filed: 2026-11-09 by WA, from RM-128's trace (the conformance owner: a latent hazard, no id).
- The mechanism (all cites origin/main 07d932612e,
packages/runtime-temporal/src):commitStep(activities.ts:5346) saves the step as a full replace (saveStateAndPromoteStaging, :5451), withpendingClientToolCalls: preservedPendingClientToolCalls(:5438).- That value is a fresh re-read of the store's map only when
mergeRootBoundChildWritesis set (:5396-5401). The workflow sets it only for a step whose sub-agents completed (workflow.ts:1733) and for the deferred-finishWithcommit (workflow.ts:1203). Otherwise it isundefined: the AgentState projection omits the field (the doc at :5357-5358). emitSuspensionMarker(:6944) builds its map from the step's projected state too (input.state.pendingClientToolCalls, :7146), not a fresh read.- So a pending entry another writer puts on the session while a step is in flight would be erased by that step's commit, with no row and no consumption.
- Why nothing reaches it today (each traced read-only at origin/main 07d932612e):
- A same-session submit during a step: a run's own entries exist only while it is suspended. A resume drains (
applyResultsAndReload,workflow.ts:502) and re-suspends at once, without stepping, if any entry remains (workflow.ts:909-922). So a submit during a step finds no entry to write into. (RM-128's dead-run entries on a new turn were the one shape with entries during a step; RM-128 consumes them in the run-start commit.) - A child's write onto the root's map: ownership routes live in
clientToolCallOwnership, written only by the store's field-scoped op and outside the projection (activities.ts:5359). A child's own entries are written on the child's session (emitSuspensionMarker, :7145). - A lapse settle:
persistLapsedDeadlines(:5717) runs only inside the drain (:5994), in the workflow's own activity sequence before any step. - A pre-submit stub: only DBOS writes
isPreSubmitStub: true(runtime-dbos/src/dbos-agent-executor.ts); Temporal has no stub path.
- A same-session submit during a step: a run's own entries exist only while it is suspended. A resume drains (
- Trigger: re-check this FU whenever anything adds a writer of
pendingClientToolCallsthat can run while a Temporal step is in flight (for example a Temporal pre-submit stub, or a lapse settle outside the drain). The fix then is a fresh re-read of the pending map incommitStepandemitSuspensionMarker, asmergeRootBoundChildWritesdoes for sub-agent steps.
FU-DBOS-WITH-MESSAGE-DRAIN-ORDER: DBOS puts a with_message message before the drained client-tool results — open (parked; owner: Temporal/DBOS bucket (parked); RM-74; needs a trace)
Surfaced by: W1 MR2b plan deviation 11, ruling 1; observed by the conformance owner (RM-74).
DBOS's with_message rows ride the run-start commit (runtime-dbos/src/lifecycle/resume.ts), but a DBOS client-tool suspension is a parked workflow consuming answers via recv. Observed: the pairing holds, but the message precedes the results. Red cells: transcript.with-message-after-answered-batch .a-message-after-results and .b-message-after-results on dbos-postgres. Fix direction: trace where the answers land relative to the message on a restart and on an attach, then apply the drain-first rule (the deferred-finishWith leg included).
Update (2026-11-15, CP-271): the observed order no longer reproduces through the cell. Since CP-271's fix a DBOS with_message over a live suspended or woken run is refused (state_already_running) before any write, and transcript.with-message-after-answered-batch meets exactly that (its submits wake the run), so its two ledgered ids now fail because the message is absent. Re-trace owed: whether any DBOS path still lets a with_message reach the drain (for example a session whose owner workflow closed with answered entries still pending, after a restart), and where its rows land there. If none remains, RM-74's dbos-postgres row is re-classified (refused by design, not misordered) with a cell id that judges the refusal; if one does, it gets its own anchored cell.
FU-DBOS-RM81-NEW-TURN-PENDING: a DBOS new turn over a terminal session leaves its pending client calls unpaired and re-stamps them — done (RM-83)
DBOS's new-turn plan (runtime-dbos/src/steps/run-commit.ts, the heal and ...restampAll(current) at origin/main 826d9665af) spared the pending ids in its heal AND adopted the stale entries onto the new live run (RM-83).
Resolution (RM-83). The turn plan calls core planNewTurnPendingSettlement (approvalDenial: 'error', DBOS's shape; markCompleted on a root; the remote children's rows placed after the settled rows) instead of the heal and restampAll, as runtime-js / CF Workflows do: a stored answer drains, anything else is not_executed (unpaired_on_continuation), every entry leaves the session through the run-start commit's restamp, and none is adopted. After the commit: the settled calls' tool_end frames (after run_started), their root bookkeeping (clearPendingClientToolCall, entries only), and, in the workflow body, onMessage per row and afterTool for an answered client call. The approval-gated check reads the live tool registry (RunCommitDeps.resolveTool). DBOS records no client-tool usage counters for these calls (it records none for any client call). Proof: the unit rm83-new-turn-settlement.test.ts. The continuation / resume / retry entries keep re-stamping (they continue the run that awaits the calls).
FU-DBOS-CLIENT-TOOL-USAGE-COUNTERS: DBOS records no client-tool usage counters — open (LOW; owner: the DBOS bucket)
Filed by: RM-83 (WA, 2026-11-09). runtime-js (and the CF DO, which runs it) records the client-tool usage counters (CLIENT_TOOL_*_COUNTER, e.g. CLIENT_TOOL_RUNTIME_RESTARTED_COUNTER) for a drained or settled call (runtime-js/src/js-agent-executor.ts consumedCallEffects). DBOS records none, for any client call: a drained answer, a timeout, or a call a new turn settles (RM-83). Operators see these calls in logs only. Temporal and CF Workflows were not checked for parity here (a source grep finds the counter constants written only under runtime-js). Wanted: the same counters at DBOS's drain and settlement sites, through its usage step.
FU-DBOS-BATCH-ANSWER-BEFORE-SIBLING-REGISTERED: DBOS answers new_run to an answer that lands before its sibling call registered — open (parked; owner: the parked DBOS bucket)
Filed: 2026-11-02, from a source read; no cell.
DBOS runs a step's client calls concurrently, each registering its own pending entry (runtime-dbos/src/workflows/shared.ts, resolver.registerPending in dispatchClientTool). An answer to a that lands before b has registered reads a batch of one, so core resolveSubmitContinuation calls DBOS's startContinuation (dbos-agent-executor.ts), which answers new_run, not awaiting_batch. The run itself still waits for both (shared.ts, Promise.all over resolver.awaitAnswer).
Not traced: what happens when b's answer takes past the 30 s live wait.
A cell needs a control that holds between the two calls' prologues; none exists, and none is being added now (orchestrator, 2026-11-01).
FU-DBOS-CONTINUATION-ENTRY-ORDER: DBOS's in-workflow continuation writes under the suspended run before its run start commits — open (parked; owner: the parked DBOS bucket)
Priority: first DBOS item when DBOS is unparked (user ruling 2026-10-11).
Surfaced by: W1 MR2b matrix-3 triage (refresh-c1-dbos.integ 6/12; debug report .superpowers/sdd/2026-10-10-cf-w1-mr2b-delivery/matrix3-dbos-c1-debug-report.md, MR2b worktree).
Once a parked DBOS turn's waiting calls are answered, the in-workflow continuation makes the session stream live again under the SUSPENDED run: R46's SuspensionGate resumes it once every call is answered (runtime-dbos/src/workflows/shared.ts:437-487, createSuspensionGate :524, whose resume fires at :537, wired to dispatcher.resumeStream at :3249-3251; dispatchClientTool waits on slot.gate.answered at :807). The post-answer work then writes under that run: the answered call's tool_end (emitToolEndStep, :824-850) and an approved tool's execution (dispatchApprovalGatedTool, :1266 then executeTool at :1296). Only after all of that does startRunCommit({ runId: dbosContinuationRunId(<suspended run>) }) run (:3498), which claims the stream and writes run_started. The R47 takeover of restored calls has the same shape (resumeStream :2378, emitToolEndStep:2393, startRunCommit :2420). This breaks the entry order commit → claim → live → run_started → content.
Pre-existing on main. MR2b's chat handler now treats a pinned submit attach as live only on the attached run's own run_started (attachedRunStartWithin). Before that run starts, the suspended run's tool outputs go through the transformer's foreign-output rule, and its other chunks are dropped. So the continuation streams again, but until this FU lands DBOS still has a user-visible residual, for approval-gated tools only:
- (a) An APPROVED tool's whole execution runs under the suspended run before the continuation's
run_started(dispatchApprovalGatedTool,shared.ts:1266, thenexecuteToolat:1296). Its live frames (state_patch, customctx.emitevents,subagent_*) are therefore not streamed: they reach the client only through the snapshot (the re-anchor,retry()or a reload), and a patch-based state view stays stale for the rest of that response. - (b) Its run time counts against the attach's live-wait bound (
DEFAULT_LIVE_WAIT_TIMEOUT_MS, 30 s). An approved tool that runs longer than the bound gets the non-retryable500run_not_live, and the client mustretry()(re-read the snapshot). On main this streamed live, so it is a regression MR2b introduces (through the pin) on DBOS approved tools, which this FU fixes.
Client-tool continuations are not affected: their post-answer work is the answered call's tool_end, which still reaches the client through the foreign-output rule. The user ruled to ship MR2b with this regression documented (2026-10-11, option (x)): docs/reference/ai-sdk.md, docs/runtimes/dbos.md, docs/reference/runtime-dbos.md and the runtime-dbos / ai-sdk changesets state it.
Fix design. Step 1 (first, before any code): settle the row order against runtime-js. The new order below moves rows between commits, and DBOS must then match runtime-js's resumed-run order exactly. Read from JS:
- JS's suspension
stepcommit already holds the assistant row and the step's other completed rows (runtime-js/src/run-loop.ts, the suspension commit'sappendMessages). - A
continueresume's run-start commit carries no rows. - Its drain (
js-agent-executor.tsdrainOrphanClientToolCalls→client-tool-resolver.tsdrainOrphans,pendingEntriesInCallOrder) then appends ONEappendcommit per call, in the model's call order. Each call's frames come first (an approved tool'stool_start/tool_end/state_patch), all under the resuming run. - So in a mixed batch an APPROVED call can come before calls whose results are already known (client results, denials). DBOS's current "assistant → resolved suspending calls → the rest" (
:3482-3487) diverges from that, and so would a run-start commit holding a known row that FOLLOWS an approved call in call order. Step 1 decides where those rows go (in the run-start commit, or each in its ownappendcommit in call order, as JS does) before the steps below are coded.
Then the entry order:
- (a) Once every waiting call is answered, the continuation records the answered outcomes with no stream write.
- It commits the continuation's run start. Its claim makes the stream live, and it writes
run_started. - R46's
resumeStreamunder the suspended run is dropped. - The post-answer chunks are then written stamped with the continuation run: the client tools'
tool_end, and an approved tool's run. Approved tools execute under the new run; their results land in that run's commits, in the order step 1 settled.
Replay safety:
- A crash between the commit and the post-answer work must replay idempotently.
- The deterministic id and
startDigestmake the commit a recorded step, and the post-answer steps re-run as recorded steps. - Tests must prove it: a crash after the commit and before each post-answer step.
Related: RM-74's DBOS half, FU-DBOS-WITH-MESSAGE-DRAIN-ORDER (the same drain, the with_message entry).
FU-DBOS-STOP-AFTER-WAKE-READ-DROPS-COMPLETED-WORK: a stop that lands after a DBOS wake's stop read drops the step's completed work — open (owner: WA)
Surfaced by: WA's DBOS O-73 MR (FU-DBOS-APPROVED-RESULT-SURVIVES-STOP, !890, batch 27), which records a woken suspension's completed work when the wake's stop read finds a stop. The window it leaves: from the wake's readStop(<suspended run>) returning null until the continuation's startRunCommit lands. A stop recorded in that interval makes the run-start refuse expected_run_stopped; the entry settles the run by its K5 (endSuspensionByStop's second branch), and the step's completed rows (an executed approved tool's result, a drained answer) are never committed. The next turn's heal (C5) pairs those calls not_executed, so the transcript stays valid, but the completed work's real result is lost, which Ruling O-73 forbids. Not measured live; the window is narrow (one store read and one commit).
Fix direction: record the completed rows in the expected_run_stopped branch too, under the same run fence, before the settle; or make the continuation's run-start carry them so a refusal records them instead of dropping them.
FU-TEMPORAL-ABANDON-SEND-STUCK: a chat message over pending client calls is refused for good on Temporal — done (!916, CP-143)
Surfaced by: W1 MR2b plan deviation 11 (by reading).
handleChatStream's path 4 on a Temporal host submits client_tool_abandoned per pending call, which only stores, then calls execute(), which meets the paused-with-pending guard (runtime-temporal/src/executor.ts, the C-4 "paused-with-pending guard") and answers 409 state_already_running. The entries are never drained, so every re-send gets the same 409 and the user is stuck. Fix direction: implement ResumeOptionsWithMessage.abandonPending (today refused framework_not_supported, executor.ts resume()'s first statement) and declare supportsAbandonOnResume, as CF Workflows and runtime-js do (W1 MR2b).
Done (2026-11-17, !916, CP-143): the Temporal executor declares supportsAbandonOnResume (runtime-temporal/src/executor.ts:432, as on origin/main fae53b26ac), so path 4 sends ONE resume({ mode: 'with_message', abandonPending: true }) entry with no submit and no execute(); its drain answers each unanswered call client_tool_abandoned before the batch hold, then appends the message. Proof: runtime-temporal's drain, executor and integ tests and the e2e chat cell on the three Temporal lanes (WA's slot-cp143b; CP-143). DBOS's half stays FU-DBOS-ABANDON-SEND-MODEL-STEP.
FU-DBOS-PAUSED-CLOSED-OWNER-WITH-MESSAGE-SKIPS-HEAL: a paused DBOS session whose workflow closed, resumed with a message, may skip the continuation heal — open (parked; owner: the parked DBOS bucket; unverified)
Surfaced by: WE-2's cross-runtime RM-29 order check for the S-C MR (2026-10-17, by reading; not reproduced).
DBOS's continuation heal runs only for a completed / failed / interrupted session (runtime-dbos/src/steps/run-commit.ts:652 root continuation, :745-748 resume, :802 retry). A session left paused whose owner workflow has closed (crashed or cancelled without its terminal commit) and then resumed with_message may reach the run-start commit with a trailing call still unpaired, so the message lands behind an unpaired tool_use (RM-29). To close: confirm the path with a DBOS integ (a paused session, its workflow cancelled, then resume({ mode: 'with_message' })), then either heal on that entry too or settle the closed owner first. Related: RM-80 (the heal's exclusion set), RM-79.
FU-DBOS-ABANDON-SEND-MODEL-STEP: a chat message over pending client calls probably runs a model step on the abandoned results alone on DBOS — open (parked; owner: the parked DBOS bucket; needs a trace)
Surfaced by: W1 MR2b plan deviation 11 (by reading).
Path 4's abandonment submits DBOS.send the errors to the parked run (runtime-dbos/src/dbos-agent-executor.ts), which then probably takes a model step on the abandoned results alone; the follow-up execute() probably meets the live workflow and is refused (runtime-dbos/src/lifecycle/execute.ts, AgentAlreadyRunningError(opts.sessionId, 'active')). Trace it, then implement abandonPending (today refused framework_not_supported).
FU-TEMPORAL-DBOS-MIXED-SEND-ABANDONS-BEFORE-REFUSAL: a send over a mixed suspension still stores per-call abandonments before it is refused on DBOS (Temporal done, !916) — open (parked; owner: the parked Temporal / DBOS bucket)
Surfaced by: S-B (the 2026-10-17 chat incident; design docs/superpowers/specs/2026-10-17-chat-send-no-presubmit-design.md, ruling C2), approved split.
On an executor without supportsAbandonOnResume (Temporal, DBOS), handleChatStream's path 4 still answers every pending call client_tool_abandoned before the send is routed, including over a suspension that also awaits children. Since S-B the handler acts on each submit's own continuation (CP-104): a new_run / deferred answer (a run took the session over) refuses the send with the typed 409 state_already_running instead of an execute() the runtime would refuse, so the message is held client-side (O-71 refusedSend) and never lost. But the client_tool_abandoned results are already stored by then, so the user's pending calls are abandoned by a message that was refused. Fix direction: the D11 with_message / abandonPending entry on Temporal and DBOS (FU-TEMPORAL-WITH-MESSAGE-DRAIN-ORDER, FU-DBOS-WITH-MESSAGE-DRAIN-ORDER, FU-TEMPORAL-ABANDON-SEND-STUCK, FU-DBOS-ABANDON-SEND-MODEL-STEP), after which the executor declares supportsAbandonOnResume and path 4 makes no submit before routing there either.
Temporal done (2026-11-17, !916, CP-143): Temporal declares supportsAbandonOnResume, so its path 4 is the one with_message / abandonPending entry and stores no per-call abandonment before routing; a send over a suspension that also awaits children is refused before any write. What stays open here is DBOS.
FU-CFW-TEMPORAL-SUBMITTED-TIMEOUT-HOOK-KIND: a submitted client_tool_timeout drains under the client-result hook label on CF Workflows / Temporal — open (LOW; owners: WB for CF Workflows, the parked Temporal bucket)
Surfaced by: S-B ruling D (the chat handler's deadline sweep now submits client_tool_timeout, the code the runtimes' own expiry writes, instead of client_tool_deadline_exceeded).
runtime-js and the CF DO treat a submitted client_tool_timeout as a timeout throughout (the drain labels it inherited: false, the counter counts it). CF Workflows (runtime-cloudflare/src/steps.ts, the drain's "Submitted client-tool result OR error" branch) and Temporal (runtime-temporal/src/activities.ts, the same branch) drain it as a client-result hook entry, where their own expiry uses timeout. Checked by reading: both build the same row (humanizeClientToolError('client_tool_timeout', …)), the same afterTool({ success: false }) error and the same errorCode; the label only decides whether afterTool fires (it is skipped for deny alone), so nothing a hook or the model sees differs today. Fix direction: label the entry from the code (submittedError === 'client_tool_timeout' → timeout) so a future per-kind hook cannot drift.
FU-DBOS-PAUSED-STOP-PARTIAL-BATCH-RESUSPENDS: a stop on a DBOS session paused on a partial batch is not honoured by the next continue — open (parked; owner: the parked DBOS bucket)
Surfaced by: the conformance owner (CP-108, batch 19), routed by the orchestrator.
With the durable interrupt flag set while a DBOS session is paused on a partially answered batch, a resume({ mode: 'continue' }) re-suspends instead of ending interrupted (it never settles). DBOS honours the stop for a complete batch. Red cells: stop.paused-flag-then-continued .partial-continue-settles and .partial-ends-interrupted on dbos-postgres. The CF Workflows half (the flag is cleared by the next explicit resume) is FU-CFW-EXPLICIT-RESUME-CLEARS-PAUSED-INTERRUPT (WB). Fix direction: as runtime-js: a pending durable interrupt makes the resume honour the stop at its first boundary, whatever the batch.
FU-CFW-EXPLICIT-RESUME-CLEARS-PAUSED-INTERRUPT: an explicit CF Workflows resume erases a stop pressed while the session was paused — open (owner: WB; CP-108)
Surfaced by: W1 MR2b Task 3 review (deferred minor), routed by the orchestrator.
interrupt() of a paused CF Workflows session only sets the durable flag (the session stays paused), and the next non-recovery resume entry clears that flag in its run-start step (runtime-cloudflare/src/steps.ts, the clearInterruptFlag for entry.kind === 'resume' && !entry.recovery), before the loop's first interrupt check. W1 MR2b exempts its own owed __submit__ continuation (trigger: { kind: 'owed_continuation' }), so a submit after a stop is honoured; an explicit resume({ mode: 'continue' }), including one past R44 on a partial batch, still continues the model after the user pressed stop. Red cells (CP-108): stop.paused-flag-then-continued .full-ends-interrupted, .full-no-model-call-after-stop, .partial-continue-settles, .partial-ends-interrupted on cfw-workflows-d1. assertClientToolBatchComplete's own docstring says a pending interrupt makes the resume only honour the stop. Fix direction: WB decides per mode: a with_message is new input (continuing is arguably right); a continue, and above all a partial-batch resume after a stop, should honour the stop, as runtime-js does. The DBOS partial half is FU-DBOS-PAUSED-STOP-PARTIAL-BATCH-RESUSPENDS.
FU-EXECUTE-OVER-FAILED-SESSION-UNANSWERED-PENDING: a new turn over a failed / completed session that still holds unanswered pending client calls can show the model a dangling tool_use — open (owner: WA for runtime-js / the CF DO)
Surfaced by: S-B fix round 1 review (2026-10-17), by reading; pre-existing, not an S-B regression.
On a supportsAbandonOnResume executor, handleChatStream's path 4 makes no submit for pending calls of a session that is not paused, the O-70 refusal does not fire for a run that is not live, and the send goes to execute() (pinned by send-no-presubmit.test.ts, "documents today's handler behaviour"). On runtime-js (and the CF DO, which runs it) nothing then answers those calls before the new input: the execute path's C5 heal excludes the pending ids (runtime-js/src/js-agent-executor.ts:6293, buildTrailingHealMessages(…, { excludeIds: Object.keys(basePendingClientToolCalls) })), executeEntry (:3700) has no orphan drain, and core failCommittedRun (core/src/run/fail-committed-run.ts:154) does not clear pendingClientToolCalls. So a run that failed while suspended on client calls leaves them pending, and the next turn's model call can see their tool_use with no tool_result. Fix direction: the execute entry drains or pairs them before its input, like resumeSession's orphan drain (drainOrphanClientToolCalls, :2728, called at :5034), in the entry's run-start commit; then prove it on runtime-js and the DO (a failed-while-suspended session, a send: every call paired, the message after the results).
FU-WITH-MESSAGE-MIXED-CHILDREN-ORDER: a with_message over a suspension that also awaits children keeps the message before the children's results — fixed by ruling A (CP-237)
Surfaced by: W1 MR2b plan deviation 11, ruling 2 (residual of option A).
Ruling A (orchestrator, 2026-10-28), fixed by CP-237: the message stays FIRST in the persisted log (append-only, as every runtime writes it), and the MODEL input pairs a child's late tool_result next to its tool_use: core pairLateToolResults, applied by buildMessagesForLLM after #181's omission, moves a result whose tool_use sits in an earlier assistant message with a non-tool row between them into that message's result group, in tool_use order, and leaves every other row in place. Every runtime building its model input there gets it (runtime-js, the CF DO, CF Workflows, Temporal; CF Workflows and Temporal sent the provider-invalid tool_use, user, tool_result order before). runtime-js (and the CF DO) now keeps the park: a with_message drives its awaited in-process children after its run-start commit, and re-suspends with no model call while one still waits. The fix direction below (append the input after the children's results) is superseded. The chat route's refusal of a send over a mixed suspension (S-B, below) is unchanged: FU-CHAT-SEND-OVER-CHILDREN-PARK.
A resume({ mode: 'with_message' }) over suspended_awaiting_children / suspended_step_partial (core suspendedStatusOf) keeps the message in the run-start commit; the children's results are recorded after the respawn (CF Workflows workflow.ts, the childrenToRespawn block; Temporal the same shape), after the message. runtime-js (and the CF DO, which runs it) does not re-attach the awaited children at all: CP-237. W1 MR2b's drain-first order covers a client-tool-only batch (planWithMessageEntry → input_after_drain); an abandonPending over a mixed suspension is the busy refusal (AgentAlreadyRunningError, state_already_running; S-B fix round 2, it was state_not_resumable). Fix direction: append the entry input after the respawn's results too, keeping the run-start commit untouched.
S-B (the 2026-10-17 chat incident; design docs/superpowers/specs/2026-10-17-chat-send-no-presubmit-design.md) found the user-visible cost of this gap. Over a session paused on 7 lapsed client calls plus an awaited child (suspended_step_partial), the chat handler's deadline sweep submitted the 7 calls client_tool_deadline_exceeded before routing; the FIRST completed the batch (a lapsed deadline counts as answered) and woke the DO continuation inside the submit request. Path 4 then skipped D11 (mixed) and submitted 7 client_tool_abandoned; the 5 s poll for "not paused" broke on the continuation's active; the O-70 refusal was skipped on the stale pending map; and execute() rejected with a non-Helix error, answered as a 200 no client reads. The message was never written. What the S-B fix changed (ai-sdk handleChatStream): a send makes no submit before routing on a supportsAbandonOnResume executor; a client-calls-only suspension takes the one D11 entry (its drain answers the overdue calls); a mixed suspension is refused with the typed 409 state_already_running before any write, which useHelixChat holds in refusedSend and retry() sends once the run ends; the sweep runs only for requests without a new user message and writes client_tool_timeout; Temporal / DBOS keep per-call abandonment but act on each submit's continuation (FU-TEMPORAL-DBOS-MIXED-SEND-ABANDONS-BEFORE-REFUSAL). Fix round 2 (found by the real-DO integ): the CF DO's /snapshot wire dropped suspendedAwaitingChildren, so the handler (reading through DOStateStoreClient.loadState) saw client calls alone and sent the D11 entry, which the DO refused state_not_resumable (a code the stock client drops the message on); the wire now carries the awaited children, and the entry itself refuses an abandonPending over a paused session that awaits children with the busy AgentAlreadyRunningError (core planWithMessageEntry), which also closes the race where the session turns mixed after the handler's read. Still owed: the message cannot be SENT over a mixed suspension until its children finish — the full mixed with_message ordering (this item's fix direction), and the RM-29 adjacency ruling it needs (the children's tool_use rows stay unpaired across the user row until their results land, and runtime-js's loop would call the model over a dangling child call), come with a later ruling.
FU-RJS-COMMIT-DROPS-SESSION-ONLY-FIELDS: runtime-js's step / start / suspension commits drop the SessionState-only fields AgentState does not carry — open (owner: orchestrator to route)
Narrowed (2026-10-29): expiresAt is MA-44 (lost on runtime-js, Temporal and CF Workflows, state.session-only-fields-kept); Temporal over the memory store rewriting createdAt is MA-45. memoryExtractedThrough is store-owned since MA-40 and is kept on every lane, and mode is not witnessed. The other fields named below stay here.
Filed by: CP-237 (diagnosis 3). runtime-js builds its commit payload from the in-memory AgentState (loadAgentState, js-agent-executor.ts) and persistViaSaveAndPromote re-reads only pendingClientToolCalls, clientToolCallOwnership, completedClientToolCalls, tracingContext and (since CP-237) suspendedAwaitingChildren from the store. The other SessionState-only fields — remotePendingCalls, suspendedStepId, memoryExtractedThrough, mode, expiresAt, failureReason, createdAt — are written as the payload carries them (absent), so a store whose saveState / saveStateAndPromoteStaging replaces the whole record (store-memory keeps only checkpointId, latestTruncateCheckpointId and the binding) loses them on every commit. CP-237 erased a children park this way (a with_message turn's start commit, with start hooks). Wanted: for each field, decide whether a commit preserves or ends it (as CP-237 did for the park: preserve by default, endsPark at the step and suspension commits), check each store's write semantics, and pin it per store. CF Workflows' saveAgentState (steps.ts:953-1020) rebuilds the session the same way and keeps only the client-tool maps; nothing reaches it between a resume's run start and its respawn scan today. The re-read is also best effort: a loadState that throws leaves the client-tool maps and tracingContext as the payload carries them, so a transient read failure erases them the same way (the park alone fails the commit instead, since CP-237's Greptile review). Decide the same per field.
FU-CHAT-SEND-OVER-CHILDREN-PARK: a chat send over a session parked on its children is still refused, though the entry can now take it — open (needs a contract decision; owner: orchestrator to route)
Filed by: CP-237 (ruling A). Since CP-237 a resume({ mode: 'with_message' }) over a children park keeps the park and pairs the children's late results in the model input, on runtime-js, the CF DO, CF Workflows and Temporal. The chat route still refuses such a send typed (409 state_already_running, O-70 / S-B) before any entry, and useHelixChat holds it in refusedSend until the run ends. Wanted: decide whether handleChatStream sends a with_message over a children park (no abandonPending) instead of refusing it, and if so on which runtimes.
FU-REFUSED-SEND-AUTO-SEND: a send refused state_already_running is held, never sent on its own — open (needs a contract decision; owner: orchestrator to route)
Surfaced by: !401's design note ("Out of scope / for decision"), the 2026-10-17 chat incident; filed by the DoD catch-up (WD, 2026-10-19).
A send refused 409 state_already_running (O-71) is kept in useHelixChat().refusedSend and sent only when the app calls retry(). An app that does not render refusedSend with a send action loses the user's message without a word: the incident's writer chat did exactly that. A stock useChat app gets the refusal as chat.error (!409 adds isRefusedSendError to recognise it) and must handle it itself. Options (!401's note): (a) keep, and document that apps must render it; (b) auto-send it with the same D-2 request key once the run it was refused for ends; (c) auto-send by default, with an opt-out. (b) / (c) are a contract change (a send the app did not call), so they need the user's decision.
FU-AISDK-CF-AMBIENT-STUB-DRIFT: ai-sdk's hand-written @helix-agents/runtime-cloudflare/do-clients stub has drifted from the real DOFrontendExecutor — open (LOW; owner: unassigned)
Filed by: WE-2 (2026-11-07), from a source-based per-tsconfig typecheck. packages/ai-sdk/src/cloudflare/runtime-cloudflare.d.ts is an ambient declare module '@helix-agents/runtime-cloudflare/do-clients', there so tsup's hashed declaration files resolve. Its DOFrontendExecutor lacks submitToolResult, and its execute(...) handle's stream() takes no options, where the real class (runtime-cloudflare/src/do/do-clients.ts:1416) takes { fromSequence }. Anything that type-resolves through the stub instead of runtime-cloudflare's dist sees a narrower API than the one it runs: two false positives in e2e's cf-do-worker tsconfig under the source-based check (conformance-worker.ts(332,42) TS2554 on handle.stream({ fromSequence: 0 }), conformance-worker.ts(1909,35) TS2339 submitToolResult). The dist-based typecheck (CI's) does not show them, and no published type is known to break: LOW.
Fix direction: first grep a fresh build's dist for whether the stub still ships (an old dist re-exported DOFrontendExecutor from runtime-cloudflare/do-clients with no ambient module). Then generate the stub from the real declarations, or drop it if tsup no longer needs it, and add a compile-time check that it matches do-clients' exports.
FU-AISDK-HELIX-CHAT-REATTACH-ERROR-PER-CHAT: useHelixChat's re-attach error outlives a chat switch — open (LOW; owner: unassigned)
Surfaced by: W1's fix of #170 / #173 (!516, 2026-10-25); orchestrator ruling: an FU, kept out of !516.
resyncError is reattachError ?? resyncError, and reattachError is hook state, not per chat. The re-attach controller is recreated for a new chat id (use-helix-chat.ts, the controller effect keyed on chat.id), but a report the OLD chat's controller made (a lost connection, ReanchorExhaustedError) stays in reattachError after the app switches to another conversation in the same mounted hook, until a send or a successful retry() there clears it. !516 fixed the same class for refusedSend (kept with its chat id) and made retry() end on a chat switch. Fix: keep the report with the chat id it was made for (as refusedSend now is), or clear it when the controller is disposed for another id; a red with a chat switch after a reported drop.
FU-AISDK-STOPPED-SEND-PROBE-NO-ABORT: the stop's snapshot probe keeps polling after an unmount or a chat change — done (LOW, test hazard; owner: WE-2; closed by !846 with CP-320)
Filed: 2026-11-11 (orchestrator, from the app's CP-320 report; lines checked by the owner at origin/main fef7348714).
- What: after an explicit
stop()the server took,useHelixChatre-anchors only once the snapshot holds the stopped send, throughprobeStoppedSendInSnapshot(ai-sdk/src/client/stopped-send-probe.ts:33-50: five paced reads, each afetchSnapshotWithRetrywith its own retries). Its fetch options excludesignal(Omit<FetchSnapshotOptions, 'signal'>, :35), so nothing stops it: the caller's guard (ai-sdk/src/react/use-helix-chat.ts:810-812) drops only the result, and the reads go on after an unmount or a chat change, through whatever globalfetchis installed by then. - Impact: no wrong state (the result is dropped); a test that swaps the global
fetchafter the hook unmounts can see the stray reads. - Done when: the probe takes an
AbortSignalthat the hook aborts on unmount and on a chat change, with a unit that unmounts mid-probe and asserts no further read. - Done (batch 21, !846, CP-320's fix):
probeStoppedSendInSnapshottakes an abortsignal(ai-sdk/src/client/stopped-send-probe.ts:36-58), which the hook aborts on unmount and on a chat change (ai-sdk/src/react/use-helix-chat.ts:772-789); unitsuse-helix-chat-chat-scope.test.tsx:307(an unmount) and:318(a chat switch), andstopped-send-probe.test.ts:90(an aborted probe reads nothing more). - Related: CP-320, FU-AISDK-HELIX-CHAT-REATTACH-ERROR-PER-CHAT.
FU-AISDK-SNAPSHOT-LAPSED-APPROVAL-STATE: a lapsed approval renders input-available in the chat snapshot until its row lands — open (LOW; owner: WA)
Surfaced by: the CP-173 + CP-174 (!490) Greptile sweep of every consumer of a saved deadline lapse (submittedAt + submittedError, no submittedResult); orchestrator ruling 2026-10-24: an FU, kept out of !490.
@helix-agents/ai-sdk's snapshot markPendingCalls (snapshot.ts, the pending-call marking) checks the approval answer before the stored error, the order !490 fixed in CF Workflows' drain. So a lapsed approval renders as input-available, a state its own docstring says never appears. It is display-only and short-lived: it lasts until the drain's timeout row lands, the client neither runs the tool nor asks again, and the resumed stream then shows the error.
Any fix must stay compatible with stock useChat / Chat auto-send (the CP-180 / CP-182 submit-contract area: what the stock ai auto-send check sees on a reload). Escalate to the orchestrator before changing a stock-visible state.
FU-INCIDENT-E2E-REMOTE-CHILD-LOST: the incident replay e2e has no lost-remote-child leg — open (owner: WM, after the incident replay MR)
Surfaced by: the incident-replay e2e design review (orchestrator ruling 4, 2026-10-19).
The 2026-10-17 incident replay Playwright spec (opennext-cloudflare-do example) covers the send-stream drop, the client-tool deadline firing on its own, the re-attach and the send over the running run. It leaves out S-C (!412): a remote child whose DO no longer holds its session, settled framework_subagent_lost. That leg needs a server-side fault that removes the child DO's state under ENABLE_TEST_FAULTS. Until it lands, !412 has no e2e tier.
FU-OPENNEXT-INCIDENT-COMPANION-VARIANT: the incident replay e2e has no persistent-companion variant — done (the companion variant MR, WM)
Surfaced by: the incident replay MR's L4b (orchestrator ruling B', 2026-10-20).
The incident replay spec consults its fact-checker as a blocking sub-agent. On the CF DO that child is a remote child the parent awaits inside the step, so while it runs the parent stays running and a send over it is the plain busy refusal. A parent paused on a client call AND an awaiting child is not reachable on the DO: a running remote child blocks the step, and a remote child that suspends on a client tool fails the parent fast (core/src/orchestration/remote-sub-agent-dispatch.ts:541, the DO /status awaitingClientTool at runtime-cloudflare/src/do/durable-object-agent-base.ts:4396-4413, docs/guide/client-executed-tools.md:865). The 2026-10-17 writer chat consulted a persistent companion. A variant with a NON-blocking persistent companion (askUser pending, the companion still running, the parent paused on its client call alone) would cover that shape, where a send takes the D11 with_message path, not the mixed refusal.
Done: incident-replay.spec.ts's companion test (K1-K3) on incident-writer-companion: the parent pauses on askUser while its non-blocking companion runs; the incident's send over it is taken (200), stored once and carried once by one model call, the pending askUser abandoned (D11, !409's no-presubmit send; a mutation that routes it through per-call submits again fails K2 alone); the companion's verdict, read from its own session, reaches the parent's next turn once. The parent DO's ref to a companion has no public read, by design.
FU-OPENNEXT-COMPANION-K3-DELIVERY-RACE: the incident replay companion test's next-turn delivery check is parked on CP-204 — ✅ fixed by CP-204 (WE-2)
Fixed: the DO notifier's refresh (refreshCompanionOutcomes) runs before each pass reads the refs: it awaits the owed verified reads, and a run's first pass reads every outstanding consult itself, so a send right after the companion ended carries its verdict. K3_DELIVERY_PARKED and its early return are removed: K3's delivery assertions are CP-204's e2e witness.
Surfaced by: the review of the companion variant MR (Greptile, confirmed in code; orchestrator ruling, 2026-10-20).
incident-replay.spec.ts's companion test asserted in K3 that the parent's next turn carries the companion's verdict (core's completion notice) exactly once. On the CF DO that is racy: the child posts /companion/child-settled after its terminal commit, and the parent answers 202 and runs its verified read of the child in the background (runtime-cloudflare/src/do/companion-settle-notice.ts); owed reads run only there and on the alarm, never before a parent turn's completion notifier. A send right after the companion ended can therefore get the outcome one turn late, and that turn's request no longer matches its recording. Recorded as CP-204 (S3 provisional, fixer WA). K3 keeps "the companion's own session ended" and the served-verdict read; its delivery assertions sit behind K3_DELIVERY_PARKED, and the recordings of both later turns stay committed. CP-204's fix re-enables them as its e2e witness (set the flag to false).
FU-JS-INPROC-COMPANION-REF-AFTER-STREAM-END: runtime-js writes an in-process companion's parent ref terminal after the child's stream ended (CP-207) — open (S3; owner: WE-2, after the CF work)
Filed by: CP-204's cross-runtime pin (WE-2). runtime-js writes the parent's ref completed only after the in-process child's runSubAgentLoop returned, when its K5 and stream end have landed (runtime-js/src/run-loop.ts :7385-7387 spawn, :7740-7742 continue at main 8cd8beb434; the failure branches, recordPersistentChildNonCompletion at :7402 / :7757, have the same order). Its notifier's ordinary pass skips a ref that is not already completed / failed before it reads the child, so a parent turn whose pass runs in between delivers the outcome one turn late. Core's deliverPendingChildCompletions (Temporal / DBOS) and CF Workflows read every undelivered persistent ref instead, but widening runtime-js's read collides with its C16 deliver-then-continue design (a child that settled under a running ref is delivered by the re-consult call as priorResult; 9 tests), so C16 stays as it is. Wanted (direction a): write the parent's ref from the child's own terminal sequence, before its stream finalizes, on the completed and failure branches alike; mind the terminate race noted at :7376-7384. The red is ready: "CP-204: a child whose K5 landed while the parent ref still reads running is delivered in the next turn and its ref synced" (branch we2-backup-cp204-inproc-red, a46a7c906b).
FU-OPENAI-VCR-INBAND-ERROR-LOAD-FLAKE: an openai-vcr record-mode test failed once under load — open (owner: WM)
Surfaced by: !470's local gate (2026-10-20). In 1 of 6 full examples/openai-vcr runs (vitest run --maxWorkers=2), server.test.ts "passes a 2xx that reports an error in-band through without recording it" failed after 7.8 s (vitest's 5 s default test timeout). It passed 8/8 alone and in the other 5 full runs; the change under test did not touch its path (one live request per mock, sent in one write). The test starts two fake upstreams and two record-mode mocks in sequence. Not reproduced since: 10 more full runs at 1-min load 5-7 passed 10/10, each under vitest's slow-test threshold (no duration printed), so the 7.8 s run looks like a one-off stall (a socket or server that did not close), not a test running near its timeout. Next: if it recurs, capture which await stalled (a per-step timestamp log in the test) before changing any timeout or splitting it. Never retried away on CI.
FU-VCR-HOLD-RUN-SCOPE: an openai-vcr hold catches the first matching request from ANY spec — open (test infra; owner: the examples' VCR)
Surfaced by: !823's merge pipeline (batch 19, job 17902733947, W1's triage, 2026-11-10). POST /__hold { id, bodyIncludes } (examples/openai-vcr/src/controls.ts) holds the FIRST model request after it whose body contains the text, whichever spec sent it. The opennext incident specs reuse one recorded send, so their requests are byte-identical: incident-replay.spec.ts's hold on the fact-checker's prompt caught incident-context.spec.ts's fact-checker call (hold incident-child-444c7944 held hash=944c731ff07fcff9 on its child session child_b4c72d98-…, while replay's own identical request was served 1.5 s later; the retry repeated it), and both specs failed. Order-dependent: locally, at two workers, replay's own request came first and both passed. The batch fix runs every incident-*.spec.ts in a one-worker Playwright project (examples/opennext-cloudflare-do/playwright.config.ts INCIDENT_SPECS). To close: a hold scoped to its own run (for example matched on a per-run marker the held request carries, as stream gates are with [mock-gate:<id>]), so a one-shot hold can never catch another spec's byte-identical request, and then the incident specs can share workers again.
FU-TEMPORAL-COMPANION-CONTINUE-REF-BEFORE-COMMIT: Temporal resets a companion's ref before the continuation's run-start commit — open (parked; owner: the parked Temporal bucket)
Surfaced by: W1 MR2b pre-flight ruling (design §2.5), confirmed by reading.
Temporal's companion continue resets the parent's ref to running, completionDelivered: false in reopenCompletedChild → resetRefForContinuation (runtime-temporal/src/activities.ts), inside the companion activity, BEFORE the continuation child workflow's run-start commit. A continuation whose entry loses leaves a ref no run backs, and the next ref sync can re-deliver the child's previous output: the shape CF Workflows had before W1 MR2b Task 6. Fix direction: move the ref into the child's run-start commit (StartRun.resumedCompanionRef, CP-103 A1), as CF Workflows now does through companionRef.
FU-CONF-CP57-DBOS-STOP-SETTLE-ORDER: a DBOS stop's handle outcome races its own committed terminal — done (the CP-57 MR, batch 28; owner: WA, cells with the conformance owner)
A DBOS stop commits its interrupted terminal a fixed 500 ms after cancelWorkflow (runtime-dbos/src/lifecycle/interrupt.ts settleCancelledOwner, DEFAULT_STATUS_UPDATE_DELAY_MS), while the handle's rejected-getResult() path (handles/active-handle.ts _fetchResult → settleDeadOwnerRun) reports that committed outcome only if it reads after it, else failed (CP-57). On a slow host getResult() can cross the 500 ms, so the handle's verdict is a race: companion.spawn-discarded-on-stop passed on !379's pipeline 2957355618, which touches no DBOS code, while its sibling stayed red in the same job.
- The fix (product): the handle awaits its own stop's committed terminal instead of a fixed window, so a stopped run reads
interrupteddeterministically. That is CP-57's fix. - The cells: CP-57's 13 cells (
scenarios/ordered-stop-settle.ts) need'ordered-stop-settle', which dbos-postgres does not claim, so they are declared gaps there. When the fix lands, the lane claims it and the cells run again; any DBOS id still red then is ledgered on its record from observed runs.
Closed (2026-11-16, the CP-57 MR): a stopped run's handle now waits, read-only and bounded, for its stop's committed terminal and reports it (runtime-dbos/src/handles/run-record-result.ts stoppedRunOutcome, from handles/active-handle.ts's getResult() catch). dbos-postgres claims 'ordered-stop-settle', the 13 cells' declared gaps are gone, and they run again. Measured by the owner's slot-cp57m (3 of 3 identical at the fix): 10 pass every id and every post-stop outcome id passes; the three still red are ledgered on their own records from those runs: CP-16 (stop.mid-llm-call .provider-call-aborted), CP-106 (companion.blocking-spawn-stopped-during-wait, three ids) and the new CP-331 (subagent.spawn-discarded-on-stop, two ids: CP-116's DBOS half). CP-57 itself is fixed with a provenByException naming WA's deterministic unit and integ, since at the base these cells race.
FU-CONF-TEMPORAL-OWNER-CLOSED-OBSERVE: CP-115's children-park cell has no deterministic verdict on Temporal until the lane can tell the parked parent's workflow closed — done (owner, slot s216b)
Surfaced by: !561's pipeline 2997143017, where subagent.new-turn-over-children-park-refused|temporal-postgres|executor admitted the new turn (.new-turn-refused, .park-unchanged, .no-input-appended, .child-result-delivered) while owner slot s179 (3 of 3) and CI's temporal-memory / temporal-redis refused it.
Temporal's execute() has no suspended-run refusal: it refuses a parked parent only through the run-start commit's ownerless proof, while the parent's workflow is still open, and supersedes the park once it has closed. So the cell's verdict depends on whether TURN-2 lands before or after the close. The cell needs 'observe-owner-closed' (runtime-js claims it: no owner outlives a park), and Temporal is a declared gap (no-race-ledgers rule). Fix: a Temporal driver member that describes the parent's workflow and waits for it closed, so the cell sends TURN-2 in the settled order. That is CP-115's Temporal red (admitted), the witness for WA's fix, which adds Temporal's explicit refusal.
Member (owner, 2026-10-31): 'observe-owner-closed' is now a driver member (SessionDriver.ownerClosed, over BackendEnv.waitOwnerClosed): Temporal describes the session's ROOT executions until closed (StartedWorkflows.waitRootsClosed; an owner still open at the bound is answered { closed: false, open }, a backend fault is a HarnessError), the JS lanes answer closed at once. stoppedRunSettles (the stopped-partial cells) waits on it where the driver offers it, which takes CP-84's window out of those cells (!599 job 17627569168, !600 job 17628075890). CP-115's two cells now wait for the parked owner's close in .parent-parked, and their Temporal gaps are lifted (a declared gap on a cell the lane can express is stale). Still open: their Temporal verdicts, ledgered from an observed 3 of 3 under the member.
Done (owner slot s216b, 2026-10-31, 3 of 3 per Temporal lane, identical sets): in the settled order (the parked parent's owner closed) Temporal REFUSES the new turn: subagent.new-turn-over-children-park-refused fails only .ref-unchanged, and subagent.with-message-over-children-park-delivers-child fails only .ref-settled. Both are RM-110's missing ref for an awaited child, ledgered there. The "admitted once it closed" reading above is not what the settled order shows; !561's admission came from a run that did not wait for the close and stays unexplained. The stopped-partial cells matched their ledger on every run (13 on temporal-memory, CI's failing lane): no repeat count proves CP-84's window gone, but the cells now act only after an observed close.
FU-CONF-CP117-TEMPORAL-ORDER-ANCHOR: CP-117's same-step spawn-then-send cell has no deterministic verdict on Temporal — open (parked; owner: conformance owner, with the parked Temporal bucket)
Surfaced by: unexpected passes of companion.spawn-then-send-same-step|temporal-redis|executor (.send-accepted) on !364's pipeline 2955540773 and !370's pipeline 2955508395, neither touching companion dispatch.
The cell forces C-6's losing order with the ref-write hold (the spawn's sub-session ref write waits until a refs read that started while it was held has answered, or the bound). On the JS lanes that forces the send to read before the ref exists. On Temporal the hold opens after-read or at-bound and the send can still see the ref, so the verdict is store timing, and the cell is a declared gap on all three Temporal lanes (no-race-ledgers rule; CP-117's Temporal half is "not covered"). Fix direction: anchor on an observed event, e.g. record the send activity's refs read and the spawn activity's write per step, and hold the spawn's write until the send's lookup has returned (not merely started), or judge from the recorded order instead of the outcome. Lands with the parked Temporal fix for O-72.
FU-HANDLE-RUN-ID: an execution handle carries no runId — done for runtime-js (b22 A1) and the CF DO; the other runtimes: FU-ENTRY-RUN-PARITY-TEMPORAL-DBOS-CFW
Closed by: b22 A1: AgentExecutionHandle.runId (with startSequence, startUIMessageCount, contextIdsTaken); path 4 (handleAbandonAndSendPath) reads the entry run from the handle (readOwnRunStart) when it names one.
Surfaced by: W1 MR2b Task 5a fix round 2.
MinimalExecutionHandle / AgentExecutionHandle carry only sessionId, preCommitFailure and stream. The D11 path (handleAbandonAndSendPath) therefore reads the entry run's startSequence with getCurrentRun after the handle returns, accepting it only when it differs from the run read before the entry. If a NEWER run supersedes the entry's run in that instant, the response anchors at the newer run's start and misses the entry's own chunks (it never replays an earlier run's). Fix direction: a runId on the handle (a cross-runtime interface change), so the handler reads getRun(runId); the same field serves FU-AISDK-ENTRY-RUN-ID-COMMITTED-CHECK.
FU-AGENT-SERVER-RESUME-OWED-WAIT: an HTTP /resume with a message waits the whole owed continuation inside the request — open (owner: WC, the HTTP entry contract)
Surfaced by: W1 MR2b Task 5a fix round 2 review (out of scope).
On CF Workflows a plain resume({ mode: 'with_message' }) on a session owed a submit continuation waits for it to settle (user ruling (b)), bounded by continuationPredecessorTimeoutMs + entryOutcomeFailSafeMs (about 12 min by default). agent-server's /resume (and any HTTP host) runs that wait inside the request. The chat handler avoids it (path 4 sends abandonPending, the busy loser), but a direct HTTP caller does not. Fix direction: a resume option (or host default) that refuses busy instead of waiting, answered 409 state_already_running (retryable), so an HTTP caller re-sends after the continuation settles.
FU-JS-WITH-MESSAGE-AFTER-CRASHED-DRAIN-ORDER: a with_message on a non-paused session holding answered pending calls commits the message before the results — done (W1, after RM-81)
Surfaced by: W1 MR2b Task 5b fix round 3 review; widened in Task 7 round 4 (an interrupted session).
planWithMessageEntry takes the drain-first path only for a paused session suspended on client calls alone. A session that still holds ANSWERED pending calls but is not paused — a run that crashed mid-drain (runtime-js leaves the session active), or an interrupted session — plans input_in_run_start, so a later plain with_message commits the message in its run-start commit and the drain appends the results after it (an RM-29 adjacency break). Applies to runtime-js (and the CF DO) and CF Workflows; pre-existing on main. Fix direction: decide the plan from "answered pending calls trail the log" (as planRetry rule 6 does), not from the session status alone.
Done. The two non-paused shapes had different causes:
- A crashed run's
activesession: coreplanWithMessageEntrynow plansinput_after_drainfor it too, when it holds client calls alone, so the drained results precede the message. CF Workflows takes the same decision. - An
interruptedsession: it holds no pending calls. Every stop's end clears them through corecommitStopTerminal(Ruling 56). The CF DO's R77 interrupted end never meets one: a stop whose K5 does not land keeps its run live (endEntryRunByStop, the stop deadline), and under Ruling O-73 a drain records an answered call before the run loop's stop boundary (CP-147, withdrawn: no reachable entry).
FU-JS-DRAIN-BULK-CLEAR-WIPES-LATER-ENTRIES: runtime-js's post-drain bulk clear empties the whole pending map — open (owner: WA)
Surfaced by: W1 MR2b Task 5b fix round 3 review (pre-existing).
After the per-entry clears, drainOrphanClientToolCalls (runtime-js/src/js-agent-executor.ts, the "BELT-AND-SUSPENDERS" bulk clear) re-reads the session and saves pendingClientToolCalls = {}. An entry written after the drain's read but before that save (a pre-submit stub registered by a racing submit, a stash carrier) is wiped with the drained ones. Fix direction: remove only the drained ids (Object.keys(pending)), through the fenced save, and pin it with a racing-submit test.
FU-JS-RULE6-UNMARKED-NOTIFICATION-REDELIVERY: planRetry rule 6 can stash an unmarked notification it cannot later prove delivered — done (W1 MR3)
Surfaced by: W1 MR2b Task 5b fix round 8 review.
Rule 6 stashes the retry's input on the last answered trailing call. It marks real user rows for the retried run, but a hidden sub-agent completion notification among them stays unmarked, and undeliveredStashRows can never prove an unmarked non-tool row delivered, so a retry after its delivery would append it again. It needs a new-turn execute() over answered pending calls, which no runtime-js flow produces today. Fix direction: mark notifications in the stash too (a marker kind that is not user input), or exclude them from rule 6's stash and re-inject them through the notifier.
Done (W1 MR3), the first direction. One marker had two meanings: ENTRY_INPUT said both "this is run R's input" (rule 4) and "this stashed row was delivered" (undeliveredStashRows). A notification needs only the second. The two are now separate:
- A new
COMMON_METADATA_KEYS.STASH_DELIVERY({ runId }, Zod-checked;markStashDeliveryRow,isStashDeliveryRowOf,isStashDeliveryRow) carries delivery only. - Rule 6 gives every unmarked non-tool row that is not real user input this marker. Rule 5 re-marks such rows for the retried run and keeps them, with or without
options.message; its non-usermessagesget the marker too. undeliveredStashRowsproves a row delivered by either marker's run.ENTRY_INPUTkeeps rule 4's meaning only, so a notification is never taken for input.
The only other stash builder, runtime-js with-message-entry-rows.ts (and so the CF DO), stashes only the entry's user input (UserMessage[]), which is already marked. DBOS's entryInput is an unrelated entry descriptor, not a stash. Tests: retry-plan.test.ts › "FU-JS-RULE6-UNMARKED-NOTIFICATION-REDELIVERY" (3 cases, red before the fix) and entry-input-row.test.ts › "stash-delivery marker".
FU-JS-LOAD-AGENT-STATE-UNTYPED-NOT-FOUND: the continuation read still throws an untyped "session not found" on runtime-js (and the CF DO, which runs it) — done (W1 MR3, taken over from WA with its OK, 2026-10-12)
Surfaced by: W1 MR2b Task 5b review minor 2 (the unreachable branch was removed); the remaining site found while filing.
WA MR1 typed runtime-js's entry refusals (sessionNotFound, FU-JS-RESUME-LOOP-UNTYPED-REFUSALS), but loadFromSession (runtime-js/src/js-agent-executor.ts) still throws Error('Cannot continue from session …: session not found'), which a host classifies as an uncoded transient failure. Fix direction: throw the existing typed sessionNotFound(sessionId) (state_session_not_found, non-retryable), with a test. The same gap on CF Workflows and Temporal is filed separately: FU-CFW-CONTINUE-SESSION-UNTYPED-NOT-FOUND (WB) and FU-TEMPORAL-CONTINUE-SESSION-UNTYPED-NOT-FOUND (the parked Temporal bucket).
Resolution (W1 MR3). loadFromSession throws core sessionNotFoundError(sessionId, { context: 'cannot continue from it' }), the one builder that also replaced runtime-js's private sessionNotFound and the other hand-built copies. Red first: js-agent-executor-continuation.test.ts › "a branch from a session that does not exist fails typed" (rejected with the plain Error before). The message keeps "Session not found", which ai-sdk's branch check still matches on: FU-AISDK-BRANCH-SOURCE-NOT-FOUND-TEXT-MATCH.
FU-AISDK-BRANCH-SOURCE-NOT-FOUND-TEXT-MATCH: ai-sdk recognises a missing branch source by its message text — open (owner: W1 / the ai-sdk bucket)
Surfaced by: W1 MR3 D2 (typing runtime-js loadFromSession).
handleChatStream's fresh-turn path rethrows a branch whose source session is missing (a caller error) instead of answering it, recognised by isBranchSessionNotFoundError (ai-sdk/src/handler/handle-chat-stream.ts), a /session not found/i match on the error's message. runtime-js, the CF DO and core's loadBranchSource (CF Workflows, Temporal) now throw a typed state_session_not_found (core sessionNotFoundError, whose message keeps the phrase); the stores' plain cloneSession errors (Source session not found: …, the JS and DBOS checkpoint branch) and Temporal's continuation read (FU-TEMPORAL-CONTINUE-SESSION-UNTYPED-NOT-FOUND) are still untyped; CF Workflows' is typed (FU-CFW-CONTINUE-SESSION-UNTYPED-NOT-FOUND, done in W1 MR3). Fix direction: type those, then check code === 'state_session_not_found' and drop the text match.
FU-CFW-CONTINUE-SESSION-UNTYPED-NOT-FOUND: CF Workflows' continuation read throws an untyped "session not found" — done (W1 MR3, orchestrator ruling 2026-10-16)
Surfaced by: W1 MR2b, filed with FU-JS-LOAD-AGENT-STATE-UNTYPED-NOT-FOUND. Owner was WB; folded into W1 MR3 by orchestrator ruling, so MR3's "typed missing-session refusals" holds on CF Workflows too.
Fixed: CF Workflows' remaining untyped missing-session throws now go through core sessionNotFoundError (state_session_not_found, non-retryable):
handle.send()'s continuation read (loadFromSession):Session not found: <id> (cannot continue from it);retry()(wasError('Agent session not found: …'));- the resume drain (
AgentSteps.applyResultsAndReload), when the session was deleted under the resumed run.
run-entry.ts's own sessionNotFound() copy (same message and code) is deleted; its 9 steps.ts callers use core's. Tests: executor.test.ts (#retry on a missing session; handle.send() after deleteSession) and steps-applyResultsAndReload.test.ts (a session deleted under the resumed run). Each was red with a plain Error. Temporal and DBOS keep their parked FUs (FU-TEMPORAL-CONTINUE-SESSION-UNTYPED-NOT-FOUND; DBOS handles/active-handle.ts's State not found for sessionId).
FU-CFW-STEP-RETRIES-PERMANENT-NOT-FOUND: CF Workflows' steps retry a permanent state_session_not_found — open (owner: the CFW workstream)
Surfaced by: W1 MR3 (orchestrator, while typing CF Workflows' missing-session refusals).
A step.do whose body throws core's state_session_not_found (for example the resume drain, AgentSteps.applyResultsAndReload, on a session deleted under the run) is retried with the engine's default retries: the engine retries any throw, whatever its class or retryable. The error can never succeed, so the retries waste the step's budget and delay the run's typed failure. Fix direction: throw a permanent HelixError (non-retryable) from a step body as the engine's non-retryable error (NonRetryableError from cloudflare:workflows), keeping its ErrorDetail for the outer catch's resolveErrorDetail, with a test that the step runs once.
FU-CFW-STEP-RESULT-CAP-FOLD-STEPS: CF Workflows steps whose result can be large only fail typed at the 1 MiB cap — open (LOW, owner: WB, the CFW-loop session)
Filed by: MA-31 (A11, Task 5 fix round 2). The step guard (guardLostClaims → assertStepResultFits, runtime-cloudflare/src/run-entry.ts) measures EVERY step result against CFW_MAX_STEP_RESULT_BYTES (1 MiB) and fails an oversized one typed and non-retryable, so nothing strands and nothing re-runs. The tool, sub-agent, remote and companion results are converted instead: a typed failed result, and the run continues. These steps can still carry large data and only fail the run:
- the sub-agent record / fold steps whose result carries the child's
customState(SubSessionFoldInput.childCustomState); ${agentType}-drain-resume;- the
load-*steps, including${agentType}-load-remaining-pending-${stepCount}(the wholeSessionState); ${agentType}-complete-hooks-*/${agentType}-fail-hooks-*(they return the whole state);${agentType}-error-committed-outcomein the outer catch after K5.
A customState nested deeper than MAX_STEP_RESULT_DEPTH (256) fails the load-* and hook steps the same way. Wanted, if the owner rules so: convert the ones that can be (a fold over the cap as a typed failed child result; the state-returning steps by returning a reference and re-reading the state outside the step).
FU-CFW-CHILD-COMPLETION-EVENT-CAP: a CF Workflows child's completion event is not measured against the engine's event cap — open (LOW, owner: WB, the CFW-loop session)
Filed by: MA-31 (A9 / A11; the A9 proposal's tool and child results were converted by A11). Not every payload the engine persists is a step result: a child notifies its parent with sendEvent (notifyParentCompletion, runtime-cloudflare/src/workflow.ts, event type sub-agent-complete), whose payload carries the child's whole output. Cloudflare caps an event payload at 1 MiB too ("Maximum event payload size", https://developers.cloudflare.com/workflows/reference/limits/). The parent fits the outcome it CONSULTS into its own step (fitChildOutcomeInStep), but the event itself is not measured, so an oversized output fails the child's notify-parent-* step, which the child logs as non-fatal, and the parent learns the outcome only by its consult. Wanted: measure the event payload before sendEvent and send a typed "too large" failure in its place, with a test. Since DI-75 a failure event also carries the child's errorDetail, so its message travels twice (error.message and errorDetail.message, plus cause): a failure reaches the cap at about half the message size it did before (the degrade path is unchanged).
FU-CFW-REPLACED-CHILD-REF-STATUS: CF Workflows records a replaced (too-large) child outcome as failed in the child's ref — open (LOW, owner: WA)
Filed by: MA-31 (A13 review). When a child's outcome is too large to record, the parent's row is the typed result-too-large failure on every runtime (O-73: the child ran). On CF Workflows the recordSubSessionMessage callers (ephemeral, remote and respawned children) pass that replaced outcome to core recordSubSessionResult as a failed outcome, which also writes the child's REF as failed (orchestration/record-sub-session-result.ts ~:218-223, the ref status derived from the outcome; runtime-cloudflare/src/steps.ts ~:4545-4550, fitSubSessionRecord's success: false), while the child itself is completed. runtime-js keeps the ref completed (A12: only the parent's row is replaced).
- Where it shows:
getState().subSessionRefsand the usagefailedCallscount. Not in companion refs, thecompanion__*tools or snapshots. - Since: A11's remote cap (
executeRemoteSubAgentCall); A13 widens it to every record throughrecordSubSessionMessage. - At entry: ✅ fixed on runtime-js / the CF DO by the remote settlement MR (WE-2). A finished remote child that a later entry settles before its input settles inside the entry's run-start commit with the child's TRUE status (core
settleRemoteChildrenBeforeInput'ssettle, Invariant S); pinned ascompletedin runtime-jsremote-orphan-reconciliation.test.tsanddo-remote-child-too-large.sqlite.test.ts. CF Workflows no longer settles at entry (RM-69 deleted its entry half: no committed remote call waits on a child there). - Wanted (the record path, still open):
recordSubSessionResulttakes the child's true ref status separately from the parent row's outcome (its remote settle now derives the status from the outcome too), with a cross-runtime test (runtime-js vs CF Workflows ref status for the same oversized child at a record). - Related, under investigation by the conformance owner (no record yet): the respawn path's
completionDeliverededge.
FU-TEMPORAL-CONTINUE-SESSION-UNTYPED-NOT-FOUND: Temporal's continuation read throws an untyped "session not found" — open (parked; owner: the parked Temporal bucket)
Surfaced by: W1 MR2b, filed with FU-JS-LOAD-AGENT-STATE-UNTYPED-NOT-FOUND.
When the continued session's state is missing, runtime-temporal/src/executor.ts (~:1107) throws a plain Error, which a host classifies as an uncoded transient failure. Fix direction: throw a typed state_session_not_found (non-retryable), with a test.
FU-CFW-ENSURE-CONTINUATION-CREATED-VS-ADOPTED: reconcileSession over-reports changed and logs every read — done (W1 MR3)
Surfaced by: W1 MR2b Task 7 review minor 1.
ensureSubmitContinuation reports created for an instance it ADOPTED (the deterministic id already existed), so reconcileSession answers changed: true and logs cfw.chat.continuation_started at info on every chat read while a continuation runs. A committed suspension behind a closed owner also reports changed: true when nothing was written. Harmless (changed means "may have moved"; one extra re-read), but noisy. Fix direction: return adopted separately (create vs get() adoption) and surface the reconciliation's write detail from settleClosedOwner.
Resolution (W1 MR3 D3.2). createInstance returns { instance, origin: 'created' | 'adopted' }; ensureSubmitContinuation answers { kind: 'instance', origin, … }; core's suspended reconciliation outcome carries runStatusWritten, and the reader's settle reports wrote. reconcileSession answers changed: true and logs cfw.chat.continuation_started only for a settle that wrote or a CREATED continuation. Red first in executor-reconcile-session.test.ts (an adopted live continuation; a no-write settle); four existing cases that pinned the over-report now answer changed: false.
FU-CFW-STALLED-CONTINUATION-SETTLE-TRACE: a committed-but-stalled owed continuation whose settle fails — trace it — done (traced; the stall's recovery fixed, W1 MR3)
Surfaced by: W1 MR2b Task 4 review (deferred minor).
An adopted __submit__ continuation that COMMITTED its run start but never started (no run_started) is terminated at the grace bound by failStalledEntry, which settles the run. If that settle fails, the run stays running behind a closed instance until a reader (reconcileSession, getHandle()) settles it. Trace whether the stored answers, already re-stamped to the continuation's run by its run-start commit, are drained by the next entry, or need retry(); pin the answer with a test.
Trace (W1 MR3 D3.3). The stored answers are never lost, and retry() is never the recovery.
- The continuation's run-start commit only re-stamps the answered calls to its run (
planPendingRestamp:adopt,submittedResultkept); its claim,run_startedand entry state (status: 'active', pending kept) follow, and its drain is a laterstep.do. Aresume()waits on it with pre-entry statuspaused, so a stall means the entry state never landed: the session is stillpausedon the re-stamped answers, with no drain. failStalledEntry's settle (owner stopped) therefore planscomplete_suspension(apausedsession that awaits calls counts as a committed suspension), neverfail_uncommitted: the run readssuspended_client_tool, the outcome is the run's own andresume()returns its handle. When that settle's write fails (not_committed) the run staysrunningbehind a closed owner; the next reader (reconcileSession,getHandle()) plans the same, and the session is owed again on the new run:reconcileSessionandresume({ mode: 'continue' })ensure a generation-1 continuation keyed on it, andwith_messagesettles and drains first.retry()refuses thepausedsession.- Fixed (user ruling, 2026-10-12): the stall's message named
retry().RunStartStalledError.recovery: 'retry' | 'resume', set by the executor from the pre-entry status it read: an entry into apausedsession getsresumeand namesexecutor.resume(agent, sessionId, { mode: 'continue' });retryablestaysfalse. - Tests:
cfw-settle-closed-owner-stalled-continuation.test.ts(red first: the recovery; pins: the answers survive a failed settle and the next read re-owes the continuation on the stalled run; a settle that lands completes the suspension and the resume gets its handle). - The re-sent submit that names the dead run meanwhile: FU-CFW-SUBMIT-ANSWERS-DEAD-CONTINUATION-RUN.
FU-CFW-SUBMIT-ANSWERS-DEAD-CONTINUATION-RUN: a re-sent submit answers new_run naming a continuation run whose owner is already closed — open (LOW, self-healing; owner: WB)
Surfaced by: the W1 MR3 trace of FU-CFW-STALLED-CONTINUATION-SETTLE-TRACE.
While a stalled __submit__ continuation's run still reads running behind a closed instance (its stall's settle did not land, and no reader has settled it yet), decideSubmitContinuation (runtime-cloudflare/src/executor.ts) answers running from the run record alone, without probing the owner, so a re-sent submit answers new_run naming the dead run. The handler's attach then gets the non-retryable run_not_live, the client re-reads the snapshot, and that read's reconcileSession settles the run and starts the owed continuation: it heals through re-anchoring, at the cost of one failed attach. Fix direction: probe the owner (cfwOwnerClosure) before answering running for a run whose instance is not this call's own, and settle it as a reader does when it is closed.
FU-DBOS-COMPANION-COMMITTED-RUN-ENTRY-FAILED: DBOS answers child_entry_failed for a companion continuation whose run did commit — open (parked; owner: the parked DBOS bucket)
Surfaced by: W1 MR2b Task 6.
A DBOS companion continuation's run-start event that fails AFTER the run committed carries committedRunId (runtime-dbos/src/steps/run-commit.ts, DbosEntryFailure). The companion path (runtime-dbos/src/steps/execute-companion-tool.ts, the startEvent mapping) answers any failed event { delivered: false, reason: 'child_entry_failed' }, although the message is the committed run's input. Fix direction: a committed run means the message was delivered ({ delivered: true }; the run's own failure reaches the parent through the completion notification), and only a pre-commit failure is child_entry_failed.
FU-CFW-DETACHED-SPAWN-CONTINUE-LOST-ENTRY: a non-blocking spawnAgent continue whose entry loses still answers spawned — done
Closed by: FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES fix round 12 (review r11 I-1). The non-blocking spawnAgent continue now awaits awaitCompanionEntryOutcome in the body like the detached sendMessage (companionEntryOutcomeTimeoutMs, 60 s) and answers { name, status: 'spawned', delivered: true } once its run committed, else { name, status: 'not_spawned', delivered: false, reason, errorDetail? } (a successful tool result). On a definite non-delivery (child_busy, child_terminated, child_entry_failed) the parent's consult map is restored, so a later waitForResult answers the child's real outcome. Tests: workflow.test.ts "round 12 ruling 2", "round 12 (FU-CFW-DETACHED-SPAWN-CONTINUE-LOST-ENTRY)".
The original entry (open, owner WD):
Surfaced by: W1 MR2b Task 6 (concern; pre-existing, no longer worse).
A detached companion__spawnAgent that continues a completed companion on CF Workflows records { name, status: 'spawned' } before the continuation's run-start commit and is not polled (W1 MR2b polls only sendMessage). If the entry loses, the child stays completed and its ref stays completed / delivered (MR2b moved the ref write into the commit, so nothing is re-delivered), but the model was told the spawn started, and a later waitForResult returns the PREVIOUS output. Fix direction: the same body-level entry poll for a detached spawn continue, with a spawn result that reports a lost start.
FU-TOOL-RESULT-UNKNOWN-ERROR-DEFAULT: a failed tool result with no error text is stored as {"error":"Unknown error"} and drops the result's own status — open (LOW, owner: unassigned; surfaced by CP-131)
createToolResultMessage({ success: false }) with no error stores {"error":"Unknown error"} (core/src/orchestration/message-builder.ts:156-166, encodeResult) and discards result. CP-131's cfw run shows the cost: CF Workflows' record-waitfor step built result: { name, status } for a completion whose status was not complete, passed no error, and the parent's row read only {"error":"Unknown error"}. That looks like a redacted message, but it is an invented one, and the real status is lost.
- Question (needs a decision): should a failed result with no error text keep the payload (e.g.
{ error: <typed reason>, result }), or should the builder refuse it so every caller must name a reason? Either way the row should never invent an error string. - Scope: every runtime's tool and companion result paths (core builder, shared).
FU-CFW-FAILED-TOOL-PARTIAL-WRITES-PARITY: CF Workflows stages a thrown tool's partial state writes; runtime-js discards them — fixed in code (tool-stopped MR 2, lands with its merge; DI-89)
Filed by: MA-31 (Task 5, ruling S10 kept the row rule identical). runtime-js stages a tool's writes only when it succeeded (if (!toolError && tracker.hasChanges()) in prepareServerToolCall, runtime-js/src/run-loop.ts), so a tool that threw loses every write it made. CF Workflows' error path (AgentSteps.executeToolCall, runtime-cloudflare/src/steps.ts, the thrown-tool branch) stages them "for partial progress" and writes their state_patch; since MA-31 a size refusal of that write is discarded (logged cf_workflow.error_path_state_write_rejected, no rethrow, no step retry). Both record the tool's own failed row, never the state_write_rejected marker. Temporal and DBOS stage inside the tool's try, so a thrown tool stages nothing. Wanted: one rule across runtimes (discard, or keep partial progress everywhere), with a conformance cell.
Ruled and fixed (2026-11-08): the conformance owner filed it as DI-89 (a retried CFW step applied the partial writes twice), and the orchestrator ruled "a failed tool's writes are discarded on every runtime". The tool-stopped MR 2 deletes CFW's error-path staging (a thrown tool, and a tool that stopped itself on a stop, stage nothing and stream no state_patch; the tracker goes back to the persisted state, so the failure's hooks never read the discarded write). Proven by state.failed-tool-writes-discarded (executor + host) and state.failed-tool-writes-no-state-patch[-host] on every lane, and the CFW unit cfw-tool-stopped-and-failed-writes.test.ts.
FU-CFW-COMPANION-NOTIFY-CLOSED-SPAWNER-STALL: a companion's completion notify retries against a closed spawning instance — fixed in code (W1's notify-gate MR, lands with its merge)
Surfaced by CP-129's design (W1, 2026-10-16). A companion child (a non-blocking spawn or a detached continuation) notifies parentWorkflowId, the instance that spawned it (parentWorkflowId: event.instanceId, runtime-cloudflare/src/workflow.ts:4013 and :4401). It does so through notifyParentCompletion (:5874), whose sendEvent (:5916-5917) runs inside a durable step.do('notify-parent-<status>') (:5934) with the engine's default retries.
- Trigger: the spawning instance closed when its turn ended (CF Workflows runs one instance per call). So
sendEventfails and the step retries before the non-fatal catch, stalling the child's tail. - No harm to delivery: the parent's next-turn notifier syncs the ref from the child's session (
steps.tscheckCompletedChildren). - Not what CP-129's cell hit: that stall was the stream write, and CP-129 part 1 fixes it; the cell is green on cfw-workflows-d1 without this.
- Fix (designed, orchestrator ruling 2026-10-16): a status-aware gate in the notify step. Read the spawning instance's status first. If it is terminal, skip the notify (debug-logged). If it is live,
sendEventwith the normal retries, and on a send failure re-check before retrying, stopping once it is terminal. - Test: a same-turn
waitForResultwith one injected transientsendEventfailure still gets the outcome. - Ships with CP-131 (event addressing to the current waiter), which works in the same code.
Fixed (W1, the notify-gate MR): notifyParentCompletion reads the spawning instance's status first and skips the notify when it is terminal; a failed send re-checks the status and stops once it is terminal; an unreadable status is treated as live (the send is attempted). runtime-cloudflare/src/__tests__/cfw-notify-parent-gate.test.ts: a closed spawner gets no event and no retried send; a live spawner with one transient sendEvent failure still receives it; a spawner that closes mid-send stops after one attempt. It shipped alone: CP-131's consult-first was dropped (CP-131 was not a lag), parked on reliability-cf-w1-parked-later-turn-wait-consult-first.
FU-CFW-NO-BINDING-COMPANION-DELIVERED-PLACEHOLDER: a CF Workflows sendMessage resume / continue with no workflow binding records delivered: true — open (LOW, owner: WD)
Surfaced by: W1 MR2b Task 6 review.
The companion step returns the core { delivered: true } as a pre-poll placeholder with its resume / continue descriptor (runtime-cloudflare/src/steps.ts, "the pre-poll placeholder only"). The workflow body replaces it with the polled outcome only when it has a workflowBinding (companionResult.resume && workflowBinding); without one the generic record step writes the placeholder, so the parent records delivered: true although no child was started. Fix direction: with no binding, record a typed failure (no child can start) instead of the placeholder.
FU-CFW-DEFERRED-COMPANION-CALL-AFTER-RESUME: a CF Workflows companion call deferred behind a same-step client tool never ran (RM-77) — done (W1, with RM-76)
Surfaced by RM-76's design (W1, 2026-10-17); confirmed by the owner's cell companion.spawn-and-finish-deferred-behind-client-tool: red on cfw-workflows-d1 (the spawn and the finishWith beside it both dropped, both tool_use left unpaired, the turn completing only through a later model step), green on js-memory / -redis / -postgres, cf-do host and js-chat-host. A step's companion calls ran only in the step's companion loop, after the sub-agent and remote phases; a phase-1 suspension (a same-step client tool or approval) commits and RETURNS before it, and the resume ran only deferred finishWith calls, through core findDeferredFinishWithCalls, which returned nothing unless EVERY trailing unpaired call was a finishWith. So a deferred companion call, and a finishWith beside it, never ran, and the next model step saw unpaired calls (the C5 heal runs only at a new turn).
- Core:
findDeferredCalls(messages, kindOf)returns the trailing unpaired calls by kind (companion,finishWith), or none when any other kind is unpaired;findDeferredFinishWithCallsis itsfinishWithview. - CF Workflows: the step's companion loop is a workflow-level runner (
runCompanionCalls, moved unchanged). The resume loads the deferred calls (steps.getDeferredCalls, in onestep.do, with ONE attempt id their frames share) and, after a stop check, runs them in the step's phase order: the companion calls through that runner (RM-76's frames included), then thefinishWithcalls. Awith_messageresume records both kinds not executed (unpaired_on_continuation) before the message (ruling 107,recordDeferredNotExecuted); a stop leaves them to the stop's terminal (run_stopped, CP-118). - Not covered: a deferred call of any other kind (a remote sub-agent call beside a client tool) still defers nothing; Temporal and DBOS are parked.
FU-TEMPORAL-DBOS-COMPANION-CALL-FRAMES: Temporal and DBOS write no tool_start / tool_end for a companion call — open (parked; RM-76's other lanes)
RM-76 fixes CF Workflows only. The cell stream.companion-call-announced is also red on temporal ×3 and dbos-postgres, where companion calls write neither frame. Fix direction: the same as CF Workflows. Announce the companion calls before they run, and write each result row's tool_end with core toolEndChunkForResultRow before the row's commit. Parked with the rest of Temporal / DBOS (Cloudflare first).
FU-DO-SUBMIT-POST-WRITE-READ-503: a CF DO submit whose post-write read fails answers a retryable 503 — open (owner: WA)
Surfaced by: W1 MR2b Task 8 review (minor), ruled to pin today's behaviour.
On the DO, runtime-js's submitToolResult runs core resolveSubmitContinuation after storing the answer; a failed re-read there rejects (by design: a re-send decides again) and reaches the DO route's submitFailureResponse as an unclassified error, answered 503 (retryable). A re-send answers already_completed, so it is safe, but it differs from the route's own post-commit read (typed, non-retryable) and from CF Workflows' Q12 wrapping (a typed non-retryable transport_error). Fix direction: one row for "the answer is stored, the continuation is unknown": non-retryable, re-anchor through the snapshot.
FU-DO-TEST-SEAM-STACK-TEXT: the DO drive test worker picks its fault target by stack text — open (LOW, owner: WA)
Surfaced by: W1 MR2b Task 8 review (minor).
runtime-cloudflare/src/__tests__/helpers/do-drive-worker.ts decides which loadState the after_submit fault hits by reading new Error().stack for resolveSubmitContinuation (and /test/hold-wake-read for wakeRequest). A rename, an inlined function or a bundler that drops names silently moves the fault to another read. Fix direction: an explicit injection seam (a read tag passed by the caller, or a wrapper keyed on the call site) instead of stack text.
FU-DO-FRONTEND-GETHANDLE-BY-SESSION: DOFrontendExecutor.getHandle requires an agent it never uses — fixed in code (lands with its MR; owner: W1)
Fixed: DOFrontendExecutor.getHandle(agent | undefined, sessionId) takes an optional agent (the shared signature elsewhere is unchanged). The host's real need, a run-scoped stop, is now a runId on every session stop: core stopSession / SessionStopExecutor.stop, JSAgentExecutor.stop, ai-sdk handleChatStop's body, CloudflareChatHandler.stop (forwarded to the DO's /interrupt / /abort, which already took one) and agent-server's /chat/{id}/interrupt|abort. Core answerNamedRunStop is the one rule (lifted from the DO's handleStop, which now calls it): a live run of the session is stopped; one that ended interrupted / failed answers that terminal with nothing written; anything else is not_stoppable; a foreign run is never stopped. Bounded by core stopRunIdSchema. Proven by units in core, runtime-js, runtime-cloudflare, ai-sdk and agent-server, and the conformance cells stop.run-scoped-stops-named-run[-host], stop.run-scoped-spares-newer-run[-host|-route-host] and stop.run-scoped-foreign-run-refused[-host] (runtime-js lanes, cf-do, js-chat-host). Temporal, DBOS and CF Workflows: FU-NAMED-STOP-OTHER-RUNTIMES.
FU-NAMED-STOP-OTHER-RUNTIMES: Temporal, DBOS and CF Workflows have no session-level stop that names its run — open (owner: post-features runtime phase)
K73 (above) is DO-first: runtime-js and the CF DO implement a session stop that names its run. Temporal, DBOS and CF Workflows executors have no SessionStopExecutor.stop; a host stops through stopSession's handle path, which is correct (the handle is signalled only while the named run is current, checked before and after the bind; a named run that is not live is answered by answerNamedRunStop) but unproven on those runtimes. Fix direction: each executor's own session stop over core stopRun with target: { kind: 'run', runId } and answerNamedRunStop, and the conformance lane claims 'session-stop', which flips the executor cells' declared gaps (stop.run-scoped-* on temporal-* / dbos-postgres / cfw-workflows-d1).
The original report:
Requested by an app's SDK session (2026-11-08). runtime-cloudflare/src/do/do-clients.ts:1653 (origin/main 0ad274bf6b) getHandle(_agent, sessionId) ignores the agent, so a host doing a run-scoped interrupt passes undefined as never. The other executors use it (runtime-js js-agent-executor.ts:6235 re-arms the client-tool deadline with it; CF Workflows executor.ts:3205 resolves the live instance by agent.name), so the shared signature stays: on DOFrontendExecutor make the agent optional or add getHandleForSession(sessionId), and/or give the chat handler a run-scoped stop({ sessionId, runId, reason }), which is the host's real need. A unit, no conformance cell.
FU-CONSUMED-PENDING-CHANGED-ANSWER-SPLIT: consumed_pending_changed is retryable on the native DO route but a 409 in the chat handler — open (LOW, owner: WA)
Surfaced by: W1 MR2b Task 8 merge ruling 1 and its review.
Core classifyEntryFailure classifies RunStartRejectedError('consumed_pending_changed') transient (a DO wake re-arms), so a native DO request answers it 503 (retryable, cause kept, nothing written). The DO clients rebuild the RunStartRejectedError from code + cause, and handleChatStream answers every RunStartRejectedError 409 state_run_start_conflict with the cause, so a chat client sees a non-retryable-looking 409 for the same condition. Both are safe (nothing was written; a re-send re-plans). Fix direction: one answer for the cause on both routes (the retryable row). Chat row: coordinate with WC (its chat contract).
FU-AISDK-ATTACH-PIN-OWNER-SESSION: on DBOS an attach for a sub-agent-owned call waited for the root's run — done (W1 MR2b matrix-2 triage, ruling 2)
Surfaced by: W1 MR2b final review (I-3).
DBOS answered the removed run-pinned kind with the OWNER session's current run, and the handler's attach pin (decideAttach, ai-sdk/src/handler/handle-chat-stream.ts) skipped the ownership check for it, so a call a sub-agent owned waited for the root stream and, behind a paused root, answered the typed live-wait 500. DBOS now answers new_run (its in-workflow continuation run), and decideAttach applies the ownership check to every attach: an answered call the run before the submits does not own re-anchors at once (answered_calls_not_owned_by_run_before). DBOS integration confirmation: client-tool-subagent-ownership-dbos.integ.test.ts and refresh-c1-dbos.integ.test.ts (12/12) are green in MR2b's final matrix.
FU-DO-LOCAL-ALREADY-COMPLETED-BEHIND-EXECUTING-WAKE: a local already_completed keeps caller_resumes while the first submit's wake still runs — open (LOW, owner: WA)
Surfaced by: W1 MR2b final review (M-9; Task 8 review Minor 3).
The DO submit route (runtime-cloudflare/src/do/durable-object-agent-base.ts:2730-2731) answers a LOCAL already_completed with the router's caller_resumes even while the first submit's wake is still executing. The handler's /resume then gets the non-retryable 409 state_already_running and the client re-anchors, so WA R68's re-send of a 503'd forwarded submit degrades to a re-anchor instead of an attach. Fix direction: call readSubmitContinuationAfterWake({ outcome: null, executing }) on the local already_completed path too; it already answers deferred behind an executing drive.
FU-JS-ABANDON-PENDING-AFTER-COMMIT-FROM-MEMORY: runtime-js needs_drain writes its abandonPending answers after the run-start commit, from memory — open (LOW, owner: WA)
Surfaced by: W1 MR2b final review (M-10).
On the needs_drain path (runtime-js/src/js-agent-executor.ts:4158-4174) the abandonPending entry stores its client_tool_abandoned answers after the run-start commit, from the call's memory. A crash in that window makes recover() drain the unanswered calls as runtime_restarted, not client_tool_abandoned (the message is still delivered from the stash). CF Workflows carries abandonPending in its instance params, so it does not drift. Owner WA: the path runs inside the CF DO (runtime-js is the DO's engine). Its non-DO half — the same path on plain runtime-js — rides along with the same fix. Fix direction: carry abandonPending durably with the entry (the stash or the run record), so recover() answers the same way.
FU-CFW-CHILD-OWNED-SUBMIT-CALLER-RESUMES: a submit to a child-owned call after the parent's cascade answers caller_resumes, not the self-wake — done (CP-160, !583)
Resolution (CP-160, !583): the parent's suspension no longer marks a child that is paused on its own call, so the child-owned batch reads ready to holdsReadyBatch / isBatchReadyOn and a submit through the root answers new_run, the root self-waking as for a root-owned call (§1.3.3's reading). A caller's resume({ mode: 'continue' }) joins that run. Proven by runtime-cloudflare/src/__tests__/cfw-cp160-child-owned-submit.wf-noiso.test.ts (real workerd: red caller_resumes at main, green at !583).
Seen: owner slot s74 (2026-10-19), CP-10's executor cells at main 4980e8a8fb on cfw-workflows-d1. The root is parked on its local sub-agent's pending call, and the answer is submitted to the root. All four shapes answered { status: 'accepted', continuation: { kind: 'caller_resumes' } }. runtime-js answers the same, and DBOS answers new_run (its own self-wake).
Why it is not a finding: the answer is typed and tells the caller to resume, and the chat route does, so the run continues (the cells pass 4/4 after the caller's resume()).
Why it is an FU: W1 MR2b's §1.3.3 self-wake reads as covering a complete batch whatever owns it. The CFW executor's startContinuation comment doesn't limit it to root-owned calls, and executor-submit-continuation.test.ts:134-159 asserts new_run for a CHILD owner. That fixture seeds the child paused, while the parent's cascade rewrites an awaited child to failed / parent_suspended (commitSuspendedStep), and isBatchReadyOn accepts only a paused owner. So the real state answers caller_resumes. W1 to settle §1.3.3's scope: either the self-wake covers the cascaded child state, or the spec and the test are scoped to root-owned batches.
FU-CONF-SUSPENSION-ORDER-CASCADED-CHILDREN: state the suspension-order rule's scope for a cascaded child's frames — open (LOW, docs; owner: the conformance owner)
WE-2's CP-186 debug run (2026-10-19, CF Workflows): a parked root's latest checkpoint keeps stream cursor 3 while the head is 12. Its local child's frames, including its tool_approval_request at seq 10, come after the cursor. That is not a defect: the snapshot's resume replays them, and the approval renders (WE-2's CFW e2e, 6/6 at main). But CLAUDE.md's suspension-order rule ("a suspending step's pending-call frames precede its suspension commit so its cursor covers them") reads as if it applied to a cascaded child's frames too. State its scope: the parent's own pending calls are covered by the parent's suspension commit, and a child's frames are delivered by the resume replay.
FU-CFW-SUBMIT-CONTINUATION-TURN-WINDOW: currentSubmitContinuation finds the superseded run by turn - 1 in a 3-run window — open (LOW, owner: WB)
Surfaced by: W1 MR2b final review (M-12).
currentSubmitContinuation (runtime-cloudflare/src/executor.ts:3785-3809) reads listRuns at offset: turn - 3, limit: 3 and takes the run with turn === current.turn - 1 as the superseded run whose __submit__ ids it recomputes. It relies on an implicit invariant: listRuns is ordered by turn, and consecutive run-starts take consecutive turns. A miss degrades a running continuation to "foreign". The with_message wait then returns to its re-plan, and since the final-review fix round (I-2) a resume({ mode: 'continue' }) on a running continuation falls back to the CP-88 wait and the busy loser. Safe (typed), but the invariant is unstated. Fix direction: read the superseded run directly (the run-start's expectCurrentRunId / seenRunId, recorded on the run), or pin the invariant in the store contract.
FU-CFW-RECONCILE-SESSION-PER-REQUEST-PROBE: reconcileSession runs on every chat POST and snapshot — one binding probe per request on CF Workflows — open (LOW, owner: WB)
Surfaced by: W1 MR2b final review (M-14).
handleChatStream (ai-sdk/src/handler/handle-chat-stream.ts:123-153) calls reconcileSession on every POST with a live run, and buildSnapshot (ai-sdk/src/handler/snapshot.ts:116) calls it on every snapshot. On CF Workflows the closed-owner settle probes the instance through the binding, so the cost is one binding call per request. FU-CFW-ENSURE-CONTINUATION-CREATED-VS-ADOPTED covers the logging, not the cost. The cost lives in the CFW executor's probe, so WB owns it; the handler call sites are coordinated with WC. Fix direction: decide in the executor from store reads first (a none plan makes no binding call) and probe only when the run is live with a stale owner, or let the handler skip the call when its own reads show nothing to reconcile.
FU-CFW-OWNER-PROBE-PERMANENT-FAILURE: an owner probe that fails on every read silently strands an owed continuation — open (LOW, owner: WB)
Surfaced by: CP-250 (the companion self-wake), probe p1, 2026-10-30.
attemptClosedOwnerReconciliation (runtime-cloudflare/src/executor.ts:2245-2276) rethrows a permanent probe error (isPermanentReaderError) and otherwise logs cfw.settle.owner_probe_failed at warn and reconciles nothing (M4: a failed read proves nothing; the next reader probes again). That is right for a transient failure. But a probe that fails on EVERY read is never escalated: each read logs one more warn, and the owed continuation (a __submit__ instance, a companion self-wake whose start was lost) is never started. Nothing reaches the caller or an error-level log. Seen in CP-250 probe p1: a test that wrapped the Workflow binding (spread + bind, or a Proxy) made instance.status() throw DataCloneError: ServiceStub cannot be serialized in this context on every call, and 358 reads ×3 never started the owed instance (fixed in the test, which now reads through the real binding: cfw-cp250-companion-self-wake.wf-noiso.test.ts). An app that wraps its binding (tracing, a mock) gets the same silent stall. Fix direction: classify a failure that cannot succeed on retry (a DataCloneError, a binding that is not a Workflow) as permanent, so it is rethrown and logged at error level; and count consecutive probe failures per run, logging at error level past a bound, so a stall that never clears is visible.
FU-CFW-ABANDON-OWED-CONSUMED-RACE-BUSY: abandon-and-send on an owed session whose batch is consumed mid-entry answers busy once — open (LOW, owner: WB)
Surfaced by: W1 MR2b final review (M-16, CFW half; the handler half is FU-AISDK-ATTACH-READ-FAILED-REANCHOR).
with_message + abandonPending on an owed session whose batch is consumed between the owed check and the ensure answers busy once (runtime-cloudflare/src/executor.ts:3016-3046; retryable AgentAlreadyRunningError, nothing written). Fix direction: re-check the owed status after the ensure and drain when it no longer holds.
FU-AISDK-ATTACH-READ-FAILED-REANCHOR: a submit attach whose decision read fails re-anchors instead of answering typed — open (LOW, owner: WC)
Surfaced by: W1 MR2b final review (M-16, handler half; the CFW half is FU-CFW-ABANDON-OWED-CONSUMED-RACE-BUSY).
decideAttach's read_failed re-anchors instead of answering typed (ai-sdk/src/handler/handle-chat-stream.ts:3196-3198, warn-logged). Fix direction: a typed retryable answer on the committed row, decided with WC's chat contract.
FU-CFW-OWED-WAIT-UNSETTLEABLE-REASON: the owed-continuation wait reports "could not settle a closed continuation" under continuation_running — open (LOW, owner: WB)
Surfaced by: W1 MR2b final review fix round (M-6).
When the continuation's instance is proven closed but 3 reader settles of its still-running run change nothing, awaitOwedContinuationSettled (runtime-cloudflare/src/executor.ts) throws RunStartOutcomeUnknownError({ reason: 'continuation_running', ownerStopped: true }), retryable (the message was not written). Core gives that combination its own message, but a caller can only tell it from "still running at the bound" by ownerStopped. Fix direction: a distinct reason (e.g. continuation_unsettled), with agent-server's /resume treating it as not committed like continuation_running, and the docs' reason lists updated.
FU-CFW-POOL-ADAPTER-SELF-WAKE: the CF Workflows workerd-pool test adapter does not model the submit's self-wake — open
Owner: the conformance owner (a harness adapter); WB is consulted if it is rebuilt on the real executor. Surfaced in W1 MR2b's slot, 2026-10-11.
- What:
packages/e2e/src/harness/setup-helpers/cfw-workflows-d1.tsis a hand-rolledAgentExecutorover the workflows pool'sAGENT_WORKFLOWbinding. It never callsCloudflareAgentExecutor: itssubmitToolResultis a store-onlypersistDurableSubmitcopy, and itsgetHandleis unsupported. After W1 MR2b, the real executor's complete-batch submit creates or adopts the run's continuation itself. This adapter cannot model that, so it keepsclientToolContinuation: 'resume'andclientToolSubmitWakesRun: false, and the cfw checkpoint-truth-matrix (helpers/checkpoint-truth-matrix.ts) still drives the explicit-resume path there. Declaring self-wake on it made 3 of 8 matrix cases (client-suspend, approval-suspend, deferred-finish) wait for a wake that never came. - Coverage today: MR2b's self-wake is covered on the real executor by the conformance lane's env (
harness/setup-helpers/cfw-miniflare.ts):smoke.client-tool-completes,submit.partial-batchandsubmit.explicit-resume-after-wakeoncfw-workflows-d1. - Fix direction: rebuild the pool adapter on the real
CloudflareAgentExecutor(as the Miniflare lane did), or retire it once nothing needs the pool row.
Not in WE MR B (2026-10-11): MR B rewrote the CF DO runtime's own readers of a DO's /sse (DI-41, DI-43); this is store-cloudflare's binding-side manager, whose reader mapping the MR B spec (§4) routes to MR C1. The same class remains in DOStreamManagerClient's non-/sse reads (CP-28): getStreamInfo (/snapshot) answers null and the getChunks* reads [] on a failed request or a non-2xx, so a DO outage still reads as "no stream" before the /sse reader (now typed) is opened.
WE MR B (one DO /sse reader) follow-ups
Surfaced while building WE MR B (spec docs/superpowers/specs/2026-10-10-do-sse-unified-reader-design.md; branch reliability-cf-we-mrb-code). Both come from running core's streamManagerContractTests with every read going through DOStreamManagerClient over a real DOAB's /sse (runtime-cloudflare/src/__tests__/integration/do-stream-client-contract.integ.test.ts). Their cases fail on main too (before MR B the reader ended cleanly instead), and they are skipped there with these ids. Routed to WE MR C, the wire work (orchestrator, 2026-10-11).
FU-CONF-RECONNECT-STREAM-READ-DISCRIMINATE: chat.reconnect-stream-info-read-fails does not tell which read failed — open (conformance; owner: the conformance owner)
CP-28's reconnect cell faults the reconnect's first stream-info read, handleChatStream's path 5 getStreamInfo, which is the third /snapshot?messages=none request after loadState and getCurrentRun (skip: 2, GL #178). Since RM-73 the reconnect reads getStreamInfo a second time in path 2b, the run-start window (handle-chat-stream.ts:641 at 62a1095838). That second read is not faulted, so a stream-info read that wrongly answers "no stream" on a failure (CP-28's own bug, reintroduced as a mutation in owner slot s167b, 2026-10-26) is hidden there. The cell passes under that mutation, and also with skip: 0, which faults loadState instead. The read-fault half's exact witness is stream.info-read-failure-rejects-host, which fails exactly .read-rejects under the same mutation.
To close:
- Fault both of the reconnect's stream-info reads:
skip: 2, times: 2. - Disarm right after the first reconnect settles.
cfDoSubmitFaults.endCellfails a fault still armed at the cell's end, and the cell's second reconnect must meet no fault. - Prove it red on target under the reintroduced CP-28 bug, on
.reconnect-not-misreadand/or.no-turn-of-its-own, and green on the fixed product. - If it cannot go red, narrow the cell's doc to "a reconnect over a failing head or stream read is never misread".
FU-DO-SSE-MIDSTREAM-TRUNCATION: the DO's live /sse never tells an attached reader about a cleanup — done (WE MR C1)
cleanupToStep on the DO's DOStreamManager now broadcasts the wire v2 rewound{atStep, atSequence, minSequence?} frame to every attached /sse reader, on every cleanup (as the StreamServer DO does). DOStreamManagerClient's reader reads its connection eagerly (a pump), prunes the buffered chunks the cleanup removed, and throws StreamTruncatedError once it is left behind the head on a live stream; the DO handle's stream() throws it when the rewind is past what it yielded. Contract G4.truncation-throws runs on the DO client (no skip). The post-endStream case stays skipped as the push-SSE divergence FU-G4-BINDING-AFTER-ENDSTREAM-RETURNS-DONE (a closed connection gets no rewound).
FU-DO-SSE-RETENTION-FLOOR-VS-G4: the /sse retention floor answers TRUNCATED where the contract expects the next run's chunks — done (WE MR C1)
Decision 1 (orchestrator, 2026-10-12): the floor is marker-based. The route answers truncated{floor} only while a cleanupToStep marker (truncated_at_step) is set (DOStreamManager.hasTruncationMarker); resetStream and reactivateStream clear it, and a reader from 0 is served the next run's chunks, as the in-DO reader does. The cross- service guarantee holds (a reconnect never silently replays only a cleanup's survivors): a reset is a deliberate new run whose old chunks no reader expects. Contract G4.resetStream-clears-truncation-marker and G4.reactivateStream-clears-truncation-marker run on the DO client (no skips).
FU-CP26-CHAT-EVICTION-E2E-ON-POOL: an end-to-end chat eviction cannot be shown on the workerd test pool — open (test infra; !397)
A test that drives the real chat route (createCloudflareChatHandler → DOStreamManagerClient → the DO's /sse) in the workerd pool and stops reading the response never gets the DO to evict, so it cannot show the CP-26 outcome end to end. Measured (W1, 2026-10-17): the paused client received the whole turn, 1 MB (120 × 8 KiB deltas) and 1.5 MB (1500 × 1 KiB), identically with !397 and with the pre-!397 buffering stream. Two layers absorb the turn before the DO sees an unread client:
DOStreamManagerClient's pump reads eagerly until 1024 chunks are unread (READER_BUFFER_HIGH_WATER; WE C2a adds a byte bound, coreREADER_BUFFER_HIGH_WATER_BYTES, 8 MiB);- the in-isolate DO → worker response pipe took the rest (its bound is not measured).
A bigger turn hits the DO SQLite row limit for the one assistant message (a 3.2 MB reply fails its save with SQLITE_TOOBIG; the owner triages that separately).
!397 is proven by the composed hops instead:
- the DO evicts by abort and releases everything (core
sse-unread-budget, workerdsse-unread-budget.cf); - both Worker-side DO readers re-open on the aborted body and deliver every chunk once, and end with the retryable
StreamDropErrorwhen the DO evicts them again before anything new arrives (runtime-cloudflareevicted-do-body-typed, restated by WE C2a); - the chat route answers a read that ends typed with an
errorevent, never a cleanfinish(ai-sdklive-stream-failure-typed).
The owner's stream.unread-client-evicted-host cell bypasses the first layer (it holds the worker's own read). It still depends on the pipe's bound: size its reply past it, or declare the gap on the pool.
WA MR1 (CF DO drive pipeline) follow-ups
Surfaced while building WA MR1 (spec docs/superpowers/specs/2026-10-08-do-drive-pipeline-design.md, "§0 As built (MR1)"; branch reliability-cf-wa-do-drive). Line pins marked "origin/main" are at 137b07d57a, the MR's base; unmarked pins are branch code. Each item says why it is not in MR1.
FU-FLAKE-CP52-RACE-HELD-WAKES-PREDICATE: the CP-52 race case's held-wake count can reach 2 — done (closed by WA MR1, 7ff0da1a77)
Seen on main: pipeline 2934653889 (f12eef982d), test:integ:rest job 17087287661 (do-nyc3). runtime-cloudflare do-run-guards.integ.test.ts, "CP-52 race (deterministic): a /resume fired while the settle-time continuation is held → exactly one continuation", timed out after 22.9 s in waitFor(settle-continuation-held), the run paused with c1 pending. Mechanism: the test worker's wake hold (do-run-guards-worker.ts) counts every ensureExecutionContinues caller. About 1 s after the settle, the unconsumed-submit alarm's recoverStrandedExecution passes its guards (the held settle caller has not taken the producer claim) and enters the hold too, so heldWakes goes from 1 to 2. A slow runner whose polls miss the window at 1 never matches heldWakes === 1. Nothing is lost: both wakes are parked in the hold. Closed by WA MR1: the predicate is heldWakes >= 1, and the exactly-once assertions are unchanged. RED: the old predicate with a 1.5 s first-poll delay fails like CI (heldWakes 2). GREEN: the new one passes with the same delay.
FU-AISDK-FAILED-TOOL-LIVE-RELOAD-STATE: a failed tool renders as available output live but as an error after a reload — open (owner: ai-sdk; fix routed to W1, client-tier cell to the conformance owner)
Found: MR B (!392), while fixing the live tool_end frame of a companion error row (2026-10-17). It's older than MR B and applies to EVERY failed tool, not only companions.
- Live:
transformToolEnd(packages/ai-sdk/src/stream-transformer.ts, ~:1054) turns a failedtool_endintotool-output-availablewithoutput: { error, errorCode? }. So auseChatclient shows the call as finished, with an output object. - Reload:
convertToolCallToPart(packages/ai-sdk/src/converter/helix-to-aisdk-converter.ts, ~:257) renders the same committed failed row asoutput-errorwitherrorText. - Effect: the same call is in a different UI state before and after a refresh or snapshot reload, so a client that renders errors differently from outputs flips on reload.
- To do:
- Pick one shape for a failed tool and make the live path and the reload path both emit it.
tool-output-errorwitherrorTextis the natural one, sinceai6.x declares it. - Keep DI-20's strict-schema contract test green for every pinned
aiversion. - Add a client-tier conformance cell: a failed tool's part state, live, equals its state after a reload.
- Pick one shape for a failed tool and make the live path and the reload path both emit it.
FU-AISDK-SEED-RESUBMITS-CONSUMED-TOOL-OUTPUT: a snapshot-seeded client re-submits an answered client tool — fixed in code (W1's CP-107 MR, lands with its merge)
In the C-1 "refresh mid hitl-continuation" cells (packages/e2e/src/helpers/refresh-char-exact-scenario.ts), client A answers the client tool tc-ask-1 and its continuation consumes the result. A whole-snapshot (snapshot-seed) client B that refreshes mid-continuation then POSTs that output again, and the server answers a 200 SSE holding data-resume-rejected { reason: 'unknown-tool-call', toolCallId: 'tc-ask-1' }. The render stays char-exact, so the cells pass. Seen on ai 6.0.49, lanes js-memory / js-redis / js-postgres / cfw-memory / cfw-d1. The seeded client treats an already-answered client-tool part as un-submitted (the seeding and the sendAutomaticallyWhen predicate, useHelixChat). Not in MR1: pre-existing client-side behaviour outside the DO drive pipeline. MR1's C-1 settle wait fails on data-resume-rejected (f4dd1ac1ae) only in the control and pending-client-tool cases; the refresh-mid cells leave it off until this is fixed. Finding: CP-107 (a different mechanism from CP-53); its client-tier cell is owner batch 19.
Fixed (W1, CP-107): a whole-snapshot seed is server truth (a client-tool part is terminal there only once its result row is committed), so its answers are history. useHelixChat keeps the answers in the messages it creates the Chat with out of whichever predicate decides (the app's own or the Helix default), beside the replayed ones; a pre-built Chat passes createHelixSendAutomaticallyWhen({ history: <seed messages> }). The hook's own resync re-anchors already used the replay seed. The refresh-mid cells now assert expectNoResumeRejected.
FU-AISDK-MIXED-BATCH-RESENDS-CONSUMED-OUTPUT: a new client-tool answer carries the trailing message's consumed ones with it — open (owner: WC, the chat delivery contract; unowned for now)
The predicate decides only WHETHER to send. When one trailing assistant message holds a consumed answer and a NEW one (the continuation calls a second client tool in the same response message), the predicate rightly fires for the new one, but the request still carries the consumed one: DefaultChatTransport posts every message, and extractResumeIntent refuses the consumed part unknown-tool-call (handler/extract-resume-intent.ts:321-323), so the client reads data-resume-rejected beside the accepted answer. HelixChatTransport.extractToolOutputs re-submits every terminal output of the last assistant message too (transport/helix-chat-transport.ts:304-318, :396-429). Older than CP-107, and the C-1 seed has it as well. Fix direction: "already answered, skip silently" — the server treats a submitted part whose call is already answered in the history as consumed (no rejection), or the transport sends only the parts the client answered since its seed.
Narrowed by CP-172 (the submit response contract): a re-sent answer whose call holds a completed marker is no longer rejected: it is submitted, core answers already_completed / consumed, and a consumed answer never replaces the continuation the new answer decided. What stays open: a consumed answer past the marker's retention (completedTombstoneRetentionMs) is still unknown-tool-call, and HelixChatTransport.extractToolOutputs still re-submits every terminal output.
Narrowed by !944: HelixChatTransport and its extractToolOutputs are removed, so only a stock client's re-sent consumed answer remains: one past the marker's retention is still unknown-tool-call.
FU-AISDK-SUBMIT-AWAITING-CLIENT-ACTION: after a reload, a client answer beside a pending approval re-sends forever — done
Closed by: the CP-180 MR, with CP-182's fix instead of a case B2. The conformance owner's reload-path cell client.stock-autosend-no-repost-loop reproduced the loop on main (s59: 19 / 8 identical answer POSTs in 5 s on cf-do / js-chat-host). Its cause was the reloaded approval part's providerExecuted: true (the persisted call; live, a server part is flagged only with its output), which the stock tool-calls predicate skips. The snapshot now renders approval parts unflagged, as live, so the client's answer waits for the approval; the same flag made ai 6.0.0 to 6.0.91's approval predicate never send a reloaded approval (CP-182). The planned B2 (a 409 awaiting_client_action) is not built: no stock predicate reaches it any more. It is parked, unpushed, on W1's local branch reliability-cf-w1-cp180-b2-backup in case a custom-sender case is ever found.
FU-AISDK-PREBUILT-CHAT-REANCHOR-RESENDS: a pre-built Chat that useHelixChat's resync re-anchors re-sends the snapshot's answers — open (owner: WC; unowned for now)
A pre-built Chat is not replay-capable, so useHelixChat's resync re-anchors it on the WHOLE snapshot (client/resync.ts reanchorNow, setMessages(resume.messages)), but the hook cannot reach that Chat's predicate: the re-anchored answers are not history for it, and the resumed stream's end POSTs them again (CP-107's mechanism). The mount seed is covered (createHelixSendAutomaticallyWhen({ history })); a re-anchor is not. Fix direction: the factory's predicate carries a markHistory(messages) method, and the hook calls it when it re-anchors a pre-built Chat whose predicate has it.
FU-AINEWEST-NO-POST-STOP-REANCHOR: useHelixChat on ai 6.0.314 makes no snapshot re-anchor after a stop — open (owner: WE-2 / client)
The owner's batch-16 diagnosis run (2026-11-08, its logged snapshot requests) shows, after a page stops a run with a sibling tool still held: on the floor pair (ai 6.0.170 / @ai-sdk/react 3.0.172) the page's useHelixChat makes a snapshot read through fetchSnapshotWithRetry right after the stop (its re-anchor); on the newest pair (6.0.314 / 3.0.317) it makes none. The finished tool result is still right after a reload on both pairs (stop.finished-shown-on-reload-client .finished-in-snapshot / .finished-output-shown pass on cf-do+ai-newest with the reload's own read held). Check whether anything else relies on that post-stop re-anchor on the newest ai without a reload: the page's run status after the stop, a pending approval's state, a call cut mid-arguments by the stop (left input-streaming locally: the hook's own stop() aborts its read before step_discarded arrives, CP-12; WE-2, RM-125 build, 2026-11-09), the stopped turn's marks (RM-122), and the re-attach controller's report. Likely cause: ai 6.0.314 reports an aborted stream as status ready, not error, so the hook's error-driven re-anchor (use-reanchor-on-error) never fires. A client cell on both pairs that judges the page's state right after a stop, without a reload, would settle it.
FU-AISDK-PREBUILT-CHAT-ONLATEANSWER-HANDLE: route onLateAnswer for a pre-built Chat through createHelixOnData — open (owner: WE-2; after !553 and #177 MR 1 land)
Filed by: #177 MR 1 (WE-2; orchestrator ruling 2026-10-27). #177 MR 1 adds createHelixOnData, an explicit handle a pre-built Chat is built with (new Chat({ onData: helix.onData }), useHelixChat({ chat, helixData: helix })), so the hook learns a failed request's code. !553 (SDK-5) ships a dev warning and docs for the same gap with onLateAnswer. Wanted: once both land, deliver onLateAnswer's signal for a pre-built Chat through the same handle, so there is one mechanism, not two (a test with a pre-built Chat; drop the warning once it works).
FU-DO-SUBMIT-ADMISSION-HOOK: no admission hook sees a client-tool submit on the DO — open (design question, owner: WA MR3)
beforeEntry runs only for request drives (/start, /resume, /retry, /subagent/:type/start|resume). A /submit-tool-result stores caller-supplied tool results and approval decisions, and the continue drive it triggers puts them in front of the model; no host hook admits it (ruling R59). Today a host must authorise submits in the worker in front of the DO. Not in MR1: the spec scopes admission to request drives; whether submits get their own hook (and with what context: the call ids, the result, the approval) is a design decision.
FU-DO-INTERRUPT-PREVIOUS-RUN-ID: a keyed send that interrupts a live run (concurrentEntry 'interrupt_previous') never tells the host which run it stopped — fixed in code (lands with its MR; owner: W1)
Fixed: the DO drive's interrupt_previous stop keeps the run its stop answered (stopping / interrupted / failed) and names it as AfterEntryContext.interruptedRunId, the native started answer's interruptedRunId, the DOFrontendExecutor handle's interruptedRunId (core AgentExecutionHandle.interruptedRunId, never on an attached handle) and handleChatStream's X-Helix-Interrupted-Run response header. None of them is set when the entry interrupted nothing. ai-sdk exports HELIX_INTERRUPTED_RUN_HEADER and HELIX_CHAT_EXPOSED_RESPONSE_HEADERS for an app's Access-Control-Expose-Headers (the handler sets no CORS headers). Proven by units in runtime-cloudflare and ai-sdk and the conformance ids chat.keyed-send-interrupts-live-run.names-interrupted-run-header, .names-interrupted-run-hook, .idle-entry-names-none (cf-do host) and client.keyed-send-interrupts-live-run.header-reaches-client (cf-do client).
The original report:
Requested by an app's SDK session (through the orchestrator, 2026-11-08). The DO's AfterEntryContext (runtime-cloudflare/src/do/drive/host.ts, origin/main 0ad274bf6b) carries the entry's own run only, and no handleChat send response names the run the send interrupted; the host can read a /status before the send, which races a run that starts between that read and the entry, or match onRunSettled's interrupted run, which is not tied to the send. Only the CF DO lets a send interrupt a live run (ai-sdk handle-chat-stream.ts:2437-2440: every other runtime refuses such a send), so the fix is DO-side: interruptedRunId on AfterEntryContext (absent when nothing was interrupted) and on the keyed send's response (an X-Helix-Interrupted-Run header). Cells: cf-do host, a keyed send over a live run under interrupt_previous names the stopped run in both places, and a send that interrupted nothing names none.
FU-CFW-CLIENT-TOOL-DEADLINE-NO-TIMER: on CF Workflows a client-tool deadline never fires without an inbound request — fixed in code by CP-138's CF Workflows MR (lands with its MR; owner: WE-2)
Fixed (CP-138, CF Workflows half): every client-tool suspension now starts one __deadline__ watcher instance (client-tool-deadline.ts, created in suspendRun). It sleeps until the run's earliest unanswered deadline, expires the lapsed calls client_tool_timeout (version-pinned, fenced on the run, first writer against a submit wins) and, once the batch is complete on the still-suspended run (whoever answered it, so a retried expiry whose reply was lost still continues), runs the owed continuation (CloudflareAgentExecutor.continueOwedSession, the path a submit and a chat read take; the watcher's call rethrows a transient failure so its step retries). A chat read (reconcileSession) runs the same expiry for a suspension behind a closed owner, so a watcher that was never created still heals on the next read. Proven by client-tool-deadline.test.ts, cfw-client-tool-deadline-wiring.test.ts and the conformance cell clienttool.deadline-advances-unattended on cfw-workflows-d1. The Temporal half stays parked, not closed: Temporal is the same class (the workflow returns suspended_client_tool and closes; expiry is the resume-time drain), tracked with plain runtime-js under FU-JS-CLIENT-TOOL-DEADLINE-NO-TIMER.
The original report:
Same class as S-A, which fixed the DO. A client-tool suspension closes the CF Workflows instance: the helper says the drain is driven by executor.resume() (client-tool-workflow-helper.ts ~:521-524), and deadlineAt is written at ~:629-656. The only continuation gate, holdsReadyBatch / isBatchReadyOn (executor.ts ~:4310-4331), is core isSubmittedBatchReadyAtLastSubmit, which needs at least one submittedAt, so a batch nobody answered is never owed. reconcileSession (~:3145) runs only from a chat read. The ParkClock / waitForEvent timeouts are for sub-agent parks, not client tools. A run whose browser left therefore stays paused until a request arrives. Fix direction (orchestrator ruling Q3): a separate small design, with a __deadline__ instance created at suspension that step.sleepUntil(min deadlineAt), then runs the expiry and ensureSubmitContinuation. Red test: suspend with a short timeout, make no request, and assert the run continues.
FU-CFW-CP197-DRAIN-LAPSE-UNIT-WITNESS: CF Workflows' drain over a lapsed client-tool call is unexercised by any conformance lane since CP-138 — done (!490, CP-173 + CP-174; owner ruling 2026-10-25)
Done: the witness landed with CP-197's CF Workflows fix in !490. It is runtime-cloudflare/src/__tests__/steps-applyResultsAndReload.test.ts "CP-173: a stop recorded before the drain — the lapsed call is NOT answered client_tool_timeout and stays pending; the submitted answer is recorded" (with the fix, f1782340f2), plus its sibling "CP-173 (fix round 1, I2): a stop that lands once the drain has started …" (with 1c4a25a31b). Both drive applyResultsAndReload directly, with no __deadline__ watcher, so the stop wins: no client_tool_timeout row and the call is left pending. CP-197's record cites them as its CF Workflows proof (owner ruling, 2026-10-25), because the cfw-workflows-d1 cell stays a declared gap.
CP-138's __deadline__ watcher (above) now expires a lapsed client-tool call of a suspended CF Workflows run before any continuation can meet it, and a continue issued while the call is still pending makes no run-start commit (owner slot s121). So the CF Workflows drain that records a lapsed deadline as completed work even after a stop (CP-197: applyResultsAndReload, recordsCompletedWork) is reached only when the watcher never expired the call: its create failed, or its rounds (DEADLINE_WATCH_ROUNDS) ran out before a reader's backstop did. No suite control reaches that, so stop.drain-lapse-after-stop is a declared gap on cfw-workflows-d1 (capability lapsed-call-stays-pending) and CP-197's ledger no longer has a CF Workflows row. Needed: a runtime-cloudflare unit witness that drives the drain over a lapsed, unexpired call after a recorded stop (no watcher) and asserts the stop wins (no client_tool_timeout row, the call left for the stop), red before CP-197's CF Workflows fix and green after it; it lands with that fix.
FU-CFW-WITH-MESSAGE-VS-DEADLINE-WATCHER: a with_message racing the CP-138 deadline watcher's continuation is refused 409 — open (decide queue-vs-refuse; the orchestrator raises it with the user, 2026-10-25)
Since CP-138, a client-tool deadline that lapses on a suspended CF Workflows run (or a batch completed while it is still suspended) makes the run's __deadline__ watcher continue the root on its own (continueOwed → ensureSubmitContinuation → a __submit__ instance). A plain resume({ mode: 'with_message' }) that arrives while that continuation's run is live is refused state_already_running (409, refuseIfOwnerLive): nothing is lost and the client may re-send, but under a stock useChat client the send can look dropped. CP-104 ruling 1 assumed a lapsed deadline owes no continuation, so it only holds when the entry beats the watcher; its test now holds the watcher (testWorkflowInjection.setDeadlineWatchBody), as do R47, C3 and hitl-cycle's TIMEOUT. Needed: a ruling: queue the message onto the watcher's continuation (or let the entry supersede it), or keep the typed refusal and make the clients re-send it. No test pins the 409 until then.
FU-JS-SELF-ROOTED-REGISTRATION-UNFENCED: fence the self-rooted client-tool registration on the run (the stale-worker ghost entry) — open (low priority)
Surfaced by MR D's D2 (Greptile on !620, 2026-10-31). runtime-js's JsClientToolResolver.registerPending for a session that is its own root writes the call's ownership through the field-scoped op and then saves the pending entry, both UNFENCED. That is parity with the code before MR D (its combined pending + ownership saveState was unfenced too), so nothing regressed. But a worker whose run was superseded while it registered can still leave a ghost pending entry and ownership entry for a call the new run never issued.
The fix: pass the registering run as the op's { fenceRunId } (and fence the pending save), so a superseded worker's registration writes nothing. Many resolver unit tests register without a run record, so they need a started run (startTestRun) first. Every other self-rooted ownership write by an executing run is fenced since MR D (the clear, the suspension's own entries, a rewind's own map; Temporal and DBOS likewise).
FU-JS-CLIENT-TOOL-DEADLINE-NO-TIMER: plain runtime-js / agent-server expire a client-tool deadline only when a resume arrives — done in process (CP-138, runtime-js half)
Resolved (WE-2, 2026-10-26) for a live process. JSAgentExecutor now runs an in-process deadline timer (runtime-js/src/client-tool-deadline-timer.ts). It is the runtime-js counterpart of the CF DO's client-tool-deadline alarm (S-A):
- It arms at a suspension, at core
nextClientToolDeadlineAt. Both rules (nextClientToolDeadlineAtandisSubmitContinuationOwed) moved from runtime-cloudflare'swake-recovery.tsto coreclient-tool/deadline.ts. - It re-arms when the session is touched (
getHandle,submitToolResultwith anagentRegistry). - At the deadline, the executor's wake expires the lapsed calls through the run-fenced, stop-guarded
expireOverdueand continues an owed batch with an ordinaryresume(). - The CF DO turns it off (
clientToolDeadlineTimer: false), since its own alarm fires instead.
Tests: runtime-js/src/__tests__/client-tool-deadline-timer.test.ts. What stays open is a process that exits or freezes after the suspension: FU-JS-CLIENT-TOOL-DEADLINE-DURABLE.
FU-JS-CLIENT-TOOL-DEADLINE-DURABLE: runtime-js's deadline timer does not survive the process — open (design: an app-supplied scheduler + executor.wake)
The CP-138 in-process timer fires only while the process that armed it lives. A serverless or scale-to-zero agent-server host, or a process that exits after the suspension and is never touched again, leaves the run suspended until the next request reaches the executor. The limit is documented in docs/reference/agent-server.md and docs/runtimes/js.md.
Direction (design session, with the user): one optional app-supplied scheduler on the executor (scheduler: { schedule(sessionId, atMs), cancel?(sessionId) }) and one entry, executor.wake(agent, sessionId), that runs whatever is due for a session: deadline expiry, owed stops, re-drives. The executor's private wakeForClientToolDeadline is shaped to become that entry's first job.
FU-JS-CLIENT-TOOL-DEADLINE-SUBAGENT: a local sub-agent's own client-tool deadline never fires on runtime-js — done (!939, CP-327) (Greptile on !536)
Fixed on runtime-js by !939: the tree root is armed at its in-process children's deadlines and its wake is decided as a submit is (CP-327).
CP-138's in-process timer arms only ROOT sessions (armClientToolDeadline skips a session with parentSessionId).
- A local sub-agent suspended on its own client-tool call keeps that call, and its deadline, in the CHILD session's
pendingClientToolCalls. - The parent's timer reads only the parent's pending map.
- So neither fires, and the tree stays paused until a request reaches the executor.
- This is not a regression (nothing fired before CP-138 either). CP-138's JS record says "fixed for root runs".
Direction: the parent's arm also schedules its awaited children's earliest deadline, and its wake advances the tree: expire the child's lapsed call, then continue the child through the parent's continuation path (the way a submit to a child's call continues the tree). Tests: a parent awaiting a local sub-agent that suspends on a 5-minute client call, the deadline lapsed, and nothing else touches the session. The tree advances once, with the child's call answered client_tool_timeout.
FU-TEMPORAL-CLIENT-TOOL-DEADLINE-SUBAGENT: a Temporal sub-agent's own client-call deadline never fires — promoted to CP-341 (2026-11-20)
Promoted (2026-11-20): the b35 cells MR's clienttool.deadline-advances-unattended-subagent measured it red on the three Temporal lanes (!952's MR pipeline 3092503096); CP-341 holds it, ledgered there. The text below is kept as written.
Owner: WA (the parked Temporal bucket). Tracks: catalogue record CP-138 (its child half). Surfaced by: CP-138's Temporal MR ("Root runs only: a sub-agent suspended on its own client call still expires at its next resume").
Problem: on Temporal only a flagged ROOT execution arms the client-tool deadline watcher (armsClientToolDeadline, runtime-temporal/src/client-tool-deadline-watch.ts; the executor sets the flag only for a root input). A sub-agent child suspended on its own client-tool call arms nothing, so its call lapses only when a later resume reaches its drain, as every run did before CP-138. runtime-js's half is CP-327 (fixed by !939).
Proof (when fixed): a conformance cell with a child suspended on its own client call that advances at the call's deadline with no request, red on the Temporal lanes first.
FU-FLAKE-TEMPORAL-CP138-WATCHER-SLEEP-WAIT: CP-138 (c) waits for a deadline-watcher timer that may never start — open (test only; owner: WA)
Filed: 2026-11-17 by the conformance owner, at the orchestrator's request. Related: CP-138.
- The test:
runtime-temporal/src/__tests__/integration/client-tool-deadline.integ.test.ts, "(c) a stop before the deadline: the watcher ends and the run never continues" (:346at origin/main 07819d59eb). It fails inwatcherSleeping(:249), which waits throughuntil(:168) for the watcher's history to hold a started timer, bounded atSLACK_MS + DEADLINE_MS(16.5 s). - Sighting: !921 pipeline 3081663421, job 18127596062 (
test:integ:rest, allowed to fail): "timed out waiting for the deadline watcher to fall asleep", 19497 ms. The file's other six cases, (a), (b) and (d)-(g), passed. !921 changes one other Temporal test only. - Candidate cause (by reading, unverified): (c) uses
DeadlineAgent, whose deadline isDEADLINE_MS= 1.5 s (:39). The watcher (client-tool-deadline-watch.ts:198-207) sleeps only when its firstclientToolDeadlineCheckanswerswait. If that check runs after the deadline on a loaded runner, it expires the call and starts the continuation with no timer. The wait then never ends, and the stop lands after the deadline, so the case no longer tests a stop during the sleep. - Fix direction: give (c) a deadline well past the watcher's start, as (b)'s
SlowDeadlineAgentdoes (6 s,:80-86), or have the wait fail at once with a named precondition once the deadline passes with no timer started. - Not seen again (2026-11-18): the test file and the watcher are unchanged through origin/main fae53b26ac. Of the 20 other
test:integ:restjobs of pipelines updated since 2026-11-17T21:00Z, none failed this file. - CI: until this lands, a recurrence is a disclosed retry under this entry.
FU-AGENT-ALREADY-RUNNING-NOT-HELIXERROR: AgentAlreadyRunningError is a plain Error, not a typed HelixError — open (fold into core 2.0.0)
Surfaced by: CP-138 (!536). A late resume() that meets the deadline timer's live continuation is refused with it, and its upgrade note has to tell consumers to match the class or name.
- The class:
core/src/errors/agent-errors.ts.AgentAlreadyRunningError extends Errorand has nocodefield. Its subclassRunStartConflictError(errors/agent-errors.ts), the run-start-commit loser, keeps the samenameand addscode: 'state_run_start_conflict'. - The mapping: core
classifyError(errors/classify-error.ts) turns aninstanceof AgentAlreadyRunningError(the subclass included) into aHelixErrorwithcode: 'state_already_running', non-retryable. So hosts that answer throughtypedErrorResponsealready send the typed 409, but a direct caller gets an untypedError. - Who throws it:
new AgentAlreadyRunningError(appears 30 times outside tests:- core:
client-tool/with-message-entry.ts,orchestration/remote-children-before-input.ts,store/checkpoint-ops.ts; - runtime-js:
js-agent-executor.ts; - runtime-cloudflare, runtime-dbos, runtime-temporal, ai-sdk and e2e.
- core:
Fix direction: make it a HelixError (code: 'state_already_running', non-retryable), with RunStartConflictError keeping state_run_start_conflict, and drop the classifyError special case. This is a core API break. Core takes a major in this release cycle (2.0.0, per the orchestrator: !535 / !527), which is the natural place to fold it in.
FU-JS-EXPIRY-FAILURE-RUNTIME-RESTARTED: a failed drain expiry turns a lapsed call into runtime_restarted — open (CP-173 / CP-174 ruling Q7; plain runtime-js)
Surfaced by: CP-173 / CP-174, investigation §2 (2026-10-20).
The resume drain's expireOverdue (runtime-js/src/js-agent-executor.ts drainOrphanClientToolCalls) is best-effort. When it fails (a store error, or version contention beyond the resolver's retries), the failure is only warned, and the drain then answers a lapsed, unanswered call runtime_restarted instead of client_tool_timeout. The answer is less precise, but correct, and nothing loops. On the CF DO, the client-tool-deadline alarm normally expires the call first, and its failures are typed and budgeted (CP-174).
A failure with a stop recorded on the run is handled: the drain reads the stop after a failed expiry and keeps the unanswered calls pending (CP-173 fix round 1, M2). What stays open is the stop-free case.
Fix direction: classify the expiry failure as the drain's own entry failure (transient → the entry's retry), or re-check the lapse when the drain declares the call lost.
FU-TEMPORAL-DBOS-SUBMIT-SAVE-STOP-GUARD: Temporal / DBOS client-tool submits read the stop, then save unguarded — open (parked; CP-173 ruling Q6, the rest)
Surfaced by: CP-173 ruling Q6 (2026-10-20).
runtime-js, the CF DO and CF Workflows now save a submit { fenceRunId, refuseIfStopped: true }, and they answer a refusal run_stopped. Temporal's (runtime-temporal/src/executor.ts, after refuseSubmitIfStopped) and DBOS's submit saves still pass { fenceRunId } only. A stop recorded between the read and the save loses: the answer is stored on a stopped run. That is benign in practice, because the stop's K5 clears the pending map; it is the same window CP-173 closed for the DO.
Fix direction: the same one-line option on each save, plus a re-read on CommitStoppedError.
FU-DO-DEADLINE-CORRUPT-SESSION-ROW-GAVE-UP: a DO deadline over a row it cannot read re-arms every 30 s and leaves the run paused — open (low)
Owner: WA. Surfaced by: the orchestrator's independent review of the CP-173 / CP-174 MR (2026-10-23).
CP-174 ends a run typed when its client-tool deadline cannot be expired, except when the deadline path cannot read one of its own rows. A corrupt session row is the reviewed case:
- The alarm's own read fails.
clientToolDeadlineTarget(runtime-cloudflare/src/do/durable-object-agent-base.ts:2011-2017) callsloadStateSync, and a permanent error there isfatal(core/src/run/classify-entry-failure.ts:106). endRunAtClientToolDeadline(:2250) callsfailCommittedRun({ requireSessionCommit: true })(:2309).failCommittedRunrethrows the permanent load error (core/src/run/fail-committed-run.ts:229).requireSessionCommitturns it intonot_committed(:288-289).not_committed(:2355-2356) is Ruling 34'sgave_up(clientToolDeadlineGaveUp,:2402). It re-arms every 30 s (EXECUTION_WATCHDOG_RETRY_MS,:395and:2422) and writes nothing, so the run stayssuspended_client_tooland its session stayspaused.
client-tool-deadline-alarm.test.ts "CP-174 (Q5) … a permanent one ends at gave_up, 30 s" pins this behaviour.
Two variants end at the same gave_up, but only after the crash-loop budget:
- the DO store's own corrupt-session error,
InvalidCustomStateError(do-state-store.ts:332-339), is not in core'sCORRUPT_ROW_ERROR_NAMES(core/src/errors/agent-errors.ts:1299-1303), so it classifiestransient; - a run row whose status fails its schema (
do-state-store.ts:3575, a rawZodError) classifiestransienttoo, and the terminal's owngetCurrentRun(:2271-2273) gives up on it.
Nothing is lost and nothing loops fast, but the client is never told. The row cannot be written either, so no terminal can land.
Fix direction: answer the stuck run on its own read path, for example a typed refusal to the next request, or end the run through a write that does not load the session. Also decide whether the DO's InvalidCustomStateError should count as permanent.
FU-SUBMIT-LAST-RETRY-STOP-REFUSAL-UNTYPED: a stop refusal on a submit's last retry rethrows raw instead of answering run_stopped — open (low)
Owner: WA. Surfaced by: the orchestrator's independent review of the CP-173 / CP-174 MR (2026-10-23).
CP-173 ruling Q6 stop-guards the submit save, and it answers a refusal run_stopped by re-reading on the next attempt. On the LAST attempt there is no next one, so the CommitStoppedError (state_commit_stopped) reaches the caller raw:
- CF Workflows
persistDurableSubmit: the refusalcontinues (runtime-cloudflare/src/executor.ts:4918-4925), the 5-attempt loop ends, andthrow lastError(:4958) rethrows it; - runtime-js
applySubmitToOwner(and so the CF DO): the retry is gatedattempt < MAX_RETRIES - 1(runtime-js/src/js-agent-executor.ts:6926), so the last attempt falls through tothrow e(:6963).
This is practically unreachable. It needs four version conflicts (or superseded re-reads), and then a stop recorded between the fifth read and its save.
Fix direction: on a CommitStoppedError, re-read the stop and answer through refuseSubmitIfStopped before giving up, whatever the attempt.
FU-CLIENT-TOOL-UNSET-TIMEOUT-NEVER-EXPIRES: with no timeout set, a client-tool call or approval never expires on some runtimes and times out at 5 min on others — open (needs a user ruling)
Surfaced in the S-A review (2026-10-17). When the tool sets no clientToolTimeoutMs, the runtimes disagree about the pending call's deadlineAt, so the same agent behaves differently depending on the runtime it runs on. "Never" is the Number.MAX_SAFE_INTEGER "no deadline" sentinel. "5 min" is DEFAULT_CLIENT_TOOL_TIMEOUT_MS (core/src/constants.ts).
| Runtime | Client tool, no timeout | Approval, no timeout | Cites |
|---|---|---|---|
| runtime-js (and agent-server) | never (tool → agent → never) | never (tool → agent → never) | client runtime-js/src/run-loop.ts:2759-2760; approval core/src/orchestration/step-iterator.ts:1669-1670 |
| CF Durable Object (runs runtime-js) | never | never | the runtime-js cites |
| CF Workflows | 5 min (tool → agent → 5 min) | 5 min (tool → agent → 5 min) | client runtime-cloudflare/src/client-tool-workflow-helper.ts:534-535; approval runtime-cloudflare/src/approval-gate-workflow-helper.ts:278-279 |
| Temporal | 5 min (tool → agent → 5 min) | 5 min (tool → agent → 5 min) | client runtime-temporal/src/activities.ts:3989-3991; approval :3959-3961 |
| DBOS | 5 min (tool → 5 min; the agent's value is not read) | never (tool → never; the agent's value is not read) | client runtime-dbos/src/workflows/shared.ts:668 (runClientToolPrologue) and :761 (dispatchClientToolInSlot); approval :1108 (runApprovalGatePrologue) and :1227 (dispatchApprovalGatedToolInSlot), whose comment says the approval/client split is deliberate |
DEFAULT_CLIENT_TOOL_TIMEOUT_MS is also read on runtime-js's legacy in-memory wait path (runtime-js/src/execution/tool-executor.ts:196), which the suspending run loop does not use. The agent option's own doc says "Falls back to DEFAULT_CLIENT_TOOL_TIMEOUT_MS if unset" (core/src/types/agent.ts:272, left as is pending the ruling), which matches CF Workflows and Temporal only. DBOS also ignores AgentConfig.clientToolTimeoutMs on both paths, which no other runtime does.
Options (user ruling):
- (a) Unify on the 5-min default, as CF Workflows, Temporal and
agent.ts:272do. This changes runtime-js, the CF DO, and DBOS approvals: an unattended call then times out and the run continues. It also makes DBOS read the agent's value. - (b) Unify on never. This changes CF Workflows, Temporal and DBOS client tools, and the docs (
agent.ts:272,tool.ts'sclientToolTimeoutMs, the client-executed tools guide).
Either way DBOS should read the agent's value. docs/runtimes/cloudflare-do.md states the DO's behaviour today (never).
FU-CLIENT-TOOL-TIMEOUT-TWO-CODES: one lapsed deadline is recorded under two different codes — fixed in code (S-B, !409: one code, client_tool_timeout)
Surfaced in the S-A review (2026-10-17). The DO's client-tool-deadline alarm and every resume drain (expireOverdue) write submittedError: 'client_tool_timeout'. handleChatStream's deadline sweep submits error: 'client_tool_deadline_exceeded' for the same event (ai-sdk/src/handler/handle-chat-stream.ts:667). The two are mutually exclusive (whoever sets submittedAt first wins), but the recorded code depends on who got there first, and client_tool_deadline_exceeded is not in CLIENT_TOOL_ERROR_CODES (core/src/client-tool/humanize-error.ts:26-32). So under the sweep's code the LLM gets the generic error text instead of the timeout explanation, drainOrphans labels the outcome inherited (runtime-js/src/client-tool-resolver.ts:968), the timeout counter is skipped (runtime-js/src/js-agent-executor.ts:3501), and Langfuse maps only client_tool_timeout to timeout (tracing-langfuse/src/langfuse-hooks.ts:21). Tests pin each code: e2e/src/__tests__/client-tool-error-code-invariant-js.integ.test.ts:45,148,155 (client_tool_timeout) and e2e/src/__tests__/concurrency-invariants-stress.integ.test.ts:437,466 (client_tool_deadline_exceeded). Ruling (orchestrator, 2026-10-17): one event, one code: client_tool_timeout. Fixed in code by S-B (!409): the sweep now runs only for a request with no new user message and submits client_tool_timeout; the stress test pins that code.
FU-DO-SUBMIT-MATERIALIZES-AGENT: every DO /submit-tool-result resolves and materializes the agent, even one that cannot start a continuation — open (owner: WA)
Surfaced by: S-B (the 2026-10-17 chat incident; S-B-rev2 §2).
handleSubmitToolResult (runtime-cloudflare/src/do/durable-object-agent-base.ts, steps 3-4: resolveAgentForLookup at ~:3408, then materializeAgent "for the continuation the submit would start" at ~:3418) runs for EVERY submit, before executor.submitToolResult: a submit whose entry is already consumed, a duplicate (already_completed), and one that leaves its batch partial (awaiting_batch) each pay two resolver calls, the adapter and the usage store. In the incident 14 submits (7 sweep, 7 abandon) did this in ~3 s of the user's request; each logged the "Rewrote sub-agent tools to DOStubTransport" / "Rewrote persistent-agent companion tools" pair. S-B removed the 14 submits from a send, not the per-submit cost. Fix direction: resolve nothing for a submit that cannot start a continuation — decide from the owning session first (the call is pending and unanswered, and the answer completes the batch per isSubmittedBatchReadyAtLastSubmit) and materialize only then; the output-schema lookup only when the call has a result to validate. A test: a consumed / partial-batch submit calls the resolver zero times, a waking one once.
FU-DO-WIRE-BUSY-REFUSAL-CURRENT-STATUS: a busy refusal rebuilt from the DO wire always says currentStatus: 'running' — open (LOW; owner: WA)
Surfaced by: S-B fix round 3 (2026-10-17), the amended item 6.
peerAnswerError's state_already_running case (runtime-cloudflare/src/do/peer-answer.ts:66) now keeps the DO's refusal text as the rebuilt AgentAlreadyRunningError's message, so the run's status and suspendedRunRefusalHint reach the caller (e.g. a with_message + abandonPending over a session that awaits children, refused by core planWithMessageEntry inside the DO). Its currentStatus is still always 'running': the busy body is exactly { error, code } (core PeerErrorBodySchema, core/src/errors/http-error.ts:53; the DO's busy answer is built at runtime-cloudflare/src/do/drive/wire-answer.ts:84-92), and the status is never read out of the message text. A typed status needs a new body field on every native route's busy answer: a cross-route wire change. No current reader of AgentAlreadyRunningError.currentStatus exists in the handler or client code (the one reader, runtime-cloudflare/src/run-entry.ts:1494, reads an in-process RunStartConflictError, never a wire rebuild). Fix direction: a typed currentStatus on the busy body (core typedErrorBody / PeerErrorBodySchema, the DO's wire-answer.ts), read by peerAnswerError.
FU-BUSY-REFUSAL-RETRYABLE-INCONSISTENT: the same busy refusal is retryable at one seam and not at two others — open (owner: WA, with core)
Surfaced by: S-B fix round 3 (2026-10-17), the runtime-js / runtime-cloudflare changeset wording; predates S-B.
One busy refusal (state_already_running) is classified three ways:
- retryable
true:handleChatStream's HTTP answer,alreadyRunningResponse(packages/ai-sdk/src/handler/handle-chat-stream.ts:3139,retryable: trueat :3147); - non-retryable: a thrown
AgentAlreadyRunningErroras core classifies it (core/src/errors/classify-error.ts:98-103,retryable: false; the class itself has noretryablefield,core/src/errors/agent-errors.ts:11); - non-retryable: the CF DO's native busy body (
runtime-cloudflare/src/do/drive/wire-answer.ts:84-92, aHelixErrorwithretryable: falsethrough coretypedErrorResponse, so the body carries noretryableand is not a 503).
This is classification drift across seams, against the structured-failure contract ("flows unchanged" from classification to every reader). Fix direction: one classification of the busy refusal, from one place (core), that every seam reads — the chat route, the DO's wire answer and classifyError — with a test at each seam.
FU-E2E-TEST-WORKER-UNTYPECHECKED: the e2e workerd test worker is checked by no tsconfig — open (LOW, owner: the next WA MR (not done in WA MR2))
packages/e2e/src/test-worker.ts (the wrangler.cloudflare.toml main) and src/helpers/do-subagent-bridge.ts are in packages/e2e/tsconfig.json's exclude list and in no other tsconfig. An ad hoc tsc shows 2 type errors already on origin/main: PersistentAgentConfig is not an export of the module it imports it from (test-worker.ts:31), and backupR2Binding is not a property of the sandbox provider config (:276). An API change can therefore break the workerd e2e lane with no typecheck failure (the same class as the cross-service workers WA MR1 Task 18 found). Fix direction: a Workers tsconfig like tsconfig.cross-service-workers.json, added to the e2e typecheck script, then fix the two errors. Not in MR1: found in the Task 18 review; the errors predate MR1.
FU-CORE-TYPECHECK-SKIPS-TEST-FILES: core's typecheck never checks its test files — open (owner: parked CI-health backlog)
packages/core/tsconfig.json excludes **/__tests__/**, and core's typecheck script is a bare tsc --noEmit on that tsconfig. CI's typecheck job (.gitlab-ci.yml, turbo run typecheck) runs exactly that script, and vitest transpiles without typechecking, so a core test that imports a renamed or removed export still passes typecheck in CI. Core is the only package whose tsconfig excludes its tests. Example: after W1 MR2a renamed assertRunStatusTransition to decideRunStatusTransition on main (137b07d57a), its new run/__tests__/reconcile-closed-owner-run.test.ts still imported the old name; only an ad hoc test typecheck during WA MR1's rebase caught it. Fix direction: a tsconfig.test.json (or dropping the exclude, with test-only types declared) added to core's typecheck script, then fix whatever it finds. Not in MR1: a CI-wide change, found during the final rebase.
More evidence (WE-2, 2026-10-24, the CP-210 (a) guard MR): orchestration/__tests__/companion-tool-dispatch.test.ts has 11 type errors at main e3f03e8196 (lines 515-518, 654, 4124, 4162, 4215, 5864, 6069 and 6088: TS2493 / TS2339 / TS2322, mostly untyped vi.fn mocks indexed or passed as typed deps). CI never reports them.
FU-TEMPORAL-SAME-STATUS-BRANCH-DEAD: Temporal's "finished by another entry" branch can no longer run — open (LOW, owner: parked Temporal bucket)
runtime-temporal/src/activities.ts:7024 (origin/main) catches RunSupersededError from updateRunStatus and returns when the run already has the same status ("finished by another entry in between", N4). Since WA MR1 (§3.15), a store answers a same-status terminal write with decideRunStatusTransition 'noop' instead of throwing, so the branch is dead. In the narrow race where another entry finishes the run between Temporal's pre-read and its write, the write is now a no-op and Temporal goes on to its stream end, which CP-85's fence drops. Fix direction: drop the branch, or re-check getCurrentRun after the write (as DBOS run-lifecycle.ts does). Not in MR1: Temporal is parked; the behaviour is correct, only the code is dead (ruling R9).
FU-DBOS-SETTLE-DEAD-OWNER-SAME-STATUS-REWRITE: DBOS settleDeadOwnerRun can write a second step_discarded and stream end — open (LOW, owner: DBOS bucket)
runtime-dbos/src/lifecycle/interrupt.ts:397-399 (origin/main) relied on updateRunStatus throwing RunSupersededError for a run already settled with the same status, returning none. Since WA MR1 the same-status write is a no-op, so a second settle of the same dead owner falls through to emitStepDiscarded (:415) and endStream whenever the stream still reads active. Fix direction: read the run's status before the fallback write and return none when it is already terminal (or branch on decideRunStatusTransition). Not in MR1: DBOS is parked (Task 4 review minor).
FU-DO-STREAM-MANAGER-MISSING-STREAM-READS-ACTIVE: DOStreamManager reports a stream that was never created as active — open (LOW, owner: the next WA MR (not done in WA MR2))
DOStreamManager.getStreamInfo (runtime-cloudflare/src/do/do-stream-manager.ts:841, origin/main) never returns null: status: (meta.status ?? 'active') (:922); getStatus() defaults to 'active' (:1033, :1575); the writer's getStatus() defaults to 'active' (:259) and ensureInitialized seeds status = 'active' (:230). The StreamManager interface says getStreamInfo returns null for a stream that does not exist, as memory, Redis and the stream DO binding do. Since R43, "failed session, no stream" is a lasting DO state, so a reader that trusts getStreamInfo / getStatus (the generic snapshot fence, a waitForLive check) sees a live-looking stream for a session that never ran. Fix direction: getStreamInfo returns null with no status row; getStatus() gets an explicit "no stream" result; audit the in-DO callers that rely on the default (the claim path's !info branch, RM-71, the writer's self-seeding); replace the drive test's raw-SQL streamCreated inspection with the public API; add a StreamManager contract scenario "a never-created stream reads null". Not in MR1: a stream-manager contract change across callers, found in the last Task 11 review round.
FU-DO-STREAM-CHUNK-TOOBIG: one stream chunk over SQLite's value cap fails DOStreamManager's write untyped — done (RM-90)
Catalogued as RM-90 and fixed there. Every stream manager declares StreamManager.limits.maxChunkBytes and refuses a chunk over it with core StateSizeExceededError (operation: 'stream_chunk'), measured as it binds, before allocating a sequence. An end is never refused: an oversized part of its terminal payload is omitted and named (omitted), and a remote consumer recovers an omitted output from the child's /status:
- the CF DO:
maxStreamChunkBytes, 1.8 MB; - store-cloudflare's StreamServer: 1.8 MB, answered as a 413 that the binding revives typed, so CF Workflows never retries the step;
- memory and Redis:
Infinityunless configured.
Model deltas are split at the producer (withChunkSplit), so a single multi-megabyte text_delta streams and completes. StreamServer now overrides onException.
FU-RECONSTRUCT-PARTIAL-REASONING-DOUBLED: reconstructPartialContent appends the thinking complete block onto its deltas — open
Surfaced by: WA's RM-90 trace (side finding 3), verified by WE-2 at main 6a2a66817c. Core utils/reconstruct-partial-content.ts (case 'thinking') appends every thinking chunk's content. That includes the isComplete: true block, which re-sends the whole reasoning (the Vercel adapter emits deltas and then the complete block), so a reconstruction of a stream carrying both doubles the reasoning text. No in-repo caller uses it today (it is only exported; snapshots no longer merge partial content), so the impact is on external callers. To close: append only isComplete: false deltas, and take the complete block's content only when no delta was seen (the same rule as the AI SDK transformer), with a test of both shapes.
FU-REDIS-STREAM-LARGE-CHUNK-PUBSUB-BUFFER: a very large chunk on an unlimited Redis stream can disconnect its subscribers — open
Surfaced by: WA's RM-90 trace. RedisStreamManager declares no chunk limit by default (RM-90: Infinity, so a 2 MB tool result that works today keeps working). But each chunk is also PUBLISHed, and Redis's default pub/sub client output buffer (client-output-buffer-limit pubsub 32mb 8mb 60) disconnects a subscriber that falls behind, so a burst of multi-megabyte chunks can drop live readers (who then re-open from their cursor). To close: measure it under a real Redis, then either document a recommended maxChunkBytes / buffer setting for deployments with large tool results, or publish a pointer for large chunks and let readers fetch the stored value.
FU-HELD-PRIOR-WRITE-UNMEASURED: a held prior's write is not size-measured, so an over-limit value fails untyped — open (owner: WA)
Filed by: the MA-31 merge of main (DO companion MR B). holdPriorResult writes the child's outcome into sub_session_refs.held_prior as JSON.stringify(held) with no size measure: the DO store (runtime-cloudflare/src/do/do-state-store.ts, holdPriorResult) and D1 (store-cloudflare/src/d1-state.ts, holdPriorResult). A value over the store's value cap therefore fails as DOStateError / D1StateError, not as core's typed state_size_exceeded (MA-31's rule for every other bind). Unlikely in practice: the child's own store already bounds the output it held (measured against maxStateBytes, 1.8 MB by default, under the 2 MB SQLite / D1 value cap; parent and child share one store class), so only a hand-set maxStateBytes near 2 MB can reach it. Even then nothing is lost or wedged: the hold is a CAS that throws BEFORE the old child session is deleted or its ref reset, the companion call fails into a typed failed tool row (MA-31 A12) and the run continues, and the ref stays undelivered, so the notifier still delivers the outcome (MA-37).
A related edge: when the answer row that carries the prior is itself too large, MA-31 A12 replaces it with the typed failed row, which no longer carries the prior. A crash between the step commit and the release then makes the next notifier pass inject a (fitted) notice too: the model is told twice, never not at all.
Wanted: measure the bind with the store's BoundSizeWitness (operation state) and refuse it typed, with a test on each store; then decide what a refused hold does to the continue (an answer row without the prior, the notice path).
FU-DO-COMPANION-START-FETCH-UNBOUNDED: the DO companion spawn / continuation /start fetches have no timeout — done (DO companion MR B)
Done by DO companion MR B (spec 2026-10-15 §3.7 "As built (MR B)"). Every /start (a first spawn's, a respawn's, a continue's, the bespoke sendMessage's completed-child continue) goes through one bounded exchange, fetchChildEntry (runtime-cloudflare/src/do/persistent-companion-tools.ts), with CHILD_START_TIMEOUT_MS (60 s, the same as CHILD_RESUME_TIMEOUT_MS; DOCompanionDispatchContext.startTimeoutMs for tests) and the R50 schema, and its answer is mapped by childEntryOutcome onto core CompanionEntryOutcome: the bound's abort is child_start_unconfirmed, the consult KEPT unconfirmed (no ref write, I-C8), and the next same-name spawn RE-PROBES it (reconcileRunningRef: confirmed by a live run of the consult, abandoned on proof it never started, else left; a failed read answers child_start_unconfirmed again without dispatching) — exactly the fix direction below. Pinned by the fake-timer cases in runtime-cloudflare/src/__tests__/do-persistent-agents.test.ts ("a first spawn whose /start hangs is aborted at CHILD_START_TIMEOUT_MS", the continue twin) and the re-probe describe. The original text follows.
Narrowed by DO companion MR A (spec 2026-10-15 §3.6): every /status read of a child is bounded by CHILD_STATUS_TIMEOUT_MS (10 s, runtime-cloudflare/src/do/persistent-companion-tools.ts): every verified read (readChildWord: the park, waitForResult, the in-tool reconcile, the owed reads), sendMessage's pre-send guard read, companion__getChildStatus's informational live read (a timeout falls back to the ref's status) and a replayed spawn's probe (replayProbeFindsNoChild: a timeout attaches; only a 404 means no child). Each has a fake-timer unit test in runtime-cloudflare/src/__tests__/do-persistent-agents.test.ts. Still open: the spawn and continuation /start (MR B, with the unconfirmed re-probe below). The original text follows.
WA MR1 bounds a companion /resume (CHILD_RESUME_TIMEOUT_MS, 60 s → child_start_unconfirmed). The spawn /start, the completed-child continuation /start and the /status polls (runtime-cloudflare/src/do/persistent-companion-tools.ts, fetchJson at :215, stub.fetch at :226, origin/main) still wait without a bound, so a hung child DO hangs the parent's tool call. Not in MR1 (ruling R49): a spawn timeout needs design. A timed-out /start may have started the child, and marking its ref failed (what core does for a failed spawn) lets the next spawn of that name wipe a live session. Fix direction: bound it with an "unconfirmed" ref outcome that the next spawn re-probes instead of respawning.
FU-CORE-RESPAWN-AFTER-TRANSPORT-FAILURE-WIPES-COMPLETED-CHILD: a failed re-consult of a completed companion marks it failed, so the next spawn wipes it — done (DO companion MR B)
Done by DO companion MR B (spec §3.7, I-C8 / I-C10; core handleSpawnAgent's entry table). The catch compensation is gone: a dep that throws is a programming or store error and writes nothing. A non-start is a typed CompanionEntryOutcome, and its ref write is by the entry's kind — a first spawn's (or its same-id replay's) definite non-start writes the ref this call created failed + delivered with the detail in the answer; a respawn's writes the ref back EXACTLY as it was (status, completedAt, completionDelivered), except that a prior this call holds and delivers in its answer is marked delivered (the held slot stays the delivery authority until released); a CONTINUE writes nothing, so the completed ref keeps its session; an unconfirmed start writes nothing (the ref stays running, re-probed by the next spawn). Pinned by core/src/orchestration/__tests__/companion-tool-dispatch.test.ts (the I-C10 / C11 cases: a busy, refused or transport-failed continue keeps completed). The original text follows.
Core handleSpawnAgent (core/src/orchestration/companion-tool-dispatch.ts:564-631, origin/main) runs the re-consult of a COMPLETED child (canContinue → continueFn) inside the same try as a first spawn. When the continuation /start fails in transport (or, since R50, answers a non-conforming 2xx), the catch compensates the ref to failed with completionDelivered: true. The next spawn of that name then takes the delete-and-respawn branch and wipes a child session that may be intact. Fix direction: compensate only a ref this call created; a re-consult that fails leaves the completed ref as it was (the sendMessage re-consult already does). Not in MR1 (ruling R49): core behaviour on every runtime, outside spec §3.9's table.
FU-CORE-REPLAYED-FIRST-SPAWN-SKIPS-USAGE-POINTER: a replayed first spawn that starts never records the parent's usage pointer — ✅ fixed by CP-212 (WE-2)
Surfaced by: DO companion MR B Task 3 review (out of scope there). Core handleSpawnAgent records the parent's kind: 'subagent' usage pointer (CompanionToolDispatchDeps.recordCompanionSpawnUsage) only when isFirstSpawn (!existingRef) and the entry started (core/src/orchestration/companion-tool-dispatch.ts, after the entry table). A same-id REPLAY of a first spawn (the ref already exists, carrying this call id, non-terminal) is not a first spawn by that test, so when the original attempt crashed between the ref write and the usage write, the replay that starts (or attaches) never records the pointer: the companion's usage is then undiscoverable from the parent's usage entries. Pre-existing (before MR B the same isFirstSpawn gated it). Fix direction: key the pointer on the ref's own parentToolCallId matching this call (isThisCallsFirstSpawn) and make the usage write idempotent by (parent, childSessionId), or record the pointer in the same write as the ref.
Fixed (CP-212): every spawn entry that starts the child session owes the pointer (a same-id replay included), and recordCompanionUsagePointer records nothing when the parent already has a subagent entry for that session. Core red "CP-212: a same-id replay of a first spawn that starts records the usage pointer".
FU-REMOTE-OWED-STOP-RETRY-SERIALISE: two concurrent owed-stop retry passes over one debt are not serialised — ✅ fixed by CP-217 (WA)
Fixed: every debt write is a compare-and-set on the status AND the debt it read (UpdateSubSessionRef.expectStopOwed: the debt's kind, at, target and attempts, on all five stores; core stopOwedMatches; contract "a debt write is a compare-and-set on the owed stop it read"). Two passes may each send a stop (harmless), but never put back a debt the other cleared.
Filed by: the independent review of !480 (CP-203). Core retryOwedRemoteStops writes a settled ref's debt with a compare-and-set on its STATUS (expectStatus: ref.status), and every debt write re-states that same settled status. So two passes that read the same debt at once (two entries of one session, or an entry and a stop cascade) both pass the check: both may re-stop the child, one pass's backoff write can put back a debt the other just cleared, and two passes past the 24 h window both stamp abandonedAt and both warn remote_stop_owed_abandoned. The impact is low (a duplicate stop is harmless, a debt put back is cleared by the next pass, a duplicate warn is noise), and the guide and the helper say "once per pass". Wanted: a compare-and-set on the debt's identity as well (its at and attempts, e.g. UpdateSubSessionRef.expectStopOwed) on all five stores, with a contract case where the second of two passes is refused.
FU-JS-WRITEOWNREF-REOPEN-RULE: runtime-js's dispatch expects whatever ref status it read, so it could re-open a settled ref — open (LOW; owner: WE-2)
Filed by: the independent review of !480 (CP-203). runtime-js writeOwnRef (runtime-js/src/execution/remote-subagent-executor.ts, CP-202) takes the status the dispatch read at entry as the status every later write expects (knownRefStatus), and writes running / completed / failed. A dispatch that read a ref another writer SETTLED (a lost-step sweep's failed + stopOwed) would pass that compare-and-set and re-open or overwrite it. Why it is unreachable today: a superseded run's dispatch registers its own ref running before any sweep can see it, so its writes expect running and are refused once the sweep settles the ref; a new run's model step mints new call ids, so nothing re-dispatches a lost call's child id. CF Workflows' persistOwnRemoteRef always expects running. Wanted: align with core's narrowed rule (CP-203, remote-sub-agent-dispatch.ts): a write that records the ref running may expect interrupted only when the ref read interrupted with no stop owed, else running; a settle write never overwrites a ref another writer settled. Red: a stale dispatch over a sweep-settled failed + stopOwed ref leaves it settled.
FU-TEMPORAL-CHILD-PHASE: Temporal still runs companion calls inside the step — open (owner: the parked Temporal bucket)
Filed by: CP-106 MR A (WE-2). runtime-js and the CF DO now run a companion call after the commit that holds it (core runChildPhase, the iterator's commitChildPhaseResults hook). Temporal reimplements the step loop and still dispatches companion calls (dispatchCompanionTools, runtime-temporal/src/workflow.ts:1395 at 232439ab39) before its commitStep (:1561), so a step a stop refuses can leave a ref and a child (CP-106). Wanted: the same phase at Temporal's step site (announce, commit, then the fenced ref write, the keyed start and the start fence, append results), with its replay and recovery entry (spec §4.4).
FU-DBOS-CHILD-DISPATCH: DBOS has not taken the child dispatch and companion identity design — open (owner: the parked DBOS bucket)
Filed by: the child dispatch and identity spec (docs/superpowers/specs/2026-10-25-child-dispatch-and-identity-design.md §9): DBOS follows MR 2 (dispatch) and MR 3 (companions minted, (parent, name)) after the CF runtimes, next to FU-DBOS-CHILD-PHASE. Must keep or replace RM-75's salt: until MR 3, a DBOS companion's workflows are keyed on the parent run that starts them (companionSpawnWorkflowId / companionContinuationIds), because its session id is still derived from the parent and its name. When MR 3 mints the companion's id, its workflow ids may follow the minted id, but never a session id a delete can bring back (RM-75's cells, session.delete-cascade on dbos-postgres, must stay green).
FU-DBOS-CHILD-PHASE: DBOS still runs companion calls inside the step — open (owner: the parked DBOS bucket)
Filed by: CP-106 MR A (WE-2). As FU-TEMPORAL-CHILD-PHASE, for DBOS: its companion calls run as durable steps (runtime-dbos/src/workflows/shared.ts:3974 at 232439ab39) before the step's one commitStepMessages commit (commitStep: :4262, a terminal step's at :4139). Wanted: the child phase after that commit, each call a durable step keyed on the committing run (childStartRequestKey).
FU-STEP-CALL-FRAMES-AFTER-STEP-COMMIT: some calls' frames follow their step's step_end and step_committed — open (LOW)
Surfaced by: RM-141's fix (2026-11-21). Related: RM-141, CP-106, FU-TEMPORAL-FINISHWITH-NO-TOOL-FRAMES, FU-TEMPORAL-CHILD-PHASE, FU-DBOS-CHILD-PHASE.
- Since RM-141 a step's
step_endis written just before the commit that records the step, after its calls' frames (ai's order). A call that runs after that commit still writes its frames after the step'sstep_endandstep_committed, so a client sees itstool-output-availableafter the step'sfinish-step. - runtime-js and the CF DO: a companion call runs in CP-106's post-commit child phase (core
isChildPhaseCall; the step iterator routes only companion calls there,step-iterator.ts:3165-3212, and runs them after the suspension commit at:4191and after the step commit at:4591, origin/main 28e535ae00). Itstool_startis announced with the step; itstool_endfollows the commit. - DBOS: a step that suspends (a client or approval call, or one waiting on a child) commits its suspension before any of its calls run (
runtime-dbos/src/workflows/shared.ts:4258, before the Phase 1 dispatch at:4301, the same commit), so every frame of its other calls follows itsstep_end. A step that does not suspend runs its calls first. - Temporal:
finishWithruns aftercommitStepand writes no tool frames at all: FU-TEMPORAL-FINISHWITH-NO-TOOL-FRAMES. - CF Workflows: none found within a run.
- Not changed by RM-141: on Temporal, CF Workflows and DBOS a stop's pairing
tool_ends for the calls it cut follow the step'sstep_discarded, and so itsstep_end, as before. - Fix direction: decide whether a call that runs after its step's commit belongs to the step on the client (
aikeeps a tool part's output in the step that announced it). Either accept and document the order, or have the client transformer place such outputs back in their step. No cell pins the order yet.
FU-DBOS-COMPANION-MISSING-TYPE-SILENT-SKIP: a DBOS companion call with no serialized companionType is skipped silently — open (LOW; owner: the DBOS bucket)
Filed by: the finish-siblings review (Greptile on !672), G1 / CP-216 MR 3b. DBOS classifies a call as a companion from its serialized isCompanion and dispatches it with its serialized companionType (runtime-dbos/src/serialize.ts). A companion-classified call whose companionType is missing is skipped with a bare continue, in Phase 3 (workflows/shared.ts, "Skip gracefully") and in the restored-finish takeover, and its call stays unpaired. Unreachable through the companion factories (each sets _isCompanionTool and _companionType together), but the Tool type has both as independent optionals, so a hand-built tool reaches it. Wanted: a typed framework_internal_error at serialization (or the two markers linked in the Tool type), with a test.
FU-CP106-CHILD-PHASE-STATE-WRITES: a child-phase call's state writes are not committed with its step — open (LOW; owner: CP-106 MR B)
Filed by: CP-106 MR A (WE-2). A child-phase call's body runs after the step's state commit, so a write it staged through ToolContext.updateState would not land in that commit. No call writes state there today: the companion tools never call updateState. Wanted (before MR B moves sub-agent calls into the phase): either commit a phase call's staged writes with its result append, or refuse a write from a phase body, with a test.
FU-CP106-JS-APPEND-REENTRY-KEY: a re-entered runtime-js sendMessage that appended into a live child run can append it twice — open (LOW; MR B must close it)
Filed by: CP-106 MR A, from Greptile's P1 on !523 (WE-2). A phase sendMessage that starts or resumes a child run is keyed (childStartRequestKey, one key per call for every route), and a re-entered call whose earlier attempt started a run for it answers delivered (core handleSendMessage: the child's run found by the call's key). A send that APPENDS into a child run that is already live starts no run, so it records no key: a re-entry of that call (runtime-js only re-enters on a resume of a suspended run, B2) appends the message again. Fix shape: stamp the appended row with the call's key (row metadata), and make the re-entry check read it. MR B must close it: it moves sub-agent sends into the phase, which widens this window.
FU-CP106-SETTLE-NOTICE-NONBLOCKING-E2E: rebuild the DO deliver-then-continue eviction cases on the non-blocking pair — done (WE-2; lands with its MR)
Why: CP-106 MR A retired three end-to-end cases from runtime-cloudflare companion-settle-notice.integ.test.ts ("C14 with a fresh call id (deliver-then-continue)", "a held prior survives an eviction between the hold and the step commit", "a held prior survives a faulted release after the step commit"). Their setup parked the parent inside a BLOCKING spawn, and under the post-commit child phase that state is unreachable.
Done: the three cases run on the worker's NON-blocking planner -> helper pair, each anchored on an observed event.
- The undelivered word. Planner's first model call spawns helper. While helper's
/startis held, planner is blocked in its child phase, so its model hold is armed race-free. Planner's next model call is held while helper finishes, so the word is recorded after that iteration's notifier ran. - Case 1 (deliver-then-continue). The held call's fresh-id spawn carries the word as
priorResult, and the next pass releases the slot with no notice. - Case 2 (an eviction between the continue's hold and its append commit). The re-entered call's answer carries
priorResult, with no notice. - Case 3 (a faulted release after the commit). The carrying row is the delivery, and the next turn releases the slot with no second delivery.
Found while rebuilding (case 2): a call re-entered after that crash met its own held slot, answered without the prior, and the next pass delivered it as a notice. Exactly once, but the channel changed because of the crash.
- Ruling (ii) (orchestrator, 2026-10-26): the re-entered call reclaims its own hold.
- Fix: core
ownHeldPrior, used by the CF DO'sholdPriorResultand runtime-js'sholdUndeliveredPrior. runtime-js reaches the same shape through CP-106's B2 resume, and re-registers the reclaimed hold so the post-commit seam releases it once. - Reds:
do-persistent-agents(and a contrast: another call's slot is not reclaimed),run-loop-persistent(B2: exactly one release against the carrying row), and integration case 2. - The replay path (Greptile on !544). A call whose earlier attempt held the prior AND started its child before the crash never reaches the hold:
sendMessage's already-started replay check answered{ delivered: true }on both runtimes, and a spawn's re-entry answered from its entry with no prior. In the child phase core now asks the runtime's optionalreclaimOwnHeldPriordep (runtime-js re-registers the hold; the CF DO's notifier releases it), else reads the slot itself (ownHeldPrior).holdPriorResultis not reused there: at replay the child may have finished this very consult, and a full hold would take the call's own answer as its prior. Reds: corecompanion-tool-dispatch(with and without the dep; another call's slot gives none),do-child-phase, andrun-loop-persistent(sendMessage and spawnAgent, each re-entered after its child started). - The C14 replay (the conformance owner's cell
companion.reconsult-evicted-keeps-prior-host, on !544). When the child had already finished the re-consult before the parent's recovery, the CF DO's spawn replay answered from the recorded outcome (recordedConsultOutcome, C14) before the reclaim. That branch now reclaims too, on both the non-blocking and the blocking answer. Reds: corecompanion-tool-dispatchanddo-child-phase.
FU-TEMPORAL-REMOTE-RESULT-BEFORE-CALL-UNPATCHED: a Temporal execution started before helix-delegation-commit-v1 commits a remote result before its call — open (RM-114 residual; owner: runtime-temporal)
Filed by: G1 / CP-216 MR 3a (WA). RM-114: before MR 3a, a Temporal step that delegates to a remote sub-agent committed each child's result row (recordSubSessionResult's append, during the remote wave) BEFORE the step's reply, which only commitStep wrote after the wave, so the history read tool_result before its tool_use and the next model request carried that order. MR 3a commits the reply first (the delegation commit) on executions past wf.patched('helix-delegation-commit-v1'). An execution that started before the patch replays false for its whole life (the patch is evaluated once, at its first activation), so it keeps the old order, and the old recovery, until it ends.
- Wanted: nothing in code (old histories must replay their commands). Drain such executions at upgrade (the upgrade note says so); this entry closes with FU-TEMPORAL-DELEGATION-PATCH-DEPRECATE.
FU-D3-TEMPORAL-DBOS-LOST-STEP: Temporal and DBOS never settle the remote child of a step lost with its run — open (owner: the parked Temporal / DBOS buckets)
Update (G1 / CP-216 MR 3a): on Temporal a workflow past helix-delegation-commit-v1 commits the delegating reply before any remote child starts, so no step is lost with its run there: its owed child is re-attached by the next entry with no input, or stopped by a new turn. Open for DBOS (MR 3b) and for Temporal executions started before the patch.
Update (G1 / CP-216 MR 3b): on DBOS too: a delegating step commits its reply before any remote child starts (commitDelegation), so a workflow that ends while the child runs (a stop cancels it, it runs out of recovery attempts, it errors) leaves the call committed; the next continue resume() or retry() with no message re-attaches to the child (planDelegationReattach), and a new turn stops it. Open only for Temporal executions started before the patch, and for the CP-202 compare-and-set below.
Filed by: CP-194 (WE-2). A remote child whose parent step was lost keeps a live ref that no entry settles. Temporal dispatches the remote call (runtime-temporal/src/workflow.ts :1364 at main 0889c0ee05) before the step's commitStep (:1561), so a run terminated, cancelled or superseded mid-step leaves such a ref; DBOS registers the ref inside the phase-1 remote step (runtime-dbos/src/steps/execute-remote-subagent.ts :312), ahead of the single step commit.
- Wanted: as runtime-js / the CF DO (CP-194,
sweepLostStepRemoteChildren) and CF Workflows (!458): after each entry's run-start commit, corelostStepRemoteRefsover the committed log, each child stopped concurrently (corestopRemoteChild, bounded), and the refs settled with a rows-freeappendcarryinglostStepRefSettlement(...).settle(stopOwedwhen the stop was not confirmed); cross-runtime tests. - Also in scope (CP-202 / CP-203 review): DBOS's in-run dispatch still writes its ref with no compare-and-set (
runtime-dbos/src/steps/execute-remote-subagent.ts:368 and :803 at main 78b21c1683), so a late write of a superseded run can revert a ref another writer settled; it should be a compare-and-set that never re-opens a settled ref, as on the other runtimes (CP-202; core's dispatch re-opens only aninterruptedref with no stop owed, CP-203). Once DBOS writesstopOwed, wire coreretryOwedRemoteStops(CP-203) at its entries too.
FU-TEMPORAL-REMOTE-HEAL-SETTLE: Temporal and DBOS heal a running remote child's call at the next entry without settling its ref — open (owner: the parked Temporal / DBOS buckets)
Update (G1 / CP-216 MR 3a): done on Temporal. Each entry's plan settles the session's remote children first (core settleRemoteChildrenBeforeInput, its settle on the run-start commit): refuse for a resume or retry carrying a message, orphan for a new turn (the orphans stopped after the commit, their refs failed). An entry with no input keeps every trailing delegated call out of its heal and re-attaches to it. Open for DBOS (MR 3b).
Update (G1 / CP-216 MR 3b): done on DBOS, the same way: the run-start plan settles the remote children of an entry with input (EntryRunPlan.settle on the run-start commit; the orphans stopped after it, best-effort, their refs failed), and an entry with no input keeps the trailing delegated calls out of its heal. Closed for both runtimes.
Filed by: the remote settlement MR (WE-2). At a new turn, resume or retry, runtime-js / the CF DO and CF Workflows read each live remote child through its transport (core settleRemoteChildrenBeforeInput / remoteChildVerdict) and pair its call with the child's real (or typed lost) result, settling the ref in the entry's run-start commit. Temporal (runtime-temporal/src/activities.ts :1412, :1503, :9811) and DBOS (runtime-dbos/src/steps/run-commit.ts :639, :734, :789) build the C5 heal with only the pending client calls excluded: a remote child's trailing call is paired not_executed, the ref stays running (no reconcile reads it), the child is never interrupted, and a child that finished loses its result. Neither runtime runs reconcileRemoteSubAgents. Temporal's recordSubSessionResult path (a result recorded during a run) does settle atomically.
- Wanted: call
settleRemoteChildrenBeforeInputin each entry's plan, carry itssettleon the run-start commit, interrupt itsinterrupts, and refuse on its typed errors, as runtime-js does; cross-runtime tests.
FU-REMOTE-LOST-INTERRUPT-DURABLE: a lost remote child's interrupt is lost on a crash between the settling commit and the interrupt — open (LOW, owner: WE-2 / the remote bucket)
Filed by: the remote settlement MR's final review (M1). A child settled lost (CP-178) is interrupted after the commit that settled it: runtime-js / the CF DO in afterRemoteSettlement (after each entry's run-start commit). A crash or eviction in between skips it for good: the ref is already terminal. A deadline_exceeded child keeps spending until its producer ends it. (CF Workflows no longer has this window: after RM-69 (!458) its lost children are D3's, settleLostStepRemoteChildren, which interrupts each child BEFORE the commit that settles its ref, so a crash in between re-runs both at the next entry.)
- Wanted: re-derive the interrupts from the run's committed rows (
framework_subagent_lost, every cause excepttool_unavailable) at recovery; an interrupt is idempotent and best-effort, so a repeat is harmless.
FU-REMOTE-LOST-INTERRUPT-BOUNDED: the lost-child interrupt is awaited inline with no time bound — done (!449)
Done: the lost-child and D4-orphan interrupts go through core stopRemoteChild (bounded by REMOTE_CHILD_STOP_TIMEOUT_MS, 10 s; never throws) on runtime-js (executor option remoteChildStopTimeoutMs) and CF Workflows. Pinned by runtime-js remote-settlement.test.ts "FU-REMOTE-LOST-INTERRUPT-BOUNDED". On CF Workflows after RM-69 (!458) the lost-child interrupt is D3's (settleLostStepRemoteChildren, the same bounded core stopRemoteChild; an unconfirmed one keeps its owed stop: cfw-rm80-remote-child-at-entry.test.ts "D3 + CP-152").
Filed by: the remote settlement MR's final review (M3). An entry awaits each lost child's transport.interrupt(); HttpRemoteAgentTransport.interrupt retries with no timeout, so an unreachable producer (the usual reason a child is lost) delays the run after it already waited out getStatus.
- Wanted: bound the interrupt, or run it without blocking the entry.
FU-REMOTE-SETTLE-LANDED-CHECK: reconcile's unknown-outcome check scans from the in-memory message count — open (LOW)
Filed by: the remote settlement MR's final review (M2). settleRemoteChildCall (runtime-js) re-reads the log from state.messages.length after a CommitOutcomeUnknownError, and an error from that read replaces the original. With a settle a missed row only ends in a refused retry; on the CP-146 path (a terminal ref, no settle) a missed row would re-append it. No path is known where the in-memory count exceeds the store's.
- Wanted: capture the store's message count before the append (or re-read the ref when settling) and keep the original error when the re-read fails.
FU-SETTLE-PARENT-CALL-VALIDATED: a commit's settle is not checked against its own rows — open (LOW)
Filed by: the remote settlement MR's final review (M5). SubSessionSettle.parentToolCallId is carried but no store (nor core settleRefusal) checks that the commit's rows answer that call, so a wrong plan could settle a ref with no result row (a terminal ref behind an unpaired call).
- Wanted: a pure core check every store runs inside the commit: each settle's call is answered by one of the commit's
role: 'tool'rows; else a typed refusal.
FU-SETTLE-REVIEW-FOCUS-TESTS: two settlement Review Focus cases have no direct test — open (LOW)
Filed by: the remote settlement MR's final review (M6). Review Focus 1 (a stop-cascade terminal ref plus a heal row in a step commit) is moot only while no step commit carries a settle; Review Focus 5 (a child paused awaiting a client on a new turn is orphaned, never refused) is covered only by the verdict unit table.
- Wanted: an entry-level test of each, on runtime-js and CF Workflows.
FU-REMOTE-START-ANSWER-IDENTITY-UNCHECKED: a validated /start / /resume answer is not matched to the session that was called — open (LOW, owner: WA MR3, the protocol MR)
Core HttpRemoteAgentTransport (core/src/transport/http-transport.ts:149 and :163, parseResponse, origin/main) and, since WA MR1, the DO companion tools validate a 2xx body with RemoteStartResponseSchema but never check that its sessionId is the session they asked for. A misrouted or proxied answer for another session would be adopted (its streamId / runId). Fix direction: compare sessionId with the request's and fail typed (RemoteProtocolValidationError) on a mismatch, in both transports. Not in MR1: a protocol-wide check (Task 15 review M3); MR3 changes the protocol.
FU-TYPED-ERROR-BODY-RETRYABLE-AND-DETAILS: the typed error body cannot carry retryable, sessionStatus or the awaited call ids — open (LOW, owner: WA MR3)
Core's typed HTTP error body is { error, code, cause? } (typedErrorBody / typedErrorResponse). The DO's state_not_resumable answer (R37) therefore loses ResumeRefusedError.sessionStatus, awaitingToolCallIds and its retryable flag (a paused partial batch is retryable), and a client rebuilds only a HelixError with the code; the awaited ids survive only in the message, which names them first. Fix direction: widen the body (and PeerErrorBodySchema) with optional retryable and details, and rebuild ResumeRefusedError on the client. Not in MR1 (ruling R52): the body is shared by every typed route, so widening it is a protocol-wide wire change.
FU-DO-SUBMIT-COMMITTED-UNCONFIRMED-CODE: the DO submit's at-bound answer cannot say "committed, outcome unconfirmed" — open (owner: WA MR3)
The DO /submit-tool-result answers a submit whose continuation was not confirmed within concurrentEntryTimeoutMs (CP-53 (b)) with 500 transport_error, non-retryable (submitContinuationTimeoutError, runtime-cloudflare/src/do/drive/submit-answer.ts:161, sent at do/durable-object-agent-base.ts:2543). The result is stored and the run continues; the message says "not confirmed" and asks for a snapshot re-read. But the typed body carries no retryable (FU-TYPED-ERROR-BODY-RETRYABLE-AND-DETAILS), so a client reading only the code cannot tell "committed, outcome unconfirmed" from "may not have arrived": transport_error means both. Core's RunStartOutcomeUnknownError, RunStartStalledError and RunStartAbandonedError (core/src/errors/agent-errors.ts) share the same ambiguity: all three are transport_error. Fix direction: a distinct committed-unconfirmed code (e.g. a state_* code mapping to 409) for these answers. Not in MR1: a wire change across every host and parseHelixChatError (orchestrator ruling: WA MR1 keeps the code and only rewords the message; a duplicate submit now answers 200 already_completed at once, so only a submit that stored the result itself can get this answer).
FU-JS-ROUTED-SUBMIT-CHILD-VALIDATION: a routed submit owned by an in-process sub-agent can skip output validation — open (LOW, owner: B4-Node, parked)
On runtime-js's in-process topology, a root-session submit for a call a sub-agent owns goes through core routeSubmitToolResult → validateSubmitAgainstOutputSchema (core/src/client-tool/route-submit.ts:251-292, origin/main). When the callback registry cannot resolve the owner's agent type, getToolByName returns nothing and the result is accepted unvalidated (:281-282). WA MR1 made the submitting session's own type fail typed (framework_agent_not_found) and validates the local-owner branch, but not this forwarded-owner lookup. The CF DO is unaffected: the owning DO validates on arrival. Fix direction: resolve the owner's type through the executor's agent tree (findAgentInTree) and fail typed on a miss. Not in MR1 (ruling R17): pre-existing, JS in-process only.
FU-SUBMIT-OWN-ROUTE-RETENTION-SOURCE: an own-session submit reads the tree root's marker with the named session's retention — open (LOW, owner: unassigned)
A submit addressed to a rooted child's own session (CP-254 rule (a)) reads the completed marker on its tree root with the completedRetentionMs its executor resolved. CF Workflows and Temporal resolve it from the NAMED session's agent type; the same call answered at the root reads it with the root agent's. DBOS resolves the real root's, and runtime-js's direct branch deliberately uses the addressed session's (js-agent-executor.ts, the CP-258 marker comment). The answers differ only when the two agents configure different completedTombstoneRetentionMs and the marker's age falls between them. Fix direction: one rule for which agent's retention reads a tree root's marker, decided once for every runtime.
FU-CFW-RETENTION-LOOKUP-SWALLOW: CF Workflows still swallows the tombstone-retention lookup — open (LOW, owner: WB)
WA MR1 records a call's completed-tombstone retention on its pending entry at registration and deleted runtime-js's getCompletedRetentionMsForRoot. CF Workflows keeps its own copy: runtime-cloudflare/src/workflow.ts:2532-2541 (origin/main) passes a getCompletedRetentionMsForRoot that catches every error and returns undefined ("best effort"), and client-tool-workflow-helper.ts:456 consumes it. A store failure or an unknown root type silently uses the constructor default. Fix direction: record the retention at registration as runtime-js does, and delete the lookup. Not in MR1 (ruling R21): a separate runtime path, WB's scope.
FU-CORE-SUBAGENT-REFRESH-ADOPTS-FOREIGN-VERSION: a step that dispatched a sub-agent adopts any newer session version before it commits — open (MED, owner: WA MR3; runtime-js and the CF DO)
core/src/orchestration/step-iterator.ts:1919 (suspension commit) and :2105 (step commit), origin/main: when the step dispatched a sub-agent call, the iterator reloads the session and adopts ANY newer version (fresh.version > nextState.version) before its version CAS. That is needed because a child's own suspension writes ownership and pending entries onto the root out of band. But it equally adopts a foreign writer's bump (a concurrent submit on the root, an admin saveState, a stale executor), and the commit then silently overwrites that write's fields. Scenario: a step runs a sub-agent call and a tool; another writer updates the root's customState (v → v+1); the step commit reloads, adopts v+1 and saves its own snapshot, losing the write. Fix direction (R38's model): the child's out-of-band writes record exactly what they wrote (base version, resulting version, fields) on a run-scoped record, and both commit sites adopt only that chain. Not in MR1: pre-existing (it predates WA MR1); R38 made the workspace path exact and left this one (Task 11 review minor 8).
FU-EXECUTE-REFUSES-NON-AGENT: execute() doesn't refuse a value that isn't an agent config — open (LOW)
execute() on every runtime, and the CF DO's materializeAgent, take the agent they are given as is. A value that isn't an agent config starts a run, which then fails with an untyped error on its first read of llmConfig, instead of a typed refusal before anything is written. Fix direction: one core check at entry, before the run-start commit.
FU-DBOS-REGISTRY-MISS-UNTYPED: DBOS throws a plain Error for an unregistered agent type — open (LOW, owner: DBOS bucket)
DBOS's process-local AgentDefinitionRegistryHolder misses throw new Error('… is not registered in AgentDefinitionRegistryHolder …'): runtime-dbos/src/workflows/standard-workflow.ts:647, workflows/persistent-workflow.ts:108, steps/execute-companion-tool.ts:465 and :1075 (origin/main). WA MR1 (MA-15) made the CF and Temporal registries throw core AgentResolutionError.notFound (framework_agent_not_found), so DBOS is the one runtime whose miss has no code. Fix direction: throw AgentResolutionError.notFound (or a typed framework_internal_error where it is an executor-registration invariant), and let isAgentNotFoundError callers treat it as a miss. Not in MR1: DBOS is parked (Task 14 review M5).
FU-RESUME-REFUSAL-AWAITING-IDS-PARITY: CF Workflows, Temporal and DBOS keep the old partial-batch refusal text — open (LOW, owner: WB for CF Workflows; parked buckets for Temporal / DBOS)
runtime-js (and the CF DO) refuse a resume of a partial client-tool batch with core ResumeRefusedError, whose message names the awaited ids first (Awaiting client tool call(s) <ids>: submit every result of the batch, then resume session '<id>'.) so they survive a typed body's 256-character scrub, and which carries awaitingToolCallIds (R37). CF Workflows (runtime-cloudflare/src/executor.ts:489), DBOS (runtime-dbos/src/lifecycle/resume.ts:374) and Temporal (runtime-temporal/src/executor.ts:921) keep the ids-last "Cannot continue session '…': still awaiting results for client tool call(s) …" text with no structured field. Fix direction: throw ResumeRefusedError there too. Not in MR1: outside the DO scope.
FU-JS-RESUME-LOOP-UNTYPED-REFUSALS: runtime-js entry methods still throw untyped "session not found" / "type mismatch" errors — done (closed by WA MR1)
runtime-js/src/js-agent-executor.ts still throws plain Errors for Agent session not found: <id> and Agent type mismatch: … (origin/main :3104, :3263, :3290, :3963, :3969, :5288, :5293), so a caller cannot classify them without matching text (classifyEntryFailure reads them as unclassified, transient). The CF DO is unaffected: its drive gate refuses both typed (state_session_not_found, state_agent_type_mismatch) before the executor runs. Fix direction: throw HelixError state_session_not_found / state_agent_type_mismatch (non-retryable). Not in MR1: the DO path is typed at the gate (Task 5 review minor).
Closed by: WA MR1. All seven throws are typed, non-retryable HelixErrors with the same message text: state_session_not_found (resume(), retry(), the reconnect handle's resume loop) and state_agent_type_mismatch (getHandle(), resume(), retry(), the resume loop). The same sweep typed two more refusals: retry() on a non-failed session is a ResumeRefusedError (state_not_resumable, sessionStatus = the status it read), which the CF DO had answered as a retryable 503 once MR1 removed its /retry precheck (now the R37 409); and resume({ mode: 'with_confirmation' }) is framework_not_supported. Tests: runtime-js retry-target-d1.test.ts and js-agent-executor-resume.test.ts ("… refusals are typed (WA MR1)"), runtime-cloudflare do-drive-pipeline.integ.test.ts (native /retry on a completed session).
FU-AISDK-RUN-ERROR-TEXT-UNSCRUBBED: the stream transformer forwards a run's own error text to the client unscrubbed — done (FU-RAW-ERROR-TEXT-SURFACES MR 1)
Decided (user, 2026-10-16, relayed by the orchestrator): "redact secrets, keep detail": strip credentials and control characters, run-level text capped at 256, tool text uncapped with its line breaks, no HTML escaping, the raw text in the server log only. Resolved by FU-RAW-ERROR-TEXT-SURFACES MR 1 (WE-2): the scrub lives at the stream write site. Every stream manager sanitizes each chunk and failStream's text, so the transformer forwards text that is already sanitized; /snapshot is sanitized, and the DO's /status error is WA's DO companion MR A. It splits by surface, not by error class: run-level text is capped at 256 characters, a tool's error text keeps its line breaks and is never capped, and both are redacted (known secrets, paths, IPs, emails), never HTML-escaped. See the section Raw error text on outward surfaces. The original entry follows.
@helix-agents/ai-sdk's stream transformer (ai-sdk/src/transformer/stream-transformer.ts, origin/main) forwards a run's own error text verbatim: an error chunk's errorText: chunk.error (:1226), tool-input-error (:1441), tool-output-error (:1458), and a failed tool_end's output.error (:990-993). WA MR1 (Task 16, R56) scrubbed every text handleChatStream itself writes (the typed entry answers, the failed-stream 409, the transformer's own error chunk) and kept code and shape; it did not change these.
The same raw text reaches clients through the DO / runtime channels too (ruling R62). The run's own failure text (SessionState.error / the failed stream's metadata.error and errorDetail) is also sent:
- by the DO's
/statusas itserrorfield (runtime-cloudflare/src/do/durable-object-agent-base.tshandleStatus,:4188,error: row.errorat:4277, origin/main; itserrorDetailcause is sanitized, itserroris not); - in the stream's fail frames (
runtime-cloudflare/src/do/do-stream-manager.ts, origin/main): the WebSocketfailevent (:626), the SSEfailframe (:634) and the WebSocket close reasonconnection.close(1011, error)(:640); - in the
/snapshoterror fieldsstreamError/streamErrorDetail(runtime-cloudflare/src/do/snapshot-wire.ts:79, origin/main).
One design ruling covers every channel: scrub the text once at the write site (where an error becomes the run's failure text), or at each send site (the transformer, /status, the stream frames, /snapshot).
- Threat model. These strings come from inside the run: a tool's thrown error, a provider error, a store error that a runtime wrapped into the run's failure. Any of them can carry internal detail (a database URL with credentials, a stack frame, a host name, infrastructure text), which then reaches the browser. They are also conversation content: the model already saw the tool error, and the user may need its detail to act.
- Open question. Where should the scrub live: at each runtime's seam where an error becomes run content, or at the transformer boundary for every client? Should it split by type (for example by
ErrorDetail.code/category: a framework or store error scrubbed, a tool's own error passed through)? Scrubbing everything truncates legitimate tool detail; scrubbing nothing leaks infrastructure text. - Needs the user's ruling before any change.
Not in MR1 (ruling R57): a UX / security trade-off in code MR1 did not otherwise touch.
FU-R66-TEMPORAL-DBOS-RUNTIME-CASE: Temporal's and DBOS's R66 entry gates have no runtime-level case — open (owners: the parked Temporal bucket; the parked DBOS bucket)
R66 (WA MR1 Task 13): an entry FINISHES, never supersedes, a live current run (running or suspended_*) whose K5 landed. Every entry gate passes finishCurrent for such a run, and core applyRunStartRules verifies it. Temporal's and DBOS's gates are pinned only by the store contract scenario run.k5-landed-finishes-suspended (it drives the store's run-start commit, not the runtime's entry):
- Temporal:
packages/runtime-temporal/src/run-entry.ts:309(expected !== null && isLiveRunStatus(expected.status)); - DBOS:
packages/runtime-dbos/src/run-entry.ts:287(same condition).
Scenario to add, per runtime. A run suspends on a client call, so it reads suspended_client_tool and the session is paused. A stop lands its K5 (the session interrupted, the run's own state commit), and the run-status write is lost (the worker died). The next entry (a new turn) commits, and the stopped run must read interrupted, not superseded. Show it red with the gate's pre-fix condition restored (expected?.status === 'running'), as runtime-js (r66-suspended-k5-finish.test.ts) and CF Workflows (r66-cfw-entry-finishes-suspended.test.ts) do.
Not in MR1: both runtimes are parked and the program is CF-first; MR1 changed their gates with the others (one rule everywhere) but adds runtime-level cases only for the CF runtimes (the CF DO runs runtime-js).
FU-UNSUPPORTED-OPERATION-ERROR-UNTYPED: core UnsupportedOperationError is a plain Error with no code — open (owner: WA MR3)
Core UnsupportedOperationError (core/src/errors/agent-errors.ts:286) extends Error, not HelixError, so core classifyEntryFailure reads it as unclassified (transient: a retryable 503 on a DO request, a re-arm on a wake). WA MR1 F4 P5 typed the one path where it reached a DO answer: a native /start carrying branch is now refused at parse with framework_not_supported (runtime-cloudflare/src/do/drive/request-schemas.ts, parseNativeStartBody). The class and its other throw sites are unchanged (orchestrator scope guard):
runtime-cloudflare/src/do/do-state-store.ts:3061:cloneSession(no DO answer reaches it now);runtime-cloudflare/src/do/do-state-store.ts:1189: a run-start commit carryingstartRun.resumedCompanionRef(CP-103 A1; the DO never passes one, and core's contract scenario matches it byname);memory-redis/src/redis-memory-store.ts:36: a non-emptymetadataFilter. The memory retriever'sisMemorySearchRefusal(memory/src/retrieval/retriever.ts, MA-29) already accepts both shapes (the class and a non-retryableHelixErrorframework_not_supported), so the change needs no edit there.
Fix direction: make it a HelixError with code: 'framework_not_supported' (retryable: false), keeping name and details, and check every catch site and the run-start-operations contract's { name } match.
FU-CF-MEMORY-EUCLIDEAN-THRESHOLD: the CF memory threshold is inverted for a euclidean index — open (found by MA-29)
Semantic search keeps a match when score >= threshold (default 0.7). For a euclidean Vectorize index the score is an L2 distance (smaller is closer), so the threshold keeps the FARTHEST memories and drops the nearest. MA-29 applies the same rule on its exact D1 path, so the two paths agree. It also never treats a failing euclidean window edge as proof of completeness. Fix direction: for euclidean, keep score <= threshold, and decide what a 0-1 threshold means for a distance (a separate maxDistance, or no default).
FU-DO-BRANCH-UNSUPPORTED: DO branching is a capability gap, refused typed but not declared — open (owner: WA MR3)
A Durable Object cannot read another DO's session, so DOStateStore.cloneSession refuses and DO branching is unsupported (CLAUDE.md, "Branch"). Until WA MR1 F4 P5 the native /start schema still accepted branch, and the request failed late, often as a retryable 503. It is now refused at parse with 500 framework_not_supported, before any read or write (do-drive-pipeline.integ.test.ts "F4 P5"). Still open: the capability is not declared anywhere a host can read before it calls (StartBody keeps the branch field). Fix direction: drop branch from the DO's public StartBody (or declare the capability), and route a branch request to a runtime that supports it in the docs.
FU-DO-RUN-SETTLED-DELIVERY-KIND: a redelivered onRunSettled carries no result / errorDetail — open (owner: WA MR3)
redeliverPendingSettle (runtime-cloudflare/src/do/drive/settle.ts) redelivers a missing mark with only sessionId, runId, drive: 'recover' and status: a host cannot tell a redelivery from a first delivery that has no detail, and a redelivered failed has no errorDetail although the session holds it. Fix direction: make the delivery kind explicit (delivery: 'first' | 'redelivered' on RunSettledContext) and load the session's errorDetail for a redelivered failed.
FU-DO-PRODUCER-CLAIM-LEASE: the producer claim is a bare boolean; the execution registration has two sources of truth — open (owner: WA MR3)
DriveHost.producerClaim.tryAcquire() returns boolean and release() releases whoever holds it, so a caller that did not take the claim can release it. The execution registration lives both in DOAB's _registration and in ExecutionState (execution, run id, status, abort controller), which executingDrive() reconciles. Fix direction: a lease type (tryAcquire(): ClaimLease | null, release only through the lease), and one source of truth for the registration that ExecutionState reads from.
FU-DO-RECOVER-REDRIVE-OVER-FAILED-STREAM: a recover re-drive reactivates a stream that already said failed — open (owner: WA MR3; must be CLOSED there, fix plus a test)
The same invariant as R77 (commit e04dade275) and its interrupted twin (commit 3b5d2000e5): announce only what is durable, and never run a session again after its run announced a terminal. WA MR3 must close this one (a fix and a test proving it), not only track it.
The DO executor has no run lease. When runtime-js onFailed's K5 commit AND its writeRunStatus('failed') both fail, runtime-js still fails the run's stream (finalizeStreamForFailure, packages/runtime-js/src/run-loop.ts) while the run stays running. The DO settle path then calls failCommittedRun (requireSessionCommit), which reports not_committed (the session write fails too); the wake's closed-owner plan for a live run with nothing committed is fail_uncommitted, so the ladder re-drives it (recover). The recover entry's ensureStreamLive (packages/runtime-js/src/js-agent-executor.ts:808-810) reactivates the failed stream and the run executes again, after its stream told every reader it failed. The paused variant: the suspension completes (complete_suspension) over a stream that already said failed.
R77's predicate does not catch it: the run is still running, not failed.
Fix direction: before a recover re-drive (and before complete_suspension), treat a live run whose own stream is failed as an announced failure: commit the session failed with the stream's recorded detail (the R77 write, plus the run's terminal status), then settle it; never reactivate the stream. Test: a DO integ case with the K5, the run-status write and every session write to failed faulted, then real alarms: exactly one run, no hook before the session commit, onRunSettled(failed) once after it.
FU-DO-ANNOUNCED-FAILURE-PAUSED-SESSION: an announced failure behind a paused session is settled over paused — open (owner: WA MR3)
R77's shape requires the session active. A paused session whose current run ended failed (its K5 not landed) is reachable through an R47 rewind that re-suspends on its restored batch (the entry state writes the session paused in its run-start commit), then fails before it suspends and loses its K5. The run and stream say failed; the session says paused. The settle path delivers onRunSettled(failed) over the paused session (R77's predicate does not match, so nothing commits it first). No second run starts without a client submit (the ladder re-drives a paused session only for an unconsumed submit), but a submit then continues a session whose run was announced failed.
Fix direction: widen the announced-terminal predicate to a paused session (the commit's idle fence already allows it: the run is ended), or prove the shape unreachable and assert it.
FU-JS-ONFAILED-K5-UNCOMMITTED-ANNOUNCES-FAILED: runtime-js announces failed when its K5 did not land — open (owner: B4-Node bucket, parked)
runtime-js onFailed (packages/runtime-js/src/run-loop.ts, the K5 commitTerminalState then writeRunStatus) writes the run status failed even when the K5 session commit failed after its 3 attempts (anything but a supersede), and its caller fails the stream (finalizeStreamForFailure). The session stays active while the run and its stream say failed: the run announces an outcome that is not durable.
The CF DO now reconciles this shape (WA MR1 F4 P1 sibling, ruling R77, commit e04dade275; module announced-terminal.ts since its interrupted twin, commit 3b5d2000e5). Its settle path and wake ladder commit the session failed with the stream's detail before onRunSettled(failed), under the crash-loop budget, and never re-drive the session into a second run. The interrupted twin is the same shape on runtime-js's interrupt paths (finalizeDurableInterrupt and onFailed's soft-interrupt branch write the run interrupted and end the stream after a K5 that did not land); the DO commits the session interrupted (no error) before onRunSettled(interrupted) and never exhausts it into failed. Standalone runtime-js and agent-server have no wake ladder, so there the session stays active behind a failed run and stream: a status read disagrees with the stream, and a later entry supersedes nothing (the run is already ended).
This breaks the "announce only what is durable" rule, the same one behind RM-29 ("persisted history never contains a tool_use without its result"), applied to terminal status: no reader is told an outcome the session does not record.
Fix direction (option A of R77, rejected for WA MR1 as a cross-runtime contract change): on a failed K5, skip writeRunStatus and the stream failure, and return "not committed" (the run stays live). Then define a JS-side contract for who finalizes it: the handle's caller, a recover path, or the next entry's run-start commit (finishCurrent cannot apply, since nothing landed). Once runtime-js does that, the DO's R77 refinement is unreachable and can go.
WA MR2 (stop as one signal) follow-ups
FU-TEMPORAL-STOP-SETTLE-CLOSED-OWNER: no Temporal reader settles a stopped run whose workflow closed without its K5 — open (parked; owner: the Temporal bucket, DoD §E)
Surfaced by: WA MR2 Task 18 (drive spec §4.6, "Abort is phased").
A Temporal abort is a stopRun (the record) plus the stop signal and cancel() of the run's workflow. Since WA MR2 F4-C (m4), the cancelled workflow commits its own K5: its terminal runs in a non-cancellable scope, and the K5 resolves the abort record (failed / framework_cancelled), so the aborting handle no longer waits for the workflow or settles the run itself. F4-C (S3) also stops the workflow from reporting a terminal that did not commit: it fails temporal_terminal_not_committed, and the handle reading its result reconciles the run (the stop's terminal, else failCommittedRun). What is left is a run whose workflow closed without its K5 and that no handle reads: a worker crash after the cancel, or a temporal_terminal_not_committed failure whose handle is gone (a child workflow's included: its parent records a failed dispatch). It stays live until the next entry settles it (its run-start commit is refused expected_run_stopped, and commitEntryRun settles the run first). No reader (getHandle, recoverCommittedRun) plans settle_stopped the way CF Workflows' settleClosedOwner does. Fix direction: adopt core planClosedOwnerReconciliation / reconcileClosedOwnerRun on Temporal's reader paths (together with FU-TEMPORAL-RECONCILE-CLOSED-OWNER-ADOPT), with the owner proven closed by describe().
FU-TEMPORAL-CHILD-CANCEL-UNTYPED-FAILURE: a Temporal child cancelled by its parent's abort fails with no framework_cancelled code — open (LOW, owner: the Temporal bucket)
Surfaced by: WA MR2 F4-C review (minor 1).
An abort cancels the parent's workflow, and Temporal's child cancellation cancels the child workflows it is waiting on. Since F4-C (m4) a cancelled workflow commits its own terminal in a non-cancellable scope, ending the run by the signalled stop. A child receives no stop signal and has no stop record, so its terminal is failed with the cancellation's own message (resolveErrorDetail of a CancelledFailure: { message }, no code). The parent's abort is framework_cancelled, but its children are untyped failures. Fix direction: in the workflow's top-level catch, map a cancellation with no stop (wf.isCancellation(err) and no inProcessStop / record) to a framework_cancelled detail (stopAbortDetail-shaped, the cancellation's message), and add an integ case: a parent aborted while waiting on a sub-agent child, asserting the child session's errorDetail.code.
FU-TEMPORAL-RESULT-SETTLE-NOT-MEMOISED: concurrent result() calls on a Temporal handle run its settle twice — open (LOW, owner: the Temporal bucket)
Surfaced by: WA MR2 F4-C review (minor 3).
Since F4-C an active handle's result() (settleResult) is not memoised until it resolves: every call that arrives before cachedResult is set runs the whole settle (settleSuspendedStop, or reconcileUncommittedTerminal for a workflow that closed without its terminal), so two concurrent callers each make the store reads and the settle writes. The writes are idempotent (commitStopTerminal / failCommittedRun never overwrite a landed outcome), so the outcome is the same, at twice the cost and with duplicate warn logs. Not caching a rejection is deliberate (a later call must retry). Fix direction: memoise the in-flight settle promise, and clear it on rejection.
FU-CI-TEMPORAL-TEST-CLIENT-NO-RETRY: the Temporal test client ran with no gRPC retry, unlike production — done
lifecycle.result-latency|temporal-* failed in CI three times on 2026-10-16 (!366 jobs 17237083992 and 17239435050, !371 job 17237635907), each on a different temporal lane: Failed to get Workflow execution history, 14 UNAVAILABLE: Connection dropped, ~27 s into the cell. The cell holds one result() long-poll open for ~27 s; a dropped connection under it failed the call at once because the harness built its client with interceptors: [] (runtime-temporal/src/__tests__/testing/test-environment.ts, the GL #87 / F5 teardown-race fix), which removes the default retry interceptor that production clients run with. Fixed: connectTestClient runs production's policy (makeGrpcRetryInterceptor with defaultGrpcRetryOptions) behind a close guard (createClosableGrpcRetry: no retry once closing, and close waits for any retry already scheduled to fire), so the teardown race stays closed. Proof: test-client-connection-drop.integ.test.ts (a proxy resets the connection under the long-poll) was red with the old client and is green; closable-grpc-retry.test.ts pins the close guard. What dropped the connections in CI is not proven (a CPU-starved or restarted service container on a loaded node are the leads); a client that retries like production no longer fails on one drop.
Runner side (the user's infrastructure; a request, not an action): the service containers run on the runner's defaults (1 CPU each, no request of their own). A job-level KUBERNETES_SERVICE_CPU_REQUEST / KUBERNETES_SERVICE_MEMORY_REQUEST is silently ignored on our runner (gitlab-runner 19.4.0, namespace ci-jobs): the job trace logs each honoured overwrite ("MemoryLimit" overwritten with "8Gi") and none for the service variables, so the runner sets no service_*_overwrite_max_allowed. Reserving CPU for the Temporal service needs either that runner setting or a job-level pod_spec patch (the runner has FF_USE_ADVANCED_POD_SPEC_CONFIGURATION on).
FU-TEMPORAL-RESULT-TRANSPORT-ERROR-REPORTED-FAILED: a transport error while waiting for a Temporal result reports the run failed — open (parked; owner: the parked Temporal bucket)
The Temporal handle's result() (runtime-temporal/src/executor.ts, the wfHandle.result() rejection branch through projectFailure) maps every rejection of the client's result() to status: 'failed'. That includes a transport error while merely WAITING for the outcome (ServiceError: Failed to get Workflow execution history, 14 UNAVAILABLE: Connection dropped). The run itself is unaffected: the server completes it, but the caller is told it failed. A production client retries a dropped connection (Temporal's default gRPC retry interceptor), so this needs an outage longer than its retries (10 attempts, backoff capped at 5 s). The test harness ran with no retry at all until FU-CI-TEMPORAL-TEST-CLIENT-NO-RETRY (above), which is how conformance lifecycle.result-latency|temporal-* surfaced it in CI on 2026-10-16. Expected: a transport error while awaiting the result is a typed, retryable transport failure (or the handle re-reads the run's persisted outcome), never failed. Repro: a TCP proxy that resets the client's connection under result(), with retries exhausted; the run completes and result() reports failed.
FU-TEMPORAL-ABORTED-TOOL-END-BEFORE-K5: Temporal writes the aborted tool_end frames before the stop's K5, so a later settle can add a second one — open (LOW, owner: the Temporal bucket)
Surfaced by: WA MR2 F4-C review (minor 5).
Both Temporal stop arms (the resume path and, since F4-C S3, the step boundary) write an aborted tool_end for each pending client-tool call BEFORE the stop's K5, so a terminal that is announced only once committed still has its per-call frames announced ahead of it. When that K5 does not land, the workflow fails temporal_terminal_not_committed and a later settle (commitStopTerminal through the handle, or the next entry) ends the run, and any discard frames it writes are a second tool_end for the same call. Readers settle a tool part on the first frame, so this is cosmetic, but the stream is not exactly-once. Fix direction: write the frames after the K5 lands (on the run's still-open stream), or skip a call whose aborted tool_end the run already wrote (runWroteChunk).
FU-TEMPORAL-STOP-AFTER-SUSPENSION-COMMIT: a stop recorded after Temporal's suspension commit is settled only at the next continuation — open (LOW, parked; owner: the Temporal bucket)
Surfaced by: WA MR2 Task 18 (the CFW analogue is Ruling 42).
The suspension commit (commitSuspendedStep) is guarded (refuseIfStopped), so a stop that lands before it ends the run. A stop that lands after it, while the workflow is still finishing its suspension (the run status, the pause), gets stopping from stopRun (the owner is still open), and nothing reads it again: the paused run keeps the record until a continuation settles it (Ruling 41, settleStoppedSuspension) or a new entry does (expected_run_stopped). Since WA MR2 F4-C (m4), a handle that reads the workflow's suspended_* result settles a recorded stop at once (settleSuspendedStop); the gap remains for a run no handle reads. CF Workflows reads the record after every suspension commit (Ruling 42). Fix direction: a readStopActivity after the suspension commit in the patched branch, ending the run by the stop.
FU-TEMPORAL-ACTIVE-SESSION-STOPPED-SUSPENSION: an active session over a stopped suspended run is refused on Temporal, settled on runtime-js — open (LOW; owner: the Temporal bucket)
Filed by: CP-115 (!566 independent review, orchestrator ruling 2026-10-28). Since CP-115, execute() refuses a new turn over a live, non-running run on both runtimes (refuseNewTurnOverSuspendedRun), but they treat a recorded stop differently when the session reads active (running / waiting_tool):
- runtime-js skips the refusal for any run with a recorded stop, so the run-start commit refuses it
expected_run_stoppedand the entry settles it and proceeds (Ruling 18); - Temporal settles a stopped suspension only when the session reads
paused(settleStoppedSuspension), so anactivesession over a stopped suspended run is refusedAgentAlreadyRunningError.
The window is short: a suspension commit leaves the session paused, so this needs a session that still reads active over a suspended run with a stop. Wanted: Temporal settles a recorded stop for any live session before the refusal, like runtime-js (a unit with the session active), or a ruling that the refusal is right there.
FU-CONF-RESULT-LATENCY-TEMPORAL-GRPC-DROP: lifecycle.result-latency drops its Temporal gRPC connection under CI load — open (parked; owner: conformance owner, with the parked Temporal bucket)
Surfaced by: !366's CI (orchestrator, 2026-10-16): lifecycle.result-latency|temporal-memory|executor, then |temporal-redis|executor, failed with 14 UNAVAILABLE: Connection dropped on "Failed to get Workflow execution history" at about 27.6 s, plus "left backend work running — reaped at cell end". The cell takes about 27 s on every run; !366 added no Temporal cells, only slower runners.
The cell's long result() wait holds one Temporal client history call open for most of its run, so a gRPC idle or keepalive limit between the harness client and the CI Temporal server can cut it. Look at: the harness Temporal client's gRPC keepalive / idle-timeout settings against the CI server, and whether the cell's result() wait can be bounded or anchored on an observed event instead of one long call.
FU-TEMPORAL-COMPANION-STOP-UNCOMMITTED-CHILD: a Temporal companion terminate of a child whose run has not committed is dropped — open (LOW, parked; owner: the Temporal bucket)
Surfaced by: WA MR2 Task 18 (the CFW analogue is Ruling 43 a).
The interim session-flag bridge stopped a child whose run had not committed yet (its first boundary read the flag). stopPersistentChild now records on the child's CURRENT run (target: current): a child workflow started but not yet past its run-start commit has no live run, so the terminate records nothing (the parent ref is still terminated, and a resume of a terminated ref is refused, resumed_companion_ref_terminated). Fix direction: as CFW does, record on the run id the parent derives for the child it started, or terminate that child workflow.
FU-DBOS-STOP-CANCEL-EXECUTING-OWNER: a DBOS stop that cancels a live owner settles after a fixed 500 ms — open (LOW, parked; owner: the DBOS bucket)
Surfaced by: WA MR2 Task 18 review (Ruling 48 D1).
A DBOS stop of a run whose owner workflow is live (interrupt(), abort(), and a continuation completing a recorded stop on a suspension, Ruling 41) signals the owner, cancelWorkflows it, waits DEFAULT_STATUS_UPDATE_DELAY_MS (500 ms) and then settles the run (settleCancelledOwner: a K5 that landed is finished, else commitStopTerminal). The cancel stops the owner's NEXT step, not one already in flight. The settle's own writes are fenced and guarded, and the owner's step commits and K5 are stop-guarded. However, an in-flight step with an unguarded write (saveState of pending entries, staging, a stream pause) that outlives the 500 ms can still land after the stop's K5. DBOS's cancellation of a workflow with child workflows (a suspended run awaiting children) is unverified: does the cancel cascade, and is each child's run settled? Fix direction: settle once the owner's status reads CANCELLED and its in-flight step finished (a DBOS getWorkflowStatus poll, bounded), rather than after a fixed delay. Add an integ case for a parent stopped while it awaits a child workflow.
The resubmit window (WA MR2 R2c, §7.1). The submit's own settle (settleRecordedDbosStop) takes the same path, and can meet an EXECUTING owner: a first submit is accepted (its marker written, DBOS.send wakes the workflow, which starts its continuation); a record-only stop is then recorded while the pending entry still exists (the woken workflow has not cleared it yet); a duplicate submit (a client retry) reads the entry, finds the stop and reaches settleRecordedDbosStop, which cancels the now-executing workflow and settles the run 500 ms later. The continuation's stop-guarded commits keep the outcome correct (it ends by the stop), but an in-flight unguarded write can land after the stop's K5, as above. The same fix (settle once the owner reads CANCELLED and its step finished) covers it; an integ case should hold the woken workflow between its recv and its pending clear while the stop and the duplicate submit land.
Related (2026-11-14): CP-328 is a concrete instance of the same class on the IDLE path. A stop of an idle persistent loop sends it shutdown, cancels it and sets the session interrupted (or failed), but the loop's finalize, started by that shutdown, ends in an unguarded session status write (runFinalizeRun) that can land after the stop's compare-and-set.
FU-CP244-DBOS-PERSISTENT-MODE: Ruling 57a on DBOS persistent sessions is unverified — open (LOW, parked; owner: the DBOS bucket)
Filed by: W1 with CP-244's fix (orchestrator ruling 2026-10-28: persistent mode out of the fix's scope).
- The contract: Ruling 57a covers any
execute()over a paused run with a recorded stop. The stop is settled first and the new turn runs. - What CP-244 fixed on DBOS: standard mode only.
executeImplsettles the stopped suspension (settleStoppedSuspension, the shared Ruling 41 helper) before its open-owner refusal. The conformance cellstop.paused-record-then-new-turnruns DBOS root sessions in standard mode, and persistent mode appears only for companion spawns, so no cell witnesses persistent mode. - Persistent mode is different:
executePersistentImplroutes the new message by the session's workflow (decideRoute): into a live (parked) workflow it is asend, not a run-start entry. Whether that workflow honours the recorded stop on its wake (settling the stopped run before it takes the message as a new turn), or continues the stopped run, is not tested. - To do: a cell (or a DBOS integ case) for a persistent session paused on client calls, with a record-only stop, then
execute(). Fix the routing or the wake if it does not settle the stop first.
FU-DBOS-RUN-STOPPED-SUBAGENT-CALL-PATH-B: a sub-agent's call of a run a DBOS stop ended answers the bare unknown call — open (LOW; owner: the DBOS bucket)
Surfaced by: WA MR2 R2c (§7.1 path (b), Ruling 56).
A DBOS stop's settle (settleCancelledOwner → clearPendingClientToolsOnInterrupt) clears the call's clientToolCallOwnership entry on the ROOT together with the owner's pending entry. A later submit for that call is answered by the router before any dispatch (ownership_missing_toolcall). R2c re-checks that bare answer on the root (refuseSubmitIfStopped with no entry), so a ROOT-owned call answers { status: 'unknown_tool_call', reason: 'run_stopped' }; a SUB-AGENT-owned call can no longer be traced to its owner session and keeps the bare unknown_tool_call (a client reads "this call never existed"). Every other runtime keeps the ownership map after a stop (core commitStopTerminal clears only the pending entries), so path (b) is decided at the owner there. Fix direction: stop clearing ownership in the DBOS stop settle (as core does), and check that the arbiter's sub-agent no-entry path (pre_submit_stub_written) can no longer accept a call of a stopped run (R2c's owner-side check runs before it); add a DBOS integ case for a sub-agent call submitted after its run was stopped.
FU-DBOS-SEND-MESSAGE-STOPPING-CHILD-INTEG: Ruling 59's stopping-child branch has never run against real DBOS — open (LOW; owner: the DBOS bucket)
Surfaced by: WA MR2 R2c (Ruling 59).
DBOS's own inbox companion__sendMessage reads the child's stop record in a recorded step (dbos.companion.child_stopping) and answers a child whose LIVE run has a recorded stop { delivered: false, reason: 'child_busy' }, sending nothing (runtime-dbos/src/steps/execute-companion-tool.ts). Only the unit tier covers the true branch (execute-companion-tool.test.ts, send-message-to-stopping-child-busy); the DBOS integ suite runs the false branch on every send, but no case holds a persistent child mid-turn (or parked on a client call) with a record-only stop while the parent's model calls sendMessage. Fix direction: a DBOS integ case with a scripted parent and child (a held child model call, requestStop on the child's run, then the parent's sendMessage) that asserts child_busy, no inbox message, and that the child ends interrupted and a later sendMessage resumes it.
FU-NOT-RESUMABLE-ERROR-CORE-HELPER: the not-resumable refusal is built three times with drifting hints — open (LOW; owner: core)
Surfaced by: WA MR2 R2c review.
The typed refusal of a resume() over a finished session (AgentNotResumableError, state_not_resumable) is built in three places: DBOS notResumableError (runtime-dbos/src/lifecycle/resume.ts:461), Temporal notResumableError (runtime-temporal/src/executor.ts:269) and CF Workflows inline (runtime-cloudflare/src/executor.ts:3332-3343 and :3506-3517). The failed-session hint already differs: DBOS says "Only interrupted or paused sessions can be resumed: use retry() for a failed session, or send a new message with execute() (the next turn).", CF Workflows and Temporal say "The agent has failed: use retry(), or send a new message with execute() (the next turn).". runtime-js and the CF DO build a ResumeRefusedError with their own wording. Fix direction: one core helper (for example notResumableError(sessionId, status)) that owns the hints for completed and failed, called by every runtime, so a client sees one message per status.
retry() over a non-failed session (WA MR2 F4 parity review m6; predates MR2). The same refusal for retry() is a typed ResumeRefusedError on runtime-js and the CF DO but a plain Error ("Cannot retry: agent status is …") on CF Workflows (runtime-cloudflare/src/executor.ts:3710), Temporal (runtime-temporal/src/executor.ts:2463) and DBOS (runtime-dbos/src/lifecycle/retry.ts:74), so a client cannot match one error across runtimes. Fix direction: the same core helper builds the retry() refusal (a typed state_not_resumable-class error with the "use resume()" hint) for every runtime.
A concurrent retry (CP-333's MR): a retry() that reads the session after another entry's run-start commit is refused there too, so on CF Workflows (runtime-cloudflare/src/executor.ts:4355-4359 at origin/main 4d2f6e9afc) the loser of two concurrent retries gets that untyped Error. Fix direction for that case: the C3 loser (AgentAlreadyRunningError) while the current run is live, as runtime-js does since !929; otherwise the typed not-resumable refusal above. CF Workflows is parked.
FU-STOP-KEEPS-WHAT-THE-RUN-BUILT: a stop that loses the race to the deltas keeps text the user may not have seen — decided, closed (user, 2026-11-02)
Decision (user, 2026-11-02): reword the ruling, no behaviour change. Stop keeps what the run produced up to when the server received the stop; the saved partial can hold text the client had not rendered yet, because stop() does not wait for its request. The docs that said "the text the client saw" now say so.
Owner: the orchestrator raised it with the user. Filed: 2026-11-02.
useHelixChat().stop() does not await its stop request (use-helix-chat.ts:740), so the deltas the model produces before the stop lands are built into the run's partial reply and kept. The partial-reply ruling (#181) speaks of "what the user saw": a stop that lands late keeps what the run built, which can be more than the client rendered. This was seen when client.stop-reaches-server released its model right after stop() on cf-do (job 17650539820: .no-full-reply and .reload-sees-stopped failed while .server-run-stopped passed). The cell now waits for the stop before it releases, so it no longer exercises this order. No behaviour change is proposed here: the question is whether the ruling's words or the behaviour should move.
FU-RUN-LEASE-RENEW-RETURNS-STOP: the lease renewal could carry the run's stop record — open (LOW, owner: W1)
Surfaced by: CP-281 (orchestrator ruling: a later five-store change, not the CP-281 MR).
Since CP-281, runtime-js reads the run's stop record after every lease renewal the store accepts (JSAgentExecutorownStopCheck, run-start.ts startRunLease({ onRenewed })): one renewal and one indexed readStop per run per ttl / 3. Folding the read into renewRunLease's atomic step (it would return the run's StopRequest, or null) saves the second round trip. It is a SessionStateStore contract change on all five stores (memory, Redis, Postgres, D1, DO-SQLite; the DO holds no lease, so DO-SQLite only keeps the shape), with the run-start contract suite extended.
FU-STOP-DEADLINE-NODE: no stop deadline outside the Cloudflare DO — open (parked; owner: B4-Node)
Surfaced by: WA MR2 (user ruling Q1 scoped the deadline to the DO).
Only the DO records a deadlineAt (stopDeadlineMs, default 30 s) and ends a run that ignores its stop at that time. runtime-js and @helix-agents/agent-server record deadlineAt: null: a run held by a tool or provider call that ignores its abort signal answers the stop 202 stopping and stays running until the call returns, and a new entry meets its live lease (run_active) meanwhile. A run whose JS owner crashed is settled only by the next entry (expected_run_stopped) or by a stop made once its lease is more than one TTL past (the owner probe's skew margin, WA MR2 Ruling 24). Fix direction: a durable deadline on the Node hosts (a timer per stop in the owning process, plus a reader that settles a live run past its deadlineAt), with the same detach rule as the DO (late writes refused by the run fence and refuseIfStopped).
The answer wording (WA MR2 F4 silent-failures review M6). Until then, core stopRun answers stopping (202) off the DO for a run nothing executes whose stop terminal did not land (not_committed or superseded, then answerFromDurable, packages/core/src/run/stop-run.ts:78-97, :163-172); its comment says "the wake or the deadline settles it", but runtime-js, Temporal and DBOS have neither (deadlineMs: null). A paused run stays paused with its record until the next entry (Ruling 41 / expected_run_stopped). The record is durable, so nothing is lost; only the 202's wording is wrong for those hosts. Fix: correct the comment and document the case in the stop-answer table.
FU-STOP-TOMBSTONE-STORE-BACKED: runtime-js keeps CP-311's pending entries and keyed-stop tombstones in its process — open (LOW, owner: WE-2)
Surfaced by: CP-311 (a Stop pressed right after Send).
JSAgentExecutor keeps the chat sends it received but has not run-started, and the short-lived tombstones a keyed stop leaves for a send that has not arrived, in its own process (PendingEntries with InMemoryStopTombstones). A stop and its send must therefore reach the same executor: behind a load balancer with several Node processes, a stop that lands on another process finds no pending entry and no tombstone, and answers as before (409 state_not_stoppable, or 404 on a first turn). The CF DO is unaffected (one DO per session; its tombstones live in its SQLite storage). Fix direction: a store-backed StopTombstoneStore (and a store-visible pending-entry record) so any process can stop a send another process received.
FU-JS-HOST-CREATE-SESSION-HOLD: no host-tier hold before a fresh session's row on js-chat-host — open (LOW, owner: WE-2)
Surfaced by: CP-311's fix slot (mutation M-FRESH-JS).
On runtime-js a fresh session's entry writes its session row (createSession) before its run-start commit. The conformance hold-chat-entry control on js-chat-host holds at the run-start commit, so the CP-311 first-turn cell's stop always finds the session row and never takes JSAgentExecutor.stop's no-session branch (the stop answered from the pending entries before the 404). That branch is proven only by a runtime-js unit that holds createSession (cp311-pending-entry-stop.test.ts). Fix direction: a js-chat-host hold before createSession, so a host-tier cell proves handleChatStream's receipt and that branch together.
FU-CP193-INMEMORY-STREAM-YIELD: runtime-js on Node with the in-memory stream manager starves its event loop on a large tool output — open (LOW, dev hosts)
Surfaced by: CP-193 (the CF DO writer now yields; orchestrator, 2026-10-19).
InMemoryStreamManager's write does no I/O either, so on runtime-js with the in-memory manager (the @helix-agents/agent-server dev setup) a tool that emits a large output through ctx.emit with no I/O of its own is one microtask chain, exactly as the CF DO was: the Node process serves no other HTTP request and flushes no SSE bytes until the tool returns. Production Node hosts use Redis (a network round trip per write) and are not affected. CF Workflows writes cross the stream-DO binding (real I/O); Temporal and DBOS write through real I/O. Fix direction: the same counted yield in the in-memory writer (setImmediate-sized, after the chunk is stored and its waiters woken), with a responsiveness test on agent-server.
FU-CONF-CFDO-FLOOD-HARNESS-COST: the cf-do conformance cell writes a large stream ~100x slower than the DO does — open (LOW, conformance harness; owner)
Surfaced by: CP-193 (owner s81 / s85, CP-136's cell; WE-2 gate f, 2026-10-20).
CP-136's cell (a tool emitting 2,000 x 16 KiB custom events, with the send's SSE reader pulling and an attach's reader held) takes ~26 s in the conformance cf-do lane: ~13 ms per write. The same flood on a real DurableObjectAgentBase under Miniflare with the same reader shape (do-large-output-throughput.integ.test.ts: one pulling and one held SSE reader, attached before the start) takes 0.30-0.36 s on main and 0.37-0.50 s with CP-193's yields: ~0.15-0.25 ms per write. So the cell's cost is the harness or its host (the conformance worker, the Node driver parsing ~33 MB per reader, parallel cells), not the product's write path. Timing-sensitive cf-do cells (a 2 s probe behind a flood) should budget for it or measure it. Fix direction: profile the cell (where its 26 s goes: worker, driver, or host load).
CP-193's own cell (owner slots s86-s88, 2026-10-20): scenarios/do-flood-responsive.ts (a no-I/O flood while the chat client reads; a mid-flood probe and stop) is kept unregistered because the lane cannot judge it. CP-193 is flipped by provenByException on WE-2's do-large-output-responsive.integ.test.ts instead.
- s86 (main + the first !466, 150,000 events): the probe went unanswered for 120 s on both, and the stop's
POST /interruptfailed withfetch failed. - s87 (main + !466 b4145f7c5e, 20,000 events): main was red (the probe answered after 139 s, final head 20,009). The fix runs crashed on the client stream-info route with
fetch failed. The Node timer was on time. - s88 (main + !466 e21b90da21, with a timeline and the cause chain): on main, the first worker-route answer came at 250.6 s and the send ended at 251 s, so the worker entry was starved too. The fix failed with
fetch failed <- cause Error ECONNRESET: connect ECONNRESET 127.0.0.1:<port>, and the run kept running (no crash). - The lane spends ~12 ms per write, against ~0.15 ms DO-only: the chat route's in-isolate reader runs inside the DO's write chain.
Owed (non-blocking): connection counts (lsof every few seconds) beside Miniflare's log, main vs the fix; workerd's stdout / stderr in the phase logs (Miniflare's handleRuntimeStdio is not on the per-worker options type from convertV4MiniflareOptions, so set it on the top-level MiniflareOptions); a profile of handleChatStream's per-chunk cost; then re-register the cell once the lane can judge it.
FU-CP193-EMIT-ABORT: ctx.emit does not check the tool's abort signal — open (LOW)
Surfaced by: CP-193 (orchestrator ruling on the writer-yield design, 2026-10-19).
A stopped run's tool that keeps calling ctx.emit (it ignores ctx.abortSignal) is not stopped by its writes: the runtime-js emit (packages/runtime-js/src/run-loop.ts, the tool context) writes regardless of the signal. On the CF DO, once the stop deadline has ended the run, each of those writes is dropped stopped_run (logged once per run), and the tool runs on to its end, holding its CPU and memory. Fix direction: emit rejects with a typed abort error once the run's signal is aborted, so an emitting tool that does not check the signal itself is still ended by its next emit; the same on every runtime's tool context.
FU-CLIENT-STOP-RUN-TARGET: a client stop targets the session's current run, not the run the client was showing — open (LOW, owner: W1 backlog)
Surfaced by CP-12 (W1, orchestrator ruling 2026-10-16). Every stop route a client calls stops the session's CURRENT live run (StopTarget { kind: 'current' }): handleChatStop, CloudflareChatHandler.stop, agent-server's /chat/{id}/interrupt|abort. The client cannot name a run, because neither the UI stream nor FrontendSnapshot exposes a runId. So a Stop pressed in one tab after another tab started a newer turn stops that newer turn. Planned end:
- expose the live run's id to the client (on the snapshot and on the stream's start);
- let the stop body carry
runId, which the routes pass asStopTarget { kind: 'run', runId }.requestStopalready supports a run target, and a stop for a run that is no longer current is then inert.
FU-COMPANION-TERMINATED-REF-GUARD-SPAWN-CONTINUE: a terminate can be dropped for a child an earlier turn created, and cannot stop a continued child — open (LOW)
Surfaced by: WA MR2 Task 17 review (Rulings 43 / 44).
The store refuses a RESUME run-start whose resumedCompanionRef names a terminated ref (resumed_companion_ref_terminated), so a resume never reopens a terminated child. Spawn and continue run-starts carry no such guard. A child created by an earlier parent turn and terminated before its run commits is not covered (the cross-turn residual), and a continued child's ref stays completed, so companion__terminateChild answers terminated: false and cannot stop it (the CF Workflows continue tracking write is unexercisable). Fix direction: a StartRun guard field for spawn and continue run-starts on every store (refuse when the parent's ref for the child is terminated), with a contract case, and a terminate that reaches a continued child's live run.
The DO and Temporal halves of the same race (WA MR2 F4 parity review m2). Spec §7.4 says every runtime answers child_terminated when a resume races a terminate; the DO and Temporal never produce it. A same-step terminateChild + sendMessage on the DO answers delivered: true and overwrites the terminated ref with running: the DO's resume branch writes the parent ref running unconditionally after /subagent/resume (runtime-cloudflare/src/do/persistent-companion-tools.ts:1187), and DOStateStore refuses resumedCompanionRef (UnsupportedOperationError), so no store gate catches it (the Ruling 43 class, open on the DO). Temporal starts the resume fire-and-forget and maps the revival cause to the C3 conflict, so it reports delivered: true too (that half is FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW in substance; add child_terminated there). Fix direction for the DO: a conditional ref write (only when the ref is not terminated) or a DO-SQLite implementation of the resumedCompanionRef guard, answering child_terminated when it refuses.
FU-DO-CHILD-STOP-RUN-FENCE: a DO child stop names no run — open, narrowed (LOW)
Now consult-scoped (MR C1): see the narrowing below; the body after it is the historical record.
Narrowed again by CP-106 MR A: the ref-write fence (M10) is done. A companion call runs in the post-commit child phase and writes its ref through the fenced addSubSessionRefs(…, { fenceRunId, refuseIfStopped: true }), stamped with that run (parent_run_id), so a late ref write of a superseded run is refused and never joins a newer run's cascade. Still open: the beginChildConsult attribution window below.
Narrowed by DO companion MR C1 (Q-A1 as narrowed, orchestrator 2026-10-17): every parent-to-child stop (the park, companion__waitForResult, companion__terminateChild, the cascade) names its consult (parentToolCallId); the child cancels an uncommitted entry of that consult, then records the stop only on the run serving it (runtime-cloudflare/src/do/drive/stop.ts handleStop, the persisted RunMetadata.parent), else answers state_not_stoppable and writes nothing. Consult-scoped, not run-scoped, so a submit continuation of the same consult is still reached. Still open: the ref-write fence (WA MR2 final review M10, below). MR C1 adds one more window of that class: beginChildConsult attributes a re-consult to the parent's CURRENT run, so a run superseded between assertRunOwnsSession and the begin attributes it to its successor (one of the two runs' cascades still posts the stop).
Surfaced by: WA MR2 Task 14 review.
The DO's child stop (/subagent/:type/interrupt|abort, body { reason } only) stops whatever run is current on the child DO (target: current). A late or racing cascade post (the stop cascade, a companion terminate) can therefore stop a newer child run than the one the parent meant: the stale-stop class the run-scoped record closes everywhere else. Fix direction: carry the expected child run id in the stop body and record with target: { kind: 'run', runId } (answer not_stoppable when it is no longer current).
WA MR2 F4-A P3 landed the child side's half: the DO stop routes accept an optional runId and record with target: { kind: 'run', runId } (a run no longer live is answered with no write). The parent's posts (the cascade's stopChildDO, a companion terminate's recordChildStopDO) still send none: the parent ref does not know the child's run id.
The parent side of the cascade has the same gap (WA MR2 final review M10, the CP-106 class: a spawn ref is not atomic with its step). DO addSubSessionRefs stamps parent_run_id from the session's CURRENT run (runtime-cloudflare/src/do/do-state-store.ts:1899-1901), and the ref write is not run-fenced. A late ref write from a detached or superseded drive is therefore attributed to the newer run and joins that run's stop cascade. Fix direction: the ref write carries its own run id and is fenced on it (with CP-106's next MR).
FU-SUBSESSION-REF-PARENT-RUN-OTHER-STORES: only DO-SQLite records the run a child ref belongs to — open, narrowed (LOW)
Narrowed by CP-106 MR A: every store now records it. SubSessionRef.parentRunId is stamped by the fenced add and re-stamped on every reopen (core reopenParentRunStamp; Postgres V24, D1 V28, the Redis hash field, memory), with contract cases. Still open: runtime-js's cascade (stopChildren) and the CF Workflows / DBOS / Temporal stop paths still select children from the whole ref list, not by parentRunId.
Surfaced by: WA MR2 F4-A D3.
The DO's stop cascade reads DOStateStore.getStopCascadeRefs(session, runId) over sub_session_refs.parent_run_id (stamped by the store with the session's current run on add, on a reopen and on a re-consult, MR C1 O9). Memory, Redis, Postgres and D1 have no such field, and SubSessionRef does not carry it: runtime-js's in-process cascade (stopChildren) and the CF Workflows / DBOS / Temporal stop paths still select children from the whole ref list. Fix direction: when another runtime's cascade selects by ref (F4-B / F4-C, or a later runtime-js change), add the column to every store with a contract case, or derive the attribution from the run's commits.
FU-MIGRATION-PIN-RETIGHTEN-AFTER-366: re-tighten the D1 migration-list pins once !366 lands — done (DO companion MR B's merge of main with !366)
Surfaced by: DO companion lifecycle MR B, Task 2 (the held prior: DO-SQLite 18, Postgres V19, D1 V23).
MR B's migrations were numbered past !366's (DO-SQLite 16 / Postgres V18 / D1 V22, which lands first), so until the rebase the lists carry a gap. Every runner tolerates it (core runSqlMigrations requires only strictly ascending versions; the DO runner applies every listed version above the stored one), but two D1 tests pinned contiguity and were relaxed to "the listed versions, strictly ascending, CURRENT_SCHEMA_VERSION last": store-cloudflare/src/__tests__/d1-migrations.test.ts (D1_SCHEMA_MIGRATIONS shape) and store-cloudflare/src/__tests__/integration/d1-collapsed-schema-parity.integ.test.ts (the two "every version row" cases, now LISTED_VERSIONS). Postgres' migrations-sql-content.test.ts lists [1..17, 19] explicitly. Once !366 is in and MR B is rebased (the numbers stay: 18 / V19 / V23 are the next free ones), restore the contiguous pins (1..CURRENT_SCHEMA_VERSION) so a future gap is caught again.
Done (2026-10-17, MR B's merge of main with !366). The lists are contiguous again (Postgres V1–V19, D1 V1–V23, DO-SQLite 1–18): d1-migrations.test.ts pins D1_SCHEMA_MIGRATIONS to 1..length with CURRENT_SCHEMA_VERSION its last version, d1-collapsed-schema-parity.integ.test.ts pins both paths' version rows to exactly 1..CURRENT_SCHEMA_VERSION (LISTED_VERSIONS is gone), Postgres' migrations-sql-content.test.ts lists [1..19], and the "gap until !366" notes at the migration sites and in the docs are removed.
FU-STORE-REOPEN-HELPERS-DUPLICATED: the reopen predicate and the generation-bump SQL are copied per store — open (LOW, tidy-up; owner: DO companion MR C or a store sweep)
Surfaced by: DO companion MR B Task 2 re-review (parked minor). The held prior's outcome generation is bumped by every reopen (a non-live ref written live), and the "is live" predicate that decides it, isLiveRefStatus, is copied in five stores (store-memory/src/in-memory-state.ts, store-postgres/src/state/postgres-state-store.ts, store-cloudflare/src/d1-state.ts, runtime-cloudflare/src/do/do-state-store.ts, and inline in Redis's UPDATE_SUB_SESSION_REF_SCRIPT / commit-script Lua), with the CASE that bumps it, REOPEN_GENERATION_SQL, copied twice (Postgres, D1). They agree today (the contract cases in core/src/testing/sub-session-operations.ts and run-start-operations.ts pin the behaviour on every store), but a status added to the live set must be added five times. Fix direction: one core helper (the live-status set beside SubSessionRefStatusSchema, and a shared SQL fragment builder for the two SQL stores), consumed by every store; the Lua keeps its own copy with a core-exported list it is generated from.
FU-DO-STOP-CASCADE-OWED-NO-ALARM: an owed stop cascade waits for an unrelated wake — open, narrowed (LOW, owner: WA)
Narrowed by DO companion MR A (spec 2026-10-15 §3.5-§3.6): the child-settled notice and the parent's owed verified reads have their own alarms (parent-notice, child-verify; runtime-cloudflare/src/do/durable-object-agent-base.tsregisterAlarmSubscribers), so neither waits for an unrelated wake.
Narrowed by MR C1 (D6): a companion whose stop post was not confirmed has a per-child owed record and the stop-cascade alarm (drive/child-stop-owed.ts), retried for childStopRetryBudgetMs, after which the alarm re-runs the cascade (its mark's only writer). Still open: a cascade whose READS failed (the stop record, the child refs, the remote transports) still has no alarm: it is re-run by the next wake only. A run left with only given-up tombstones whose cascade re-run did not write the mark gets no alarm either (the same half).
Surfaced by: WA MR2 F4-A S2 (the F4 detached-drive review's M8).
A cascade whose reads failed (or whose DO was evicted mid-cascade) leaves its mark absent, so the next wake re-runs it (redeliverPendingSettle, the owed-settle set). Once the run is terminal nothing arms an alarm for it: the re-run waits for the next request, entry or alarm the DO gets for another reason. Fix direction: an alarm subscriber that fires while a stopped current run (or an owed entry) lacks its cascade mark, with a backoff.
FU-DO-CHILD-SETTLED-NOTICE-ENDLESS-RETRY: a DO companion's child-settled notice retries forever against a parent that keeps answering 5xx — open (LOW, owner: the conformance owner)
Surfaced by: the companion-lifetime trace for the user's session-scoped ruling (owner, 2026-10-20; companions outlive an idle Stop and a failed parent, C3).
A child DO's settle posts /companion/child-settled to its parent (ringParent, runtime-cloudflare/src/do/companion-settle-notice.ts:173-292 at main 16ae386528). On a 5xx or no answer it re-arms the parent-notice alarm with a 1 s doubling backoff capped at 5 minutes (BACKOFF_CAP_MS, :74) and, by design (:9), no give-up by time. A parent DO that answers 5xx permanently (a broken deploy, a corrupt row) therefore keeps the child's alarm firing every 5 minutes forever, and nothing says so past the per-attempt logs. Nothing is lost: the outcome stays on the child, and a parent read tool's verified read still finds it. Fix direction: after a bound (attempts or age), stop re-arming and write one terminal error log naming the parent and the child, or keep the 5-minute cadence but log the give-up threshold once.
FU-DO-CASCADE-REMOTE-STOP-UNCONFIRMED: a remote ephemeral child's failed stop post counts as done — ✅ fixed by CP-217 (WA)
Fixed: every unconfirmed stop of a remote child is owed on its ref and retried by the stop-cascade alarm, one debt model with the companions' (spec 2026-10-26).
Surfaced by: DO companion MR C1 (spec 2026-10-17 §7). The cascade's per-child confirmation (D6) covers persistent companions only: a remote ephemeral child (stopRemoteChild, runtime-cloudflare/src/do/drive/stop-children.ts) whose transport throws or times out is logged and counted done, so nothing retries it. Fix direction: the same owed record and stop-cascade retry, keyed on the remote ref, with the transport's typed errors classified like postChildStop.
FU-REMOTE-CHILD-PARENT-LEASE: a remote child its parent cannot reach keeps working until a retried stop reaches it — open (MEDIUM; owner: design session / user agenda)
Filed by: CP-217 (user rule R4, 2026-10-26). A parent's stop is durable at once and retried until the child confirms it, but while the parent cannot reach the child (a partition, a producer down) the child keeps calling its model. Wanted (defence in depth against runaway spend): a parent-liveness lease the child checks at each step boundary, so a partitioned child stops itself within a bound; the parent renews it while it runs, including while it is suspended. Cost: a per-step check on the child, and a renewal on the parent.
FU-TEMPORAL-REMOTE-CHILD-STOP-OWED / FU-DBOS-REMOTE-CHILD-STOP-OWED: Temporal and DBOS owe no unconfirmed remote stop — open (MEDIUM; owner: parked, CF-first)
Filed by: CP-217. Temporal's core dispatch sends one interrupt on cancel (even for an abort) and drops a failure; a parent INTERRUPT never cancels the remote activity at all. DBOS sends a remote child no stop (lifecycle/interrupt.ts). Wanted: the same debt (core remoteStopDebtUpdate) at their stop sites and a pacer (a Temporal timer / a DBOS scheduled workflow) running core owedRemoteStopsSource; the Temporal interrupt gap goes to the conformance owner as a finding candidate.
FU-CFW-REMOTE-STOP-PACER: CF Workflows retries a remote stop only at entries and cascades — open (MEDIUM; owner: WA, CP-217 MR 2)
Filed by: CP-217 (spec §5.2). CF Workflows' in-flight dispatch sends its stop once, and core retryOwedRemoteStops runs there over SETTLED refs only, at entries and cascades. Wanted (MR 2): the dispatch owes an unconfirmed stop on its live ref, and a durable __stops__<run> instance loops step.sleep to the next due debt (core owedRemoteStopsSource) until none is left or the window ends.
FU-DO-STOP-DEBT-PER-DELEGATION: a sub-session ref holds one owed stop, so a second delegation's unconfirmed stop is owed only through a cascade re-run — open (MEDIUM; owner: WA)
Filed by: CP-217 (independent review of !555). SubSessionRef.stopOwed is one debt. When a companion owes consult A's stop and the parent's cascade stops consult B, A's debt is kept until its own run is proven over (core stopEndsEarlierDelegation). If B's stop then goes unconfirmed (timed out, failed), it has no slot. Interim (in !555): the DO cascade answers unrecorded for that child, so it writes no mark. Its re-run (the wake's settle redelivery, on every alarm tick while A's debt keeps the alarm armed) posts B again, and once A's debt is settled the re-run owes B normally (stop.child_stop_unowed_earlier_debt_kept, warn). The CP-106 B6 fence stop (oweChildStopOnRef) logs the same warn and writes nothing; its stopped parent run's cascade then takes the same path. B's re-posts follow A's retry cadence, not B's own schedule. Core remoteStopDebtUpdate returns {} for this case, and its doc says a caller must surface it. Wanted: debts keyed per delegation (a ref owes each delegation's stop separately, each with its own budget and compare-and-set identity), across all five stores and both pacers. Then the cascade owes B durably at once, and the interim goes.
FU-RJS-DELETE-ALL-CALLS-EXECUTOR: runtime-js's legacy executeAllCalls is dead code with an unarmed owed-stop path — open (LOW; owner: WA)
Filed by: CP-217 (Greptile on !555, F4). runtime-js/src/execution/all-calls-executor.ts executeAllCalls has no production caller: the run loop replaced it, the package does not export it (execution/index.ts re-exports it, but src/index.ts exports neither), and only tool-sequential-execution.test.ts calls it. Its remote calls go through executeSingleRemoteSubAgentCall without owedStopSchedule or armOwedStops, so a debt it wrote would not be armed. Wanted: delete executeAllCalls and its test (port any case the run loop's own tests lack), rather than wire the pacer into a path nothing runs.
FU-RJS-RECONCILE-ORPHAN-ARM-WITNESS: the resume-time reconcile path's D4 orphan arm has no witness — open (LOW; owner: WA)
Filed by: CP-217 (Greptile on !555, F4). orphanRemoteChild arms the owed-stop pacer after its debt write. On a new turn its D4 orphan runs before the entry's lost-step sweep, whose arm already covers the debt (remote-owed-stop-arming.test.ts guards that, green before the fix). The resume-time reconcile path (reconcileRemoteSubAgents, a D4 orphan after the sweep) is where the arm matters, but no test reaches it: the sweep settles a ref whose call is in no committed row first. Wanted: build a witness (a resume where the sweep does NOT settle the ref first, red without the arm) or prove that branch unreachable and remove the arm.
FU-DO-COMPANION-STOP-GIVE-UP-PROVEN-GONE: a companion whose stop was given up keeps a running ref even when the child is proven gone — open (LOW, owner: WA)
Surfaced by: MR C1 (ruling O7, D6 kept). Since CP-217, past owedStopSchedule.windowMs (24 h) the debt is abandoned (stop.remote_stop_owed_abandoned, warn) and the ref left running. The user's D6 amendment has not been answered: it proposes that a PROVEN-gone child has its ref written failed with a typed framework_child_unreachable at the give-up. A typed 404 on the stop post already counts as CONFIRMED (and settles the record), so "proven gone" at the give-up needs another signal: the parent's ref read reporting no child session, or a repeated typed not-found on the child's /status. MR C1 ships D6 option (1). The drop-in point is core retryOwedRemoteStops's abandon branch (a per-runtime hook), which would decide from the last post's detail. Fix direction: at the give-up, probe the child's /status (twice); a typed not-found both times writes the ref failed + framework_child_unreachable, anything else keeps it running.
FU-STOP-DELIVERY-FAILURES-INVISIBLE: several stop-delivery failures leave no trace above debug — open (LOW, owner: WA)
Surfaced by: WA MR2 F4 silent-failures review M1, M2, M8.
Four places swallow a failure that bears on whether a stop reached its target, with no log or a debug line only:
- runtime-js skips a live remote child whose transport does not resolve with a bare
continue(runtime-js/src/execution/in-process-runs.ts:279-280), which includes every remote child ofJSAgentExecutor.stop(sessionId)called without an agent. The DO's cascade logsstop.child_unreachablefor the same case. Ruling 51 accepts the skip, not its silence. - CF Workflows logs a failed stop signal to an open instance at debug (
cfw.stop.signal_failed,runtime-cloudflare/src/run-entry.ts:645); the DO, Temporal and DBOS log the same failure at warn. - CF Workflows logs a failed
terminate()of a pending child that has not started at debug (runtime-cloudflare/src/workflow.ts:3521). The parent ends stopped, and that child then starts with no stop record and runs unstopped; the only trace is a debug line. A parent waiting on sub-agents whose wake-up was lost is bounded only bysubAgentEventTimeout(60 s by default): the per-interval timeout re-checks the in-memoryinterruptDetected, not the stop record. - CF Workflows' lenient
loadStatereads the stop withreadStop(...).catch(() => null)(runtime-cloudflare/src/steps.ts:643): a cleanup load reportsaborted: falseon a read failure without a trace (a debugging gap; it decides nothing terminal).
Fix direction: log the first three at warn (the runtime-js skip with the child and ref ids), log the fourth at warn, and re-check the stop record in the sub-agent wait's per-interval timeout.
FU-DO-STOP-SETTLE-DETAIL-AND-MARK: the DO's stop settle reads an abort's detail from the live session, and an unguarded mark write skips the hook — open (LOW, owner: WA)
Surfaced by: WA MR2 F4 silent-failures review M3, M4.
settleStoppedRun(runtime-cloudflare/src/do/durable-object-agent-base.ts:1363-1385) reads afailedrun'serrorDetailfromloadState(sessionId)when it runs, and its callers (:1297,:1354) pass noknownDetail. It runs as avoidafter anawait, so a new turn that has started on the session by then gives the hookfailedwith no detail, or a newer run's detail, instead of the abort'sframework_cancelled. The DO already hasstopAbortDetail(reason), andonStoppedRunSettledByEntrypasses the detail it knows. A throw beforesettleRun(theloadState) also skipsafterRelease, so the alarm is not rescheduled and the hook and cascade redelivery wait for an unrelated wake (logged at error asstop.settle_failed).cascadeStopwrites the cascade mark with an unguardedhost.kv.put(runtime-cloudflare/src/do/drive/settle.ts:453) insidesettleRun's try/finally. If the write throws,deliverHook(:373-377) is skipped and the error propagates (logged asdrive.settle_failed/stop.settle_failed); redelivery depends on a later wake.- The stop route's
already_endedbranch reads itserrorDetailfrom the current session too (runtime-cloudflare/src/do/drive/stop.ts:519-522; WA MR2 Task 12 deferred minor), so a newer turn can lend it a newer run's detail.
Fix direction: compute the abort's detail from the stop record it is settling (stopAbortDetail(stop.reason)) instead of the live session, and catch and log the mark write (a failed mark means the next pass re-runs the idempotent cascade), so the hook is still delivered.
FU-DO-STOP-DEADLINE-SETTLE-EDGES: the DO stop deadline can settle a run whose terminal it did not write, or leave a detached run unsettled — open (LOW, owner: WA)
Surfaced by: WA MR2 F4 silent-failures review M5; F4 detached-drive review M1, M9.
Three edges of endRunAtStopDeadline (runtime-cloudflare/src/do/drive/stop.ts) and its settle:
- A run that is still
runningwhose own K5 landed before an eviction makescommitStopTerminalansweralready_endedwith the committed status (:512-520). The step then deliversonRunSettledthroughsettleRun({ via: 'stop_deadline' })and returnsended, but the run status and its stream were never finished, sosettle.ts's precondition for thatvia("the caller has already written K5 and the stream terminal") is false. The status reported is truthful; the stream staysactiveuntil the wake'sfinish_committed. - A detach whose K5 did not land (
not_committed,:534) leaves the drive detached, so its own settle is a no-op (settle.ts:214-225). The drive's own resolution K5 then lands legitimately (the run is still live andallowKindlets it through), and the 1 s retry reads the run as not live and returnsnone. NoonRunSettled, cascade or Ruling 35 reset follows in this instance and no alarm stays armed; delivery waits for an unrelated wake or the next run-start's owed record. - The deadline delivers
onRunSettledwithsuperseded(:521,:640), while the DO otherwise never settles a superseded live run (FU-DO-SUPERSEDED-RUN-NO-SETTLE). The two disagree. - A permanent deadline failure keeps the DO waking (WA MR2 final review M14; Task 13 deferred minor). A run whose stop deadline fails permanently (
gave_upfor a paused or non-endable run) re-arms the alarm everyEXECUTION_WATCHDOG_RETRY_MS(30 s) indefinitely (durable-object-agent-base.ts:1432-1440)._stopDeadlineRetrylives in memory, so after an eviction the alarm fires at once, then every 30 s. The DO never goes quiet. Fix direction: a durable per-run "gave up" marker that stops the re-arm. - An R27 finish on the deadline's permanent path leaves a stale crash-loop record, and the entry-only callers of that path could skip the R27 check (Task 13 (D) deferred minor; from the ledger, not re-verified).
Fix direction: treat the first as none and let the wake's finish_committed finish and settle the run; remember the detached run id and settle it on the retry when it ended meanwhile; decide superseded settles together with FU-DO-SUPERSEDED-RUN-NO-SETTLE.
Done (DO drive settle 409 MR, design note 2026-10-17-do-drive-settle-409-design.md §3.1, ruling Q2): the deadline's drivePhase no longer reads the drive's settled flag. It is the run's truth (drivePhaseOf over the stop probe's executingRunOf, the same input as stage 1's ExecutingDrive.phase): a registered drive whose run has already ended or suspended in the store but whose handle.result() has not resolved reports tail, so a deadline that fires in that window answers in_tail and retries instead of detaching a drive that is ending its own run (and orphaning its settle). The listed edges are unchanged: the first (a run still running whose K5 landed) still reads executing, since the run record says so.
FU-DO-STOP-ENTRY-BOUNDARIES: a stop's reach over entries that have not committed, and a 503 after a recorded stop — open (LOW, owner: WA)
Surfaced by: WA MR2 F4 detached-drive review M5, M6, M7; the final review's M5 (agent-server).
handleStopanswers a typed 503 after a stop it already recorded when a later step throws (loadStatein corestopRun,getCurrentRunSyncinexecutingRunId), with no deadline alarm armed:rescheduleAlarmsruns only on astoppinganswer (runtime-cloudflare/src/do/drive/stop.ts:167-176,:243-250). The 503 body's premise is "nothing durable was promised", but the record is durable. A retry converges; a caller that gives up leaves a record-only stop, and on a paused run nothing arms an alarm. agent-server has the same 503-after-record: itsstopResponse(agent-server/src/http/handler.ts) answers a retryable 503 whenstopRun's laterloadStateorcommitStopTerminalread throws after the record (there is no alarm to arm on Node: FU-STOP-DEADLINE-NODE).- The spec's "an entry in flight that has not committed: its signal is cancelled" covers only a REGISTERED drive (
durable-object-agent-base.ts:1262-1275,cancelPendingEntry). Not cancelled: a request entry in admission (gate,materializeAgent,beforeEntry), aninterrupt_previousentry parked instopPrevious(up todeadlineAt + 5 s) and a wake drive before registration. The stop answers for the old run and the new entry commits afterwards. - A registered entry stuck BEFORE its run-start commit (a hang inside the Ruling 18
settleStoppedRuncascade, a hung consumer usage store) is never detached:registrationOf(durable-object-agent-base.ts:605-615) needs the entry's run to be current, andendRunAtStopDeadlinekeys on the current run. It holds the registration and 409s every entry up to the 4 h keepalive cap. Outside §3.4's "run" scope.
Fix direction: arm the alarm for any recorded stop whatever the answer (or return the 503 only before the record); have an entry re-check for a stop requested after its admission, or document the boundary; give the deadline a way to detach a registration whose run never became current.
FU-DO-DETACHED-DRIVE-HOOKS-AND-TERMINAL-SHAPE: a detached DO drive still fires agent hooks, and the deadline's terminal differs from a drive's — open (LOW, owner: WA)
Surfaced by: WA MR2 F4 detached-drive review M3, M4.
- A drive the stop deadline detached still fires agent-level hooks when its tool finally returns:
afterTool,onStateChange,onMessagefor a step that never commits, thenonAgentSuperseded(runtime-js/src/run-loop.ts:755-768, through the refused fenced K5) for a run the deadline endedinterrupted/failed. A Langfuse span then closesWARNING/supersededfor an interrupted run, and a tool that never returns closes no span. No store write. Fix direction: document it next todrive.detached_settle, or have the run loop reportalready_endedas its stop outcome rather thansuperseded. commitStopTerminalends the stream with no terminal payload (core/src/run/commit-stop-terminal.ts:337), while a drive's interrupt goes throughendStreamWithTerminalInfo(state,final_usage). The stuck LLM call's usage still lands after the K5 throughDOUsageStore.recordEntry(no run fence). Parity and observability only.
FU-JS-RETRY-POST-RUN-START-SEAMS-UNFENCED: runtime-js retry()'s failure seams after its run-start are not run-fenced — fenced in WA MR2 (Ruling 69); one narrow piece open (LOW, owner: runtime-js)
Surfaced by: WA MR2 F4 detached-drive review M2; the final review's I1 (an MR2 regression: before MR2 a stuck entry held the registration, so no newer run could exist).
- Fixed (Ruling 69). The two seams after
retry()'s run-start commit wrote without a run fence: the FIX BcompareAndSetStatus(['active'] → 'failed')andpersistRunFailure(a status CAS, an unfencedsaveState, a reactivated, unfencedfailStream,onAgentFail). A/retrydrive stuck indrainOrphanClientToolCallsand detached by the DO's stop deadline could, when it threw, flip a newer run'sactivesession tofailed. Both are replaced byfailRetryRun(runtime-js/src/js-agent-executor.ts), which ends the run through corefailCommittedRun: K5 → run status → stream, each fenced on the retry run, so a run that is no longer the session's current live run writes nothing.onAgentFailstill fires only forstate_history_incomplete/framework_cancelled, and only when this call ended the run. Test:runtime-js/src/__tests__/retry-post-run-start-fence.test.ts(red before the fix). No DO integ: the detached/retryinterleaving needs a stashed retry drain held in an approved tool plus a failure inside a newer run's commit-to-live window, so it is pinned at the runtime-js seam the DO runs. - Open: Ruling 19 is not applied to
retry()'s drain.retry()'s drain of a re-stashed input (drainOrphanClientToolCalls, W1 MR2b rule 5) carriesrefuseIfStopped, but its catch has noCommitStoppedErrorbranch. A stop recorded for the retry run between its run-start and that drain ends the runfailedwith the commit error's detail (now fenced, throughfailRetryRun), not by the stop's kind.resume()and the continuation entries route it throughendEntryRunByStop. Fix direction: the same branch inretry()'s catch, returningsettledHandle, with a unit test over a stashed retry (thestopAfterNextRunStartshape).
FU-DBOS-ABORT-EDGES: DBOS abort() can fire a second onAgentFail, and a persistent idle abort writes failed with no detail — open (LOW, owner: the DBOS bucket)
Surfaced by: WA MR2 F4 silent-failures review M7; F4-B review minors.
abortImpltreatsanswer.status === 'failed'as "this abort ended the run" (runtime-dbos/src/lifecycle/abort.ts:142), butstopRunalso answersfailedforalready_ended(a run that failed by itself between the record and the K5). The abort then firesonAgentFailwithAborted: <reason>after the workflow's own one. Narrow race; wrong hook payload. Fix direction: gate on the stop's own terminal (read the run's record and requireisRunEndedByStop, or use the commit outcome).- The one remaining
failedwrite of F4-B P2, a persistent loop idling between turns (markIdleSessionFailed,abort.ts:287-308), callscompareAndSetStatus(..., 'failed')with noerror/errorDetail; every other abort endsfailedwithframework_cancelled. Fix direction: pass the abort'sstopAbortDetail(the CAS takes anerroroption, and the session'serrorDetailneeds the detail as well). - Its branch for an owner probe that cannot read (
ownerProbe(owner).catch(() => 'unknown'),abort.ts:178-185: signal and mark failed anyway) has no test;persistent-mode-abort.integcovers the open loop only. - A race-path abort fires no
onAgentFail(WA MR2 Task 18 deferred minor; from the ledger, not re-verified). - When the first bullet is fixed, give the rule one home (final review M2): the DO decides it with
StopResult.endedHere(runtime-cloudflare/src/do/drive/stop.ts:176-180), DBOS withendedByThisAbort = answer.status === 'failed'. Consider corestopRunreporting whether its own K5 committed.
FU-CFW-COMPANION-TERMINATE-MISSING-INSTANCE: a CF Workflows terminate of a tracked child instance that does not exist is a tool failure — open (LOW, owner: WD)
Surfaced by: WA MR2 F4-B review minor.
stopUnstartedChild (runtime-cloudflare/src/steps.ts:7254) terminates a tracked child instance when the child has no session. F4-B S1 made a failed terminate() throw unless inspectCfwOwner reads the instance closed. An instance the engine no longer has (get() / terminate() answers not found, or inspectCfwOwner throws) reads as "not closed", so the callback throws and core answers a tool failure, rolls the ref back to running and reports "not recorded" for a child that is gone and has nothing to stop. Fix direction: classify a not-found instance as closed (liveness: 'closed'), with a unit case for a missing instance.
FU-COMPANION-TERMINATE-UNKNOWN-OUTCOME: a terminate whose stop cannot be confirmed rolls the ref back over a stop that may exist — open (LOW, owner: core)
Surfaced by: WA MR2 F4-A review minors 3, 4.
handleTerminateChild writes the ref terminated first and rolls it back when stopPersistentChild throws (core/src/orchestration/companion-tool-dispatch.ts:1386-1418). Two gaps:
- (minor 3) The rollback is a blind write from the pre-stop snapshot (
childRef.status,completionDelivered): a status the child or another writer wrote while the callback ran (the child completing, a concurrent sendMessage) is overwritten, and a failed rollback leaves the refterminatedwhile the tool reports failure (logged at error only). - (minor 4) A callback that cannot confirm a stop it may have recorded reports "not recorded". CF Workflows, DBOS and Temporal re-read the record after a throw (
childStopRecorded, and Temporal'sreadStopre-check), and a failed read proves nothing, so it throws. The DO'srecordChildStopDO(runtime-cloudflare/src/do/persistent-companion-tools.ts:824) has no follow-up read at all: a child DO that recorded the abort but answered after the 10 s bound, or a lost response, throws. The model is told the terminate failed and the ref goes back torunning, while the child is stopped and its result is never delivered.
Fix direction: distinguish "not recorded" (a definite refusal) from "unknown" in the callback contract; an unknown outcome keeps the ref terminated (the stop record wins) and the tool answers terminated: true with a warning, or a distinct retryable failure; make the rollback a conditional write (only when the ref still holds the value this call wrote).
FU-CFW-WITH-MESSAGE-ABORT-REFUSAL-RACE: CF Workflows' with_message abort refusal is decided from two reads — open (LOW, owner: WB)
Surfaced by: WA MR2 F4-B review minor (the m1 lost-race edge).
After the owed or running continuation wait, resume({ mode: 'with_message' }) reads the session (loadState) and then, separately, the current run and its stop record (endedByAbort, runtime-cloudflare/src/executor.ts:3157, :3553). The two reads are not one snapshot. A caller that lost a concurrent entry race, or whose wait ended on a continuation that failed on its own while an older abort is the latest stop record it reads, can be refused state_not_resumable with the "The agent has failed: use retry()" hint for an abort that was not the run it waited on. Fix direction: read the run the wait settled on (the continuation's run id) and refuse only when THAT run ended by an abort; otherwise treat the session as finished (the next turn).
FU-STOP-CASCADE-NON-BLOCKING-COMPANIONS: the stop cascade stops non-blocking companions only on runtime-js and the DO — open (LOW, owner: the CF Workflows, Temporal and DBOS buckets)
Surfaced by: WA MR2 F4 parity review m5.
A stopped parent run stops its live children on runtime-js (in-process children, persistent refs; remote ones when the parent was not executing) and on the CF DO (every live child through its route, with a durable mark). CF Workflows stops only the pending children of a sub-agent wait; Temporal and DBOS stop none. The per-runtime parity table documents the divergence (docs/internals/concepts.md, "Child cascade"), but CP-23 covers only blocking companions on Temporal and CFW; nothing tracks the non-blocking companion half or DBOS. Ruling 39 says a stop cascades to background companions. Fix direction: on each runtime, a stopped run's settle records an abort/interrupt on every live child ref of that run (stopPersistentChild per ref, the DO's rule), with the run attribution of FU-SUBSESSION-REF-PARENT-RUN-OTHER-STORES.
FU-HANDLE-STOP-DELETED-SESSION: a handle stop on a deleted session resolves on some runtimes and rejects on others — open (LOW, owner: orchestrator to assign)
Surfaced by: WA MR2 F4 parity review m7.
Core stopRun throws state_session_not_found for a missing session (core/src/run/stop-run.ts:137). The runtime-js handle answers not_stoppable with currentStatus: 'none' (only executor.stop throws), the CF Workflows abort() (runtime-cloudflare/src/executor.ts:1781) and DBOS resolve, while the CF Workflows / Temporal interrupt() (runtime-cloudflare/src/executor.ts:4739, runtime-temporal/src/executor.ts:1054) reject. An edge case (a handle used after deleteSession); the contract (concepts.md) does not say which. Fix direction: decide one answer (the JS handle's resolved not_stoppable / 'none'), implement it in the rejecting handles and pin it in the conformance suite.
FU-CFW-INTERRUPT-OVER-COMPLETION-HOOK: a CF Workflows run interrupted after onAgentComplete gets no later terminal hook — open (LOW; WH area)
Surfaced by: WA MR2 Task 17 review.
On CF Workflows a stop that lands after the run's onAgentComplete step and before its K5 ends the run interrupted (the K5 is refused, refuseIfStopped, and the stop's terminal commits), but no later hook reports it: the last terminal hook the host saw says complete. runtime-js fires onAgentFail in the same window (the last terminal hook is authoritative, as for complete → superseded). Fix direction: fire the stop's terminal hook after a stop resolution that follows onAgentComplete, matching runtime-js.
FU-STOP-PAUSED-DISCARDED-CALLS-NO-TOOL-END: a stop of a paused client-tool run writes no per-call tool_end — open (LOW)
Surfaced by: WA MR2 Ruling 53.
commitStopTerminal on a suspended_client_tool run clears pendingClientToolCalls, writes run_interrupted and ends the stream, but writes no aborted tool_end for each discarded call. A UI sees the calls end only through run_interrupted (and the next turn's C5 heal, which pairs them with not-executed results), so a client that renders per-call state shows them pending until reload or the next turn. Not a regression: main's runtime_restarted end was written by a later resume's drain, not by the stop. Fix direction: commitStopTerminal writes one discardedToolEndChunk per discarded call before run_interrupted, for a run that owns its stream.
FU-DO-SUPERSEDED-RUN-NO-SETTLE: a stranded live run a DO request entry supersedes gets no onRunSettled('superseded') — open (LOW)
Surfaced by: WA MR2 R2a fix round 3 (Ruling 61 review).
A DO run that is still live (running, nothing executing it after an eviction, with no stop record) is superseded by the next request entry's run-start commit. Nothing delivers onRunSettled for it with status superseded: its drive is gone, the wake ladder settles only the current run, and the owed-settle set (Rulings 60 / 61) records only runs whose session held their terminal (completed, failed, interrupted), because superseded has no terminal commit to snapshot. A host that counts settles per run sees that run never settle. Fix direction: the run-start choke point (commitRunStartSync) records owed(superseded) for a replaced live run (the commit's own status write), with no errorDetail, and the owed pass delivers it like the others.
FU-TEST-DO-COMPANION-SUPERSEDED-AFTERTOOL-UNREGISTERED: a CP-161 fix 5 test asserts no afterTool on a hook it never registers — open (LOW, test only; owner: the CF DO bucket)
Surfaced by: W1's read-only audit during the CP-301 fix (2026-11-14), checked by the owner at origin/main 3fee8e37bf; not CP-301's change.
packages/runtime-cloudflare/src/__tests__/do-companion-notifier.test.ts:1226 ("superseded while the spawn holds (its claim insert) …") puts afterTool on the agent's own hooks (:1256) and calls runtime-js runLoop directly (:1290) with no agentHooks. runLoop takes its hooks only from input.agentHooks (runtime-js/src/run-loop.ts:1093, else a no-op manager), so the hook is never registered and expect(afterTool).not.toHaveBeenCalled() (:1313) cannot fail. The test's other checks (the run ends superseded, no failed tool_end, nothing logged at error) still hold.
Fix: pass a manager built from the agent's hooks as agentHooks, and add a positive control in the same file: the same hook fires for a companion call that is not superseded, so the negative check can go red (mutate the regular-tool catch's rethrow and watch it fail). While there, check the other direct-runLoop tests that set an agent's hooks without agentHooks for the same pattern.
FU-DBOS-TERMINATED-CHILD-PARKED-RECV: a terminated DBOS companion parked in recv waits for its answer — open (LOW, parked; owner: the DBOS bucket)
Surfaced by: WA MR2 Task 18 review.
companion__terminateChild records an abort on the child's current run and sends the child's loop shutdown. A child whose run is parked in DBOS.recv(toolCallId) (suspended on a client tool) keeps waiting for that answer or its deadline before it reads the record and ends by the stop: DBOS has no stop deadline. Fix direction: the terminate also wakes the parked recv (a control message on that topic, or a cancel then a settle as the parent's stop does).
FU-DBOS-DRAIN-CHUNKS-BEFORE-STOP-READ: DBOS emits a drained call's tool_end before the wake's stop read — open (LOW, parked; owner: the DBOS bucket)
Surfaced by: WA MR2 Task 18 fix round 2.
At the DBOS main continuation site, each answered client call's tool_end is emitted at its own recv wake, before the wake-time stop read. A stop recorded meanwhile ends the run without committing those results (the drain append is refused), but their tool_end frames are already on the stream. Fix direction: read the stop before emitting, which needs a recorded step per call.
FU-TEMPORAL-STOP-ON-ENDED-RUN: a Temporal handle's stop records a stop on a run that has already ended — open (LOW, parked; owner: the Temporal bucket)
Surfaced by: WA MR2 final review M1 (and the Task 6 deferred minor "a run target records for any run id", whose runtime side this is).
Neither interruptRun nor abortRun (runtime-temporal/src/executor.ts, around :1031-1100) checks that the handle's run is still live before stopRunById → core stopRun with target: { kind: 'run' }. The store contract records a run target as given ("a stop for a run that never commits is inert"), so a stale handle writes a stop row on an ended run. A run that failed by itself and then gets a stale abort() reads as "ended by its stop" (isRunEndedByStop, core/src/run/stopped-submit.ts:55), and a late submit for its unpaired call answers run_stopped instead of the bare unknown call. docs/upgrade-guides/stop-request.md says "a handle on a run that already ended resolves and writes nothing", which holds on runtime-js, the DO and CF Workflows. Fix direction: the liveness pre-check of runtime-js/src/execution/in-process-runs.ts:196-198 (the DO's endedRunAnswer, CFW's abortRun check), with a unit case for a stale handle over a run that failed by itself.
FU-JS-STOP-ENTRY-EDGES: runtime-js stop edges around entries, reconnect handles and the cascade — open (LOW, owner: runtime-js)
Surfaced by: WA MR2 final review M6, M12; the Task 8 and Task 9 deferred minors. Items marked "not re-verified" were triaged from the ledger, not re-read at the fix head.
- A drain refused by a stop over a superseded run rethrows the raw error (M6).
endEntryRunByStop(runtime-js/src/js-agent-executor.ts) callssettledRunResult, which returnsnullfor asupersededrun (run-start.ts), so the caller gets the rawCommitStoppedError. Fix direction: a typedRunSupersededError(statussuperseded). - An approved tool runs before any stop read (M12).
drainOrphanClientToolCallsre-executes an approved tool with a signal that never aborts and reads no stop first. A stop recorded between the continuation's run-start and its drain still runs the tool's side effect; only the append after it is refused. Fix direction: read the run's stop before each re-execution (as DBOS does after FU-DBOS-DRAIN-CHUNKS-BEFORE-STOP-READ). - An abort that upgrades an interrupt after the soft path started leaves the
run_interruptedchunk and theonAgentFailpayload disagreeing; no test (Task 8; not re-verified). - An in-process child's late signal after its own suspension commit is not settled unless the parent cascades; the next entry settles it via
expected_run_stopped(Task 9). - Concurrent
resume()calls on one reconnect handle share theresumedRunbox (js-agent-executor.ts, the reconnect handle'sresume, around:3733-3784), so one call's stop can target the other's run (Task 9; still present). stopChildren's recursion uses the root's remote transports (deps.remoteTransportFor,execution/in-process-runs.ts), so a grandchild's remote child resolves through the root agent's tools (Task 9; not re-verified).- A delayed abort whose run id is undefined loses its kind (Task 9 fix round 2; not re-verified).
FU-JS-EXECUTE-HEAL-REMOTE-SUBAGENT-CALLS: execute()'s new-turn heal does not exclude running remote sub-agent calls — done (superseded by S-C, !412)
Surfaced by: WA MR2 Task 11 deferred minor (pre-existing; the ledger called it an FU candidate).
retry() and resume()'s trailing heal (C5) exclude a running remote sub-agent's call (a remote ref, isRemoteSubSessionRef, that is running), so the loop pairs it with the child's real outcome. execute()'s new-turn heal does not, so a new turn can pair a still-running remote child's call with a not-executed result. Fix direction: the same excludeIds as retry() (js-agent-executor.ts, planRetry's buildTrailingHealMessages call; execute()'s passes only the pending client-tool ids).
Done (S-C, !412, orchestrator ruling 2026-10-18), not by the fix direction above. A new turn now settles its remote children before the input (core settleRemoteChildrenBeforeInput with running: 'orphan'): a finished or lost child's result is committed before the message, and those calls are excluded from the heal (execute() passes the pending client-tool ids and remoteAtEntry.rows' call ids). A child still running is stopped (the D4 orphan rule) and its call keeps the heal's not-executed (unpaired_on_continuation) result, by design.
FU-COMPANION-STOP-REF-STATUS-EDGES: a companion's parent ref can disagree with its stopped run — open (LOW, owner: runtime-js)
Surfaced by: WA MR2 Task 8 and Task 9 deferred minors (from the ledger, not re-verified).
- A companion re-spawn over a stopped live run records the ref
failedeven when the run settledinterrupted. - A companion continue whose status write is lost, followed by a stop, leaves the ref
running. - No test covers the companion ref update when
stopChildrenends a companion that was not executing (execution/in-process-runs.ts, theref?.mode === 'persistent'branch).
Fix direction: derive the ref's status from the settled run (settledRunResult), and add the missing test.
FU-CFW-SUSPENDED-PARENT-INTERRUPTED-CHILD: a CF Workflows child that ends interrupted while its parent is suspended is never respawned or drained — open (LOW, owner: WD)
Surfaced by: WA MR2 final review M11 (the controller's Task 17 final-review note).
scanChildrenForRespawn (runtime-cloudflare/src/steps.ts) picks only failed: 'parent_suspended' and paused children. An awaited child that a direct stop ended interrupted while its parent was suspended is neither respawned nor drained on the parent's resume. Whether the parent's call still gets a tool_result is not established, and no test covers it. Fix direction: a CFW engine test (.workflow.test.ts) that interrupts an awaited child while its parent is suspended, then resumes the parent; pair the call with the child's interrupted outcome if it is not.
FU-CFW-STOP-READER-EDGES: two CF Workflows stop-reader edges — open (LOW, owner: WD)
Surfaced by: WA MR2 Task 13 (A) and Task 17 deferred minors (from the ledger, not re-verified).
- The CFW child consult builds its
finishedanswer from a session read taken before the reconcile, so it can report the pre-reconcile session. - After a stale wake-up, a stale completion waiter can delay a result by about 30 s.
FU-WA-MR2-STORE-AND-CORE-STOP-MINORS: small store and core leftovers of the stop record — open (LOW, owner: core / the store owners)
Surfaced by: WA MR2 final review M7, M8, M9; the Task 1, 3, 4, 5 and 7 deferred minors.
StopRequestSchemaaccepts any number forrequestedAt/deadlineAt(core/src/run/stop-request.ts, M8). Fix direction:.int().nonnegative()(a corrupt row is already permanent,parseStoredStopRequest).- Redis dead code (M7).
commitHeaderBasestill sendsstopsKey(store-redis/src/redis-state.ts), which no script reads (the Lua usesKEYS[7]);planPromotetestsif H.runId ~= ''twice in a row (store-redis/src/commit-script.ts). commitStopTerminal's logs (M9). Its warn logs the INPUT kind after an abort upgrade, and anot_committedis logged twice (core/src/run/commit-stop-terminal.ts, and again instop-run.ts).- Tests. No Postgres test races two
requestStopcalls; the contract (core/src/testing/stop-operations.ts) has no{ allowKind: 'abort' }commit over a stored interrupt (not re-verified). - D1
requestStopspells the live-current SQL two ways (cosmetic; not re-verified).
FU-DO-STOP-DRIVE-MINORS: DO stop diagnostics and storage leftovers — open (LOW, owner: WA)
Surfaced by: WA MR2 Task 12 and Task 14 deferred minors.
- The KV marks are never pruned. Per-run
__helix_settle_delivered:*and cascade marks grow DO storage indefinitely. Fix direction: prune a run's marks once its settle and cascade are delivered and the run is no longer the current one (or with the session's deletion). rearmrecords nolastFailurewhen its own-run read throws (runtime-cloudflare/src/do/drive/run-drive.ts,rearm→ownRunOf), so an exhausted crash loop loses its cause (diagnostic).stopRemoteChild's logs are namedpersistent_agent.stop.*(do/drive/stop-children.ts), notstop.*(cosmetic).
FU-WA-MR2-STOP-TEST-GAPS: stop paths with no test — open (LOW, owner: WA / runtime-js)
Surfaced by: WA MR2 deferred minors (Tasks 9, 13, 19; fix round R2a).
- runtime-js: a reconnect-resume handle's stop on a run that has already ended (Task 9).
- The DO: the DOAB 30 s
EXECUTION_WATCHDOG_RETRY_MSmapping (unit), andonReconcileFailure'sfinishedarm (do/drive/reconcile-wake.ts) (Task 13). - The
finishedarm offailCommittedRuncallers:do/drive/settle.ts,run-drive.ts,stop.ts, DBOSrun-commit.ts; and a core test of a K5 that lands between the R27 check and the commit (Task 13 (B)). - agent-server: the
stopViaHandlefallback's status mapping for a paused run (agent-server/src/agent-server.ts, Task 19). - The lost-class row of Ruling 62 (a request entry over an announced terminal whose commit is lost) (R2a).
FU-TEMPORAL-REPLAY-TEST-FIXED-WORKFLOW-IDS: companion-resume-replay fails on a long-lived Temporal server that holds a stale child — done (the replay run-nonce MR, stacked on the batch 28 owner records; owner: WA)
Surfaced by: WA MR2's final matrix: run against a local temporal server start-dev (Docker, TEMPORAL_ADDRESS), not against the per-job CI service. Pre-existing on main.
packages/runtime-temporal/src/__tests__/replay/companion-resume-replay.integ.test.ts records its history under fixed session ids (replay-cp103-parent). Its resume child workflow id is therefore fixed too: agent__replay-resume-parent__replay-cp103-parent__persistent__replay-cp103-parent-agent-helper__resume__0.
The failure. If a server still holds a RUNNING workflow under that id (for example from an earlier run that was cut short), the resume's startChild is refused:
- the log shows
persistent_agent.resume_failedwithWorkflow execution already started; - the test then fails
resumed companion never completed(its 15 s poll at:168-170).
Reproduction.
- Start a workflow under that id on an idle task queue, then run the file. It fails identically at main
6ab56e113aand at WA MR2a2daf249c3. - On a clean server, both pass (3/3 each).
Where it does not happen. CI (a fresh temporal service per job) and HELIX_TEMPORAL_BACKEND=test-server (ephemeral) never hold state from an earlier run.
Also hit by a local slot (2026-11-16). A slot that started one Temporal container and ran the replay/ files twice on it, once alone and once inside the full runtime-temporal integ suite, failed the second run the same way (Workflow execution already started, then resumed companion never completed). A driver that runs these files more than once starts a fresh Temporal for each run, or leaves replay/ out of the second.
Fix direction: suffix the test's session ids with a per-run nonce. The replay compares the recorded history against itself, so ids need not be stable across runs.
Closed (2026-11-16, the replay run-nonce MR): companion-resume-replay records its parent under replay-cp103-parent-<nonce> (companion-resume-replay.integ.test.ts:134), so the resume child's workflow id is new on every recording, and a new case (:214) records twice more on the same server, each completing under its own workflow id. Its other recordings take recordScenario's default nonce (FU-TEMPORAL-REPLAY-FIXED-IDS).
Raw error text on outward surfaces (FU-RAW-ERROR-TEXT-SURFACES) follow-ups
FU-RAW-ERROR-TEXT-SURFACES (filed with WA's DO companion MR A): no raw error text reaches a client or a model on any surface. MR 1 (WE-2) covers every client and remote-parent surface: all four stream managers sanitize on write, and the raw response fields (agent-server /status error, the DO's /snapshot error fields, the DO's superseded-submit answer, verbose INVALID_RESULT, ai-sdk's Express body, SSE builder error event and reloaded tool errorText, the CFW stream / lock DO 500 bodies) are sanitized. The DO's /status error is WA MR A's. The design, the 45-site audit and the per-surface rules: 2026-10-16-raw-error-text-surfaces-design.md. Upgrade: error-text-redaction.md.
FU-RAW-ERROR-TEXT-MODEL-SINK: a failed tool's error reaches the model raw — open (MR 2)
A thrown tool error, a failed sub-agent and a failed remote child are written into the model's history (the tool result message) with the raw text on all five runtimes (core createToolResultMessage / createSubAgentResultMessage in orchestration/message-builder.ts, the step iterator's sub-agent outcome, record-sub-session-result, and each runtime's own tool and remote-dispatch paths). A secret in a store or tool error is sent to the LLM provider.
Ruling (orchestrator, 2026-10-16): sanitize in the core message builders every runtime uses, with a new core redactCredentials (the credential subset of redactKnownPatterns: database URLs with credentials, JWTs, sk- / pk- keys, bearer tokens, password=-style pairs, card numbers). redactKnownPatterns becomes redactCredentials plus paths, IPs and emails, so the two cannot drift. Never sanitizeErrorMessage here: no 256 cap and no path redaction, so a model keeps "ENOENT '/workspace/src/a.ts'" and can recover. Guards per runtime (JS, DO, CFW, Temporal, DBOS): a tool throwing SECRET_ERROR_TEXT puts no secret in the persisted message or the model request, and a path survives.
FU-EPHEMERAL-ROW-TYPED-DETAIL-PARITY: only CF Workflows stamps a failed ephemeral sub-agent's typed detail on its row — open (LOW, owner: WA, after the CF backlog)
Filed by: DI-75 (review I3). Since DI-75, CF Workflows records a failed ephemeral (plain) sub-agent's row with the child's classified errorDetail in metadata[COMMON_METADATA_KEYS.ERROR_DETAIL], through core's client preset (sanitizeErrorDetail: code, category and retryable kept, the free text redacted): the first wave, the wait bound and the gamma-cascade respawn (runtime-cloudflare/src/workflow.ts, the recordSubSessionMessage calls). A reload's tool_end then carries the child's errorCode (core toolEndChunkForResultRow, steps.ts:3359). No other runtime stamps it on that row:
- runtime-js and the CF DO (it runs runtime-js): core
step-iterator.ts:1884-1890,createSubAgentResultMessagewitherroronly. - DBOS:
runtime-dbos/src/workflows/shared.ts:3600(local sub-agents) and:3612(remote), no detail. - Temporal:
runtime-temporal/src/workflow-dispatch.ts:403(the failed row), no detail.
Data point: runtime-js execution/awaited-children.ts:173 already stamps a detail on a related row (an awaited child's classified entry failure), RAW. Wanted: every runtime stamps the child's sanitized classified detail on a failed plain sub-agent's row (one core helper beside createSubAgentResultMessage), with a cross-runtime test. The row's error TEXT stays FU-RAW-ERROR-TEXT-MODEL-SINK's (its own redactCredentials ruling, never the client preset).
FU-TOOL-ROW-TYPED-ERROR-CODE: a regular tool's thrown error keeps only its message on the result row and its tool_end — fixed in code (batch 39)
Surfaced by: WE-2's trace for the b36 sandbox MR (2026-11-19). Related: FU-EPHEMERAL-ROW-TYPED-DETAIL-PARITY (the same mechanism on a failed ephemeral sub-agent's row), the structured terminal-failure spec (2026-07-28; terminal failures only).
- A tool that throws a typed
HelixError(workspace_command_timeout,workspace_unavailable, an app's own code) is recorded with its message only. On runtime-js and the CF DO (runtime-js/src/run-loop.ts:5459-5464, origin/main d6dd0022d7),createToolResultMessage({ success: false, error: toolError.message }). Coremessage-builder.ts:237-266writes the content andhelixToolFailed. - The row's
metadata[ERROR_DETAIL]is the only source oftool_end.errorCode(coretool-end-chunk-for-result-row.ts:52-57). Today only sub-session, remote-child and awaited-child rows write it (corerecord-sub-session-result.ts:308,remote-children-before-input.ts:292/:434; runtime-jsawaited-children.ts:116,js-agent-executor.ts:802; runtime-cloudflaresteps.ts:6762). So no regular tool's code reaches the stream, a reload or an app reading the log. The model still gets the message text. - runtime-js's failed
ToolExecutionResultdoes carryerrorDetail: toErrorDetail(rejected)(execution/tool-executor.ts:705), but only telemetry reads it (telemetry/run-telemetry.ts:273,:613, aserrorType), and coretoErrorDetail(errors/error-detail.ts:108-131) keeps acodeonly for aHelixError: a plainErrorkeeps its message,retryableand flags. - CF Workflows, Temporal and DBOS each build their own failed tool row: to trace per runtime.
- Fix direction: one core builder for a failed tool row that stamps the thrown error's sanitized detail (core
sanitizeErrorDetail's client preset: code, category, retryable kept, free text redacted, as DI-75's ephemeral rows), used at every runtime's tool site, sotool_endand a reload carryerrorCode. Whether the code also reaches the model's content is a separate design question.
Fixed in code (batch 39, origin/main 28e535ae00):
- Core
typedToolErrorDetail(core/src/orchestration/tool-error-detail.ts:41-59) records a thrownHelixError, or a coreWorkspaceErrorthat carries a stringcode, as its sanitized typed detail (sanitizeErrorDetail's client preset);typedToolErrorMetadata(:66-71) puts it on the failed row ashelixErrorDetail, andtoolEndChunkForResultRowreads its code throughtoolResultErrorCode(:74-77). Any other thrown value, a plainErrorwith a third party'scodeincluded, adds nothing (FU-TOOL-ROW-THIRD-PARTY-ERROR-CODE). - Every runtime's tool site stamps it: runtime-js and the CF DO (
runtime-js/src/run-loop.ts:5493), CF Workflows (runtime-cloudflare/src/steps.ts:6160), Temporal (runtime-temporal/src/activities.ts:2865,:3439-3440,:5539, andtool-result-messages.ts:68) and DBOS (runtime-dbos/src/steps/execute-tool.ts:561-562, carried to the workflow's row atworkflows/shared.ts:2731and:4526), the same commit. - The row's content and the
tool_enderror text are unchanged. - Conformance: tool.typed-error-on-row (executor tier) passes every id on js-memory and cfw-workflows-d1 in the owner's slot s-toolrow (2026-11-21) at 6e6f6a3843, on the fix branch's head 91fd2c3dfe. With the fix's product files restored to origin/main 60cb09f994, both lanes fail exactly .typed-detail-on-row, .typed-tool-end-code, .reload-parity and .workspace-error-code; .plain-unchanged and .model-content-unchanged pass. The cell's other executor lanes were not run in the slot (it is container-free).
FU-TOOL-ROW-THIRD-PARTY-ERROR-CODE: a regular tool's thrown error with a third party's code keeps only its message — open (LOW)
Filed by: FU-TOOL-ROW-TYPED-ERROR-CODE (ruling 2026-11-21: its option B, deferred). Since that fix, a failed regular tool row carries metadata.helixErrorDetail, and its tool_end carries errorCode, only for a typed error core owns: a HelixError, or a core WorkspaceError with a string code (core orchestration/tool-error-detail.ts, typedToolErrorDetail). A plain Error that carries its own code property records nothing. That covers a Node system error (ENOENT, ECONNREFUSED) and a provider or HTTP client error.
- Why deferred: those codes come from code the platform does not own (an open set, possibly changing between library versions). Recording them through core
toErrorDetailwould change the detail of every caller, not just tool rows. - Wanted, if a need appears: an opt-in that records a plain error's own
codeon the tool row only, guarded to a bounded shape (a string of at most 64 characters matching^[A-Za-z][A-Za-z0-9_.-]*$, else dropped), with nocategoryorretryable, on every runtime's tool site, with tests per runtime.
FU-CFW-RESPAWN-STOPPED-STEP-SINGLE-COPY-FIT: a stopped respawn whose one copy of the reason exceeds the step cap still records "Failed to re-dispatch sub-agent" — open (LOW, owner: WA)
Filed by: DI-75's Greptile round (!497; orchestrator, 2026-10-24). Since !497, CF Workflows' stopped-respawn step (runtime-cloudflare/src/workflow.ts, the …-respawn-stopped-… step.do) carries the child's failure text once. But it has no fit of its own: a stop reason whose ONE copy is over CFW_MAX_STEP_RESULT_BYTES (1 MiB) still fails the step at assertStepResultFits (through guardLostClaims), and the parent records Failed to re-dispatch sub-agent: … with no typed detail. Nothing caps a stop reason today (FU-STOP-REASON-API-CAP). Pre-existing, not DI-75's. Wanted: fit the stopped outcome inside the step, so an oversized reason degrades to the typed state_size_exceeded answer that fitChildOutcomeInStep already gives, with a test at just over the cap.
FU-CFW-BOUND-OUTCOME-FAILED-DETAIL-UNFITTED: boundOutcomeOf's failed case returns the raw message and detail inside a step with no fit — open (LOW, owner: WA)
Filed by: DI-75's Greptile round (!497; orchestrator, 2026-10-24). CF Workflows' boundOutcomeOf (runtime-cloudflare/src/workflow.ts, the wait-bound outcome) fits its completed case, but its failed case returns session.error and the session's raw errorDetail unfitted, inside the bound steps. The bound-step callers discard that result, but the step-result cap is checked on it anyway: an oversized child failure text can fail the bound step instead of answering a typed outcome. Pre-existing, not DI-75's. Wanted: fit the failed case as the completed one is (and give the parent the sanitized copy, core's sanitizeErrorDetail, as every other parent-facing copy), with a test at the cap.
FU-STOP-REASON-API-CAP: a stop's reason is an unbounded string — open (LOW, owner: WA; needs a contract decision)
Filed by: DI-75's Greptile round (!497; orchestrator, 2026-10-24). core's stopRequestSchema (core/src/run/stop-request.ts) takes reason as a plain z.string(), and every stop route, handle and companion terminate passes the caller's reason through unchanged (the CF executor's abort(reason); stopAbortDetail copies it). A very large reason then reaches every size-capped write that carries it (the CF Workflows step results above, a DO /status body, stream frames). Question to decide: cap the reason at the API boundary (a typed validation_error above N bytes, or a truncation with a marker), and at what N, on every runtime at once.
FU-TEMPORAL-DBOS-COMPANION-FAILURE-ROWS-SANITIZE: Temporal's blocking companion rows and DBOS's own spawn path carry a child's failure text raw — parked (Temporal / DBOS parked)
DI-70 (WA, spec 2026-10-20-di70-companion-failure-text-sanitized-design.md) sanitizes every copy of a persistent companion's failure text built for the parent inside core's builders (the park answer, every priorResult, the failure notice). The notice reaches Temporal and DBOS through core buildFailureMessage, so it is clean there too. Two parent-facing copies are each runtime's own and stay raw:
- Temporal's blocking companion results (
runtime-temporal/src/workflow-dispatch.ts:856,:1059,error: childResult?.error ?? 'child failed'): the child's raw text in a failed tool row, so in every later model call (itstool_end.erroris sanitized by the stream managers). - DBOS's bespoke spawn handler (
runtime-dbos/src/steps/execute-companion-tool.ts): its failed-park text was not traced.
Fix when the runtimes are unparked: answer through core companionSpawnResultFromTerminal (as runtime-js, the CF DO and CF Workflows do; this flips helixToolFailed on those rows), or at least pass the text through sanitizeErrorMessage. Guard: a blocking companion failing with SECRET_ERROR_TEXT puts no marker in the parent's history while the child keeps it.
CP-118 follow-ups (a stop terminal pairs the stopped run's unfinished calls)
CP-118 (a stop that refuses a post-commit result append left the call's tool_use unpaired, RM-29) is fixed on CF Workflows, runtime-js and the CF DO by two rules. Ruling O-73: a stop refuses NEW work, never the record of completed work, so the result row of a call whose work finished (a tool that ran, a child outcome that arrived, a drained client answer) is appended run-fenced but not stop-guarded, and new work (a tool, a child, the next step) is gated on the stop. The stop pairing: the stop's terminal pairs every trailing call with no result, except the pending client-tool calls, with a typed run_stopped not-executed result (core pairUnfinishedCalls) before its K5. DBOS has no window: a step's rows (assistant, tool, sub-agent, finishWith, companion results) land in one commitStepMessages, a refusal discards the whole step, and the later commits of a step whose suspension already landed are deliberately not stop-guarded; its resume drain rides the continuation's run-start commit. At a woken suspension's stop read it records the step's completed work first (FU-DBOS-APPROVED-RESULT-SURVIVES-STOP, done); a stop that lands after that read still drops it (FU-DBOS-STOP-AFTER-WAKE-READ-DROPS-COMPLETED-WORK).
FU-181-TEMPORAL-DBOS-STOP-ABORTS-CALL: a Temporal / DBOS interrupt never cuts the model call, so the reply kept is the complete one — open (DBOS; Temporal done; owner: the DBOS bucket)
Filed by: WE-2 with #181 (user ruling D2, 2026-10-28). Conformance: CP-16. .cut-at-stop of transcript.stopped-turn-partial-persisted was red on the Temporal lanes, ledgered under CP-16, and passes since the Temporal fix (WA's slot-cp16b); it passes on dbos-postgres (MA-43, owner slot s198).
Temporal half done (2026-11-17, CP-16, b31 origin/main 4d2f6e9afc): runLLMStep reads the run's stop record while the model call runs (core watchOwnStop; the call's signal is AbortSignal.any of the attempt fence and the watch, runtime-temporal/src/activities.ts:4221 there), so a stop cuts the call, and an interrupt keeps the cut reply (CF Workflows' rule: the cut result's own row, else the text the step streamed). Proof: WA's slot-cp16b (stop.mid-llm-call, usage.aborted-call-usage-reported and transcript.stopped-turn-partial-persisted pass on the three Temporal lanes, 3 of 3) and its targeted run of stop.record-only-mid-llm-call (3 of 3). What stays open here is DBOS's half and the Temporal abort race below.
- What happens: on runtime-js, the CF DO and CF Workflows a stop aborts the in-flight model call. #181 keeps the text the run produced up to when the server received the stop (the cut-stream reply), flagged
helixInterrupted. Temporal's interrupt is a signal, so the call runs to its end and the step ends at its commit, which the stop refuses. DBOS's record-only stop is read after the call, and a cancelled workflow is settled out of band. Both keep the COMPLETE reply, flagged: more text than the run had produced when the stop arrived. - Fix direction: the stop aborts the call's adapter signal (a heartbeat-tied abort on Temporal, like
llm-attempt-fence.ts; a stop poll inside DBOS'scallLLMstep). The partial must then be the text the step streamed, whatever the aborted call answers, as core's step iterator and CF Workflows' stop branch do: the adapter's cut-stream text, else the deltas the step forwarded when it answered an error or threw. Today an interrupt never aborts Temporal's call (activities.tspassesfence.signal, which only an activity cancellation aborts,llm-attempt-fence.ts:83-87), and only an abort cancels the workflow (executor.ts:1096), which keeps nothing (D4). So Temporal has no interrupt-with-error-answer path yet. - Temporal abort race (not fixed by #181): an abort cancels the workflow. A tool activity that is still running can stage its writes under
step-Nafter the stop terminal has discarded that step's staging (finalizeTerminalRun). The next retry entry'scleanupOrphanedStagingdrops them, but a new turn'sstep-Ncommit could promote them first.
FU-TEMPORAL-LLM-CALL-STOP-PATCH-GATE: CP-16's mid-call stop branch has no workflow patch, so a mixed-version fleet can break replay — open (owner: WA)
- Filed 2026-11-17 by the conformance owner, from WA's trace (2026-11-17), at the orchestrator's request. Related: CP-16, FU-181-TEMPORAL-DBOS-STOP-ABORTS-CALL.
- Where:
runtime-temporal/src/workflow.ts:1494-1503(origin/main 4d2f6e9afc): astoppedMidCallresult goes tostopAtBoundary(), which schedulesreadStopActivity(:453-454), with nowf.patchedgate; its siblings are gated (finishSiblings,recoverInvalidFinish,:1483-1488). - What breaks (WA's trace): old workflow code given a new activity's
stoppedMidCallresult recordscommitStep; a replay on new code schedulesreadStopActivitythere instead, a persistent non-determinism error. The comment at:1490-1493covers only an old activity with no field. - Interim: b31 shipped a deploy line in CP-16's changeset (
.changeset/temporal-llm-call-stop.md:17at origin/main 4d2f6e9afc, released in runtime-temporal 17.6.0's changelog): upgrade every worker of a task queue in one step. - Fix: its own
wf.patchedgate (the name chosen in the fix MR), plus a replay test of an old-workflow history over a new activity result; the deploy line goes when it lands.
FU-181-DBOS-SETTLE-KEEPS-RETRIED-ATTEMPT-TEXT: a DBOS interrupt during an LLM step's retry backoff keeps the failed attempt's text — open (owner: the DBOS bucket)
Filed by: WE-2 with #181 (r4's independent review, finding 2; ruled by the orchestrator, 2026-10-28). Conformance: no cell yet (no observable anchor, below).
- What happens: an LLM attempt streams text, then answers a retryable error, and a stop lands in the retry backoff, before the next attempt.
- runtime-js / the CF DO keep nothing. The step iterator's
awaitRetryends the step as an abort with no kept reply. Pinned byinterrupted-partial-error-answer.test.ts, "a stop during the retry backoff keeps nothing". - CF Workflows keeps nothing. Attempt 2's pre-call
state.abortedcheck ends the step before any model call orstep_start(steps.ts, the LLM step's abort check). - DBOS retries
callLLMStepitself (@DBOS.step({ retriesAllowed: true, maxAttempts: 3 … }), a newattemptIdper retry). A cancel during the backoff is settled by the cancelled-owner settle, whoseinterruptedPartialFromChunkskeeps the text of the run's LASTstep_startattempt: attempt 1's, the failed one.
- runtime-js / the CF DO keep nothing. The step iterator's
- Rule (orchestrator): an attempt that ended in a retryable error was being replaced, so a stop in its backoff keeps nothing, and every runtime must agree.
- Fix direction: DBOS's settle keeps nothing for an attempt that ended in a retryable error. For example, mark the retried attempt (a
step_discarded-style marker, or an attempt-ended record) before the retry, and have the settle skip such an attempt. - Why no cell yet (owner, 2026-10-28): DBOS writes each attempt's
step_startBEFORE its model call (call-llm.ts), so a cell that holds attempt 2 at its start already has attempt 2'sstep_start, and the settle keeps nothing there: DBOS passes. The divergence lives only in DBOS's own step-retry backoff, before attempt 2'sstep_start, which the harness cannot anchor (no race ledgers). Once DBOS marks a retried attempt, that marker is the anchor, and the owner writes the cell.
FU-181-RJS-KEEP-CRASH-CLEANUP-BOUND: a crash between runtime-js's kept-reply commit and the interrupt's cleanup leaves the stopped step's chunks — open (owner: the orchestrator)
Filed by: WE-2 with #181's integ sweep (CP-200's fix, 2026-10-28).
- What happens: on runtime-js and the CF DO, an interrupt keeps the stopped step's reply in an
appendcommit BEFORErun_interrupted(the pairing'stool_endframes must precede it, CP-118 M5) and so before the orphan cleanup. The cleanup is bounded by the checkpoint read before that commit (#181's CP-200 fix). A crash in the window between the kept commit and the cleanup leaves a later recovery's cleanup bounded by the latest checkpoint, the kept commit, which records the stopped step's count: that step's chunks stay, and an away reader is not told. - CF Workflows has no such window: it keeps its rows after the interrupt's cleanup (
prepareInterrupt). - Fix direction: bound a recovery's cleanup by the run's last checkpoint that is not a kept-reply
append, or record the pre-keep bound on the kept commit.
FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS: a step whose commit a stop refused pairs its completed tools run_stopped — promoted to RM-120 (2026-11-07)
Promoted: to the finding RM-120 (S2; owner and orchestrator ruling, 2026-11-07, after a live repro in an app's rehearsal on the CF DO before !754). RM-120 is open on CF Workflows, Temporal and DBOS, by reading: on CF Workflows a stopped step with only regular tools is discarded (InflightStepPolicy), so its completed call is paired run_stopped too. stop.mid-tool-keeps-finished on those lanes decides it.
Closed for runtime-js and the CF DO (2026-11-07, Save-First Tools MR 1, !754): the reply is saved before its tools start, so a stop after the save commits the step run-fenced, not stop-guarded (Ruling O-73), with every result that came back and its staged state (core commitStoppedSavedStep, core/src/orchestration/step-iterator.ts), except a call the stop cut (cutByStop: a cut wait is not completed work, CP-277), whose row is dropped; a call with no result is paired run_stopped. No tool starts before the save (every preparation, body and sub-agent dispatch follows the saved-calls commit), so a stop before it, or a refused save, keeps #181's reply with nothing started; a reply whose calls start no body (client calls or __finish__ only) is not saved at all. Open for CF Workflows, Temporal and DBOS until Save-First Tools reaches them.
Filed by: WE-2 with #181, from its independent review; ruled by the orchestrator (2026-10-28). Related: RM-29 (the pairing contract), Ruling 66 (step atomicity), O-73 (completed work).
- What happens: a stop that refuses a step's commit discards the step (Ruling 66). Since #181 an interrupt keeps the step's reply, and pairs every call of it
run_stopped, including a phase-1 tool that RAN (its side effect happened; its result row was part of the refused commit). The model may run it again on a resume. Before #181 the whole step vanished, so a re-run was possible then too. - What #181 did: the
run_stoppedtext no longer claims the call did not run: "Tool call result was not recorded: the run was stopped before it was recorded, and the tool may have run". - Open question: should a refused step's completed tool results be recorded (with their state writes, per O-73), against Ruling 66's step atomicity?
FU-NOT-EXECUTED-FRAMES-COMMITTED-STEP-ATTEMPT: not-executed frames that pair calls of COMMITTED steps carry no attempt or a fresh one — open (low; owner: the orchestrator)
Filed by: WE-2 with #181 r8 (the batch-3 tool_end@8 audit, 2026-10-29). Related: #181 r8, where a kept reply's run_stopped frames carry the STOPPED step's own attempt on every runtime; CP-118, RM-29, RM-91.
- What happens: several stop and failure paths pair calls left unanswered in COMMITTED history, and their
tool_endframes do not carry the attempt of the step that made the call:- runtime-js
appendStopPairing(stop-pairing.ts, CP-118: unfinished calls at the stop terminal) writes through the run-level writer: UNSTAMPED; - CF Workflows
recordDeferredNotExecuted(with_messagedeferred calls) and its run-stopped pairing at the deadline / K5 stamp a FRESHnewAttemptId(); - CF Workflows RM-91
pairOwedCallsOnFailure→stopPairingRows(no in-flight attempt) keeps the fresh-id fallback; - core
discardedToolEndChunk(a rewind's abortedtool_end, written at the entry) is stamped fresh by each runtime's entry writer.
- runtime-js
- Consequence (small):
- An UNSTAMPED frame fails the checkpoint-truth matrix's
chunk.attempt-stampedrule wherever a cell exercises it (runtime-js's stop pairing). - A fresh id is inconsistent with the step's own attempt.
- The UI is unaffected: ai-sdk's
filterSupersededAttemptsdoes not attempt-scope tool results, so atool_endpasses it through whatever its attempt. A replayed backlog keeps the frame, and nodata-attempt-supersededfires.
- An UNSTAMPED frame fails the checkpoint-truth matrix's
- Fix direction: a frame that pairs a call of a committed step carries that step's
committedAttemptId(from itsstep_committedmarker, or the checkpoint that committed the call). A matrix or conformance cell would check that every own content chunk after a stop carries an attemptId.
FU-181-RUN-STOPPED-TEXT-NEVER-STARTED: a kept reply's calls that provably never started read "the tool may have run" — open (design; owner: the orchestrator; same root as FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS)
Filed by: WE-2 with #181 (batch 3's test:cloudflare, the CF DO's interrupt-before-spawn case). Related: CP-106 (companion.blocking-spawn-stopped-during-wait-reason pins the run_stopped reason).
- What happens: core
interruptedPartialRowspairs EVERY call of an interrupt's kept reply with the onenot_executed/run_stoppedsynthetic (createNotExecutedToolResult). Its text is always "Tool call result was not recorded: … the tool may have run" (notExecutedErrorText). That is right for a step whose commit the stop refused (its tools may have run), but imprecise where the runtime KNOWS a call never started. Examples: the abort check before the tool batch, a call cut off by the stop, and the DO's interrupt before a spawn ran. - Constraint: the synthetic's KIND must stay
not_executed/run_stopped. Clients read it (the O1 contract), and CP-106's cell pins it for a stop during a blocking spawn's wait. - Fix direction: keep the kind and reason, and let the caller say whether the call started, for example an optional
started: falsedetail on the synthetic. The text then reads "the run was stopped before the tool ran" when known.interruptedPartialRowsgets that knowledge from its call site: the pre-batch abort and a cut call never started a tool; a refused commit may have. - 2026-11-08 (conformance owner): the queued-call shape (a concurrencyGroup or sequential call a stop skips before its body starts) is RM-123's, judged by
stop.queued-call-not-failed-text[-host](.stopped-text-not-may-have-run). The orchestrator scoped RM-123's text fix to cover a cutcompanion__waitForResulttoo (stop.mid-tool-keeps-finished-beside-wait-hostlogs the generic "may have run" text today). The other shapes stay here. - 2026-11-08 (RM-123): both shapes are fixed with the field this entry proposed,
helixSynthetic.started.- A call a stop skipped before its body started (a queued
concurrencyGroupcall, a sequential tool's next call) is pairedstarted: false: "the run was stopped before the tool started". - A cut
companion__waitForResultis pairedstarted: true: "the run was stopped while this call was waiting". - runtime-js / the CF DO know it, and core's step iterator and
pairUnfinishedCalls({ detailOf })carry it. - Still open here: an interrupt's kept partial reply (
interruptedPartialRows), the pre-batch abort, and the DO's interrupt before a spawn. - A companion call a stop cut after its announce, before its body began (its child phase not reached), is paired
started: falsetoo (runtime-js knows: a fresh announce notes it, the body's start replaces the note). - The wait text is proven at unit level (core
transcript.test.tspins the literal andstarted: true; runtime-jsstop-skipped-call-not-started.test.tspinsappendStopPairing's waiting row and frame). Its cells (stop.mid-tool-keeps-finished-beside-wait-textand-host) are withdrawn until FU-CONF-INPROCESS-COMPANION-BODY-ANCHOR; when they return and save-first is wired on CF Workflows / Temporal / DBOS, the executor one reds there on the text until this FU lands on that runtime (their stop pairings do not get the detail yet).
- A call a stop skipped before its body started (a queued
FU-CONF-INPROCESS-COMPANION-BODY-ANCHOR: no observable anchor for a companion wait's body starting — open (LOW; owner: WA, next after RM-123)
Filed: 2026-11-08 by WA (RM-123's cells slot, drv-c123).
- What happens: a cell that stops "once the wait has started" anchors on the wait's stored
tool_start, but the wait's body starts later: on runtime-js an in-process companion call is announced at the step's prepare and runs in the child phase after the step's commit; on the CF DO the override wait is announced at its prepare and its body (the late-start check, then the wait loop) starts a moment after. A stop landing in between is paired, correctly,started: false("not executed … before the tool started"), not the waiting text. In drv-c123 that window was hit once in nine js runs (the executor twin, js-redis run 1) and once in five cf-do runs (the-hosttwin, under the mg mutation). Both wait-text cells (stop.mid-tool-keeps-finished-beside-wait-textand-host) are withdrawn (no race ledger); the text is proven at unit level (FU-181-RUN-STOPPED-TEXT-NEVER-STARTED). - Design (orchestrator, 2026-11-08): a
beforeToolwitness on the bundled wait-parent agent (agents/save-first.tssfWaitParentAgent): the hook records a tool witness for the wait's call through the bundled agents' ports when its body begins, and the cells hold the stop until that witness. Still to check: thatbeforeToolfires on the DO override path (executeServerToolWithHooksfor a_companionDispatchOverridewait) and for an in-process companion (runtime-js's child phase). Both twins then return EXACT.
FU-DBOS-TOOL-SIGNAL-NEVER-ABORTS: a DBOS tool's abort signal never aborts on a stop — open (LOW; owner: the DBOS bucket)
Filed: 2026-11-08 by WA (FU-TOOL-STOPPED-RESULT-TYPED).
- What happens:
steps/execute-tool.tsgives a toolargs.signal ?? new AbortController().signal, and no caller passes a signal, so a DBOS tool'sctx.abortSignalnever aborts: no tool can stop cooperatively on a stop, and the user's ruling (a) / (b) (a tool rejecting with its signal's reason, or aToolStoppedError, is pairedrun_stopped) cannot trigger on DBOS. - What's owed: wire the tool step's signal to the run's stop record, classify the rejection with core
isToolStoppedRejection, and pair the callrun_stopped,started: true. The DBOSstop.tool-*cells are declared gaps naming this FU.
FU-SDK6-PRECOMMIT-ENTRY-HOOK-STOP: a RunStoppedError from afterSubAgent for a remote child's completion delivered as a session resumes is not honoured — open (owner: W1; post-features)
Filed by: W1 with SDK-6 (orchestrator ruling, 2026-11-10). runtime-js and the CF DO settle a remote child that finished while its parent was down at the parent's next entry, BEFORE that entry's run-start commit (js-agent-executor.ts fireRemoteCompletion, called from entryCompletions in executeEntry / resumeSession / retryEntry). There is no run to stop yet, so a RunStoppedError thrown from that afterSubAgent is not honoured (noRunStopHookManager, the not-honoured warn). It is handled as any throw there: the child's completion becomes its failure, afterSubAgent is re-fired best-effort with it, and the entry goes on. A hook that wants to stop the turn can throw from the resumed run's beforeLLMCall. A fix would carry the stop to the run the entry then starts.
FU-JS-REGISTER-RUN-AT-COMMIT: runtime-js registers a run for in-process stop signals only when its loop starts, so a Stop landing during a resume's drain aborts nothing in flight — open (owner: W1; post-features)
Filed by: W1 with SDK-6's D-i trace (orchestrator ruling, 2026-11-10). An entry's run is registered in this process's stop registry (inProcessRuns.register(runId, runController), js-agent-executor.ts:10465, inside runRegisteredLoop) only once its loop starts (runWithNewRunLoop). Every entry path that runs the run's hooks or tools before that comes after the run-start commit and the lease, but before the registration (lines at reliability-w1-sdk6-run-stopped-error 5b7cc1a344, before SDK-6's entry-path edits):
| point | run-start commit | startLease | the loop (registration) |
|---|---|---|---|
| executeEntry consumed answers :6341 | :6254 | :6316 | :6406 |
| resumeSession consumed answers :8257, drain :8267 | :7902 | :7989 | :8438 |
| retryEntry drain :8933 | :8850 | :8890 | :9009 |
| resumeLoop drain :10958 | :10896 | :10930 | :11030 |
An external Stop landing there answers stopping, because the probe reads the live lease, and records the run's stop row. But no signal is aborted, because the run is not registered yet. So an approved tool already running in the drain keeps its ctx.abortSignal un-aborted. The recorded stop is still honoured: the drain starts no later approved call (Ruling O-73), and the loop's pollStop ends the run before any model call. SDK-6's hook stop at these points uses the same recorded fact (stopTaken = the stop row). A fix would register the run at its run-start commit.
FU-TOOL-STOPPED-RESULT-TYPED: a tool that rejects with its own abort reason after a stop is recorded as a failed call, not as stopped — runtime-js / CF DO done (MR 1); CF Workflows: MR 2; Temporal: wired (CP-62); DBOS open under its stop follow-up (owner: WA; user ruling 2026-11-07)
Filed by: the conformance owner with WA's Save-First cells (RM-120 / RM-121 records). Related: RM-120, RM-121, RM-123, FU-181-RUN-STOPPED-TEXT-NEVER-STARTED; a reporting app's holdForStop, which rejects with ctx.abortSignal.reason.
- What happens: a tool whose body rejects after the stop fired gets a real failed result row (
helixToolFailed), whatever it rejected with. So a tool that does the cooperative thing (rejecting with the signal's own reason) reads "Failed" after a reload, beside calls the stop cut, which read stopped (not_executed/run_stopped). - Ruling (user ruling 2026-11-07, relayed by the orchestrator):
- (a) A rejection that IS the tool's own
ctx.abortSignal.reason, after the abort, is pairedrun_stopped. This is a default change, named in the release notes. Any other error after the abort stays a real failed result. - (b) Core exports an opt-in
ToolStoppedError, which is also pairedrun_stopped. - Both apply on every runtime.
- (a) A rejection that IS the tool's own
- First step of the MR: grep every bundled conformance agent for a rejection with
signal.reason/ctx.abortSignal.reason, and switch those to a non-reason error. Save-First'ssf_slowalready rejects with its own Error (packages/e2e/src/conformance/agents/save-first.ts); the reporting app'sholdForStopis the shape that rejects with the reason. - Cells: (a) gets its own sibling scenario, never a new id inside a scenario whose cells prove a fixed finding. Cells go on every lane, js and the CF DO first; the lanes not yet wired are declared gaps naming the phase that flips them.
- 2026-11-08 (WA, the orchestrator's split): MR 1 lands core (
ToolStoppedError, honoured only once the signal aborted;isToolStoppedRejection; thestarted: truepairing) and runtime-js / the CF DO, with the cellsstop.tool-rejects-with-own-reason,stop.tool-throws-tool-stopped-errorand the guardstop.tool-rejects-other-error-stays-failed(executor on the js lanes,-hoston cf-do). MR 2 (same batch) wires CF Workflows, whose tool signal does abort on a stop, and stops staging a failed or stopped tool's partial writes there (DI-89; the orchestrator's contract: discarded on every runtime). Temporal and DBOS: their tool signal never aborts on a stop (FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS, FU-DBOS-TOOL-SIGNAL-NEVER-ABORTS); their cells are gaps naming those. - Done for runtime-js and the CF DO (MR 1): a tool that stops itself on the stop (rejecting with its own
ctx.abortSignal.reason, or aToolStoppedError) is paired not executedrun_stopped, withhelixSynthetic.started: true; any other rejection stays a real failed result. Proven by thestop.tool-*executor cells on the js lanes x3 and their-hosttwins on cf-do (MR 1's slots: 39 checks OK and 177 checks OK, 0 failed). The lanes that honour it claim the'tool-stop-signal'flag capability. CF Workflows is MR 2. Temporal and DBOS stay under FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS / FU-DBOS-TOOL-SIGNAL-NEVER-ABORTS: their tool signal never aborts on a stop. - MR 2 (CF Workflows):
executeToolCall's catch classifies the raw rejection against the tool's own stop watch (watchOwnStop, which aborts with aStopAbortError) and returns the stop's own pairing row,run_stopped,started: true, with the onetool_endthat row's text (finishStoppedInBodyCall); a step committed under the stop keeps it and the stop's pairing finds the call paired. Its writes are discarded (DI-89). But a phase-1-only step a stop refuses is DISCARDED (InflightStepPolicy, RM-69), the common case, so the row never lands on CF Workflows yet: the cfwstop.tool-*cells stay gaps (no'tool-stop-signal') until Save-First Tools' CF Workflows phase (FU-CFW-DISCARDED-STEP-STOPPED-SELF-DETAIL; drv-c-ts2 found it at the step-level fix's head). - Temporal (CP-62's Temporal fix, 2026-11-17): the tool activity watches the run's stop record (core
watchOwnStop,activities.ts:2439-2441, origin/main 0aed8d1250, !913, batch 29) and classifies the raw rejection (:2682-2686, same pin): a tool that rejects with its signal's reason, or aToolStoppedErrorafter the abort, is not executedrun_stopped,started: true; any other error stays a real failure (tool-stop-signal.test.ts:95-146, same pin). An approved tool's typed row lands (the approve drain's run-fenced append,:167, same pin). A regular tool's row does not: the step commit refuses under the stop (refuseIfStopped), so the step is discarded, as on CF Workflows. The temporalstop.tool-*cells stay gaps and flip with RM-120's Temporal half. The abort case is FU-TEMPORAL-ABORT-RECORDS-STOPPED-TOOL.
FU-CFW-DISCARDED-STEP-STOPPED-SELF-DETAIL: a CFW stop discards a phase-1-only tool step, so a tool's own row (stopped itself, or failed) never lands — open (the common case; lands with Save-First Tools' CF Workflows phase; owner: WA)
Filed: 2026-11-08 by WA (tool-stopped MR 2). Raised the same day: drv-c-ts2 showed it is the common case, not an edge (the orchestrator).
- What happens: CF Workflows commits a tool step once, after its last result. When a stop refuses that commit and the step holds only phase-1 results (regular tools),
InflightStepPolicyDISCARDS the whole step (RM-69): the tool's ownToolCallResultrows go with it, and the stop's K5 (pairStoppedRunCalls) pairs every call from history with the defaultrun_stoppedtext ("the run was stopped before it was recorded, and the tool may have run"). So on CF Workflows a tool that stops itself on the stop (FU-TOOL-STOPPED-RESULT-TYPED) never showsstarted: true, and a tool that failed after the stop loses its real failed row. Only a step with later work (a sub-agent, remote, companion or phase-2 result) commits under the stop and keeps them. - Proven by: the cfw-workflows-d1
stop.tool-rejects-with-own-reason/stop.tool-throws-tool-stopped-error/stop.tool-rejects-other-error-stays-failedcells at the step-level fix's head (drv-c-ts2): one defaultrun_stoppedrow each, in every run. - What's owed: Save-First Tools' CF Workflows phase (spec §6: a step with an executable tool call commits its reply first, and its results commit in the step commit). When it lands, CF Workflows claims
'tool-stop-signal'(thestop.tool-*cfw gaps flip), and a stop no longer drops a finished tool's row.
FU-181-REFUSED-STEP-DOUBLE-TOOL-END: a refused tool step's ran tool gets two tool_end frames on the stream — open (owner: the orchestrator; same root as FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS)
Narrowed (2026-11-10, Save-First): on runtime-js and the CF Durable Object the refused step's ran tool is closed whenever the stopped step's own commit lands. A stopped saved step commits the ran tools' real rows with the stop's pairings in one step commit (commitStoppedSavedStep, core/src/orchestration/step-iterator.ts:3666-3745 at a220d5dfa8: appendMessages: [...kept, ...paired] at :3736, recordsCompletedWork at :3738), and the pairing runs over those kept rows (:3696), so a call with its real row gets no run_stopped row and no second frame. It is reached from the abort check after the tools (:3762) and from a refused commit's catch (:3799), so interrupt and abort alike. A tool that stopped in its own body writes no tool_end of its own (runtime-js/src/run-loop.ts:5262-5264), and no tool runs before the saved commit (Save-First). Still open on runtime-js / the DO: when that stopped step's commit itself fails, its catch (:3740-3742) returns only the pairings' frames as framed, the run loop hands them to the stop terminal (runtime-js/src/run-loop.ts:2715-2717, announced: stopFramed at :2731), and a ran tool whose kept row was lost with the commit is paired run_stopped there: its live tool_end with the real result is followed by an aborted one. Reachable only on a store fault or a fence at that commit. Nearest pin: runtime-js/src/__tests__/stop-framed-step-commit-lost.test.ts (its tool stops in its body, so it pins the unfinished call's one frame, not a ran tool's); no cell or unit pins the ran-tool residual. Fix direction (for whoever takes it): treat a kept row's call as framed, and have the stop terminal re-append its real row instead of pairing it. The DBOS / Temporal concurrent-settler half below stays open.
Narrowed (2026-11-06, CP-280): the runtime-js in-process delegated-call path (WA's 2026-11-05 sighting) is closed by CP-280's fix (!745): an in-process call cut by a stop writes no terminal frame of its own, so the stop pairing is the one writer, held by subagent.stop-cut-call-live-text-host on js-chat-host and the fix's no-cut-rule mutation. The refused step's ran tool and the DBOS / Temporal concurrent settlers stay open.
Narrowed (2026-11-06, CP-277): the DO remote-call sighting is CP-277's, closed by its fix (!737): a remote call whose wait the stop cut writes nothing of its own, so the stop pairing is the one writer, held by subagent.stop-cut-call-one-terminal-frame-host on cf-do. The refused step's ran tool and the DBOS / Temporal concurrent settlers below stay open.
Filed by: WE-2 with #181's independent-review round (orchestrator, 2026-10-28).
- What happens: a tool that ran in a step whose commit a stop then refused already wrote its live
tool_end(with its real result) to the stream. An interrupt keeps the step's reply and pairs that callrun_stopped, writing a second, abortedtool_endfor the same call before its commit. The persisted row is therun_stoppedpairing; a live client sees the real result, then the stopped one. - Same root as FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS: what a refused step records for the tools that ran. Settle both together.
- Known limit, concurrent settlers (Greptile ac3acd21f9 on !576): on DBOS (two settlers of one stopped run) and on Temporal (a timed-out attempt racing its retry), the keep's version CAS lands ONE flagged reply in history, but each settler writes the kept rows'
run_stoppedtool_endframes once, before its append. So a live reader can see the pairingtool_endtwice. This affects only the live stream; history is unaffected, and a replay of the snapshot shows one. - Sighting, a delegated call (WA, 2026-11-05): a stop while a delegated child runs writes two
tool_endframes for the call on runtime-js: an aborted one from the dispatch, then the pairing's. The row isrun_stopped. Whether theannouncedframes add a third reading is unproven.
Narrowed (2026-11-07, Save-First Tools MR 1, !754): on runtime-js and the CF DO a stopped saved step keeps its ran tool's real result (FU-REFUSED-STEP-COMPLETED-TOOL-RESULTS, closed there), and a call whose tool_end the stopped step already wrote, when the step's commit did not land, gets no second frame at the stop's pairing or at the K5 (runtime-js/src/run-loop.ts, the stopFramed set fed to both pairings' announced). It is pinned on the iterator side (core step-iterator-delegation.test.ts, "the step commit does not land: the outcome names the framed call") and, since 2026-11-09, through the real runtime-js executor (runtime-js/src/__tests__/stop-framed-step-commit-lost.test.ts: the stopped step's commit fails once, and in a second case the stop pairing's append fails too, so the K5 pairs; each keeps one row and one tool_end; removing stopFramed from the stop pairing fails both cases, from the K5 only the second). subagent.stop-cut-call-one-terminal-frame-host on cf-do covers the COMMITTED path only. CF Workflows, Temporal and DBOS stay open.
FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS: a Temporal stop refuses the record of completed work and leaves calls unpaired — open (parked; owner: the Temporal bucket)
Surfaced by: CP-118 (survey §3/§5); widened by Ruling O-73 and the approve-drain half of FU-STOP-APPROVE-DRAIN-DROPS-EXECUTED-RESULT (folded in here).
Narrowed (CP-119, the Temporal fix, origin/main 0aed8d1250, !912, batch 29): the approve-path result append (applyResultsAndReload, activities.ts:6846-6852) now records an approved tool's executed result run-fenced, not stop-guarded, and an approved tool no longer starts once its run has a stop (:6818-6821: it stays pending, and the drain ends the run by the stop, :7179-7180). What stays open here (CP-332's Temporal half, !918, also changed these terminal paths):
- the stop terminal's pairing:
commitTerminalState(activities.ts:8602) runs no corepairUnfinishedCalls(run_stopped, pending client-tool ids excluded) before the K5; - the workflow's appends of
runPhase2FinishWith's and the deferredfinishWith/__finish__results (workflow.ts:1254,:2019), through the stop-guardedappendMessagesactivity (activities.ts:8135); - the resume drain's answered-call rows (
applyResultsAndReload's gated append,refuseIfStopped,activities.ts:6665), completed work under O-73; - the abort fix, FU-TEMPORAL-ABORT-RECORDS-STOPPED-TOOL (CP-62's Temporal fix wired the tool's abortSignal).
Abort gap (explicit): abort() cancels the Temporal workflow (executor.ts:1234, if (stop.kind === 'abort') await handle.cancel();). The workflow commits the stop's K5 while an approved tool still runs in the worker, so that tool's late result is fenced off (the run has ended) and the next turn heals the call not executed (unpaired_on_continuation). A late result is lost under abort until (4) lands. The CP-119 integ pins this case (approve-result-survives-stop.integ.test.ts:342, origin/main 0aed8d1250, "abort() on Temporal: the held tool's signal fires, but the workflow ends framework_cancelled first …") and must flip with (4). CP-119 stays open for it.
Temporal commits a step's server-tool, sub-agent, remote and companion results with the assistant row in one commitStep, but these appends follow a step commit and are refuseIfStopped: runPhase2FinishWith's result append after commitStep (workflow.ts ~1580-1601 → activities.appendMessages, activities.ts ~6893), the deferred finishWith append (workflow.ts ~1143; the activity reads the stop first, a stop after that read refuses it), the resume drain rows (activities.ts ~5501) and the approve-path result append (activities.ts ~5704; the outer catch at ~5720 swallows the CommitStoppedError, so an approved tool that RAN loses its result and its pending entry is cleared). A stop in the window refuses the append, the run ends by its stop (commitStoppedOf, stopTerminalSession), and the call's tool_use stays unpaired until a new turn's C5 heal. Fix direction, as CF Workflows and runtime-js do: (1) O-73 — append each completed result run-fenced, NOT stop-guarded (drop refuseIfStopped at those sites, keep the run fence), and read the run's stop before starting an approved tool or a finishWith (with a stop: do not start it; an approval stays pending); (2) pair at the stop terminal (the commitTerminalState activity's stop branch) with core pairUnfinishedCalls(history, { reason: 'run_stopped', excludeIds: <pending client-tool ids read before the K5 clears them> }) before the K5, in one append commit under the stop's { allowKind }, with the aborted tool_end frames first, nothing once the K5 landed (CF Workflows' pairStoppedRunCalls).
Also a site (finish-siblings !670, Greptile on !672): the deferred __finish__ record: its acknowledgement comes back from runPhase2FinishWith (deriveDeferredFromTranscript) and is appended through the same activities.appendMessages (workflow.ts ~1167, refuseIfStopped), so a stop between that activity and the append refuses the record of a finish that was honoured. runtime-js (recordsCompletedWork) and CF Workflows (appendDeferredFinish → appendCommitted(…, { recordsCompletedWork: true })) record it run-fenced. Also (FU-TOOL-STOPPED-RESULT-TYPED, 2026-11-08; wired 2026-11-17 by CP-62's Temporal fix): each tool activity now watches the run's stop record and aborts the tool's signal on a stop, and a tool that stops itself is not executed run_stopped, started: true. A regular tool's step commit still refuses under the stop, so its row is discarded with the step; the stop.tool-* temporal cells are declared gaps naming RM-120's Temporal half.
FU-TEMPORAL-ABORT-RECORDS-STOPPED-TOOL: a Temporal abort ends the run before its stopped tool's typed row is recorded (CP-62 (B), CP-119) — open (owner: WA)
Filed: 2026-11-17 with CP-62's Temporal fix (the tool signal half, CP-62 (A)). Related: CP-62, CP-119, FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS item (4), FU-TOOL-STOPPED-RESULT-TYPED.
- The gap: CP-62 (A) delivers a stop to a running server tool: core
watchOwnStopruns in the tool activity and reads the run's stop record everystopPollIntervalMs(default 500 ms;activities.ts:372-375, origin/main 0aed8d1250, !913, batch 29). Butabort()cancels the workflow (executor.ts:1234, same pin), whose K5 lands at once, while the watch aborts the tool up to one poll later. So the stopped tool's typedrun_stoppedrow is fenced off by the ended run, and the next turn heals the callunpaired_on_continuation. Pinned byapprove-result-survives-stop.integ.test.ts:342(same pin), which asserts the signal fires and the row is fenced off. - Seen in CI: !918's MR pipeline 3081382355, job 18124090562 (
test:integ:rest, allowed to fail):approve-result-survives-stop.integ.test.ts's abort pin failed "the ended run's late result is fenced off: expected … to have a length of +0 but got 1". Measured locally (W1, 2026-11-17, on main's code too): the K5 lands about 50-75 ms afterabort()and the typedrun_stoppedrow's run-fenced append about 350 ms later, so the pin holds only by that margin; under CI load the row landed under a still-live run. The pin is timing-dependent until this FU's fix. - Seen again (2026-11-19): !931's MR pipeline 3087168720,
test:integ:restjob 18170781628 (a disclosed retry of that job's earlier failures; !931 is records only, so this is main's code): the abort pin failed the same way atapprove-result-survives-stop.integ.test.ts:374(origin/main fae53b26ac), "the ended run's late result is fenced off: expected [ { role: 'tool', …(4) } ] to have a length of +0 but got 1". runtime-temporal readTest Files 1 failed | 45 passed | 2 skipped (48),Tests 1 failed | 294 passed | 11 skipped (306); every other package in the job passed. Not retried (orchestrator ruling: the MR lands through its batch's pipeline). - Fix direction: the tool activity races the tool against the watch; once the watch fires it records the typed row (
createNotExecutedToolResult(call, 'run_stopped', { started: true }); for an approved tool a run-fenced append, as completed work) and returns, without waiting for a tool that ignores its signal. The workflow's abort waits for that activity before its K5: the tool and drain activities heartbeat and are proxied withcancellationType: WAIT_CANCELLATION_COMPLETED, the wait bounded by the watch / heartbeat interval. - Replay: this changes when the workflow resolves the activity await, so it goes behind an input flag set by the executor on root inputs and carried to child args (as RM-74's is); unflagged workflows keep today's proxy.
- Also: wire
runPhase2FinishWith's tool signal, which (A) leaves a fresh, never-abortedAbortController(activities.ts:4998, same pin). - Tests: an integ with a never-resolving tool (the abort settles within the bound, the typed row kept); the
:342pin flips to the kept row;stop.mid-tool-ignores-abortstays green; replay tests for flagged and unflagged histories; mutations (cancellationTypereverted, the flag never set). - Open for the builder's plan (O-73): a tool that ignores its signal and finishes after the race returned: is its late real result recorded, replacing the typed row, or dropped?
- Pin relaxed until this fix (2026-11-19): the abort case failed the same way on both
test:integ:restruns of batch 33's MR pipeline 3088897524 (jobs 18180827594 and its disclosed retry 18187261188). Until this FU's fix, the case accepts either outcome for the stopped tool's typed row: fenced off by the ended run (the next turn heals the callunpaired_on_continuation), or kept asrun_stopped(started: true) when its append lands while the run is still live. Every other assertion is kept, the tool's signal firing included. This FU's fix makes the case assert "kept" only.
FU-DBOS-APPROVED-RESULT-SURVIVES-STOP: DBOS's approval continuation drops an approved tool's executed result when a stop ends the suspension (CP-119) — done (!890, batch 27; owner: WA)
Surfaced by: the conformance cell approval.approved-tool-result-survives-stop (CP-119). It corrects the CP-118 survey's "DBOS has no window" for the approve path.
What happens: a session suspended on an approval is approved, and DBOS's continuation executes the approved tool. A stop recorded for that run (core requestStop, the run-scoped StopRequest) while the tool runs is honoured: the continuation reads the suspended run's stop right after the wake and ends the turn interrupted (runtime-dbos/src/workflows/shared.ts:4258-4259 at 9ca43ae3ee). But endSuspensionByStop clears the staged step rows first (:2091-2092), so the approved tool's executed result is never committed (Ruling 21 there, which Ruling O-73 supersedes for completed work). Traced by WA, read by the owner.
Corrected 2026-11-15: this entry used to say the stop was IGNORED and waited on CP-120. CP-120 is withdrawn: the "ignored" stop was the conformance responder resuming the stopped session itself (FU-CONF-RESPONDER-RESUMES-AFTER-WOKEN-RUN-ENDED). With that fixed, the cell fails only CP-119's two result ids on dbos-postgres, ledgered under CP-119.
Fix direction: apply Ruling O-73 there, as CF Workflows and runtime-js do:
- append the executed tool's result, and its staged state writes, run-fenced but NOT stop-guarded;
- read the run's stop before starting the approved tool; with a stop, do not run it, and the call stays pending.
Then flip the CP-119 cell on dbos-postgres.
Closed (2026-11-15, !890, batch 27): at both wake stop reads (the in-step continuation and the restored-calls takeover) DBOS now records the suspending step's completed work before the turn ends by the stop (recordCompletedWorkAtStop, runtime-dbos/src/workflows/shared.ts:2178-2235 at origin/main c72fe62f8f): an executed tool's real result with its staged state, a denial, a real client answer the wake drained, a finished regular tool or sub-agent, in ONE step commit that is run-fenced and not stop-guarded; then onMessage, and afterTool for an executed tool. A suspending call's lapse and an approved tool the stop kept from starting are not completed work (isCompletedWork, :1489-1499); a regular tool that ran and failed keeps its failure row whatever its error text. The approve branch reads the run's stop before executeTool (:1371-1391); with one the tool does not start and its call stays unpaired (Ruling 56) for the next turn's heal (C5) or new-turn settlement (RM-81). Rule R holds: the commit promotes the step's whole staging, and no excluded call staged a write (a tool stages only after its body returns). The record uses the step commit's completed-work mode (steps/run-commit.ts:1724-1760): the plain step commit is keyed on (run, stepCount), which the step's own suspension commit already holds, so it would write nothing. Tests: approved-result-survives-stop.integ.test.ts (interrupt and abort), seven units in run-agent-loop-one-turn.test.ts and four in commit-checkpoint-truth.test.ts. The CP-119 cell leaves its ledger on dbos-postgres. The window between the wake's read and the continuation's run-start stays open: FU-DBOS-STOP-AFTER-WAKE-READ-DROPS-COMPLETED-WORK.
FU-CONF-RESPONDER-RESUMES-AFTER-WOKEN-RUN-ENDED: the conformance responder resumed a session whose woken run had ended by a stop — done (!889 for an ended run, !894 for a live one, batch 27; owner: WA)
Surfaced by: CP-120 (WA's slots p120 and p120b at origin/main 9ca43ae3ee, read by the owner). On a backend whose submit wakes the waiting run (clientToolSubmitWakesRun, of the lanes only dbos-postgres with clientToolContinuation: 'resume'), the executor driver's client-tool / approval responder followed every answered round with its own resume({ mode: 'continue' }) (e2e/src/conformance/drivers/executor-driver.ts:811 and :1297-1299 at 9ca43ae3ee), whatever the submit had answered. When the woken run ended by a stop it honoured, that resume started a new turn over the interrupted session; the turn completed, and the chain reported its outcome as the run's, so the stop looked lost. CP-120 ("DBOS ignores a recorded stop") was filed from that and is withdrawn.
Closed: on a waking backend, a round whose last accepted submit answered new_run or deferred (CP-104: the backend started or carries the continuation) AND whose wake wait read the woken run ENDED (completed, failed or interrupted, from the same read that ended the wait) is followed by re-attaching to that run through executor.getHandle, which reports its own outcome; the chain never resumes it. Every other case continues as before, a woken run still live included. For the CP-120 cell this is deterministic: DBOS clears an answered pending entry only in the continuation's run-start (runtime-dbos/src/steps/run-commit.ts:1049-1059 at 9ca43ae3ee), which a stop at the wake never reaches, and the stop's K5 clears it together with the interrupted status (:2030-2040). Units in executor-driver.test.ts: new_run and deferred over a run that ended by a stop re-attach and report its outcome with no resume; negative controls: a woken run that clears its entry while live still resumes, caller_resumes still resumes, and a non-waking backend still resumes on new_run.
Closed, the wider version (2026-11-15, !894, batch 27): the ended-run condition is dropped. On a waking backend, a round whose last accepted submit answered new_run or deferred is followed by re-attaching through executor.getHandle, live or ended, and never by a resume of the chain's own, as CP-104 has a caller do. It changes only dbos-postgres: cfw-workflows-d1 (clientToolContinuation: 'auto') already re-attached on every round. It needed DI-78's fix, in the same MR: DBOS's re-attach handle now decides from its run record and waits while the run is live. The three cells WA's slot-a saw fail under this version (approval.approved-tool-throw-recorded-failed, history.branch-at-suspended-checkpoint-paired and transcript.co-emitted-subagent-dispatched) are DI-78's proof. The unit that was the negative control (a woken run that clears its entry while live) now pins the re-attach.
FU-CONF-DBOS-RECONNECT-AFTER-RESTART-CELL: DI-78's restart face has no cell of its own — open (LOW, test only; owner: the conformance owner)
Filed: 2026-11-15, with DI-78's fix. DI-78's title names three faces of one defect: a run recovering after a process restart, a run paused on its suspension, and a continuation a client-tool submit started. Its cells cover the last two (the responder's re-attach, three cells on dbos-postgres), and DI-56's cell a running turn. A run that recovers keeps its runId and stays running, so the fixed result() takes the same branch (it waits, or reports the run's K5), and the record is fixed on that path. A cell that reconnects to a run a simulated restart recovered would prove it directly. Wanted: a dbos-postgres executor cell that restarts the runtime mid-run, calls getHandle(...), and asserts result() waits for the run's real outcome; a declared gap on the lanes that have no restart control.
FU-CONF-APPROVAL-LAPSE-TOOL-END-CELL: an approval that lapses has no cell for its tool_end — open (LOW, test only; owner: the conformance owner)
Filed: 2026-11-16, with DI-95's fix. approval.denied-call-ended covers a denial on every executor lane. An approval that LAPSES (no answer before its deadline) takes a different exit on DBOS (DI-95's fourth), and no cell reaches it: it needs a bundled agent with an approval-gated tool and clientToolTimeoutMs. DBOS's lapse face is covered by the runtime-dbos integ (approval-not-run-tool-end.integ.test.ts). Wanted: that agent (with its fingerprint, provenance and registry pins) and a lapse leg in the cell, judged like the denial: exactly one tool_end for the call, carrying the row's outcome, before the next step, on every executor lane.
FU-CFW-COMPLETED-WORK-ROW-AND-STATE-ATOMIC: a CF Workflows tool's result row and its staged state write are two commits — open (LOW; owner: the CF Workflows bucket)
Surfaced by: CP-118 fix round 2 (Ruling O-73).
On CF Workflows, a finished tool's record is two separate run-fenced commits:
- its result row:
steps.appendCommitted(…, { recordsCompletedWork: true }); - its staged customState writes:
mergeStagedIntoStatefor phase-2 / deferredfinishWith, and the approve drain's finalwriteEntryState, both throughwriteEntryState, run-fenced and not stop-guarded.
Neither is stop-guarded, so a stop cannot split them.
The window: a CRASH or eviction of the workflow instance after the row's commit and before the state commit (or the reverse, at sites that write state first). The transcript then shows the tool's real result while customState lacks its writes. runtime-js / the CF DO do not have the gap, because a deferred finishWith's state rides its unguarded step commit.
Fix direction: a store commit that carries both the result rows and the tool's staged state promote atomically, on all 5 stores:
- either
appendMessageswith a staged-promote option, or a new completed-work commit kind; - with its contract cases in
core/src/testing; - the CF Workflows result sites then use it. MR-sized, because of the store-contract change.
FU-FAIL-COMMITTED-RUN-PAIRS-OWED-CALLS: a run that crashes outside its loop leaves its owed deferred calls to the next entry's heal — open
Owner: unassigned (reliability orchestrator to route). Finding: RM-91 (its stated residual). Surfaced by: W1's RM-91 fix.
RM-91 pairs a FAILED run's owed deferred calls (a resumed run's finishWith, CF Workflows' companion calls and re-spawned children) unpaired_on_continuation before its K5, inside the run (runtime-js / the CF DO onFailed, CF Workflows' catch-all). A run that crashes OUTSIDE its loop is ended by core failCommittedRun (a later entry or reader, the CF DO's wake), which commits the terminal with no pairing, so the failed session keeps the calls unpaired until the next entry's C5 heal (a new turn or retry()). To close: failCommittedRun pairs the trailing unpaired calls but the pending ones before its K5, as the stop settle should (FU-STOP-SETTLE-PAIRS-UNFINISHED-CALLS). Temporal and DBOS do not pair at a failure at all yet (the owner's catalogue ids, parked).
FU-JS-PAIRING-FRAMES-BEFORE-FENCED-APPEND: a superseded run may stream not-executed frames for calls the new owner runs — open
Owner: unassigned (reliability orchestrator to route). Finding: none (a narrow stream window). Surfaced by: the orchestrator's review of !465 (RM-91).
appendStopPairing (runtime-js/src/stop-pairing.ts, ~:95 then ~:117) writes each pairing row's not-executed tool_end frame BEFORE the run-fenced append (R23: a commit's frames precede it, so its cursor covers them). A run superseded between the new run's run-start commit and its claimStream (CP-85 drops another run's chunks only once the stream is claimed) has already streamed those frames when its append is refused, for calls the new owner may still run; the client then sees a call end "not executed" and later its real result. The CP-118 stop path and the RM-91 failure path share the order. Fixing it means either writing the frames after the commit (against R23) or a stream fence that holds from the new run's commit, not its claim; a cell would hold a takeover in that window.
FU-JS-ONFAILED-HOOK-BEFORE-PAIRING: a run superseded at its failure pairing fires onAgentFail then onAgentSuperseded — open
Owner: unassigned (reliability orchestrator to route). Finding: none (hook order). Surfaced by: the orchestrator's review of !465 (RM-91).
runtime-js onFailed (runtime-js/src/run-loop.ts, ~:4790) fires onAgentFail before the RM-91 pairing append (~:4809) and the K5. A takeover that refuses the pairing (RunSupersededError) ends the run superseded, so both terminal hooks fire, contradicting stopAsSuperseded's "never onAgentFail". The K5-refused path has the same order (the hook fires before commitTerminalState). To close: fire onAgentFail only once the K5 landed (as R40's complete → fail order allows), or document the pair like FU-HOOKS-COMPLETE-THEN-SUPERSEDED-WINDOW; the CF DO inherits it.
FU-STOP-SETTLE-PAIRS-UNFINISHED-CALLS: a stop committed for a run nothing executes does not pair its unfinished calls — narrowed (Save-First Tools MR 1: a saved reply's own calls are paired); open for the rest
Save-First Tools MR 1 (2026-11-06): narrowed. Core commitStopTerminal now pairs a trailing SAVED reply's (spec 2026-11-05 §6) own unfinished calls run_stopped before its K5: aborted tool_end frames, then one append under the stop's { allowKind }, recomputed per attempt, nothing once the K5 landed. It leaves pending client calls, the reply's __finish__ (no body: a continue honours it, RM-116; the next turn's C5 heal pairs it otherwise) and any call a sub-session ref of the run names (same parentToolCallId, and dispatchId when both carry one: the stop cascades to that child and a continue re-attaches it, G1 B2). Measured on core (commit-stop-terminal.test.ts), runtime-js through core and store-memory (store-memory/src/__tests__/commit-stop-terminal.test.ts), the CF DO (do-stop.integ.test.ts, and the cf-do cell transcript.lost-delegation-message-entry-paired-host's .lost-row-stop-paired) and CF Workflows (cfw-settle-closed-owner.test.ts); Temporal and DBOS reach the function, untested there. Still open: a trailing reply that is NOT a saved reply (the step saved nothing first) keeps its calls for the next entry's C5 heal.
CP-106 MR A (ruling B4, 2026-10-25): kept open. MR A pairs a companion call a stop cuts short where the run itself executes (CP-118 appendStopPairing: run_stopped). These terminals still leave such a call for the next entry's C5 heal (unpaired_on_continuation). RM-29 holds again at that heal; only the recorded reason differs.
Surfaced by: CP-118 review (I2).
Core commitStopTerminal (the stop's terminal for a run nothing executes) writes the K5, the run status and the stream, but no pairing, and CP-118 did not widen it. A call such a run leaves with no result stays unpaired until the next turn's C5 heal (RM-29 holds again only then). Sites: the CF DO's drive/stop.ts (~487, a stranded or paused run) and drive/settle.ts (~576, the R77 announced-terminal settle); the DO stop deadline (stopDeadlineMs: a deferred finishWith whose tool ignores the abort for 30 s is ended by commitStopTerminal, its call unpaired); CF Workflows readers' settle_stopped (settleClosedOwner, e.g. an instance that died after a refused append and before its own handleStop); and a stopped parent paused on suspended_awaiting_children (its sub-agent calls). Fix direction: a Pick-ed appendMessages pairing step inside commitStopTerminal (CP-106 MR A's reuse point for pairUnfinishedCalls), under the stop's { allowKind }, excluding the pending client-tool ids read before the K5 clears them, and never once the run's K5 landed; or a runtime wrapper that pairs, then calls it.
FU-STOP-APPROVE-DRAIN-DROPS-EXECUTED-RESULT: an approved tool runs, then a stop refuses its result append and the result is lost — done for CF Workflows, runtime-js and the CF DO (CP-118 fix round 1, Ruling O-73); the Temporal half is folded into FU-TEMPORAL-STOP-PAIRS-UNFINISHED-CALLS
Surfaced by: CP-118 (survey §1a / §3, steps.ts ~6810, activities.ts ~5704 / ~5720).
The resume drain executed an approved tool, then a stop refused its result append; CF Workflows' and Temporal's drains swallowed the CommitStoppedError, so the tool's side effects happened but its result was never committed. Fixed by Ruling O-73 on CF Workflows (applyResultsAndReload: the approved tool's result is appendCommitted(…, { recordsCompletedWork: true }); the stop is read before the tool runs, and with one the tool does not run and the call stays pending; a CommitStoppedError there is no longer swallowed) and on runtime-js / the CF DO (drainOrphanClientToolCalls: the result row is not stop-guarded; the stop gates the approved tool and the entry ends by it). The same holds for a deferred / phase-2 finishWith and every other completed result: they are recorded, never labelled "not executed". Proven by steps-applyResultsAndReload.test.ts (O-73 cases), stop-completed-work.test.ts and runtime-js stop-pairs-unfinished-calls.test.ts.
FU-APPROVED-TOOL-REDRIVE-IDEMPOTENCY: an approved tool can run more than once on runtime-js and the CF DO — open (owner: WA)
Filed by: MA-31 (Task 3 fix round 1, CP-10 ruling). The approve drain commits an approved tool's state write itself (executeSingleToolCall's commitWrites, runtime-js/src/execution/tool-executor.ts). A retryable or unclassified store fault there is a ToolStateCommitFault: the entry fails with the call still pending, and the next re-drive runs the tool again. Under a persistent retryable fault on the CF DO that is up to the wake's crash-loop maxWakeResumeAttempts times. After an unknown commit outcome (isCommitOutcomeUnknownError) the first write may have landed, so a re-run's .push() appends twice. The approved tool's tool_start (and beforeTool) already went out, with no tool_end (accepted: the call stays pending). Wanted: an idempotency key for the approved execution, or a durably recorded outcome (result plus state write) that a re-drive reads instead of re-running the tool.
DO companion lifecycle MR A (spec 2026-10-15) follow-ups
Surfaced while building and reviewing DO companion MR A (the consult protocol: CP-39, CP-125). Line numbers are at the MR A head. MR B / MR C items that the spec already plans are recorded on their own FUs (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY, FU-DO-COMPANION-START-FETCH-UNBOUNDED, FU-DO-STOP-CASCADE-OWED-NO-ALARM).
FU-RAW-ERROR-TEXT-SURFACES: raw failure text still reaches a client or a model on three surfaces — open (MED, security, owner: WE-2)
Surfaced by: MR A Task 13 (a credentialed postgres URL in a companion's failure). MR A sanitizes the DO /statuserror (root /status and /subagent/:type/status, handleStatus → statusErrorField, runtime-cloudflare/src/do/status-error.ts; commit 4b18ef3d26), so getStatus().error is the sanitizeErrorMessage text. The same raw text still leaves on:
- The DO stream's failure.
DOStreamManager.failStream(runtime-cloudflare/src/do/do-stream-manager.ts:585) writes itserrorargument raw intostream_meta, the WebSocket event and the SSEfailframe; the DO settle passes the run's rawresult.error(durable-object-agent-base.ts:4575-4576).DOStubTransportparses that frame'serror(do-stub-transport.ts:239), and core's remote sub-agent dispatch putsevent.errorin the parent's tool result (core/src/orchestration/remote-sub-agent-dispatch.ts:517), so a non-persistent remote sub-agent's secret can reach the parent's MODEL, and the frame reaches any client attached to the child's stream. - The CF Workflows stream's failure.
DurableStreamManager.failStream(store-cloudflare/src/durable-stream.ts:996), called with the step's rawinput.error(runtime-cloudflare/src/steps.ts:5785). - agent-server's
/status.error: outcomeState.error(agent-server/src/agent-server.ts:752, the comment calls it "intentionally raw"), and the synthesized end event'serror: currentState?.error(:614).
The invariant to hold (one MR, all surfaces): no raw error text reaches a client or a model on any surface. The raw text stays in the session row and in server logs. Fix direction: sanitize at each writer with core sanitizeErrorMessage (as errorDetail.message already is), with a red-first test per surface carrying a credentialed URL.
FU-DO-CHILD-VERIFY-ATTEMPTS-KEY-CLEANUP: a child's failed-read count outlives its consult — open (LOW, owner: WA, DO companion MR C)
Surfaced by: MR A Task 12 review (M7). The parent counts consecutive failed verified reads per child in DO KV (__helix_child_verify_attempts:<childSessionId>, childVerifyAttemptsKey, runtime-cloudflare/src/do/companion-settle-notice.ts:324). The key is deleted only after a good read (:524) or a give-up (:554). It is keyed by the child, not the consult, and beginChildConsult does not reset it, so a later consult of the same child (a respawn after failed / terminated, a continue) starts from a stale count: its 404s can give up before 8 of its own. Nothing deletes it with the parent's storage either. Fix direction: reset the count in beginChildConsult's caller (or key it by consult), and delete it with the ref.
FU-DO-SENDMESSAGE-C4-REMOTE-GRANDCHILD: a DO sendMessage can resume a companion parked on a remote grandchild — open (LOW; owner: WA, DO companion MR C)
Filed by: CP-106 MR A re-derivation (WE-2). The bespoke DO sendMessage's C-4 guard refuses only a child that is paused with awaitingClientTool (runtime-cloudflare/src/do/persistent-companion-tools.ts:2174 at 232439ab39). A companion parked on its own remote sub-agent call is also paused, with the flag false, so its /resume with a message is sent and appended to a run waiting on a grandchild. Pre-existing (not a CP-106 change). Fix direction: carry the suspension kind on the child's /status (with FU-DO-SENDMESSAGE-TWO-STATUS-READS) and refuse any suspension other than one the message resolves.
FU-DO-SENDMESSAGE-TWO-STATUS-READS: a DO sendMessage to a running child reads its /status twice — open (LOW, owner: WA)
CP-40 (C2) update: C2 (DO companion MR C part 2) did not fold the reads (orchestrator ruling: not needed for CP-40, kept out to keep the diff small). The guard now answers the C-4 case typed (child_awaiting_client with the calls of the child's CP-157 pendingCalls listing, or child_busy with none listed) instead of throwing, and /status now carries everything the fold needs (committedOutcome: 'suspension', pendingCalls, stopping).
Surfaced by: MR A Task 10 (concern 5, review M1). On a running ref, companion__sendMessage first runs the in-tool reconcile's verified read (CP-39), then its session-level pre-send guard read (runtime-cloudflare/src/do/persistent-companion-tools.ts:1436-1443; the code comment cites this FU). Spec §3.6 says the reconcile replaces the guard. It is kept because the guard checks stopping, the C-4 paused child and a session-level terminal under another consult, which the verified read does not report. Fix direction (MR C): carry those three on the consult-aware /status (with the suspension's pendingCalls) and fold the two reads into one.
FU-COMPANION-ALREADY-RUNNING-TYPED: a same-name spawn of a live companion is an untyped tool error — fixed in core (CP-40 (C2) T1; DBOS: FU-DBOS-COMPANION-TYPED-NAME-REFUSAL)
Fixed (CP-40 (C2) T1): resolveChildName returns { kind: 'resolved', name } | { kind: 'refused', name, outcome }; core's dispatcher answers the refusal as not_spawned (child_busy, child_interrupted, child_awaiting_client), so runtime-js, the CF DO, CF Workflows and Temporal answer it typed. The original record follows.
CP-106 MR A (2026-10-25): the child phase runs a step's calls in call order, so a SAME-STEP second spawn of a name now meets this error too (it used to race the claim and answer the typed child_busy). Agreed with WA: MR A asserts it by kind (expectSameStepSpawnRefused in runtime-js/src/__tests__/companion-first-spawn-entry.test.ts: CP-161 R1, K1 A', K1 B, zero-IO and I1), and CP-40 C2 flips those assertions to not_spawned / child_busy on its main merge.
Surfaced by: DO companion MR B pre-flight (§4) and Task 7 / 8 (seven rejects.toThrow(/already running/) assertions). Core resolveChildName (core/src/tools/companion/helpers.ts:78-81) throws a plain Error('Agent '<name>' is already running. Use sendMessage to communicate with it.') (and the CP-68 interrupted sibling just above it) for a name whose ref is in ACTIVE_STATUSES, so the model gets an untyped error row while every other non-start of MR B is a typed, successful not_spawned answer (CompanionSpawnResult), and tests can pin it only by message text. Fix direction (MR C, with child_not_supported): answer it on the contract — a not_spawned reason (child_running, or child_busy with a detail naming the live run) and a child_interrupted-style reason for the CP-68 case — from one core builder, so no runtime and no test keys on the text; keep the guidance in the detail's message.
FU-DBOS-COMPANION-TYPED-NAME-REFUSAL: DBOS's bespoke spawn still throws a same-name refusal — open (LOW, owner: DBOS, parked)
Surfaced by: CP-40 (C2) T1. DBOS's spawn does not route through core's dispatcher: its resolveChildNameForSpawn (runtime-dbos/src/persistent/child-session-id.ts) throws the refusal's message for child_busy / child_interrupted (an ok: false tool result, as before) and does not refuse a paused_awaiting_client ref (it never did). Fix direction (when DBOS is unparked): answer the refusal as the typed not_spawned result, as core's dispatcher does.
FU-DO-COMPANION-CLIENT-TOOL-SURFACING: a CF DO companion with client / approval tools is refused, not supported — open (design, unowned)
Surfaced by: DO companion MR C (spec 2026-10-15-do-companion-lifecycle-design.md D2 (a)); CP-40 (C2) built D2 (c) + (b) instead. A DO companion's child has no route to a client, so C2 refuses such a companion at spawn (refuseChild, framework_companion_not_supported) and at its own consult entry (materialize step 2c). Fix direction (D2 (a), a design with the user): route a companion child's awaited calls to the root's client (the CP-157 remote park is the model), then lift the refusal.
FU-CFW-COMPANION-CLIENT-TOOL-SPAWN-REFUSAL: CF Workflows has no spawn-time refusal of a companion whose calls nothing can answer — open (LOW, unowned)
Surfaced by: CP-40 (C2) (MRC §2). CF Workflows shares one store, so a companion's client call is answerable through the root and C2 answers its suspension suspended (T5) instead of refusing it. If a CFW deployment cannot answer it (no client attached to the root), the parent learns only from the suspended answer. Fix direction: decide (with the user) whether CFW should refuse like the DO or keep answering suspended.
FU-TEMPORAL-DBOS-COMPANION-PARK-TYPED-ARMS: Temporal and DBOS do not answer a suspended companion typed — open (parked with Temporal / DBOS)
Surfaced by: CP-40 (C2). Temporal and DBOS park on a companion with their own workflow code and have no suspended arm: a companion paused on client / approval calls is not answered suspended with its calls and its ref is not recorded paused_awaiting_client. Temporal's name refusals do flow through core (typed, T1). Fix direction (when unparked): map a suspended child to core's suspended arm and the paused ref, as runtime-js (T4) and CF Workflows (T5) do.
FU-JS-COMPANION-SUSPENDED-CONTINUATION: who drives a runtime-js companion after its awaited calls are answered — done
Surfaced by: CP-40 (C2) ruling 5. On runtime-js a client can answer a paused companion's call through the root (shared store). The parent was answered suspended (T4) and its ref reads paused_awaiting_client; nothing then drives the child's continuation, and its later outcome reaching the parent (notifier delivery after a suspended answer) is unproven. Pairs with CP-128 (the root ownership write), MR D. Fix direction: decide the driver (the root's resume, as CP-10 drives in-process children) and pin the delivery.
Done:
- No live ancestor: CP-250's self-wake drives the companion, and the outcome is written to the parent's ref undelivered, for the root's next turn.
- The pause window: CP-315 closes the window where nothing drove it.
- The blocking consult: it never stays parked on a paused companion (
companion.blocking-spawn-answers-suspended-*; CP-250's helper spawns blocking). - A companion's own call answered through a live root: CP-315's W3 guard.
The residual is FU-JS-COMPANION-SELF-WAKE-CROSS-PROCESS-ANSWER.
FU-STORE-TERMINAL-CAS-DROPS-DESCENDANT-PARK: a status CAS off live clears a still-paused descendant's park — open (reachable by reading, not shown; owner: W1)
Surfaced by: CP-279 (S1, fixed). Its F2 keeps a DESCENDANT's parked remote call through the root's terminal commit (core remotePendingCallsAfterTerminal: an entry whose parked parent's run is still suspended is kept), on every terminal commit site: runtime-js's step and K5 commits, CF Workflows' K5, and core's failCommittedRun, commitStopTerminal and failSessionWithoutRun.
The gap: the store's status CAS (compareAndSetStatus) still clears the whole map when it moves a session to a status that is not live. That's the CP-157 contract, remote-pending-calls.cleared-by-terminal-cas (core/src/testing/state-operations.ts:718), and every store implements it (store-memory/src/in-memory-state.ts:1506, store-postgres/src/state/postgres-state-store.ts:2830, store-redis/src/redis-state.ts:4309, store-cloudflare/src/d1-state.ts:2802). A root moved off live by a status-only writer would therefore drop a still-paused companion's call, and its answer could no longer be routed. The writers today:
- agent-server's expired-session cleanup (
agent-server/src/cleanup.ts:245); - runtime-js's
active→failedCAS (js-agent-executor.ts:9674); - DBOS's interrupt settle (
runtime-dbos/src/lifecycle/interrupt.ts:684); - core's checkpoint ops (
core/src/store/checkpoint-ops.ts:242,:306).
Not shown: no conformance cell reaches a status CAS on a root that holds a descendant's park. CP-279's local CF Workflows diagnostic logged no status CAS on its path, and CP-279 is fixed through the terminal commits, not the CAS. A run that shows it reopens CP-279, or gets its own S1 id.
Fix direction: the CAS keeps what remotePendingCallsAfterTerminal keeps. Either the store applies the same rule in its atomic step (it needs the parked parents' current runs), or the status-only writers go through a commit that does, together with CP-282's field-scoped parked-map op (updateRemotePendingCalls, which CP-282's MR adds: the CAS still clears the map whole there, pinned by RPC-7c).
FU-STORE-BARE-CREATE-RUN-NOT-A-RUN-START: on D1, a bare createRun racing an idle-fenced write is not refused — open (LOW; no product path; owner: W1)
Surfaced by: CP-282's review of D1's idle-fence binds (owner and orchestrator rulings, 2026-11-07).
The gap: createRun is a public, required store method, but it is not a run start: it sets no session pointer (current_run_id). Every runtime starts runs through the guarded run-start commit, and no product src calls a bare createRun. A custom loop built on the store interface could, though ("bring your own loop").
- Before the write: every store's fence resolves the current run as the session pointer, else the session's highest-turn run, so a bare run that already exists is the current run, and an idle fence expecting another run is refused. Pinned by RPC-5c (
core/src/testing/remote-pending-calls-operations.ts, every store). - Racing the write (the gap): on memory (synchronous), Redis (inside the Lua script), Postgres (the fence runs under the state row's
FOR UPDATE;createRuntakesFOR SHAREon that row) and the DO (transactionSync), the check and the write are one atomic step, so there is no window. D1's pre-read check and its guarded UPDATE are separate statements, and the UPDATE re-checks onlycurrent_run_id IS ?(still NULL after a barecreateRun) and "no live run with the EXPECTED id". A barecreateRuncommitted in between passes both, for both D1 writers:saveStatefenceIdle(store-cloudflare/src/d1-state.ts:5270-5271) andupdateRemotePendingCalls(:6593-6597). A run-start commit in between does not pass: it moves the pointer (RPC-5d, D1 through the seam).
Fix (b), for a follow-up MR: both D1 UPDATEs check "no live run of the session other than the expected one" instead of the expected id (one predicate per writer), red first with RPC-5e on D1 through the seam: a bare createRun committed between the op's read and its write refuses the op, nothing written. Because it touches saveState's shared fence, the MR runs the store-cloudflare and runtime-cloudflare integ suites. Kept out of CP-282's MR to hold its scope (orchestrator, 2026-11-07).
FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN: runtime-js companion sites answer a child suspended on its own children as today — open (LOW, unowned)
Narrowed (2026-11-06, CP-279; orchestrator ruling): a companion whose run is suspended_awaiting_children on a REMOTE child parked on client / approval calls is a paused companion, on runtime-js and CF Workflows. Its parked calls are on its park root (remotePendingCalls), so the blocking spawn answers suspended with them and the ref reads paused_awaiting_client; the root's terminal keeps such a call while the companion's run is suspended (core remotePendingCallsAfterTerminal), and an answer through an ended root wakes the companion (its CP-250 self-wake). Only a wait on LOCAL children (no remote park) stays here (runtime-js; CF Workflows' twin is CP-249 / Q3).
Narrowed (2026-10-29): the suspended_step_partial half is reachable and is CP-248 (WA's CP-214 red). The suspended_awaiting_children half stays here: probe B could not reach it on runtime-js (inconclusive). A conclusive re-probe either widens CP-248 / CP-249 or closes this as unreachable. CP-249 is CF Workflows' awaiting_children strand.
Surfaced by: CP-40 (C2) T4 (Q3's runtime-js twin). The companion child loop reports only suspended_client_tool; a companion whose run is suspended_awaiting_children or suspended_step_partial kept the legacy semantics (the sites read it as an end). [inferred: not reachable for an in-process companion today.] Fix direction: answer it as live (as the DO does, Q-B1) when it becomes reachable.
Narrowed (CP-214): suspended_step_partial is answered suspended (the ref paused_awaiting_client) on runtime-js, CF Workflows and the CF DO since CP-214; it was reachable (a server tool beside a client call). Only suspended_awaiting_children remains, pending CP-214's reachability probe and the orchestrator's ruling.
FU-CONF-CLIENT-TIER-CURRENT-AI: the conformance client tier runs only ai 6.0.49, older than any current app — open (owner: the conformance owner)
Reshaped (user ruling 2026-11-07): test only the versions we support. @helix-agents/ai-sdk's peer range rises to the floor ai ^6.0.170 / @ai-sdk/react ^3.0.172 (matching llm-vercel; a breaking peer bump, in the release notes). The client tier runs two pairs: the floor (6.0.170 / 3.0.172) and the newest pair at least two weeks old (6.0.314 / 3.0.317 now), moved forward periodically; the 6.0.49 tier and the ai-sdk unit pins below the floor (6.0.0 / 6.0.49 / 6.0.91) go. A new CI shard runs the newest pair. CP-284's client-tier red lands on the newest pair (CP-284 is fixed on unit proof until then).
Filed: 2026-11-07, with CP-284. The client tier (the real useHelixChat / Chat over each host) builds against packages/e2e's ai and @ai-sdk/react devDeps: ai 6.0.49 and @ai-sdk/react 3.0.51. The ai-sdk package's UI-chunk schema contract pins more versions (6.0.0 to 6.0.281), but only for chunk parsing, never for a real client. So a client behaviour that changed in a later ai passes CI and breaks real apps: CP-284 is the first such break seen. The member: the client-tier lanes run on a second, current client pair (the latest ai 6.x and its @ai-sdk/react), through aliased packages as the schema contract does, every client cell judged on both, with a self-test that the second pair is really loaded. Then the pinned old pair can move forward without losing coverage.
Widened (2026-11-07, Save-First Tools' cell plan): the member also gives js-chat-host a client-tier tool gate (hold-tool-gate is the cf-do bridge's only), so a client cell that holds a server tool mid-step (Save-First's reload-mid-tool cell) runs on both hosts instead of a declared js-chat-host gap.
2026-11-08: the ai-sdk half lands with the client-pairs MR's follow-on (branch conformance-owner-ai-floor): the peer ranges rise to ai ^6.0.170 / @ai-sdk/react ^3.0.172 (and embedding-vercel's ai peer), the unit pins below the floor go (each multi-version table runs the locked 6.0.170 as its floor row), and the UI-chunk contract's compile-time guard reads 6.0.170's UIMessageChunk. core / sdk / the examples follow in the next MR.
FU-AISDK-BELOW-FLOOR-HANDLING: ai-sdk code and comments that exist only for ai releases below the 6.0.170 floor — open (LOW, owner: the conformance owner)
Filed: 2026-11-08, with the ai floor bump (FU-CONF-CLIENT-TIER-CURRENT-AI). The floor MR changed only the peer ranges, the unit pins and the docs; product code that handles releases now below the floor stays as it was. To review (each keeps working on supported releases; the question is only whether its below-floor branch or wording can go): client/reattach.ts (DI-83's ai <= 6.0.89 resume order), client/resync.ts and react/use-helix-chat.ts (the shared activeResponse on ai <= 6.0.91), and types.ts's AISDKFileEvent note (releases that drop a file part's providerMetadata: the floor copies it, so the filename now reaches every supported client). Fix: drop what only served the old releases, with the touched package's full suites.
FU-STOCK-MIXED-STEP-DELEGATED-APPROVAL: a root step parked on its own approval and a sub-agent's keeps one step, which a stock useChat on ai 6.0.158+ never sends — open (LOW, owner: the conformance owner)
Filed: 2026-11-07, with CP-284's fix. CP-284 starts a new step before a child's first part after the root's unanswered delegating call, so ai's stock auto-send predicates (which read only the last step) can complete it. It adds no boundary when that step also holds an unanswered part of the root's OWN that the user may answer (its own approval, or a call not known to be server-executed), so that mixed step keeps its single-step shape exactly: on ai 6.0.0 to 6.0.91 with findTool wired it sends both answers together, as before. On 6.0.158 and later the delegating call's unanswered part keeps the predicate false, as before the fix. Also: with no findTool, a root server tool still running when a child's first frame arrives counts as such a part, so live adds no boundary there, while a reload (the tool's result committed) does. Pinned by delegated-approval-autosend.test.ts ("the mixed step's shape is today's") and transformer/delegated-step-boundary.test.ts (the running-call shapes). Fix: render the root's own pending parts of that step after the boundary (a reordering the live stream cannot do once streamed), or an app-side predicate (useHelixChat's per-answer predicate is not affected).
FU-AISDK-NESTED-DELEGATION-RELOAD-PARTS: a reload renders a delegated call only while it is pending, so live and reloaded messages differ in nested and finished delegations — open (LOW, owner: the conformance owner)
Filed: 2026-11-07, with CP-284's fix (CP-157 / CP-186 scope). Live, the root's message holds every forwarded child frame: an intermediate child's own delegating call (nested) and a finished child's answered call (sequential delegations). The snapshot renders only the pending leaf calls (markRemotePendingCalls, insertLocalChildCallParts), so those parts and their step boundaries are not in the reloaded message. The last step, which ai's predicates read, is the same. Pinned by delegated-approval-autosend.test.ts ("nested: the same last step"; "sequential: the same last step"). Fix: render a delegation's forwarded parts from the child's committed rows on reload.
FU-CONF-CF-DO-GENUINELY-REMOTE-CHILD-PRODUCER: no host-tier producer for a genuinely remote sub-agent child on the CF DO — open (owner: the conformance owner)
Filed by: CP-279 (2026-11-06). Every remote child the cf-do lane builds is a sub-agent declared local and turned into a sibling Durable Object at drive time (remote-subagent-rewrite); the executor-tier remote-child capability is claimed only by the runtime-js and CF Workflows lanes. A DO companion that declares a genuinely remote sub-agent tool (createRemoteSubAgentTool) is not refused by C2, which treats remote tools as opaque, so CP-279's shape may reach the DO through it, unverified. Wanted: a cf-do worker producer the DO can call over a transport as a genuinely remote child (pausing on an approval), then CP-279's -host cell on cf-do, and the contrast cell that a locally declared tree is refused at spawn.
FU-CONF-TRUTH-CF-ADAPTER-ABORT: the checkpoint-truth matrix's cf adapters cannot stop a failed cell's run, so it bleeds into the next cell — open (owner: the conformance owner)
Filed: 2026-11-07, from Save-First Tools MR 1's slots (WA). The cf-do-d1 and CF Workflows pool adapters of the checkpoint-truth matrix refuse handle.abort() and handle.interrupt() (e2e/src/harness/setup-helpers/cf-do-d1.ts, the shared unsupported() throw, used by abort, interrupt, and also getState, canResume, resume, send and retry; the same pattern in cfw-workflows-d1.ts). A cell that fails while its run is held cannot end that run in its cleanup, so the run keeps going and consumes the next cell's scripted model replies. Seen in WA's slot slot-s10r: two refusals, then cf-do-d1's text-terminal settled failed ("MockLLMAdapter: No more responses configured"). In slot-c1, with no cell failing early, there were no refusals and text-terminal passed. The cost is a misattributed red, never a false green: the bled cell fails. Fix: the adapters stop a cell's run in cleanup (through the DO's stop route or the instance's terminate), or each cell gets its own session namespace and mock script, so a leftover run cannot read another cell's replies. A self-test fails a held cell, then asserts the next cell's script is untouched.
FU-CONF-EXECUTOR-RUN-START-REJECT-CONTROL: no control injects a run-start rejection with a given cause on the js lanes or DBOS — open (owner: the conformance owner)
Filed by: CP-278 (2026-11-06). cf-do's fault-do-store runStartRejected injects a cause, but the CF DO already classifies through classifyEntryFailure and runs no client-tool deadline timer. Wanted: an executor-tier control that makes a named session's next run-start commit throw RunStartRejectedError with a chosen cause, on the runtime-js lanes and dbos-postgres, so CP-278's sites can be witnessed by cells, not only by unit tests.
FU-JS-STEP-PARTIAL-AWAITING-DESCENDANT: a step_partial companion whose client calls are answered but which still awaits its own child is answered as a stale pause — open (LOW; related: CP-249, FU-JS-COMPANION-SUSPENDED-AWAITING-CHILDREN)
Surfaced by: !584's review (CP-214). runtime-js awaitAttachedChildTerminal (run-loop.ts, the replay-attach park): a suspended_step_partial companion child whose client / approval calls are all answered, but whose step still awaits its OWN child, lists no pending calls. After one lease TTL its parent reads it as a stale pause and answers suspended with no calls, so the parent is not told the child waits on a descendant. It borders CP-249 (a companion awaiting its own children). Fix direction: decide with CP-249's contract (answer the descendant's pending calls).
FU-DO-NOTICE-FORWARD-COMPAT: a settle notice with a status the parent DO does not know is rejected, not retried — open (LOW; owner: the DO)
Surfaced by: !584's review (CP-214). The parent DO parses a child-settled notice with the strict childSettledNoticeSchema (companion-settle-notice.ts) and answers an unknown status 400, which the child treats as not retryable: the notice is given up (rejected). So any future widening of the ringing statuses drops notices while a deploy rolls: a parent DO still on the old code never hears the new status (CP-214 widened it with suspended_step_partial). Then a blocking park answers unreachable after its 30 s bound, and a non-blocking ref stays running until the model polls again after the upgrade. Fix direction: answer an unknown status with a retryable code, so the notice stays owed and lands once the parent runs the new code.
FU-JS-ATTACH-PARENT-STOPPED-KIND: the runtime-js replay attach answers every parent stop interrupt — open (LOW, unowned)
Surfaced by: CP-40 (C2) re-check (L5). awaitAttachedChildTerminal answers { status: 'parent_stopped', kind: 'interrupt' } for any abort of the parent's signal, so an abort stop is reported as an interrupt. Fix direction: carry the stop's kind (the DO's parentStopOf).
FU-DO-STATE-STORE-FOREIGN-SESSION-REFUSE: a DO state store write naming another session is not refused — open (LOW, owner: WA, after CP-40)
Surfaced by: CP-188 (withdrawn as a duplicate of CP-144; design note cp188-design-note.md). A DOStateStore holds one session. Its commit paths refuse a foreign id (requireStateRowSync + the run fence), and loadState answers null, but some mutators ignore the id: an unfenced saveState with a matching version, mergeCustomState, updateStatus, compareAndSetStatus, incrementStepCount / incrementResumeCount write row 1; addSubSessionRefsInternal and stageChanges insert rows under the foreign id; createSession throws untyped. No runtime path calls them with a foreign id today. Fix direction (hardening): refuse a foreign sessionId on every write, typed (keeping the respawn wipe → createSession path and the first-save INSERT working), with a node-sqlite test per mutator.
FU-COMPANION-REOPEN-IN-RUN-START-COMMIT: a re-consulted companion's session reopens after its run-start commit, not in it — done (CP-333, !929)
Surfaced by: CP-155 (S1; spec 2026-10-19-cp155-companion-continue-entry-design.md §3.1, ruling R1 (a)). runtime-js's continue (and CF Workflows' __continue__ instance, in writeEntryState) reopens the child session (completed → active) in a fenced saveState AFTER the run-start commit, so between the two (or after a crash there) the session still reads the previous turn's completed while its new run is live. Core's run-anchored ref sync (readChildNotice, CP-155: the companion read tools, CF Workflows' checkCompletedChildren, runtime-js's notifier, core deliverPendingChildCompletions) keeps every companion reader truthful over that window, and the turn's own commits rewrite the status, but the session row itself is stale until then. On CF Workflows a crash there is completed by the engine's re-executed run-start step; on runtime-js nothing re-drives the committed run: readers stay truthful and waitForResult's orphan detection settles the ref failed for a fresh respawn. Related: CP-164 (S2, runtime-js, parked with JS) — recover such an orphaned in-process companion run as interrupted (resumable, memory kept) instead of failed + a fresh respawn. Fix direction (ruling R1 (b)): an atomic reopen inside the commit, RunEntryState.status: 'active' applied by all five stores (memory / Redis / Postgres / D1 / DO-SQLite) in the run-start's atomic step, with a run-start contract case; root execute() and CF Workflows could adopt it too.
Done (!929, CP-333): the companion continue's run-start commit writes the session's active status and clears a previous failure in its own atomic step (runtime-js/src/run-loop.ts:11124-11125 and CF Workflows' __continue__ entry, runtime-cloudflare/src/steps.ts:1956-1957, as on origin/main fae53b26ac), as every other entry now does.
FU-TEMPORAL-DBOS-COMPANION-CONTINUE-ENTRY: Temporal and DBOS reopen a re-consulted companion before its run-start commit — open (parked with Temporal / DBOS)
Surfaced by: CP-155 (spec §2, §3.3). Temporal's continue reopens the child (a version-pinned saveState to active plus the __helix.companionContinue marker) and resets the ref before the workflow's startChild, compensated by three rollback seams (runtime-temporal/src/activities.ts, workflow-dispatch.ts); DBOS CASes completed → active first (runtime-dbos/src/steps/execute-companion-tool.ts) and resets the ref only after a confirmed start, with no CAS undo. DBOS keeps its own session-only copies of three rules that runtime-js / CF Workflows / core now anchor on the child's run or name: its resolveChildStatus ref sync (execute-companion-tool.ts, no run anchor; core's readChildNotice is the read to route it through, FU-DBOS-COMPANION-READS-RUN-ANCHORED) its sendMessage I1-race child_exited check, and its pollChildSessionTerminal outcome read (execute-companion-tool.ts:955-976), which sets completionDelivered from the session status. DBOS has no continue window today (its CAS reopens the session before the start), so neither is wrong there yet. Fix direction: adopt runtime-js's CP-155 shape — nothing written before the child's run-start commit but the prior hold, entryStatuses: ['completed'] plus resumedCompanionRef in that commit, the reopen after it fenced on the new run, every outcome answered through core companionStartOutcome (CP-155's continueStartOutcome, renamed by CP-161) — then delete Temporal's marker and rollback seams.
FU-CORE-REPLAY-KEEPS-RECORDED-COMPLETED-AT: a replayed blocking spawn re-stamps the ref's completedAt — open (LOW, owner: WA)
Surfaced by: MR A Task 2 review (m2), Task 10 (concern 1). A replayed spawn answered from the recorded outcome (ruling C14) goes through applyParkResult, which writes completedAt: Date.now() (core/src/orchestration/companion-tool-dispatch.ts:536), so the ref's completion time becomes the replay's time, not the child's. recordedConsultOutcome's answer has no timestamp (:290-294). Fix direction (core): an optional completedAt on the recorded answer (the DO dep reads it from the ref), kept by applyParkResult on a replay.
FU-DO-REF-WRITER-GUARD-BRACKET-CALL: the I-C1 ref-writer guard misses a bracket or dot-less call — open (LOW)
Surfaced by: MR A Task 10 scoped re-review (parked minor). The guard (runtime-cloudflare/src/__tests__/do-companion-ref-writers.test.ts:24, /\.updateSubSessionRef\s*\(/) catches store.updateSubSessionRef( but not store['updateSubSessionRef']( or a destructured updateSubSessionRef(. No such call exists today. Fix direction: match the bare identifier too, with a self-test fixture for each form.
FU-DO-COMPANION-ORPHANED-CONSULT-ADOPTION: a recovered parent's fresh-id spawn does not adopt its orphaned consult — open (MED, owner: WA, DO companion MR C)
Surfaced by: MR A final review (I1). C14's replay answer (CompanionToolDispatchDeps.recordedConsultOutcome) and the in-tool reconcile's skip (reconcileRunningRef's consultOfThisCall) key on the tool-call id. A DO recovery truncates to the committed prefix and re-calls the model (runtime-js/src/run-loop.ts, the orphan truncateMessages; Ruling 66 discards the uncommitted step), and a real provider (Anthropic, OpenAI) mints a fresh tool_use id, so the re-emitted spawn is not a replay. Today, by when the child's outcome is recorded:
- before the recovered model call: the notifier injects it, and the fresh-id spawn then CONTINUES the child (a second run of the instruction the child already answered);
- never yet (the child is live): the spawn answers the "already running" tool error;
- during the recovered model call: the spawn is a continue over an undelivered outcome. Narrowed by MR B (deliver-then-continue): that continue HOLDS the outcome, delivers it as
priorResultbeside its own answer AND runs the child again — the verdict is no longer lost (exactly one delivery in history), but the child still runs the instruction a second time, as in the first bullet (pinned byruntime-cloudflare/src/do/__tests__/companion-settle-notice.integ.test.ts, "C14 with a fresh call id (deliver-then-continue)").
So after MR B every fresh-id order either runs the child twice or (a live child) answers "already running"; none loses the verdict.
Design (MR C): an orphan is a consult whose tool call is not in the parent's committed history (the step that issued it never committed). A new companion__spawnAgent / companion__sendMessage for that child re-points the consult to the new call id (a store CAS: consult_tool_call_id from the orphan to the new call, only while the orphan is still the ref's consult and its call is absent from the committed log) and parks on it (blocking) or answers spawned (non-blocking), instead of answering busy or continuing. The conformance sibling companion.parent-evicted-mid-park-host must script a fresh id on the replay to witness it.
FU-DO-COMPANION-PARK-OTHER-CONSULT-LOOPS: an untimed wait on an unconfirmed consult over a consult-less run never ends — open (LOW, owner: WA, DO companion MR C3 part 2)
Surfaced by: MR A final review (M1). MR C3 part 1 answers a confirmed consult from its run's own end once that run has ended. MR C3 part 2: companion__waitForResult never calls reconcileLiveRef. An earlier version of this entry said it ran once before the wait's loop, but the line cited was getChildStatus. So any UNCONFIRMED consult (a lost start, or a lost start answer with its run never seen live) under an untimed wait re-read other_consult until the parent's stop, once a consult-less run was on the child. The wait's loop now runs the same decision (settleUnconfirmedOverRun). Limits:
- The outcome is unknown, and the answer says so. The child's fence reads its current run only, so a run of the consult that a newer run replaced reads
fenced, as a start that never arrived does. The ref is settledfailedand the wait answersfailedwithreason: 'outcome_unknown'. Telling them apart needs a lookup of a run by consult on the child, which is new protocol, left out. A later wait on that ref answers a plainfailed: the cause is not persisted. - What nothing can decide is bounded. A fence that does not decide, or a child with a session and no run, counts against the failure window: the wait answers
unreachable, writing nothing. - A live consult-less run still keeps an untimed wait going, until it commits.
- A park takes the same decision. A park follows a started entry, so its consult is confirmed, unless
recordStartedansweredstale: the consult moved to a parallel call's, and the park reads that consult. Its own run current is the adopt row: the real word answers. Over a consult-less run the fence decides, and the answer isfailedwhoseerrorDetail.messagesays the outcome is unknown. It has no typedreason: that needsChildTerminalResultand the spawn result, and can be added without breaking anyone. Unit-tested only. - A child with a session and no run was an unbounded wait too. It is bounded here, typed
unreachable, and writes nothing. No route to that state was traced, so "reachable" is unproven. A route that produces it, or a wait that outlasts the failure window, makes it a finding.
Do not build b1 from the C3 design note. It said a stranded reset's continuation run "is doing X's work", and proposed parent: 'inherit' on it. A trace at 109b656d82 (reading only) shows otherwise. At a reset the current run is either live-suspended, so its continuation already inherits the parent, or ended with its own K5, so its end is the consult's real word and inheriting would set a second run against it. No writer of active over an ended current run was found on the DO: run-start commits active in its own transaction, and every state save is version-pinned, so a state read before a K5 cannot land after it. That is a failed search, not a proof. If that holds, the wake's reset only ever sees a suspended_* run. The comment in drive/announced-terminal.ts now says so.
FU-CONF-CP40-C3-OWED-CELLS: cf-do cells CP-40 C3 part 1 still owes — open (LOW, owner: WA, conformance)
Surfaced by: CP-40 C3 part 1, owner review. Unit-tested only so far: the superseded_by_entry branch, the parent fence over an unconfirmed consult (fenceUnconfirmed), and a current run serving another consult (sendMessage verifies first, which makes that last one narrow, but it is not proven unreachable). Owed: a cell for each; the DO's live-X recovery before an entry; and an after-answer drop arm on /resume. The fence-in-the-wait fix has its cell (companion.unconfirmed-wait-settles-host); a run that ran with its answer lost is unit-tested only: holding it needs a control that forwards a request and drops its answer.
FU-DO-COMPANION-INTERRUPTED-REF-CONSULTLESS-END: an interrupted ref over a child whose consult-less run ended stays interrupted — open (LOW, owner: unassigned, DO companion)
Surfaced by: GL #174 (the in-tool reconcile now reads every live ref, interrupted included). The residual case: the ref is interrupted, and the child's interrupted session was then driven by an entry that serves no consult (a direct /resume or /start on the companion DO, not through the parent). The verified read answers other_consult with no consult (readChildWord), so the parent records nothing, by MR B's rule that a consult-less run is never adopted (reconcileLiveRef). The ref stays interrupted while the child's session is completed or failed: sendMessage answers child_exited on every retry, and a same-name spawn is refused. Fix direction: decide what a consult-less ending means for the parent. Either the ref takes the child's session-level terminal (with no word to deliver), or sendMessage continues a completed child whatever the ref says. Cover it with a DO case over DOStateStore.
FU-DO-COMPANION-INTERRUPTED-DEBOUNCE-LIVE-RESUME: the interrupted debounce keeps a recorded interrupted word across a live read of the same consult — open (LOW, owner: WA, DO companion)
Surfaced by: MR C1 plan review (ruling O3). The park and companion__waitForResult share InterruptedDebounce (runtime-cloudflare/src/do/persistent-companion-tools.ts) under the wait's rule: a verified read that finds a live run for the SAME consult leaves the recorded interrupted word in place, so it still counts. That live run is a continue-mode resume of the child that inherits parent (a direct /resume of the child DO): the park or wait can answer interrupted while the resumed run is live. Fix direction: a live read whose run is not the recorded word's run resets the count (both loops at once).
FU-DO-COMPANION-INLINE-DELIVERY-UNRECORDED: an inline-delivered word has no structural record, so a timed-out respawn re-delivers it once — open (LOW, owner: WA, DO companion MR C)
Surfaced by: MR B Task 8b review (the orchestrator ruling of 2026-10-16: an adopted word is delivered only if the parent's committed history records it). DOStateStore.abandonUnconfirmedConsult decides an adopted word's completion_delivered with isChildWordRecordedSync (runtime-cloudflare/src/do/do-state-store.ts), which reads only the structural keys of core companion-held-prior.ts: the notifier's notice keyed { subSessionId, runId }, or the held slot's answer row while the slot holds that run. The two INLINE delivery channels leave no such record: markChildOutcomeDelivered (a companion__waitForResult answer or the blocking park: the answering call's id is not on the word) and releasePriorResult (the slot is cleared; its mark is a flag the next beginChildConsult resets). So a RESPAWN whose /start times out — core's respawn branch writes the ref running before dispatch, the timeout leaves the consult unconfirmed, and the re-probe (reconcileRunningRef) adopts the child's committed word — adopts an inline-delivered word UNDELIVERED, and the next notifier pass (or the next continue's hold) delivers it a second time: (a) a waitForResult / park answered it and marked it, with no notice and no slot; (b) the held-prior pass released it after the respawn's generation bump. A visible duplicate, deduplicated by nothing, never a loss (as ruled: the alternative, trusting the flag, retires unseen words). Fix direction (MR C): a { toolCallId, runId } delivery tombstone on the ref, written by both inline channels in their own CAS, which isChildWordRecordedSync reads through isCompanionAnswerCarryingPrior(messages, tombstone.toolCallId) when tombstone.runId is the adopted run (Task 9b: a row for the call is a record only if it CARRIES the prior — isCompanionAnswerCarryingPrior / isHeldPriorDelivered, never any row by call id) — or a structural tombstone of its own — the same key the held slot uses, surviving the slot's release and the consult's reset.
FU-DO-COMPANION-MR-B-UNREAD-FIELDS: an MR B field has no reader — open (LOW, owner: WA, DO companion MR C)
Surfaced by: DO companion MR B Task 7 / Task 8 reviews (parked for the docs pass). abandoned?: true on reconcileRunningRef's answer (persistent-companion-tools.ts, ReconcileRunningRefResult) is set when the re-probe's abandon landed and logged, but the spawn tool decides on read alone and lets core's dispatch act on the ref's new status.
Fix direction (MR C): give it a reader (the orphan adoption of FU-DO-COMPANION-ORPHANED-CONSULT-ADOPTION may want it) or drop it, so no dead field survives MR C.
FU-COMPANION-NOTIFIER-MARK-SEPARATE-WRITE: the completion notifier's injection and its delivered mark are two writes — done on runtime-js, the CF DO and CF Workflows (DO companion MR B); Temporal / DBOS parked
Done by DO companion MR B (orchestrator ruling C16, "fold it if local"; Task 4 runtime-js, Task 9 CF Workflows, the DO through runtime-js's loop). The two writes stay two writes by design (a retry of the mark must never re-append), and the re-injection is closed structurally: every notice the notifier appends carries its key, metadata.helixCompanionNotice = { subSessionId, runId } (core COMMON_METADATA_KEYS.COMPANION_NOTICE, stamped by buildCompletionMessage / buildFailureMessage's notice argument), and a pass that finds a notice with that key already committed (findCommittedCompanionNotice) marks the ref delivered without injecting again — runtime-js's ordinary pass (run-loop.ts), CF Workflows' checkCompletedChildren (steps.ts; its dedupeCompletionMessages keys on the same metadata, never on content). Only a key WITH a run dedupes (a { subSessionId } key would match a re-consulted child's first notice and drop its second outcome), so with no run id the flag alone decides and a mark crash can double-inject, never lose. The DO's notifier (consultOutcomeNotifier) names the consult run (CompanionNotifier.outcomeOf returns runId, required). Pinned by runtime-js/src/__tests__/run-loop-persistent.test.ts ("FU-COMPANION-NOTIFIER-MARK-SEPARATE-WRITE" case) and runtime-cloudflare/src/__tests__/cfw-persistent-companion.test.ts. Still open (parked): core deliverPendingChildCompletions (the Temporal / DBOS notifier) appends through buildPendingCompletionNotifications with no key and marks in a separate write, so an eviction between the two re-injects there (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY's runtimes). The original text follows.
Surfaced by: MR A final review (M2). The notifier appends the completion rows in an append commit, then marks each ref completionDelivered with a separate updateSubSessionRef (runtime-js/src/run-loop.ts, the Stage-2 injector). An eviction between the two re-injects the completion on recovery, because the recovery keeps the append commit's checkpoint. Pre-existing logic, harmless in process; on the DO (MR A wires the notifier there) it is newly reachable. Fix direction: mark in the same commit, or dedupe on recovery by the injected row's ref id.
FU-DBOS-NESTED-HITL-UNDER-COMPANION: a client call raised by a DBOS companion's ephemeral sub-agent is surfaced to no one — open (MED)
Surfaced by: the Greptile review of the CP-247 / CP-130 MR (!589). Root → persistent companion → the companion's EPHEMERAL sub-agent, which raises a client (or approval) call. DBOS awaits the sub-agent inline, so the companion's run stays running, and the pending entry lives on the sub-agent's session, not the companion's: the companion readers (c2AwaitedCalls, runtime-dbos/src/steps/execute-companion-tool.ts) find no pause, and a blocking spawn parks to its bound. The call's ownership is registered on the root, but CP-130's filter (ActiveHandle._ownedUnderCompanion, runtime-dbos/src/handles/active-handle.ts) rightly no longer counts a call owned under a companion as the root's own suspension (the contract: a companion's pause is the companion's). Before CP-130 the root's handle surfaced the call as suspended_client_tool; now nothing does, and the call waits silently until its clientToolTimeoutMs. This is doubly-nested HITL, documented unsupported on DBOS (GitLab #73; docs/runtimes/dbos.md Known Limitations). Fix options: (1) a bounded walk in c2AwaitedCalls over the companion's live EPHEMERAL sub-session refs, reporting a descendant's committed pause (its run C2-suspended and its stream paused, read via state.streamId) as the companion's awaited calls, so the blocking park and waitForResult answer suspended and the root's ownership map still routes the user's submit; or (2) a typed spawn-time refusal of the shape (as the CF DO's D2 (c)). Either needs a real-DBOS nested integ test first (whether a nested sub-agent's suspension pauses the stream it writes to). The conformance owner dedupes or mints the finding id.
FU-DBOS-COMPANION-READS-RUN-ANCHORED: DBOS's own companion readers still take a child's word and output from separate session reads — open (parked with DBOS)
Surfaced by: CP-155 fix round 1 (the read order; spec 2026-10-19-cp155-companion-continue-entry-design.md §8.2). Every other companion reader takes a persistent child's word, output and run key from ONE run-anchored read (core readChildNotice / readChildTerminalOutcome): core's read tools and notifier, runtime-js's notifier, CF Workflows' notifier and companion wait. DBOS keeps three readers of its own, each reading the child session with no run anchor:
resolveChildStatus(runtime-dbos/src/steps/execute-companion-tool.ts:1606, its read at:1618): maps arunningref from the session alone, with no run anchor (and swallows a read error as "still running").handleWaitForResult's answer (:1713, then the output read at:1718): the word fromresolveChildStatus(or the ref), the output from a secondloadState. The race: a whole child turn landing between the two reads answers the old word with the new turn's output, and the delivered mark (status,completionDelivered: true) overwrites the new turn's ref, so its outcome is lost.pollChildSessionTerminal(:955, its read at:970, inside a recorded step): the blocking spawn's outcome from the session alone, under the same race.
Why not folded into CP-155: not a drop-in route through readChildNotice, and barely reachable today. The K5 link itself holds on DBOS: each persistent turn's K5 is a state commit under the turn's run, and the loop exit (finalizeLoop → runFinalizeRun) moves the session with updateStatus / saveState, which write no checkpoint, so the last turn's K5 stays the latest commit (DBOS's notifier and getChildStatus already go through core deliverPendingChildCompletions / executeCompanionToolDispatch and read DBOS children settled). But the three readers have DBOS semantics of their own: they map an interrupted session to terminated and an unknown terminal status to completed; the blocking spawn deliberately keeps the ref running for replay safety; and pollChildSessionTerminal reads inside a recorded step. The races are hardly reachable: a DBOS session turns terminal only when its loop exits, so a whole turn landing between two adjacent reads needs a new loop to start AND idle out between them; and DBOS's CAS reopens the session before a continue's start, so resolveChildStatus sees no previous-turn completed under a live run. DBOS is parked. Narrowed by CP-69: a persistent turn now writes its terminal to the session while its loop lives on, so a previous turn's completed under a new turn's live run IS reachable; the blocking park now watches the turn's own run, and resolveChildStatus reads a terminal session as running while the child's current run is live, both inside their recorded steps. The word and the output are still two reads (the race above). Related: FU-DBOS-BLOCKING-SPAWN-SEMANTICS, FU-TEMPORAL-DBOS-COMPANION-CONTINUE-ENTRY. Fix direction: route the three reads through core readChildNotice, keeping DBOS's interrupted mapping and the blocking spawn's replay-safe running ref (the decision inside the recorded step), with DBOS integration runs.
FU-DBOS-ATOMIC-EXPECT-OWNER-TOKEN: DBOS's run finalize checks the run's owner by a read, then writes in a separate call — open (from CP-310; no known path today)
Filed: 2026-11-10 (WA, with CP-310's fix).
- The window: DBOS
runFinalizeRun(runtime-dbos/src/steps/run-lifecycle.ts:163at af276bd0d7) checks the run's owner by a read (getRun, thenrun.ownerToken !== args.ownerTokenat :189) and writes the terminal status in a separate call (updateRunStatus, :225). A run another execution creates under the same id between that read and that write would still be finalized. - Why no known path today: CP-310's salted companion run ids (
companionSpawnRunId/companionRespawnRunId/companionMessageRunId/companionContinuationIds,runtime-dbos/src/persistent/child-session-id.ts) make such a same-id run impossible for companions. - Fix: an optional
expectOwnerTokenonupdateRunStatus, checked in the store's own atomic step, on every store (memory, Redis, Postgres, D1, DO-SQLite), with a contract case incore/src/testing;runFinalizeRunpasses it, and the read-side check stays as the logged no-op. It also covers the run-status writers that do not finalize throughrunFinalizeRun:runSuspendRun(steps/run-commit.ts:2105) and the stop fallback (lifecycle/interrupt.ts:549).
FU-CFW-NOTIFIER-NEVER-SETTLED-CHILD: CF Workflows' notifier never notifies a non-blocking child that never commits a terminal — open (LOW, pre-existing)
Surfaced by: CP-155 read-order review (M5). A CF Workflows non-blocking child's ref stays running until the parent's notifier syncs it from a settled child read (checkCompletedChildren, runtime-cloudflare/src/steps.ts). Two states never settle: an instance that closed without its K5 (a live run, no K5: nothing in the parent's notifier reconciles the closed owner), and a never-run child whose error-run-exists failure write was skipped (session active, no run). The ref stays running and no notice is sent unless a reader settles the child (a parent's consult, getHandle()). Same at aace0064aa and on main. CP-155 fix round 1 maps one neighbouring state: a turn that ended failed without its K5 over the previous turn's terminal session now syncs the ref failed and is notified. Fix direction: for the never-run child, sync a running ref over an active session with no run to failed once the child's own entry has ended (it needs the entry's outcome, not just the session); for the closed owner, a closed-owner probe (or a consult) from the notifier.
Completion strategies and preserved thinking (MR 1) follow-ups
MR 1 of the completion-strategies spec (2026-10-04-completion-strategies-and-preserved-thinking-design.md) wired capability-driven completion strategies, the early refusal, the prompt-prefix check, ordered parts and the fail-closed skills / memory paths into core, llm-vercel, runtime-js, the CF DO and CF Workflows. These items are the Temporal and DBOS halves, the risks MR 1 could not close, and items for the conformance owner.
FU-COMPLETION-TEMPORAL: Temporal does not resolve completion strategies — done
Owner: completion strategies MR 2 (Temporal). Closed by: completion strategies MR 2 (completion-strategies-temporal-mr2): runLLMStep resolves capabilities per call, applies forced_tool / native_output, runs the prefix check before step_start and returns the forced call's outcome (forcedTransition), which the workflow applies behind wf.patched('helix-completion-strategy-v1'); the entry activities refuse an unsupported agent; a terminal LLM error keeps its ErrorDetail (DI-24). As built: §0b of the parent spec. The text below is the original gap.
runLLMStep (runtime-temporal/src/activities.ts) passes LEGACY_RESOLUTION to prepareForcedCompletionCall, so every forced call is the legacy named-tool request, and forced completion on Opus 5.5 / Sonnet 5.5 / Fable 5.1 / Mythos 5.1 fails (a 400 per attempt, then framework_forced_completion_exhausted). The workflow classifies a forced call's error with the package-local R1 shim (forced-error-r1.ts: every error step is a recoverable miss, because the activity → workflow JSON boundary loses the error's class, retryable and code), and isTerminalLLMStepError keeps the !isForcedCall exemption (no error chunk). To close: resolve capabilities per call inside runLLMStep (resolveCompletionCapabilities), carry ErrorDetail and the applied strategy across RunLLMStepResult, delete the R1 shim and use resolveForcedCallTransition, route the native_output protocol classification and native finishWith through phase 2, early refusal in the startRunCommit activity (checkCompletionSupport), the prefix check and stampPromptFingerprint in runLLMStep, suppress text_delta on native_output, set completionStrategy / capabilityProfile on hooks and the attempt_started chunk, rename forcedToolMissing (it now also carries framework_completion_strategy_unavailable). Every changed workflow-side branch goes behind wf.patched('helix-completion-strategy-v1') with a runReplayHistory test over a pre-change history (spec §7). Proof: the both-profiles cells of e2e/src/__tests__/forced-completion-parity.integ.test.ts on temporal-*.
FU-TEMPORAL-DELEGATION-PATCH-DEPRECATE: retire the pre-G1 Temporal workflow branch — open
Owner: runtime-temporal. Surfaced by: G1 / CP-216 MR 3a. The delegation commit, its stamp in runLLMStep and the entry re-attach are gated by wf.patched('helix-delegation-commit-v1') (DELEGATION_COMMIT_PATCH, runtime-temporal/src/delegation-commit-patch.ts), evaluated once per execution. To close, once no workflow started before MR 3a is open: wf.deprecatePatch for one release, then drop the unpatched branch (the delegationCommit flag's absent case); FU-TEMPORAL-REMOTE-RESULT-BEFORE-CALL-UNPATCHED closes with it.
FU-DBOS-ADOPT-WORKFLOW-PATCHES: DBOS workflow changes are guarded by a drain, not by DBOS.patch — open (decision for the user)
Owner: runtime-dbos. Surfaced by: G1 / CP-216 MR 3b. runtime-dbos changes its workflows' recorded step sequence with a deploy-time drain ("drain in-flight DBOS workflows before upgrading"): a workflow recorded on the old code and recovered on the new one fails typed at the first new step (DBOSUnexpectedStepError, pinned by integration/delegation-commit.integ.test.ts), or continues on the new code when its recording ended before that step. The DBOS SDK in use (4.17.x) ships DBOS.patch(name) / DBOS.deprecatePatch(name), which would let an in-flight workflow replay its old path, as Temporal's wf.patched does. The decision:
- Keep the drain (today). No app configuration; every workflow change is a breaking deploy note.
- Adopt
DBOS.patch. Recorded workflows survive a deploy. Cost:DBOS.patchthrows unless the APP'sDBOSConfigsetsenablePatching: true, so every DBOS app must turn it on (a new hard configuration requirement, itself a breaking change), and each patch is retired later withdeprecatePatch(a release per patch). The scaffold isruntime-dbos/src/patches/patches.ts.
FU-TEMPORAL-STEP-END-PATCH-DEPRECATE: retire the inline step_end branch — open
Owner: runtime-temporal. Surfaced by: RM-141 (2026-11-21).
The workflow passes runLLMStep deferStepEnd: wf.patched('helix-step-end-after-tools-v1') (STEP_END_AFTER_TOOLS_PATCH, runtime-temporal/src/step-end-patch.ts): with it, the activity returns the step's step_end held and the commit writes it after the step's tool frames; without it (a workflow started before RM-141), the activity writes it inline, as before. To close, once no workflow started before RM-141 is open: replace wf.patched with wf.deprecatePatch('helix-step-end-after-tools-v1') for one release, then delete the inline branch and the deferStepEnd input.
FU-TEMPORAL-COMPLETION-PATCH-DEPRECATE: retire the pre-MR-2 workflow branch — open
Owner: runtime-temporal. Surfaced by: completion strategies MR 2 (spec 2026-10-06 §6, T4).
Every workflow-side decision of MR 2 is gated by wf.patched('helix-completion-strategy-v1') (COMPLETION_STRATEGY_PATCH, runtime-temporal/src/completion-strategy-contract.ts); the unpatched branch, and a runLLMStep result without resultVersion, run the pre-MR-2 classification (legacy-forced-classification.ts). To close, once no workflow started before MR 2 is open: replace wf.patched with wf.deprecatePatch('helix-completion-strategy-v1') for one release, then delete legacy-forced-classification.ts, the unpatched branches (forcedStepTransition / nonForcedFailure's legacy paths, the gated catch and P17 sites). Update (WA MR2 Task 18, the orchestrator's fresh-start ruling): pre-WA-MR2 Temporal histories are drained, not replayed, so the pre-MR-2 fixtures were replaced by histories recorded on the WA MR2 code (src/__tests__/replay/fixtures/recorded/). The unpatched completion branch therefore no longer has any recorded history: it is dead code, and this FU can close by deleting it outright (no deprecatePatch release is needed). When retiring the patch, also evaluate wf.patched unconditionally at the companion-refusal site (runtime-temporal/src/workflow-dispatch.ts ~:1037): today it is evaluated only when a refusal is present — deterministic, but data-dependent. The retry-entry argument (retryEntry: true to initializeAgentState, ruling P22) is not gated: it is an argument, which replay never compares (see FU-TEMPORAL-ROLLING-DEPLOY-CROSSOVER-TEST).
FU-ORDINARY-PATH-TRUNCATION-NO-CODE: a run cut off at the model's token limit fails with no error code — open (owner: the orchestrator; a new public code is the user's ruling)
Filed: 2026-11-07, with DI-85. A max_tokens stop on the ordinary path fails the run with { message: 'Response truncated: token limit reached' } and no code on runtime-js, the CF DO and CF Workflows. The forced-completion path treats it as recoverable and retries. Core has no code for it, so an app cannot tell a truncated turn from a generic failure. DI-85 left it out on purpose: a code here is a new public error code (e.g. provider_output_truncated), which the orchestrator puts to the user. If they rule one, it rides DI-85's helper (errorStopReasonDetail), and DI-85's cell gets a .max-tokens-coded id.
FU-TEMPORAL-DBOS-FORCED-ERROR-STOP-CODE: Temporal's and DBOS's forced-completion entries do not reach DI-85's classification — open (owner: the Temporal and DBOS buckets, with DI-26)
Filed: 2026-11-07, with DI-85. DI-85 made one core classification (errorStopReasonDetail) for a model call that stops on content_filter, refusal, error or unknown, and wired it on runtime-js, the CF DO and CF Workflows, both on the ordinary terminal path and at the forced-completion entries. Temporal's and DBOS's forced entries build their failure without it, and their ordinary path still completes such a run (DI-26). DI-26's fix must use the helper on both paths, and its cell completion.error-stop-reason-coded then passes on those lanes.
FU-TEMPORAL-FORCED-MESSAGE-STRICT-TOOL: Temporal still tells a strict completion tool to answer "in the required structured format" (DI-84, pending) — open
Owner: WA (runtime-temporal). Surfaced by: the audio app on core 9.2, after the DI-80 bridge (DI-84).
A strict expected tool gets no output format on a native_output forced call (the DI-80 bridge, prepareForcedCompletionCall), so the correction message must name the tool instead of pointing at a format. core records the tool's strictness at phase entry (ForcedCompletionState.expectedToolStrict, from resolveExpectedTools), and buildForcedCompletionMessage then sends Final Step's instruction (finalStepInstruction). runtime-js, the CF DO (core's step iterator) and CF Workflows (steps.ts, both entry sites) pass the snapshot. Temporal's runLLMStep activity resolves the model's real capabilities, so it takes the no-format path, but its workflow (workflow.ts) enters the phase with only the expectedToolNames the activity returns, so a strict tool there still gets the old text (the conformance cell completion.strict-forced-call-names-tool on the temporal lanes). To close: runLLMStep returns the snapshot beside expectedToolNames and the workflow passes it (an old history lacks the field and keeps the old text; the message reaches the store as activity input, which replay does not compare). DBOS is out of scope: it passes LEGACY_RESOLUTION, so every forced call is forced_tool (a named tool choice, or 'required' over the narrowed set of several finishWith tools). Final Step (Release 5 on Temporal) replaces this path.
FU-TEMPORAL-FINISH-SIBLINGS-PATCH-DEPRECATE: retire the pre-finish-siblings plan fold — open
Owner: runtime-temporal. Surfaced by: the finish-siblings batch (spec 2026-11-03-finish-step-sibling-calls-design.md §3.1, §3.4).
A new Temporal execution passes finishSiblings: wf.patched('helix-finish-siblings-v1') (FINISH_SIBLINGS_PATCH, runtime-temporal/src/completion-strategy-contract.ts) to runLLMStep. Without it, runLLMStep folds a __finish__ step's sibling plan back into the terminal plan core produced before the batch (foldFinishIntoTerminalPlan, runtime-temporal/src/legacy-finish-plan.ts), so a history recorded before the patch replays the activity results it recorded. To close, once no workflow started before the patch can still be replayed: replace wf.patched with wf.deprecatePatch('helix-finish-siblings-v1') for one release, then delete legacy-finish-plan.ts, its test and the finishSiblings input flag.
FU-CLONE-INDEX-BOUNDARY-FORCED-PHASE: a message-index clone carries no forced phase — open
Owner: the clone-boundary design, with FU-BRANCH-SOURCE-FORCED-PHASE-PARITY. Surfaced by: !325 (RM-72), user decision 2026-10-08.
Intended rule (user): a message-index clone (cloneSession(from, to, { fromMessageIndex: N })) carries the forcedCompletion of the latest-WRITTEN checkpoint with messageCount <= N, or no phase when there is none. (A whole-log clone already takes the latest committed checkpoint's phase.)
Why deferred: no store records checkpoint write order. "Latest" must mean write order (the commit chain the pointer walks), not stepCount (not monotonic across turns) and not the highest messageCount; a rewind (truncate checkpoint) leaves discarded-history checkpoints behind, and walking back from the latest-written one is the only correct pick. Redis keeps the session's checkpoint index as a ZSET scored by stepCount; Postgres and D1 have only the millisecond timestamp column (ties are possible) and the ms stamp inside the checkpoint id; memory alone has an insertion-ordered array. The interface reads do not help: listCheckpoints returns metadata ordered by stepCount, listRunCheckpointMetas is per run.
Options: (1) a write-order sequence: a nullable seq column (Postgres, D1) and a Redis list or sequence score, read through a write-ordered listing, a store-contract change (recommended); (2) order by (timestamp, id), no schema change but wrong for commits in the same millisecond; (3) walk the chain from the pointer using runId and each truncate checkpoint's keepCount to skip discarded history, existing reads only, a different algorithm to confirm.
Interim (!325): a message-index clone carries NO forced phase (cloneStateWithoutCheckpoint(live, undefined)), never the live row's. Pinned by the store contract scenario clone.live-state in core/src/testing/commit-checkpoint-operations.ts (message-index case, forcedCompletion: undefined). The fix adds the contract cases: N on a checkpoint's messageCount, between two, below all, and after a rewind with a different-phase discarded checkpoint.
FU-CLONE-STREAM-OWNER-RUN-FENCE: a superseded run that owned a stream under another session id is never CP-85-fenced — open (LOW, owner: CF backlog)
Surfaced by CP-129 (W1, WA review). A session can own a stream whose id is not its own: a clone's run claims and writes its source's stream (runtime-js RM-72). Its chunks name the clone (agentId !== streamId), so chunkFenceRunId (session-based, core stream/run-markers.ts) never fences them. Once a newer run claims that stream, the old run's late chunks to the ACTIVE stream still land (pre-existing; CP-129's decideStreamWrite classifies them by the owner run only to keep them out of the guest drop). Session-based fencing must stay: fencing every chunk by run would drop children's chunks, which never carry the owner run. Fix direction (WA): a run-scoped fence keyed on the stream's owner history: drop a chunk whose agentId !== streamId and whose runId was ever the stream's owner but is not the current one (each manager records the owners it has had).
FU-DO-STREAM-GUEST-TO-MISSING-STREAM: on the two Durable Object stream managers a child's chunk to a stream that no longer exists appends instead of dropping — open (LOW, owner: CF backlog)
Surfaced by WA's review of CP-129 (!378). Core decideStreamWrite treats a stream that no longer exists (missing: deleted, or expired) as non-active: a child's chunk is dropped and an own chunk is refused. store-memory and store-redis apply that (Redis: the write script answers GUEST_DROPPED / missing when the stream's meta is gone, e.g. after its TTL).
- The two DO managers cannot tell a missing stream from a fresh one on the write path. runtime-cloudflare's
DOStreamManager.getStatus()defaults toactivewith no status row, and the in-process manager never answers "absent" (RM-71). store-cloudflare'sStreamServerinfers existence from several markers (/info'sexists), and the binding'screateWriter/inits the stream. - So a guest chunk to a wiped DO stream appends (re-creating it) instead of dropping.
- Rare: a DO stream is wiped only by a storage reset.
- Fix direction: pass "missing" to the decision from each DO's own existence rule (StreamServer:
existsfalse;DOStreamManager: no status row and noinit_claimed), and add a contract hook that removes a stream so the case is aGC.*contract case on every manager.
FU-FRESH-HISTORY-TRAILING-UNPAIRED: a plain execute({ history }) ending in unpaired calls is not paired — open
RM-117 pairs the seed of a branch on every runtime. A fresh session seeded by history: with no branch has the same shape: if the caller's history ends in calls with no result, the first input leaves them unpaired mid-history. It is left out of RM-117 by ruling, because the rows are the caller's own. To decide: pair them the same way, or refuse the entry typed.
FU-SDK7-STOP-NOTICE-OTHER-RUNTIMES: CF Workflows, Temporal and DBOS write no stopped-turn notice — open (owner: WA; after the features window)
Surfaced by: SDK-7 (user ruling 2026-11-09: runtime-js and the CF DO first).
- What: a turn stopped before any reply leaves
… U1. On runtime-js and the DO, the next input entry writes coreplanStoppedTurnNotice's notice beforeU2. CF Workflows, Temporal and DBOS write none, so their next turn's model still seesU1 U2. - Gaps: the
stop.no-reply-notice*executor cells are declared gaps on those lanes. - Each needs: the planner at its new-turn and
with_messageentries, in its run-start commit before the input. Temporal and DBOS must do it inside their deterministic entry activity / step.
FU-ENTRY-CONTEXT-OTHER-RUNTIMES: CF Workflows, Temporal, DBOS and agent-server refuse an entry's context — open (owner: WE-2; after the features window)
Surfaced by: entry context (spec 2026-11-11, batch 22: runtime-js and the CF DO first).
- What: an entry's
contextpieces are committed on runtime-js and the CF DO. CF Workflows, Temporal and DBOS refuse a non-emptycontextonexecute(),resume()andretry(), and agent-server onPOST /startandPOST /resume, withframework_not_supportedbefore any write (corerefuseEntryContext). - Gaps: the
context.*entry cells are declared gaps on those lanes;context.entry-refused-unwiredis their cell. - Each needs: the rows in its entry's run-start commit after the input ('data' before 'app'), the drain-first carrier,
planRetry's context argument at its retry site, and the digest's ids. Temporal and DBOS inside their deterministic entry activity / step.
FU-ENTRY-CONTEXT-CONTINUE-DRAIN: a continue that answers pending tool calls refuses entry context — open (owner: WE-2; after the features window)
Surfaced by: entry context (spec 2026-11-11, batch 22), runtime-js unit entry-context-resume-retry.test.ts.
- What: a
continue's pieces ride its run-start commit, but its drain appends a submitted client-tool result (and an awaited child's result) after that commit, so the pieces would land between the call and its result. On runtime-js and the CF DO such acontinuerefuses a non-emptycontextwithframework_not_supportedbefore the commit; acontinueof a stopped turn (its calls healed in the same commit) takes it. - Needs: the drain-first carrier for
continue(the pieces stashed on the carrier entry, delivered after the last result) without thewith_messageplan's not-executed rule for a deferredfinishWith, which acontinueruns. No caller needs it in batch 22: the chat route sends no context on a tool-result request.
FU-PROVIDER-OPTIONS-SCHEMA-TYPE-MISMATCH: an undefined member nested in an option type-checks but is refused at entry — open (WA, unscheduled)
Owner: WA, its own MR after !919. Surfaced by: the llm-vercel caller-parts fix (!919), which normalises the adapter's own input instead.
Problem: core's ProviderOptions (core/src/types/provider-options.ts:18, origin/main 07819d59eb) is built on the AI SDK's JSONValue, whose object members may be undefined; its schema providerOptionsSchema (:24-27) takes an option value through jsonValueSchema(), whose object arm takes no undefined member (core/src/state/json-merge-patch.ts:36). So a nested member (an object inside an option) left undefined type-checks and is refused at runtime, typed; a top-level option member left undefined passes. Seen in !919's repro: anthropic.caller = { type: 'direct', toolId: undefined }, refused invalid_union at anthropic.caller. It is reached only through in-process app, tool and hook input: contextPieceSchema (core/src/types/context.ts:202-211, parsed at core/src/orchestration/entry-context.ts:86 from execute() / resume() options) and contextItemSchema (:72-99). Loud, no data loss. Not reached by store reads (every store JSON-serialises on write) or HTTP bodies (JSON), per WA's trace.
Fix direction: drop undefined members before the JSON check in the core schema, with a unit on a nested undefined member through both schemas.
FU-CONF-DO-STEP-COMMIT-FENCE: no conformance cell proves the CF DO step commit's run fence — fixed in code (lands with its MR)
Surfaced by: Save-First §14 MR α's record slot (slot-sfar, 2026-11-09). Mutation M8a removed DOStateStore.saveStateAndPromoteStaging's superseded-run fence (this.assertFenceSync(sessionId, checkpointMeta.runId)), and no α row went red: save-first.zombie-result-refused-host's late result goes through the DO's APPEND commit, whose fence (M8b) the cell does prove. W1 confirmed no β cell reaches the step commit from a stale owner either (an evict kills every pre-crash body).
- Owed: a cell where a body still alive in a LIVE isolate after its run was detached (
stopDeadlineMs) finishes into the STEP commit rather than the append. For example a zombie with no saved row before it, or a step whose results commit throughsaveStateAndPromoteStagingafter a deadline detach. Judged with M8a.
Closed by: a store contract case, not a cell. fence-operations.ts has "commit.rejects-superseded-run: saveStateAndPromoteStaging by a superseded run with the current expectedVersion is rejected and writes nothing". The superseded run passes the CURRENT version, so only the run fence can refuse it. It runs on every store through stateStoreContractTests, DO-SQLite included. Proof (local slots, 2026-11-10): every store green with the case (slot-dfc). With M8a, the DO contract file reds exactly the three step-commit fence cases (slot-dfc2): the two in fence-operations.ts, plus "refuseIfStopped runs AFTER the superseded-run fence" in stop-operations.ts.
Why no cell: at runtime, a stale DO body's step commit is also refused by the version CAS. The fence is decided first (do-state-store.ts:3834), then the CAS (:3842). The run loop passes expectedVersion: state.version (runtime-js run-loop.ts:4067). The stop that detaches a run writes its K5 through commitState, which bumps the version (do-state-store.ts:4882), so a body that outlives its detach carries a stale version. A cell would see the CAS refusal under M8a too and could not isolate the fence. The contract case can (lines at main 41ea410715).
FU-RESULT-NOT-IN-HISTORY-NO-PRODUCER: the not-executed reason result_not_in_history has no producer — open (owner: WA; a product decision for the user)
Surfaced by: WA's §5.6 answer for the Save-First §14 cells (2026-11-09, read at main 7b92366025).
- What: core defines
result_not_in_history(orchestration/transcript.ts:32,76,89: the reason and its text, "this call's result is not part of this history, and the tool may have run"), and only a text unit reads it (saved-call-recovery.test.ts:345). Nothing inpackages/*/srcwrites it. - Why: spec 2026-11-05 §5.6 meant it for a rewind or branch at a saved checkpoint. The batch-14 ruling (§4.3) made
from_checkpointat a saved append a recovery andretry({ checkpointId })there anoutcome_unknownheal, so the rewind half is gone (the orchestrator ruled the shipped contract stands, 2026-11-09); a branch's heal still pairsunpaired_on_continuation(CP-286). - To decide: wire it where a branch / clone heals a saved row (RM-117's delegation cell pins the old reason, so it changes on purpose), or delete the reason and its text.
FU-BRANCH-REENTRY-OWN-CALLS: a JS branch re-entering its own session leaves its own calls unpaired under its input — open
Surfaced by: RM-117's excludeIds review.
A second execute() with the same checkpointId / messageIndex branch on a target that exists skips cloneSession and loads the target. The entry's checks are skipped too (isCheckpointBranch: the clone is the gate, and here there is no clone), and the new-turn settlement is skipped (!branch). RM-117 leaves the target's own pending calls and reconcile-owned remote calls out of its pairing, as it must: a pending entry would stay live beside a row, and a remote child's outcome can only come from its producer. So the input lands after them. Seen on the strict mock: a target paused on a client call fails with the unpaired-call error, with the pending entry kept. A live run is still refused at the run-start commit. RM-117 did not change this shape; before it the same run failed the same way. Fix: treat a re-entry as the continuation it is (the entry's checks and settleRemoteChildrenBeforeInput / planNewTurnPendingSettlement), or refuse it. Pinned in branch-trailing-heal.test.ts.
FU-BRANCH-SOURCE-FORCED-PHASE-PARITY: Temporal and CF Workflows branches carry no forced phase and restart stepCount at 0 — open
Owners: WB (CF Workflows step-loop session, CF backlog) for CF Workflows; the Temporal bucket (parked) for Temporal. Surfaced by: !325 (RM-72), user decision 2026-10-08. Related: FU-CLONE-INDEX-BOUNDARY-FORCED-PHASE.
Problem: Temporal and CF Workflows branch through core loadBranchSource (core/src/store/checkpoint-ops.ts), which carries only the messages and customState. A branch there carries NO forced phase on any branch kind and starts at stepCount 0, unlike cloneSession (JS, DBOS), which takes the checkpoint's stepCount and phase: a silent divergence between runtimes.
Fix direction: core loadBranchSource reuses the same core clone-state rule !325 built, cloneStateFromCheckpoint / cloneStateWithoutCheckpoint (boundary-wins forcedCompletion, the stepCount policy), so Temporal and CF Workflows inherit it from one place. No per-runtime patches (DoD §F2).
FU-COMPANION-SEND-DURING-FINAL-CALL: a send during a companion's final call is delivered, never read — done (!918, CP-332)
Owner: W1. Tracks: catalogue records CP-332, RM-139. Surfaced by: the CP-330 re-trace (2026-11-16).
Problem: on runtime-js, CF Workflows and Temporal, companion__sendMessage to a companion whose run is live appends the message under that run and answers delivered: true. When the message lands after the run's last history load (during its final model call), the run commits its finish and its K5 without reading it: the companion settles on its earlier answer, and the message is read only at its next consult. The CF DO (a /resume, refused child_busy while the child runs) and DBOS (the live loop's inbox, taken as its next turn) deliver only into a run that reads the message.
Fix (user ruling 2026-11-16): before a run commits its finish it re-checks its log; a row that arrived after its last history load gets one more step instead of the finish, and a late append makes the terminal commit retry, so delivered means this run reads it. runtime-js, CF Workflows and Temporal in one change.
Proof: companion.send-during-final-call-read (executor) and companion.send-during-final-call-read-host (host) flip to pass on every ledgered lane with .delivered still passing; dbos-postgres and cf-do stay green.
Done (2026-11-17, !918, CP-332): a run never finishes over a row it has not read: the store guards G1-G3 refuse a terminal or step commit over a moved count and an append into a run whose K5 landed (answered child_exited), and each runtime's finish re-checks its log and takes one more step (runtime-js and CF Workflows; Temporal behind its lateInputFinish input flag). The two cells flip on all 8 ledgered lanes, 3 of 3, with .delivered passing; dbos-postgres and cf-do stay green (W1's slot s-cp332 run 4). Temporal's result append and CF Workflows' superseded-results append no longer drop a result whose call is not trailing (RM-139): a late row between a call and its result leaves the result in the log.
FU-DOCS-AFTERLLMCALL-LATE-INPUT-REDO: the hooks guide does not say an overtaken finish fires afterLLMCall for a reply that is not the turn's answer — done (docs only; owner: W1; !925)
- Filed 2026-11-18 by the conformance owner, at the orchestrator's request (an app session found it after the b31 release). Related: CP-332, FU-COMPANION-SEND-DURING-FINAL-CALL.
- Since CP-332, a finish overtaken by a row the run has not read is not taken: the run takes one more step.
- When the row lands during the final model call, the finish is not taken and the step is redone over it, the reply discarded: runtime-js and the CF DO (
core/src/orchestration/step-iterator.ts:2996,redo_step), CF Workflows (runtime-cloudflare/src/workflow.ts:5822-5832,redoThisStep) and Temporal for a text reply (runtime-temporal/src/workflow.ts:1925-1927). A Temporal step that called a finish tool is instead committed after the row (:1928-1930), and the next step reads both. - When the row lands while the completion hooks run, every runtime keeps the reply in the log before the row, takes one more step, and
onAgentCompletefires again at the finish that lands (ruling D2,runtime-js/src/run-loop.ts:6786-6788). - Either way the overtaken call has already fired
afterLLMCallwith its successful result (step-iterator.ts:2301, before the commit).
- When the row lands during the final model call, the finish is not taken and the step is redone over it, the reply discarded: runtime-js and the CF DO (
- Where:
docs/guide/hooks.md:288-291(origin/main 4d2f6e9afc) describes per-attempt firing for a retried call only, so a host that takesafterLLMCall(or the firstonAgentComplete) as the turn's final answer can record a reply the run does not keep. - Fix: one paragraph there for runtime-js, the CF DO, CF Workflows and Temporal: the run's answer is the last finish's, its result and its last
onAgentComplete.
Done (!925, docs/guide/hooks.md, as on origin/main fae53b26ac): one paragraph for runtime-js, the CF DO, CF Workflows and Temporal: the run's answer is its last finish, the handle's result() and the last onAgentComplete. (The earlier bullet that CF Workflows and Temporal keep the reply on a model-call overtake was wrong; corrected above.)
FU-COMPANION-SEND-UNSYNCED-REF: sendMessage reads a companion's ref without the lazy sync — done (!932, CP-330)
Owner: W1, after CP-332 (its fix also touches handleSendMessage). Tracks: catalogue record CP-330. Surfaced by: the CP-330 re-trace and its fixed cell (2026-11-17).
Problem: core's handleSendMessage reads the parent's refs with a plain getSubSessionRefs, while listChildren, getChildStatus and waitForResult read them through the lazy sync resolveAndSyncSubSessionRefs. On CF Workflows and Temporal nothing writes a non-blocking companion's ref when its detached continuation completes, until a later sync, so a send made right after that answer reads a running ref over a completed session and is refused child_exited. runtime-js writes the ref in process; DBOS has its own send.
Fix direction: handleSendMessage reads its refs through resolveAndSyncSubSessionRefs, as its sibling handlers do: one change in core for both runtimes.
Proof: companion.send-after-continuation flips to pass on temporal-memory, temporal-redis, temporal-postgres and cfw-workflows-d1, while the js lanes and dbos-postgres stay green.
Done (!932, CP-330): handleSendMessage syncs the named companion's ref through syncSubSessionRef, the per-ref body it now shares with resolveAndSyncSubSessionRefs (core/src/orchestration/companion-tool-dispatch.ts:2183, :2526, as on origin/main f0abcc475b): one change in core, as the fix direction said. The proof held: in !932's pipeline 3087479794 companion.send-after-continuation passed on temporal-memory, temporal-redis, temporal-postgres and cfw-workflows-d1 (unexpected-pass against CP-330's ledger, now dropped), and the js lanes and dbos-postgres stayed green.
FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW: a companion resume that loses its run-start still reports delivered: true on Temporal and CF Workflows — open for Temporal (the CF Workflows half done in W1 MR2b)
Owners: Temporal half: the parked Temporal bucket (orchestrator). CF Workflows half: done (W1 MR2b). Tracks: catalogue record CP-105. Surfaced by: completion-strategies test hardening MR 1, Task 16 (CP-103).
CF Workflows half: done (W1 MR2b, Task 6). The outcome is core CompanionDeliveryOutcome ({ delivered: true } | { delivered: false, reason, errorDetail? }, six reasons). A sendMessage resume of an interrupted child, and a detached continue of a completed one, record the tool result only after the parent polled, in its body, for the child's exact run id (awaitCompanionEntryOutcome, companionEntryOutcomeTimeoutMs, 60 s): the run exists → delivered: true; the child instance closed with no run → child_busy (a lost run-start) or child_entry_failed with its errorDetail (e.g. resumed_companion_ref_missing, a registry miss); no outcome → child_start_unconfirmed. The mechanism is a store poll, not the event + waitForEvent this entry first proposed (orchestrator Q1: "same outcome, engine-specific mechanism"). The continue path's ref write moved into the child's run-start commit (companionRef). Tests: runtime-cloudflare/src/__tests__/cfw-companion-entry-poll.test.ts, workflow.test.ts, cfw-persistent-companion.test.ts, cfw-persistent.wf-noiso.test.ts / cfw-continue.wf-noiso.test.ts (real engine). The CF DO and DBOS answer the same union (child_entry_failed with errorDetail).
Temporal half (still open). Temporal must report the same core CompanionDeliveryOutcome, by its own mechanism (a signal from the child, or an activity read of the child's run-start commit; Q1's contract is the outcome, not the mechanism). Today it reports delivered: true once the continuation's child workflow starts, and answers child_start_failed (runtime-temporal/src/workflow-dispatch.ts ~:1051 / ~:1080) when the child workflow did not start: a reason OUTSIDE the core union (COMPANION_NOT_DELIVERED_REASONS). Map it into the union (e.g. child_entry_failed with its errorDetail) and parse the result at the boundary with core parseCompanionSendMessageResult, as runtime-js and CF Workflows do. The rest of this entry is the Temporal half's analysis.
companion__sendMessage to an interrupted persistent companion starts the resumed child fire-and-forget: Temporal's wf.startChild (runtime-temporal/src/workflow-dispatch.ts, the __resume__ child) and CF Workflows' createChildInstance (the __resume__ instance). Since CP-103 the parent's ref is written running only by the child's own run-start commit (StartRun.resumedCompanionRef, A1), and a resume refused for completion support is reported before any child starts (a tool error from the parent's companion activity on Temporal, from its companion step on CF Workflows). But when the child's run-start LOSES to another entry (child_busy), the resume correctly writes no ref (on CF Workflows the ref stays interrupted; on Temporal it keeps its prior status, the stale running of CP-73, since Temporal never records interrupted on a ref) while the parent's tool result still says delivered: true: neither workflow learns the entry's outcome. runtime-js reports { delivered: false, reason: 'child_busy' } here. CP-103 is fixed; this symptom is tracked as CP-105.
Both halves must land with the SAME delivery contract, so the runtimes do not diverge:
- the resumed child signals its entry outcome to the parent (Temporal:
SignalExternalWorkflowExecution, gated in the child; CF Workflows: its engine equivalent, an event to the parent instance); - the parent applies a bounded, patch-gated wait for that outcome after starting the child (Temporal:
wf.conditionwith a timer bound behindwf.patched; CF Workflows: its engine equivalent, a boundedwaitForEvent); - the tool result reports
{ delivered: false, reason: 'child_busy' }on a lost start, and{ delivered: false, reason: 'child_start_unconfirmed' }on a timeout (DBOS's existing reason).
Each half needs its replay / re-execution test (a pre-change history replays). No control forces child_busy on Temporal or CF Workflows today, so the CP-105 conformance cell needs one first.
A missing parent ref has the same symptom (CP-103 A1, Task 16B). Since A1 the resumed child's run-start commit carries StartRun.resumedCompanionRef, and every store refuses it typed when the parent's ref (or the parent session) is gone: RunStartRejectedError('resumed_companion_ref_missing'), nothing written. runtime-js reports that as a typed tool error. On Temporal the parent has already answered delivered: true, and the child workflow ends failed / state_run_start_conflict with details.cause: 'resumed_companion_ref_missing'. Today's outcome is pinned by runtime-temporal/src/__tests__/companion-resume-missing-ref.integ.test.ts; the fix must flip its delivered: true assertion. The delivery contract above must report this case as a typed failure, never child_busy (it is not a lost claim: no other entry won). To do that:
- Temporal's
lostClaimType(runtime-temporal/src/lost-claim.ts:24) keys on the failure type only, andtoActivityFailuregives this cause the samestate_run_start_conflicttype as a lost run-start. Soworkflow.tstreats it as astart_conflictloser. The child's entry outcome must tell the two apart, either with its own failure type or by checkingdetails.cause, or CP-105'schild_busyreport will mislabel it. - CF Workflows passes the field since Task 16 Step 5 (the
companionResume__resume__instance;companionRefsince W1 MR2b, also on the__continue__instance). Its outcome before W1 MR2b (nowchild_entry_failed, above): the child instance returnsfailed/state_run_start_conflictwith causeresumed_companion_ref_missing(errorDetail.cause; thestart_rejectedlost-claim record,LostClaimRecordSchemainruntime-cloudflare/src/run-entry.ts, carries the cause acrossstep.do) and sends the parent instance anerroredsub-agent-complete event, while the parent'ssendMessagehas already recorded{ delivered: true }.runtime-cloudflare/src/__tests__/completion-children.wf-noiso.test.ts("a resume entry whose parent ref is gone") pins the CHILD side only: it creates the resume instance directly, with no parent run. Unlike Temporal'scompanion-resume-missing-ref.integ.test.ts, no CF Workflows test pins the parent'sdelivered: true, so the CFW half of the fix must first add a parent-side red case (a parent run whosesendMessageresumes a child whose ref is gone by the child's commit) and flip it. It must report the case typed from that cause, neverchild_busy. DBOS also records it as astart_rejectedlost claim (runtime-dbos/src/steps/run-commit.ts:255); any DBOS resume path must not report it aschild_busyeither. - One definition of "lost claim" (!325 final review, type-design M1). The run-start reject causes are written out by hand in four places: core
isLostClaimError(which excludesresumed_companion_ref_missing), CF WorkflowsLostClaimRecordSchema/lostClaimRecordOf(runtime-cloudflare/src/run-entry.ts) and DBOSLostClaimRecord/lostClaimRecordOf(runtime-dbos/src/steps/run-commit.ts), both of which still accept it as a lost claim. When this FU lands, export from core aRunStartRejectCauseenum (Zod) and its lost-claim subset, derive the CFW and DBOS lists from them, and decide once, in core, that this cause is never a lost claim (or give it its own error class), so a fifth cause cannot drift. Narrowed (2026-11-06, CP-278, !743): core now has the one table (REJECT_CAUSE_CLASS,rejectCauseClass, incore/src/run/reject-cause-class.ts, shared withclassifyEntryFailure), andisLostClaimErrorreads it: only a cause classedlostis a lost claim. Still open: the CF Workflows and DBOSlostClaimRecordOflists are written by hand, and are not derived from that table.
A child agent type missing from the registry has the same symptom. The parent's completion pre-check (Temporal completionRefusal, CF Workflows companionResumeRefusal in runtime-cloudflare/src/steps.ts) reads a registry miss (core isAgentNotFoundError, framework_agent_not_found, on CF Workflows and Temporal since WA MR1; any other registry throw is a failed companion call) as "no refusal", so the resume starts and the parent records delivered: true. The child entry then fails with the registry's own error before its run-start commit (CF Workflows: workflow.ts's entry registry.get(agentType), then the pre-commit failure path, an errored event to the parent; the ref stays interrupted). On CF Workflows one deployment serves the parent and the child, so such a miss is conclusive at the parent. The delivery contract above must report it typed too (the child's entry failure), never child_busy.
FU-CFW-INVALID-INPUT-UNTYPED: a CF Workflows instance whose input fails its parse returns an untyped failure and notifies no parent — open
Owner: the CF backlog (W1). Surfaced by: completion-strategies test hardening MR 1, Task 16 Step 5 review (minor 2).
runAgentWorkflow parses the instance input with AgentWorkflowInputSchema.safeParse and, on failure, returns { status: 'failed', error: 'Invalid workflow input: …', errorDetail: { message } } (runtime-cloudflare/src/workflow.ts, the "Validate workflow input" block, ~:594-604). The failure is loud, not swallowed, but:
- It carries no code. A parse failure is a known classification (
validation_error);resolveErrorDetail's "never invent a code" rule does not apply to an error the runtime itself classifies. Type itvalidation_error(category fromcategoryFromCode, non-retryable), so an executor waiting on the instance's result, and a reader of the run that never started, get a typed error. - It notifies no parent. A child instance whose input does not parse never sends its
parentWorkflowIda sub-agent-complete event, so a blocking parent waits outsubAgentEventTimeout. Decide whether a child whose payload still yields a parseableparentWorkflowId(andparentToolCallId,sessionId) notifies the parenterroredwith the typed detail; a payload too broken to name its parent cannot.
Since Task 16 Step 5 the parse has a cross-field rule (companionResume, renamed companionRef in W1 MR2b, requires parentSessionId), a new trigger for direct create() callers. The product's own __resume__ instance always satisfies it, so no product path hits this today. Tests: a unit case per point (workflow.ts over a bad payload: the typed result; a child with a parseable parentWorkflowId: the event, or a written decision not to send one).
FU-TEMPORAL-WAITFORRESULT-RESUME-RACE: an untimed companion__waitForResult can report a resuming companion interrupted before its run-start commit — open (known bug; owner: the Temporal backlog, parked with Temporal)
Surfaced by: completion strategies test hardening MR 1, Task 16 (CP-103), owner review 2026-10-08.
On Temporal, a waitForResult issued in the same step as a sendMessage resume of an interrupted companion reports interrupted when the resumed child's entry takes more than about 2 s to reach its run-start commit: core's isAwaitingResume returns true for ORPHAN_CONFIRMATIONS (3) consecutive observations of an interrupted ref with no live child run (core/src/orchestration/companion-tool-dispatch.ts, the awaitingResumeObservations debounce). The live-run guard only narrows the window to "before the run-start commit". The model may then send again. This is a timing heuristic (observation counts plus about 2 s), which DoD §F rejects, so it is a known bug, not accepted behaviour. CF Workflows waits for the child's event, so it is not affected. runtime-js (and the CF DO, which runs it) relies on the same debounce, with a narrower window: the child resumes in-process, but between core's handleSendMessage returning and the child's run-start commit, resumeChildAgent (runtime-js/src/run-loop.ts) runs a loadState, a loadLLMHistory and a getSubSessionRefs. On a remote store (Postgres, Redis) under load that can exceed the ~2 s debounce, and a same-step waitForResult then reports a false interrupted. Low probability; the same fix covers both runtimes.
Fix direction: with A1 (the parent's ref is written inside the resumed child's run-start commit, plan Task 16B, design note §2.8 (a)) a committed resume start is observable, so the wait keys on "run-start committed or refused" instead of observation counts. Needs the same entry-outcome signal as FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW, so plan them together. Documented as a known bug in docs/guide/sub-agents.md.
FU-TEMPORAL-REPLAY-FIXED-IDS: the replay recorder's workflow ids collide on a shared Temporal server — done (the replay run-nonce MR, stacked on the batch 28 owner records; owner: WA)
Owner: runtime-temporal test infra. Surfaced by: completion strategies MR 2 Task 11 review.
recordScenario (runtime-temporal/src/__tests__/replay/replay-scenarios.ts) runs each scenario under the fixed session id replay-<scenario name>, so its workflow ids are fixed too. The fixtures were recorded, and the replay test was run, on the per-process test server (HELIX_TEMPORAL_BACKEND=test-server), where that is harmless. On the default Docker backend (TEMPORAL_ADDRESS), one server is shared by every test process, so two concurrent runs of the fresh-history case (or a run while an earlier one's workflow is still open) collide on the id. To close: suffix the session id with a per-run nonce in the fresh-history path (the committed fixtures keep their recorded ids).
Also hit inside one run (2026-11-16). The full runtime-temporal integ on a FRESH Docker Temporal failed subagent-tool-start-replay's fresh recording with WorkflowExecutionAlreadyStartedError on test__replay-sub-parent__replay-local-subagent: completion-strategy-replay's fresh-history case records the same local-subagent scenario, and the two files ran in parallel. A fresh server does not avoid it; the nonce does. In CI (test:integ:rest, per-job Docker Temporal) the two files ran back to back in each of the 5 runs of 2026-11-15 (subagent-tool-start-replay starting 2 to 5 s after completion-strategy-replay ended) and passed, so CI has not hit it yet; a slower or more parallel runner can.
Closed (2026-11-16, the replay run-nonce MR): batch 27 had already narrowed it, giving recordScenario an explicit opts.sessionId and subagent-tool-start-replay its own (replay-local-subagent-cp280), so the collision above, measured before batch 27, no longer reaches that pair. The fix closes it for every recording: recordScenario's DEFAULT session is now replay-<scenario name>-<8-char nonce> (replay-scenarios.ts:268), with an explicit opts.sessionId still winning; the two fixture recorders pin replay-<scenario name> so re-recorded fixtures keep their ids (record-replay-histories.integ.test.ts:62, record-delegation-replay.integ.test.ts:61); subagent-tool-start-replay drops its now redundant override. replay-ids.integ.test.ts:50: a second recording of one scenario while the first is still open (its model call held) completes under its own workflow id.
FU-TEMPORAL-STEP-START-FENCE-WINDOW: a late step_start from a timed-out attempt can abort the current attempt — open
Owner: runtime-temporal. Surfaced by: completion strategies MR 2 final review (decision D-A).
D-A moved skills resolution and beforeLLMCall ahead of the attempt's step_start in runLLMStep (runtime-temporal/src/activities.ts ~:3192-3352). That widens the window in which a timed-out earlier attempt of the same step writes its step_start after the retry's: the llm-attempt fence (isLlmAttemptSuperseded, runtime-temporal/src/llm-attempt-fence.ts ~:47-54) orders attempts by step_start position in the stream, so it then treats the CURRENT attempt as superseded and aborts its model call. Rare (it needs a timed-out attempt plus a narrow interleaving), and the window existed before D-A; D-A only made it longer. To close: check the activity's cancellation immediately before writing step_start, or stamp attempt order independently of write order (e.g. the activity attempt number on step_start, compared by the fence).
A narrower residual remains after any such fix: a server-side activity timeout (or a result serialisation failure) between the held chunk's write and the activity's completion retries an attempt whose chunk already landed.
FU-CFW-LLM-ERROR-CHUNK-BEFORE-HOOKS: CF Workflows writes the held LLM error chunk before its hooks run — open
Owner: runtime-cloudflare. Surfaced by: completion strategies MR 2 review (Temporal G1).
CF Workflows executeAgentStep (runtime-cloudflare/src/steps.ts ~:2029-2095) writes the held error chunk (errorReport.settle) and then runs planStepProcessing, afterLLMCall and usage recording, all inside step.do with llmStepRetry (runtime-cloudflare/src/workflow.ts ~:2244). A throwing hook after a terminal error retries the step after the client already stopped reading at the error chunk, which violates E5 on CFW. runtime-js and DBOS are not affected (a hook throw there does not re-run the model call). To close: write the held report as the step body's last action, as Temporal does after MR 2 (runtime-temporal/src/activities.ts ~:3883-3900), with a test that a rejecting afterLLMCall after a terminal error writes no error chunk.
FU-TEMPORAL-REFUSED-CALL-EMPTY-STEP: a call refused before the model commits an empty step — open
Owner: runtime-temporal. Surfaced by: completion strategies MR 2 review.
A call runLLMStep refuses before the model (a prompt-prefix change on an ordinary or forced call, no completion strategy; refusedCall in runtime-temporal/src/activities.ts) returns nextState.stepCount = N+1 with no messages, and the workflow commits that step via commitStep before the terminal. The step checkpoint has no assistant row and step_committed names an attempt that never wrote step_start. Harmless today: the UI treats an unseen attempt as unscoped, retry() re-refuses, and the pre-MR-2 tool-missing path did the same. To close: skip the step commit for a refused call, or do not advance stepCount for it.
FU-COMPLETION-RETRYABLE-STEP-ERROR-CORE: lift isRetryableLLMStepError into core — open
Owner: completion strategies MR 3 (DBOS). Surfaced by: completion strategies MR 2 (ruling P13).
The rule "an LLM error step is retried when its detail is retryable: true or has no code" exists twice: runtime-cloudflare/src/steps.ts (isRetryableStepError, CF Workflows) and runtime-temporal/src/activities.ts (isRetryableLLMStepError, a copy kept package-local because MR 2 did not change core). DBOS needs it too. To close: export one helper from core (next to isTerminalLLMStepError), use it in all three runtimes, and delete the copies.
FU-COMPLETION-RETRY-AFTER-FINISHWITH-EXHAUSTION: retry() after a finishWith miss or a provider error ends the phase re-forces the spent attempt — open
Owner: completion strategies (cross-runtime). Surfaced by: the !312 wave 2a re-review (ruling P22 follow-on).
When a forced finishWith call misses (the tool fails) on the phase's last attempt, the phase fails in a phase-only commit after the step's rows. That commit is not a retry target, so retry()'s default target (planRetry: the failed run's last step commit, or a later append commit with no assistant row, such as the phase-2 tool results) still holds the phase pending at the attempt the run already spent. The retried run then re-forces that attempt instead of starting fresh, on both runtime-js and Temporal (on Temporal, retryRestoredForcedPhase in runtime-temporal/src/retry-forced-phase.ts reads a forced step's own transition commit only for a step target; an append target is restored as given). The same happens after a provider error at any attempt, retryable or not, on both runtimes (the non-retryable case is pinned by e2e/src/__tests__/forced-completion-retry-phase-parity.integ.test.ts): its forced step records no row, so the default target is the commit before it, which holds the phase pending. Only a miss the forced step recorded (under either strategy: a __finish__ answer, or a native_output prose or schema-invalid answer) is not re-forced (on Temporal its transition shares the step's counts and is read). To close: decide the intended semantics of a retry after an exhausted phase (re-force the last attempt, or restore the failed phase so the retried run works and re-enters a fresh phase on its next trigger), then make every runtime's retry target and restored phase match it, with a cross-runtime parity test.
FU-RETRY-RESTORES-PREVIOUS-TURN-PHASE: retry() of a new turn that failed on its first step restores the previous turn's forced phase — done on runtime-js, the CF DO and CF Workflows; open on Temporal / DBOS (parked)
Done (W1, DI-40) on runtime-js, the CF DO and CF Workflows. A run's input checkpoint now records the state the run entered with: StartRun.entryState ({ forcedCompletion: ForcedCompletionState | null; clearOutput?: boolean }) is written onto the row and into the input checkpoint inside the run-start commit on every store (memory, Redis, Postgres, D1, DO-SQLite; contract cases run-start.entry-state.*). A new turn passes { forcedCompletion: null, clearOutput: true }, a retry the restored phase, and a from_checkpoint resume the target's; a continue resume passes nothing, because the stored row is its entry. Open on Temporal and DBOS (parked): their turn and retry entries still pass no entryState, so the mechanism below still holds there. Fix: pass it at their run-start sites (runtime-temporal/src/activities.ts initializeAgentState / retry, runtime-dbos/src/steps/run-commit.ts), as runtime-js / CF Workflows do.
Owner: a dedicated cross-runtime MR after !325, in the CF backlog (workstream WF), owner TBD by the orchestrator (user ruling: out of scope for the completion-strategies test-hardening MR, which must not touch the run-start or retry path). Tracks: catalogue record DI-40. Surfaced by: !325, Task 7A (RM-72) probes.
Mechanism. On runtime-js's store path, a new turn's run-start commit (commitRunStart, runtime-js/src/js-agent-executor.ts:2909) writes the run's input checkpoint from the session row as it was BEFORE the entry. The entry-state write that gives the turn its WORKING state (no phase, from createInitialAgentState) comes after it (writeEntryState, :2987; a plain saveState unless customState changed). So the input checkpoint holds the previous turn's forcedCompletion, or the clone's. When the run fails on its first step, planRetry rule 2 targets that entry (core/src/store/retry-plan.ts:323-324, fieldsOf :113-119) and restores its phase (js-agent-executor.ts:4024). The retried run then opens with a forced call although the failed run's entry was WORKING, so the model never does a work call on the turn's question. DBOS likely (inferred, not probed): DBOS retry() restores forcedCompletion: restored?.forcedCompletion from the same kind of target (runtime-dbos/src/steps/run-commit.ts:743) and runs with isResume: true (runtime-dbos/src/lifecycle/retry.ts:143), which skips the first step's new-turn reset. The CF DO may be clear: its claim writes the hardened entry state (toSessionState) in the run-start transaction (not verified).
Probes (store-memory + JSAgentExecutor, after the RM-72 store fix and the toSessionState hardening; the first step errors non-recoverably, then retry() with no message):
- (a) a plain continuation of a session left
interruptedin a pending phase,execute('next turn'): the retried first call isforced_completion(its last user message isnext turn); - (b) a branch at a checkpoint whose phase is
pending(the P2-18 shape; the clone carries the checkpoint's phase by the RM-72 rule):forced_completion; - (c) a
messageIndexbranch of a source left in a pending phase: wasforced_completion; nowwork, because a message-index clone no longer copies the live phase (RM-72). Kept as the control.
To close: decide where a new turn's entry phase is recorded (e.g. clear forcedCompletion in a new-turn run-start commit, or record the entry's phase on the run for planRetry), apply it on runtime-js, the CF DO, DBOS, Temporal and CF Workflows, with a cross-runtime retry test of shapes (a) and (b).
FU-RUN-START-COMPLETED-WITHOUT-OUTPUT-WINDOW: a new turn commits a row reading completed with no output — done (CP-333, !929)
A new turn over a completed session commits, in its run-start commit, an entry state that clears the output (clearOutput, DI-40) and moves current_run_id to the new live run, but keeps the status: the status changes only at the runtime's next, separate save (runtime-js writeEntryState, or the companion continuation's saveState). In that window the row reads completed, no output, current run live. Measured by WA on store-postgres (an audit trigger, 2026-11-17): every run, twice per run, about 8 to 164 ms. A reader there takes the turn as finished with no answer: runtime-js getHandle().result() resolves completed at once with output: undefined, naming the live run. A runtime-js retry() over a failed session has the same window: its row reads failed, with the failed run's error, under the new live run, and result() reports that failure.
Where: every store's run-start entry-state write (memory, Redis, Postgres, D1) and its callers on runtime-js (new turn, companion continuation, rewind, retry over failed) and CF Workflows (new turn, rewind). The CF DO has none: its run-start commit and entry save are one transactionSync. Temporal and DBOS: FU-TEMPORAL-DBOS-RUN-START-ENTRY-STATUS.
Fix: let the run-start commit write the entry's session status with its entry state, and clear the entry's prior error, errorDetail and failureReason (RunEntryState.clearError, ruled 2026-11-18), so no committed row pairs a live run with the previous terminal's output or error; or make the readers decide from the current run. A store contract case and a runtime-js seam pin prove it (no cell can hold the window).
Also (harness): companion.send-after-continuation's predicate (send-after-continuation.ts:130-131) stringifies an absent output and throws, so a poll in the window crashes the cell. Make it read a missing output as "not answered yet".
Tracks: catalogue record CP-333. Surfaced by: CI job 18128925136 (!922), WA's probe.
Done (!929, CP-333 and CP-335): a run-start commit now writes the entry's session status (RunEntryState.status) and clears a previous failure (clearError) on all five stores, and every runtime-js and CF Workflows entry passes them (a new turn, a rewind, a with_message turn, a retry, a companion continue); the CF DO, which had no window, passes the same fields. An entry that throws after its run-start commit ends the session too. The readers (getHandle, /status) report one run's fields across a new turn's run-start (CP-335). Proven by WA's slot-cp333 rev 7 (256 checks OK): the run-start contract on every store and the runtime-js, agent-server and CF Workflows window pins, red at the base and green at the fix. Not done, by ruling (2026-11-18): send-after-continuation's poll stays unguarded, so a recurrence of the window crashes loudly.
FU-TEMPORAL-DBOS-RUN-START-ENTRY-STATUS: a Temporal / DBOS turn's row reads the previous turn's status and output while its run is live — Temporal done (!936, CP-334); DBOS open (parked; owner: WA)
Temporal's and DBOS's run-start commits carry no entry state (commitEntryRun, activities.ts:1870; commitDbosEntryRun, run-commit.ts:979): the status turns active only in the later writeEntryState (activities.ts:1979, run-commit.ts:1106). In that window the row reads the previous turn's completed, with its output, under the new live run. Temporal's reconnect result() resolves such a row at once (executor.ts:2099-2109), so a getHandle() there reports the previous turn's output as the live turn's result. DBOS's handle answers from its run (DI-78); only its row is stale.
Fix: CP-333's part A field (the entry's status in the run-start commit's entry state), passed by the Temporal and DBOS entry builders with clearOutput; a Temporal seam pin proves it. Order: Temporal right after CP-333's fix; DBOS stays parked. Tracks: CP-334. Related: CP-333, FU-COMPANION-REOPEN-IN-RUN-START-COMMIT.
Temporal's reconnect also reads the session row and the current run in two reads (state-helpers.ts:37 and :46), the shape of CP-335; its fix folds in here.
Temporal done (!936, CP-334): every Temporal entry's run-start commit now carries its entry state (runtime-temporal/src/activities.ts:12030-12037, initializeAgentState; :1745-1752, :1816-1824 and :1935-1945, startRunCommit's rewind, with_message and retry entries, as on origin/main f0abcc475b), so the row reads active with no previous output under the new live run, and getHandle has CP-335's fence (executor.ts:1849-1891). Still open: DBOS (parked): commitDbosEntryRun still carries no entry state, so its row is stale in the window; its reconnect handle answers from the run (DI-78), so no reader is known to report it.
FU-CFW-RECONNECT-OUTPUT-FROM-LIVE-ROW: a CF Workflows reconnect that finishes its run can report another turn's output — fixed in code (lands with its MR; severity: CP-336)
CF Workflows' reconnect result() settles its captured run (executor.ts:5860-5863) and then reads the session row (:5872); a completed settle answers that row's output (:5873-5879). A run that already ended is guarded (no settle, :2284-2288, then the C3 guard reports its own record, :5888-5891), so only a run this reader finishes itself is exposed: if a new turn's run-start commit lands between the settle and the read, the captured run is reported completed with the new turn's output (none after CP-333). Traced from source, not run. Fix: take the captured run's outcome from its own K5 (core readRunEndOutcome), never from a later read of the session row. Tracks: CP-336. Related: CP-335.
Fixed in code (as on branch reliability-cp336-cfw-reconnect-own-output dee60def80): the reconnect result() answers a run it finished from that run's own K5 through core readRunOutcome (executor.ts:5887-5900), not the readRunEndOutcome named above (no such reader exists). Proven by the unit pins CP-336 cites.
FU-TEMPORAL-COMPLETION-CONFORMANCE: DI-24 / completion-strategy conformance cells for Temporal — open
Owner: the conformance owner (this work does not edit packages/e2e/src/conformance/** or docs/conformance/findings.md). Surfaced by: completion strategies MR 2.
The DI-24 Temporal cells' ledger update and the runtime-temporal citation re-pins landed in MR 2 (the conformance-owner commit in MR !312: discipline.error-step-detail passes on temporal-*, and DI-24's ledger lists only dbos-postgres). What remains is capability-profile conformance coverage for Temporal: forced-completion scenarios on the anthropic-5.5 profile (and any other per-profile completion-strategy scenario) wait on FU-COMPLETION-CONFORMANCE-WORKER-CAPABILITIES' per-cell profile selection. To close: run those scenarios on the temporal-* cells once the conformance worker can select a profile per cell.
FU-COMPLETION-TEST-HARDENING-MR1: deferred test-audit items for the MR 1 runtimes — done
Closed by: the completion-strategies test-hardening MR (!325; plan docs/superpowers/plans/2026-10-07-completion-strategies-test-hardening-mr1.md, fix design docs/superpowers/specs/2026-10-08-completion-hardening-fixes-design.md). Each audit item and its test (paths under packages/; "CFW workflow test" is runtime-cloudflare/src/__tests__/workflows/completion-strategies.workflow.test.ts, "runtime-js completion test" is runtime-js/src/__tests__/completion-strategies.test.ts):
- P1-4: CFW workflow test, "an agent with llmConfigOverride on an unresolved base model IS refused (CF Workflows never runs the override)".
- P1-5: runtime-js completion test ("maxCompletionRetries: 0 on an unresolved model is admitted …"); the CF DO through DOAB
/startine2e/src/__tests__/completion-strategies-do-entry.cf.test.ts; CF Workflows in the CFW workflow test. - P1-6, P2-15:
runtime-cloudflare/src/__tests__/completion-children.wf-noiso.test.ts(sub-agent child and companion first spawn refused at their own start; native forced completion in a child). Wording corrected: the bullet's "the child row isfailedwith the detail" describes a ROOTexecute()refusal (ruling R17). A refused CHILD writes nothing, on CF Workflows as on runtime-js (spec §4.6): the parent's tool row is a failed call carrying the refusal message, and the child has no session, run or message. The one difference is the parent's ref, tracked as FU-COMPLETION-CHILD-REFUSAL-REF-PARITY. - P1-7:
runtime-cloudflare/src/__tests__/steps-legacy-forced-golden.test.tswith__tests__/fixtures/cfw-legacy-forced-request.golden.json(CFW's legacy forced request vs the pre-MR-1 code); the parity suites compare each lane's request shape (e2e/src/helpers/parity-request-shape.ts: system-prompt hash,outputFormat,minOutputTokens,maxOutputTokens) in the forced-completion, continuation, finishWith-failure, protocol-violation and retry-phase parity files (.integ,.cfand the CF Workflows engine.wflanes), and pass under--sequence.shuffle. - P1-8, P1-9: runtime-js completion test (
handle.resume()refusal; companion re-consult refusal) andruntime-js/src/__tests__/run-loop-persistent.test.ts(interrupted-companion resume refusal). - P1-10: the faithful mock (
core/src/testing/faithful-provider.ts,core/src/llm/mock-adapter.ts; testscore/src/__tests__/llm/faithful-mock.test.ts); every scripted native answer migrated to the newnative_outputresponse. - P2-11: runtime-js completion test,
e2e/src/__tests__/completion-strategies-ui.integ.test.ts, CFW workflow test. - P2-12 and the CFW unknown provider code: CFW workflow test (an unclassified and an unknown-code forced error exhaust
llmStepRetry: exactly oneerrorchunk, no code) andruntime-cloudflare/src/__tests__/steps-completion-strategies.test.ts; the rule lifted to corewithoutUnknownCode(core/src/__tests__/errors/error-detail.test.ts) and used by runtime-js, Temporal and CF Workflows; cross-runtime cases ine2e/src/__tests__/forced-completion-parity.integ.test.ts(JS, Temporal) ande2e/src/__tests__/forced-completion-parity-cfw-engine.wf.test.ts(CFW). - P2-16:
core/src/testing/message-operations.ts(includeThinking: falsestrips reasoningparts), on all five stores. - P2-17:
core/src/orchestration/__tests__/step-iterator.test.ts(memory), runtime-js completion test (skills),steps-completion-strategies.test.tsand the CFW workflow test. It holds on a history-bound profile (anthropic-5.5); a legacyforced_toolcall trims the prompt and never loads the skills catalog, which the runtime-js completion test andsteps-completion-strategies.test.tsassert as their legacy case. - P2-18: runtime-js completion test ("a new turn on a branch of a pending checkpoint", both profiles); the branch's loaded phase is RM-72's (below).
- P2-20, P3-29:
completion-strategies-do-entry.cf.test.ts(native forced completion through DOAB/starton the harness DO; the schema refusal; a new turn on an existing session). - P2-21:
llm-vercel/src/__tests__/capabilities.test.ts("every capability-table row resolves to its exact profile"). - P2-22:
steps-completion-strategies.test.ts(tool-set prefix refusal; cache-marker exemption). - P2-24:
agent-server/src/__tests__/http-completion-refusal.test.ts(a real submit never refuses, the next/resumeanswers the typed 400; the stubbed tests relabelled as error-to-HTTP mapping tests) andagent-server/src/__tests__/chat-refusal.test.ts(POST /chat). - P3-25, P3-26:
core/src/__tests__/orchestration/step-processor-native.test.ts. - P3-27: runtime-js completion test ("prompt-prefix check is per model on JSAgentExecutor").
- P3-34: CFW workflow test.
- P3-35 and the quality items: the classify half of
step-processor-native.test.tsis driven by the real plan; the "would reject" headers are true since P1-10; the runtime-js completion test callsexpectReplayedReasoning; the five tests that bypass DOAB keep their cases, relabelled as DO-SQLite persistence tests: the audit's three (completion-strategies-do.cf.test.ts,forced-completion-do.cf.test.ts,e2e/.../forced-completion-parity.cf.test.ts) and, for consistency, the two sibling e2e.cfparity files built the same way. DOAB itself is covered bycompletion-strategies-do-entry.cf.test.ts; the byte-exactness claims are corrected and backed by the golden (P1-7); the weak assertions are exact; the fixed 300 ms wait is an idle probe; and the do-entry andrefusal.wf-noiso.test.tssession ids carry a random suffix.
Runtime bugs the new tests found (the plan's D1 rule: diagnose, fix the runtime, never weaken the test), each fixed on !325 and recorded in its catalogue record and in the fix design note docs/superpowers/specs/2026-10-08-completion-hardening-fixes-design.md ("the note" below):
- RM-72 (fixed):
cloneSessioncopied the source's live status, output, errors and forced phase; the note §1 (the core clone helpers, all four cloning stores, runtime-js's new-turn hardening). - CP-103 (fixed): a refused or lost interrupted-companion resume left the parent's ref
running; the note §2, with A1 (the ref inside the child's run-start commit) in §2.8 (a). Closes FU-COMPLETION-COMPANION-RESUME-REFUSAL-REF's ref half. - W1 (CP-103 / CP-73): an untimed Temporal
waitForResulton a stalerunningref polled until the activity timed out; the note §2.2 (I-W1, with the live-run guard on both branches). - B1 / B1b (DoD §F2): masking defaults in the companion tools' child reads and the ref sync's write; the note §2.8 (b) and (b').
- DI-39 (fixed): a refused chat resume or submit answered
200SSE with no code, and agent-server/startan untyped500; the note §3. - The CFW unknown provider code (D2, P2-12): CF Workflows kept an unknown provider code on a retry-exhausted forced error; the note §4 (core
withoutUnknownCode).
Found and NOT fixed here, each tracked: DI-40 (FU-RETRY-RESTORES-PREVIOUS-TURN-PHASE), CP-105 (FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW), the Temporal waitForResult resume race (FU-TEMPORAL-WAITFORRESULT-RESUME-RACE), DI-41 (a cf-do reconnect to a failed run answered as finished; catalogue record), and the DBOS request gap MA-24 (pinned by the parity suites' dbosKnownGapShapes, owned by FU-COMPLETION-DBOS; MA-27, the other one, is fixed by the DBOS live-agent MR).
Owner: completion strategies (a dedicated MR after !312, before MR 3). Surfaced by: the !312 test audit, which also covered MR 1 (!301). !312 closed the Temporal items; these are the deferred ones for runtime-js, the CF DO, CF Workflows, core, the stores, llm-vercel and agent-server, by the audit's gap number:
- P1-4 (CFW): an agent with
llmConfigOverrideon an unresolved base model IS refused at entry (CF Workflows never runs the override); assert the refusal code inworkflows/completion-strategies.workflow.test.ts. - P1-5 (JS, CF DO, CFW):
maxCompletionRetries: 0on an unresolved model is not refused: it starts, runs its work call and on a trigger failsframework_forced_completion_exhaustedwith no forced call. - P1-6: CFW child and companion refusal at their own start (the parent sees a failed sub-agent call, the child row is
failedwith the detail), and native forced completion in a CFW child. - P1-7 (CFW + parity snapshot): a full-request golden of CFW's legacy forced request, and
buildSnapshotin the parity suites extended with a system-prompt hash,outputFormat,minOutputTokensandmaxOutputTokens. - P1-8: JS
handle.resume()refusal (rejects typed before any write). - P1-9: JS companion re-consult (
continuePersistentChild) and interrupted-companion resume (resumeChildAgent) refusals: the reopen CAS is not taken, the child keeps its status, the parent's tool result reports the refusal. - P1-10 (faithful mock, spec §6.4): the faithful mock returns a native JSON answer only when
outputFormatwas sent, can emit redacted blocks, binds non-cache toolproviderOptionsand detects a droppedpartsarray. Today it rejects only a forced tool choice and bad signatures, so a runtime that dropsoutputFormatstill completes. - P2-11 (JS, CFW): native invalid-output miss end to end: one hidden assistant row, one correction, attempt 2's salt, success, and
convertToUIMessagesomitting the miss. - P2-12: CFW E5 on retry exhaustion: exactly one
errorchunk, and an unclassified forced error that exhaustsllmStepRetrypersists a detail with nocode. - P2-15 (CFW): companion first-spawn refusal (spawn result
failed, the child session's state). - P2-16: store contract:
includeThinking: falsestrips reasoningpartsand keeps text / tool-call refs, on all five stores (core/src/testing/message-operations.ts). - P2-17 (JS, CFW): a skills or memory failure on a forced call leaves the phase
pendingand the retried step completes. - P2-18 (JS): a branch (
cloneSession) of a pending phase, under both profiles, resolves the strategy fresh on its first forced call and completes. - P2-20: DOAB-level native forced completion:
/starton the harness DO with ananthropic-5.5adapter (native input, the synthetic row in DO SQLite, chunks carrying the strategy). - P2-21: exact profiles for every 5.5-generation and new row (
claude-mythos-5-1,claude-mythos-5,claude-mythos-preview,claude-fable-5, a 5.5 Vertex id) inllm-vercelcapability tests; the drift guard checks "non-legacy" only. - P2-22 (CFW): the tool-set prefix refusal and the runtime-level cache-marker exemption of the prefix check.
- P2-24: agent-server
/chatsend for a refused agent (status and bodyuseChatsees), and a real-path submit test or an explicit note that a JS submit never refuses. - P3-25: core: a native text answer whose
partshold two text blocks (the first becomes the call ref, the second is dropped). - P3-26: core: a native
tool_callsanswer with__finish__plus another call is a protocol violation. - P3-27: JS runtime-level same-model rule: an
llmConfigOverridemodel switch after signed thinking is not refused. - P3-29: CF DO
/startrefusal forframework_completion_schema_unsupportedand for a new turn on an existing session. - P3-34: CFW: an instance recorded before the
completion-supportstep existed checks fresh on replay. - P3-35: the quality fixes below.
- CFW unknown provider code (from !312 SF1): when a retryable forced error with a code that is not an
ErrorCodeexhaustsllmStepRetry, CF Workflows keeps that code on the persistederrorDetail(failPendingPhasedrops it from the phase only; the K5 persists the input detail as is), while runtime-js and, since !312, Temporal strip it (coreclassifyForcedCompletionResult; Temporal drops it inrunLLMStepfrom the retryable forced-call failure it rethrows, so only the forced call's own LLM error loses it and a non-provider failure keeps its code, as on runtime-js). Align CF Workflows to core's rule, with a test.
Quality items (audit §2) in non-Temporal files:
- Mock fidelity: test headers that say the faithful mock "would reject" a mis-built forced call (
step-iterator-native.test.ts,workflows/completion-strategies.workflow.test.ts) hold only for narrowed tools and trimmed prompts until P1-10 lands;runtime-js/.../completion-strategies.test.tsnever callsexpectReplayedReasoning. - CF DO tests that bypass DOAB:
completion-strategies-do.cf.test.ts,forced-completion-do.cf.test.tsande2e/.../forced-completion-parity.cf.test.tsrunJSAgentExecutorover aDOStateStore; relabel them (DO-SQLite persistence, not the DOAB routes) or move them behind DOAB (P2-20). - Over-claimed byte-exactness:
steps-completion-strategies.test.ts's header claims the CFW legacy request is covered byte-exact (four directive fields are asserted); the parity suites' "byte-identical snapshots" compare a narrow projection (P1-7). Their order dependence on earlierits breaks under--sequence.shuffle. - Vacuous case:
step-processor-native.test.ts's classify half passes a hard-codedoutputSchemaValid: falseand an unrelated step result; drive it from the real plan. - agent-server stubbed submit tests:
http-completion-refusal.test.tsreplacesAgentServerwith a stub; label them as error-to-HTTP mapping tests (P2-24). - Weak assertions:
completion-strategies-do.cf.test.ts(parts.length > 0) andruntime-js/.../completion-strategies.test.ts(forced.length >= 1). - Timing and ids:
completion-strategies-do-entry.cf.test.tswaits a fixed 300 ms "for a wrongly woken run" (wait on a positive signal instead); it andrefusal.wf-noiso.test.tsbuild ids fromDate.now()alone (add a random suffix).
FU-TEMPORAL-ROLLING-DEPLOY-CROSSOVER-TEST: no live old-bundle → new-bundle crossover test — open
Owner: runtime-temporal. Surfaced by: the !312 test audit (P3-33).
A live crossover test is not built: a workflow begun on pre-MR-2 code that continues on an upgraded worker should record the helix-completion-strategy-v1 patch marker mid-history and take the new path. It needs the old code live at test time, either a build of fdc56520a5 or a vendored workflow bundle (about 8 MB) plus the old activities and their own @temporalio/activity instance, which is not viable in CI today. The marker semantics are pinned by the pre-MR-2 replay fixtures (old histories replay deterministically on the new code) and by the fresh-history marker assertions. This test would add an end-to-end proof. To close: build the two-bundle harness and assert the marker position and the new path after the crossover.
The overlap also mixes the retry entry, whose change is an activity argument, not a patch (ruling P22): a retry() workflow passes retryEntry: true to initializeAgentState. Replay is unaffected (commands match on activity type and id, never arguments). But a retry whose startRunCommit ran on an old worker restores the target's raw phase (no correction for a forced step's own transition commit, retryRestoredForcedPhase), and a new worker's initializeAgentState then keeps it, so a retry after an exhausted forced phase re-forces the spent attempt once (a budget-0 terminal_text / max_tokens retry restores no phase instead of failed). When the hook half runs on an old worker, or the workflow task runs old code (no argument), the phase is reset as before the upgrade. The crossover harness should cover this mix too.
FU-COMPLETION-DBOS: DBOS does not resolve completion strategies — open
Owner: completion strategies MR 3 (DBOS).
The DBOS gaps are the ones Temporal had before MR 2 (LEGACY_RESOLUTION in workflows/shared.ts, the R1 shim in workflows/forced-error-r1.ts, the forced exemption in steps/call-llm.ts, no refusal, no prefix check), plus the spec §6 / §5.2 prerequisites: two of them are done by the DBOS live-agent MR (batch 28: every LLMConfig field reaches the model, MA-27, and the systemPrompt function is rendered per call with the live state, MA-23, so the prefix check can apply once this item lands); the system prompt lacks the completion instruction and workspace fragment parity; the directive must move from the workflow body (no adapter) into callLLMStep; early refusal at registerAgent and execute(). Recovery tests must replay old-shape recorded step results through the new bodies.
DI-71 note: DBOS classifies forced answers with core's classifyForcedCompletionResult / recordForcedCompletionMiss on planStepProcessing's plan (workflows/shared.ts), so its schema-invalid misses name the issue paths, and an exhaustion's errorDetail.cause carries the last miss. That was not run on DBOS. The cause can still be lost on the way to the handle and the session (DI-24). Prove both when this item lands.
DI-94 note: the R1 shim is why a forced call's typed provider error loses its code on DBOS (the run fails with the exhausted error instead of the provider's detail); completion.forced-provider-error-code on dbos-postgres flips when this item lands.
FU-HELIX-ERROR-APP-CODE-OPTION: an app can throw its own error code only by casting HelixErrorOptions.code — done (!864: the option widened)
Closed by: W1's !864 (batch 23) took the second option. HelixErrorOptions.code, HelixError.code and ai-sdk's HelixStreamError.code widen to ErrorCode | (string & {}), as ErrorDetail.code already was (breaking for types: an exhaustive switch (error.code) needs a default branch, or narrow first with isKnownErrorCode). Two behaviour changes ride with it. D1: helixErrorFromDetail keeps a detail's code that is not an ErrorCode, so an extension or app code reaches the typed HTTP body as itself; only an absent or empty code, or a bare status number read off a thrown non-Error, is still framework_internal_error. D2: a DO client's DOPeerError and fetchSnapshotWithRetry's SnapshotUnavailableError keep a peer's own code instead of transport_error (a body with no code still gets it). The entry below is the plan as filed.
Owner: core (the user's ruling first). Surfaced by: DI-93's design (WE-2, 2026-11-13): an app's provider wrapper throws new HelixError({ code: <its own code> as ErrorCode, category: 'provider', retryable: false }) on a usage-limit refusal, because HelixErrorOptions.code and HelixError.code are the closed framework ErrorCode union, while ErrorDetail.code is already ErrorCode | (string & {}) for extension codes. Two options:
- Document the attached-detail route: throw an
Errorcarrying anerrorDetail(corefromErrorDetail, theErrorWithDetailshape), whichresolveErrorDetailalready honours anywhere on the cause chain, and say so in the error-handling guide. No type change. - Widen the option: accept
ErrorCode | (string & {})inHelixErrorOptions.code.HelixError.codewidens with it, which breaks consumers' exhaustiveswitchonerror.codeat compile time (isKnownErrorCodenarrows it back).
FU-COMPLETION-DBOS-RETRY-PHASE-PARITY: DBOS retry() restores a different forced phase from runtime-js and Temporal — open
Owner: completion strategies MR 3 (DBOS). Surfaced by: the !312 delta audit (after rulings P22 / P23).
runtime-js and Temporal now restore the same forced phase on retry() (ruling P23; proven by the e2e forced-completion-retry-phase-parity.integ.test.ts, which runs only those two runtimes). DBOS does not, because it never puts a phase transition in a checkpoint:
- the phase is written by
persistForcedCompletion, a plain fencedsaveState(no checkpoint;runtime-dbos/src/steps/load-state.ts:206); applyForcedTransitionappends the correction or entry instruction BEFORE it persists the phase (workflows/shared.ts:1776then:1783), so thatappendcheckpoint holds the pre-transition phase;retry()starts the workflow withisResume: true(lifecycle/retry.ts:143), so the restored phase (steps/run-commit.ts:742) is kept (the new-turn reset atworkflows/shared.ts:2559is skipped).
Outcomes (read from the code, not yet tested): a budget-0 terminal_text / max_tokens retry restores no phase (runtime-js and Temporal restore failed); a forced-step target and a correction-append target restore the pre-transition phase, so the spent attempt is re-forced; a budget > 0 phase-entry append target carries no phase, so the pending phase is dropped (runtime-js and Temporal continue it). CF Workflows looks consistent by structure (not verified here) but has no test. To close: make DBOS's retry restore what runtime-js restores per trigger and per target, and add DBOS and CF Workflows lanes to forced-completion-retry-phase-parity.integ.test.ts. Related: FU-COMPLETION-RETRY-AFTER-FINISHWITH-EXHAUSTION (the semantics of a retry after an exhausted phase, on every runtime).
FU-COMPLETION-DBOS-SKILLS-RECOVERY-STRAND: DBOS keeps a throwing skills provider non-fatal until it can fail closed in a step — open
Owner: completion strategies MR 3. Surfaced by: MR 1 Task 13 review; ruling R24 (whole-branch review A1).
Core resolveSkillsCatalog fails closed (framework_skills_unavailable, retryable), and so do runtime-js, the CF DO, CF Workflows and Temporal. DBOS resolves the catalog in the workflow body, not in a step, so failing closed there would reject the body outside any step-retry or terminal path and could strand a recovering run active; and DBOS has no history-bound thinking path in MR 1, so failing closed only adds a failure mode. MR 1 therefore keeps the pre-0.50 non-fatal behaviour on DBOS: both workflow bodies call resolveDbosSkillsCatalog (runtime-dbos/src/workflows/standard-workflow.ts, used by standardAgentWorkflow and persistentAgentWorkflow), which logs a warn (errorId: 'dbos.skills.catalog_unavailable') and runs the turn with an empty catalog (skills-catalog-nonfatal.test.ts). To close (MR 3): resolve the catalog inside a checkpointed @DBOS.step (also FU-SKILL-DBOS-CATALOG-IN-WORKFLOW-BODY), fail closed there with framework_skills_unavailable, and route the failure through finalizeFailure; then delete resolveDbosSkillsCatalog.
FU-COMPLETION-LIVE-ANTHROPIC-STRUCTURED-PROFILE: no live run covers the anthropic-structured forced request — open (coverage)
Owner: llm-vercel (routed by the reliability orchestrator). Surfaced by: the GPT-6 Luna / Claude 5.5 model bump (WM, 2026-10-19).
The project's LiteLLM proxy keys serve only claude-sonnet-5-5 and claude-haiku-5-5 on the Anthropic side (user decision: Claude 4.x dropped), so completion-strategies-live-anthropic.integ.test.ts case 1 no longer runs on claude-sonnet-5 (anthropic-structured). It now runs claude-haiku-5-5 with a capabilities override that declares forced tool choice with history-bound thinking, so the forced_tool wire path (a named __finish__ choice, full tools and prompt) is still live. What no live run covers any more is the anthropic-structured request: forced_tool with thinking NOT bound to history, i.e. one tool, the trimmed prompt and reasoning disabled. Its mapping and request shape stay covered by packages/llm-vercel/src/__tests__/capabilities.test.ts, capabilities-drift.test.ts and the request-boundary test forced-completion-anthropic-boundary.integ.test.ts (no network). legacy (every OpenAI model) stays live through gpt-6-luna. To close, if live coverage is wanted: allowlist one anthropic-structured model (e.g. claude-sonnet-5) on the proxy keys and restore a case on it.
FU-E2E-ANTHROPIC-SDK-BUMP: the repo's tests pin @ai-sdk/anthropic 3.0.38, which cannot ask for readable thinking — open: the two wire shims only (deps, test pins)
Owner: llm-vercel (routed by the reliability orchestrator). Surfaced by: the Claude 5.5 bump (WM, 2026-10-19).
Claude 5.5 models default thinking display to omitted (empty thinking text). @ai-sdk/anthropic sends providerOptions.anthropic.thinking.display from 3.0.61 (released 2026-03-16); 3.0.38, which llm-vercel pins as its dev dependency and the e2e package resolves, drops it. So thinking-llm.integ.test.ts adds display: 'summarized' on the wire through a fetch wrapper (and completion-strategies-live-anthropic.integ.test.ts defaults it the same way). Users on 3.0.61+ need no workaround: the docs (docs/llm/vercel.md, Anthropic Extended Thinking; the llm-vercel README) say to set display: 'summarized'. Moving the repo to a newer release changes the Anthropic wire the request-boundary regressions pin byte for byte, and the peer range (>=3.0.38) lets users run those releases today, so this is about real users' requests, not just our pin (sent to the conformance owner as a candidate finding). Measured with forced-completion-anthropic-boundary.integ.test.ts: on 3.0.61 the forced (forced_tool) request is unchanged, but the native_output request carries fine-grained-tool-streaming-2025-05-14 instead of the structured-outputs-2025-11-13 beta (1 of 4 fails); on 3.0.100 and 3.0.127 the forced request also changes: max_tokens defaults to 128000 instead of 4096, an explicit thinking: { type: 'disabled' } is sent, and tools carry eager_input_streaming and strict: true (2 of 4 fail). Rating (conformance owner, 2026-10-19): not a catalogue record yet: a request-shape change across releases, no defect witnessed (the conformance lanes run the scripted model, never a provider SDK's wire, so the witness is this request-boundary integ plus a live probe). The native_output arm is the one that matters: the anthropic-5.5 profile assumes the provider enforces the output format, so if 3.0.61+ (no structured-outputs-2025-11-13 beta) returns unenforced or free text, forced completion degrades on a release the peer range claims to support (S2 if it fails loudly, S1 if it yields wrong output). The forced_tool arm is lower risk unless a probe shows a 400 or changed output. Live probe (2026-10-19): an outputSchema agent driven into forced completion (a tool the prompt requires first, maxSteps: 1) on claude-sonnet-5-5 and claude-haiku-5-5 with 3.0.38, 3.0.61 and 3.0.127. All 6 cells sent the native_output request (output_format set) and completed with a schema-valid object; no invalid object was returned as a success (n=1 per cell, a weak sample). Only the beta header differs: 3.0.38 adds structured-outputs-2025-11-13 to the forced call, 3.0.61 sends no structured-outputs beta, 3.0.127 sends it on every call. The forced_tool arm was not probed: both 5.5 models resolve to native_output. Still open: a larger sample, the forced_tool arm on the legacy / anthropic-structured profiles, and the peer-range cap or two-version matrix below. Until then, cap the peer range at the tested window, or run this boundary integ against both 3.0.38 and the latest release (a pinned alias devDep, as @helix-agents/ai-sdk does for ai 6.0.x) so request-shape drift fails CI. To close: bump llm-vercel's dev pin and e2e's range together, re-baseline those pins (and decide whether the 128000 default and strict are acceptable for the legacy / anthropic-structured forced request, or need an explicit maxOutputTokens / tool option), then drop the two wire shims.
Bumped (WA, 2026-11-04): llm-vercel's dev pin, e2e's pin and the peer floor all moved to 3.0.128, the first release whose model table has an entry of its own for Sonnet 5.5, Opus 5.5, Haiku 5.5 and Fable 5.1 (own entries landed at 3.0.121, 3.0.125 and 3.0.128; Mythos 5.1 still rides the provider's default for an unknown id). The pin had a third consequence beside readable thinking and request-shape drift: 3.0.38 did not know the 5.5 generation, so llmConfig.strictOutput was the documented no-op there (docs/guide/finishing-agents.md) and every boundary test ran a request no current app sends (read, not run: 3.0.38 dist :1092; 3.0.135 :1696, :1709, :3970). The re-baselined pins name what the provider changed: the max_tokens default of 128000, a literal thinking: { type: 'disabled' }, eager_input_streaming and strict on the tool, and the structured-outputs-2025-11-13 beta on the forced request. All accepted for the legacy / anthropic-structured forced request. Still open: drop the two wire shims once the live job has run on this pin.
FU-STRICT-TOOL-RUN-START-CHECK: a user's strict tool is refused at the step, not at run start — open
Owner: llm-vercel + core. Surfaced by: DI-82 (WA, 2026-11-04).
The adapter refuses a strict tool whose schema the Anthropic subset cannot express, typed and before the model call, but only when the step that offers it runs. An outputSchema agent's __finish__ gets the same answer earlier, at run start, through checkOutputFormat, but only when its forced calls use native_output (checkCompletionSupport checks the format for that strategy alone): on the forced_tool strategy a strict __finish__ is, like a defineTool({ strict: true }) tool, refused only at its first step, after its run committed. The subset and its refusal apply only on a model with structured output (the anthropic-structured and anthropic-5.5 profiles); elsewhere the provider drops strict and the schema goes as written. Closing that needs a core hook that hands the agent's strict tools to the adapter before the run-start commit, on every runtime. The orchestrator accepted the step-time refusal for DI-82 (2026-11-04).
FU-ANTHROPIC-EAGER-INPUT-STRICT: does eager_input_streaming undercut strictOutput? — open (not yet measured)
Owner: llm-vercel (routed to the conformance owner for a call). Surfaced by: the 3.0.128 bump (WA, 2026-11-04).
@ai-sdk/anthropic 3.0.128 sends eager_input_streaming: true on tools by default, beside strict: true on a strictOutput agent's __finish__. Anthropic's tool-use notes say that with the field on the API no longer coerces or validates a tool's input. If that holds for a strict tool, strictOutput gives no guarantee on any recent provider. No request has shown an invalid __finish__ input under both fields, and the notes do not say which field wins. completion-strategies-live-anthropic case 10 logs both fields, which shows only that they are sent.
The measurement is doable on test:live-llm with the real key, when someone wants it: a larger sample that tries to provoke a schema miss with the field on and with it off. It becomes a finding if an invalid __finish__ input appears on a strictOutput agent with both fields on and not with the field off, because docs/guide/finishing-agents.md says strictOutput forces schema-conforming output; its severity is decided then. The likely fix: the adapter sets providerOptions.anthropic.eagerInputStreaming: false on a strict tool, the per-tool switch 3.0.128 exposes.
FU-ANTHROPIC-VCR-HARNESS: no record/replay harness for Anthropic calls — open (test infra)
Owner: reliability orchestrator (route). Surfaced by: the Claude 5.5 bump (WM, 2026-10-19).
The example apps' record/replay proxy (examples/openai-vcr) speaks the OpenAI API only, so an Anthropic path (for example an outputSchema agent on claude-haiku-5-5 finishing through native_output) is proven only by the gated live e2e files (strict-structured-output-live-anthropic, completion-strategies-live-anthropic), run by hand or by the manual test:live-llm CI job. To close: extend the recorder to /v1/messages (Anthropic SSE framing, x-api-key injection, the same header allowlist and leak refusal) and add a replayed outputSchema run on a Claude 5.5 model to a CI-replayed suite.
FU-AI-SDK-MAIN-ENTRY-AI-COST: the @helix-agents/ai-sdk main entry still imports ai statically — open (LOW, SDK startup cost)
Design spec §4.7 (docs/superpowers/specs/2026-10-21-sdk-startup-cost-design.md): the @helix-agents/ai-sdk main entry imports ai at load (generateId is synchronous, so loading it lazily would be an API change). The Workers-facing @helix-agents/ai-sdk/cloudflare entry does not. Measured with the startup-guard probe on @helix-agents/ai-sdk (import; median of 5 fresh probes, source maps off, Node v24.19.0, 2026-10-21): helix 0, thirdParty 169, importMs 57.24, heapMB 9.4. So the cost is third-party zod instances from ai and @ai-sdk/gateway, none of Helix's own. To close: make the one eager use lazy behind an async API, or move generateId to a small local helper.
FU-AI-SDK-OPENAI-MAJOR-SPLIT: the repo depends on both @ai-sdk/openai 2.x and 3.x — open (deps)
Owner: examples / runtime-dbos (routed by the reliability orchestrator). Surfaced by: the GPT-6 Luna bump (WM, 2026-10-19).
Twelve package.json files take @ai-sdk/openai ^2.0.135 (the ai 5 line) and five take ^3.0.130 (the ai 6 line), while every package uses ai 6. The bump raised both floors to the first patches that treat gpt-6-* as a reasoning model (developer role, no sampling parameters); it did not unify the majors. To close: move every package to @ai-sdk/openai 3.x, run each affected example and the runtime-dbos real-LLM round-trip, and drop the 2.x line from the lockfile.
FU-SCHEMA-ISSUE-CUSTOM-MESSAGE: a failed .refine reaches the model as $: custom — open (needs a ruling)
Owner: core. Surfaced by: the review of DI-81 (WA, 2026-11-06); not a finding (conformance owner).
describeSchemaIssues (DI-71) never passes on Zod's message, because a custom refinement can build it from the value, so a .refine / .superRefine failure reaches the model as just $: custom: a rejected __finish__ (DI-81) or a forced-completion miss tells the model where, not what. Proposed rule (the conformance owner's, for a ruling before anyone builds it): forward a custom issue's message only when the schema author wrote it as a static string, through the same sanitizer and length cap as tool error text, and keep $: custom when the message is built by a function (it can carry the rejected value). Unit tests for both cases. Not in Release 2: today's text fails safe (orchestrator, 2026-11-06).
FU-OPENAI-TOOL-SCHEMA-LOOKAROUND: GPT-6 rejects a tool schema whose pattern uses lookaround — open (llm-vercel)
Owner: llm-vercel (routed by the reliability orchestrator). Surfaced by: the GPT-6 Luna re-recording (WM, 2026-10-19). Rated by the conformance owner: not a catalogue record. The failure is up front and typed, and names the field; see DI-71 for the late, unnamed Anthropic counterpart.
GPT-6 Luna answers 400 invalid_json_schema (regex lookaround is not supported. Found at $.properties.<path>.pattern) to any tool whose JSON Schema has a lookaround pattern. Zod emits one for .email() / z.email() (its default regex), z.hostname(), z.iso.duration() and any custom .regex() that uses lookaround (zod 4.3.6; the other string formats emit none). The run fails on its first call with provider_invalid_request; the provider text, path included, survives in the result's, the session's and the stream's errorDetail. An outputSchema with such a pattern fails the same way on the first call (the legacy profile offers __finish__ from step 1). The approval-gates example and the two docs snippets use z.string().email({ pattern: z.regexes.html5Email }), and docs/guide/tools.md explains why. To close, pick a design: llm-vercel refuses such a tool schema at run entry, typed and naming the path, for every model whose provider rejects it; or drops the pattern as DI-71 does for a native-output schema, if the tool's input Zod parse is proven on every path (the framework validates tool input before execute). Either way it uses DI-71's detector (lookaroundPatternPaths in llm-vercel's capabilities/anthropic-schema.ts). Tool schemas never pass through llm-vercel's JSON conversion: aiTool({ inputSchema }) hands the Zod schema to the AI SDK, and core has no run-entry hook for tools, so this needs its own design.
FU-COMPLETION-BEDROCK-VERTEX-NATIVE-OUTPUT: native output through Bedrock / Vertex providers is unverified — open (risk)
Owner: llm-vercel. Surfaced by: MR 1 final review.
A 5.5-generation model reached through @ai-sdk/amazon-bedrock or @ai-sdk/google-vertex (Anthropic) resolves to anthropic-5.5 → native_output; so do AI Gateway and OpenRouter ids. For every Anthropic-family model the adapter sets providerOptions.anthropic.structuredOutputMode: 'outputFormat'; the extra provider-specific key is added only for @ai-sdk/anthropic-backed models with a custom provider name (its providerOptionsName getter, which Vertex's Anthropic models share). Whether @ai-sdk/amazon-bedrock (and the gateway / OpenRouter providers) read anthropic.structuredOutputMode at all, and whether they send Output.object as native structured output rather than a forced json tool (a 400 on 5.5), is not verified; none of those packages is installed in the repo. Vertex's Anthropic models run @ai-sdk/anthropic's messages model, so they most likely honour the key, but that is unverified too. Documented as a limitation in docs/llm/vercel.md. To close: a wire-level test with each provider and a captured fetch, or a live probe; if a provider forces a tool, map its own structured-output option or resolve such models to unresolved with a clear reason.
FU-LIVE-MAX-EFFORT-REFUSAL: a work step at effort: max ended stop=refusal in 3 of 5 live sessions — open (question)
Surfaced by: DI-80's live case 7, job 17782651314 (WA, 2026-11-05). claude-sonnet-5-5, arm thinking-max, one prompt (STRICT_FORCED_MESSAGE). Not checked against earlier jobs. All 5 no-thinking and all 5 high-effort sessions reached a forced call.
thinking-max#1:phase=work status=200 stop=refusal blocks=[thinking] in=1921 out=4664thinking-max#3:phase=work status=200 stop=refusal blocks=[thinking] in=1921 out=5226thinking-max#5:phase=work status=200 stop=refusal blocks=[thinking] in=1921 out=9617
The question: how did those runs end? Case 7 now logs the final status of a session that never forced, so its next run answers it. A refusal that fails the run typed is correct and needs nothing more; a run that completes or hangs on one is a finding.
FU-NATIVE-OUTPUT-FORCED-TOOL-CALL: a forced native_output call answered with a tool call — open (design question)
Owner: WA (core forced completion + llm-vercel). Surfaced by: DI-80's live case 8 (WA, 2026-11-04). The conformance owner ruled it an FU, not a finding: a forced call's other tool call is a recoverable miss and the budget ends typed, so the behaviour matches the contract and a scripted cell would pass.
The numbers. Case 8 of completion-strategies-live-anthropic (claude-sonnet-5-5, effort: 'max', a 6000 cap) over three test:live-llm jobs, by how the work step ended: tool_use passed (1 job), max_tokens failed (framework_completion_protocol_violation, 2 jobs). In the third job every thinking case's work step truncated.
The chain, from the traces, not reproduced at another cap: thinking uses the whole cap, so the work step makes no tool call; the run enters forced completion; native_output keeps the tools visible; the model asks for the tool it still needs, on every attempt; the budget ends the run typed.
Fix directions to weigh:
- The repair text says no tool can run in this phase. It is strategy-neutral today, and it names a directive
native_outputdoes not send. - A
max_tokenswork step with no tool call may not belong in the forced phase at all: the work was not finished, so an ordinary next step could fit better.
tool_choice: 'none' is not a direction: @ai-sdk/anthropic then sends no tools, which changes the tool list of a history-bound call.
What makes it a finding: this failure at a realistic cap, or a rate on an ordinary agent. A 6000 cap with effort: 'max' is the test's own condition; the guides' examples use 4096.
Measured at the guides' cap (case 9d, 2026-11-04): five runs at 4096 and effort: 'high', all five work steps ended tool_use and completed. That kept this an FU with no design change for the release. It is five runs of one prompt: it does not rule out a rate of about one in five, and it says nothing about other prompts or about effort: 'max'.
FU-COMPLETION-LIVE-NATIVE-CONTINUATION: live proof that a synthetic completion call replays under binding enforcement — open (the proof is unverified)
Cases 9e to 9h (2026-11-05): the check runs for this key and model, and our /v1/messages route hides it.
- Case 9f (job 17759399130), on the proxy's
/anthropicpass-through: an honest replay was accepted, an edited first message was refused 400 undererror, and the same edit underdrop_blockwas a 200 reportingthinking_dropped. - On
/v1/messages, LiteLLM v1.104.0 strips thinking and retries a binding 400, so a 200 undererrorsays nothing. That explains the 200s of cases 9, 9b and 9c, and it means cases 3, 5 and 8 say nothing about the binding. - The
drop_blockreport survives the proxy. Case 9e (the same edit,/v1/messages) reportedthinking_dropped,first_message_rewritten. Case 9g (onlystrictdiffers on the forced call) reportedthinking_dropped,tool_schema_changed,sections=tools, sostrictis compared. - Case 9h (job 17763256471), an agent setting only
effort: 'medium'on real tools, three sessions: nothinkingfield went out, and each response held one thinking block with empty text and a signature (1644, 1380, 2020 characters). It was replayed as received, with empty text, and the forced call returned 200. So empty-text replay is measured on normal-shape traffic. Case 9h asks for no binding, so its 200s say nothing about it. Whether the proxy removes those blocks on our route is a separate measurement against a fake upstream, not this job.
Case 9c (2026-11-04): a corrupted signature on a completed work step's block was accepted (200, input_transformations: []), as were the edits of cases 9 and 9b. An invalid block_binding value is refused, so the field arrives.
Reopened (WA, 2026-11-04): "under enforcement" below rests on a 200, and the live job's controls did not go red: on the CI route block_binding and the beta header reach the API (case 9a; input_transformations is on every response), yet an edited tool (case 9) and an edited first message (case 9b) both returned 200. Anthropic's docs call that the check not running for the request. To close: case 5's session under drop_block, or on the pass-through route, where a control goes red (case 9f).
Owner: e2e. Surfaced by: MR 1 final review.
Case 5 of packages/e2e/src/__tests__/completion-strategies-live-anthropic.integ.test.ts continues case 3's session (claude-sonnet-5-5, thinking, prefix_mismatch_behavior: 'error') with a new user turn after a forced completion the model answered with JSON text, and asserts the continuation is accepted (200) with the synthetic tool_use + tool_result in its history and the byte-prefix property intact across the whole session. Its first live run (effort max, $0.24) never reached that path: the model answered every forced call with a real __finish__ call. Ruling R22 moved the case to effort high, with up to three attempts, and a SKIP when none reaches the path. The R22 run (2026-10-05, $0.0263) reached it on attempt 1: the native JSON-text answer was replayed as the synthetic call in turn 2, both turn-2 requests returned 200 under enforcement, and the byte-prefix chain held across all six requests.
FU-COMPLETION-OUTPUT-CONFIG-FORMAT: send output_config.format once @ai-sdk/anthropic supports it — done
Done (WA, 2026-11-04). The peer floor is 3.0.128, which sends the GA output_config.format. The adapter still has no runtime provider-version check: the peer range is the floor.
Owner: llm-vercel.
@ai-sdk/anthropic@3.0.38 (then the peer floor) sent native structured output as the beta output_format field, which Sonnet 5.5 accepts (gate G1, 2026-10-04). When a provider release sends the GA output_config.format, raise the peer floor, drop any beta header dependence, re-run the wire-level boundary test (forced-completion-anthropic-boundary.integ.test.ts) and the key-gated live suite. The adapter has no runtime provider-version check (spec §3.3 envisaged one); add it then if older versions must be refused.
FU-COMPLETION-RM35-PARTS: a __finish__ tool-call finish drops the response's parts (RM-35) — done
Done (W1, RM-35). llm-vercel returns a __finish__ step as tool_calls, and core's tool_calls branch keeps the assistant message with the step's ordered parts. The runtime stamps helixPrefix on it like any other assistant message (step-iterator.ts / CFW steps.ts, a history-bound model only), and the native-output normalisation shares the same branch. Proven by e2e/src/__tests__/finish-co-emission-real-adapter.integ.test.ts (turn 1's persisted assistant keeps its signed reasoning part, runtime-js and CF Workflows).
Owner: the RM-35 owner. Related: RM-35 (catalogue).
llm-vercel maps any step whose tool calls include __finish__ to a structured_output result, and core persists no assistant row for it, so that response's ordered parts (and signed thinking) are not kept. Under native_output the model may also answer by calling __finish__ itself (gate G4.5), which takes this path. The next turn's history lacks that response entirely rather than replaying it, so preserved-thinking continuity is lost for it (not a 400). When RM-35's fix persists the assistant row, carry parts (and helixPrefix) on it and share the persistence code with the native-output normalisation in planStepProcessing (spec §15).
The MR !301 Greptile review re-raised this as its G2 (a signed __finish__ response is not persisted). Parked by the R26 brief: it is RM-35 (open catalogue finding, another owner), unchanged by MR !301.
FU-COMPLETION-DO-CLIENT-TYPED-CODES: DOWorkflowExecutor drops a refusal's typed code — done (its open half removed with the code, WE MR B)
Closed by WE MR B: DOWorkflowExecutor is deleted (the user's ruling, 2026-10-11: "delete it if we don't need it. don't worry about backwards compatibility"). It never reached a Durable Object (relative URLs, no x-partykit-room, a session:-prefixed id; DI-49), and nothing used what only it offered. The half below that stayed open was entirely its code. DOFrontendExecutor's half was done in !316; its remaining code-less non-2xx answers (413 payload_too_large, 411 length_required) are DI-47, fixed in WE MR B. The history below is kept as written.
Priority raised (WA MR1, verified from source): DOWorkflowExecutor is a public export (runtime-cloudflare/src/index.ts:210), and its submitToolResult throws a plain Error (submitToolResult failed: HTTP <status> <body>) on any non-2xx other than a 404 or a recognised 400 (do/do-executor.ts:722-724; an unknown 400 shape at :716-718). Behind handleChatStream, the client-tool submit catch classifies it with core (ai-sdk/src/handler/handle-chat-stream.ts:2272, classifyEntryFailure): a plain Error has no known code, so it stays a code-less per-intent data-resume-rejected (:848), and when every intent failed the answer is a 200 SSE holding only those rejections (rejectedOnlyResponse, :857 / :864, :3142). So a DO's typed submit answer (409 state_run_superseded, the CP-53 bound's 500 transport_error, a continuation's typed failure, a relayed refusal) reaches a useHelixChat client as a 200 with no code: the W1 silent-loss shape. DOFrontendExecutor (below) is not affected.
Done in WA MR1: DOWorkflowExecutor's execute, resume, retry and the handle's resume read the native body with core PeerErrorBodySchema (do/peer-answer.ts peerAnswerError, the same builder DOFrontendExecutor uses), so they throw typed errors. Still open: submitToolResult (do/do-executor.ts) still turns a non-400 failure into a plain Error (submitToolResult failed: HTTP <status> …), so the DO submit route's new typed answers (409 state_run_superseded, 503 framework_agent_resolution_failed, a relayed refusal) lose their code there. Its resume() client-side /status pre-check also still throws plain Error('Cannot resume: Agent already completed' / 'Agent has failed') (do-executor.ts:520-523; CP-60 and DI-31 cite those strings).
Owner (split, orchestrator routing; restated by WA MR1): cf-we-do-streams, WE-2's MR B owns every DO error answer, this submitToolResult one included (the shared DO error mapping, 2026-10-08), and closes this FU; MR2b Task 8 owns only the 200 path. W1 MR2b Task 8 did that half: DOWorkflowExecutor.submitToolResult parses a 200 body with core SubmitToolResultResponseSchema (its continuation is required; a body without one is a ZodError; do-workflow-executor-typed-errors.test.ts). Every non-2xx path is still MR B's. Pre-existing for every non-409 entry failure; surfaced by MR 1 Task 15.
Done (the DOFrontendExecutor half): CF-D2 MR 1 (!316, B1). do/do-clients.ts's typedPeerError (origin/main 2e626e7f6a :161, called at :1088 /start, :1246 /resume, :1366 and :1401 /submit-tool-result) parses a DO entry route's body with PeerErrorBodySchema and throws a typed DOPeerError carrying the code (retryable only on 503), so execute, resume and submitToolResult keep framework_completion_strategy_unavailable. Pinned by runtime-cloudflare/src/__tests__/do-frontend-executor-typed-errors.test.ts:88 (origin/main 2e626e7f6a).
Still open: DOWorkflowExecutor (do/do-executor.ts: execute Failed to start agent, resume Failed to resume agent, retry and submitToolResult) still turns a non-409 DO failure into a plain Error, so a caller cannot read the code without parsing the message. To close: WE MR B's shared mapping (a DO HTTP error / SSE fail frame ErrorDetail → typed error), adopted in all of DOWorkflowExecutor, with a test per call. DOStubTransport is the separate FU-DO-STUB-TRANSPORT-TYPED-CODE-LOST (W1 MR2b).
FU-COMPLETION-CHAT-SEND-TYPED-REFUSAL: a chat send's completion-support refusal carries no code — done
Closed by: CF-D2 MR 1 (!316, B1) for the send and the DO client, and the completion-strategies test-hardening MR (!325), Task 15 for the chat resume and client-tool submit: every handleChatStream host answers a refused chat send, resume or submit with the typed 400 { error, code } before any stream opens. Tests: ai-sdk handler/handle-chat-stream-pre-commit-failure.test.ts (!316), handler/handle-chat-stream-resume-typed-errors.test.ts, react/send-refusal-400.test.tsx; agent-server chat-refusal.test.ts; e2e chat-send-refusal-do.cf.test.ts and the js-chat-host case in completion-strategies-ui.integ.test.ts; conformance DI-39 (fixed).
Owner: ai-sdk. Surfaced by the MR 1 final re-review.
agent-server's /start, /resume and submit routes answer a completion-support refusal with a typed 400 { error, code }, as the DO does. A chat send (POST /chat, ai-sdk handleChatStream) instead turns the executor's refusal into a 200 SSE data-resume-rejected that carries only the message (handle-chat-stream.ts), so a useHelixChat caller cannot tell framework_completion_strategy_unavailable from any other rejection. The docs list only the typed routes, so they are accurate. To close: carry the refusal's code on the rejection (as parseHelixChatError reads for a failed stream), with host- and client-tier tests.
FU-COMPLETION-DO-SUBMIT-REFUSAL-CHILD-PATHS: the DO submit refusal's sub-agent paths are untested; a refused root can still stall — open (partly done)
Owner: runtime-cloudflare. Surfaced by the R24 re-review.
The DO /submit-tool-result completion-support gate runs in the root's local resolver and in a sub-agent DO's forwarded-submit shortcut, and a root relays an owning sub-agent DO's typed 400. Only the root's local path is tested (completion-strategies-do-entry.cf.test.ts); the sub-agent gate and the relay are not. In the shortcut the gate runs before the unknown-call check, so an unknown call on a refusable child answers 400 instead of 404. Separately, when the ROOT agent becomes refusable but the pending call belongs to a child DO, the forwarded submit is accepted, the child finishes, and the root's wake refuses and leaves the root suspended. To close: test both sub-agent paths, order the gate after the unknown-call check, and fail a suspended root whose wake is refused (as the active branch does).
Done (R26 G4): forwardToOwner now runs the same completion-support gate on the ROOT's agent and adapter before forwarding (typed 400, nothing written, nothing forwarded), so a refusable root no longer lets a child finish into a refused wake. Tested on real DO instances (ForwardSubmitAgentServer in packages/e2e/src/test-worker.ts, cases in completion-strategies-do-entry.cf.test.ts): a refusable root refuses before forwarding (child stays pending, no rows, no model call); a refusable child refuses the forwarded submit in its shortcut and the root relays the 400 verbatim; a control proves the seeded forward lands on the child. The cells seed the root's ownership entry: on the DO runtime a remote sub-agent that suspends on a client tool fails fast at the parent (RemoteSubAgentClientToolUnsupportedError, core/src/errors/agent-errors.ts, GitLab #107) and a child's registerPending writes ownership into its own DO store, so no natural flow reaches forwardToOwner today.
Not done, by decision: reordering the shortcut gate after the unknown-call check. In the shortcut a call with no pending entry is the submit-before-register race: the resolver writes a pre-submit stub and answers accepted (runtime-js/src/client-tool-resolver.ts, the rootSessionId branch of submitOnce); unknown_tool_call is only returned for a missing session, which the shortcut's isSubAgent check already excludes. Moving the gate after the resolver would write a stub on a refusable child, and the 404 it was meant to restore is unreachable there.
Remaining: fail a suspended (paused) root whose wake is refused, as the active branch does.
FU-COMPLETION-COMPANION-RESUME-REFUSAL-REF: a refused companion resume leaves the parent's ref running — open (partly done)
Done (the ref half): catalogue record CP-103 (fixed), the completion-strategies test-hardening MR (!325), Tasks 16 and 16B (design note docs/superpowers/specs/2026-10-08-completion-hardening-fixes-design.md §2). Core handleSendMessage no longer writes the ref when it defers the message to a resume. The resumed child's own run-start commit writes it, atomically: StartRun.resumedCompanionRef: { parentSessionId } (A1) turns the parent's ref running, completionDelivered: false in the same atomic step as the run record, its rows and the input checkpoint, on memory, Redis, Postgres and D1. A refused or lost start writes no ref, so the ref keeps its prior status (interrupted on runtime-js and CF Workflows, which record it; on Temporal the stale running of CP-73); a missing ref fails the commit typed (RunStartRejectedError('resumed_companion_ref_missing'), nothing written). The original fix direction (the description below: "mark the ref failed on any child-start throw") was not taken: a compensating write can race the child and is lost on a crash. Per runtime:
- runtime-js (
resumeChildAgent), Temporal (thecompanionResumechild entry,initializeAgentState) and CF Workflows (thecompanionRef__resume__and__continue__instances since W1 MR2b,startRunCommit) pass the field. Temporal and CF Workflows also refuse an unfinishable resume in the parent (Temporal's companion activity, CF Workflows' companion step) before any child starts, so the tool result is the refusal, notdelivered: true. - The CF Durable Object does not use the field. A DO without
subAgentNamespacerefuses persistent agents (400persistent_agents_no_subagent_namespace), and DO-SQLite refuses the field (UnsupportedOperationError). The namespaced DO keeps its own write: its companion tools call the child DO's/resumefirst and write the ref only after it succeeds (two Durable Objects, so not atomic). - DBOS is unaffected: it writes the ref only after a delivered inbox send.
Tests: core orchestration/__tests__/companion-tool-dispatch.test.ts; the store contract scenario commit.start-companion-ref-atomic (core/src/testing/run-start-operations.ts) on all five stores (DO-SQLite in 'refused' mode); runtime-js/src/__tests__/run-loop-persistent.test.ts; runtime-temporal/src/__tests__/activities-companion-resume-ref.test.ts, dispatch-companion.test.ts, companion-resume-missing-ref.integ.test.ts and replay/companion-resume-replay.integ.test.ts; runtime-cloudflare/src/__tests__/completion-children.wf-noiso.test.ts, cfw-persistent-companion.test.ts and do-persistent-agents.test.ts (the namespaced-DO contrast); runtime-dbos/src/__tests__/execute-companion-tool.test.ts (the busy-inbox pin); e2e persistent-subagent-parity-harness.integ.test.ts; conformance companion.resume-refused-keeps-ref and companion.resume-busy-keeps-ref. Remaining on Temporal and CF Workflows: a resume that loses its run-start (or whose parent ref is gone) still reports delivered: true (CP-105, FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW).
Still open: on runtime-js a refused ephemeral sub-agent emits subagent_start before its refusal (runtime-js/src/run-loop.ts: the subagent_start write comes before the child-start try that calls refuseUnsupportedCompletion). To close: emit subagent_start only after the child start succeeded, with a unit case.
Owner: runtime-js / core companion dispatch. Pre-existing for any resumeChildAgent throw; surfaced by MR 1 Task 14.
When the early refusal rejects a persistent companion's resume, the parent's SubSessionRef for it stays running. Also, on runtime-js a refused ephemeral sub-agent emits subagent_start before its refusal. To close: mark the ref failed (typed) on any child-start throw, and emit subagent_start only after the child start succeeded, with a unit case each.
FU-COMPLETION-CHILD-REFUSAL-REF-PARITY: a child refused at its own start leaves a failed ref on CF Workflows and no ref on runtime-js — open (LOW, owner: runtime-cloudflare, the CF backlog)
Surfaced by: the completion-strategies test-hardening MR (!325), Task 12 (P1-6, P2-15).
A sub-agent child or a persistent companion's first spawn whose model the adapter cannot finish is refused at the child's own start and writes nothing on both runtimes (spec §4.6): no child session, run or message, and the parent's tool row is a failed call carrying the refusal message. The parent's ref differs:
- CF Workflows registers the ref (
registerSubSessions) before the child instance runs its recordedcompletion-supportstep, so the refused child ends with afailedref whosesubSessionIdnames no session (a client that follows the ref to load the child gets not-found). Pinned byruntime-cloudflare/src/__tests__/completion-children.wf-noiso.test.ts(the P1-6 and P2-15 cases assert thefailedref). - runtime-js writes a sub-agent child's ref only after the child's start succeeded (
run-loop.ts,addSubSessionRefsafter the child run), so a refused sub-agent child leaves no ref. A persistent companion's first spawn is different since CP-161: core inserts its ref before the dep, and the refusal is answered typed (not_spawned/child_entry_failedwith theframework_completion_*detail), the ref writtenfailed, delivered (I-C10), and no child session created (the refusal precedescreateSession).
Related, the same on every runtime: the parent's tool row carries only the refusal message, with no code (core SubAgentResultInput has only error?: string), so a parent recognises a child refusal by text alone.
Fix direction: pick one contract and apply it on every runtime (runtime-js, the CF DO, CF Workflows, Temporal, DBOS): either a refused child leaves no ref, or a typed failed ref, with a parity case in the e2e sub-agent parity suite; flip the completion-children.wf-noiso.test.ts assertion if CF Workflows changes.
FU-COMPLETION-CFW-REPLAY-REFUSAL: CF Workflows re-runs the completion-support check on replay — done
Owner: runtime-cloudflare. Surfaced by: MR 1 Task 20 review.
entryRefusal (runtime-cloudflare/src/workflow.ts) runs checkCompletionSupport in the workflow body outside a step.do, so it runs again on every replay. A deploy that changes how an outputSchema agent's model resolves (a capability-table change, a removed or changed capabilities override) can fail an instance whose run already committed. Documented (CF Workflows runtime page, migration guide). To close: record the refusal decision for the entry in a step.do (so a replay reuses it) or skip the check once the instance's run-start commit is recorded.
Done (R26 G3): the check runs inside step.do('<agentType>-completion-support') at the top of the body's main try, returning { ok: true } | { ok: false, error: { message, code } } as data (no retries); the typed non-retryable HelixError is rebuilt outside the step and ends through the unchanged pre-commit refusal path. A replay reuses the recorded decision. Proven by two cases in runtime-cloudflare/src/__tests__/workflows/completion-strategies.workflow.test.ts (a recorded { ok: true } runs to completion although the live adapter would refuse; a recorded refusal fails typed although the live adapter would admit). An instance recorded before this step existed checks fresh on replay (the old behaviour).
FU-COMPLETION-DEMOTE-NO-K5: the DO's failed demote writes no K5 checkpoint — open (partly done)
Done in WA MR1: the failed demote branch is deleted. Crash-loop exhaustion and a refused wake end the run through failCommittedRun({ requireSessionCommit: true }), which writes the K5 terminal commit with the detail, then settle it. Still open: callers of ensureExecutionContinues other than the wake ladder (the submit continuation) do not check the interrupt flag.
Owner: runtime-cloudflare (DO wake recovery). Pre-existing; surfaced by MR 1 Task 15 review.
demoteStrandedExecution(..., 'failed', ...) (used for auto_resume_exhausted and, since MR 1, for a stranded run whose agent is now refused) ends the session failed without the K5 commitState terminal commit and writes errorDetail after the run status, against the terminal order (K5 → run status → stream). Also, callers of ensureExecutionContinues other than the wake path skip the interrupt-flag check. The recovery test covers only the reset path (run already completed). To close: route the demote through the K5 terminal commit and add the failed-path recovery case.
FU-COMPLETION-CONFORMANCE-WORKER-CAPABILITIES: the conformance worker's ProxyModel has no capabilities — open (partly done)
Owner: the conformance owner (this session does not edit packages/e2e/src/conformance/**).
Partly done (!325, Task 16A, H1 / H2): every lane's adapter now applies a per-model capability rule (conformance/agents/completion-support.ts): in-process through ScriptedModel's new resolveCapabilities option (harness/completion-support.ts, createConformanceModel), inside workerd through ProxyModel's (the cf-do DO and the cfw workflow body). The rule only refuses (refuse-completion-model, switch-completion-support): every other model id resolves to LEGACY_RESOLUTION. Still open: selecting a native_output / history-bound profile per cell, and the forced-completion scenarios for both profiles.
packages/e2e/src/harness/cf-do/worker/conformance-worker.ts builds new ProxyModel(new BridgeClient(...), c) without a capability profile, so a cf-do conformance cell always runs the legacy profile and cannot exercise native_output, the prefix check or the early refusal on the DO. The CF Workflows harness added by conformance plan 0c MR-1b (packages/e2e/src/harness/cfw/worker/cfw-deps.ts, cfw-conformance-worker.ts) builds its ProxyModel the same way, so a cfw cell is legacy too. The MR 1 e2e harness (ProxyModel, the bridge protocol) already forwards resolveCapabilities / checkOutputFormat and carries outputFormat, minOutputTokens, parts and providerOptions; the worker needs a way to select the profile per cell. Then: forced-completion scenarios for both profiles (spec §11, "conformance").
FU-COMPLETION-DI28-CITATIONS: DI-28 cites lines MR 1 moved — open
Owner: the conformance owner (see the catalogue citation-pinning rule).
DI-28's fix evidence (packages/e2e/src/conformance/catalogue/discipline/DI-28.ts:20, mirrored in docs/conformance/findings.md:240) cites e2e/src/__tests__/forced-completion-parity.integ.test.ts:410 and forced-completion-parity.cf.test.ts:240, which MR 1 Task 24 reshuffled. Its contrast citations docs/guide/agents.md:294-296 and docs/guide/finishing-agents.md:243-246 also moved with MR 1's docs. Re-cite with the MR number, pinned to the merge commit.
FU-COMPLETION-WF-NOISO-LONG-PATH: some *.wf-noiso files cannot start from a long worktree path — open (local only)
Owner: runtime-cloudflare test infra.
Nine long-named *.wf-noiso.test.ts files fail to start when the repository lives under a long path (for example a ~/.superset/worktrees/... worktree): the workerd temporary directory path exceeds the platform limit. CI checkouts are short and unaffected. To close: shorten the per-file temp dir (hash the file name) in scripts/run-wf-noiso.mjs or the pool config.
FU-COMPLETION-FAILED-RUN-PHASE-STATE: a failed run's still-pending forced phase differs across runtimes — open
Owner: runtime-js / runtime-cloudflare. Surfaced by: MR 1 whole-branch review A4 (parked by ruling R24).
When a run fails for a reason other than the forced call's own outcome while its forced-completion phase is still pending (a skills or memory failure on a forced call, a throwing generateStep), CF Workflows fails the phase in its K5 terminal commit and writes the exhausted diagnostic, while runtime-js and the CF DO leave the phase pending for the same failures. retry() is unaffected on both (it restores the phase from the step commit, never from K5). The divergence is stated in docs/internals/concepts.md (Completion strategies). To close: align the runtime-js / DO onFailed K5 with CF Workflows, or document a CF-Workflows-only contract, with a cross-runtime case. Temporal matches CF Workflows since completion strategies MR 2 (its catch's K5 commit passes failPendingForcedPhase).
FU-COMPLETION-CFW-EXHAUSTED-CHUNK: CF Workflows writes the K5 exhausted chunk before the K5 commit — open
Owner: runtime-cloudflare (CF Workflows). Surfaced by: MR 1 whole-branch reviews A5 + B7 (parked by ruling R24).
CF Workflows writes the exhausted forced_completion chunk for a failed run's pending phase best-effort BEFORE its K5 commit. A step.do retry or a crash re-writes it, and a K5 that fails every attempt leaves the stream showing exhausted for a phase that is still pending in state. The chunk is diagnostic only (no reader acts on it). To close: write it after the K5 commit lands, once per run (e.g. keyed on the K5 checkpoint id).
FU-COMPLETION-PREFIX-REFUSAL-STEP-START: a prefix refusal leaves an attempt's step_start unpaired — open
Narrowed (2026-11-10): CF Workflows and Temporal only. Core checkPrefixStability is called only at runtime-cloudflare/src/steps.ts:3323 and runtime-temporal/src/activities.ts:3901 (a220d5dfa8). runtime-js and the CF Durable Object stopped refusing a changed prefix in batch 13 (234725409d, Reasoning Recovery): the shared step iterator recovers once instead, so neither a prefix-refusal step_start nor a framework_prompt_prefix_changed session arises there.
Owner: core (step iterator) / runtime-cloudflare (CF Workflows). Surfaced by: MR 1 whole-branch review A7 (parked by ruling R24).
The prompt-prefix check (framework_prompt_prefix_changed) runs after the attempt's step_start is written (runtime-js step-iterator.ts, around the buildLLMCallbacks call; CF Workflows steps.ts, the prefix check after the attempt's step_start), so the stream holds a step_start with no step_committed / step_discarded and no attempt_started; the run fails right after. To close: move the check above buildLLMCallbacks (before step_start), or write step_discarded for the attempt before failing, with a stream-order test on both runtimes. Temporal does not inherit it: completion strategies MR 2 refuses before step_start (plan decision D-A).
FU-COMPLETION-OVERRIDE-SCHEMA-UNSUPPORTED-TEST: no runtime-level test of a per-call schema refusal under an override — open
Owner: runtime-js / runtime-cloudflare. Surfaced by the R26 re-review.
With appliesLlmConfigOverride, an agent with llmConfigOverride is not refused at entry, and a forced call whose overridden model is native-only with a schema it cannot take fails with framework_completion_schema_unsupported. That path is covered only in parts (the llm-vercel checkOutputFormat test, the core forced-call-error tests). To close: a runtime-js case (native-only resolver keyed on the override model, an unrepresentable schema) and a DO cell for an override to an unsupported model.
FU-COMPLETION-PREFIX-CHANGED-RECOVERY: no recovery operation for a framework_prompt_prefix_changed session — open
Narrowed (2026-11-10): CF Workflows and Temporal only. Core checkPrefixStability is called only at runtime-cloudflare/src/steps.ts:3323 and runtime-temporal/src/activities.ts:3901 (a220d5dfa8). runtime-js and the CF Durable Object stopped refusing a changed prefix in batch 13 (234725409d, Reasoning Recovery): the shared step iterator recovers once instead, so neither a prefix-refusal step_start nor a framework_prompt_prefix_changed session arises there.
Status: closing with Reasoning Recovery (spec 2026-11-05-reasoning-recovery-design.md): runtime-js and the CF DO are wired, and CF Workflows, Temporal and DBOS follow in its MRs 2 to 4. MR 1's owed cell, reasoning.lost-marker-self-heals-host (cf-do, js-chat-host), now runs on the 'fault-step-commit' control: a recovered reply whose step commit is refused loses its marker, the run fails, and the next turn is refused and recovers the same way. It is an integration guard with no mutation red: Reasoning Recovery has no reset cap (resetsInRun only picks the log level, and counts markers the lost commit never wrote), so no product line makes it fail alone.
Owner: core. Surfaced by: the MR !301 superpowers review (recommendation 3; parked by the R26 brief). Future work.
A session on a model with history-bound thinking that fails with framework_prompt_prefix_changed (its system prompt or tool set changed under signed reasoning blocks) can only be abandoned today: every further call replays the same signed blocks against the changed prefix. To close: a recovery operation (for example a retry / resume option, or a store-level rewrite) that strips the signed reasoning parts from the session's earlier assistant rows so the history is no longer bound to the old prefix, then lets the run continue under the new prompt, with a test on each runtime that has the prefix check.
FU-RUN-START-PLAN-TYPE-HYGIENE: the companion-resume ref is typed by hand on three runtime plan types — open (LOW, owner: the next run-start / CF MR, CF backlog)
Surfaced by: !325 final review (type-design L1, L8, and M2: the companionResume && parentSessionId → resumedCompanionRef derivation is written separately in Temporal initializeAgentState and CF Workflows workflow.ts; one core resumedCompanionRefOf helper should own it). StartRun.resumedCompanionRef is redeclared, with the same conditional spread, on runtime-js RunStartPlan (runtime-js/src/run-start.ts), Temporal EntryRunPlan (runtime-temporal/src/run-entry.ts) and CF Workflows CfwEntryRunPlan (runtime-cloudflare/src/run-entry.ts), and CF Workflows carries it twice (StartRunCommitInput.resumedCompanionRef and the per-attempt plan, merged in steps.ts). CF Workflows' refined AgentWorkflowInputSchema still infers the unrefined type, so workflow.ts casts parentSessionId as string, and createChildInstance takes untyped params. No behaviour is wrong today. Fix direction: base the plan types on Pick<StartRun, 'truncateTo' | 'entryStepCount' | 'pendingRestamp' | 'resumedCompanionRef'> with one core helper that copies the defined optionals into StartRun; pass the entry-constant ref once on CF Workflows; have the CFW input schema .transform into resumedCompanionRef (removing the cast) and type the child-create params as AgentWorkflowInput (satisfies, as Temporal does).
FU-CONF-HARNESS-TYPE-HYGIENE: conformance harness literals and mirrored types are not tied to their schemas — open (LOW, owner: the conformance owner)
Surfaced by: !325 final review (type-design L6, L7). Owner files, so not changed in !325:
e2e/src/harness/completion-support.tscreateConformanceModel(opts: Omit<ScriptedModelOptions, 'resolveCapabilities'>)acceptscapabilities, whichScriptedModelthen refuses at construction (core keeps the runtime check). Omitcapabilitiestoo, and consider making coreScriptedModelOptionsa union so the exclusivity is compile-time.e2e/src/harness/cf-do/bridge-protocol.ts:CompletionSupportTableSchemahard-codes'conf/switch'instead of deriving fromSWITCHABLE_COMPLETION_MODELS, admits'default'(never emitted),'refused' | 'default'is spelled three times,DriverCompletionSupport.settakesmodelId: string,RunStartCommitFields.expectCurrentRunStatusisz.string()whereRunStatusSchemaexists, andRunStartSettleReport/RunStartHoldSettled/RunStartEnterResulthand-mirror their schemas. Derive the literals and addExpect<Equal<…>>type assertions.
FU-CHECKPOINT-ID-COLLISION: two checkpoints of one session can get the same id; D1 and Redis then report a refused commit applied — done (RM-138, !926)
Filed: 2026-11-17 by the conformance owner (RM-138), from a records gate's one-off failure of store.list-checkpoints-paging-tied-stepcount on the in-memory store. Related: RM-138, RM-54, RM-38.
- The cause: core
generateCheckpointId(core/src/types/checkpoint.ts:225-229) is the session id, the stepCount,Date.now()and 6 hex chars ofMath.random(): 24 random bits per (session, stepCount, millisecond). Every commit path mints with it (corebuildCommitCheckpoint,commit-checkpoint.ts:23; Redis's own commit paths), and so does each store'screateCheckpoint. Measured: RM-54's concurrent batch repeats an id about once in 10,000 runs. - What a repeat does (by reading, origin/main 0aed8d1250):
- D1: the batch fails on the primary key, and
resolveWriteFailure(store-cloudflare/src/d1-state.ts:1490-1512) reads backWHERE id = ? AND session_id = ?, finds the EARLIER checkpoint and returns the commit as applied: nothing was written. - Redis: the commit script's re-send check (
store-redis/src/commit-script.ts:1255-1261, before the plan) finds the earlier commit's recordedcommitReplyunder the repeated id and answers it as this commit's: applied, nothing written, as on D1. The TS read-back after a transport fault (redis-state.ts:2167) has the same id-only hazard. An earlier checkpoint with no recorded reply (a directcreateCheckpoint) is overwritten instead (writeCheckpoint,commit-script.ts:305-319), and so is one a directcreateCheckpointrepeats; a rewind to it restores the later state. (Corrected 2026-11-18 from W1's trace.) - Postgres, DO-SQLite: the insert fails and the commit answers
CommitNotAppliedError(retryable). - In-memory: the map is overwritten and the session index lists the id twice.
- D1: the batch fails on the primary key, and
- Fix direction: mint the id's random part with
crypto.randomUUID()(or 128 bits fromcrypto.getRandomValues). Make every read-back and re-send check match a per-attempt commit nonce, never the id alone (D1's read-back, Redis's re-send check and read-back), so a repeated id can never read as applied. - Tests: a per-store unit that forces equal ids through a stubbed generator, pinning each store's answer (D1 and the rest: not applied, never applied; Redis and memory: refused, never an overwrite). RM-54's paging case stays as the witness.
- CI: until this lands, a recurrence of
store.list-checkpoints-paging-tied-stepcounton any pipeline goes to the orchestrator, never a silent retry.
Done (!926, RM-138): checkpoint ids are cpv2 with 128 random bits; every guarded commit's read-back and re-send check matches a per-attempt nonce (D1 V38, Postgres V36, Redis on the checkpoint's meta), Redis refuses a held id (CHECKPOINT_EXISTS, also its direct createCheckpoint), and memory re-mints until free. Proven per store by core's checkpoint-id-collision contract through the id seam (W1's s-rm138 run 2 and s-rm138r). The paging witness passes on all five stores; the CI line above is retired.
FU-CHECKPOINT-STATE-STATUS-TYPE: Checkpoint.state.status is typed as the AgentState union, not what commits store — open (LOW, owner: core)
Surfaced by: !325 final review (type-design L13). Commit checkpoints store a SessionStatus in state.status, but Checkpoint.state is typed AgentState, whose status is the AgentStatus union; core's clone helper (checkpointSessionStatus) therefore reads it as unknown and normalises both. Fix direction: type Checkpoint.state.status as stored (SessionStatus | AgentStatusValue, or a dedicated checkpoint-state type) so the normaliser takes a typed input. Pre-existing; no behaviour change.
D-2 MR 1 (request keys) follow-ups
Spec: 2026-10-13-d2-request-key-design.md (read its "As built" first).
FU-D2-REJECT-HEADER-KEYLESS-ONLY: the internal reject header is still sent on keyed chat entries — done (D-2 MR 1.5b)
Resolved by D-2 MR 1.5b: the header is keyless-only, a send's response is its own run's (from its start, pinned to its runId), and the O-70 rule is a DO invariant (!377). See the spec's "As built (MR 1.5)". The original entry follows.
Spec §5 makes x-helix-reject-concurrent-entry keyless-only. MR 1 still sends it on keyed entries (DOFrontendExecutor.execute / resume; orchestrator-approved deviation). The header is the DO's only O-70 trigger (checkConcurrency's suspended_run rule), and dropping it would let a keyed new send follow interrupt_previous without the pinned send response. A keyed duplicate is unaffected: the DO answers it from its key before the concurrency check. MR 1.5 (CP-13's interrupt half) makes the header keyless-only together with:
- the pinned send response (
pinnedRunChunkson the send's own run, from itsstartSequence); - a DO-level O-70 rule that does not depend on the header.
FU-CORE-HANDLE-RUNID: core's AgentExecutionHandle has no runId, so a chat send's run is read, not named — done in core and runtime-js (b22 A1); the other runtimes: FU-ENTRY-RUN-PARITY-TEMPORAL-DBOS-CFW
Closed by: b22 A1: core's handle carries runId / startSequence / startUIMessageCount / contextIdsTaken, guaranteed by an executor that declares reportsEntryRun (runtime-js; the CF DO's client already named its run). MinimalExecutionHandle.runId stays optional: the current-run read stays as the fallback for an executor without the flag.
FU-CHAT-SEND-PINNED-STREAM-EMPTY-TYPED-DROP: a chat send whose pinned stream ends with no chunk of its own run answers an empty 200 — open (LOW; defence in depth for CP-337; no fix chosen)
Filed: 2026-11-19 at the orchestrator's request, from WA's CP-337 plan. Related: CP-337 (the CF Workflows cause, fixed at the entry wait), FU-CORE-HANDLE-RUNID (the pinned send response).
- The gap: a send streams its run's own chunks from the run's start (
pinnedRunChunkson the handle's stream,handle-chat-stream.ts). When that stream ends before any chunk of the run arrives (CP-337: a handle returned before its run claimed the stream, so the reader found the previous turn's ended stream), the response is a 200 with an empty assistant reply. Nothing tells the client the reply is missing; the run's answer appears only on a reload. - Suggested direction (WA, not chosen): a send whose pinned stream ends with no chunk of its own run answers a typed drop, e.g. the non-retryable
transport_stream_drop/run_not_livethe submit attach uses (Q12), so the client re-reads the snapshot. To weigh: a run that legitimately writes nothing before a clean end, and the attach-path rules that already exist fornew_run/deferred. - Scope: every runtime's send path through
handleChatStream; CP-337's fix removes the known cause on CF Workflows, so this is the backstop, not the fix.
FU-ENTRY-RUN-PARITY-TEMPORAL-DBOS-CFW: Temporal, DBOS and CF Workflows handles do not name their entry run — open (b23, spec 2026-11-11-entry-input-answer-design.md §8)
Each mints its run id caller-side, and its entry wait already reads the committed run and drops it (Temporal awaitEntryOutcome, DBOS awaitExecutorEntry, CF Workflows awaitEntryRunStart). Fix: return the RunMetadata from the wait, put runId / startSequence / startUIMessageCount on the handle, set HelixError.entryAnswer at the entry seam, declare reportsEntryRun. Until then handleChatStream decides their answers from the session's current run, and unknown cannot be resolved by a keyed re-send there (FU-D2-RUNTIMES-REQUEST-KEY). Paths that read no run leave the fields unset: CF Workflows' explicit entryStartTimeoutMs return, its operator-paused return, the adopted-continuation handle, and DBOS's null outcome. DBOS's handle classes already expose runId (public since A1, which made core's AgentExecutionHandle.runId a public member); their startSequence / startUIMessageCount are still unset.
Only the CF DO's handle (DOExecutionHandleImpl) exposes its run (runId + startSequence). The runtime-js, Temporal, DBOS and CF Workflows handles do not, so handleChatStream takes a send's run from stateStore.getCurrentRun() after execute() (readOwnRunStart in ai-sdk/src/handler/handle-chat-stream.ts). That is safe today only because a later entry can take the session over before that read solely by interrupting a live run, which only a DO on interrupt_previous does. Add runId (and the run's startSequence) to core's AgentExecutionHandle on all five runtimes, make MinimalExecutionHandle.runId required, and keep the current-run read only as a cross-check. Required before any other runtime lets a send interrupt a live run. Seen in batch 29: RM-74's leg-a integ read handle.runId on Temporal, which is unset, and now reads the resumed run from the store (runtime-temporal/src/__tests__/integration/with-message-drain-first.integ.test.ts:230-233, origin/main 0aed8d1250).
FU-D2-RUNTIMES-REQUEST-KEY: CF Workflows, Temporal and DBOS ignore request keys — open
They declare supportsRequestKey: false. A key passed to execute() / resume() / retry() is ignored with one request_key.ignored warning per executor (core ignoredRequestKeyWarner), and handleChatStream sends keyless to them. Each needs planKeyedEntry before its entry and the key on its run-start commit, as runtime-js (JSAgentExecutor.enterKeyed) and the DO drive (planKeyedDrive) have.
FU-CONF-CFDO-SEND-AFTER-KEYED-START: no cf-do cell witnesses CP-229 on the Durable Object — open, deferred (orchestrator, 2026-10-24: low value while CP-157 / G1 / DI-76 are open)
CP-229 (runtime-js's handle.send() re-used a keyed execute()'s requestKey, so the store refused the send as state_request_key_conflict; fixed in !508, 7010be85b9) is witnessed on the js executor lanes only (session.send-after-keyed-execute-continues). The CF Durable Object runs runtime-js, so the same fix covers it, but no cell shows the DO's path.
- Is the path reachable on the DO? No DO route calls
handle.send(): every native/start(and every chat send) is its ownexecutor.execute()with that request's key (do/drive/run-drive.tscase 'start'), and the drive itself reads onlyhandle.result(). But the publicafterEntrydrive hook receives the run'sAgentExecutionHandle(do/drive/host.tsAfterEntryContext.handle), so app code on a DO can awaitresult()and then callhandle.send(…)on a run that a keyed/startstarted. That is the one DO path, and it is app code, not a route. - The witness, if wanted: a cf-do host cell, sibling of the executor scenario. The cf-do test worker's DO subclass gets an
afterEntrycontrol armed per cell: it awaitshandle.result(), callshandle.send(message)and records the send's outcome (status, or the error's code) for an observe channel. The cell sends a keyed chat POST (X-Helix-Request-Key, which the DO turns into the drive'srequestKey), waits for the recorded send outcome under a named settle, and checks that the session holds both user turns. js-chat-host has noafterEntry: a declared gap. Red at !508's 22fa724145 and green at 7010be85b9, as the executor cell. - Size: medium, about one working day. One worker control and its capability, one observe channel, the scenario, the harness self-tests (
test:conformance-harness) and driver unit tests, then one cf-do slot (red and green builds). The worker's main module may export only functions or classes. - Value: low to medium. The code path is the runtime-js send callback the js cells already witness, reached only through app code in
afterEntry. The case for it is DO-first coverage of a public hook surface.
FU-D2-REMOTE-TRANSPORTS-REQUEST-KEY: remote transports do not send a key yet — open (D-2 MR 2)
The DO's /subagent/:type/start|resume routes plan a keyed request (MR 1, with its DO integ). The remote transports (DOStubTransport, the HTTP transport) still send none, so a re-POST after an unknown outcome still goes through CP-95's RemoteEntryOutcomeUnknownError path. MR 2 sends the deterministic key and its conformance twin on CP-95.
FU-D2-CHAT-TOOL-RESULT-KEYLESS: a tool-result chat POST stays keyless — open
handleChatStream forwards a request key on a send (execute) and on an abandon-and-send (resume with_message), not on a tool-result POST (path 3: submit + resume continue). A duplicate there is already safe: the re-submit is idempotent, and its continuation attaches through the existing busy-attach fallback. Keying it would record the continuation under the key.
FU-D2-MIGRATION-RENUMBER: the request-key migrations are numbered provisionally — done (D-2 MR 1, rebased onto !358)
W1's deleteSession cascade (!358) took Postgres V17, D1 V21 and DO-SQLite 15. MR 1's request-key migrations are Postgres V18, D1 V22 and DO-SQLite 16, with each store's migration-test pins moved to them.
FU-D2-KEY-ONTO-TAKEOVER-CONTINUATION: the key stays on a dead owner's run after a takeover — open
A keyed retry of a runtime-js run whose owner died takes the run over as a continue resume (found and fixed in MR 1 before merge: it used to attach and wait forever). The dead run keeps the key (now superseded), and the continuation records none, so a later duplicate of the same request is 409 state_request_already_completed (the client re-anchors on the snapshot, which shows the continuation) instead of attaching to the continuation. Moving the key onto the continuation needs every store's run-start commit to transfer a superseded holder's key in the same atomic step.
FU-D2-REVIEW-MINORS: deferred minors from MR 1's final review — open
- Digest determinism (M-3): a DO
transformRequestthat rewrites the input, or per-request session metadata layered into each message, makes the handler's digest differ from the recorded one, so a true duplicate gets 409state_request_key_conflict. Document that both must be deterministic across retries, or digest content and files only. - Digest width (M-4):
computeRequestDigestis 32-bit FNV-1a; a collision attaches a different request to the old run. Use the adjacent 64-bit hash. - agent-server window (M-5): a duplicate arriving between the original's
createSessionand its run-start commit sees no key holder and gets 409ALREADY_RUNNINGinstead of attaching. instanceofintypedErrorBody(M-7):runIdis attached by class; a duplicated core copy loses it. Key it on the code plus arunIdfield.- Client (M-8):
useHelixChathas no dedicatedstate_request_already_completedre-anchor (withresync,retry()'s snapshot check covers it), andretry()withoutresyncreuses the last send's key even when the failed request was a keyless continuation. - Small (M-9): agent-server
/chatvalidates the key on any POST (alsomessages: []); PostgresisRequestKeyUniqueViolationis a hand-written guard on driver errors (repo preference: Zod).
Note for FU-CHAT-HANDLE-RESPONSE-MESSAGE-ID (filed by !356)
Removing streamHandle's post-commit run re-read needs the run's startUIMessageCount on the execution handle. An attached handle has the same need: handleChatStream reads its run's start from the current run (attachedRunStart). Both can move to the handle together.
Redis store: multi-step writes and existence checks (surfaced by RM-50)
RM-50 (!269) made createSession and appendMessages single Lua scripts. The audit behind it found the multi-step writes below, where a concurrent reader can see a partial state. Each still needs a fix. "Delete race" means the write runs between a separate existence check and its own commands, so it can recreate a status-less session hash that loadState rejects with InvalidStoredStateError: the old appendMessages shape. The fix pattern is the HEXISTS sessionId guard inside one script, as in APPEND_MESSAGES_ATOMIC_SCRIPT.
FU-REDIS-RM50-01: remaining multi-step Redis writes — open
Owner: unassigned (reliability orchestrator to route). All sites are in packages/store-redis/src/redis-state.ts.
| Site | Shape | Partial state a reader can observe |
|---|---|---|
saveState | Lua, then createSessionCheckpoint (EXISTS + pipeline), then a separate pointer HSET | checkpointId stays empty until the trailing HSET, so getLatestCheckpoint falls back to stepCount ordering. The trailing HSET can race a delete. The dangling pointer is RM-38 |
promoteStaging | EXISTS/HGET/LLEN reads, then a Lua script with no in-script existence check | delete race; its no-staging path builds a checkpoint from separate reads |
createCheckpoint / createSessionCheckpoint | EXISTS + pipeline (pointer last) | delete race |
incrementStepCount / incrementResumeCount | EXISTS + pipeline HINCRBY | delete race |
mergeCustomState | EXISTS + SMEMBERS + a non-MULTI pipeline | half-applied ops (e.g. DEL list before RPUSH reads []); mixed old/new keys |
cloneSession | loadState + createSession + HSET + appendMessages | the target is visible without its messages. RM-27, cross-reference only; owned elsewhere |
deleteSession | HGETALL + index Lua + pipelines + SCAN + DEL | the long window is where every delete race above lands; a same-id create running concurrently has its keys deleted by the later DELs |
compareAndSetStatus | EXISTS + Lua + EXPIRE + HGET + index EVAL | the status:* index lags the hash |
createRun / updateRunStatus | EXISTS / HMGET + pipeline | the run is briefly missing from listRuns; a delete race recreates a partial run hash |
addSubSessionRefs | none: the session check, the type check of every ref key and the writes are one atomic step | |
updateSubSessionRef | EXISTS + pipeline | a delete race recreates a partial ref hash |
truncateMessages | none: the trim and the messageCount rewrite (from LLEN, only when the hash exists) are atomic, and serialize with the atomic appendMessages | |
loadState (a reader) | two non-transactional pipelines | a concurrent atomic write can yield a mixed version (old array keys, new list contents) |
Also: saveState and saveStateAndPromoteStaging never move status:* index entries, so listSessions({ status }) drifts.
FU-REDIS-RM50-02: sessionExists (EXISTS) vs the append / create guard (HEXISTS sessionId) — open (LOW)
sessionExists treats any key as a session, but createSession and appendMessages test the sessionId field. They disagree on a legacy status-less hash left by the pre-RM-50 appendMessages. A caller that creates-if-missing on sessionExists then skips the create, and its next append throws Session not found (runtime-js/src/run-loop.ts:~2638, and CF Workflows' saveAgentState create-if-missing). Fix direction: make sessionExists test HEXISTS sessionId. Recorded as evidence under RM-50 / CP-75.
FU-REDIS-RM50-03: persistent-spawn initial-message append error is swallowed — open (LOW)
runtime-js/src/run-loop.ts:4754-4761 catches a failed appendMessages of a re-spawned persistent child's initial user message and only logs a warning. The child then runs without its initial message. This is reachable only if the session vanishes right after createSession. Fix direction: rethrow, or fail the spawn with a typed error. Recorded as evidence under RM-50 / CP-75.
Eviction recovery (DO wake auto-resume) follow-ups
Deferred items surfaced by the four-lens code review of the eviction-recovery change (DO wake auto-resume + crash-loop guard + companion wait hardening + JS orphan detection).
FU-C2A-BACKLOG-BEYOND-READER-BUDGET: a reader's backlog (cursor → caught-up) is read whole before the reader returns — open (its own MR after C2b, orchestrator ruling D3 2026-10-18)
WE C2a bounds both Durable Object /sse readers' unread buffers in bytes (core READER_BUFFER_HIGH_WATER_BYTES, 8 MiB) and count, but only past the backlog boundary: createResumableReader returns once it has read the route's caught-up{head} frame (its catchUpHead is a number at return), so the history replay before it is read whole into memory. Applying the budget there would deadlock reader creation on a backlog larger than the budget (found by the C2a security review; tests … a backlog larger than the byte budget before caught-up … in do-stream-manager-client-reopen.test.ts / durable-stream-reopen.test.ts). C2a caps it instead (READER_BACKLOG_MAX_BYTES, 32 MiB; orchestrator ruling (b), 2026-10-17): past it the reader ends with StreamTruncatedError and the client re-anchors from the snapshot. Bounding the backlog by the buffer budget itself needs the head before the replay, e.g. a head-first frame: a wire change, C2b's.
FU-C2B-REOPEN-RUN-CHANGE-TYPED-SIGNAL: a re-opened reader that finds a newer run ends as a plain drop — done (WE C2b)
WE C2a: a Durable Object /sse reader that re-opens after a drop and then reads a chunk of a NEWER run of the stream's own session (another run took the stream while it was detached) ends with StreamDropError (core StreamReopenBudget.endsRead, packages/core/src/stream/reopen-policy.ts), so the client re-anchors from the snapshot. A typed signal (the run changed, not a transport drop) is a wire addition: C2b's evicted{resumeFrom, reason} frame work should refine it (orchestrator ruling 2026-10-17, design note §7.2).
Done in WE C2b: such a read ends with StreamRunChangedError (transport_stream_run_changed, retryable; a StreamDropError, so every drop handler keeps its recovery) on all three readers (the DO run handle's stream(), DOStreamManagerClient.createResumableReader, the store-cloudflare binding reader). useHelixChat re-anchors on the route's error event for it without reading a code (ai-sdk react/typed-drop-reanchor.test.tsx).
FU-C2B-BROWSER-RECONNECT-G3-G4: the browser still shows an error for a stream it could have resumed — open (awaits the user's ruling; owner: orchestrator)
Surfaced by: WE C2b (!416), out of scope; filed by the DoD catch-up (WD, 2026-10-19). Design record: docs/superpowers/specs/2026-10-18-do-sse-reopen-and-evicted-design.md §4.
The server-side DO readers re-open on a drop or an eviction, but the browser leg (browser ↔ worker) does not: a drop there reaches useChat as status: 'error', and useHelixChat / useReanchorOnError re-anchor from the snapshot, so the user sees an error flash and a re-render. G3 (resume without a visible error) and G4 (detect a clean-EOF truncation in the browser) both need an opt-in custom ChatTransport, a divergence from the stock ai transport that needs the user's ruling first.
FU-EVICT-01: ai-sdk HelixChatTransport swallows /resume 409 and drops the user message — done (stale: WE MR C1's postEntry rejects a refused /resume; !944 removes the transport)
Closed 2026-11-19: WE MR C1's postEntry rejects a non-2xx /start or /resume (CP-07's fix evidence), and !944 removes createHelixChatTransport altogether (user ruling 2026-11-19).
packages/ai-sdk/src/transport/helix-chat-transport.ts (~:243-265) never checks res.ok on submit; a 409 from /resume (likelier now that wake auto-resume keeps stranded sessions active) silently drops the accompanying user message. Pre-existing; surface a typed error or retry.
FU-EVICT-03: JS runtime marks a SUSPENDED persistent child's ref completed — open
runtime-js/src/run-loop.ts (~:4338-4350): the spawn IIFE treats a child run that exits at a suspension boundary as completion and writes the ref completed. Consequence: core's paused_awaiting_client ref-status exemption (added for orphan detection) is dead code on runtime-js — nothing ever writes that status outside the store contract tests. Align the ref status with the child's real outcome.
FU-EVICT-04: usage-store counters for wake auto-resume / exhaustion — open
Fleet operability of eviction recovery is logs-only. Precedent exists (CLIENT_TOOL_RUNTIME_RESTARTED_COUNTER, SCHEMA_LIMITS_FALLBACK_COUNTER); add counters for auto-resume attempts and exhaustions (framework_auto_resume_exhausted, a K5 terminal since WA MR1) so operators can count them fleet-wide without log aggregation.
FU-EVICT-05: dead-onFire audit of relative-deadline alarm subscribers — open
AlarmScheduler.fireExpired(now) can never fire a subscriber whose compute() returns Date.now() + interval (deadline always future). All three current subscribers (heartbeat, interrupt-poll, execution-watchdog) have this shape — their onFire bodies are unreachable under real alarms and their real work happens elsewhere (documented on registerSubscriber). Either give subscribers durable next-fire deadlines or remove the misleading onFire bodies.
FU-EVICT-07: residual copy-tests in hibernation-recovery.test.ts — open
shouldRejectInterrupt / simulateSuccessPathOnComplete (~:39-115) re-implement production logic locally — the anti-pattern whose simulateOnStart sibling was deleted by the eviction-recovery change. Replace with tests against the real code paths or delete.
FU-EVICT-08: park-failure compensation can still lose a recovered child's output — fixed in code (DO companion MR A); flips with the owner's ledger
Fixed in code by DO companion MR A (spec 2026-10-15, CP-125, D1); not done until its witnesses (the CP-125 cells) are ledgered on main. The failure window no longer writes anything: markChildFailedAndAbort is deleted, and the park and companion__waitForResult answer the typed unreachable (runtime-cloudflare/src/do/persistent-companion-tools.ts parkOnChild, buildWaitForResultTool; core waitForResultUnreachable / companionUnreachableMessage), leaving the ref running and the child running. A child that recovers and completes reaches its parent through the child-settled notice and the parent's verified read, and the completion notifier delivers it (consultOutcomeNotifier). The reconciliation sweep this item asked for is the in-tool verified read plus the durable notice (D3 (a) chose push over a turn-start sweep). Guard: the I-C1 ref-writer test (runtime-cloudflare/src/__tests__/do-companion-ref-writers.test.ts). Witnesses: the conformance CP-125 cells (below, in the MR description). Residual (I1, as left by MR B): a recovered parent's model mints a fresh tool-call id, so its re-emitted spawn is not a C14 replay; a recorded outcome reaches the model exactly once, through the notifier or as the re-emitted continue's priorResult (deliver-then-continue, held on the ref and released after the step commit), but either way that continue runs the child again (FU-DO-COMPANION-ORPHANED-CONSULT-ADOPTION, MR C). Temporal / DBOS continues still clear an undelivered outcome (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY, parked); CF Workflows delivers it. The original text follows.
When the poll failure window trips on TRANSIENT unreachability, the ref is marked failed/delivered and the child is best-effort aborted. If the abort ALSO fails (child unreachable) and the child later recovers and completes, its output is never delivered (notifier skips delivered refs). Residual accepted for now — the LLM is told the wait failed and can re-spawn — but a reconciliation sweep (compare ref vs live child status on parent wake) would close it.
FU-EVICT-09: crash-loop attempt over-count on a transient recovery CAS loss — open
recoverStrandedExecution persists the incremented crash-loop record (nextRecord, attempt+1) BEFORE calling ensureExecutionContinues. If ensureExecutionContinuesImpl's version-pinned active → paused reset CAS then LOSES (e.g. a concurrent /submit-tool-result bumped the version between the two loads), the impl returns early without running an LLM step, yet the attempt was already counted. Repeated pathological interleaving could reach maxWakeResumeAttempts slightly early → a premature exhaustion (framework_auto_resume_exhausted). Self-healing (the concurrent producer that won the CAS drives the run → committed message progress → counter resets on the next recovery) and /retry-recoverable, so Minor. A clean fix: have ensureExecutionContinues report whether it actually started a resume, and only persist the incremented record on a real attempt. Deferred to preserve the pure-planner / thin-recovery split.
FU-EVICT-10: unpinned re-arm CAS after a failed post-reset resume — open
The paused → active re-arm in ensureExecutionContinuesImpl's resume .catch (fired only when WE reset active → paused and the resume then threw in its cleanup phase) is NOT version-pinned. It is gated on didResetFromActive && stateAfter.status === 'paused', so it cannot resurrect a genuine HITL suspension, and it is self-correcting (the next recovery cycle reads the interrupt flag first and demotes to interrupted, honoring any user interrupt). Verified non-resurrecting; pinning it to the post-reset version would make the intent explicit. Minor.
Remote skill loading (@helix-agents/skill-cli) follow-ups
FU-SKILL-CLI-MANIFEST-READ-UNWRAPPED: runCli doesn't wrap manifest read/parse — RESOLVED
Status: resolved · Severity: low (UX polish)
runCli now wraps the manifest readFile / JSON.parse in a try/catch that prints a helix-skills: <message> diagnostic and returns exit code 2 (matching the manifest-validation path); the bakeSkills call is likewise wrapped (exit 1 on a resolve/bake throw). A missing or malformed helix.skills.json no longer surfaces as an unhandled rejection. Regression tests: cli.test.ts › runCli error paths (missing manifest → 2; invalid JSON → 2).
FU-SKILL-CLI-MARKETPLACE-NO-ZOD: marketplace/plugin JSON parsed without a Zod schema — open (LOW)
Status: open · Severity: low (defense-in-depth)
ClaudeMarketplaceSource parses marketplace.json / plugin.json via JSON.parse + a TypeScript interface cast, not a Zod schema — malformed marketplace metadata yields an opaque error. Add a Zod schema for these structures for defense-in-depth (consistent with the rest of skill-cli, where the manifest and lockfile are Zod-validated).
Also lacking a malformed-marketplace-JSON test: the untrusted-external-data case (a marketplace.json / plugin.json that parses as JSON but has the wrong shape) has no test today. Add one alongside the Zod schema so the hardened validation has teeth.
FU-SKILL-CLI-INTEGRITY-EXCLUDES-FRONTMATTER: lockfile integrity hash omits skill frontmatter — RESOLVED
Status: resolved · Severity: low (edge case under mutable tags/branches only)
packages/skill-cli/src/bake.ts previously built the integrity file-map from the PARSED body (files[${meta.name}/SKILL.md] = body), not the raw SKILL.md — so the frontmatter (name / description / allowed-tools) was excluded from skillIntegrity(files), and a frontmatter-only change did NOT move the lockfile integrity hash.
Fix (landed): walkSkills now carries the raw SKILL.md text onto WalkedSkill.raw, exposed via ResolvedSkillSource.getRaw(name); bake.ts builds the integrity file-map's ${meta.name}/SKILL.md entry from resolved.getRaw(meta.name) (the full raw file incl. frontmatter) instead of the parsed body. A frontmatter-only change now moves the hash and is detected by helix-skills sync --check. Regression test: bake.test.ts › integrity hashes the raw SKILL.md — a frontmatter-only change moves the hash. (The codegen still uses the parsed body, unchanged.)
FU-SKILL-CLI-INTEGRITY-PER-MANIFEST-ENTRY: lockfile integrity is per-manifest-entry, not strictly per-skill — open (LOW)
Status: open · Severity: low (doc/wording accuracy)
packages/skill-cli/src/bake.ts aggregates ALL selected skills of one manifest entry into a single files map keyed by the manifest entry's localName, then writes one integrity per lock.skills[localName]. A multi-skill entry (e.g. a claude-marketplace plugin that ships several skills, or a git source whose include lists more than one) therefore gets ONE combined tree-hash spanning all its selected skills — not one hash per skill.
Doc wording reconciled: the README + docs/reference/skill-cli.md no longer claim "one tree-hash per skill" — they now say "one integrity hash per manifest entry (over its selected skills' raw files)", which matches the implementation. The integrity.ts docstring was updated likewise. The item is kept open as a design choice: switching the lockfile to truly per-skill lock entries (one integrity per resolved skill name) is still a possible future change, but the doc inaccuracy that motivated this item is now fixed.
FU-SKILL-CLI-TEMPDIR-CLEANUP: degit temp dirs never removed — RESOLVED
Status: resolved (2026-10-25) · Severity: was low, raised to medium: it filled a dev disk.
packages/skill-cli/src/git-source.ts and packages/skill-cli/src/marketplace-source.ts cloned into mkdtemp(<tmpdir>/helix-skill-cli-*) and never removed the dir. Every helix-skills sync / --check left one full repo clone, and two for a marketplace whose plugin lives in another repo. The video-gen app's hyperframes sync left 25 clones of about 330 MB each in one day (8.3 GB), which tripped the conformance owner's disk-floor abort.
Fix:
degitToTempDirremoves its own partial dir when the clone fails.GitRepoSource.resolve()andClaudeMarketplaceSource.resolve()remove every clone they made, in afinallyafter the walk, which has already read every body and resource into memory.- A dir an injected
materializereturns is the caller's, and is never removed.
Tests: src/__tests__/temp-clone-cleanup.test.ts uses a stubbed degit and checks:
- success, for git and for a marketplace with a second plugin clone;
- each failure path: a failed clone, a missing plugin, a missing
marketplace.json, a failed plugin clone; - that an injected dir survives.
FU-SKILL-CLI-BIN-SYMLINK-NOOP: helix-skills exited 0 without baking when started through a symlink — RESOLVED
Status: resolved (2026-10-25) · Severity: high (silent): a consumer's CI went green with stale skills.
The bin ran only when import.meta.url === `file://${process.argv[1]}`. A package manager starts it through the node_modules/.bin/helix-skills symlink, where Node resolves the module to its real path but argv[1] stays the link. So npx helix-skills sync (and --check) printed nothing, baked nothing and exited 0. The video-gen app worked around it with a wrapper script that calls bakeSkills directly.
Fix: src/cli.ts is now only the bin, and always runs runCli. The library half (parseCliArgs, runCli) moved to src/run-cli.ts.
Test: src/__tests__/bin-symlink.test.ts runs the BUILT bin dist/cli.js with plain node through a node_modules/.bin/helix-skills symlink. It checks that sync bakes, and that a missing manifest exits 2 with a helix-skills: error. A local run without a build skips it; in CI (CI_JOB_ID) a missing dist fails.
FU-SKILL-FS-SYMLINK-REALPATH: readResource containment check is lexical, not realpath — open (LOW)
Status: open · Severity: low (security-hardening)
fileSystemSkillProvider.readResource (packages/skill-fs/src/filesystem-provider.ts) enforces skill-dir containment with a lexical check (resolve + relative), NOT realpath. A symlink that lives INSIDE a skill dir but points OUTSIDE it is therefore followed by read_skill_file — and the path argument is LLM-supplied (runtime-untrusted input), so a crafted skill resource tree could exfiltrate files outside the skill dir. Disclosed in the docs as operator-trusted-only (v1 limitation #2), so this is hardening, not a live bug for operator-provisioned dirs.
Hardening: after the lexical check passes, realpath BOTH the target file and the skill dir and re-assert containment, returning null on escape. Trade-off to decide first: this would break operators who intentionally symlink skill resources to shared content outside the dir — so decide opt-in (a provider option) vs default-on before implementing.
FU-SKILL-DBOS-CATALOG-IN-WORKFLOW-BODY: DBOS resolves the skills catalog in workflow-body code — open (LOW)
Status: open · Severity: low
DBOS resolves resolveSkillsCatalog directly in @DBOS.workflow body code (packages/runtime-dbos/src/workflows/standard-workflow.ts / persistent-workflow.ts), NOT inside a @DBOS.step. For an in-code provider this is fine (deterministic data, no IO). For a fileSystemSkillProvider it runs replay-sensitive node:fs IO in workflow code — a soft determinism/cache-churn issue. It cannot corrupt committed state (the LLM result is checkpointed downstream) and degrades non-fatally, but it is the reason fileSystemSkillProvider is discouraged on DBOS (see the Skills cross-runtime matrix in docs/internals/concepts.md). Fix: wrap the DBOS catalog resolution in a checkpointed @DBOS.step, mirroring Temporal (which resolves it inside the per-step activity).
FU-SKILL-RESOURCE-FILES-MARKED-UNTRUSTED: a skill's own resource files reach the model marked untrusted, its body does not — open (feature / design, user ruling needed)
Raised by: an SDK consumer (2026-11-19, via the orchestrator). Related: the skills spec (2026-06-03), its read_skill_file section, and the remote-skill spec's deferrals.
- Today:
read_skill_filewraps every Level-3 file in<skill_file … untrusted="true">and adds a note telling the model not to follow instructions inside it (coreskills/skill-tools.ts), as the skills spec specifies.load_skillbodies and preloaded bodies carry no such mark. So a first-party, pinned skill whose instructions live inreferences/*.mdtells the model to follow the skill while the framework tells it not to follow the files. - No override: both skill tool names are reserved, and only the framework's own tools may use them;
SkillProviderhas no trust field. - Measured impact: none yet. The consumer's A/B (Sonnet, n = 26) showed no measurable adherence loss. So this is a correctness item for the trust model, not a defect firing.
- Proposed direction (pending the user's ruling): an opt-in
trustedonSkillProvider. When set, no wrapper or note, and the catalog header says resource files are part of the skill's instructions. The default stays as today. The remote-skill spec's deferred hardening (bodies are untrusted input too) is the other side of the same model. - To close: the user rules; then the option, its docs, and a core unit for both modes.
- Related: the roadmap's trust-levels item (user ruling: trust levels last). This may be decided with it or ahead of it, by the user's ruling.
FU-STARTUP-GUARD-ZOD3-COPIES: the dist check and countZodCopies only recognise zod 4 — open (LOW, SDK startup cost)
ZOD_MARKER (packages/startup-guard/src/dist-check.mjs) matches zod 4's $constructor("$ZodType", ...). A stray zod 3 copy inlined into a dist (or a Worker bundle) has no such marker and is invisible to the check. Add a zod 3 marker (for example ZodFirstPartyTypeKind) next to it.
FU-STARTUP-GUARD-DIST-WORKER-BUNDLE: the dist check flags store-cloudflare's local test-worker bundle — open (LOW, local only; owner: WE-2)
Surfaced by: the CP-203 gate (WE-2, 2026-10-22). checkWorkersDists() (packages/startup-guard/src/dist-check.mjs) scans every file under a Workers-facing package's dist/. store-cloudflare's test:integration first runs build:worker, which writes dist/worker/test-worker.js: a Miniflare test bundle (tsup.worker.config.ts, noExternal match-all) that inlines zod on purpose and is never published. Once it exists, startup-guard's test fails "no Workers-facing dist inlines zod or immer" with that file. So a local test:integration of store-cloudflare followed by pnpm test reds the guard until dist/worker is removed. CI cannot hit it: the build job runs plain tsup (no dist/worker in its artifacts), test:unit tests only those artifacts, and build:worker runs only in the separate test:integ:rest pod, which never runs the guard. Wanted: the scan skips store-cloudflare/dist/worker/ (or writes the test bundle outside dist/), with a comment on why, and a red.
FU-SKILL-CORE-YAML-IN-WORKER-BARREL: @helix-agents/core barrel drags yaml (CJS) into Worker bundles — done
Status: done · Severity: low · Closed by: branch sdk-startup/mr3-load-less (SDK startup cost MR 3/3)
Fixed by the first option below: parseSkillFile / ParseResult left core's main entry for the new subpath @helix-agents/core/skill-file, which skill-fs and skill-cli import, so core's main entry no longer loads yaml (proved by packages/startup-guard/src/__tests__/core-main-entry.test.ts, ESM and CJS). The store-cloudflare test worker's yaml stub is removed (its bundle is byte-identical without it).
Lifting parseSkillFile into @helix-agents/core (commit 0a8a94c1) added a yaml dependency to core. parseSkillFile is re-exported from the core barrel (@helix-agents/core) and statically imports yaml, a CommonJS package whose yaml/dist/log.js does require('process'). Any consumer that bundles the core barrel into a Cloudflare Worker without nodejs_compat and without tree-shaking parseSkillFile out now pulls yaml in, and esbuild's __require('process') shim makes workerd fail at startup ("Dynamic require of 'process' is not supported"). Core can't be marked sideEffects: false (it has a required import-time enablePatches() in state/immer-state-tracker.ts), and its dist is a single bundled index.js, so per-file sideEffects won't help either.
This surfaced in @helix-agents/store-cloudflare's test worker (tsup.worker.config.ts, noExternal match-all, no node compat), which is worked around by stubbing yaml there (the store worker never parses SKILL.md). Real Worker apps typically enable nodejs_compat (which provides process) and the other Cloudflare CI lanes pass, so impact is low — but the core barrel is advertised as Workers-safe and a Node-CJS dep now taints it. Fix options: expose parseSkillFile via a dedicated core subpath (e.g. @helix-agents/core/skill-file) kept out of the main barrel and consumed by skill-fs/skill-cli; or move the parser to a Node-only package (skill-cli would depend on skill-fs). Either keeps the Workers-facing barrel yaml-free.
Deep test-coverage review (post-FU-sweep) — deferred breadth gaps
A deep 8-audit test-coverage review of the recently-shipped arc (Temporal parallel-tool execution, history-continuation gate, parent_session_id round-trip, DBOS HITL, cross-runtime correctness waves, stream/DO + O5, providerExecuted/lifecycle/usage) found one production bug (D1 saveStateAndPromoteStaging dropped parent_session_id — fixed + contract test added) and a set of coverage gaps. The closeable parity/breadth gaps were closed in the same sweep (CF parallel-tool compose, Temporal activity-retry idempotency, postgres re-stage upsert, legacy providerExecuted round-trip, onStateChange plain-tool parity, JS scalar-LWW exact, Temporal parallel-tool redis/postgres legs, DBOS in-place-push composes + doc correction, DBOS previousRunId unit). The items below are deferred breadth extensions (each documented in its area's test header where applicable).
FU-DBOS-INTEG-SUITE-GATED-IN-CI: runtime-dbos integration suite runs 0 tests in routine CI — done (batch 26, !884; owner: WA)
Severity: high — the runtime-dbos integration suite (~49 files / ~88 tests) is gated behind RUNTIME_DBOS_INTEG=1 (GL #108), and the test:integ:dbos CI job does NOT set it, so the suite registers zero tests in MR/main pipelines. DBOS HITL guarantees whose ONLY coverage is in packages/runtime-dbos/src/__tests__/integration/* are unprotected in CI (a regression merges green). Load-bearing arbiter/rollback/timeout contracts DO have non-gated unit + e2e-parity backstops (so this is depth-not-sole- protection for most), but ~6 flows are gated-only: multi-parallel suspended.toolCallIds, persistent multi-turn 2nd submit, persistent SEND-path hooks, F1 resume-time-hooks, sub-agent child-owned suspend/forwarded-concurrent, submit-after-completion-timeout, predicate checkpoint-replay determinism. Action: either fix GL #108 (DBOS-singleton reset / event-driven waits) and flip include, or set RUNTIME_DBOS_INTEG=1 on the test:integ:dbos job (the suite passes locally with it set — verify stability first). Decide deliberately; the gate was added for a reason.
Casualty (2026-10-23): CP-155's ref sync (d919c9b584, 2026-10-20) turned the negative control at companion-continuation.integ.test.ts:208 red, and nothing ran it until W1's !483 gates (FU-DBOS-SENDMESSAGE-FAILED-NEGATIVE-CONTROL-STALE).
Closed (2026-11-15, batch 26, !884): test:integ:dbos runs the suite on every MR and train pipeline (ci/test-matrix.yml: RUNTIME_DBOS_INTEG=1, parallel: 2 with --shard, and RUNTIME_DBOS_INTEG_REQUIRE_TESTS=1, whose guard in packages/runtime-dbos/scripts/run-integ-tests.mjs fails a paused suite or a shard that ran zero tests). GL #108's flakes were fixed first (!874 in batch 25; CP-270 and CP-328, !881 in batch 26). First real runs, batch 26's !887: merge-result pipeline 3070902232 and train pipeline 3070968920, each shard 1/2 167 passed / 1 skipped and shard 2/2 130 passed / 1 skipped, 297 / 2 over 103 files, every job green. WA keeps a per-shard ledger of the batch pipelines. Release need (user ruling 2026-11-16): stays as is, no flip. test:integ:dbos has gated the release since batch 26 through the train gate and the child matrix; optional: true covers only a main push that has no test jobs. Green batches on record: 26, 27 and 28. Batch 29 (!908) added the CI comment and the repo pin test (core/src/__tests__/repo/ci-heavy-once.test.ts:287, origin/main 0aed8d1250).
FU-DBOS-SENDMESSAGE-FAILED-NEGATIVE-CONTROL-STALE: DBOS's sendMessage-to-a-failed-child negative control forges a state the product now heals — open (LOW, test bug; owner: the DBOS bucket, parked)
runtime-dbos/src/__tests__/integration/companion-continuation.integ.test.ts:208 ('sendMessage to a FAILED/TERMINATED child still throws (negative control)', main e4a79cda2a) runs a companion to completed, then sets the parent's ref to failed directly and expects exec 2's companion__sendMessage to answer "not active". It answers {"delivered":true} (W1's gates g1062 / g1062b, 2026-10-23, red at main 8cd8beb434 at the :305 assertion).
Diagnosis, inferred from source (not run): since CP-155 (d919c9b584, 2026-10-20), core deliverPendingChildCompletions, which DBOS runs every parent turn (runtime-dbos/src/steps/deliver-child-completions.ts:30), re-syncs an undelivered terminal ref to the child's own word (core/src/orchestration/completion-delivery.ts:73-83). So exec 2 turns the forged failed ref back to completed before the tool runs, and sendMessage continues the completed child (Plan 3c). The product's refusal is intact: a failed ref throws "not active" (runtime-dbos/src/steps/execute-companion-tool.ts:1253-1259). Not a finding (orchestrator ruling, 2026-10-23), on that basis.
Fix: make the child genuinely fail (a scripted model error, so its session and ref both read failed) instead of forging the ref, and keep the "not active" assertion. Fallback: if the genuinely-failed child is still answered {"delivered":true}, the diagnosis is wrong: file a conformance finding instead (orchestrator ruling, 2026-10-23).
FU-CFW-CORE-HOOK-PARITY: CFW Workflows lacks non-suspend core-hook parity coverage — open
lifecycle-hooks-parity.wf-noiso.test.ts covers only the suspend/resume scenario on cfw-workflows-d1; the full core-hook set (onAgentStart, onMessage, onStateChange, beforeTool, afterTool, onAgentComplete) is not asserted on that backend (agent-hooks-parity.integ.test.ts covers JS/Temporal/DBOS; CF-DO has agent-hooks-invocation.test.ts, CFW has no equivalent). Action: add a CFW workflow-pool core-hook invocation test via the recordedHooks bridge.
FU-CF-STREAM-CHUNK-PARITY: 31-variant chunk round-trip + terminal-state idempotency not extended to CF stream managers — open
stream-chunk-roundtrip-parity (31 variants) and stream-terminal-state-parity (endStream/failStream idempotency, latestSequence) run only InMem+Redis. The DO-internal / binding-side managers run the G1–G4 concurrency contract but not the chunk-fidelity sweep nor the terminal-state idempotency assertions, so a DO/D1 chunk variant that loses a field through the SQLite JSON round-trip wouldn't be caught. Action: add a workerd stream-chunk-roundtrip-parity.cf over DOStreamManager + extend terminal-state assertions to the DO managers.
FU-EXPIRED-SESSION-D1-COVERAGE: expired-session cleanup has no D1/DO coverage — open
expired-session-cleanup-parity.integ.test.ts runs js+temporal (memory/redis/ postgres) only; cf-do-d1 is hard-filtered (Node-context suite). D1's listSessions/expiresAt round-trip + CAS-to-failed sweep is unverified end-to-end (honest gap, acknowledged in the test header). Action: add expired-session-cleanup-parity.cf.test.ts over selectViable(CF_BACKENDS, { only: ['cf-do'] }).
FU-MIXED-STATUS-RESUME-CONTINUE-PARITY: mixed-status resume→continue run-record only on Temporal — open
The suspend(turn1)→resume(turn2)→continue(turn3) run-record sequence (3 runs, turns [1,2,3], mixed status vector) is pinned only on Temporal (multi-turn-continuation.integ.test.ts). JS, DBOS, CF have no equivalent; CF continuation run-record-per-turn is documented-as-parity but never directly asserted (CF is also excluded from the e2e continuation floor). Action: add the 3-turn mixed sequence to the e2e continuation suite as a describe.each(getViableBackends({requires:['hitl']})) block + a listRuns() .length === 2 assertion to the CF DO continuation test.
FU-REVIEW-MINORS: assorted low-risk coverage minors — open (LOW)
Branch+history: interaction untested (all runtimes keep branch baseMessages semantics — distinct code path); 3rd-turn continuation history-drop relies on the > 0 gate but isn't explicitly tested at turn 3; previousRunId === undefined on a first run not asserted; remote sub-agent resume dimension is JS+memory only (not fanned out across runtimes). Each is cheap to add.
Persistent sub-agents convergence — Phase 4 (CFW) follow-ups
Surfaced during the Phase 4 (Cloudflare Workflows) convergence + its code review. None block Phase 4 acceptance — the convergence is correct and verified (18 unit tests + real-workerd cfw-persistent.wf-noiso.test.ts).
FU-CFW-COMPLETION-AT-LEAST-ONCE: completion-message append is not idempotent on step retry — ✅ FIXED
The parent's per-step completion injector appends hidden completion messages in one step.do and persists completionDelivered in a SEPARATE step.do (workflow.ts ${agentType}-inject-completions + -mark-completions-delivered). Splitting them (Phase 4 review fix) means a flag-write failure retries WITHOUT re-appending. The residual: if the APPEND step.do itself crashes mid-write, Cloudflare retries it and appendMessages (non-idempotent) double-appends — the framework-wide appendMessages-in-step.do pattern, not specific to this path. Blast radius is one workflow run (CFW has no multi-turn execute yet). Fixed: the inject-completions step.do now re-reads the recent message tail and drops completion messages already present (dedupeCompletionMessages in steps.ts, keyed on content + metadata.source === 'sub-agent-completion') before appendMessages, so a step retry after a partial write no longer double-appends. Pure dedup helper unit-tested in dedupe-completion-messages.test.ts. Residual (1) resolved by DO companion MR B (Task 9): dedupeCompletionMessages now keys on the notice's structural key, metadata.helixCompanionNotice = { subSessionId, runId } (core companionNoticeKey), never on content: a candidate whose run-keyed notice is already in the tail (or earlier in the batch) is dropped, a same-output re-spawn's notice (a new run) is kept, and a key without a run is always appended (the flag decides). Note for long-lived parents: checkCompletedChildren reads the parent's whole committed history (getAllMessages) once per pass whenever a held slot or a run-keyed candidate exists — acceptable now; bound it (a tail window keyed on the notice's sequence) if parents grow long. Note (MR B final review M4): dedupeCompletionMessages never dedupes a notice without a runId (by design: the flag decides for it), so a re-executed inject-completions step could append such a notice twice; practically unreachable on CFW, because a completed CFW child always has a current run and every notice checkCompletedChildren builds is run-keyed. The residual text follows for the record. Residuals (LOW, from the review): (1) the dedup window is sized to the current batch, and the dedup key is the message CONTENT — so in the rare case of ≥3 completions in one batch that includes a same-output explicit re-spawn whose prior identical completion is still within the tail window, the new (legitimate) completion would be wrongly dropped (a lost notification, never a duplicate). Common single-child batches are unaffected (the prior completion is separated by the intervening LLM step + spawn tool-result, outside a window of 1). Fully robust fix = a per-delivery unique key in the completion message metadata (e.g. subSessionId) instead of content — a core buildCompletionMessage change, deferred. (2) Only the pure dedupeCompletionMessages helper is unit-tested; the inject-step wiring (getMessageCount + tail offset/window) is not (a step.do retry is hard to simulate in a unit test). Both are acceptable for this LOW item (CFW has no multi-turn execute yet).
FU-TERMINATE-FORCEFUL-ABORT: durable-runtime terminateChild does not abort the running child workflow — open (LOW, CFW + DBOS + Temporal)
terminatePersistentChild is a no-op on all three durable runtimes (CFW steps.ts buildCfwDispatchDeps, DBOS, Temporal activities.ts): termination relies on the durable interrupt flag core's handleTerminateChild sets, which the child workflow observes at its next interrupt-check poll. This matches pre-convergence behavior (flag-only; never aborted the child workflow), so it is NOT a regression and is UNIFORM across CFW/DBOS/Temporal (deliberately — a Temporal-only forceful cancel would make Temporal inconsistent with the others). Gap: a running child blocked inside a long tool/LLM step keeps consuming resources after terminate returns. Action (cross-runtime):
- CFW: forceful
workflowBinding.get(childWorkflowId).abort()— needs the workflow binding threaded intoAgentSteps+ a durable child-workflow-id registry (the id gains a__respawn__Nsuffix on re-spawn). - Temporal:
wf.getExternalWorkflowHandle(childWorkflowId).cancel()— carries a child-workflow-id-reconstruction risk across a parent suspend/resume boundary (the child was spawned under the original parent workflow id; a resumed parent has a__resume-Nid), so the id must be tracked durably, not reconstructed. - DBOS: the equivalent cancel on the child DBOS workflow handle.
FU-PERSISTENT-COMPANION-TOOLS-PER-DRIVE-SCHEMAS: buildPersistentCompanionTools builds zod schemas on every drive — open (LOW, SDK startup cost)
buildPersistentCompanionTools (runtime-cloudflare/src/do/persistent-companion-tools.ts) builds z.object / z.enum schemas each time a Durable Object drives an agent that has persistent companions. This is a hot path, not startup (the startup guard does not see it). Cache one schema set per companion-name list, or build them with lazySchemaFactory.
FU-WORKSPACE-FAILFAST-PERSISTENT-CHILD (Phase 6 / C8): persistent-child workspace fail-fast — DBOS fixed; CFW covered; Temporal FIXED (TR-a)
Phase 6 closed the C8 gap where a persistent CHILD agent declaring a workspace bypassed the parent-executor assertRuntimeSupportsWorkspaces and failed LATE. Status per runtime:
- DBOS — FIXED.
serialize.tsnow captureshasWorkspaceonSerializedAgent(theWorkspaceConfigitself isn't JSON-serializable), andhandleSpawnAgentasserts fail-fast BEFORE any ref/session/workflow side effects. Unit-tested. - CFW Workflows — already covered. The persistent child self-asserts at its own workflow entry (
workflow.ts:522), so it fails at child-start (not late at tool-call). Minor follow-up (LOW): a blocking parent currently waits for thestep.waitForEventtimeout rather than failing at the spawn; add a spawn-dispatch assert (child config is reachable atworkflow.ts:2826viapaConfig.agent.workspace) so the failure surfaces at the parent immediately. - Temporal — FIXED (Temporal rewiring TR-a). The child run-start activity
initializeAgentState(activities.ts) callsassertRuntimeSupportsWorkspaces, so a persistent child declaring a workspace fails fast at startup with the shared message (the throw surfaces within themaximumAttempts: 3cap, like other deterministic init failures). The child workflow's top-level catch RETURNS{status:'failed'}(does not throw), so the blocking dispatch inworkflow-dispatch.tsinspects the resolvedchildResult.statusto mark the parent's SubSessionRef'failed'.sub-agent-fail-fast-temporal.integ.test.tsis now un-skipped and green against a real Temporal server (unlike CFW it fails at the parent immediately — no blocking-spawn timeout). NOTE: the earlier claim that Temporal v7 companion dispatch "is not implemented" was STALE; it was largely working before the rewiring (verified by the persistent integ suite).
FU-DBOS-BLOCKING-SPAWN-SEMANTICS: DBOS blocking-spawn blocks-until-idle and mis-reports a failed child as completed — fixed (CP-69, lands with its MR)
Narrowed (2026-10-29): the misreport of a PAUSED child (answered running at the bound, never C2's suspended) is CP-247 (fixed by !589). The wait itself (a blocking spawn holds the parent's turn until the child's idle-out) is CP-69, fixed here.
Catalogued as CP-69 (runtime conformance catalogue, packages/e2e/src/conformance/catalogue/control-plane/CP-69.ts; see docs/conformance/findings.md), with CP-124 (companion__waitForResult never saw a stopped turn), whose root it shares.
From the Phase 5 review. A persistent child ran every turn with skipSessionStatusUpdate:true, so its SESSION status stayed active until the recv loop idled out, and finalizeLoop then wrote status:'completed'. A blocking spawn polled that session: (1) it blocked until the child's persistentIdleTimeoutMs (default 24 h) rather than until the child's first turn; (2) a child whose turn FAILED but then idled out could be reported completed.
Fixed (Option B, per-turn session truth). Each persistent turn's K5 writes its terminal (completed, failed, interrupted) to the session, as on every other runtime; the flag is gone. The loop's exit is its own durable marker (metadata.dbosLoopFinalized, runtime-dbos/src/persistent/loop-finalized.ts): finalizeLoop records it (in the finalize / entry-failure step) and a persistent turn's entry clears it (R43 (b)'s late message). Every reader that took a terminal session as "the loop exited" reads the marker or the loop's DBOS workflow status instead (execute()'s route and inbox wait, companion__sendMessage, a spawn re-consult, the inbox check), each liveness read recorded. The blocking park watches the turn's own run; waitForResult and the park answer a stopped turn interrupted (resumable). Consequence (2) closes with it: each turn's outcome is its own, and finalizeLoop keeps the last one (runFinalizeRun's FU-A2-19 gate).
FU-DBOS-TERMINATE-IDLE-LOOP: companion__terminateChild over a companion between turns leaves its DBOS loop to idle out — open (LOW)
Surfaced by: CP-69 (per-turn session truth). A DBOS companion between turns reads its last turn's completed, so core's terminate answers { terminated: false, status: 'completed' } (the same answer as every runtime) and writes nothing; its recv loop stays parked in recv until its persistentIdleTimeoutMs (24 h by default), holding a DBOS workflow open. Nothing is lost: a later re-consult takes that loop. Fix direction: on DBOS, a terminate over an idle loop (no live run, no recorded exit) could also send it a shutdown, freeing it before the idle timeout, without changing the answer. Not in CP-69's MR (orchestrator ruling).
FU-DBOS-COMPANION-INTEG-GAPS: DBOS persistent companion lacks end-to-end delivery + crash/recover integ tests — open (MED)
From the Phase 5 review. (1) No integ test proves the Phase 5e notifier actually DELIVERS a non-blocking child's completion to the parent (unit-proven + integ-non-regression only — companion-non-blocking finishes the parent before the child completes, so the notifier is never the delivering path). (2) No crash/recover integ test exercises the C5 blocking-spawn body-replay (the design said this "MUST be integ-verified before claiming C5"). Add: (a) a parent with a turn AFTER a non-blocking child completes, asserting the hidden sub-agent-completion message appears; (b) a restart-mid-blocking-spawn test asserting no duplicate child + a single consistent tool result. Env: PG on 5434
- Redis (see the Phase 5 design doc's env note). Pattern available: the Temporal rewiring (TR-b) proves gap (a) with a race-free pre-seed test — seed a terminal-but-undelivered non-blocking child ref + child state, run the parent one turn, assert the hidden
sub-agent-completionmessage + the ref'scompletionDeliveredflip (seepersistent-subagent-temporal.integ.test.ts"completion notifier delivers..."). The same pattern ports directly to DBOS and sidesteps the shared-MockLLM concurrent-spawn race.
FU-TEMPORAL-NOTIFIER-LIVE-NONBLOCKING-RACE: live non-blocking spawn not integ-tested on Temporal (shared MockLLM queue) — open (LOW, test-infra)
The Temporal completion notifier (TR-b) is proven by a race-free pre-seed integ test (see FU-DBOS-COMPANION-INTEG-GAPS above) + core unit tests (completion-delivery.test.ts). A LIVE non-blocking spawn (parent spawns, child runs concurrently, completes, parent's next turn delivers) is NOT integ-tested: parent + child share one MockLLMAdapter response queue, so the concurrent child races the parent for responses (persistent-subagent-temporal.integ.test.ts:200 stays it.skip-ped for this reason). Fix needs per-agent LLM response queues in the Temporal test harness (or a deterministic child that consumes no LLM responses). Same test-infra limitation as DBOS; not a runtime bug.
FU-DO-COMPANION-FRESH-START-DUPLICATE: two narrow races in which a DO companion's fresh-session start wipes the CURRENT child — ✅ closed (CP-210; (a) by !476, (b) by !488)
Filed by: the independent review of !484 (CP-205). Since CP-205 every DO companion /start except a replay that found no child carries the fresh-session headers, and the child's planRespawn wipes a non-live session of the same stored parent. Two narrow orders reach it with a child of the parent's CURRENT incarnation:
- (a) ✅ closed by CP-161's claim (!476): two same-name first spawns in one step. Before CP-161 both read no ref (the snapshot cannot see a sibling's insert), so both started the child, and if the first child ended before the second
/startlanded, that start wiped it (data loss since CP-205). The ref insert is now the claim (addSubSessionRefs{ inserted }): the loser sees the winner's ref. A live winner answerschild_busybefore any child IO; a failed or terminated one is a respawn, by design; a COMPLETED one is a continue (continuePersistentChild), which on the CF DO posts a plain/startthat asks for no fresh session, so nothing is wiped. Pinned by the core unit "FU-DO-COMPANION-FRESH-START-DUPLICATE (a): a claim lost to ANOTHER call's COMPLETED ref continues it; never a spawn or delete", mutation-proven:classify()withcanContinueforced false, and the lost-claim branch skipped (the pre-CP-161 shape), each fail "the winner's completed child is continued: expected "spy" to be called once, but got 0 times". - (b) ✅ fixed (CP-210, a regression from CP-205 !484): a headered
/startdelivered after an eviction replay's run ended. The parent sends its first spawn's/start, is evicted before the answer, and its replay's probe finds no child (404), so the replay starts it plainly; if the original request is delivered only after that run ended, its fresh-session ask wiped it.planRespawnnow answers the busy 409 (duplicate_consult, never a wipe) when the child's current run already serves the start's consult and the start names no newer parent-ref generation than that consult started at. The child records the generation with the consult (parentRefGeneration.ofConsult, written in the claim block with a parent-bound run-start commit), so a later run of the consult that no parent-ref start opens (a client-tool or approval resume) keeps it; a consult never recorded (a companion continue's own) is bounded by the DO-wide record. A newer generation of the same consult (CP-161's reset replay) is a new parent-ref incarnation and wipes as before. Miniflare reds "CP-210 (FU-DO-COMPANION-FRESH-START-DUPLICATE (b)): a fresh-session start repeating the consult its child already served is refused busy, never wiped" and "…: a reset of the same consult at a newer generation is not a duplicate: the child starts fresh".
FU-REVIEW-CONVERGENCE-FOUR-PASS: deferred items from the (two) four-reviewer convergence passes — open (LOW-MED)
TWO four-independent-reviewer passes over the convergence branch. Round 1 (correctness / replay-safety / cloudflare / test-coverage) → fixed 84ad171 (Temporal blocking inline-output R2-I1; CFW waitForResult dedup R3-C1; CF-DO sendMessage terminated guard R3-I6; timeout comment R1-C1) + 54ae452 (R1-C2 paused-child terminate flag ✅; R3-I5 registry-cast removed via the new CompanionParentConfig ✅; R3-I3 D1 V14 comment ✅). Round 2 (concurrency / type-design / fix-verification / store-layer) → review #3 verified all round-1 fixes correct; fixed 63fb801 (streamManager optional → removed 3 as unknown as StreamManager casts; mode union → removed a cast; UpdateSubSessionRef.status += paused_awaiting_client; dropped unused CompanionParentConfig.name; RoutingMockLLMAdapter opts spread; completionDelivered doc fix; new cross-store contract test for status-only-preserves-completionDelivered). Deferred-items round (the user's "fix deferred items"): a7db7cd (R1-C3 dispatcher Zod arg-validation ✅; terminated-ref notifier skip-early ✅), 710213e (R2-C-1 ✅ — extractRootErrorMessage cause-unwrap + C8 assert moved after createSession so a non-blocking workspace child's failure is persisted + delivered), + the parallel-auto-name spawn DOCUMENTED (spawn tool name description + dispatcher comment steer the LLM to name concurrent same-type children — see below). Round 3 (concurrency / edge-cases / parity-audit / comment-accuracy) → review #1 verified the deferred-items fixes correct; fixed 7d9f229: DIV-1 (CRITICAL — waitForResult double-delivery on JS+Temporal: core now sets completionDelivered on the inline-delivered ref) ✅; C4 canonical signal (sendMessage guard now checks non-empty pendingClientToolCalls, not status==='paused') ✅; extractRootErrorMessage hostile-getter guard ✅; Zod hardening (name .min(1), timeout .positive()) ✅; a batch of comment-rot (DBOS waitForChildTerminal/registry/deps JSDoc, Temporal dispatch header, drifted line refs, DIV-4 DBOS no-resumeMetadata documented as intentional) ✅. Accepted/architectural (NOT bugs): DIV-2 — CF-DO listChildren reads refs directly without resolveAndSyncSubSessionRefs (a DO can't loadState a sibling DO), so a just-completed non-blocking child shows running until the parent's next checkCompletedChildren turn; DIV-3 CF-DO sendMessage C4 is best-effort under a transient /status failure; DIV-5 CFW 'spawning' interim status is never surfaced to the LLM. Round 4 (verify-fixes / resource-DoS / public-API-semver / test-quality) → review #1 verified all round-3 fixes correct; fixed e852b08: R4-M1 (orphan ref on a failed spawn — core now compensates the ref to 'failed' so the name stays re-spawnable, mirroring DBOS) ✅; R4-I3 (terminate sets completionDelivered:true so terminated refs aren't re-scanned each turn) ✅; R4#4-I1 (parity-harness defensive store reset between scenarios) ✅. The public-API/semver audit (R4#3) is captured in docs/superpowers/specs/2026-06-06-phase8-changeset-semver-analysis.md (core needs a MAJOR bump). Tracked (disproportionate / known): waitForResult unbounded wait on Temporal (activity-proxy + heartbeat tuning, or a Zod timeout.max() cap); per-turn O(N) notifier loadState (scales with RUNNING children — a store-side non-delivered index would bound it); RoutingMockLLMAdapter unbounded key Map (test-only, negligible). children; store-atomic add-if-absent remains the proper fix).
Still deferred (cosmetic / disproportionate only):
- R1-minor: stale
completedAton re-spawn. The re-spawn path resets status→running+completionDelivered:falsebut leaves the prior run'scompletedAt(UpdateSubSessionRef can't null a field). Add a clear mechanism or set it explicitly. Data-quality only (never read for control flow). - R3-I2 (cosmetic):
runtimeName: 'CloudFlare Workflows'uses non-canonical brand casing ('Cloudflare'). Left as-is to avoid churning the sharedassertRuntimeSupportsWorkspacesmessage contract + changelog references. - R4 test gaps: the C6 core call-order assertion ✅ (now has teeth via the CTD-stub
opLogtest + the Temporal recording-store test) and the harness blocking-FAILURE scenario ✅ (blocking-child-fails) are DONE under FU-TEST-AUDIT-PERSISTENT-SUBAGENTS. Remaining: non-blocking LIVE completion-delivery e2e (only unit + pre-seed today).
Already tracked elsewhere: R2-I2 = FU-DBOS-BLOCKING-SPAWN-SEMANTICS; R3-I4 = FU-CFW-COMPLETION-AT-LEAST-ONCE.
FU-TEST-AUDIT-PERSISTENT-SUBAGENTS: comprehensive test-coverage audit of the persistent-subagent feature — high-leverage deterministic gaps FILLED; timing-sensitive integ gaps tracked (LOW-MED)
A 6-slice deep audit (core-unit / runtime-adapters / integ-cross-runtime / e2e / stores / parity-harness) of the entire persistent-subagent test suite. Established that happy paths + the marquee bugs (C1–C8, DIV-1, R4-M1, terminate- truth) were well covered, but real gaps clustered in core-unit error branches, zero-coverage helpers, and a Temporal-adapter blind spot. Filled (committed):
companion/helpers.test.ts(NEW, 14):generateAutoNameMAX+1/gaps/prefix- isolation/substring-prefix/ephemeral-skip;resolveChildNameauto/provided/ active+interrupted-dup-throw/terminal-reuse; ACTIVE/TERMINAL set membership +paused_awaiting_clientin neither.companion-tool-dispatch.test.ts(+21): spawn error branches, sendMessage branches incl. C6 append-before-flag ORDERING (newopLogstub gives it real teeth — the prior test asserted both writes but not order), listChildren/ getChildStatus/waitForResult/terminateChild branches, arg-validation edges.dispatch-companion.test.ts(NEW, 20, runtime-temporal): the CRITICAL gap — Temporal had ZERO directexecuteCompanionToolCallunit tests vs CFW's 14. Mirrors CFW + pins Temporal-adapter seams (spawnMetadata/resumeMetadata lifting, error mapping, C6 ordering via recording-store, DIV-1 flag).- cross-store contract (
sub-session-operations.ts):paused_awaiting_clientround-trip — runs on all 5 stores (verified memory + real PG + Redis). - parity harness:
blocking-child-failsscenario (JS + Temporal) — child runs then errors; parent completes, ref recordedfailed. session-types.test.ts(+5):assertPersistentSubSessionRef(was zero).routing-mock-adapter.test.ts(+2):gracefulExhaust:falseregistered-key throw vs unregistered-key always-graceful.run-loop-persistent.test.ts: replaced theexpect(true).toBe(true)placebo companion-guard test with a real one (inject a CompanionTool into an agent with no persistentAgents → hits + asserts the guard).persistent-subagent-js.integ.test.ts: the duplicate-active-name e2e was would-pass-if-reverted (conditionalif (isError)); now documents it as a timing-dependent SMOKE test (deterministic teeth live in the 3 unit tests above) + added a CROSS-BRANCH singleton-name invariant (exactly one terminalworkerref) that breaks on a bad revert regardless of timing.
Review pass over the audit additions (4 independent reviewers: test-teeth / fixture-fidelity / adjacent-coverage / cross-runtime-consistency). ZERO critical; all actionable items fixed: core handleSendMessage terminal-child-with-stale- pending-map fall-through (the !childIsTerminal clause — delivered:false, not a C4 throw); Temporal hook-firing (beforeTool/afterTool success+failure — the file claimed to pin hooks but every agent had none) + unresolvable-parent-agentType branch; CFW mock updateSubSessionRef narrowed to the real-store updatable-field whitelist (was a permissive full-spread); generateAutoName descending-order (counter > maxCounter false arm); DBOS failed-child waitForResult inline branch; fixed an overstated e2e comment (the singleton toHaveLength(1) is structural-by- construction, NOT a guard-revert detector) + a wrong helper test title.
Remaining (timing-sensitive / cross-runtime integ — tracked, not blocking): JS live non-blocking delivery between turns + C1 CAS-loser + C4 e2e + waitForResult timeout/abort; Temporal sendMessage-to-interrupted + terminate- running + live non-blocking (see FU-TEMPORAL-NOTIFIER-LIVE-NONBLOCKING-RACE) + dedup-across-2-turns + non-blocking workspace-failure integ; DBOS notifier- delivers + C5-replay + @DBOS.step replay-once + blocking-inline-failure (see FU-DBOS-COMPANION-INTEG-GAPS); CFW at-least-once (FU-CFW-COMPLETION-AT-LEAST- ONCE) + waitForResult handler branches; CF-DO terminate-running + C4 fail-open; Postgres V11 column-on-populated-table. A fully-deterministic duplicate-active- name e2e needs a child-side sync barrier (gated tool) — disproportionate vs the unit teeth that already exist.
FU-PARITY-HARNESS-TABLE-DRIVEN: table-driven cross-runtime persistent-subagent harness — JS+Temporal SHIPPED; DBOS/CF drivers remaining (LOW-MED)
The table-driven harness is built: packages/e2e/src/__tests__/persistent-subagent-parity-harness.integ.test.ts. A shared scenario library (runtime-agnostic: per-agentType RoutingMockLLMAdapter scripts + a ref-based assertion) runs against a thin per-runtime DRIVER. Drivers implemented: JS (in-process JSAgentExecutor) + Temporal (docker runner). 4 deterministic blocking/sequential scenarios (blocking-spawn-completes, two-blocking-spawns-auto-named, terminate-truth-preserves-completed-ref, wait-for-result-after-blocking-child) × 2 drivers = 8 tests, green. Assertions normalize on getSubSessionRefs + child loadState (every runtime persists these; Temporal/DBOS don't persist companion ToolResultMessages). The DRIVERS array makes adding a driver one line.
Remaining:
- DBOS driver — blocked on FU-DBOS-BLOCKING-SPAWN-SEMANTICS. Every current scenario is blocking-spawn-based; DBOS blocking-spawn polls the child SESSION status until idle-timeout, so it would hang/mis-report. Add the DBOS driver once that lands (or add a non-blocking + completion-notifier scenario set that DBOS handles cleanly).
- CF-DO / CFW drivers — run in the separate workerd vitest pool, so they need adapter wrappers (can't share the Node
describeloop directly). - More scenarios to fold in: non-blocking spawn + completion-notifier delivery (needs the timing handled — see the 7b pre-seed pattern), sendMessage happy-path, getChildStatus, re-spawn (skip on DBOS — idempotency), and stream-event parity for
companion__spawnAgent(tool_start/tool_end) on Temporal/DBOS (JS has it atpersistent-subagent-parity.integ.test.ts:1398). Design:docs/superpowers/specs/2026-06-05-persistent-subagents-parity-harness-design.md.
FU-DBOS-ROUTING-LLM-CONVERGE: DBOS persistent integ uses a local routing LLM instead of the shared core one — open (LOW, convergence)
Phase 7a added a shared RoutingMockLLMAdapter to core (session/agent-routing mock LLM). persistent-subagent-dbos.integ.test.ts still uses a LOCAL createRoutingLLM (lines ~85-140) with its own AgentScript format, predating the shared adapter. Converging it to RoutingMockLLMAdapter (route by agentType, setResponses(key, MockResponse[])) removes the duplication and lets the DBOS suite share the same exhaust-isolation semantics. Deferred because it's a sizable script-format rewrite on the finicky DBOS integ env (PG :5434 + Redis) with no behavior change — the local adapter works. The DBOS non-blocking-spawn skip is a separate concurrent-DBOS-workflow constraint, not a queue race, so it is NOT unblocked by the routing adapter alone (unlike Temporal, where Phase 7b un-skipped the live non-blocking spawn via the shared adapter — persistent-subagent-temporal-routing.integ.test.ts).
FU-TEMPORAL-HASPERSISTENT-PER-ENTRY-ACTIVITY: notifier gate costs one extra activity per workflow entry — open (LOW, perf)
TR-b gates the per-turn completion notifier on agentDeclaresPersistentChildren, resolved via an ACTIVITY once at each workflow entry (fresh + resume) so the gate survives suspend/resume without plumbing a flag through AgentWorkflowInput + both child-spawn sites. Cost: one extra activity round-trip per workflow entry for ALL agents, including non-persistent ones (DBOS reads agentSerialized.persistentAgents?.length inline — no activity — because it embeds the serialized agent in the workflow input). Optimization: thread a hasPersistentAgents boolean onto AgentWorkflowInput (set by the executor for the parent and by dispatchCompanionTools for children — the child's config is reachable there via the registry), eliminating the per-entry activity. Behavior is already correct; this is purely a round-trip saving.
FU-DBOS-DETERMINISM-SCAN-COMPANION: the workflow-body Date.now() scan doesn't follow into core companion handlers — open (LOW)
From the Phase 5 review. The routed read/terminate handlers run core's executeCompanionToolDispatch in the DBOS workflow body, and core uses bare Date.now() in resolveAndSyncSubSessionRefs / handleTerminateChild (completedAt). The no-date-now-in-workflow-body.test.ts scan only checks the three local workflow files, so it no longer enforces the invariant for code that genuinely runs in the body. Runtime impact is BENIGN (those timestamps are one-time-idempotent — guarded by terminal-status skips — and never feed control flow), so it's a test-coverage gap, not a behavioral bug. Fix: extend the scan to follow imported companion handlers, or add a core lint rule for dispatcher timestamp helpers.
FU-DBOS-SPAWN-REPLAY-DUPLICATE: auto-named spawn can duplicate a child on workflow body-replay — ✅ FIXED (one residual)
From the Phase 5 review (flagged as a pre-existing property, NOT a Phase 5 regression — the old fire-and-forget spawn shared it). Companion handlers run in the DBOS workflow body and re-run on recovery (and inside CFW's retryable step.do). For an AUTO-named spawn, a body-replay re-ran resolveChildName → generateAutoName which, seeing the prior ref still at running, returned the NEXT name (researcher-2) and addSubSessionRefs+startPersistentWorkflow a brand-new child; an EXPLICIT name threw already running. Fixed: spawn now keys idempotency on the deterministic toolCall.id. At handler entry, if a SubSessionRef already carries this toolCall.id, its name is reused so the derived child id is identical across replay and addSubSessionRefs / spawnPersistentChild are idempotent. First-spawn refs store the real toolCall.id as parentToolCallId. Applied to core handleSpawnAgent (JS / Temporal) and the DBOS bespoke handler. CFW Workflows and CF-DO also route through core handleSpawnAgent but had been passing a CONSTANT 'companion-spawnAgent' id — surfaced in code review: with the new id-keyed dedup that constant would collide distinct spawns onto one child, so both call sites (steps.ts handleSpawnAgent, do/persistent-companion-tools.ts) now thread the real per-call toolCall.id. All 5 runtimes covered. Unit-tested on core, DBOS, and the CFW path (distinct-spawns-no-collision + same-id-replay). Residual (LOW): an explicit-name RE-spawn (a new tool call after a terminal run) resets the existing ref via updateSubSessionRef, which doesn't carry parentToolCallId, so a replay of the RE-spawn degrades to a deterministic no-dup error (not a duplicate). Fully closing it needs parentToolCallId in UpdateSubSessionRef across all 5 stores — deferred (explicit-name re-spawn is rarer).
FU-COMPANION-WAIT-STOP-ANSWER-PARITY: a stop that cuts a parent's companion__waitForResult may answer the wait differently on the CF DO and on core's poll — open (LOW; owner: WA)
Filed by: the conformance owner with WA's Save-First cells (RM-120 / RM-121 records). Related: CP-277 (a cut wait is not completed work), FU-COMPANION-WAIT-FAILED-MESSAGE, FU-181-RUN-STOPPED-TEXT-NEVER-STARTED.
- What is known: the CF DO's wait tool is its own (
buildWaitForResultTool,packages/runtime-cloudflare/src/do/persistent-companion-tools.ts). A stop of the parent while it waits pairs the cut waitrun_stopped, with the generic "the tool may have run" text. WA'sstop.mid-tool-keeps-finished-beside-wait-hostpasses on cf-do 3 of 3 in slot-c3 and slot-q1 (its observe line logs the cut wait's row). - What is not known: core's poll (
companion-tool-dispatch.ts, the wait loop runtime-js uses) exits a cancelled wait by throwing a plainError('companion__waitForResult cancelled'). Whether that reaches the transcript as a failed call or is pairedrun_stoppedby the stop is read, never observed: the cell's JS contrast host (js-chat-host) wires no tool gate, so its cell is a declared gap that names this record. - To close: an executor-tier sibling of the beside-wait cell on the runtime-js lanes, which holds the companion with a tool gate, then either aligns core's answer with the DO's (
run_stopped) or records the difference as a finding. The text half ("the tool may have run" for a wait that provably ended) is FU-181-RUN-STOPPED-TEXT-NEVER-STARTED's.
FU-COMPANION-WAIT-FAILED-MESSAGE: a failed waitForResult answer carries no reason, on every runtime — open (LOW)
Surfaced by the FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES review (concern 1; orchestrator ruling: a cross-runtime FU, not this MR). { name, status: 'failed' } is the model's only signal of a child failure, and no companion tool exposes why (getChildStatus returns lastOutput only for completed). Action: fill the existing optional failed.message of CompanionWaitResultSchema with the child's saved error message (length-bounded) on EVERY runtime: core (loadState(child).errorDetail), the DO (the /status body), DBOS (loadState), CF Workflows (the completion event's error.message, else loadState), with a schema-parse test per runtime.
Status (2026-11-19): the CF DO half is done: since 2e5afe376b (DO companion lifecycle MR A) a failed child's committed word answers message with its error detail's message, unbounded and unsanitized (its ended-ref fallback still answers none). Open for core (runtime-js and Temporal), CF Workflows (both its completion-event and read-path answers) and DBOS, each still answering a failed child with no reason (traced at origin/main 4f0e1d1807). The fix also length-bounds the DO's message, as the Action asks.
FU-CFW-WAITFORRESULT-STALE-BUFFERED-EVENT: a CFW waitForResult answered an earlier run's buffered completion as the current run's — done
Closed by: FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES fix rounds 9-10 (review r8 m3, r9 I-1; conformance CP-126, cell companion.wait-not-answered-by-stale-completion|cfw-workflows-d1|executor). A companion's completion event (sub-agent-complete:<child>) that no wait consumed stays buffered in the parent instance: run R1 completed while no wait was parked (its outcome then reached the parent through the ref sync and the pre-park step), or a store read answered it. A later companion__sendMessage started R2 (or a re-spawn started a new incarnation), and the next companion__waitForResult dequeued R1's event and answered R1's output as R2's (a silent wrong answer).
- Fix (fail closed, round 10): the parent instance keeps
companionConsults(child session id → the call id of the consult it started LAST for that child: spawn, continue, resume; set besidecompanionInstances, rebuilt identically on replay). A child's events reach only the instance that started its run (parentWorkflowId), so that is every consult whose events the wait can receive.waitForResultaccepts ONLY an event naming it, and drops an event of another consult, one naming no consult (warned once per park), and any event when the instance started no consult for the child; after a drop the park's next step is the store read (readCommittedChildOutcome, served by that consult), then it listens again. It also reads the store after every rejected slice. - Round 9's current-run read (
readCurrentRunConsult,<prefix>-event-run-<n>) is removed: it failed open (a failed read, or a re-spawned child whose new run had not committed its start, accepted the stale event). - Round 11 (review r10): the map is also set at the CP-82 respawn of a persistent child, and pinned at the
sendMessagecontinue / resume sites by tests (mutation-proved). EVERY park now drops an event naming no consult (one warning per park): the blocking spawn and continuation keep only their own call's events, the ephemeral park its sub-agent call's. The second witnesscompanion.wait-after-spawn-continue-not-stale(a non-blockingspawnAgentcontinue, then a same-step wait): the continue site writes the map before the wait parks, so R1's buffered event is dropped; and the pre-park step and the deadline answer accept a terminal ref only when the child's current run'sparent.toolCallIdis the map's consult (terminalRefServesConsult), so R1's terminal ref is never R2's answer. - Round 12 (review r11 I-1): a detached continue (
sendMessage, non-blockingspawnAgent) or a resume whose entry definitely never starts (child_busy,child_terminated,child_entry_failed) restores the previous consult (a replay restores identically), so a later wait answers the child's real outcome instead oftimeout/runningfor the rest of the instance; an unconfirmed entry keeps the new consult, and a timeout reports the child's TRUE persisted status. The non-blockingspawnAgentcontinue awaits its entry (FU-CFW-DETACHED-SPAWN-CONTINUE-LOST-ENTRY, done). - Tests (
workflow.test.ts"round 9 m3", "round 10 …";cfw-park-replay.test.ts"round 10 (CP-126)"): the witness; the re-spawn and read-blip shapes; the newest-consult overwrite; fail-closed (a) no consult named, (b) a failed store read, (c) no consult started here; a replay that rebuilds the map mid-park; foreign events on the blocking spawn and the ephemeral park.
FU-CFW-BLOCKING-SPAWN-CONTINUE-RESTORE: a blocking spawnAgent continue whose entry never starts leaves its consult in the map — open (LOW)
Surfaced by: FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES review r12 m-1 (pre-existing since round 10; rare).
- Where (
packages/runtime-cloudflare/src/workflow.tsat a0743d559e): the continue branch writescompanionConsults[childSessionId] = companionCall.id(:4426) before theif (!detached)split (:4428); the blocking continue's park (:4428-4518) waits on the completion event and learns no entry outcome, so it never callsrestoreConsultUnlessStarted(only the detachedsendMessage/spawnAgentcontinue and the resume do). - Probe shape: C completed (R1,
spawn-1); a BLOCKINGcompanion__spawnAgent(C)continuebc-2has its entry refused or failed before its run start; its park waits the fullsubAgentEventTimeouton a closed instance and answersTimeout waiting for continued child agent; the map still namesbc-2, so every latercompanion__waitForResult(C)in this instance answers{ status: 'timeout', currentStatus: 'completed' }after its full bound (the detached case answers R1). The status is true; the answer is a non-answer. - Fix direction: let the blocking continue's park learn its entry outcome (e.g. its consult treats "the continuation instance closed with no run" as a definite non-delivery), answer it typed, and restore
priorConsult.
FU-COMPANION-RESPAWN-AFTER-UNSTARTED-NO-USAGE-POINTER: a respawn over an I-C10 failed ref never records the companion's usage pointer — ✅ fixed by CP-212 (WE-2)
Surfaced by: CP-161 review 2 (m5). A first spawn that did not start writes the ref failed and no usage pointer. The model's next spawn of that name is a respawn (isFirstSpawn false in core, recordUsage: false on CF Workflows), so the child's usage is never discoverable from the parent's rollup. Pre-existing on every runtime. Fix direction: record the pointer for the first STARTED entry of a subSessionId (e.g. key it on "no pointer yet" rather than "no ref yet").
Fixed (CP-212): the pointer is owed by every spawn entry that starts the child session (core's dispatcher gate and CF Workflows' spawnEntry.recordUsage), kept to one per child session by the helper. Reds: core units, the runtime-js companion-usage-scoping integ and the CF Workflows cfw-persistent.wf-noiso case "CP-212: a respawn after a first spawn that never started keeps the companion's cost discoverable from the parent, with one pointer".
FU-DO-RESET-BEFORE-CONSULT-LATE-NOTICE: on the CF DO a late notice of the previous consult can land between a respawn's reset and its new consult — open (LOW, owner: the WA bucket)
Surfaced by: CP-161 fix round 2 (rule 4 on the DO). The reset (core) lands before the DO dep's beginChildConsult. The DO's outcome recording is consult-scoped (recordChildOutcome), so a notice of the previous consult that arrives in that gap still matches and writes the previous outcome's status on the new generation's live ref; the next /start's recordChildRunStarted then reopens it again (a second bump). Reachable only when the previous consult's run was still ending when its ref went terminal (a terminate). The DO's reopen also does not stamp the call on the ref (its readers use the consult row). Fix direction: record the generation a consult began at on the consult row (a DO-SQLite column) and scope recordChildOutcome to it.
Narrowed by CP-161 fix round 4 (m-E): a REPLAY of a respawn's reset (core's respawn + isReplay: the reset landed, the first attempt died) used to attach to the child DO on ANY /status answer, so a child still holding the OLD consult's session (its first attempt's /start never landed) was attached to, and the park could answer the old consult's outcome as the respawn's. It now attaches only when the child's /status names this consult (parentToolCallId); otherwise it begins the consult and starts fresh with the respawn header (persistent-companion-tools.ts, replayProbeFindsNoChild). The late-notice gap above remains.
FU-DO-STATE-STORE-KV-REQUIRED: a DOStateStore built without kv has no ref generation floor — open (LOW, owner: the WA bucket)
Surfaced by: !476's independent review (2026-10-23), below 50% confidence. DOStateStoreOptions.kv is optional (runtime-cloudflare/src/do/do-state-store.ts:900). Without it the parent-side sub-session ref generation floor is 0 (subSessionRefClaimGeneration, :1097) and a respawn wipe raises no floor (only an error log, :3948), so such a store reproduces the CP-161 Greptile P1 defect (after a wipe, the next claim of a child that recorded a higher generation is refused stale_parent_ref_generation). Unreachable through DurableObjectAgentBase, which always passes ctx.storage.kv (durable-object-agent-base.ts:1314); only a hand-built store can hit it. Fix direction: make DOStateStoreOptions.kv required, or refuse a ref wipe (a typed error) when it is absent.
FU-DO-TERMINATE-INTERRUPTED-RESUME-RACE: on the CF DO a terminate racing a same-step sendMessage to an interrupted child can abort the resumed run and answer terminated: false — open (LOW, owner: the WA bucket)
Surfaced by: CP-161 review 6 (n-1). The DO terminate's stop dep (persistent-companion-tools.ts, stopPersistentChild) posts its stop only while the ref is non-terminal at the generation the terminate read (CP-161 fix 5, I-1). That is exact for a respawn, whose reset bumps the generation before beginChildConsult moves the consult row. A continue or resume, though, begins its consult at the OLD generation: the reopen bump is recordChildRunStarted's (do-state-store.ts), after the child answered. A continue from completed is refused by the terminal check, but an interrupted ref is not. Take one step with terminateChild(x) and sendMessage(x), x interrupted at g:
- sendMessage begins c2 and posts
/resume. - The stop dep reads the row (c2) and the ref (
interrupted, g), and posts a stop naming c2. - The child accepts the
/resume, then records the stop on c2's run and aborts it. recordChildRunStartedreopens the ref torunningat g + 1 before the terminate's write.- The terminate's
terminatedat g is then stale, so it answersterminated: false, yet it aborted the resumed run.
The other order is the intended one: when the terminate's write lands first, recordChildRunStarted refuses (terminated), and stopping c2 is what prevents an orphan run. Persisted state stays truthful (the run's abort is recorded at g + 1 and notified); only the two tool answers disagree with what happened. runtime-js does not share this (runServesRef on target.consult). Fix direction: when the stop dep posted a stop for a consult other than target.consult, have the terminate's stale answer report terminated: true with the stopped run; or record the generation a consult began at on the consult row, as FU-DO-RESET-BEFORE-CONSULT-LATE-NOTICE's fix direction proposes, and scope the stop to it.
FU-DBOS-COMPANION-PARENT-RUN-FENCE: DBOS's bespoke companion spawn and sendMessage carry no parent-run fence — open (parked: DBOS)
Surfaced by: CP-161 review 5. DBOS's own handleSpawnAgent / handleSendMessage (runtime-dbos/src/steps/execute-companion-tool.ts) do not route through core's dispatcher, so CP-161 fix 4's assertParentRunOwned before every child mutation and the in-store parent-run fences (the reset's SubSessionRefExpect.parentRunId, the reopen's ResumedCompanionRef.expect.parentRunId) do not reach them; only the handler's top-of-call D2 recorded step guards them. A run superseded after that step can still reset, continue or resume a child. Fix direction: route both through core's dispatcher (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY), or pass parentRunId to their reset / reopen CASes and re-prove ownership (a recorded step) before each child mutation.
FU-SUPERSEDE-IN-TOOL-SIBLINGS-AFTER-TERMINAL-HOOK: a supersede raised inside a tool body leaves the step's parallel siblings running after the run's terminal hook — open (LOW, owner: the WA bucket)
Surfaced by: CP-161 review 6 (n-4). Since CP-161 fix 5 (I-5), a tool body that throws its own run's RunSupersededError ends the run superseded at once (runtime-js's regular-tool and companion catches, so the CF DO too). The step's parallel calls run under Promise.all (step-iterator.ts), which fails fast: siblings whose bodies were already dispatched keep running after stopAsSuperseded returns and onAgentSuperseded fires. Their afterTool / onMessage hooks fire after the run's terminal hook, and their tool_end frames reach the stream until the newer run claims it (then the run-id fence drops them). Nothing persists: the step commit never runs, and later sequential calls are refused by D2 at their start. Documented in the hooks guide. Fix direction: on a supersede, have the step iterator await Promise.allSettled(toolPromises) before it returns, so no sibling hook fires after the terminal hook.
FU-TEMPORAL-COMPANION-RUN-PARENT: Temporal's companion child runs record no StartRun.parent (and its reopen no parent-run fence) — open (parked: Temporal)
Surfaced by: CP-161 review 4 (m-B). Temporal's child runs record no parent (activities.ts, the companion continuation / resume run-start passes none), so readers of the run's consult (core runServesRef, isOwnResetReplay, a terminate's target.consult) cannot tell a Temporal child run of another call from this call's. Fix round 4 made isOwnResetReplay read a parentless run as possibly this call's (no reset-replay delete of a live child's session; the replay attaches), and the others adopt / stop a parentless run as before. Its reopen also passes no ResumedCompanionRef.expect.parentRunId (the in-store parent-run fence, I-A). Fix direction: pass StartRun.parent (the consult) and expect.parentRunId (the parent run) in the Temporal companion child's run-start commit, as runtime-js and CF Workflows do.
FU-SPAWNER-STRANDS-CLAIM-OTHER-RUNTIMES: on runtime-js, Temporal and DBOS a spawner that dies between its claim and its child's start strands the name running — open (runtime-js: LOW, owner: the WA bucket; Temporal / DBOS parked)
Surfaced by: CP-161 review 7 (I-1 (a)), narrowed by fix 9. On CF Workflows the stranded claim is now reclaimed by the next spawn of the name (core isStrandedSpawnRef + the CFW reclaimStrandedSpawnRef dep; the joint model's "reclaim" row; race table cp161-reclaim-race-table.md). The CF DO already frees it: its pre-dispatch reconcile re-probes an unconfirmed consult and fences the child (requestParentRefFence) before abandonUnconfirmedConsult. The other runtimes do not (race table §6):
- runtime-js (catalogue CP-190; reachable: a process crash between the claim and the child's run-start inside
startPersistentChildEntry; a store ERROR there is now released by the call itself, CP-161 final I-2). OnlywaitForResult's process-local orphan detection frees it;spawnAgentanswers "already running". Fix direction: supplyreclaimStrandedSpawnRefthere, with (ii) = the step's call ids (JS runs a step's tools in parallel) and (i) adapted (a JS step's assistant row commits with its results, so the claiming call is in no log), and give JS child startsparentRunFencefirst. - Temporal (catalogue CP-191; parked; reachable narrowly: the parent workflow CLOSED between the dispatch activity that wrote the ref and
wf.startChild). Child runs record no parent (FU-TEMPORAL-COMPANION-RUN-PARENT), so (iii) cannot be decided strictly, and the child carries noparentRunFence. - DBOS (catalogue CP-192; parked): recovery re-runs the spawn; the residual is an external
cancelWorkflowor recovery-attempt exhaustion between the ref insert and the child's start. DBOS is not on the CP-161 CAS model (FU-TEMPORAL-DBOS-COMPANION-CLAIM-CALLERS).
FU-CFW-RESPAWN-REFUSED-CHILD-SILENT: on CF Workflows a respawn whose child is refused after its parent turn ended is never announced — closed (by design, ruling D1)
Surfaced by: CP-161 final re-review (n-3). A CF Workflows respawn whose entry poll timed out answers child_start_unconfirmed; when the parent's turn then ends, the child's parentRunFence refuses its run-start (parent_run_superseded), and the refused child releases the parent's reset with the respawn's EXACT restore (releaseSpawnClaimForRefusedChild), so the model only ever heard "unconfirmed" for that respawn. Closed by the orchestrator's ruling D1 (2026-10-20): a refused child did no work, so no next-turn report is owed; a first spawn's refused child is now released failed, delivered, too (logged companion.spawn.refused_child_report_dropped), and only a child that actually ran is reported. Since RM-69 such a spawn's tool_use may be in no committed row at all (the parent's step was lost), so a report would name a call the model never made.
FU-CFW-SPAWN-CLAIM-CALL-ID-TYPE: a CF Workflows child with parentSpawnClaim but no parentToolCallId only logs and leaves the claim running — open (LOW, owner: the WA bucket)
Surfaced by: !476's independent review (2026-10-23), below 50% confidence. A spawn child refused by its parent-run fence releases the parent's claim only when it knows its own call: with parentSpawnClaim set and parentToolCallId absent it logs cf_workflow.companion.refused_child_release_no_call and releases nothing (runtime-cloudflare/src/workflow.ts:6097-6110), so the claim stays running. Unreachable today: the companion create step sets both (parentToolCallId: companionCall.id and parentSpawnClaim, workflow.ts:3120-3131). Fix direction: make it unrepresentable: one parentSpawnClaim: { refGeneration, entryKind, parentToolCallId } field, or a Zod / type tightening of the child's input (dto.ts, parentSpawnClaim) that requires the call id with the claim.
FU-CFW-STRANDED-CLAIM-OTHER-READERS: on CF Workflows only a spawnAgent that meets the name reclaims a stranded claim — open (LOW, owner: the WA bucket)
Surfaced by: CP-161 fix 9's race table (R4). The reclaim runs where a spawn meets another call's live claim of its name. companion__waitForResult, companion__sendMessage and companion__getChildStatus on a stranded name still answer from a running ref, and an auto-named strand is never met again (generateAutoName skips used names), so it stays running in listChildren (cosmetic). Fix direction: apply the same predicate (isStrandedSpawnRef, with the run's own call ids) in the CFW lazy sync's reader (checkCompletedChildren), releasing through the same fenced CAS. A stranded respawn reset is reclaimed failed, delivered either way (its exact restore is gone; a held prior is still delivered by the held-prior pass).
FU-TEMPORAL-DBOS-COMPANION-CLAIM-CALLERS: Temporal's and DBOS's companion spawns do not act on the claim's answer — open (parked: Temporal / DBOS)
Surfaced by: CP-161 fix round 2. Their stores implement the joint ref model's claim / reset / release contract (shared stores), and Temporal's activity passes a lost claim's child_busy through as the tool result. But DBOS's bespoke spawn handler (runtime-dbos/src/steps/execute-companion-tool.ts) ignores addSubSessionRefs' answer and resets unconditionally, and neither runtime scopes its outcome writes by generation. Fix direction: route both through core's dispatcher (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY), or apply the claim / reset / release CAS at their own sites.
FU-REDIS-EMPTY-CALL-ID-ANY: Redis reads an empty expected call id as "any call" where the other stores compare strictly — open (LOW, owner: the WA bucket)
Surfaced by: !476's independent review (2026-10-23), below 50% confidence. updateSubSessionRef's call-scoped expect (CP-161, the late release) passes update.expect?.parentToolCallId ?? '' to the Lua script, which reads '' as "any call" (store-redis/src/redis-state.ts:4797-4799). So an EMPTY expect.parentToolCallId matches every ref on Redis, while memory, Postgres, D1 and DO-SQLite compare it strictly. Tool-call ids are never empty, so this is unreachable, but it is a contract divergence. Fix direction: pass "no expect" and "expect the empty id" apart (a separate flag ARGV) and compare strictly on Redis; add a contract case to core/src/testing/sub-session-ref-operations.ts (beside release.call-scoped): an empty expected call id matches no ref.
FU-CONF-CFW-HELD-PRIOR-FAULT-WITNESS: no conformance cell witnesses CF Workflows' held prior on a continue — open (LOW, owner: conformance owner)
MR B implements a held prior on CF Workflows too: the continue dep holds a completed, undelivered ref and the record steps deliver and release it after the commit. A normal non-blocking flow never reaches it there: a child's ref turns terminal only in the parent's next iteration's lazy sync (checkCompletedChildren), which delivers it as a notice in that same iteration. So companion.prior-result-parity-spawn / -send are declared gaps on cfw-workflows-d1 (it does not claim continue-holds-prior). The path is reachable through a fault window: a failure between the record's appendCommitted and its delivered mark, or a mark step that throws.
- The cell: fault the
completionDeliveredwrite once (a harness store fault), so the ref stayscompletedand undelivered; the next continue must carrypriorResultexactly once and no notice may repeat it.
FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY: a lost spawnAgent continue answers differently per runtime — open, narrowed to Temporal / DBOS (parked)
Narrowed by DO companion MR B (spec 2026-10-15 §3.2 / §3.7 "As built (MR B)", D4). runtime-js, the CF Durable Object and CF Workflows now share core's contract: CompanionEntryOutcome from every spawn / continue dep, CompanionSpawnResult (parseCompanionSpawnResult) to the model, and deliver-then-continue (priorResult, the held prior). Each item carried below is done there:
inputCommittedon the/subagentenvelope (RemoteAgentErrorResponse); the DO parent keeps the consult unconfirmed on any non-2xx carrying it (childEntryOutcome), and on a transport-throwing continue and a non-conforming 2xx too (the interim restores are gone; the unconfirmed consult is re-probed, never respawned).- A failed park carries
errorDetail(ChildTerminalResult.failed.errorDetail; the DO'srecordedWordOf; thefailedarm ofCompanionSpawnResult). - Core's park catch compensates nothing (a throw writes no ref); the only
failedcore writes is a first spawn's definite non-start, with the child's detail in the answer (I-C10); a respawn's non-start restores the ref exactly. - The non-replay 409 is
not_spawned/child_busy, a successful tool result (the advice text is gone). - A continue over an undelivered recorded outcome delivers it AND starts the new consult (
priorResulton every arm). - runtime-js's plain delivered mark is resolved: the claim is the store's
holdPriorResultCAS (one holder per outcome), so two parallel continues of one child cannot both claim it (the loser answerschild_busy); and it is now atomic with the parent's commit, not before it — the hold never marks delivered, the release after the step commit does (C16 resolved; the two crash windows are pinned on runtime-js, the DO and CF Workflows). - The spawn tools' output schemas describe every arm (
CompanionSpawnResultSchema; load-bearing on the DO). - A fresh call id over an outcome recorded BEFORE the recovered model call still runs the child twice; one recorded DURING it now delivers the outcome as
priorResultand runs the child again (no loss; both FU-DO-COMPANION-ORPHANED-CONSULT-ADOPTION, MR C).
C14's 404 window (accepted residual): the DO's re-probe takes a 404 "no session" as proof the consult never started; a /start whose child-side handling outlasts the 60 s bound before createSession answers 404 while still in flight, so the ref reads failed briefly, the respawn's /start gets 409 (the ref back to failed, the consult restored, a verified read owed), and the child's notice for the kept consult records the real outcome — converges, no loss, a wrong failed in between.
Core holds a respawn's prior only for a dispatcher-parked runtime (waitForChildTerminal supplied), because only the D4 answer can carry it; CF Workflows' spawn path is the handoff (FU-COMPANION-SPAWN-HANDOFF-TYPED), so a CFW RESPAWN never holds (its CONTINUE holds in steps.ts's continue dep).
Still open (parked, owner: the Temporal / DBOS parity MR): the FIRST spawn too (CP-161, 2026-10-19): runtime-js and CF Workflows now answer a first spawn / respawn only after the child's run-start commit (spawned means it committed; a non-start is a typed not_spawned, blocking or not, and a first spawn's child_busy writes no ref: core companionSpawnNonStartRefUpdate), but Temporal's first spawn answers before wf.startChild's child commits (runtime-temporal/src/workflow-dispatch.ts, a startChild throw only logged) and DBOS's bespoke handleSpawnAgent answers { name, status: 'spawned' } after enqueueing the child workflow, whose commit is its own later step (runtime-dbos/src/steps/execute-companion-tool.ts); fix direction: poll the child's run like CF Workflows (its deterministic run id) and answer through core companionStartOutcome / companionSpawnResultFromEntry. Temporal's deps answer { started: true } before the child starts and hold no prior (runtime-temporal/src/activities.ts, type-only in MR B), and its model sees core's spawn handoff; DBOS's bespoke spawn / sendMessage handlers answer { name, status: 'spawned' }, hold no prior and reset the ref's delivery over an undelivered outcome. The conformance cell companion.prior-result-parity is a declared gap on those lanes ("Temporal's and DBOS's continue paths neither hold nor deliver a prior result"; orchestrator ruling 2026-10-16). The original text follows.
Surfaced by: FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES review r12 m-3. CF Workflows now answers a non-blocking companion__spawnAgent continue with its entry outcome: { name, status: 'spawned', delivered: true } once the continuation's run committed, else { name, status: 'not_spawned', delivered: false, reason, errorDetail? } (workflow.ts at a0743d559e :4579-4595). The other runtimes, for the same agent definition (citations at a0743d559e):
- CF Durable Object: a failed continue is a tool ERROR, through core's continue compensation (
core/src/orchestration/companion-tool-dispatch.ts:654-682). - runtime-js: a lost reopen CAS answers
spawnedfor a continue that never ran (runtime-js/src/run-loop.ts:6593-6604: the CAS loss resolves the child's previous terminal result). - Temporal / DBOS:
spawnedbefore the continuation starts (no entry outcome).
Fix direction: every runtime reports a spawnAgent continue's entry outcome with CFW's shape (or a core-level contract for it), as CompanionDeliveryOutcome did for sendMessage (CP-105); related: FU-COMPANION-BUSY-DELIVERED-TEMPORAL-CFW.
Owner and plan: WA, DO companion MR B (spec 2026-10-15 §3.7, D4 (a): core CompanionEntryOutcome / CompanionSpawnResult on every spawn). Carried from DO companion MR A into MR B (each found in MR A's reviews, none fixed there; line numbers at the MR A head):
- A non-2xx after the child's run-start commit is answered as a definite non-start. The
/subagentprotocol envelope (core/src/types/remote-protocol.ts:266,RemoteAgentErrorResponse) carries noinputCommitted, so a typed refusal the child answers AFTER its run-start commit (astate_commit_stopped, a transient failure after the commit) makes the parent restore the previous consult (restorePriorConsult,runtime-cloudflare/src/do/persistent-companion-tools.ts:815, called at:849,:1563) while a run serving the new consult exists; that run's notice is thenstaleand its answer can be lost. Fix: the envelope carriesinputCommitted, and the parent keeps the consult aschild_start_unconfirmed(spec §3.7 amendment, Task 9 review I1). MR A also restores on a transport-throwing continue and on a non-conforming/resume2xx (interim; the spec's end state keeps the consult unconfirmed and re-probes it). - A failed park drops its
errorDetail.ChildTerminalResult'sfailedarm has noerrorDetail(core/src/orchestration/companion-tool-dispatch.ts:341), and the DO park'sterminalOfanswers{ status: 'failed' }(persistent-companion-tools.ts:1933), so a blocking spawn'sfailedanswer carries no detail although the recorded outcome has one. MR B'sCompanionSpawnResultfailedarm carries it. - Core's park catch writes
failedon a parent store error.handleSpawnAgentcompensates the reffailed+ delivered whenwaitForChildTerminalthrows (companion-tool-dispatch.ts:776-797). Since MR A the DO park rethrows only a parent store error (a child read failure answersunreachable), so that write runs against the very store that failed, over a child that may be live (I-C10: only a first spawn's definite non-start writesfailed). - The non-replay 409 advice. A non-replay spawn whose
/startgets 409ALREADY_RUNNINGthrows a tool error telling the model towaitForResult/sendMessage/terminateChild(persistent-companion-tools.ts:1055-1060). Under D4 it becomesnot_spawned/child_busy, a successful tool result, and the advice text goes. - A continue over an undelivered recorded outcome: deliver it, then continue (MR B,
priorResult). A same-namecompanion__spawnAgenton acompleted/failedchild, or acompanion__sendMessageto acompletedone, whose previous outcome is not delivered yet (the child ended after the parent's completion notifier ran, e.g. during the model call) starts the new consult and clears that outcome before the model saw it, on every runtime. On runtime-js the notifier marks delivered only at the top of the parent's next step, so a same-step re-consult (the critic loop) hits it every time. MR A tried a guard that answered the outcome and started nothing (I2); it was dropped (orchestrator ruling 2026-10-17: a same-turn re-consult answerednot_spawned, worse than before). MR B delivers the outcome AND starts the new consult in the same answer; DBOS's bespoke spawn / sendMessage handlers have the same gap. Also left for MR B:- A fresh call id over an outcome recorded BEFORE the recovered model call runs the child twice. The notifier injects the outcome (no loss), and the recovered parent's fresh-id spawn then continues the child, a second run of the instruction it already answered. FU-DO-COMPANION-ORPHANED-CONSULT-ADOPTION (MR C) closes it.
FU-COMPANION-SPAWN-HANDOFF-TYPED: CF Workflows and Temporal read core's spawn handoff by a cast, and Temporal hands it to the model — open
Surfaced by: DO companion MR B Task 3 (controller ruling (A), 2026-10-16). Core handleSpawnAgent answers on two shapes (CompanionSpawnAnswer, core/src/orchestration/companion-tool-dispatch.ts): the D4 CompanionSpawnResult on a runtime that supplies waitForChildTerminal (runtime-js, the CF DO), and the CompanionSpawnHandoff { name, childSessionId, agentType, mode, initialMessage } (no status) on a runtime that starts and awaits the child in its own workflow body. Two consumers read the handoff by a cast (citations at BASE f6cef81551):
- CF Workflows
runtime-cloudflare/src/steps.tsexecuteCompanionSpawn(:8029-8060) casts the dispatcher'stoolResultto the handoff and builds the model's result itself ({ name, status: 'spawning' | 'spawned' }), so its model never sees the handoff. But a D4 answer from core would readmeta.modeasundefined(a blocking spawn would answerspawnedand never park) with no compile error: the cast hides the shape. - Temporal
runtime-temporal/src/activities.ts(:9036-9055) casts the sametoolResultforspawnMetadataand leavesresult.resultas the dispatcher'stoolResult, whichworkflow-dispatch.ts:755-760hands to the model as the spawn's tool result (createToolResultMessage({ result: compResult.result })). Temporal's model therefore sees the handoff{ name, childSessionId, agentType, mode, initialMessage }, not a D4CompanionSpawnResult(spawned/not_spawned,priorResult), andtools/companion/spawn-agent.ts'soutputSchema(CompanionSpawnResultSchema) does not describe what Temporal returns. Not fixed in MR B (Temporal parked). - Neither path delivers a held prior:
priorResultis dropped on the handoff path.
Fix direction: when CF Workflows' / Temporal's parity lands (FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY), move both off the cast — narrow on 'status' in result (or parseCompanionSpawnResult), take the handoff's fields from the dep's own input (the no-op spawnPersistentChild captures its SpawnPersistentChildInput, as CFW's continueCapture does) — answer the model with the D4 shape on every runtime, then drop CompanionSpawnHandoff from core.
FU-CFW-WAITFORRESULT-CLOSED-CHILD-TIMEOUT: a CFW waitForResult on a child whose instance closed without notifying times out forever — open, narrowed (LOW, pre-existing)
Surfaced by the FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES review (M-7). The companion__waitForResult park never consults a closed child (consultClosedChildRun) on its timeout, as the blocking-spawn park does, so a child whose instance closed without its completion event answers { status: 'timeout', currentStatus: 'running' } on every repeat wait (and O1 tells the model to repeat).
- Narrowed by FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES fix round 8: the park now reads the child's persisted state after every floor and as its last step before the timeout answer (
readCommittedChildOutcome), so a child whose current run COMMITTED its outcome (its own K5) is answered with it at the deadline, notified or not (since round 10, after every slice that ends without an answer). - Still open: a child whose instance closed with NOTHING committed (a crash before its K5) is never reconciled by this park, so it still times out on every repeat wait. Action: run the closed-owner reconcile (
consultClosedChildRun, any consult of the child) at the park's deadline consult.
FU-DO-BLOCKING-SPAWN-UNREACHABLE-ZOMBIE: a DO blocking spawn / waitForResult marks an unreachable but live child failed + delivered — fixed in code (DO companion MR A); flips with the owner's ledger
Fixed in code by DO companion MR A (spec 2026-10-15, CP-125, user decision D1 (b)); not done until its witnesses (the CP-125 cells) are ledgered on main. After companionPollFailureWindowMs (30 s) of failed verified reads, the blocking spawn answers { name, status: 'unreachable', message } and companion__waitForResult answers { name, status: 'unreachable', currentStatus, message }, both successful tool results. Nothing is written and nothing is aborted: the ref stays running, the child runs on, and its own committed word (the child-settled notice → the parent's verified read → recordChildOutcome) or companion__terminateChild ends the ref. markChildFailedAndAbort is deleted, and a unit guard (do-companion-ref-writers.test.ts) fails if any src/do file other than do-state-store.ts calls .updateSubSessionRef( (I-C1). A parent store failure during the wait is rethrown, never answered unreachable (companion.child_read.store_failed). Residual: core's park catch still compensates the ref failed on such a store error (MR B, under FU-SPAWN-CONTINUE-ENTRY-OUTCOME-PARITY). Witnesses: the CP-125 conformance cells on the owner branch (companion.unreachable-child-blocking-host, -wait-host). The original text follows.
Owner: the WA DO companion group (with CP-39 / CP-40 / CP-23), next. Surfaced by the FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES parity check (fix round 4). Not fixed there (DO-only, pre-existing).
- The site (
packages/runtime-cloudflare/src/do/persistent-companion-tools.tsat 79ecf2a101):DEFAULT_POLL_FAILURE_WINDOW_MS = 30_000(:147);markChildFailedAndAbort(:201-229):updateSubSessionRef(… status: 'failed', completedAt, completionDelivered: true)at:207-212, then a best-effortstopChildDO(…, kind: 'abort')at:222-227;- called when every
/statuspoll failed for the window: the blocking-spawn parkpollChildToTerminal(:1578, the call at:1633) andcompanion__waitForResult(the call at:1436).
- The consequence: a child that is ALIVE but unreachable (a wedged isolate, a network partition) gets its ref
failed+ delivered; if the abort post also fails, the child keeps running and spending tokens, its eventual completion is never delivered (the ref says delivered), andcompanion__spawnAgentunder the same name deletes its session and starts fresh. The same zombie class C-1 fixed on CF Workflows. - Fix direction: never write a terminal ref over a run that may be live: record the stop first and write the ref only on its confirmed terminal (as CF Workflows' ephemeral park, C-1 option iii), else leave the ref and answer the parent's failure.
FU-STOP-DEADLINE-CFW: a stop deadline on CF Workflows, matching the DO's D-5 contract — open
Owner: the CFW workstream, after the DO. The DO has stopDeadlineMs (30 s default): a run that ignores its stop is ended and its drive detached. CF Workflows has none, so a CFW run that ignores its stop stays live until its next step boundary that reads the stop record, with NO bound. (Line numbers at 79ecf2a101.)
- Where CFW reads the stop today (
packages/runtime-cloudflare/src/): the body'sreadOwnStop(workflow.ts:1236) at the loop top (:2464), before sub-agent spawns (:3038), the cascade (:3595), the deferred finishWith (:2364), a resuspend (:2261) and after a suspension (:1220); the steps' state load (steps.ts:675-676,state.aborted) andexecuteAgentStep's pre-LLM check (steps.ts:1625-1626); the terminal commits (steps.ts:5245,:5446). - Long steps with no stop check inside them:
- The LLM call (
executeAgentStep,workflow.ts:2528→steps.ts:2162llmAdapter.generateStep(...)): the stop is read once before the call (steps.ts:1625), and NOabortSignalis passed togenerateStep(coreLLMGenerateInputhas one,core/src/llm/adapter.ts:300), so a long generation runs to its end. - Tool executions (
executeToolCall,steps.ts:3188; the drain's approved tools,workflow.ts:2885-2888): a 5 ssetIntervalreads the stop record and aborts the tool'sabortSignal(steps.ts:3251-3270), but this is cooperative only: a tool that ignores the signal runs to its end, and the step does not end early. - Hooks get a never-aborted signal (
steps.ts:758, "Cloudflare doesn't have cancellation signals"). - Waits: the ephemeral sub-agent park races an interrupt-event wait and reads the stop when woken (
workflow.ts:3283,:3301), so a stop that is signalled wakes it. The companion parks (blocking spawn:3999,companion__waitForResult:4160, blocking continuation:4414) wait only on the child's completion event and re-slice on every rejection (C-1 option B): they do NOT read the stop between slices, so a stop reaches them only at the park's bound (up tosubAgentEventTimeout). - The companion entry poll (
awaitCompanionEntryOutcome,workflow.tsat a0743d559e: the detachedsendMessagecontinue:4529, the non-blockingspawnAgentcontinue:4565, thesendMessageresume:4664) reads neither the stop record nor the interrupt event: a parent stopped while it polls a slow-booting child keeps polling for up tocompanionEntryOutcomeTimeoutMs(60 s) before its next stop-guarded commit or loop-top read ends the run (FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES review r12 m-2). Fix: read the stop record per probe, or race theinterruptevent. - Remote sub-agent calls (
executeRemoteSubAgentCall,steps.ts:4241) poll the stop at:4497.
- The LLM call (
- Fix direction:
- a recorded stop with a deadline (
StopRequest.deadlineAt, as the DO's) that a CFW reader, or the engine, can enforce by terminating the instance and settling the run through the closed-owner path'ssettle_stopped(framework_cancelled/interrupted), as the DO's deadline detaches its drive; - the per-step abort-signal plumbing (pass a stop-driven
abortSignaltogenerateStep, and to hooks); - the companion parks read the stop between slices (or race the interrupt event, as the ephemeral park does).
- a recorded stop with a deadline (
FU-CFW-WAITFORRESULT-STALE-PARK: CFW waitForResult can park forever on an already-completed child — open (LOW, pre-existing)
CFW's handleWaitForResult (steps.ts) is NOT routed through core (it must park in the workflow body) and reads getSubSessionRefs verbatim — no lazy ref-sync. If a non-blocking child already completed (its sub-agent-complete event already fired) but the parent ref is stale running, waitForResult returns a blocking descriptor and the body parks on an event that won't fire again → waits until timeout. Pre-existing (unchanged by Phase 4). Action: run the synchronous read-portion of handleWaitForResult through the same lazy-sync used by the other handlers (return the result inline when the child is already terminal, park only otherwise).
Persistent-cacheable-memory + cache-strategy follow-ups (post-MR !214)
Surfaced by the eight-review pass after MR !214 merged. Full triage: docs/superpowers/specs/2026-05-31-deferred-items-triage.md. Two of the four deferred items (multimodal recall query, unreachable format branch) were fixed in the memory-recall-followup-cleanups branch. The two below are dedicated efforts.
FU-MEM-FROMSTART-RETRY: retry({ mode: 'from_start' }) double-claims and is cross-runtime divergent — ✅ done
Status: done · Closed by: branch from-start-retry-fix
Resolution: Removed RetryOptions.mode and the 'from_start' value entirely across core and all four runtimes. retry() now has a single behavior — restore from a checkpoint (latest or checkpointId) and re-run the triggering message — with a genesis fallback: if no checkpoint exists, restart fresh from the triggering message instead of throwing. from_start was broken on every runtime since introduction and had zero test coverage.
FU-RETRY-GENESIS-PARSE-WEDGE: genesis-state stateSchema.parse({}) can wedge a session in active — ✅ done
Status: done · Closed by: branch from-start-retry-fix
Resolution: Wrapped the entire post-CAS retry() body in a try/catch (JS, Temporal, CF) that best-effort rolls the session back active→failed on ANY error (including a ZodError from stateSchema.parse({})), then re-throws the original error. Tests in js-agent-executor-retry-genesis.test.ts and executor-retry-genesis.integ.test.ts pin the rollback and verify a second retry() can succeed after the parse-throw.
FU-RETRY-BOGUS-CHECKPOINTID: explicit-but-unresolvable checkpointId silently wipes state — ✅ done
Status: done · Closed by: branch from-start-retry-fix
Resolution: Added an explicit guard after getCheckpoint(checkpointId): if the caller provided an explicit checkpointId and the store returns null, throw Checkpoint '${checkpointId}' not found for session ${sessionId}. instead of falling through to genesis. The general post-CAS rollback (FIX B) covers this error too — the session is returned to failed and no truncation occurs. Tests in both js-agent-executor-retry-genesis.test.ts and executor-retry-genesis.integ.test.ts verify the error message, rollback, and transcript preservation.
FU-CF-DO-RETRY-DOUBLE-CAS: Cloudflare Durable-Object /retry double-claims the session and never runs — ✅ closed
Status: ✅ closed — handleRetry now does a read-only status precheck; the single authoritative ['failed']→'active' CAS is owned by JSAgentExecutor.retry(). CF-DO /retry runs again instead of wedging. · Closed by: branch fix/cf-do-retry-double-cas. · Severity: high (correctness — CF-DO /retry was non-functional) · Pre-existing: introduced by the v7 stateless-suspension redesign (d7fab7fb2), NOT by the from-start-retry-fix work. Surfaced during the post-!220 exhaustive coverage audit.
Resolution: Removed handleRetry's own compareAndSetStatus(sessionId, ['failed'], 'active') (durable-object-agent-base.ts:1843) and replaced it with a READ-ONLY precheck: load state, return 400 if no state exists, 409 if status is not 'failed' — without mutating status. The single atomic ['failed']→'active' transition is now owned exclusively by JSAgentExecutor.retry() (js-agent-executor.ts:2547), which is also used directly by the JS runtime. The precheck is advisory — if state flips between the read-only load and the inner CAS, retry()'s own CAS-failure handling + rollback applies, since handleRetry no longer pre-claims. The two post-resolve rollback paths (updateStatus(sessionId, 'failed') on agent-resolution / missing-namespace errors) are now idempotent no-ops (the store is still 'failed' at those points), kept as defense-in-depth. New boundary coverage: packages/e2e/src/__tests__/cloudflare-do-retry-protocol.cf.test.ts (real-workerd DO) — (a) /retry on a failed session reaches 'completed' with materialized output (pre-fix wedged; teeth-verified by reverting the precheck → RED again), (b) non-failed session → 409, session untouched. This closes the prior zero-coverage gap on the CF-DO /retry route.
Symptom: DurableObjectAgentBase.handleRetry (runtime-cloudflare/src/do/durable-object-agent-base.ts:1843) does its own compareAndSetStatus(sessionId, ['failed'], 'active'), then delegates to JSAgentExecutor.retry() (:1940), which performs the samecompareAndSetStatus(['failed'], 'active') (runtime-js/src/js-agent-executor.ts:2547). By the time the inner CAS runs the status is already 'active', so it fails and retry() throws "Cannot retry: agent status is 'active'…" — and that throw happens before the inner post-CAS rollback try-block (:2567), so the session is left wedged in 'active'. The throw propagates inside the un-awaited execution IIFE, so the HTTP /retry response is still 200 with a runId, but the run never starts and no subsequent retry() can succeed (the ['failed'] CAS never matches again).
Why it went unnoticed: the CF-DO /retry boundary has zero test coverage — the cloudflare-do-* suite exercises /start, /status, /interrupt, and persistent-agent flows, but never /retry. (The Workflows-runtime retry path, executor.ts, is a separate code path and is covered.)
Recommended fix: make handleRetry perform a read-only status precheck (load state, return 409 early if not 'failed') and let JSAgentExecutor.retry() own the single atomic CAS — eliminating the double transition while preserving the clean early-409. Then add a DO /retry boundary test (checkpoint retry + non-failed-status 409) to the cloudflare-do-* suite. Small, self-contained; warrants its own MR rather than riding on the from_start removal.
FU-CF-MEM-COUNTER: Cloudflare re-extracts the whole transcript every step — done
Status: done. MA-36 gave CF Workflows a per-run watermark and a range read; MA-40 replaced the MemoryManager's per-instance counter with a persisted, store-owned per-session watermark (SessionState.memoryExtractedThrough, advanced by a CAS after the memories are stored, lowered by every truncate), read by runtime-js, the CF Durable Object and CF Workflows. The original record follows. · Severity: low (efficiency, not correctness)
Symptom: Cloudflare builds a fresh MemoryManager per durable step (runtime-cloudflare/src/steps.ts:4530, dispatched at workflow.ts:1984), resetting lastExtractionMessageCount (packages/memory/src/memory-manager.ts:57) to 0. Realtime extraction therefore re-walks the entire filtered transcript every step (~O(steps²) extractor + dedup LLM calls; prompt cost grows with conversation length).
Not a correctness bug: there is no store-layer hash key; dedup is the heuristic LLM/similarity pass in storeExtractedMemories, which suppresses re-extracted content (small tail-risk of stray dupes only). The counter is meaningless in Cloudflare's stateless-per-step model; the JS runtime keeps one MemoryManager per runLoop so the counter accumulates correctly there. Temporal does no realtime extraction.
Recommended fix: persist the extraction offset in durable customState (or a per-session memory-store marker) and rehydrate it per step. Touches the MemoryManagerLike contract → warrants a planned change. The same externalization would let Temporal adopt realtime extraction cleanly. Needs a brainstorm before a plan.
FU-TEMPORAL-DBOS-MEMORY-WATERMARK: Temporal and DBOS run no realtime memory extraction, so they do not use the persisted watermark — open (parked with Temporal / DBOS)
Owner: the parked Temporal and DBOS buckets. Surfaced by: MA-40 (W1).
MA-40 persists the realtime extraction watermark (SessionState.memoryExtractedThrough) on every store and uses it on runtime-js, the CF Durable Object and CF Workflows. Temporal and DBOS do not call extractAndStore at all: they run no realtime extraction (generation: { mode: 'realtime' } extracts nothing there; only the async / on-completion path runs). Their stores still keep the field correct (it is store-owned: the truncates lower it, saveState never writes it). To close: when either runtime adopts realtime extraction, extract from the stored watermark and advance it with advanceMemoryWatermark after the memories are stored, as core's runRealtimeMemoryExtraction does, with a two-turn red like runtime-js/src/__tests__/memory-extraction-watermark.test.ts.
FU-REDIS-STAGE-UPSERT-BY-TOOLCALLID: RedisStateStore.stageChanges double-applies a retried toolCallId — ✅ closed
Status: ✅ closed — RedisStateStore.stageChanges now upserts by toolCallId; idempotent under Temporal activity retry; parity with Postgres + in-memory.
Symptom (before fix): RedisStateStore.stageChanges did an unconditional rpush, and promoteStaging's Lua applies every LRANGE entry with no dedup by toolCallId. A retried stageChanges for the same toolCallId therefore double-applied its writes at promotion (e.g. an /items/- append duplicated into the array).
Why it became reachable: runtime-temporal now stages per-tool via executeToolActivity — a Temporal activity with at-least-once retry semantics (maximumAttempts: 3). Temporal+Redis is the first per-tool-staging-on-Redis combo; before the parallel-tools split, Redis staging was only ever exercised once per step.
Closure: stageChanges now runs STAGE_CHANGES_UPSERT_SCRIPT — a single EVAL that scans the staging LIST for an entry whose decoded toolCallId matches and LSET-replaces it IN PLACE (preserving its FIFO position), else RPUSHes a new tail entry (TTL applied inside the script). The scan+write is one atomic EVAL, so it can't race concurrent parallel-tool staging on the same step, and a retry collapses to a single applied entry. This is the Redis analogue of Postgres's ON CONFLICT (session_id, step_id, tool_call_id) DO UPDATE and the in-memory findIndex → replace-in-place; all three stores are now idempotent-by-toolCallId. promoteStaging's Lua and the size-limit pre-flight guard are unchanged. Corrupt / old-shape entries that fail cjson.decode are treated as non-matching (can't alias a live toolCallId), mirroring getStagedChanges / the promoteStaging Lua.
Teeth: the cross-store staging contract (packages/core/src/testing/staging-operations.ts) gained three idempotency-by-toolCallId assertions (re-stage replaces / applies append once; distinct ids stay separate FIFO entries; middle re-stage preserves position) — these run against ALL stores incl. Redis, Postgres, and in-memory in CI. Redis-specific coverage lives in packages/store-redis/src/__tests__/redis-staging-fifo.integ.test.ts (Idempotency by toolCallId describe). Both verified locally against a live Redis; reverting only the redis-state.ts impl re-fails the new tests ([t1,t2,t3,t2] duplicate + double-applied append).
TII-C1+SUBAGENT (Cross-runtime sub-agent matrix) follow-ups
TII-C1+SUBAGENT: Cross-runtime sub-agent multiplexed stream resume matrix — ✅ landed (with 2 open follow-ups)
Status: ✅ landed Closed by: the commit series implementing docs/superpowers/plans/2026-05-12-cross-runtime-subagent-resume-matrix.md (20 commits ending at HEAD on omnara/stateless-suspension-redesign).
Summary: Extended the single-agent cross-runtime-resume-matrix.{integ,cf}.test.ts with parent-with- sub-agent topologies across 6 runtime × store × stream-manager combos (JS+memory, JS+redis, JS+postgres+redis, JS+D1+DO, DBOS+postgres+redis, Temporal+memory, Temporal+redis+redis) plus the CF-DO slice. 30 active tests pass with vitest run; 67 env-gated tests skip cleanly without REDIS_URL / POSTGRES_URL; it.todo markers preserve documented gaps (CF Workflows combo — blocked on Miniflare). S6 doubly-nested HITL is now a negative-invariant test, not it.todo — see FU-MATRIX-DOUBLY-NESTED-HITL (resolved, documented-as-unsupported).
Scenarios covered: single sub-agent baseline (S1), multi-turn with sub-agents (S2 — see FU-MATRIX-MULTITURN), nested γ-cascade (S3), γ-fan-out (S4), child failure (S5), doubly-nested HITL (S6 — negative-invariant, see FU-MATRIX-DOUBLY-NESTED-HITL), cleanupToStep boundary (S7), remote sub-agent (S8), multi-tab race (S9), fromSequence past end (S10), mid-cursor resume (S11), and stream_resync on path-3 resume (S12).
Side fix landed during matrix work: runtime-temporalworkflow.ts was missing the parent's tool_end chunk for subagent__<name> after subagent_end on the success path of local sub-agent dispatch. Mirrors runtime-js's paired-close pattern. The matrix's S1 path-6 tool_end assertion (commit 808f36273) surfaced the bug; the fix lives in commit 9f6723900.
Two follow-ups still open (tracked separately):
FU-MATRIX-MULTITURN— no portable cross-runtime API for multi- turn continuation; S2 documents the gap by skipping Temporal rows with descriptive name suffixes.FU-MATRIX-DOUBLY-NESTED-HITL— ✅ resolved (documented-as- unsupported, GL #73): child suspending on its own client-tool returnscompletedinstead ofsuspended_client_toolon both JS and Temporal. S6 pins this as a negative-invariant assertion.
FU-DBOS-LOGGING-GAPS: Bare catch {} blocks in runtime-dbos lose operator observability — ✅ closed
Status: done (post-release follow-up cluster, alongside Path B Phase 2). Surfaced by: pr-review-toolkit:silent-failure-hunter audit of the v6→v7 stateless-suspension MR (commit landing audit performed 2026-05-13).
Closure: all 4 bare catch {} sites now emit a typed log line at the appropriate severity. Control flow is unchanged — the authoritative fallback mechanism (CAS gates, failStream/endStream, TTL fallbacks) still owns user-observable correctness. The log lines are observability-only:
packages/runtime-dbos/src/workflows/shared.ts:1122-1133/1144-1155(now widened to include the warn log line):args.logger?.warn?.('dbos.workflow.emit_error_chunk_failed; failStream remains authoritative', {...})and the analogousemit_output_chunk_failed.packages/runtime-dbos/src/handles/active-handle.ts:295→this.cfg.logger?.debug?.('dbos.active_handle.poll_load_state_failed; will retry on next interval', {...}). Debug because a single missed poll-iteration is fully tolerable; a sustained pattern is visible at this level.packages/runtime-dbos/src/lifecycle/resume.ts:161, 167→ both.catch()handlers now logwarnwith structured context. The control flow degrades gracefully to a misleadingAgentAlreadyRunningErrorif the store is genuinely broken, so observability is the only operator hook for diagnosing that case.packages/runtime-temporal/src/workflow.ts:1660-1666→ top-level workflow-failure escape hatch now logserror(notwarn) viawf.log.error('temporal.workflow.persist_terminal_state_failed; durable state may be stuck in non-terminal status', {...}). This is the canonical place where durable state can drift from the in-memory result, so it gets the highest severity.
No tests changed; verified runtime-dbos 364/364 + runtime-temporal 243/243 + integration usage-tracking 4/4 still pass after the edits.
Description: Four sites in runtime-dbos swallow errors with a bare catch {} (no log line). The user-observable contract is preserved at each site by a bounded primary mechanism (CAS gates, authoritative failStream/endStream, TTL fallbacks), so these are NOT bugs — but the operator loses signal when transient infra failures occur. Inconsistent with sibling sites in the same package that DO log at debug/warn.
Sites:
packages/runtime-dbos/src/workflows/shared.ts:1112-1123and1133-1144—dispatcher.emitErrorChunk/dispatcher.emitOutputChunk"best-effort" catches. If miniflare D1 / Redis pubsub is degraded, the canary signal is suppressed. Comment ("failStream is authoritative") is defensible but aargs.logger?.warn?.(...)log line would aid investigation.packages/runtime-dbos/src/handles/active-handle.ts:295—_pollForPendingClientTool'sloadStatecatch. State-store flakes during the poll race are unobservable.packages/runtime-dbos/src/lifecycle/resume.ts:161, 167—loadState(sessionId).catch(() => null)andgetCurrentRun(sessionId).catch(() => null)in the[CROSS-RUNTIME]active-session guard. If the state store is genuinely broken,hasPendingClientToolsreads false and the code throwsAgentAlreadyRunningError— misleading classification.packages/runtime-temporal/src/workflow.ts:1660-1666— innerpersistTerminalState"best-effort" catch in the top-level error path. IfpersistTerminalStatefails, the workflow still returns{status:'failed', error}from in-memory, but durable state may be stuck in 'active', which the nextexecute(sessionId)would trip on with a misleading "already active" rejection.
Action: Replace each bare catch with catch (err) { args.logger?.warn?.('<operation> failed; <fallback mechanism> remains authoritative', { sessionId, runId, error: ... }); }. No control-flow change — just add the log line.
Effort: ~2 hours. Priority: low — observability polish.
FU-DBOS-RECONNECT-STREAM-UNIT-TEST: Multi-turn stream reactivation lacks focused unit test — ✅ closed
Status: done (post-release follow-up cluster, alongside Path B Phase 2). Surfaced by: pr-review-toolkit:pr-test-analyzer audit of the v6→v7 stateless-suspension MR. Rated 6/10 (not blocking).
Closure: added 4 focused unit tests in packages/runtime-dbos/src/__tests__/lifecycle/execute.test.ts that pin the reactivateStream → initStream ordering contract. The shared MockCfg.streamManager now exposes reactivateStream as a first-class mock (was previously absent because no existing test exercised the path), and makeCfg accepts overrides for getStreamInfo, reactivateStream, and initStream.
The tests:
- Reactivates when prior stream is
'ended'— captures call order via shared array; asserts['reactivateStream', 'initStream']exactly. - Reactivates when prior stream is
'failed'— parity case for thefailStream()-produced status. - Skips reactivation when prior stream is
'active'— negative case ensures the guard isn't fired wastefully on an already-live stream. - First-turn execute (no prior stream) — null-stream path; neither reactivate nor unexpected calls should fire on the reactivation branch.
Pins the documented contract so any future refactor that silently drops reactivation in favor of a different mechanism will fail these tests even when the end-to-end coverage (standard-mode-multi-message.integ.test.ts
cross-runtime-subagent-resume-matrix-multiturn.integ.test.tsS2 paths 3/4/6) continues to pass for the wrong reason.
Description: packages/runtime-dbos/src/lifecycle/execute.ts:366-388 calls streamManager.getStreamInfo() and conditionally streamManager.reactivateStream() when status is 'ended' or 'failed'. This is the multi-turn continuation reactivation path (without it, the second execute() on a session whose first turn ended would throw Cannot write to stream in 'ended' state).
The path is covered end-to-end by:
runtime-dbos/.../integration/standard-mode-multi-message.integ.test.ts:36—second execute on same sessionId sees prior messagespackages/e2e/.../cross-runtime-subagent-resume-matrix-multiturn.integ.test.tsS2 paths 3, 4, 6
…but there is no direct unit test in packages/runtime-dbos/src/__tests__/lifecycle/execute.test.ts that mocks getStreamInfo: () => ({status: 'ended'}) and asserts reactivateStream is invoked before initStream. A refactor that silently drops reactivation in favor of a different mechanism could pass the unit tests while breaking the documented contract.
Action: Add a focused unit test:
it('reactivates ended stream on continuation', async () => {
const reactivateStream = vi.fn().mockResolvedValue(undefined);
const streamManager = {
getStreamInfo: vi.fn().mockResolvedValue({ status: 'ended' }),
reactivateStream,
initStream: vi.fn().mockResolvedValue(undefined),
} as never;
const stateStore = { /* mock with existing session that triggers
isContinuation = true */ } as never;
await executeImpl({ stateStore, streamManager, ... }, agent, opts);
expect(reactivateStream).toHaveBeenCalledWith('test-session-id');
expect(reactivateStream).toHaveBeenCalledBefore(streamManager.initStream as never);
});Effort: ~1 hour. Priority: low — defensive test coverage.
FU-RUNTIME-JS-STATUS-NORMALIZATION: persistViaSaveAndPromote writes AgentStatusValue into SessionStatus column
Status: closed — fixed in commit 7c2944b8b (see below). Surfaced by: the DBOS fix agent for FU-DBOS-ERROR-CHUNK (commit 2ff95b0ed, partially reverted in 8f88e5d4d). The agent attempted an in-place mutation fix that broke 4 runtime-js unit tests; reverting left the underlying bug unfixed.
Description: packages/runtime-js/src/run-loop.tspersistViaSaveAndPromote(input) does a structural cast of input.state (typed AgentState<TState, TOutput>) to SessionState<TState, TOutput>. The two types differ on the status field:
AgentState.statusis anAgentStatusValue('running','waiting_tool','completed','failed','interrupted','paused').SessionState.statusis aSessionStatus('active','completed','failed','interrupted','paused').'running'and'waiting_tool'both map to'active'.
The structural cast does NOT convert the field. As a result, saveStateAndPromoteStaging may persist 'running' into the states.status column when the row should hold 'active'. loadState later rejects the value via SessionStatusSchema.parse, producing the ZodError cluster that breaks ~100 of 240 CF workerd tests (see FU-D1-STATUS-SCHEMA-DRIFT).
The prior in-place mutation fix (2ff95b0ed) broke 4 unit tests because sessionState aliases input.state; mutating status also corrupted the live in-memory AgentStatusValue that the runLoop switch reads after the call.
Fix: Build persistedState as a fresh spread object before the saveStateAndPromoteStaging call, converting only the persisted copy's status:
const persistedState: SessionState<TState, TOutput> = {
...sessionState,
status: agentStatusToSessionStatus(sessionState.status as unknown as AgentStatusValue),
};Validation:
- runtime-js: 532/532 pass (all 4 previously-broken completion-retry tests green)
- store-cloudflare unit: 264/264 pass
- CF workerd e2e: 225 pass / 6 fail (all 6 are the pre-existing edge-case failures unrelated to this bug)
- typecheck: 55/55 tasks pass
FU-D1-STATUS-SCHEMA-DRIFTalso closed by this fix (schema widening from2ff95b0edstays; write-side normalization eliminates the primary Zod rejection).
FU-DBOS-ERROR-CHUNK: DBOS failStream doesn't emit AI SDK error chunk
Status: ✅ closed — see "Done items" section. Closed by: commit e7506723e.
FU-D1-STATUS-SCHEMA-DRIFT: D1StateStore.loadState Zod schema rejects valid status
Status: closed — root cause was FU-RUNTIME-JS-STATUS-NORMALIZATION (write-side bug, not a read-side schema gap). Fixed together in the same commit as that follow-up. The schema widening in state-schemas.ts (from 2ff95b0ed) stays as a defense-in-depth measure to accept both AgentStatusValue and SessionStatus literals in checkpoint state blobs. Surfaced by: npm run test:cloudflare in packages/e2e — most CF integration tests fail with:
ZodError: Invalid option: expected one of "active"|"completed"|"failed"|"interrupted"|"paused"
at D1StateStore.loadState (packages/store-cloudflare/src/d1-state.ts:2591:37)The actual rejected value was 'running' — an AgentStatusValue written to D1 by persistViaSaveAndPromote's structural cast without converting the status field. Normalizing at write time (new persistedState object in run-loop.ts) eliminates the rejection.
Verified pre-existing: Reverting packages/store-cloudflare/, packages/runtime-cloudflare/, and packages/ai-sdk/ to baseline 2d5c3d786 reproduces the same failures identically — TII-C1+SUBAGENT matrix work did not introduce this.
FU-MATRIX-DOUBLY-NESTED-HITL: Sub-agent client-tool suspension — documented as unsupported
Status: ✅ resolved (documented-as-unsupported; path 3 chosen) — GL #73 Surfaced by: cross-runtime-subagent-resume-matrix-suspended.integ.test.ts S6 scenario in the cross-runtime sub-agent matrix.
Description: When a child sub-agent calls a client-executed tool (via defineTool({ execute: 'client' })), the expectation is that the child's session suspends with status 'suspended_client_tool' and the parent waits for a submitToolResult against the child's sub-session id. In practice, both runtime-js and runtime-temporal return parent status 'completed' immediately — the child's client-tool dispatch does not propagate a suspension up through the ephemeral sub-agent boundary.
Resolution (path 3): Doubly-nested HITL (ephemeral sub-agent calling a client-executed/approval-gated tool) is documented as unsupported in docs/internals/concepts.md (Client-Executed Tools section). The matrix test S6 is now a negative-invariant assertion (not it.todo): it pins the current behavior (parent 'completed', exactly one subagent_start/subagent_end pair) so any future change that starts supporting nested HITL fails the assertion and forces a conscious revisit. Users needing inner-agent human approval should either hoist the client-executed tool to the top-level agent, or use a persistent sub-agent (stable session id, drivable via follow-up turns) instead of an ephemeral one.
Deferred-but-available paths if this is ever scheduled for real support: (1) fix the framework to propagate child suspension across the ephemeral boundary, or (2) add an explicit inheritClientTools option on createSubAgentTool.
Priority: medium — affects users building agents-that-call-agents where the inner agent needs human approval. The single-level HITL case (parent agent's client tool) works correctly across all runtimes; only the doubly-nested ephemeral case is unsupported.
FU-DBOS-REJECTED-FINISH-FRAMES: a rejected __finish__ step's rows get no tool_end frame on DBOS — open
Owner: runtime-dbos. Surfaced by: the review of DI-81's Temporal/DBOS MR (WA, 2026-11-06).
A rejected __finish__ step (DI-81) does not end the run, so its rows (the rejected call's error, its siblings' finish_output_rejected) need their tool_end frames, or a client that saw the calls streamed keeps them open until a reload. runtime-js, the CF DO, CF Workflows and Temporal write them, built from the rows; DBOS stages the rows in the workflow body, where a frame write needs a step of its own, so it writes none. The rows are committed, so a snapshot shows them.
FU-DBOS-FINISHWITH-OUTPUT-SCHEMA: a DBOS finishWith output is neither transformed nor checked against the agent's outputSchema — done (finishing v3 MR 2, DI-91)
Finishing API v3 (user rulings 2026-11-14) removed
finishWithTransformon every runtime, and an invalidfinishWithresult now fails the run (framework_completion_output_contract_violation) instead of going back to the model. Finishing v3 MR 2 adds the check on DBOS, CF Workflows and Temporal (coredecideOutputWithoutGuardat each runtime's finishWith site; DI-91 fixed). The text below is as filed.
Owner: runtime-dbos (DI-91). Surfaced by: the finish-step sibling-calls design (spec 2026-11-03 §7) and DI-90's trace (W1, 2026-11-10).
DBOS completes a run with the finishWith tool's raw result (workflows/shared.ts:4500, :4593-4603 at main 4e26f6cb0b): _finishWithTransform is never applied (serialize.ts does not carry the function into the workflow) and nothing checks the value against the agent's outputSchema. runtime-js and the CF DO check it (DI-90, fixed): a valid output completes as the schema's parsed value, an invalid one is a failed call the model can correct, and a forced invalid call is a recoverable miss. The fix applies the transform inside the tool's own step, then the same check. CF Workflows and Temporal apply the transform but not the check: DI-91 covers all three.
FU-TEMPORAL-CHILD-MAX-STEPS: Temporal child workflows never receive their agent's maxSteps
Status: open. Owner: CF-D1 MR 2 (Temporal round). Surfaced by: CF-D1 MR 1a final review (I-5).
Temporal sub-agent / companion child workflows are started from inside the parent workflow (workflow-dispatch.ts spawn / continue / resume sites, workflow.ts:~541), which has no agent registry, so childArgs never carry the child agent's maxSteps. Before DI-28 an absent budget meant "uncapped", so a child's own maxSteps was never honoured; applying the new default there would have capped every child at 50 regardless (a regression for children with maxSteps > 50 or Infinity). MR 1a keeps children uncapped (workflowMaxSteps). Fix: have the activities that plan a spawn / continuation / resume return the child's resolved budget (wireMaxSteps(childAgent.maxSteps)) and put it into childArgs; then drop the child exception in workflowMaxSteps. Add a child-cap parity case.
FU-RESUMECOUNT-WINNER-BUMP: SessionState.resumeCount was not counted on Temporal and DBOS
Status: ✅ resolved (S5 snapshot MR). Surfaced by: the S5 rebase onto CF-D1 MR 1a.
resumeCount is documented as the number of resume / retry entries. On origin/main, Temporal and DBOS bumped it with incrementResumeCount BEFORE their instance started (a loser write), for both resume() and retry(); S5 removed those calls ("losers write nothing", workflow ids from the run id) without a winner-path replacement, so it stayed 0. Fix: as on Cloudflare Workflows, the run-start WINNER of an executor resume / retry entry bumps it once in its entry state, never on an activity / step re-execution whose commit already landed, and a losing caller never writes it: Temporal in startRunCommit (TemporalRunStartEntry.countsResume, set for an executor resume(), not for a parent's re-spawn of a parent_suspended child, which main never counted either), DBOS in runStartRunCommit. runtime-js (and the CF DO, which runs its loop) never counted it on main and still does not; docs/guide/querying.md says so per runtime. Tests: runtime-temporal/src/__tests__/resume-count-winner.test.ts, runtime-dbos/src/__tests__/resume-count-winner.test.ts, the real-server Temporal commit-checkpoint-truth.integ.test.ts (concurrent resume / retry, sequential resumes), the DBOS integ dbos-cas-races / c3-single-winner / resume-standard / retry-*, and the e2e resume-count-parity{,-dbos}.integ.test.ts (JS ×3 at 0, Temporal ×3, DBOS).
FU-CFW-ORPHANED-EXECUTE-CLAIM: a crash between a CONTINUATION's claim and create() strands the session
Status: ✅ resolved (S5 snapshot MR, rebase onto CF-D1 MR 1a). Surfaced by: CF-D1 MR 1a final review (I-3) and DoD audit (D1).
MR 1a's execute() claimed the session BEFORE workflowBinding.create(), so a Worker cancelled between the claim and create() left a claim with no instance (a new session was reclaimable; a continuation's → active CAS stranded it).
Fix: there is no executor-side claim any more. The single-winner gate is the instance's own run-start commit, and execute() / resume() / retry() write nothing before create() except a write-once createSession for a NEW session (an active row with no run). A cancel there leaves no run, and the next execute() simply runs: its run-start commit finds no live run. Tests: the runtime-cloudflare executor unit case "FU-CFW-ORPHANED-EXECUTE-CLAIM" and the real-engine cfw-terminal-truth.wf-noiso.test.ts "D1" case.
FU-JS-PAUSED-AWAITING-CHILDREN-EXECUTE: runtime-js execute() continues a parent suspended on a sub-agent
Status: open. Owner: CF-D1 MR 3 (JS / DBOS round). Surfaced by: CF-D1 MR 1a final review (I-6).
A parent suspended on a sub-agent's pending client tool is paused with an empty pendingClientToolCalls (the wait lives in suspendedAwaitingChildren / suspendedStepId), so runtime-js's paused-with-pending guard lets execute() start a new turn and orphan the child's pending call. CF Workflows refuses this case since MR 1a; add the same check to runtime-js (and so the CF DO) and DBOS.
FU-CFW-E2E-MULTITURN-ROW: add CF Workflows to the e2e multi-turn matrix
Status: open. Owner: conformance owner (plan 0c, the CFW executor lane). Surfaced by: CF-D1 MR 1a (CP-88).
CF Workflows multi-turn continuation is proven on the real engine in runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts, but the e2e cross-runtime families (cross-runtime-subagent-resume-matrix-multiturn, client-tool-multiturn-resume) have no CFW combo: the e2e CFW harness adapter (e2e/src/harness/setup-helpers/cfw-workflows-d1.ts) is hand-rolled and builds its own fixed agent__<name>__<sessionId> instance ids instead of driving the real CloudflareAgentExecutor. Add the CFW rows once plan 0c's real-executor CFW lane exists, and retire the adapter's hand-rolled ids.
Unblocked (plan 0c MR-1b): the real-executor env exists: setupCfwMiniflare (e2e/src/harness/setup-helpers/cfw-miniflare.ts, the cfw-workflows-d1 row) runs the real CloudflareAgentExecutor from Node over Miniflare's Workflows binding. The hand-rolled adapter is now the cfw-workflows-d1-pool row. The two families still hand-write their combos, so adding the CFW rows is still open.
Plan 0c MR-2: the cfw-workflows-d1 conformance lane is running. Its multi-turn cells pass through the real executor: history.prior-turns, lifecycle.retry-after-turn and the branch cells. That covers plain continuation, not these two families' shapes: a sub-agent resumed across turns, and a client tool suspended and resumed in a later turn. No cfw cell asserts either (the nearest, history.companion-continue, fails on cfw under CP-23), so this item stays open. It no longer waits on the lane; it waits on those shapes.
FU-MATRIX-MULTITURN: portable cross-runtime multi-turn continuation
Status: ✅ resolved for JS / DBOS / Temporal / CF-DO (GL #74) and CF Workflows (GL #109, CF-D1 MR 1a, CP-88/CP-89). Surfaced by: cross-runtime-subagent-resume-matrix-multiturn.integ.test.ts S2 in the cross-runtime sub-agent matrix.
Resolution: AgentExecutor.execute(agent, msg, { sessionId }) now continues a completed session into a new turn portably (Option A — execute() is the continuation verb; no new interface method). A spike refuted the original premise (a Temporal workflow-id collision): Temporal's default WorkflowIdReusePolicy already starts the new run once turn 1 has CLOSED, and initializeAgentState already preserves history. The real blocker was that turn 1's completion left the sessionId-keyed stream in a terminal 'ended' state and the durable runtimes' execute() never reactivated it. Fix: a reactivateStream- on-continuation block, mirroring what runtime-js / runtime-dbos and Temporal's own retry() already do.
Per-runtime status:
- runtime-js / runtime-dbos: already worked (matrix S2 js/dbos rows).
- runtime-temporal: fixed — added the reactivation block to
execute(); matrix S2 Temporal rows re-enabled. - CF-DO: already worked — its DO
/starthandler delegates toJSAgentExecutor.execute(), which has the reactivation block. Regression-guard test added (proven load-bearing). - CFW Workflows: fixed in CF-D1 MR 1a:
execute()claims the session before creating the instance, passesentry: 'new' | 'continue', runs each turn in its own write-once instance (…__turn__<n>, keyed on the session's resume counter), reactivates the stream and clears a leftover interrupt flag;getHandle()resolves the live instance. Proven on the real engine under vitest-pool-workers (runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts). The e2e matrix has no CFW row yet: see FU-CFW-E2E-MULTITURN-ROW.
Portable contract floor = completed. A still-running session rejects with AgentAlreadyRunningError (mutex preserved — Temporal via the status === 'running' guard, others via CAS). Continuation from interrupted / failed / paused is not part of the portable contract (use resume() / retry()); runtime-js's broader execute() acceptance set is a documented JS-only superset — tracked separately as GL #110 (decide between dev-warning vs deprecation vs status quo).
Design + plan: docs/superpowers/specs/2026-05-23-cross-runtime-multiturn-continuation-design.md.
FU-JS-EXECUTE-SUPERSET: runtime-js accepts a broader prior-state set than the portable contract — ✅ closed (status-quo)
Status: ✅ closed as documented status-quo — GL #110. Decision: do NOT tighten runtime-js (design decision #3); document the superset; pin the portable floor with a cross-runtime test. A dev-warning on non-floor JS execute() continuation remains an OPEN OPTION under GL #110 if ever desired, but is a deliberate UX/behavior change out of scope here. Priority: low Surfaced by: the #74 audit (FU-MATRIX-MULTITURN resolution sweep).
Description: runtime-js's execute() CAS at packages/runtime-js/src/js-agent-executor.ts:1641-1646 accepts prior states ['paused', 'interrupted', 'completed', 'failed'], while the portable cross-runtime contract is completed only. JS code that continues a failed / interrupted session through execute() silently works on JS but breaks when ported to Temporal / CF-DO / DBOS, where the correct verbs are resume() / retry().
Closure (decision: status-quo + floor pinned):
Per the cross-runtime multi-turn continuation design (docs/superpowers/specs/2026-05-23-cross-runtime-multiturn-continuation-design.md, decision #3): runtime-js KEEPS its broader execute() CAS acceptance set as a documented, non-portable superset. We do NOT tighten it — no JS users break — and the framework simply stops promising the broader set cross-runtime. The portable floor is the intersection (completed).
This is now enforced by test, not documentation alone:
- Portable floor pinned cross-runtime:
execute()continuation from acompletedprior run runs across every viable backend (js-{memory,redis,postgres}, temporal-{memory,redis,postgres}, dbos-postgres) in thedescribe.each(getViableBackends({ requires: ['hitl'], excludes: ['cfw-workflows'] }))block atpackages/e2e/src/__tests__/session-model-continuation.integ.test.ts("execute() continuation floor (fromcompleted)"). TEETH: the row fails if any runtime rejected continuation-from-completed. - JS superset pinned (JS-only): two JS-runtime unit tests at
packages/runtime-js/src/__tests__/js-agent-executor-continuation.test.ts("execute() continuation SUPERSET (JS-specific, GL #110)") assert JSexecute()continues frominterrupted(real interrupt) andfailed(store CAS transition) prior statuses. TEETH: each fails if JS tightened its CAS source-set — a future tightening becomes a deliberate, test-visible change rather than a silent regression.
The divergence is documented at docs/internals/concepts.md ("Multi-turn continuation" — "runtime-js additionally accepts a broader prior-state set on execute(); that is a JS-specific superset, not a cross-runtime promise") and docs/internals/session-model.md.
Open option (GL #110, NOT in scope here): a dev-mode warning on non-floor JS execute() continuation (gathers real-usage signal before any tighten) — or a full tighten in a major release. Either is a deliberate UX/behavior change; neither blocks real workloads today.
A.2 (Temporal HITL rewrite) follow-ups
FU-A2-01: Per-call agent hooks not plumbed through Temporal runtime
Status: ✅ done — see "Done items" section. Closed by: commit 2693eb18f (+ changeset da8bd52ea).
FU-A2-02: SUBMIT_TOOL_RESULT_SIGNAL_NAME no-op signal call still in executor.ts
Status: ✅ done — see "Done items" section. Closed by: commit 869787e5c.
FU-A2-03: Harden persistent sub-agent edge cases under __resume-N
Status: ✅ done — see "Done items" section. Closed by: commit 3ebed60f7.
FU-A2-04: workflow.ts size still 1,589 LOC after rewrite — ✅ closed
Status: done (post-release follow-up cluster). Surfaced by: A.2 Task 3.2 implementer's self-review.
Description (historical): Spec target was ~300-400 LOC; original landing was 1,152 LOC, and it had since grown to 1,692 LOC as FU-A2-09 (γ-cascade re-spawn) and other v7-stateless suspension work landed. The bulk of the extra was inlined sub-agent / companion-tool / remote-sub-agent dispatch (~700 LOC across dispatchSubAgents, dispatchCompanionTools, dispatchRemoteSubAgents) plus chunk emission (subagent_start / subagent_end / tool_end) needed to settle SSE consumers, plus the parent-suspended re-spawn loop.
Closure: extracted all three dispatch functions + their context interfaces (SubAgentDispatchContext, SubAgentCall, SubAgentDispatchResult, CompanionDispatchContext, CompanionToolCall, RemoteSubAgentCall) into a new sibling file packages/runtime-temporal/src/workflow-dispatch.ts.
Sizing impact:
workflow.ts: 1692 → 1002 LOC (-690 LOC, -41%)workflow-dispatch.ts: 776 LOC (new file, all dispatch + types)
The workflow body itself (mode dispatch + main loop + suspension branches) is now the dominant content of workflow.ts rather than being buried under 700 lines of dispatch helpers.
ESM cycle handling: Both dispatchSubAgents and dispatchCompanionTools need to start child workflows of the same function they were extracted from (agentWorkflow). To avoid a workflow.ts ↔ workflow-dispatch.ts cycle, the dispatchers accept agentWorkflow as a function parameter typed against a structural AgentWorkflowFn = (input: AgentWorkflowInput) => Promise<AgentWorkflowResult> alias declared in workflow-dispatch.ts. The two callsites in agentWorkflow pass agentWorkflow explicitly. dispatchRemoteSubAgents doesn't spawn child workflows so it doesn't need the parameter.
Verification:
- typecheck: clean across both packages
- runtime-temporal unit: 245/245 pass
- runtime-temporal integration (against Docker Temporal): 49/49 pass
- e2e parity (lifecycle-hooks-parity + client-tool-happy-parity on
temporal-memory): 6/6 pass
Behavior-preserving refactor — no test changes, no signature changes on the public surface, no determinism risk introduced. The cycle- breaking parameter pass is the only contract change, and it's purely internal to the runtime-temporal package.
FU-A2-05: Cross-runtime e2e tests need migration from signal-based to durable submit/resume patterns
Status: ✅ done — see "Done items" below for closing summary. Surfaced by: A.2 Task 4.4 deeper e2e validation (post-6b5f4b309).
Description: A.2 Task 4.2 migrated packages/runtime-temporal/src/__tests__/ files to the new model. But the cross-runtime e2e files in packages/e2e/src/__tests__/*temporal*.integ.test.ts still depend on pre-A.2 patterns. Notable failures:
client-tool-double-submit-temporal.integ.test.ts,client-tool-already-completed-durable-temporal.integ.test.ts— callhandle.signal('submitToolResult', ...)directly. Post-A.2 the workflow exits on suspension so the signal hits a completed workflow →WorkflowNotFoundError: Completed workflow. Migrate to useexecutor.submitToolResult()+ durable observation.client-tool-abort-temporal.integ.test.ts— interrupt during HITL test expects status'interrupted'but gets'suspended_client_tool'. Same path as C-3 but for a slightly different scenario; needs the same durable-flag-on-resume fix to apply. Possibly fixed already by A.2 commita907f1698; verify.client-tool-temporal-crash-recovery.integ.test.ts— both tests fail with'suspended_client_tool'instead of'completed'. Same root cause as C-5 but in a crash-recovery scenario. The__finish__terminal-step detection may not be hitting on the recovery path.persistent-subagent-temporal.integ.test.ts— 6 tests failing withexpected 'running' to be 'completed'. Persistent sub-agents not completing — likely a workflow-body rewrite issue with persistent sub-agent dispatch (the inlineddispatchSubAgentsmay have edge cases).client-tool-subagent-ownership-temporal.integ.test.ts— sub-agent ownership routing test fails. Potentially related to sub-agent ownership writes through the new commitSuspendedStep path.remote-subagent-output-persistence-temporal.integ.test.ts— 1 test failing on remote sub-agent + Temporal interaction.
Action: Per-file triage similar to Task 4.2, but for cross-runtime e2e files. Per-file outcomes likely:
- migrate (replace signal sends with
executor.submitToolResult+ durable poll) - delete (if testing pre-A.2-only behavior covered elsewhere)
- diagnose-and-fix (if it surfaces a real A.2 regression like persistent-subagent)
Effort: ~3-5 days for triage + per-file migrations + diagnosing the persistent-subagent regression. Priority: HIGH — these are real test failures on the main e2e path. Production functionality may be affected for the persistent-subagent and remote-subagent paths.
FU-A2-07: Sub-agent ownership routing — StaleStateError on parent's commitSuspendedStep
Status: ✅ activity-level fix landed (commit landing this entry update). The commitSuspendedStep save now retries on cross-session StaleStateError, reloads, merges clientToolCallOwnership, and re-saves with the fresh version. Confirmed by activities-commit.test.ts "FU-A2-07: retries on StaleStateError and merges with fresh clientToolCallOwnership" + the FU-A2-07 e2e test reaches its later assertions (no longer fails on the first parent commit).
Surfaced by: FU-A2-05 Phase 2 migration (commit 8f3c889bf). File: packages/e2e/src/__tests__/client-tool-subagent-ownership-temporal.integ.test.ts (test still skipped pending FU-A2-09; see below).
Description (closed): When a parent dispatches a sub-agent that suspends with a pending client-tool entry, the parent's commitSuspendedStep save threw StaleStateError: State version mismatch ... expected version 1, but store has version 2. Root cause: the child's own commitSuspendedStep activity calls writeRootOwnership(rootSessionId=parent, ...) which loads + mutates + saves the parent's session row — bumping the parent's version between the moment the parent's commitSuspendedStep activity loaded input.state and the moment it tries to save with expectedVersion: input.state.version.
Fix: bounded retry (5 attempts) inside commitSuspendedStep. On StaleStateError, reload the latest persisted state, merge the clientToolCallOwnership map (associative — child's added entries + parent's locally-computed entries), and re-save with the fresh version.
FU-A2-09: Temporal sub-agent resume after failed:parent_suspended doesn't drive child to completion
Status: done — see "Done items" section.
FU-A2-08: Remote sub-agent SubSessionRef metadata not persisted on Temporal
Status: ✅ done — see "Done items" section. Closed by: commit 5085756d0.
Effort: ~0.5 day. Priority: medium — affects remote sub-agent observability.
FU-A2-06: AgentWorkflowResult status enum extension is a DTO breaking change
Status: ✅ done — see "Done items" section. Closed by: commit e221ac4d3.
FU-A2-38: Re-enable Cloudflare DO real-DO end-to-end tests (G1 + G2)
Status: ✅ done — see "Done items" section. Closed by: commit 30432ad34. Surfaced by: v7 stateless-suspension code review #X (expected +0 to be 5 workerd-suite triage).
Description: Two test files under packages/e2e/src/__tests__/ were committed in 9cf39bd66 wip: preserve doc agent work before stash recovery without the supporting test-worker.ts fixtures and wrangler.toml durable-object bindings. They have NEVER passed on this branch:
cloudflare-do-persistent-agents.cf.test.ts— referencespersistentAgentTestLLMimport +PERSISTENT_AGENT_SERVERdurable object namespace; both undefined at runtime.cloudflare-do-interrupt-protocol.cf.test.ts— referencesinterruptTestLLMimport +INTERRUPT_AGENT_SERVERdurable object namespace; both undefined at runtime.
Both test files are currently describe.skip with an inline comment explaining the situation (re-grep 9cf39bd66 in those files).
Action:
- Add
persistentAgentTestLLM+PersistentAgentTestServerDO class exports inpackages/e2e/src/test-worker.ts(mirror thesubAgentTestLLMpattern). - Add
interruptTestLLM+InterruptTestServerDO class exports. - Add the two new DO namespace bindings (
PERSISTENT_AGENT_SERVER,INTERRUPT_AGENT_SERVER) topackages/e2e/wrangler.toml. - Remove
.skipfrom thedescribe(...)blocks in both test files.
Effort: ~1 day. Priority: medium — this is the only real-DO coverage of the v7 interrupt protocol (G2) and persistent companion-spawn (G1) paths. Currently those paths are exercised only via D1-with-JS shims, not the workerd DurableObjectAgentBase directly.
FU-A2-43: Pre-existing DBOS HITL gaps surfaced by lifecycle-hooks-parity + approval-gate-hook-parity + client-tool-happy-parity — ✅ closed
Status: ✅ closed — fully subsumed by GL #75 (DBOS approval-gate suspension + onMessage/onStateChange) and GL #111 (Batches A–D), shipped across MRs !201–!206. Every failing row this FU enumerated is now un-skipped on dbos-postgres and passing; see the per-row closure map below. Surfaced by: round-3 CI verification after DBOS_CAPS was tagged with 'hitl' in P1.10 — the parity tests now run for dbos-postgres but fail because DBOS's DBOS.recv() blocking model differs from the durable-state-only suspension model used by JS/Temporal/CF.
Closure — per-row map (verified against the actual test files):
The root cause was structural: DBOS_CAPS lacked the 'hitl' capability tag, so getViableBackends({ requires: ['hitl'] }) silently excluded dbos-postgres from the parity matrix. Tagging it (now at packages/e2e/src/harness/capability-sets.ts:55, DBOS_CAPS) made the three parity files run for DBOS — surfacing the gaps below, all since closed. With the gaps closed, every DBOS-specific skip gate in the three files collapsed to the generic env-only backend.skipReason gate (no runtime-conditional skips remain):
lifecycle-hooks-parity.integ.test.ts— client-tool happy path row + tracingContext round-trip row: closed by Batch D (per-callagent.hooksresolution on DBOS, GL #75/#111) + the GL #75 suspension work. Both rows are now gated only byitOrSkip(backend.skipReason ? it.skip : it).lifecycle-hooks-parity.integ.test.ts— approval-gate deny row (previously failed withexpected 'completed' to be 'suspended_client_tool'because DBOS did not suspend on approval-gates): closed by FU-DBOS-APPROVAL-GATES (GL #75 — addeddispatchApprovalGatedTool). The row now assertssuspended_client_tooland runs ondbos-postgres.approval-gate-hook-parity.integ.test.ts— approve path (all four hooks) + deny path (onMessage only): closed by FU-DBOS-ONMESSAGE-ONSTATECHANGE (DBOS now firesonMessage/onStateChangefrom the workflow body / executeTool step in the canonical cross-runtime order). TheitOrSkipPendingMessageStateGapgate is nowbackend.skipReason ? it.skip : it(env-only); both rows run ondbos-postgres.client-tool-happy-parity.integ.test.ts— scenario 1 (suspended.toolCallIds result shape) + scenario 2 (exactly-once submit arbiter): closed by GL #111 Batch A. Scenario 3 (client_tool_timeouttool_endchunk): closed by GL #111 Batch B. All three scenarios are gated only by the genericbackend.skipReason(theitOrSkipTimeoutOnDbosgate is now identical toitOrSkip).
Orthogonal items NOT part of A2-43's scope (tracked separately): FU-TRACING-LANGFUSE-PENDING-RUNS-OVERWRITE-ON-RECV-WAKE (still open — observability degradation on DBOS Branch 1 recv-wake) and the per-user-message onMessage cross-runtime divergence (FU-USER-INPUT-ONMESSAGE-CROSS-RUNTIME, since closed). These are distinct from the lifecycle/approval/client-tool parity rows A2-43 enumerated and are not gated by A2-43's closure.
Historical detail (pre-closure):
Failing tests (all dbos-postgres only):
packages/e2e/src/__tests__/lifecycle-hooks-parity.integ.test.ts:
client-tool happy path: onAgentSuspended + onAgentResumed fire exactly once with linked sessionId— 30s timeout (hooks never observed)tracingContext set by onAgentSuspended persists through resume → completion— 30s timeout (same root cause)approval-gate deny path: onAgentSuspended + onAgentResumed both fire—expected 'completed' to be 'suspended_client_tool'(DBOS does not suspend on approval-gates; resolves immediately)
packages/e2e/src/__tests__/approval-gate-hook-parity.integ.test.ts:
approve path fires { beforeTool, afterTool, onMessage, onStateChange } with correct payloadsdeny path fires onMessage only — beforeTool / afterTool / onStateChange do NOT fire
packages/e2e/src/__tests__/client-tool-happy-parity.integ.test.ts:
submitToolResult routes client-tool result back to LLM for next turn— ✅ closed (Batch A, gap 1)exactly one submit wins; conversation reflects the winning result— ✅ closed (Batch A, gap 2)emits tool_end with client_tool_timeout error and continues on next step— ✅ closed (Batch B, gap 3)
Action:
- (Short term) The lifecycle-hooks-parity tests are skipped on
dbos-postgresvia theitOrSkipDbosgate (seepackages/e2e/src/__tests__/lifecycle-hooks-parity.integ.test.ts). - (Medium term) Decide whether DBOS should grow a true suspend-on-pending-approval path that mirrors the JS/Temporal/CF semantics, OR refine
Capabilityto expose'hitl-approval-gate'separately from'hitl'so DBOS only opts into the subset it can actually serve. - (Long term) Alternatively, add a DBOS-specific test variant that asserts the
DBOS.recv()model directly rather than asserting parity with the suspension-based runtimes.
Effort: ~2-3 days for option 2 (capability split) — preferred because it gives DBOS a clean opt-out without losing parity coverage on JS/Temporal/CF. Priority: medium — DBOS HITL works in production via DBOS.recv; the gap is in cross-runtime parity assertions, not user-visible functionality.
Update: the approval-gate suspension portion is now resolved — see FU-DBOS-APPROVAL-GATES below (GL #75). The remaining e2e parity gaps are split to GL #111.
FU-DBOS-OWN-PENDING-CLEANUP-NITS: the ownership-race cleanup of a DBOS sub-agent registration — open (LOW; owner: W1)
Surfaced by: the orchestrator's review of !532 (merged). In runtime-dbos/src/steps/client-tool.ts: removeOwnPendingEntry sits between writeOwnership's JSDoc and writeOwnership (two stacked JSDoc blocks); a non-stale error from it replaces the OwnershipConflictError the caller should see (catch it, log the cleanup failure, rethrow the original conflict); and its OCC loop returns silently after OWNERSHIP_MAX_RETRIES (log it). Fix the next time a change touches that file.
FU-DBOS-APPROVAL-GATES: DBOS approval-gate suspension
Status: ✅ resolved for approval-gate suspension (GL #75); e2e HITL parity gaps split to GL #111.
runtime-dbos previously rejected submitToolResult({kind:'approval-response'}) and had no requireApproval dispatch path. Closed by adding dispatchApprovalGatedTool parallel to dispatchClientTool, reusing the shared pendingClientToolCalls map + DBOS.recv(toolCallId) suspension primitive. Function-form requireApproval was initially evaluated inline (fail-closed); it is now step-wrapped/checkpointed as of Batch C (gap 5 — see the Batch C update below). onAgentSuspended/onAgentResumed now fire for approval-gate + client-tool auto-continue. 9 package-level integration tests cover approve/deny/predicate-variants/shape/hooks/persistent-mode.
Deferred (GL #111): the cross-runtime e2e parity tests were blocked by 5 pre-existing DBOS gaps the parity tests assert — suspended.toolCallIds result shape (gap 1, ✅ closed Batch A), exactly-once submit (gap 2, ✅ closed Batch A), client_tool_timeout tool_end chunk (gap 3, ✅ closed Batch B), per-call agent.hooks resolution (gap 4, ✅ closed Batch D), and checkpointing the requireApproval function-form predicate (gap 5, ✅ closed Batch C). All five are now closed.
Batch A update (#111 gaps 1 + 2 closed): the suspended.toolCallIds result-shape gap and the exactly-once-submit arbiter gap are closed in the feat/dbos-hitl-parity-gaps-1-2 MR. ActiveHandle now populates suspended: { toolCallIds } from the durable pendingClientToolCalls map; markPendingSubmittedBestEffort is now the OCC arbiter, so the first concurrent submit observes submittedAt == null and wins (accepted), subsequent ones see the stamped value and return already_completed. The corresponding e2e parity rows (client-tool-happy-parity.integ.test.ts scenarios 1 + 2) are un-skipped on dbos-postgres and pass. Scenario 3 (timeout tool_end chunk) is closed in Batch B (gap 3, see below).
Batch C update (#111 gap 5 closed): the function-form requireApproval predicate is now evaluated inside a @DBOS.step (ApprovalGateStep.evaluateApprovalGatePredicateStep in packages/runtime-dbos/src/steps/approval-gate.ts), so its boolean result is checkpointed in DBOS workflow history. On replay or step retry, DBOS returns the recorded boolean WITHOUT re-invoking the predicate — the suspend-vs-run decision is replay-deterministic even when the predicate is non-pure (clock / random / external reads). Fail-closed semantics (throw → true) live INSIDE the step so the checkpointed value is always a boolean — never a thrown error — and the workflow body never observes the throw. Throws emit a warn with errorId: 'dbos.approval_gate.predicate_threw'. The previous "MUST be pure" caveat on evaluateApprovalGate is lifted; runtime-dbos now matches runtime-temporal's activity-wrapped predicate evaluation. Locked by 7 new unit tests in __tests__/steps/approval-gate-step.test.ts (per-branch return-value contract + fail-closed errorId warn) and 3 new integration tests in __tests__/integration/approval-gate-predicate-checkpoint.integ.test.ts (predicate-called-exactly-once, throw-caught-inside-step, single evaluateApprovalGatePredicateStep entry in DBOS.listWorkflowSteps(workflowID) with the boolean output). The pre-existing GL #75 approval-gate suite (12 tests) passes unchanged.
Batch B update (#111 gap 3 closed): the client_tool_timeout tool_end chunk gap is closed in the feat/dbos-hitl-parity-gap-3 MR. dispatchClientTool now emits a paired tool_end chunk for both outcomes (submit success + timeout failure) via a new emitToolEndStep (@DBOS.step), emitted BEFORE the afterTool hook to match the canonical cross-runtime order (tool_end chunk → afterTool); the timeout tool-result message carries the humanized error text plus the CLIENT_TOOL_ERROR_CODE metadata, matching JS/Temporal/CF. The emitToolEndStep failure path logs errorId: dbos.client_tool.emit_tool_end_failed and never loses the turn outcome. Server-tool failures are unaffected (verbatim error, no error code). The aborted branch in client-tool-resolver.ts is defensive — exercised by the resolver unit tests (client-tool-resolver.test.ts) but unreachable from the DBOS workflow body, because cancellation rejects DBOS.recv rather than firing the abort signal (cleanup runs in lifecycle/interrupt.ts), so dispatchClientTool's tool_end emit never runs for abort. e2e parity scenario 3 (emits tool_end with client_tool_timeout error and continues on next step) is un-skipped on dbos-postgres and passes; all 3 client-tool-happy-parity scenarios now pass on dbos-postgres.
Batch D update (#111 gap 4 closed): per-call agent.hooks (and options.hooks / options.hookManager) now fire on runtime-dbos. Implementation lives in packages/runtime-dbos/src/observability/hook-manager-registry.ts (process-local Map<workflowId, HookManager>) + packages/runtime-dbos/src/observability/merge-hook-manager.ts (mirrors runtime-js/src/js-agent-executor.ts:3065-3108 buildHookManager — constructor → agent.hooks → options.hooks order; options.hookManager parents via createChild()). executeImpl / resumeImpl / retryImpl compute the merged manager BEFORE DBOS.startWorkflow and register under the new workflowId; HookStep.runHookStep and ExecuteToolStep.executeToolStep resolve via HookManagerRegistry.get(DBOS.workflowID) ?? <static fallback> on every step invocation. Cleanup runs in a non-blocking .finally() on dbosHandle.getResult(); start-failure paths delete the entry eagerly. Cross-worker recovery falls back to the constructor-bound static (documented split-brain risk matching Temporal/CF replaceAgent semantics at runtime-cloudflare/src/registry.ts:182-193). Persistent-mode SEND path leaves the existing registry entry alone (subsequent inbox messages still see the first execute()'s per-call hooks); cleanup for persistent workflows is bounded by process lifetime — tracked as FU-DBOS-PERSISTENT-HOOK-CLEANUP below. Locked by 15 new unit tests (__tests__/hook-manager-registry.test.ts, __tests__/merge-hook-manager.test.ts) + 6 new integration tests (__tests__/integration/per-call-hooks.integ.test.ts). Un-skips 2 of 5 newly-viable e2e parity rows on dbos-postgres; the remaining 3 expose orthogonal DBOS gaps (FU-DBOS-RESUME-RUNID-CLIENT-TOOL still open; FU-DBOS-ONMESSAGE-ONSTATECHANGE closed by follow-up — see below — un-skipping the 2 remaining approval-gate-hook-parity rows). All 5 #111 gaps (1, 2, 3, 4, 5) are now closed.
FU-DBOS-RESUME-RUNID-CLIENT-TOOL: resumed runId equals suspended runId on DBOS — ✅ closed
Status: ✅ closed — Option D (test relaxation + documented divergence) per the FU's own closure criteria #4. Per-runtime conditional assertion in lifecycle-hooks-parity.integ.test.ts Scenario 1 + documented divergence in docs/internals/concepts.md > "Resume runId semantics".
Symptom: on DBOS, the cross-runtime client-tool resume path detects the existing PENDING workflow (blocked on DBOS.recv) and returns a ResumedHandle wrapping the SAME workflow rather than starting a fresh resume-scoped workflow (lifecycle/resume.ts:139-273 — the documented [CROSS-RUNTIME] branch). The resumed run's runId equals the suspended run's runId. JS / Temporal / CF allocate a new runId per resume. DBOS Branch 2 (resume of an interrupted / paused workflow) DOES allocate a fresh runId, matching the reference runtimes.
Closure:
DBOS Branch 1's previousRunId: runId self-loop at packages/runtime-dbos/src/workflows/shared.ts:1579-1601 is semantically correct: "recv-driven in-place continuation under the same runId". The runId equality is the explicit signal of in-place continuation, not a defect. Closure delivered via:
- Test relaxation (
packages/e2e/src/__tests__/lifecycle-hooks-parity.integ.test.tsScenario 1): replaced the unconditionalresumedCtx.runId !== suspendedCtx.runIdassertion with a per-runtime conditional — DBOS asserts equality (in-place continuation), other runtimes assert inequality (fresh resume workflow). TherunIds.has(suspendedCtx.runId)ANDrunIds.has(resumedCtx.runId)invariants onstateStore.listRuns()continue to pass on DBOS via same-key identity. TheitOrSkipPendingResumeRunIdGapskip gate was removed. - Documented divergence (
docs/internals/concepts.md> "Resume runId semantics"): cross-runtime contract forrunId/previousRunIdsemantics.previousRunIdis the load-bearing linkage signal for span stitching on both shapes; hook consumers MUST use it (not therunId !== previousRunIdinequality). - Independent grep evidence: zero downstream production consumers assume
runId !== previousRunId.packages/tracing-langfuse/src/langfuse-hooks.ts:521-522usespreviousRunIdonly as metadata. Parent-span linkage usestracingContext.lastActiveSpanId/rootSpanId.packages/core/src/hooks/types.ts:375-380definespreviousRunIdas optional with no equality contract.
Hook consumer contract:
Tracing-langfuse and audit-hook consumers stitch the resumed span as a child of the suspending span via previousRunId, which is available in the onAgentResumed payload on every runtime:
- Same-runId case (DBOS Branch 1): Observer sees
onAgentResumed({ runId: X, previousRunId: X, sessionId }). Treat as a self-loop "resume of run X in place" — emit a child span keyed on(sessionId, runId)withparentSpan = (sessionId, previousRunId). The self-loop is the explicit signal that this is a DBOS recv-driven auto-continue, not a fresh resume. - Distinct-runId case (JS / Temporal / CF, and DBOS Branch 2 — the new-resume-workflow path): Observer sees
onAgentResumed({ runId: Y, previousRunId: X, sessionId }). Treat as the canonical "new run resumes prior run" and emit a child span keyed on(sessionId, Y)withparentSpan = (sessionId, X).
The previousRunId field is the load-bearing linkage signal — NOT the requirement that runId !== previousRunId. This contract is now canonical: future hook consumers that depend on the inequality alone should be updated.
FU-TRACING-LANGFUSE-PENDING-RUNS-OVERWRITE-ON-RECV-WAKE — ✅ closed
Status: ✅ closed — registerResumedRun now preserves an existing pending entry's accumulated data when one is already registered under params.runId (the DBOS Branch 1 self-loop). Behavior is byte-for-byte identical on the no-collision path (JS / Temporal / CF / DBOS Branch 2 allocate a fresh runId → existing is undefined → fresh entry, exactly as before). Surfaced by: FU sweep #2 brainstorm (revisit of FU-DBOS-RESUME-RUNID-CLIENT-TOOL).
Symptom (pre-fix): on DBOS Branch 1 recv-driven resume (same runId across the suspend/resume boundary — the documented self-loop where runId === previousRunId, see docs/internals/concepts.md "Resume runId semantics"), StatelessSpanEmitter.registerResumedRun did this.pendingRuns.set(params.runId, {...fresh...}), which OVERWROTE the suspended run's existing pendingRuns entry (containing toolInputs, subAgentInputs, the original suspend-time startTime, and original metadata). The suspended entry is still populated at suspension time on DBOS Branch 1 (no finalize happens at suspend; pendingRuns.delete only fires inside emitRootSpan or cleanup()). On JS / Temporal / CF / DBOS Branch 2 there was no collision because params.runId is a fresh key.
Closure: registerResumedRun now reads any existing entry under params.runId and preserves it: startTime: existing?.startTime ?? Date.now(), toolInputs: existing?.toolInputs ?? new Map(), subAgentInputs: existing?.subAgentInputs ?? new Map(), metadata: params.metadata ?? existing?.metadata. sessionId/agentName/input/parentSpanId come from the resume params (the resume is the authoritative source for those). The fix carries a comment citing the concepts-doc "Resume runId semantics" section and this FU name. This preserves the full suspend→resume span-duration window and any in-flight tool/sub-agent input buffered before suspension.
Tests that lock this in: two new unit tests in packages/tracing-langfuse/src/__tests__/suspend-resume-hooks.test.ts (describe('registerResumedRun — DBOS Branch 1 self-loop ...')): (1) same-runId resume preserves accumulated toolInputs/subAgentInputs and original startTime (RED-first against the plain .set(fresh), GREEN after the fix; teeth-verified by reverting to .set(fresh) → preservation test fails); (2) no-collision case — a NEW runId produces a fresh empty entry and leaves the suspended run's entry untouched (behavior unchanged). Package suite green.
Scope note: observability-only (no correctness impact) and only on DBOS Branch 1, but landed as a clean guard to tighten cross-runtime observability parity.
FU-DBOS-ONMESSAGE-ONSTATECHANGE: DBOS does not fire onMessage / onStateChange hooks — ✅ closed
Status: ✅ closed — DBOS now dispatches onMessage and onStateChange from the workflow body / executeTool step, matching the cross-runtime canonical order beforeTool → execute → onStateChange → onMessage → afterTool. Surfaced by: packages/e2e/src/__tests__/approval-gate-hook-parity.integ.test.ts approve path + deny path. The previous itOrSkipPendingMessageStateGap gate was lifted on dbos-postgres; both rows now run.
Closure summary: added the missing hook fires in two sites that mirror the JS / Temporal / CF emission points:
onStateChange— insiderunExecuteTool(packages/runtime-dbos/src/steps/execute-tool.ts), fired whentracker.hasChanges()is true. Payload{ previousState, newState, source: 'tool', toolName }matchesrun-loop.ts:2063exactly.previousStateis astructuredCloneofcustomStatecaptured BEFORE the tool runs. Hook throws are NON-FATAL (parity withrun-loop.ts:2072-2081— logged atwarnwith errorIddbos.execute_tool.on_state_change_hook_failed, tool result preserved).onMessage— in the workflow body (packages/runtime-dbos/src/workflows/shared.ts), fired AFTER appending: (a) the assistant message, (b) each phase-1 tool-result message (including the approval-gate deny synthetic tool-result), (c) the phase-2 (finishWith) tool-result, (d) the phase-3 (companion) tool-result. Routed throughargs.dispatcher.runHookso the invocation is checkpointed byHookStep. Hook throws are FATAL (parity withstep-iterator.ts:734/:1235returning{ kind: 'failed' }— the throw propagates out of the workflow body so DBOS retry / recovery surfaces the failure).
Canonical order fix: to land onMessage between onStateChange and afterTool, the afterTool invocation was moved OUT of runExecuteTool and deferred to the workflow body. ToolStepResult now carries durationMs?: number as the deferred-afterTool signal — present on results from runExecuteTool (regular tool / approve-resume tool), undefined on client-tool / deny / unknown-tool results (the client-tool path fires its own afterTool inline; deny never fires afterTool; unknown-tool never reached tool.execute). This mirrors runtime-js's deferredAfterTool pattern (run-loop.ts:2173-2180 + step-iterator.ts:1238-1257) and converges DBOS on the same cross-runtime order as runtime-temporal's in-activity sequencing (activities.ts:545-549).
Locked by: 9 new unit tests (5 in execute-tool.test.ts + 4 in run-agent-loop-one-turn.test.ts, later expanded to 6 in the fix-round with strengthened canonical-order + phase-2 finishWith + phase-3 companion coverage) + 1 new integration test + 2 newly-passing e2e parity rows.
- Unit:
__tests__/execute-tool.test.ts(onStateChange payload + decoupling + non-fatal throw + step-body order);__tests__/run-agent-loop-one-turn.test.ts(assistant onMessage, phase-1 tool-result onMessage, canonical order, fatal throw, phase-2 finishWith onMessage + deferred afterTool, phase-3 companion onMessage). - Integ:
__tests__/integration/on-message-on-state-change.integ.test.ts(real Postgres; full canonical order assertion with order counter + strictly-monotonic-and-contiguousmessageIndexassertion). - E2e:
approval-gate-hook-parity.integ.test.tsapprove + deny rows ondbos-postgres.
GL #111: with this closed, all 5 #111 gaps are resolved AND the DBOS cross-runtime hook surface is at full parity with JS / Temporal / CF for the { beforeTool, afterTool, onMessage, onStateChange } set (for the assistant + tool-result message paths; per-user-message onMessage cross-runtime divergence is tracked separately under FU-USER-INPUT-ONMESSAGE-CROSS-RUNTIME).
FU-USER-INPUT-ONMESSAGE-CROSS-RUNTIME: per-user-message onMessage fires on JS/CF but not on Temporal/DBOS — ✅ closed
Status: ✅ closed — joint Temporal + DBOS fix landed on feat/dbos-followups-post-111 (FU sweep #2). No JS / CF code changed — this brought the lagging runtimes up to the reference behavior.
Symptom (before closure): the onMessage hook fires for every appended message on runtime-js and the Cloudflare DO path (via runtime-cloudflare/src/steps.ts) — including each user-input message at the top of a turn — but did NOT fire for user-input messages on runtime-temporal and runtime-dbos. All four runtimes already agreed on assistant + tool-result message fires.
Verified citations (still valid for the reference runtimes):
- JS fires per-user-message
onMessageafter the initial append +onAgentStart:packages/runtime-js/src/run-loop.ts:542-551. - CF DO fires per-user-message
onMessageafter the initial append:packages/runtime-cloudflare/src/steps.ts:611-622.
Closure:
runtime-temporal(packages/runtime-temporal/src/activities.tsininitializeAgentState— declared at line 4824, fire site immediately after theonAgentStartsave block): added a per-user-messageonMessageloop usinginvokeHookWithStateTracking.messageIndexis computed frompreAppendMessageCount(read from the durable message log viastateStore.getMessageCounton continuations, derived fromconversationMessages.lengthon fresh sessions) plus the loop index — mirroring the JS reference'sstate.messages.length - input.newMessages.lengthformula across multi-message inputs and continuations. Resume entries continue to be handled byapplyResultsAndReload(the per-message firing loop ~line 2512 within the function declared at ~line 2086) which firesonMessagefor drained tool-result messages only.runtime-dbos(packages/runtime-dbos/src/workflows/shared.tsinrunAgentLoopOneTurn, immediately after theonAgentResumed/onAgentStartbranch): capturedpreAppendMessageCountviadispatcher.getMessageCount({ sessionId })BEFOREappendMessages(replay-safe throughLoadStateStep.getMessageCountStep, which delegates toSessionStateStore.getMessageCount— an O(1) indexed count that avoids thegetMessages10k cap), then fires per-user-messageonMessageviadispatcher.runHookgated byargs.isResume !== true. Persistent-mode coverage is automatic: eachrunAgentLoopOneTurncall inpersistent-workflow.ts(lines 425, 520, 585) is invoked withisResumedefaulted (= false), so every inbox-message turn fires per-user-messageonMessage— matching how JS fires it on eachexecute()call. Locked in bypackages/runtime-dbos/src/__tests__/integration/onmessage-persistent.integ.test.ts.
Test that locks this in: packages/e2e/src/__tests__/user-input-onmessage-parity.integ.test.ts. The harness-driven matrix runs 4 scenarios (single-message initial, multi-message initial, continuation pre-append-count, resume-path suppression) across 7 Node-context backends (js-{memory,redis,postgres}, temporal-{memory,redis,postgres}, dbos-postgres). Pre-closure: 12 PASS on js-{memory,redis,postgres}, 16 FAIL on temporal-{memory,redis,postgres} + dbos-postgres. Post-closure: 28/28 GREEN. CF DO is the reference implementation (runtime-cloudflare/src/steps.ts:611-622) and is not enumerated as a tested backend here — workerd-context coverage is deferred to the future IMPL_PENDING_O5 lift. Persistent-mode DBOS coverage is locked separately by packages/runtime-dbos/src/__tests__/integration/onmessage-persistent.integ.test.ts.
Changesets: .changeset/temporal-user-input-onmessage.md + .changeset/dbos-user-input-onmessage.md (both patch).
FU-TEMPORAL-HISTORY-PREFIX-PRE-CLAIM-DROP: history-prefix dropped on Temporal fresh sessions because executor pre-claims — ✅ closed
Status: ✅ closed — bounded fix in initializeAgentState (Option A from the original fix sketch). The brainstorm-corrected condition is existing !== null && existingMessageCount === 0 (read via stateStore.getMessageCount), not existing.messages.length === 0 — the latter wouldn't typecheck because SessionState has no messages field (messages are stored separately and read through the store's message API).
Surfaced by: FU#2 fix-round (feat/dbos-followups-post-111) attempt at a "fresh session with history: prefix" scenario in packages/e2e/src/__tests__/user-input-onmessage-parity.integ.test.ts. The scenario passed on JS but failed on Temporal with messageIndex === 0 instead of === history.length.
Symptom (pre-fix): when executor.execute(agent, { message, history: [...] }, { sessionId }) ran against a fresh session on runtime-temporal, the history: prefix was silently dropped from the message log AND the per-user-message onMessage fired with messageIndex === 0 (instead of === history.length).
Root cause: Temporal's executor pre-claims the session via stateStore.createSession() at packages/runtime-temporal/src/executor.ts:434-443 BEFORE the workflow starts (this is by design — it closes a startup race with concurrent execute() calls). By the time initializeAgentState activity runs:
existing !== nullis always true (the session was just created by the pre-claim).- The
messagesToAppendternary atactivities.ts:5096-5097takes thenewUserMessages-only branch (skippingconversationMessages). - The
preAppendMessageCountternary atactivities.ts:5115-5118takes thegetMessageCountbranch, which returns 0 (the pre-claim created an empty session).
Net effect: conversationMessages (the resolved history prefix) was computed at line 4934 but never used on fresh sessions when the executor pre-claims. The two ternaries were structurally unreachable for the existing === null branch on the user-input entry point.
Closure: runtime-temporal/src/activities.ts initializeAgentState now reads existingMessageCount once via stateStore.getMessageCount (only when existing !== null to avoid an unnecessary round trip) and derives treatAsFresh = existing === null || existingMessageCount === 0. Both messagesToAppend and preAppendMessageCount derive from treatAsFresh — pre-claimed-but-empty rows now take the history-prefix prepend path, with preAppendMessageCount === conversationMessages.length. The existing customState merge / version carry-over branch above (sessionState = { ...existing, ... }) still runs against existing regardless — only the message-log derivation changes. Continuations (real prior messages, existingMessageCount > 0) take exactly the pre-fix path. Side-effect audit: fresh+no-history is unchanged (conversationMessages.length === 0), and the second existingMessageCount read used by preAppendMessageCount is deduplicated against the one feeding treatAsFresh.
Test that locks this in: Scenario 5 in packages/e2e/src/__tests__/user-input-onmessage-parity.integ.test.ts ("fresh session with history prefix"). Asserts (a) the persisted message log starts with the 3-entry history prefix followed by the new user message; (b) onMessage fires exactly once for the new user message with messageIndex === history.length; (c) the run record's startMessageCount (when the store tracks it) is at least history.length + 1. DBOS is skipped pending its sibling FU (FU-DBOS-HISTORY-PREFIX-NEVER-HONORED) because DBOS discards history: entirely at normalizeAgentInput — separate root cause.
Changeset: .changeset/temporal-history-prefix-pre-claim.md (patch bump).
Scope clarification: This was NOT a regression introduced by FU#2 — the history-prefix-drop predated FU#2; FU#2 simply added an onMessage fire site whose messageIndex derivation surfaced the symptom. The onMessage cross-runtime parity (the FU#2 invariant) held on the user-input paths that scenarios 1-4 exercise; scenario 5 was attempting to extend coverage and surfaced this separate bug, which is now also closed.
FU-TEMPORAL-HISTORY-PREFIX-ON-CONTINUATION: Temporal drops caller-provided history: on continuations — ⛔ reframed / superseded
Status: reframed — the original "Temporal drops it, CF appends it, match CF" framing was WRONG. Superseded by FU-CROSS-RUNTIME-HISTORY-CONTINUATION-DOUBLING (below). Temporal's drop turns out to be the least-broken behavior; "matching CF" would ship a history-doubling bug into Temporal.
Why reframed (Wave 1 investigation, full 4-runtime read): history: is a fresh-session seeding input (see AgentInputObject.history JSDoc at packages/core/src/types/executor.ts:64-74 — "provide the full conversation history directly" / caller-owns-the-store; and the documented continuation pattern in docs/guide/concepts.md:184-198 passes only { message } and relies on auto-loaded history, with the explicit "stored once per session, not duplicated per run — O(n) not O(n²)" principle). Passing history: on a session that already owns prior messages is undefined / unsupported misuse, and EVERY runtime that honors it on a continuation produces a doubled durable log:
- JS (
packages/runtime-js/src/js-agent-executor.ts:3201-3203, 3246, 3275-3280): on a same-session continuation it appends[...inputHistory, ...newMessages]ON TOP of the already-persisted prior messages → durable log doubled; the in-memory LLM view (:3201override) also diverges from the store. - CF (
packages/runtime-cloudflare/src/steps.ts:538-541, 596-597): appends[...providedHistory, ...userMessages]unconditionally; on a continuation-with-history the prior messages are already stored → doubled;onMessagemessageIndex(:617) is also miscomputed against the provided-history length. - DBOS (post-
FU-DBOS-HISTORY-PREFIX-NEVER-HONORED,packages/runtime-dbos/src/workflows/shared.ts): the history-prefix prepend is gated onargs.isResume !== true, so a continuationexecute()(which isisResume !== true) also prepends → same doubling. (Chosen deliberately for parity with JS/CF on the supported fresh-session path; see that FU's closure.) - Temporal (
packages/runtime-temporal/src/activities.ts treatAsFreshbranch): the ONLY runtime that does NOT double — it dropshistory:on continuation. Silent, but not corrupting.
So the real defect is cross-runtime history-doubling on continuation, NOT a Temporal-specific gap. Tracked as the FU below.
FU-CROSS-RUNTIME-HISTORY-CONTINUATION-DOUBLING: history: on a continuation doubles the durable message log (JS / CF / DBOS) — ✅ closed
Status: ✅ closed — gated the history: prefix on an empty durable log in ALL runtimes. On a non-empty session history: is now a no-op (drop) + uniform <runtime>.history_dropped_on_continuation warn (js. / cf. / dbos. / temporal., carrying { sessionId, droppedCount, existingMessageCount }); the fresh-session seeding path is unchanged. Per-runtime predicates: JS (js-agent-executor.ts) honors inputHistory only when existingMessageCount === 0 on an implicit continuation (the in-memory LLM view now matches the store; branch path unaffected); CF (steps.ts) prepends providedHistory only when the durable log is empty AND fixed the onMessage messageIndex to derive from the durable count (existingMessageCount + conversationMessages.length + i); DBOS (workflows/shared.ts) tightened the gate from !isResume to !isResume && preAppendMessageCount === 0; Temporal (activities.ts) kept its already-correct treatAsFresh drop and added the observable warn. Locked in by the cross-runtime parity test packages/e2e/src/__tests__/history-continuation-noop-parity.integ.test.ts (asserts no doubling + fresh-seed preserved) plus focused per-runtime unit tests (js-agent-executor-continuation.test.ts, runtime-cloudflare/src/__tests__/steps.test.ts, run-agent-loop-one-turn.test.ts, runtime-temporal/.../activities-hooks.test.ts). RED→GREEN teeth-verified: js-memory failed (doubled) before the gate and passes after; temporal-memory passed throughout (it already dropped); reverting the JS gate re-reds the parity test. Changeset: .changeset/cross-runtime-history-continuation-no-double.md. Surfaced by: Wave 1 history-prefix investigation (the read that reframed FU-TEMPORAL-HISTORY-PREFIX-ON-CONTINUATION).
Symptom: executor.execute(agent, { message, history: [...] }, { sessionId }) against a session that ALREADY has persisted messages appends the provided history ON TOP of the existing log on JS, CF, and DBOS — duplicating the prior conversation in the durable store (and, on JS, diverging the in-memory LLM view from the store). Temporal alone drops it (silently). No runtime produces a correct, non-duplicating result. history: is intended as fresh-session seeding (per AgentInputObject.history JSDoc + docs/guide/concepts.md continuation pattern).
Clean cross-runtime fix (the portable contract): gate the history: prefix prepend on "the durable message log is empty" in ALL runtimes — i.e. promote Temporal's already-correct fresh-only predicate (existingMessageCount === 0) into the portable contract, rather than promoting CF's unconditional-prepend (the doubling behavior):
- JS
js-agent-executor.tscontinuation path: only seedhistorywhen the loaded session has no prior messages. - CF
steps.ts:538-597: gate the[...providedHistory, ...]prepend on empty existing state; fix theonMessagemessageIndex(:617) to use the durable count. - DBOS
shared.ts: change the prefix gate from!isResumeto!isResume && preAppendMessageCount === 0(DBOS already readsgetMessageCountpre-append, so this is a one-line gate tightening). - Temporal
activities.ts: keep the drop, but make it observable —logger.warnwitherrorId: 'temporal.history_dropped_on_continuation'when!treatAsFresh && conversationMessages.length > 0(and, ideally, all four runtimes warn uniformly whenhistory:is passed on a non-empty session).
Lock it in: a cross-store/cross-runtime parity test (in the packages/core/src/testing harness family + an e2e parity row) asserting that history: on a non-empty session does NOT duplicate the durable log on ANY runtime (and either no-ops or warns uniformly). This is the right place to enforce the single portable contract rather than per-runtime conditionals.
Out of scope for Wave 1: this is a focused cross-runtime change touching all four runtimes + a new parity test; it should be its own MR. Wave 1 fixed the supported fresh-session path (FU-DBOS-HISTORY-PREFIX-NEVER-HONORED) and reframed the continuation question to this FU.
FU-TEMPORAL-INPUT-STATE-ON-CONTINUATION: Temporal applies input state over the committed state on a continuation — ✅ closed (CP-228: the entry activity honours it on a fresh entry only and logs temporal.state_dropped_on_continuation with the dropped keys otherwise; an activity retry after the commit reads freshness from the committed run)
U23 / CP-223 made input state (AgentInputObject.state) seed a fresh session only on runtime-js, the CF Durable Object and CF Workflows, with each runtime's history: gate (above). Temporal's entry activity still prefers it on every entry: runtime-temporal/src/activities.ts:9752-9755 (origin/main cb84ce3406), resolvedInitialState = input.initialState !== undefined ? input.initialState : (baseCustomState ?? input.customState). The conformance owner's s116 saw state.continuation-input-state-keeps-committed red ×3 on the Temporal lanes (.committed-state-kept). The fix is the same gate: honour it only when the activity's treatAsFresh holds (the history drop's predicate), else keep the committed state and log temporal.state_dropped_on_continuation with the dropped keys. DBOS is CP-224 (input state never applied at all).
FU-177-MR2-TEMPORAL-DBOS-ENTRY-STATE: per-turn entry state, start signals and statePatch on Temporal and DBOS (#177 MR 2) — open
Why: #177 MR 1 implements per-turn entry state, the start commit and one start signal per run on runtime-js, the Cloudflare Durable Object and Cloudflare Workflows. Temporal and DBOS keep their previous hook behaviour (onAgentStart fires for every non-resume run, a retry included, with startSignal: true; onAgentRetry never fires), and refuse a non-empty statePatch typed (framework_not_supported, core refuseStatePatchUntilSupported, orchestrator ruling (a), 2026-10-26) rather than drop it silently.
Do: #177 MR 2: wire planEntryState, the run-start commit's entryState, the start commit, the start-signal rules and the stash patches into both runtimes; remove refuseStatePatchUntilSupported and its two unit pins (runtime-temporal executor-state-patch-refusal.test.ts, runtime-dbos dbos-agent-executor.test.ts "#177" block). When Temporal adopts start commits, also remove core planRetry's entry state commit candidate (the non-terminal state commit at the run-start count, kept for Temporal's writeEntryState; Greptile on !554) and its pin in retry-plan.test.ts. Flip writesTurnStartCommit (packages/e2e/src/helpers/checkpoint-truth-matrix.ts) to true for temporal / dbos, so the matrix's boundary.*.input-commit cell expects exactly the rowless start commit there too.
FU-177-ONAGENTSTART-INPUT-PARITY: owed start work gives onAgentStart and onMessage the same input rows — open
Filed by: the review of #177 MR 1 (!554), 2026-10-27. When an entry owes its turn's start but brings no new messages (a recovery before the start commit, a rewind past it, a continue that owes it, a CF Workflows successor), onMessage gets the turn's input (core turnInputMessages), but onAgentStart's input is [], or on runtime-js every user row in the log when newMessages is undefined (run-loop.ts's start block).
Wanted: onAgentStart's input is the same rows onMessage gets for that entry (the entry's new messages, else the turn's input), on runtime-js, the CF DO and CF Workflows (and Temporal / DBOS with #177 MR 2), with a unit pin per runtime.
FU-CHILD-IDENTITY-MR2-REMOTE-STATEPATCH: a remote child's verdict statePatch needs the producer to opt in — open (owner: child-identity MR 2)
Filed by: the agent-server statePatch gate (WE-2; orchestrator ruling 2026-10-27, security review of !554). agent-server's HTTP /start and /resume refuse a body statePatch unless the producer's host opts in (AgentServerConfig.clientState: { statePatch: 'accept' } or a ClientStateHook; by default the typed 400; /start's state stays accepted). Child-identity MR 2 sends the child-dispatch verdict's statePatch (#177 spec §2.5) in a child's start. Wanted: MR 2's plan includes, for a REMOTE child: the producer opts in (docs for the producer's host, the cross-service guide), and a cross-service test where a verdict patch reaches a remote child through an opted-in producer, plus one where a default producer refuses it typed.
FU-177-SPEC-CONSUMED-PENDING-SENTENCE: the #177 spec names ConsumedPendingChangedError — ✅ done (#177 MR 1, after !541 merged)
Filed by: #177 MR 1 (WE-2; orchestrator ruling 2026-10-27). MR 1 adds core ConsumedPendingChangedError (an append's consumePending refusal: state_concurrency_conflict, reason: 'consumed_pending_changed', non-retryable; agreed with the design session). The #177 spec lives in !541, still open, so its §2.2 sentence naming the error cannot land with MR 1. Wanted: when !541 merges, add that one §2.2 sentence and the as-built note. Done: !541 merged first; #177 MR 1 (rebased onto it) adds the §2.2 sentence and the spec's §10 "As built".
FU-DBOS-STARTMESSAGECOUNT-IGNORES-SEEDED-HISTORY: DBOS run-record counts ignore the seeded history: prefix on a fresh session → msg-N UI-id collision on reconnect/SSR — ✅ closed
Status: ✅ closed — RECLASSIFIED from "low priority / arguably defensible metadata nuance" to a real UI-correctness bug. The "unverified concern" in the original writeup (does the same pre-run snapshot cause a msg-N id collision?) was VERIFIED to be a real collision. Bounded one-runtime executor change (no workflow-body / replay-determinism change). Surfaced by: Wave 1 — the e2e user-input-onmessage-parity Scenario 5 DBOS row (Assertion 3 on startMessageCount).
Symptom: on a fresh-session execute(agent, { message, history: [...] }, { sessionId }), DBOS recorded run.startMessageCount === 0 AND run.startUIMessageCount === 0, because the executor captures both counts from the store BEFORE DBOS.startWorkflow (packages/runtime-dbos/src/lifecycle/execute.ts, read pre-workflow) while the history prefix + new user message are appended INSIDE the workflow body (runAgentLoopOneTurn, shared.ts). JS/Temporal capture both counts post-append (= history.length + newMessages.length). The persisted message log + the per-user-message onMessage messageIndex ARE correct on DBOS (proven by Scenario 5 Assertions 1+2); only the run-record baseline counts diverged.
Root cause (the real bug): the UI-id derivation keys off startUIMessageCount. On the live/reconnect/SSR path the in-flight assistant bubble's id is deriveExistingMessageId(startUIMessageCount) → msg-${startUIMessageCount} (ai-sdk/src/handler/handle-chat-stream.ts; the same value seeds the SSR partial in snapshot.ts:482-488). The persisted converter assigns msg-${uiIndex} to each UI-visible (non-tool, non-hidden) message in order (helix-to-aisdk-converter.ts). With startUIMessageCount === 0 the bubble id is msg-0, which COLLIDES with the FIRST seeded-history message (also msg-0). AI SDK useChat keys by message id, so the bubble dedupes against the history message and is DROPPED on reconnect/SSR — the same bug class as the already-fixed SF-C1 / Review-1-H2 duplicate-bubble issues; the fresh-session-with-history: path was the remaining hole.
Fix: on the FRESH STANDARD path only (gated on standardStartMessageCount === 0, i.e. genuinely fresh), execute.ts now PROJECTS the about-to-be-seeded history into the run-record counts via a new projectSeededHistoryCounts(input, base, base) helper. The helper parses the input with parseAgentInput (mirroring the body's normalizeAgentInput) and applies the SAME m.role !== 'system' filter as shared.ts's filteredHistory; it returns null (→ keep base counts) when the input doesn't parse as AgentInput or carries no honored history, so the no-history fresh path is unchanged. Both projected counts are base + filteredHistory.length + newMessages.length (JS-parity). The same projection is mirrored on the PERSISTENT fresh-start path (!sessionExisted → its body runs the same runAgentLoopOneTurn), carried ONLY into the START-path initialRunMeta; the persistent SEND/continuation path keeps the unprojected base counts (a live workflow does not re-seed history — its preAppendMessageCount > 0 drops the prefix). The branch path was already correct (it counts messagesPage.total post-clone) and the standard continuation path keeps the SF-C1 UI-merged base counts — neither is touched.
Gate-match + JS-parity reasoning (the careful part): the body seeds iff args.isResume !== true && preAppendMessageCount === 0 && filteredHistory.length > 0. On both fresh executor paths isResume is false and the durable count is 0, so the gate reduces to "history present" — exactly what the helper checks. Value note: the SPEC originally proposed startUIMessageCount = base + history.length (excluding the new user message). That was WRONG and would have left the collision in a different slot: the in-flight assistant bubble's UI index is the count of UI-visible messages that existed when the run STARTED, and the new user message is part of the turn's input (persisted before the run record), so it MUST be counted. JS reads startUIMessageCount = loadUIMessageCount AFTER persisting [...filteredHistory, ...newMessages] (js-agent-executor.ts _initializeForExecute), i.e. history.length + newMessages.length. For a 3-entry history + 1 new user message the correct value is 4 → msg-4, which is collision-free (history = msg-0..2, new user = msg-3) AND matches the committed assistant id. We implemented the JS-parity value (4), NOT the spec's history.length (3), to avoid introducing a NEW divergence. startMessageCount and startUIMessageCount coincide here because none of the seeded rows are tool rows.
Teeth: unit coverage in packages/runtime-dbos/src/__tests__/lifecycle/execute.test.ts (two new tests: "fresh session with history: projects seeded history + new message into runMeta counts" asserts intercepted runMeta.startMessageCount === 3 and startUIMessageCount === 3 for a 2-entry filtered history + 1 message on a fresh session; "…excludes system messages from the projection" asserts a system-bearing 3-entry history still projects to 3, mirroring shared.ts's filter). Verified RED-first (pre-fix → both 0); GREEN after; teeth-verified (reverting the standard-path runMeta wiring back to the base counts → both RED again, expected 0 to be 3; restored → GREEN). The e2e gate in user-input-onmessage-parity.integ.test.ts Scenario 5 un-gated the DBOS row of Assertion 3 (startMessageCount >= history.length + 1) AND added Assertion 4 (no-collision): run.startUIMessageCount === history.length + 1 and the derived msg-${startUIMessageCount} is not any seeded-history msg-N. Verified RED-first on dbos-postgres (pre-fix → expected 0 to be greater than or equal to 4); GREEN after, all 35 backend×scenario runs pass (JS/Temporal already passed the un-gated assertion). Changeset: .changeset/dbos-fresh-history-uimsgcount.md.
FU-TEMPORAL-CONTINUATION-RUN-RECORD: Temporal execute() continuation does not write a run record (listRuns under-reports turns) — ✅ closed
Status: ✅ closed (Wave 4) — bounded one-runtime executor change, no workflow-body / replay-determinism change. Surfaced by: Wave 2's execute() continuation floor e2e parity test (packages/e2e/src/__tests__/session-model-continuation.integ.test.ts), whose runs.length >= 2 assertion failed on temporal-{memory,redis,postgres} (got 1).
Symptom: a continuation execute(agent, msg, { sessionId }) on a completed Temporal session does NOT create a second run record. stateStore.listRuns(sessionId) returns 1 run after two execute() turns, vs 2 on JS / DBOS / CF. Root cause: the executor's createRun call is gated behind if (!existingState) (packages/runtime-temporal/src/executor.ts:434, createRun at :488-500), so only the FIRST execute() writes a run record; continuation turns reuse the sessionId-derived workflow id (Temporal's default workflow-id-reuse) and skip createRun. This defeats the F3 intent (observable runs via listRuns) for continuation turns.
Reference behavior: DBOS writes a run record per execute() turn (the DBOS continuation test in session-model-continuation.integ.test.ts asserts runs.length === 2); JS / CF create per-turn run records too. Temporal is the outlier.
Fix sketch: hoist createRun out of the if (!existingState) guard so a continuation execute() writes a run record with a fresh runId and turn = priorRuns + 1 (load prior run count or derive turn from the session). Care needed around: (a) the workflow-body's run-status transition (it currently transitions the run created at start), (b) the runId vs the reused sessionId-derived workflow id, (c) startMessageCount/startUIMessageCount capture for the continuation turn. This is a contained Temporal-runtime change deserving its own MR + 4-battery (turn numbering + workflow-id-reuse interaction).
Scope (historical): Wave 1 gated the floor test's runs.length >= 2 assertion to non-Temporal runtimes (with a comment), keeping it as real teeth on JS/DBOS/CF and excluding Temporal pending this FU. The floor test still asserted the portable contract (message-history continuity) on Temporal, which passed.
Closure (Wave 4): the createRun call in TemporalAgentExecutor.execute() was hoisted out of the if (!existingState) claim guard, so EVERY execute() turn writes a fresh run record — not just the first. The metadata is computed per-branch:
- Fresh session (
!existingState):turn=1,startSequence=0, message counts from the inboundhistory + newMessages(unchanged behavior). - Continuation turn (
existingStatepresent):turn = listRuns().total + 1,startSequence = getStreamInfo().latestSequence(live stream head),startMessageCount = getMessages().total,startUIMessageCount = loadUIMessageCount(...)— the same cursorsresume()already uses.createSessionremains gated to the fresh path (a continuation session already exists).
stateStore.listRuns(sessionId) now reports per-turn runs on Temporal exactly like JS / DBOS / CF (distinct runIds, monotonically-increasing turn).
What this fix restores — and what it does NOT: the hoist restores run-record EXISTENCE + per-turn turn numbering. It does NOT add a terminal run-status transition. The ONLY in-workflow run-status write is commitSuspendedStep.updateRunStatus(input.state.runId, ...), which fires only on a SUSPENSION boundary; it targets THIS turn's runId (threaded via workflowInput.runId → projectSessionToAgent), so a suspending turn transitions its own record independently. A plain completing continuation never hits that path — its run record stays at the initial running status the executor wrote at createRun time. The terminal completion path (persistTerminalState) does not touch run records at all (so there was never a stale-run-transition risk either way). Run records are observability-grade — nothing load-bearing reads the terminal status.
Replay determinism (the careful part): createRun runs in the executor (client side), NOT inside the deterministic workflow body — so there is no workflow-history change and no new non-determinism. The workflow reads input.runId (a plain serialized field) deterministically on every replay. The continuation createRun (and the metadata reads that feed it) is best-effort: failures are logged at warn and do not fail execute(). Because those metadata reads run AFTER startWorkflow, they live INSIDE the same best-effort try/catch as createRun (Waves 3-5 deep-review FIX 1) — a read hiccup degrades to a skipped run record rather than rejecting execute() and orphaning the already-started workflow. This matches the fresh-session and resume() paths.
Teeth: integration coverage in packages/runtime-temporal/src/__tests__/integration/multi-turn-continuation.integ.test.ts ("completes two-turn continuation and sees turn-1 history") now asserts listRuns(sessionId).runs.length === 2, distinct runIds, turn 1 then 2, and runs[1].startMessageCount > 0. Verified RED-first (reverting the hoist → expected 1 to be 2); GREEN after. Unit coverage in packages/runtime-temporal/src/__tests__/executor-run-records.test.ts (the former "does NOT call createRun when continuing an existing session" test, flipped to "calls createRun (turn = priorRuns + 1) on a continuation execute()") pins turn=2 + the live-cursor metadata. The single-turn case is still exactly 1 run (the fresh-session createRun test asserts toHaveBeenCalledTimes(1)). The previously-gated Temporal exclusion in the cross-runtime execute() continuation floor e2e parity test was removed — listRuns(sessionId).runs.length >= 2 now runs on Temporal too. Changeset: .changeset/temporal-continuation-run-record.md.
FU-DBOS-HISTORY-PREFIX-NEVER-HONORED: DBOS executes drop the history: input prefix entirely — ✅ closed
Status: ✅ closed — bounded one-runtime fix (initial entry only), no architectural change. Surfaced by: FU sweep #2 brainstorm (Scenario 5 design for FU-TEMPORAL-HISTORY-PREFIX-PRE-CLAIM-DROP).
Symptom (before fix): executor.execute(agent, { message, history: [...] }, { sessionId }) against runtime-dbos silently dropped the history: prefix. normalizeAgentInput (packages/runtime-dbos/src/workflows/shared.ts) extracted only newMessages from parseAgentInput, discarding the history field. The persistent message log started at the new user message; onMessage fired with messageIndex === 0 regardless of history.length. JS / CF / Temporal(fresh) all honored it. The history field IS preserved across the DBOS serialization boundary (plain JSON on StandardWorkflowInput.input) — no serialization gap, purely a normalize-time drop.
Closure: normalizeAgentInput now returns { newMessages, history } (it already had history available from parseAgentInput, which returns { newMessages, initialState, history }). In runAgentLoopOneTurn, on INITIAL entry only (args.isResume !== true), the history prefix is prepended: messagesToAppend = [...historyPrefix, ...userMessages] where historyPrefix = args.isResume !== true ? (history ?? []) : []. The single dispatcher.appendMessages call writes history + new user messages atomically. preAppendMessageCount is still read from the durable log BEFORE the append (timing unchanged) and is now gated on messagesToAppend.length === 0 (was userMessages.length === 0). The resume path is untouched — historyPrefix is empty when args.isResume === true, so resume continues to drain tool-results without re-prepending history.
onMessage scoping (FU-USER-INPUT-ONMESSAGE-CROSS-RUNTIME preserved): the per-user-message onMessage loop fires ONLY for userMessages, never the history prefix (history is prior context, not new user input — matches JS, which fires onMessage only for input.newMessages at run-loop.ts:542-551, not the loaded/provided history prefix). The index math is messageIndex = preAppendMessageCount + historyPrefix.length + i. For a fresh session (preAppendMessageCount === 0) with a 3-entry history, the single new user message fires at messageIndex === 3 === history.length — matching JS's baseIndex = state.messages.length - input.newMessages.length (where state.messages already includes the prepended history, so the formula resolves to history.length). On resume / no-history turns, historyPrefix.length === 0, so the index reduces to the prior preAppendMessageCount + i.
Reference behavior (cross-runtime contract this now matches):
- runtime-js (
packages/runtime-js/src/js-agent-executor.ts:3246):state.messages = [...conversationMessages, ...newMessages]. - runtime-cloudflare (
packages/runtime-cloudflare/src/steps.ts):allMessagesToSave = [...conversationMessages, ...userMessages]. - runtime-temporal (post FU-TEMPORAL-HISTORY-PREFIX-PRE-CLAIM-DROP):
treatAsFresh ? [...conversationMessages, ...newUserMessages] : newUserMessages.
Teeth: unit coverage in packages/runtime-dbos/src/__tests__/run-agent-loop-one-turn.test.ts ("prepends history prefix on initial entry; onMessage fires only for user message at index history.length") asserts (a) the first appendMessages call carries [h1, h2, h3, newUserMsg], and (b) onMessage fires exactly once — for the new user message at messageIndex === history.length, NOT for either historical user entry. Verified RED-first (history dropped → first append carries only the user message → fails); GREEN after the prepend. Reverting the prepend (messagesToAppend = [...userMessages]) re-reds the append assertion; dropping the + historyPrefix.length index offset re-reds the messageIndex assertion. The e2e gate in packages/e2e/src/__tests__/user-input-onmessage-parity.integ.test.ts Scenario 5 (un-skipped on DBOS by this closure) is the cross-runtime contract: history lands in the log, messageIndex === history.length on a fresh session — assertions the unit test mirrors exactly.
FU-DBOS-PERSISTENT-HOOK-CLEANUP: explicit cleanup of HookManagerRegistry entries on persistent-workflow terminate — ✅ closed
Status: ✅ closed — single-call fix, no architectural change. Surfaced by: packages/runtime-dbos/src/lifecycle/execute.ts executePersistentImpl start path. Persistent workflows are long-lived (recv-loop until idle-timeout / abort / crash recovery), so the getResult() .finally() cleanup approach used by standard / resume / retry did not apply.
Symptom (before fix): in long-running processes that spawn many persistent agents and let them terminate via idle-timeout, the HookManagerRegistry retained entries for terminated workflows until process exit. Bounded by process lifetime — no production impact for typical short-lived workers; observable as growing-Map memory for long-lived processes managing thousands of persistent agents.
Closure: added HookStep.cleanupPerCallHookManagerStep (@DBOS.step, idempotent because HookManagerRegistry.delete is a Map.delete no-op when the key is absent — safe across replay), called directly from the persistentAgentWorkflow body's try/finally wrapper. The try/finally was hoisted (in FU#4 fix-round H3) to wrap the entire workflow body — beginRun, the LiveModelRegistryHolder lookup, AND runPersistentLoop all sit under the same cleanup umbrella, so the step fires on every terminal path: idle-timeout, shutdown via inbox, shutdown via control, recv cancellation, uncaught errors propagating out of the recv loop, AND pre-loop setup throws from beginRun or the liveModel missing-check. DBOS.workflowID is captured at the top of persistentAgentWorkflow and used directly in the same scope's finally so the cleanup uses the same key that executeImpl used at HookManagerRegistry.set time (no threading through RunPersistentLoopArgs needed). Cross-worker recovery semantics are unchanged: a recovered worker doesn't have the per-call registry entry to begin with and falls back to the constructor-bound HookStep.hookManager, matching the documented split-brain shared with runtime-cloudflare and runtime-temporal.
Teeth: workflow-body assertions live in packages/runtime-dbos/src/__tests__/persistent-workflow-cleanup.test.ts (mocks the DBOS SDK to drive the workflow body and asserts HookStep.cleanupPerCallHookManagerStep is called with the right workflowId on each terminal path), and direct unit coverage for the step body lives in packages/runtime-dbos/src/__tests__/cleanup-step.test.ts (registry delete + idempotency). Commenting out the new await HookStep.cleanupPerCallHookManagerStep({ workflowId }) call in the persistent workflow body locally fails the cleanup-test assertions with the original symptom; restoring re-greens.
Follow-up (cross-runtime, not DBOS-specific): an empty/whitespace submitToolResult({ error: '' }) passes through to an empty tool_end.error string (humanizeClientToolError's default branch returns the code verbatim). This is identical across JS / Temporal / Cloudflare / DBOS, so it should be fixed once in packages/core (either the humanize-error.ts default branch or a .min(1) on the submit error schema). Tracked here; out of scope for Batch B.
Design + plan: docs/superpowers/specs/2026-05-23-dbos-approval-gate-suspension-design.md.
FU-DBOS-PERSISTENT-HOOK-CLEANUP-AFTER-CANCEL: a stopped persistent loop leaks its HookManagerRegistry entry — open (LOW, parked: DBOS; owner: the DBOS bucket)
Surfaced by: WA's slot dbos-probe (2026-11-14, origin/main 3fee8e37bf), checked by the owner. Every iteration whose persistent loop a stop cancelled (57 of 60, the correct outcomes included) logged warn runtime-dbos: persistentAgentWorkflow HookManagerRegistry cleanup failed — entry may leak (dbos.hook_manager_registry.cleanup_failed, error Workflow <id> has been cancelled); the 3 loops a stop left running (CP-270) did not.
The closed FU-DBOS-PERSISTENT-HOOK-CLEANUP above runs the cleanup as a DBOS step, HookStep.cleanupPerCallHookManagerStep, from the workflow body's finally (runtime-dbos/src/workflows/persistent-workflow.ts:270-281 at 3fee8e37bf). Its closure lists recv cancellation among the covered paths, but a cancelled workflow cannot run a step, so on every stop that cancels a loop the step throws, the warn is logged and the entry stays until the process exits. The idle-timeout and shutdown exits are covered.
Fix direction: the delete is a process-local Map.delete, so run it outside a DBOS step on the cancel path (or in a non-step finally), and add a unit check that a cancelled workflow body removes its entry.
FU-DBOS-PERSISTENT-HANDLE-RUN-RECORD-WAIT: the persistent-mode execute handle answers result() from one session read (RM-127) — done (!895, batch 27; owner: WA)
Filed: 2026-11-15, from WA's trace while fixing DI-78 (orchestrator ruling 2026-11-15). RM-127 is not the ReconnectHandle: it is the persistent-mode handle's no-DBOS-handle branch, runtime-dbos/src/handles/active-handle.ts:435-467 (origin/main 9ca43ae3ee), which reads the session once, answers completed, else failed / "Unknown error", and keeps that first answer (result() at :193-202). DI-78's fix does not touch it. Fix: route that branch through the same run-record decision and wait as the ReconnectHandle (resultOfRun: core readRunOutcome, follow a superseded run into its continuation, report a suspension, wait while live), and never cache a read that was not final. RM-127's cell, persistent.turn-handle-result-truthful on dbos-postgres, is its proof.
Closed (2026-11-15, !895, batch 27): the persistent turn handle's no-DBOS-handle branch now decides from its own turn's run record through resultOfRunRecord, extracted from DI-78's ReconnectHandle into runtime-dbos/src/handles/run-record-result.ts and shared by both handles. A turn whose run has not started yet (the loop took the message mid-turn) waits while the loop is open; a loop that closed with no run for the turn answers a typed failure (closedLoopTurnFailure, as CF Workflows' closedEntryFailure: cancelled is framework_cancelled, an errored loop keeps the engine's message, one that ended on its own is { message }), never another turn's outcome. A rejected read is no longer kept. RM-127's cell was tightened with the fix: its two "not failed" ids are retired and four positive ids judge that each handle answers completed, and only once its own turn's answer is in the log.
FU-DBOS-PERSISTENT-CT-MULTITURN: persistent recv-loop multi-turn client-tool re-arm — ✅ closed
Status: ✅ closed — bounded one-call fix (not architectural). The it.skip is now it and the test passes in ~1s. Surfaced by: packages/runtime-dbos/src/__tests__/integration/client-tools-persistent.integ.test.ts "client tool inside persistent workflow turn — submit resumes, multi-turn flow continues" (was it.skip).
Closure: the original FU framed this as "unbounded design work / architectural" — that framing was overstated. The actual root cause was a single missing reactivateStream call between persistent recv-loop turns; the fix is the persistent-mode analog of the standard-mode reactivate call that already lives at packages/runtime-dbos/src/lifecycle/execute.ts:441-461 (for standard-mode between-execute() continuation).
Actual root cause: turn N's finalizeRun → endStream puts the session-level stream in 'ended' state. Turn N+1's dispatchClientTool then (a) registers the new pending entry at shared.ts:336-344, (b) calls emitToolStartStep at shared.ts:355, which throws Cannot write to stream <id> in 'ended' state from runEmitToolStart (steps/stream-lifecycle.ts:278-298), and (c) the catch block at shared.ts:365-378 rolls back the pending entry via clearPendingClientToolCallStep and re-throws. Net observable: no visible pending for tc-2, no tool_start chunk on the stream — which is what the test's waitForCondition('turn-2 pending tool tc-2') saw as a timeout. (The persist-then-rollback sequence is observationally indistinguishable from a never-persist failure mode, but the fix-round corrected the FU's earlier "never persisted" framing because future debugging must start from the actual call order.)
Fix (commit 0805c755f + 330182bba + fix-round soft-fail in this branch):
- Added
runReactivateStreamplain function +StreamLifecycleSteps.reactivateStreamStep@DBOS.stepinpackages/runtime-dbos/src/steps/stream-lifecycle.ts. The plain function guards ongetStreamInfoso it is idempotent and safe to call against an'active'stream (no-op) — the underlying Redis store throws when called against an active stream, so this guard is load-bearing. Fix-round soft-fails onstreamManager.reactivateStreamrejection (matches standard-mode.catch(err => warn)atlifecycle/execute.ts:455-460); if reactivate truly fails the nextemitToolStartStepthrows and theshared.ts:365-378catch block rolls back cleanly. - Added
reactivateStreamto theStepDispatcherinterface (packages/runtime-dbos/src/steps/index.ts) and wired it into all five dispatcher construction sites (standard-workflow + persistent-workflow production factories, plus three test mocks). - Called
dispatcher.reactivateStream({ streamId })inpackages/runtime-dbos/src/workflows/persistent-workflow.tsimmediately before BOTHrunAgentLoopOneTurninvocations in the recv loop (drained-message branch at line 436, regular inbox-message branch at line 501). The initialrunAgentLoopOneTurnat line 342 does NOT need reactivation — the stream was just initialized fresh byinitStreamStep.
Teeth: the integ test now (a) state-store invariants for both turns (pending tc-1 / cleared → tc-2 / cleared, both completedClientToolCalls markers), AND (b) chunk-level invariants: after turn-1 ends, capture streamManager.getStreamInfo(sessionId).latestSequence as the high-water; after turn-2 ends, use createResumableReader(sessionId, { fromSequence: turn1HighWater }) to read only turn-2 chunks and assert a tool_start chunk for tc-2 (the literal write that was failing) plus a text_delta containing 'blue' (turn-2 final text). With reactivate removed locally, the test fails with the original waitForCondition timeout AND would fail the chunk assertion (the tool_start for tc-2 is never written because the throw aborts dispatch). Fix-round also adds call-site assertions on the mock dispatcher.reactivateStream in persistent-workflow.test.ts (no-call on initial turn, one call on drained branch, two calls on regular-inbox 2-message run) with comment-out teeth verification on each.
Cross-runtime symmetry: all three stream-manager backends reject writes to 'ended'/'failed' streams identically (packages/store-memory/src/in-memory-stream.ts:216-226, packages/store-redis/src/redis-stream.ts:303-347 Lua + :602-618 JS-side, packages/store-cloudflare/src/stream-durable-object.ts:328-342). The JS and CF runtimes avoid this DBOS-persistent-mode-only failure because their executors reactivate BEFORE each turn (packages/runtime-js/src/js-agent-executor.ts:1763-1771, packages/runtime-cloudflare/src/executor.ts:985-993) — and DBOS standard-mode does the same at packages/runtime-dbos/src/lifecycle/execute.ts:441-461 (with .catch(err => warn) soft-fail). DBOS persistent-mode was the outlier because it processes inbox-message turns inside the long-lived workflow body, bypassing the executor-level reactivate. The fix adds the reactivate inside the recv loop so persistent-mode now matches the other three runtimes' behavior at the workflow-body level (and matches standard-mode's soft-fail semantics — see fix-round in this branch).
Working coverage (single-turn HITL in persistent mode, retained): persistent-client-tool-interrupt.integ.test.ts, persistent-companion-client-tool.integ.test.ts. Multi-turn standard-mode flows are exercised by standard-mode-multi-message.integ.test.ts and the e2e parity matrix.
FU-DBOS-SEED-BRANCH-PERSISTENT-PATHS: extend seedStateSchemaDefaults integ coverage to branch + persistent-start paths — ✅ closed
Status: ✅ closed — dedicated integ coverage added at both call sites.
Surfaced by: integ-fix fix-round (pr-test-analyzer crit 5 follow-on). The helper is wired from THREE call sites in lifecycle/execute.ts — the standard-path entry (~line 494), the branch-from-checkpoint path (~line 227), and the persistent start path (~line 825). The new state-schema-seed.integ.test.ts exercises the standard-path entry comprehensively (spurious-write, dropped-keys warn, legacy-invalid warn); the other two call sites are covered transitively (the helper is the same code), but a dedicated integ test would lock them against future drift if any of the three call sites diverges.
Closed by: two new integ tests in packages/runtime-dbos/src/__tests__/integration/state-schema-seed.integ.test.ts:
- Branch-path seeding (
branch-path: seedStateSchemaDefaults seeds new schema keys on the branch target (FU#5)) — runs a source session with a narrowstateSchema(onlycounter), then branches into a new session viaexecutor.execute(branchAgent, ..., { branch: { fromSessionId, checkpointId } })wherebranchAgentdeclares an extended schema (addsnewKey: z.string().default('default-value')). Asserts (a) the branched session's persistedcustomStatecontainsnewKey: 'default-value', and (b) an observe tool running INSIDE the branched workflow body sees the seeded default viactx.getState(). Teeth-verified: commenting out the branch-siteseedStateSchemaDefaultscall atexecute.ts:227makes the test fail withexpected undefined to be 'default-value'. - Persistent-start seeding (
persistent-start: seedStateSchemaDefaults seeds defaults visible to tools in the persistent workflow body (FU#5)) — defines adefaultMode: 'persistent'agent whosestateSchemadeclarespersistentKey: z.string().default('seeded-default'), executes it withmode: 'persistent', and asserts an observe tool running INSIDE the persistent workflow body sees the seeded default viactx.getState(). Uses a 3 spersistentIdleTimeoutMsso the workflow exits cleanly forcleanup(). Teeth-verified: commenting out the persistent-startseedStateSchemaDefaultscall atexecute.ts:825makes the test fail withexpected undefined to be 'seeded-default'.
Both tests are integration-only and run under RUNTIME_DBOS_INTEG=1; recorded in .changeset/dbos-seed-branch-persistent-coverage.md (patch bump).
FU-GL-113: handle.result() race in client-tools-standard.integ.test.ts timeout test — ✅ closed
Status: ✅ closed — auto-resolved by Batch B's standard timeout test rewrite. Tracks GL #113.
Surfaced by: Batch A's _pollForPendingClientTool addition (commit d3fb7b497) made handle.result() return suspended_client_tool at ~50ms before the in-workflow DBOS.recv deadline fired. The pre-existing timeout test asserted result.status === 'completed' directly and failed deterministically against the new poll-driven semantics.
Closed by: Batch B's client-tools-standard.integ.test.ts rewrite (commit e2ba46569, refined in the Batch B fix-round). The test now exercises the canonical cross-runtime suspend→resume pattern:
execute → handle.result() === 'suspended_client_tool'
→ (no submit; deadline elapses inside the live workflow)
→ resume(agent, sessionId) → resumeHandle.result() === 'completed'Asserts the timeout tool_end chunk (errorCode: 'client_tool_timeout', humanized error text, result undefined) and the tool-result message's CLIENT_TOOL_ERROR_CODE metadata. Header comment at packages/runtime-dbos/src/__tests__/integration/client-tools-standard.integ.test.ts:140-156 documents the "REGRESSION LOCK" narrative. The equivalent e2e timeout-finalization test (agent-server-submit-http.integ.test.ts, Bug #3 in the integ-fix tier-3) was rewritten in the same pattern (commit e2ba46569). All three timeout flows (per-package standard, e2e HTTP, e2e parity scenario 3) now consistently use suspend→resume.
FU-GL-112: v0.7 merge-train pipeline thrash on !199 — ✅ closed
Status: ✅ closed — resolved by runner contention easing; no in-code action required. Tracks GL #112.
Surfaced by: the v0.7 merge train when MR !199 (Batches A→B) ran into 6 stuck no_updates_running jobs from runner contention with the parallel MR !200, followed by transient flakes in:
test:integ:rest— Redis pub/sub timing under load.test:examples:research-assistant-cloudflare-do— Playwright transient flake.test:unit— turbo task-graph race whenai-sdk#teststarted beforeruntime-cloudflare#buildfinished on a cache miss.
Closed by: runner contention easing (the parallel MR !200 merged; cache-miss/contention conditions cleared). All retries eventually succeeded on the original !199 run. The root cause was CI infrastructure capacity, not an in-code defect — there is no code change that would close it. The individual transient-flake patterns are tracked separately:
- Redis pub/sub timing is a known intermittent under heavy CI parallelism; absorbed by the standard 3-retry pattern.
- Playwright DO-example flakes are tracked under the example-app testing discipline (see CLAUDE.md
docs/dev/development-workflow.md). - Turbo task-graph correctness is enforced by the package dependency graph (CLAUDE.md "Package Dependencies"). The specific
ai-sdk#testvsruntime-cloudflare#buildrace was an order-of-arrival artifact under cache-miss + worker-contention, not a missing^builddeclaration.
Out of scope for closure: no individual code change. If similar thrash recurs under future merge trains, options include (a) reducing parallel-MR concurrency on the runner pool, (b) sticky-CPU on the test:unit turbo workers, or (c) bumping the retry count on the affected jobs.
FU-A2-42: Pre-existing runtime-temporal integration test failures (12 tests, mostly conversation continuation + initial-state)
Status: ✅ closed — see "Done items" section. Closed by: commit f4a169588.
FU-A2-41: Redis customState pipeline → Lua atomic script
Status: ✅ done — landed on branch omnara/stateless-suspension-redesign. Surfaced by: second-round adversarial review #5 finding P2-A. Closed by: atomic SAVE_STATE_ATOMIC_SCRIPT in packages/store-redis/src/redis-state.ts.
Description (closed): RedisStateStore.saveState's customState write previously ran as a non-atomic pipeline AFTER the Lua version check: SADD/DEL/RPUSH for arrays + HSET for scalars + orphan cleanup. A crash mid-pipeline left the session hash at the new version while customState was partially updated. An orphan-recovery loop on the next saveState swept up dangling list entries, but the window was unbounded for paused sessions.
Fix: the version check, main hash field write, and customState replacement now run inside a single EVAL call — SAVE_STATE_ATOMIC_SCRIPT. The script:
- CAS-checks
expectedVersionagainst the stored value. - HSETs every main hash field plus the new version in one call.
- Applies the session-key TTL.
- (When customState is provided) clears the prior customState structures, writes the new baseline as scalars (HSET) + arrays (SADD into the array-keys index, RPUSH into per-key lists), and applies the customState TTLs.
The customState replacement is structural: keys absent from the new baseline are dropped, mirroring the in-memory store's deep-clone-then-apply semantics. The orphan-recovery loop is gone because partially-applied state is now structurally impossible — any crash before the script completes leaves the prior state untouched.
Encoding choice — variadic ARGV instead of cjson: the script uses a flat [expectedVersion, ttl, listPrefix, mainCount, ...kvs, 'apply'|'skip', scalarCount, ...sKvs, arrayCount, ...arrays] layout. cjson would be cleaner but ioredis-mock (used by unit-test files under packages/store-redis/src/__tests__/) ships without a Lua cjson binding. The variadic format works on both real Redis and the mock — eliminating the need for an allowSequentialFallback downgrade on the saveState path.
Verification: packages/store-redis/src/__tests__/integration/save-state-atomic.integ.test.ts covers happy-path round-trip, replacement semantics (absent-key-drop), scalar↔array type transitions, CAS rejection (no partial writes on stale version), 10-way parallel-save serialization (exactly-one-wins; durable state is the winner's complete payload), no-orphan-list-keys after a sequence of array/scalar/delete transitions, and empty-customState clears.
FU-A2-40: CFW Workflows γ-cascade re-spawn
Status: ✅ done — see "Done items" section. Closed by: commit 6cbd78808. Surfaced by: second-round adversarial review #6 finding P1.7.
Description: When a parent CFW Workflow suspends mid-sub-agent dispatch, in-flight children are abandoned. Unlike runtime-temporal (which marks children as failureReason: 'parent_suspended' and re-spawns them on parent resume — see runtime-temporal/src/workflow.ts:995-1178, closing FU-A2-09), CFW Workflows has no equivalent path:
- Suspension commit doesn't mark children with
parent_suspended. applyResultsAndReloaddoesn't detect or surface them.- The workflow body doesn't have a re-spawn loop on resume.
After CFW parent resumes, in-flight children are abandoned. The parent loops back to the LLM with stale suspendedAwaitingChildren in state, the LLM produces duplicate sub-agent calls or the session deadlocks waiting for results that won't arrive.
Workaround until fix lands: the workflow body now logs a clear warning when suspendedAwaitingChildren exists after a resume drain (see packages/runtime-cloudflare/src/workflow.ts near "FU-A2-40"). Operators can configure agents that suspend mid-sub-agent dispatch to restart from a checkpoint upstream of the dispatch.
Action:
- Add
parent_suspendedmarking in CFW suspension commit (mirror the TemporalcommitSuspendedStepflow). - Extend CFW
applyResultsAndReloadto returnchildrenToRespawnmatching the Temporal contract. - Add a re-spawn loop in
runtime-cloudflare/src/workflow.tsresume branch — likely 100-150 LOC adapting the Temporal pattern. - Add a workerd unit test asserting end-to-end re-spawn behavior.
Effort: ~2-3 days. Priority: medium-high — blocks production use of CFW Workflows for any agent that suspends mid-sub-agent dispatch (a common HITL pattern with researcher / writer hierarchies).
FU-A2-39: Lazy-load Node-only harness setup helpers so workerd .cf.test.ts files can reuse harness-driven tests
Status: ✅ done — see "Done items" section. Closed by: commit 0d789d40d. Surfaced by: v7 stateless-suspension code review #X workerd-suite triage.
Description: packages/e2e/src/harness/backend-descriptor.ts eagerly imports every setup helper at module top-level:
import { setupJsRedis } from './setup-helpers/js-redis.js'; // → ioredis (Node-only)
import { setupJsPostgres } from './setup-helpers/js-postgres.js'; // → pg (Node-only)
import { setupTemporalRedis } from './setup-helpers/temporal-redis.js';
// ... etcWhen a .cf.test.ts file in workerd pulls in the harness (directly or via importing a .integ.test.ts companion), workerd's parser rejects ioredis/pg source at module-load with SyntaxError: Unexpected token ':', before any test code runs.
This blocks the C2-style "re-run the same integ test in workerd via a 1-line import companion" pattern, e.g. lifecycle-hooks-parity.cf.test.ts (currently .skip'd, see file).
Action:
- Convert eager imports in
backend-descriptor.tsto lazy:register('js-redis', { setup: () => import('./setup-helpers/js-redis.js').then(m => m.setupJsRedis), envRequirements: ['NODE'] }). - Update
getViableBackendsto honorenvRequirements: ['NODE']the same way it honors['WORKERD']— synthesizeskipReasonfor Node-only backends in workerd. - Re-enable
lifecycle-hooks-parity.cf.test.ts(remove thedescribe.skip, restore theimport './lifecycle-hooks-parity.integ.test.js').
Effort: ~0.5–1 day. Priority: medium — the .cf.test.ts import-companion pattern is the only way to cover hooks parity for CF DO + CFW Workflows in workerd context. Until this lands, CF backends in cross-runtime parity tests fall back to indirect coverage via dedicated workerd tests (less convenient when adding new hook contracts).
FU-TYPE-SAFETY-2026-05: Eliminate any + forced casts across all packages
Status: ✅ closed — see "Done items" section. Closed by: commits 4fd490327 (Stages A + B.1 + B.3), 47e3faa18 (Stages B.2 + B.4 + C + D + ToolContext.getState docs), c94ca7606 (close-out review findings). Surfaced by: four parallel pr-review-toolkit:type-design-analyzer agents (2026-05-10) run after the BC removal pass landed.
Summary: Eliminated any annotations and forced as Type casts across all packages (Stages A through D). The full migration write-up lives in the "Done items" section below; the short version is that ~200+ violations across the framework were resolved through a combination of (a) shared type aliases (AnyTool, AnyAgentConfig), (b) Zod schemas with _DRIFT_ASSERTIONS satisfies _SchemaTypeDriftAssertions compile-time drift guards (so runtime schemas can't drift from the TypeScript types they validate), and (c) typed test-mock helpers replacing as any. See the Done section for per-stage detail.
FU-A2-18: Re-enable usage-tracking.integ.test.ts on DBOS — ✅ closed
Status: done (post-release follow-up cluster, alongside Path B Phase 2) Surfaced by: Skipped + TODO audit during exhaustive coverage review.
Description (historical): packages/runtime-dbos/src/__tests__/integration/usage-tracking.integ.test.ts:46 was wrapped in describe.skip by commit be5e793bc (2026-05-07) to silence a "fresh CI node" timing flake — the DBOS workflow's first usage-store entry write intermittently didn't land before executor.result() resolved, surfacing as expected 'failed' to be 'completed' + expected null not to be null assertions.
Closure: the underlying timing flake no longer reproduces. Verified by:
- Re-enabling
describe.skip→describe(no other code changes required). - Stress-loop locally: 10/10 runs clean under
RUNTIME_DBOS_INTEG=1against the docker-compose Postgres store. - Full DBOS-suite ordering: usage-tracking passes 4/4 in 1.1s even when interleaved with the other 88+ runtime-dbos integration tests in the same vitest process.
⚠ Verification was local-only. The broader runtime-dbos integration suite is paused in CI by the vitest.integ.config.ts RUNTIME_DBOS_INTEG gate (see GL #108). This means CI does NOT exercise this file as a regression guard today. If the original timing flake re-emerges on a CI runner, the test will not catch it until GL #108 is resolved and the suite is un-paused. The script wrapper at packages/runtime-dbos/scripts/run-integ-tests.mjs now prints a loud warning when the suite is silently no-test'd so the pattern doesn't slip past contributors.
The most likely fixers (sequenced earlier in this branch's work train):
- The
#104ActiveHandle._fetchResultrace fix (f9dcabe39, formerlybfa101151) tightened the post-submit / pre-clear ordering in DBOS — adjacent to but not the same as the usage write path. - Phase-B Stage-3 / Round-5 DBOS hardening (merged into main via
f942fc27f) tightened multiple step-body ordering invariants. - DBOS SDK bumps post-2026-05-07.
No code changes were needed to re-enable; the describe.skip was the only blocker.
Action:
- Diagnose the flake — likely a beforeAll timeout on cold-start DBOS.
- Either bump beforeAll timeout or split heavy setup out.
- Replace
describe.skipwithdescribe.
Effort: ~1-2 hours (diagnose) + cleanup. Priority: medium — coverage gap on a tier-1 supported runtime.
FU-A2-44: Round-3 P3.R3-CONC partial closure — Redis state-store atomic Lua scripts
Status: ✅ partial close — patchMetadata, updateStatus, setInterruptFlag, compareAndSetStatus script hoist all landed. Surfaced by: round-3 review #3 P1.C1 / P1.C3 + P2 cluster.
Closed by these scripts (in packages/store-redis/src/redis-state.ts):
PATCH_METADATA_SCRIPT— read + merge + write all in one EVAL. Two concurrent patches with disjoint keys can no longer race (TOCTOU between the JS-sideHGETandHSETis gone). Cross-store contract is preserved: per-key last-write-wins; the win is that N parallel patches on N disjoint keys retain all N.UPDATE_STATUS_ATOMIC_SCRIPT— status field write, interrupt-context handling, secondary-index ZREM/ZADD, and TTL refresh all in one EVAL. Concurrent transitions can no longer leave a session indexed in twostatus:*ZSETs.SET_INTERRUPT_FLAG_SCRIPT— HSET (reason + timestamp) + EXPIRE in one EVAL. Closes the narrow window where a process crash between the HSET and the EXPIRE left the flag without a TTL. Cross-store contract is preserved: last-writer-wins onreason.COMPARE_AND_SET_STATUS_SCRIPT(P3.R3-MISC) — hoisted from a per-calltemplate.join('\n')to a module-levelstatic readonlyconst. Saves a string allocation per CAS call; makes the script cacheable forEVALSHAif/when we opt in.
Verification: packages/store-redis/src/__tests__/integration/atomic-state-ops.integ.test.ts covers patchMetadata round-trip, additive merging, last-write-wins semantics on conflicts, 20-way parallel-patch retention of all keys, session-not-found error; updateStatus atomic + secondary-index move + interruptContext clear-on-leave; setInterruptFlag TTL atomicity + last-write-wins reason; compareAndSetStatus hoisted-const sequence test + interruptContext preservation on transitions to interrupted/paused.
Round-3 Redis-stream closures included in this batch:
INIT_STREAM_SCRIPT— atomic HSETNX status + sequence/createdAt + HSET updatedAt + EXPIRE meta/chunks all in one EVAL (P3.R3-CONC.3). Closes the window where a crash between HSETNX status and EXPIRE left the stream key without a TTL.getStreamCountswitched fromKEYS patterntoSCAN pattern(P3.R3-MISC). Non-blocking on large keyspaces.- NOT_ACTIVE chunk-write rejection now logs at the throw site (P3.R3-OBS) so chunk drops are visible in production traces regardless of caller-side catch behavior.
RedisStreamManagerconstructor accepts aloggeroption (defaults tonoopLogger) and warns at construction whenmaxChunks: 0is configured (P3.R3-PERF.6 — operators see the unbounded-stream cliff at startup, not buried in production traces).chunkParseCachemoved from a single module-levelMapshared across everyRedisStreamManagerto a per-instanceChunkParseCachewith proper LRU eviction (P2 closure). Test isolation improved; steady-state cache hit rate also improved (LRU vs FIFO).
Round-3 batch 4 closures (this commit):
deleteSessionTOCTOU fix (P3.R3-CONC): the cleanup ZREM list now enumerates ALL validSessionStatusvalues (not just the one read back fromHGETALL hash.status), so a concurrentupdateStatusbetween the HGETALL and the index Lua can't leave the sessionId orphaned in a status index we didn't touch. ZREM on absent members is a no-op, so over-enumerating is safe. The same fix extends to the per-status run indexes added by P3.R3-PERF.1 below.listRuns({ status })per-status secondary index (P3.R3-PERF.1): addedsessionRunsByStatusKey()helper.createRunZADDs into the initial 'running' index;updateRunStatusZREMs old + ZADDs new when status changes.listRunswith a status filter now does an O(log N + page) ZRANGE on the per-status index instead of loading every run and filtering in JS.deleteSessioncleans up all per-status run indexes.cleanupOrphanedStagingDatapipelining + observability (P3.R3-PERF.5 + P3.R3-OBS): each SCAN page now batches itsLINDEX 0calls into one pipeline and DELs into a single multi-key DEL — turns 2*pageSize round-trips into 2. On completion logs a structuredredis-state.cleanup_orphaned_staging.summarywith{ scannedKeys, deletedExpired, deletedUnparseable, totalDeleted, durationMs, maxAgeMs }.RedisStateStoreOptions.logger?: Loggeradded (P3.R3-OBS) — defaults tonoopLogger. Currently consumed by the cleanup summary log; reserved for future operator-visible warnings.
Still open under P3.R3-CONC (deferred):
RedisLockManagerfencing-token INCR not atomic with the lock acquisition (low priority — Redlock's mutex still bounds the race pergetNextFencingTokencall). Tracked here for completeness; not blocking production.
A.2 — Coverage + demo + streaming items migrated from deleted branch-scoped review docs
The following items were surfaced during the branch-scoped audit / review / verification cluster (branch-audit-review.md, test-state-deep-review.md, exhaustive-coverage-review.md, exhaustive-fixes-report.md, code-review-v7.md, demo-review-v7.md, doc-audit-v7.md, resumable-streaming-deep-review.md, and the six sub-project-*-verification.md files) — all of which were deleted as part of branch cleanup. These items remained open at the time of deletion; recording them here preserves the trail.
Also tracked in GitLab issues (each FU code below maps into one of the three umbrella issues; the issue body has the full context):
FU-DEMO-01,FU-DEMO-02,FU-DEMO-04,FU-DEMO-05→#100— opennext-cloudflare-do demo polish + securityFU-STREAMING-01throughFU-STREAMING-05→#101— v7.0.1 resumable-streaming hardeningFU-A2-16,FU-A2-17,FU-A2-19,FU-A2-31,FU-A2-CF-MISLABELED-FILES,FU-A2-CF-SETUP-EXPLICIT,FU-A2-AI-SDK-V7-MIGRATION,FU-A2-HARNESS-SMOKE-WORKERD,FU-A2-LIFECYCLE-HOOKS-E2E,FU-A2-SUB-AGENT-USAGE-PARITY,FU-A2-DO-STREAM-MANAGER-PARITY,FU-A2-EXPIRED-SESSION-CROSS-STORE,FU-A2-CF-TEST-MIRROR→#102— v7 test-infra coverage gaps (12 items)
FU-A2-16: CFW Workflows real workerd HITL e2e
Status: ✅ closed (real-workerd workflow-body HITL coverage landed; a residual harness-matrix gap remains tracked under O5 — see "Residual" below). Severity (historical): medium-high — tier-1 supported runtime with 100% mocked-step tests Surfaced by: exhaustive-coverage-review.md §2.2. Closed by: Wave 4 test-only commit (this branch).
Description (historical): All CFW Workflows tests used mocked workflow steps; full-execution.workflow.test.ts had every real-workflow test marked describe.skip, and hitl-cycle.workflow.test.ts only exercised the activity surface. The stated blocker — "full workflow-body invocation requires @cloudflare/vitest-pool-workers v0.9.0+ features that aren't available yet" — was stale: v0.9.14 is installed and introspectWorkflowInstance IS available. The real blocker was the miniflare compatibilityDate pin (2024-01-01), which predates the RPC compat surface that backs introspectWorkflowInstance's unsafeGetInstanceModifier binding (workerd warns: needs the rpc flag or compat date ≥ 2024-04-03).
Closure (real coverage now landed):
- Bumped
packages/runtime-cloudflare/vitest.workflows.config.tsminiflarecompatibilityDateto2024-09-23(a Workflows-era date), which enables the introspection RPC binding. No regressions in the existing workflow suites. - Added a same-isolate test-injection seam to
packages/runtime-cloudflare/src/test-workflow.ts(testWorkflowInjection) so a workerd-context test can register a real agent config + scriptedMockLLMAdapterresponses BEFOREenv.AGENT_WORKFLOW.create(...); the REALTestAgentWorkflow.run→runAgentWorkflowbody then resolves them. (The test runner +mainworker share one isolate under@cloudflare/vitest-pool-workers, so module-level state is shared.) - Rewrote
packages/runtime-cloudflare/src/__tests__/workflows/hitl-cycle.workflow.test.tsinto a real-workerd HITL suite: it drives a REAL Workflow instance viaenv.AGENT_WORKFLOW.create(...)+introspectWorkflowInstance(...).waitForStatus('complete'), reads the workflow body's returnedAgentWorkflowResult, and verifies the full v7 stateless-suspension cycle end-to-end against real D1:- client-tool: execute → real LLM step → client-tool register →
suspended_client_tool+ pending persisted to D1 +onAgentSuspended(1×) → durable submit →create({mode:'resume'})→ drain →completed- pending cleared +
onAgentResumed(1×).
- pending cleared +
- approval-gate deny: execute →
suspended_client_tool→ deny submit ({approved:false}) → resume →completed.
- client-tool: execute → real LLM step → client-tool register →
- Un-skipped the real-workflow block in
full-execution.workflow.test.ts(real completion path + unregistered-agent error path) and removed the stale "v0.9.0+ not available" note. The only remainingdescribe.skipthere is the legitimate partyserver-StreamServer-headers one.
Teeth-checked: neutralizing the client-tool LLM response flips the result to completed (suspend assertions fail); corrupting the previousRunId expectation fails against the real suspended-run UUID — both assertions are load-bearing, not vacuous.
Residual (tracked under O5, NOT this FU): the coverage above lives in the runtime-cloudflare package and choreographs the two-instance suspend/resume by writing D1 directly (what executor.submitToolResult() does). It is NOT the harness-driven cfw-workflows-d1 BackendEnv row in the e2e matrix — that still requires the full O5 lift: a workerd-resident worker exposing an HTTP/RPC surface for CloudflareAgentExecutor's execute/submitToolResult/resume, wired into setupCfwWorkflowsD1. Until O5 lands, getViableBackends continues to hard-filter cfw-workflows-d1 (IMPL_PENDING_O5) so the harness-matrix tests (lifecycle-hooks-parity.{integ,cf}.test.ts, etc.) skip that row cleanly. See the O5 entry under "D (Test infrastructure overhaul) follow-ups".
FU-A2-17: CF DO persistent sub-agents real-DO test expansion
Status: ✅ done — see "Done items" section. Severity: medium — persistent sub-agents only unit-tested with mocks Effort: ~1 day Surfaced by: exhaustive-coverage-review.md §2.3.
Description: cloudflare-do-persistent-agents.cf.test.ts (closed via FU-A2-38 G1) exists but the coverage is narrow. Expand to exercise the full companion-spawn matrix (blocking + non-blocking modes, child failure, parent interrupt during child run, cascading sub-agents) against a real DO instance.
Note: CF DO interrupt protocol test is already closed via FU-A2-38 G2 — do not duplicate.
FU-A2-19: DBOS bug-as-test in persistent-companion-client-tool — ✅ closed
Status: done (post-release follow-up sweep) Severity (historical): medium — test asserted buggy behavior Effort: ~30 min (fix + 4 new unit tests + assertion inversion) Surfaced by: exhaustive-fixes-report.md §304 (deleted).
Description (historical): persistent-companion-client-tool.integ.test.ts:600 asserted the buggy DBOS behavior where the persistent-loop's clean exit overwrote a prior turn's status:'failed' run record with status:'completed' + completionReason:'idle_timeout'. The failure trace was lost; no programmatic record (session-level OR run-level) of the LLM error survived once the persistent loop exited.
Closure:
Root cause in
packages/runtime-dbos/src/steps/run-lifecycle.ts'srunFinalizeRun. The function unconditionally wroteargs.statustoupdateRunStatus, so a later finalize call with status='completed' overwrote a prior status='failed'.Fix:
runFinalizeRunnow detects the specific failed→completed downgrade. When the incoming status is 'completed', it reads the current run record viastateStore.getRun(runId). If the current status is 'failed', the call returns without re-writing — preserving the failure trace and the original completionReason.Why only failed→completed: other transitions are unaffected:
- failed → failed: fine (refresh metadata)
- completed → completed: fine (idempotent — re-write run)
- running → completed: canonical happy path (re-write)
- running → failed: real failure write (re-write)
The narrowing keeps the fix scoped to the exact pre-fix bug class.
Test inversion: the integration test at
persistent-companion-client-tool.integ.test.ts:600now assertsstatus === 'failed'(the correct post-fix outcome).New unit tests: 4 new tests in
packages/runtime-dbos/src/__tests__/run-lifecycle.test.ts'srunFinalizeRundescribe block pin the new behavior:- failed→completed preservation (the fix)
- completed→completed idempotent path (not gated)
- failed→failed metadata refresh (not gated)
- running→completed canonical happy path (not gated)
runtime-dbos: 372 → 376 unit tests.
FU-A2-31: Migrate 9 client-tool families to shared harness
Status: open Effort: ~24 hours Surfaced by: exhaustive-coverage-review.md §6.
Description: Only 4 of 152 e2e files (~2%) use the shared matrix harness. Nine client-tool-family integration suites are still hand-rolled per-runtime — migrate them to harness-driven matrix tests to gain cross-runtime parity for free and reduce hand-maintained boilerplate. Post-merge work.
FU-A2-CF-MISLABELED-FILES: mislabeled CF test files — ✅ closed
Status: done (Wave 5 test-infra sweep) Effort: ~2 hours total Surfaced by: exhaustive-fixes-report.md §300.
Description (historical): Test files named cloudflare-*.integ.test.ts (and cloudflare-*.cf.test.ts) claimed runtime-cloudflare coverage but instantiate JSAgentExecutor via the cloudflare-pool-setup.ts / cloudflare-executor-setup.ts helpers (which wire JSAgentExecutor + D1 store, NOT CloudflareAgentExecutor). Root-cause: those setup helpers made JS-executor usage implicit. The established resolution (begun in the v7 stateless-suspension redesign) renames each mislabeled file to a *-d1-with-js.{integ,cf}.test.ts twin and rewrites its header docstring + describe() label to state explicitly that the agent loop runs on JSAgentExecutor and only the store/stream layer is Cloudflare-specific.
Closure: The earlier rename pass created the corrected *-d1-with-js twins but left the original cloudflare-* files in place as committed duplicates — both copies ran (double test runs) and the originals kept perpetuating the exact mislabel this FU is about. This sweep removed the 7 stale duplicates (their *-d1-with-js twins carry identical test bodies + the corrected naming/docstrings — verified by diff: only docstring / describe / comment label lines differed, identical it() titles and counts):
usage-cloudflare.integ.test.ts→usage-d1-with-js.integ.test.tsmulti-message-execute-cloudflare.integ.test.ts→multi-message-execute-d1-with-js.integ.test.tstool-attachments-cloudflare.integ.test.ts→tool-attachments-d1-with-js.integ.test.tsai-sdk-cloudflare.integ.test.ts→ai-sdk-d1-with-js.integ.test.tsai-sdk-cloudflare-do.cf.test.ts→ai-sdk-d1-with-js.cf.test.tscloudflare-edge-cases.cf.test.ts→edge-cases-d1-with-js.cf.test.tsfinishwith-tools.cf.test.ts→finishwith-tools-d1-with-js.cf.test.ts
Count reconciliation: the FU estimated "8 remaining"; the actual count of mislabeled-files-with-a-corrected-twin was 7 (all deleted here). The remaining cloudflare-do-*.cf.test.ts files are correctly named — they exercise real Durable Objects without the JS-executor helper. Live code-comment cross-references in runtime-cloudflare / runtime-temporal finishwith/workflow tests and usage-subagent-parity.integ.test.ts were repointed at the surviving *-d1-with-js twins (the stale (CloudflareAgentExecutor) annotation on the deleted usage-cloudflare reference was also removed — it was always wrong; that suite used JSAgentExecutor). Historical docs/superpowers/ plan/spec snapshots were intentionally left as-is (point-in-time records).
FU-A2-CF-SETUP-EXPLICIT: Make cloudflare-pool-setup.ts JS-executor usage explicit — ✅ closed
Status: done (post-release follow-up sweep) Effort: ~30 min (doc comments only — no rename) Surfaced by: exhaustive-fixes-report.md §301 (deleted).
Description (historical):packages/e2e/src/helpers/cloudflare-pool-setup.ts and packages/e2e/src/helpers/cloudflare-executor-setup.ts both instantiate JSAgentExecutor from @helix-agents/runtime-js, but their file names imply Cloudflare DO / Workflows execution. A reader looking for the CFW Workflows test wiring would land in the wrong place.
Closure: added prominent EXECUTOR CHOICE sections at the top of each file's docstring explaining:
- The "cloudflare" in the name refers to the STORE/ENVIRONMENT, not the executor runtime.
- Why the JS-executor + Cloudflare-store combination exists: tests that exercise store/stream-manager contracts don't need the workflow-orchestration runtime; the JS executor keeps timing deterministic and failures localized to the store layer.
- A "when in doubt" decision tree pointing readers to the right helper for their test's needs (
cloudflare-pool-setupfor D1+DO under workerd-pool,cloudflare-executor-setupfor D1 under Miniflare, orruntime-cloudflare/src/__tests__/for actual workflow-step semantics).
A rename was considered but rejected — the existing names match the naming convention of other helpers in the same directory (temporal-setup.ts, redis-setup.ts, etc.) and renaming would churn ~30 test files for cosmetic gain. Explicit doc comments are sufficient.
FU-A2-AI-SDK-V7-MIGRATION: AI SDK e2e tests still use v6 API — ✅ closed (stale)
Status: done — closed as STALE after an independent re-audit (Wave 5 test-infra sweep). No migration work was required. Effort: ~half-day (estimated) — N/A, premises no longer hold. Surfaced by: exhaustive-fixes-report.md §249.
Description (historical): packages/ai-sdk/src/__tests__/e2e/* was said to still use the deprecated v6 API; the ask was to migrate to v7 handleChatStream so the suite exercises the current public surface.
Audit / closure: every premise of this FU is now false:
- The referenced directory does not exist. There is no
packages/ai-sdk/src/__tests__/e2e/. The ai-sdk tests live under__tests__/(unit) and__tests__/integration/(*.integ.test.ts). - The suite already exercises the current v7 surface. 28 test files import/drive
handleChatStream,StreamTransformer,createReplayEvents, andcreateHelixChatTransportdirectly — the canonical v7 chat-stream/transformer API. The pre-v7 frontend-handler API (FrontendHandler/createFrontendHandler/handleFrontendRequest) was removed in the v8 redesign (see CHANGELOG) — it no longer exists to migrate off of. The pre-v7 frontend-handler API (class/factory/request-handler) has no definition or usage in ai-sdk source or tests; the nameFrontendHandlersurvives only in JSDoc describing the new replacements. - The remaining
v6/useChat/tool-input-endreferences are descriptive or negative, not real deprecated-API usage:- "AI SDK v6" mentions refer to the Vercel AI SDK v6 UIMessage format (orthogonal to the Helix v6→v7 redesign this FU implied), all in JSDoc.
useChatmentions describe the framework's ownuseHelixChatas a "drop-in replacement for useChat", andtransport-submit-tool-result-server-side.test.tsspecifically validates the non-useChatserver-side dispatcher path.tool-input-endappears only in comments/assertions confirming the transformer emits the NEWtool-input-availableevent instead of the deprecatedtool-input-end— the deprecated event is never emitted or asserted-present.
No genuine deprecated v6 call exists to migrate; the suite is already on the v7 public surface. Closed with no code change.
FU-A2-HARNESS-SMOKE-WORKERD: harness-smoke.integ.test.ts:69-71 workerd assertion bug — ✅ closed
Status: done (post-release follow-up sweep) Effort: ~20 min (context guard + comment) Surfaced by: exhaustive-coverage-review.md §321 (deleted).
Description (historical): The cf-do-d1 and cfw-workflows-d1 are absent from Node context results assertion in packages/e2e/src/harness/__tests__/harness-smoke.integ.test.ts holds in Node context (where the file currently runs via the .integ.test.ts extension), but would fail if mirrored to a .cf.test.ts companion — under workerd those backends ARE present because their setup helpers can load there.
Closure: added a runtime context-guard so the assertion only runs when typeof process.versions.node === 'string' (i.e., Node context). A future .cf.test.ts mirror would now see the test correctly skip rather than produce a misleading green result or a mystery failure. Also expanded the surrounding comment to make the context-specificity explicit.
The fix is intentionally defensive — there's no immediate workerd mirror, but the brittle assumption that "this file is Node-only" no longer lives implicitly in the file extension.
FU-A2-LIFECYCLE-HOOKS-E2E: onAgentSuspended/Resumed per-runtime e2e
Status: ✅ closed — all runtimes now have lifecycle-hook coverage, including the previously-missing CFW Workflows leg. Surfaced by: exhaustive-fixes-report.md §299. Closed by: Wave 4 test-only commit (this branch), alongside FU-A2-16.
Description (historical): The lifecycle hooks were silently broken on runtime-js and runtime-temporal until this branch (see fixed items #9 and #10 in the deleted exhaustive-fixes-report). Per-runtime e2e coverage was needed across all runtimes to prevent regression.
Closure (per-runtime status):
- JS / Temporal / DBOS / CF-DO: covered by the harness-driven matrix in
packages/e2e/src/__tests__/lifecycle-hooks-parity.integ.test.ts(getViableBackends({ requires: ['hitl'] })) + the.cf.test.tscompanion for the CF-DO workerd leg.onAgentSuspended/onAgentResumedare asserted exactly-once with theclient_tooldiscriminator, thepreviousRunIdlinkage (load-bearing on every runtime), and a runtime-conditionalrunId-equality assertion (DBOS Branch 1 self-loop vs fresh-runId runtimes). - CFW Workflows: NOW covered by
packages/runtime-cloudflare/src/__tests__/workflows/hitl-cycle.workflow.test.ts(real workerd via@cloudflare/vitest-pool-workers+introspectWorkflowInstance— see FU-A2-16 for the mechanism). Both hooks fire exactly-once across the real two-instance suspend/resume cycle, andonAgentResumed.previousRunId === suspendedCtx.runIdis asserted against the real suspended-run UUID (teeth-checked).
The CFW Workflows leg is NOT wired into the shared getViableBackends matrix (the harness cfw-workflows-d1 BackendEnv row stays IMPL_PENDING_O5 — see FU-A2-16 "Residual" + the O5 entry), because the matrix needs a CloudflareAgentExecutor HTTP/RPC surface inside workerd. The package-level workflow test exercises the same hook contract directly against the real workflow body, so the contract is covered even though the matrix row remains gated.
FU-A2-SUB-AGENT-USAGE-PARITY: Sub-agent usage scoping parity (Temporal/CF/DBOS)
Status: ✅ closed (matrix parity test added; the one runtime divergence that was temporarily gated has since been root-caused + fixed under FU-TEMPORAL-POSTGRES-SUBAGENT-OVERSTEP, so all viable backends now run). Effort: ~half-day Surfaced by: exhaustive-fixes-report.md §299 (P2 list).
Description: Sub-agent token-usage scoping (whether child usage is attributed to parent's run vs reported separately) is verified on runtime-js but not on Temporal/CF/DBOS. Add parity tests.
The portable contract (the "Residual E" usage-scoping model from packages/runtime-js/src/__tests__/usage-scoping.integ.test.ts): when a parent agent spawns a child sub-agent and the child records usage,
- the parent's DIRECT rollup (
getRollup(sessionId, { includeSubAgents: false })) shows the sub-agent dispatch but NOT the child's tool call / custom metrics (those are scoped to the child's session); - the child's usage lives under the child session, discoverable via the parent's
subagentusage entrysubSessionId; and - the AGGREGATED rollup (
getRollup(sessionId, { includeSubAgents: true })) surfaces the child's custom metrics summed exactly once (no double-count, no omission).
Closure: new matrix-driven parity test packages/e2e/src/__tests__/usage-subagent-parity.integ.test.ts, driven by getViableBackends({ requires: ['sub-agents'] }), pins all three invariants against a single shared script. Verified locally (Docker Redis + Postgres + Docker Temporal dev server) on six backends — js-{memory,redis,postgres}, temporal-{memory,redis}, dbos-postgres: 6 passed. The scoping contract held identically on every passing backend (the read path goes through usageStore.getRollup(...) directly, mirroring the per-runtime usage suites, to avoid the executor-wiring asymmetry where some harness setup helpers don't thread usageStore into the executor they build, so handle.getUsage() would return null). Teeth-checked: neutralizing the child's recordUsage() calls makes the child-rollup + aggregated-rollup assertions fail (expected undefined to be 1), proving the aggregation assertions are not vacuous. cf-do-d1 / cfw-workflows-d1 are in the matrix but skip in Node (workerd-only, IMPL_PENDING_O5); CF DO sub-agent usage scoping is exercised via usage-d1-with-js.integ.test.ts (D1UsageStore driven by the JS executor).
One backend gated: temporal-postgres is in the matrix but SKIPPED — its sub-agent → parent-finish step loop genuinely diverges (the parent over-steps past its FINISH output). This is NOT a usage-scoping bug (where the run completes, the child rollup is correctly scoped); it's a Temporal × Postgres-state step-loop divergence tracked separately as FU-TEMPORAL-POSTGRES-SUBAGENT-OVERSTEP below.
FU-TEMPORAL-POSTGRES-SUBAGENT-OVERSTEP: parent over-steps past FINISH after a sub-agent completes (Temporal + Postgres state) — ✅ closed
Status: ✅ closed Severity: medium — a parent → sub-agent → parent-finish flow did NOT reach a clean completed status on the temporal-postgres backend. Surfaced by: the usage-subagent-parity.integ.test.ts matrix (FU-A2-SUB-AGENT-USAGE-PARITY).
Description: On the temporal-postgres backend (TemporalAgentExecutor + PostgresStateStore), after a child sub-agent completed and its result was recorded on the parent session, the parent's next LLM step threw and the run failed (status: 'failed') instead of completing. Confirmed deterministic and store-specific — temporal-memory / temporal-redis completed the SAME script in four LLM calls, and js-postgres / dbos-postgres also completed cleanly.
Root cause (NOT a usage-scoping or result-reload defect):PostgresStateStore.saveState silently dropped parent_session_id. Both saveState UPDATE branches (the skipCheckpointCreation path and the checkpoint-creating path) wrote root_session_id but omitted parent_session_id — only saveStateAndPromoteStaging had been fixed in an earlier cross-store-parity pass (code review #8 finding I1); saveState was the remaining outlier (the in-memory, Redis, and D1 stores all round-trip the field).
The Temporal child sub-agent's initializeAgentState persists its parentSessionId via saveState. With the dropped column the child loaded back from Postgres with parentSessionId: undefined, so when the child completed, persistTerminalState's !state.parentSessionId guard mistook the child for a ROOT session and finalized (endStream) the SHARED parent stream. The parent's next runLLMStep then threw Cannot write to stream … in 'ended' state from the LLM stream callback (after generateStep had already consumed the FINISH response). Because runLLMStep runs as a Temporal activity with a 3-attempt retry policy, Temporal re-ran the step — consuming an extra mock LLM response — and the run failed. temporal-memory / temporal-redis round-trip parentSessionId, so their children correctly skip shared-stream finalization; js-postgres / dbos-postgres drive the child stream lifecycle differently and never depended on the loaded child parentSessionId here.
Fix: add parent_session_id to the UPDATE column list (and parameter array) in BOTH saveState branches of packages/store-postgres/src/state/postgres-state-store.ts, mirroring the already-correct saveStateAndPromoteStaging.
Teeth:
- Cross-store contract test (
packages/core/src/testing/state-operations.ts→ "should round-trip v7 suspension fields through saveState") now also assertsparentSessionIdround-trips throughsaveState. Verified RED on Postgres without the fix (expected undefined to be 'parent-sess-xyz'), GREEN with it; memory / Redis / D1 already passed. - The
isTemporalPostgresgate inusage-subagent-parity.integ.test.tswas removed —temporal-postgresnow runs with full teeth (all 7 viable backends GREEN).
Changeset: .changeset/temporal-postgres-subagent-overstep.md (@helix-agents/store-postgres patch).
FU-A2-DO-STREAM-MANAGER-PARITY: DOStreamManager round-trip parity test — ✅ closed
Status: ✅ closed (Wave 3) — reframed: the runtime-cloudflare DOStreamManager ALREADY had a full contract-harness parity test (packages/runtime-cloudflare/src/__tests__/do-stream-manager-contract.cf.test.ts, real workerd via cloudflare:test). The genuine remaining gap was the SEPARATE binding-side store-cloudflare DurableObjectStreamManager (stub.fetch + SSE consumer), which had bespoke unit tests but never ran the shared streamManagerContractTests factory. Wave 3 added that parity test (packages/store-cloudflare/src/__tests__/durable-stream-contract.integ.test.ts, real StreamServer DO in Miniflare), which both fills this parity gap AND verifies the G4 fix (FU-G4-STORE-CF-STREAM-DO). All G1–G5 contract guarantees are now asserted against the binding-side manager. Effort: ~2 hours Surfaced by: exhaustive-fixes-report.md §299 (P2 list).
Description: No round-trip parity test exists for DOStreamManager (write → read → assert chunk sequence integrity). Add one mirroring the in-memory/Redis parity tests.
FU-A2-EXPIRED-SESSION-CROSS-STORE: expiredSessionCleanup extension to Temporal+CF
Status: Temporal portion ✅ closed; CF DO portion open (gated on the O5 workerd-harness lift). FU-A2-21 closed the cross-store parity for memory/Redis/Postgres/D1. Effort: ~2 hours Surfaced by: exhaustive-fixes-report.md §299 (P2 list).
Description (original): Extend the expiredSessionCleanup parity test to cover Temporal + CF DO runtimes — previously only memory + the three remote stores had it (per FU-A2-21).
Why this is portable: expiredSessionCleanup (packages/agent-server/src/cleanup.ts) is runtime-agnostic — it touches ONLY the SessionStateStore abstraction (listSessions → loadState → compareAndSetStatus) and never invokes the executor. So the contract ("identify past-expiry sessions, CAS them to failed/session_expired, leave future-expiry and terminal sessions untouched") is the same on every backend the harness can stand up; the runtime is just the vehicle that constructs (and, on Temporal, writes through) the store.
Closure (Temporal): the parity test (packages/e2e/src/__tests__/expired-session-cleanup-parity.integ.test.ts) broadened its backend filter from getViableBackends({ only: ['js'] }) to getViableBackends({ excludes: ['dbos', 'cfw-workflows'] }). In Node context this resolves to six backends — js-{memory,redis,postgres} + temporal-{memory,redis,postgres} — so all five scenarios run on Temporal (30 logical runs). Verified locally against Docker Redis + Postgres + the in-process Temporal test environment: all 30 pass. No runtime divergence was found — the store-abstraction assertion is fully portable, and the Temporal harness's store wiring persists expiresAt / status identically to the JS path.
Remaining gap (CF DO): this suite is not yet wired to run on cf-do-d1 in the workerd pool. O5 Phase 1 landed real CF setup helpers and made cf-do-d1 a live parity participant for the lifecycle-hooks-parity suite, but it did so via a context-split: the Node and CF backend registries are now separate (NODE_BACKENDS vs CF_BACKENDS) so the workerd bundle never statically references Node-only loaders. This integ file drives its matrix via getViableBackends, which reads NODE_BACKENDS only — so it can never surface cf-do-d1.
A previous .cf.test.ts companion that merely re-imported this integ file relied on the pre-O5 assumption that cf-do-d1 lived in the single shared registry with an IMPL_PENDING_O5 skip marker (and would auto-activate once the marker dropped). The context-split invalidated that assumption: after O5 the re-import resolves to zero viable backends in workerd, registers zero suites, and fails collection (No test suite found). That dead companion was removed (fix-forward after the O5 Phase 1 merge); expiredSessionCleanup cross-store coverage continues to run Node-only via the integ file (six js/temporal backends).
When this suite is CF-wired in a later O5 phase, it will follow the lifecycle-hooks-parity pattern — a Node-import-free shared scenario module plus a dedicated CF entrypoint over selectViable(CF_BACKENDS, …) — NOT a re-import companion. That also requires the cf-do-d1 harness SessionStateStore shim to implement listSessions / compareAndSetStatus (the O5 Phase 1 shim only covers the lifecycle-hooks scenario's needs). The DOStateStore cross-store contract is exercised today by the dedicated workerd-context .cf.test.ts suites under packages/store-cloudflare / packages/e2e, just not yet through this parity harness. Tracked here + under O5 in docs/dev/test-infrastructure-roadmap.md.
Scope note: dbos and cfw-workflows are excluded by intent. DBOS would only re-exercise the postgres store already covered by js-postgres / temporal-postgres, and CFW Workflows shares CF DO's not-yet-CF-wired harness gap for this suite with no distinct store to add. The FU's explicit ask was Temporal + CF DO.
FU-A2-CF-TEST-MIRROR: C-1..C-5 contract tests need *.cf.test.ts mirrors
Status: ✅ closed (Wave 5 — two genuinely-missing mirrors landed; two already covered; one reframed as redundant, see mapping below). Effort: ~half-day Surfaced by: exhaustive-fixes-report.md §299 (P2 list).
Description: Five contract tests (C-1..C-5 in the deleted coverage review) run in Node context only. Mirror them to .cf.test.ts for real workerd-context coverage.
Closure. The "C-1..C-5" labels map to the five shared-contract-factory suites that the Cloudflare store packages run. Mapped against what already had real-workerd coverage:
| # | Contract suite (shared factory) | Implementation under test | Node-context suite | Real-workerd .cf.test.ts? |
|---|---|---|---|---|
| C-1 | stateStoreContractTests | D1StateStore | store-cloudflare/.../integration/state-store-contract.integ.test.ts (Miniflare) | ADDED — store-cloudflare/.../state-store-contract.cf.test.ts |
| C-2 | usageStoreContractTests | D1UsageStore | store-cloudflare/.../integration/usage-store-contract.integ.test.ts (Miniflare) | ADDED — store-cloudflare/.../usage-store-contract.cf.test.ts |
| C-3 | streamManagerContractTests | DOStreamManager (in-process DO) | — | already real-workerd: runtime-cloudflare/.../do-stream-manager-contract.cf.test.ts (Wave 2) |
| C-4 | streamManagerContractTests | DurableObjectStreamManager (binding-side) | store-cloudflare/.../durable-stream-contract.integ.test.ts (Miniflare, real StreamServer DO) | reframed redundant — see below |
| C-5 | (no distinct CF store) | — | — | n/a |
What was added (genuinely missing). store-cloudflare had NO cf-pool harness at all (no vitest.cf.config.ts, no wrangler.cf.toml, no test:cloudflare script). Added that infra (mirroring runtime-cloudflare's exactly, NOT a parallel invention) plus the two D1 mirrors. Each runs the SAME shared factory from @helix-agents/core/testing against the D1Database bound by workerd itself (env.AGENT_DB via cloudflare:test), not Miniflare's Node-side D1 proxy. runMigration / table-clearing are inlined in the .cf.test.ts (the Node helper miniflare-setup.ts imports miniflare, unavailable in the workerd pool). A D1Database binding is NOT DO-IO-restricted, so — unlike the in-process DOStreamManager mirror — the store + every assertion run directly in the worker context with no runInDurableObject proxy. Result: 157 tests (137 state-store + 20 usage-store) green on first run in real workerd, no skipKnownFailures, no opt-outs.
Teeth verified. (1) Skipping runMigration makes all 20 usage tests fail with no such table: __agents_usage: SQLITE_ERROR from workerd's internal cloudflare-internal:d1-api — proving the test hits the live workerd D1, not a no-op binding. (2) Wrapping the store in a Proxy that returns a wrong queryUsage total (999) makes 20/20 fail with meaningful diffs (expected 999 to be 3, wrong lengths, wrong pagination caps) — proving the contract assertions catch a semantic violation against the workerd-bound store. Both sabotages reverted; suite green.
Why C-4 is NOT given a separate .cf.test.ts (reframed, not skipped). The binding-side DurableObjectStreamManager mirror (durable-stream-contract.integ.test.ts) already runs the contract against the real partyserver StreamServer Durable Object inside Miniflare (createRealStreamContext() loads the bundled worker; the inline test-double is not used). For a DO accessed via its HTTP/SSE binding surface, that path is real-workerd-equivalent: the DO body executes in workerd, and the manager talks to it over the same fetch/SSE surface it uses in production. A separate .cf.test.ts would re-host the identical StreamServer + contract with no new code path exercised — a vacuous duplicate. The distinct in-process DOStreamManager surface (the part that genuinely benefits from runInDurableObject workerd hosting) is already covered by C-3's do-stream-manager-contract.cf.test.ts. This mirrors the Wave-3 FU-A2-DO-STREAM-MANAGER-PARITY reframing discipline (reframe rather than blindly produce a no-value duplicate). The push-SSE late-attach G4 nuance on this binding path is separately tracked by FU-G4-BINDING-AFTER-ENDSTREAM-RETURNS-DONE.
C-5 has no distinct Cloudflare store implementation to mirror (the fifth slot in the deleted review did not correspond to a CF store contract); nothing to add.
Demo (Round-3 web-app-testing) follow-ups
FU-DEMO-01: remote-agents-temporal/approve.ts v7 migration — ✅ closed (verified migrated)
Status: done (verified during post-release follow-up sweep — the migration already landed in an earlier work train and the example typechecks + builds clean). Severity: WAS P0 for that example; the original audit predated the actual migration. Effort: ~5 min (verify + clean up stale comment). Surfaced by: demo-review-v7.md §232-242 (deleted; original audit).
Closure:
examples/remote-agents-temporal/src/approve.ts— already uses the v7 durable-submit pattern. The file's own docstring (lines 14-17) explicitly calls out the v6→v7 migration: "v6 usedhandle.signal('submitToolResult', payload)— that signal was deleted in v7's stateless-suspension redesign (Task 3.3) so this script HAD to migrate." The body now usesexecutor.submitToolResult({ kind: 'client-tool-result', ... })followed byexecutor.resume(), with proper handling ofunknown_tool_callandalready_completedresponses.examples/remote-agents-temporal/src/client.ts— the only residual issue was a stale comment block (lines 95-101) that still described the old v6 behavior ("The signal-basedsubmitToolResultflow used here continues to work in v7..."), which was false post-A.2 stateless-suspension. Cleaned up to correctly describe the v7 surface (Temporal now emits'suspended_client_tool'etc., approve.ts uses durable submit).example-remote-agents-temporal—npm run typecheckandnpm run buildboth pass cleanly. The example is not fail-fast anymore; running the end-to-end flow requires a Temporal + Postgres+Redis stack but the code itself is correct.
Nothing else to ship for this entry.
FU-DEMO-02: client-tools README v6 snippet + nextjs-redis snapshot helper — ✅ closed (verified outdated)
Status: done (verified during post-release follow-up sweep — the two pending sub-items have already been resolved by intervening work; this entry stays in the docs as a back-reference). Effort: ~1 hour (audit). Surfaced by: demo-review-v7.md §245-251 (deleted; original audit).
Closure:
examples/client-tools/README.md— re-audited 2026-05-18 post-release. The README no longer contains any v6 snippet; the current "How the client-tool flow works" walkthrough documents the v7useHelixChat/createHelixChatTransport/submitToolResultpath end-to-end. No diff needed. Searches for "v6" / "deprecated" return zero matches in the README.examples/nextjs-redissnapshot fallback — the framework helper the demo was waiting on isbuildSnapshot(exported from@helix-agents/ai-sdkatsrc/index.ts:199asexport { buildSnapshot, type BuildSnapshotDeps } from './handler/snapshot.js').examples/nextjs-redis/src/lib/agent-client.ts:141already uses it directly:tsexport async function getSnapshot( sessionId: string ): Promise<FrontendSnapshot<ChatState> | null> { return buildSnapshot<ChatState>(deps, { sessionId }); }No fallback path is needed because the helper exists; the "snapshot fallback path still expects a v7 framework helper that isn't shipped yet" sub-item in the original audit pre-dated the
buildSnapshotexport landing.
Nothing to ship for this entry — included here only so the original audit trail stays auditable.
FU-DEMO-04: opennext-cloudflare-do — coordinator UI hardcoded child name — ✅ closed
Status: done (verified during May 18 post-release follow-up audit for GL #100). Severity: medium (demo polish, not framework) Effort: done via option 1 (system prompt constraint — the cheapest fix in the original issue). Surfaced by: Round-3b web-app-testing live-LLM verification.
Closure: the CoordinatorAgent system prompt explicitly instructs the LLM NOT to pass a name argument to companion__spawnAgent, so auto-naming produces the canonical researcher-1 that the UI polls. See the in-code rationale at examples/opennext-cloudflare-do/src/app/coordinator/[sessionId]/ChatClient.tsx:83-97. A future framework enhancement could add GET /api/coordinator/<id>/children for dynamic name discovery via getSubSessionRefs, but that's an enhancement, not a closure requirement.
FU-DEMO-05: opennext-cloudflare-do — unauthenticated /debug route — ✅ closed
Status: done (verified during May 18 post-release follow-up audit for GL #100). Severity: medium (information disclosure if demo deployed without auth) Effort: done via env-var gate (option 2 in the original issue). Surfaced by: Round-2 web-app-testing curl probes.
Closure: both /api/chat/[sessionId]/debug and /api/debug/[sessionId] routes are gated by gateDebugRoute() from examples/opennext-cloudflare-do/src/lib/debug-gate.ts. Default behavior is 404 (no signal that the route exists, so scrapers don't get a "look harder" hint). Set ENABLE_DEBUG_ROUTES=true on the worker (typically via .dev.vars for local development) to enable. Strict === 'true' comparison documented to prevent future "helpful" refactors from accepting '1' / 'TRUE' / 'yes'.
Streaming hardening (v7.0.1+)
FU-STREAMING-01: Concurrent writer + reader race in core — ✅ closed
Status: done (closed by Sub-project A streaming-hardening, MR !184). Severity: 7/10 Surfaced by: resumable-streaming-deep-review.md §261.
Closure: the cross-impl streamManagerContractTests harness in @helix-agents/core/testing codifies the no-gap + global-monotonic invariant via G1.concurrent-writers (5 writers × 10 chunks, asserts no dups, no gaps, monotonic-as-yielded). Wired into all three implementations (in-memory, Redis, DOStreamManager). The race-detection test in packages/e2e/src/__tests__/streaming-races.integ.test.ts exercises the same shape against every viable backend under adversarial load.
FU-STREAMING-02: Redis getChunksFromSequence array-index assumption — ✅ closed (verified already resolved)
Status: done (verified during post-release follow-up sweep; the original audit predated the current implementation OR the fix landed in passing without back-reference). Severity (historical): 8/10 Surfaced by: resumable-streaming-deep-review.md §262 (deleted).
Description (historical): getChunksFromSequence was reported to assume array indices map 1:1 to sequence numbers in the Redis chunk list — fragile against producers that skip sequences (e.g., retry- then-resume).
Closure: the current implementation at packages/store-redis/src/redis-stream.ts:1212-1231 does NOT make the array-index assumption. It:
- Loads the entire list (
LRANGE 0 -1). - Filters by
parsed._sequence > fromSequenceper entry. - Sorts the filtered set by sequence before returning.
The class-level comment at line 231 explicitly states "we filter by _sequence (not list index) in getChunksFromSequence." The sort step's comment at line 1228 explicitly handles "out-of-order list insertion (concurrent writers)."
Both alternatives the original audit proposed (invariant verification, or a (seq, chunk) Hash) are unnecessary — the filter-then-sort already provides the same robustness without the storage-shape change. The cost is one full-list scan per getChunksFromSequence call, which is acceptable because the JS-side parse-cache (mentioned at line 1206) amortizes parsing across reads.
FU-STREAMING-03: DO SSE late-attach race vs concurrent writes — ✅ closed
Status: done (closed by Sub-project A streaming-hardening, MR !184). Severity: high (CF DO production path) Surfaced by: resumable-streaming-deep-review.md §263.
Closure: root cause identified as a lost-notification race in DOStreamManager.createResumableReader's iterator wait loop: a write landing between the status check and waiter registration fired notifyWaiters() before the reader's waiter was registered, so the reader hung up to 5 seconds on the safety timer. Fixed by reordering the loop to register-before-poll (waiter registered BEFORE the storage SELECT; any write either lands before the register and is caught by SELECT, or fires the waiter). The G2.late-attach-during-burst contract test and the FU-STREAMING-03 repro race-detection test both exercise the scenario and verify sub-100ms delivery.
FU-STREAMING-04: cleanupToStep + attached reader interaction — ✅ closed
Status: done (closed by Sub-project A streaming-hardening, MR !184). Surfaced by: resumable-streaming-deep-review.md §264.
Closure: behavior pinned as G4 of the StreamManager concurrency contract: cleanupToStep(N) that deletes chunks past a reader's cursor MUST cause the next iterator.next() call to throw StreamTruncatedError (new typed class, exported from @helix-agents/core). Implemented across all three impls via a per-stream truncated_at_step marker that the reader consults on each iteration. Contract tests: G4.truncation-throws + G4.truncation-after-endStream-returns-done + G4.caught-up-on-active-stream-does-not-throw + G4.resetStream-clears-truncation-marker. Two known gaps tracked separately:
FU-G4-STORE-CF-STREAM-DO: binding-sideDurableObjectStreamManagerpath lacks G4 (requires new SSE event type).FU-G4-PERF-POLISH: ✅ closed — sticky O(N) reader rebuild while the truncation marker stays set (bounded with a per-reader marker cache on redis + DO; the clear-on-reset sub-issue predated this FU and was already done).
FU-STREAMING-05: Inline mock DO scaffolding divergence from real Phase 8 semantics — ✅ closed
Status: done (closed by Sub-project A streaming-hardening, MR !184). Surfaced by: resumable-streaming-deep-review.md §265.
Closure: the 240-LOC packages/runtime-cloudflare/src/__tests__/helpers/mock-sql.ts helper (whose header explicitly admitted "NOT a full SQLite implementation") was DELETED entirely. The 5 dependent test files (do-stream-manager-resumable.test.ts, -paused, -pause-cas, -cleanup-to-step, -info-metadata) migrated to .cf.test.ts and now run inside real workerd via cloudflare:test's runInDurableObject. 26 tests preserved with all original assertion intent. The shared helpers/stream-manager-cf-helpers.ts provides the runInDurableObject boilerplate to keep the 5 files thin.
FU-G4-STORE-CF-STREAM-DO: StreamDurableObject lacks G4 truncation enforcement — ✅ closed
Status: ✅ closed (Wave 3) — the binding-side DO stream path now surfaces G4 truncation. Severity: medium (parity gap, not data loss) Surfaced by: Batch 3 spec review of streaming-hardening MR (Sub-project A).
Closure (Wave 3): wired a truncated event end-to-end through the binding-side (store-cloudflare) DO stream path, which previously deleted orphaned chunks on cleanupToStep but never surfaced StreamTruncatedError to attached readers:
packages/core/src/store/stream-event.ts— addedStreamTruncatedEvent({ type: 'truncated', truncatedAtStep, atSequence? }) to theStreamEventunion + ZodStreamEventSchema+ a type guard (with a new core unit test instore/__tests__/stream-event.test.ts).packages/store-cloudflare/src/stream-durable-object.ts—handleCleanupToStepnow writes atruncated_at_stepmarker and broadcasts thetruncatedevent to attached WS + SSE connections;handleReset/handleReactivateclear the marker (matching memory/redis/DOStreamManager).packages/store-cloudflare/src/durable-stream.ts— the binding-side SSE reader (createSSEReaderCore) recognizestruncatedand throwsStreamTruncatedError(close-before-throw discipline; subsequentnext()returns done).packages/ai-sdk/src/cloudflare/sse-parser.ts—parseDOSSEStreamsurfaces the newtruncatedSSE event (ParsedStreamTruncated+isParsedTruncated) instead of dropping it.
Best-effort nuance (documented): the binding-side reader is a push-SSE consumer with no per-next() marker poll, so G4 truncation is event-driven — the throw fires only if the reader is attached when cleanup broadcasts. A reader that attaches AFTER cleanup (replay /sse?fromSequence=X) won't get a truncated event. The G4.truncation-throws contract case attaches before cleanup, so it passes (active-stream throw enforced). One sibling contract case — G4.truncation-after-endStream-returns-done — IS opted out for this manager via skipKnownFailures with a documented push-SSE justification (not faked): see FU-G4-BINDING-AFTER-ENDSTREAM-RETURNS-DONE below. Its portable safety property is pinned by a dedicated binding-side test instead.
Locked by: the new binding-side contract-parity test packages/store-cloudflare/src/__tests__/durable-stream-contract.integ.test.ts (runs the shared streamManagerContractTests against the real StreamServer DO in Miniflare — this also closes FU-A2-DO-STREAM-MANAGER-PARITY) plus the binding-side push-SSE behavior tests in packages/store-cloudflare/src/__tests__/integration/durable-stream-late-attach-g4.integ.test.ts.
Residual — handleResume (paused→active) marker-clear has no direct read surface (Waves 3-5 deep-review L1, accepted-no-test): handleResume (stream-durable-object.ts:841) clears the truncated_at_step meta row with an UNCONDITIONAL DELETE FROM meta WHERE key = 'truncated_at_step' on the paused→active transition (mirroring the handleReactivate:877 / handleReset:1119 DELETEs). The deep-review noted this clear is only "trivially" tested — the existing reactivate contract case passes for a different reason (the push-SSE reader does not poll the marker). We investigated adding a DIRECT "marker is gone after resume" assertion and found the DO exposes NO read surface for the truncated_at_step row:
/infodeliberately excludes it from itsSELECT … WHERE key IN (…)allow-list (handleInfo:912).- The DO never reads the row back for any behavioral decision — the only consumer of G4 truncation is the binding-side reader, which reacts to the broadcast
truncatedEVENT emitted atcleanupToSteptime (durable-stream.tsSSEReaderState.truncated), NOT the meta row. - The SSE attach path (
setupSSEConnection:462) does not read the marker or emit atruncatedframe on attach, so a fresh post-resume reader can't observe its presence/absence. - Miniflare 3.x exposes no public API to query an arbitrary user DO's internal SQLite (
meta) table from the test (onlygetDurableObjectNamespace+ HTTP stubs).
Asserting the row directly would require EITHER a production code change (add a test-only read endpoint to the DO) OR reaching into Miniflare internals in a way that does not map to the real meta table — both contortions the deep-review explicitly said to avoid for this harmless-on-push-path Low. The clear is therefore verified by code-read + the unconditional SQL DELETE, and is structurally identical to the handleReactivate / handleReset DELETEs that ARE exercised (the shared contract's G4.resetStream-clears-truncation-marker case + the binding-side late-attach tests). No test added; tracked here. Re-evaluate if a future read surface for the marker is added (e.g. surfacing truncatedAtStep on /info), at which point the direct assertion becomes feasible.
Description: The G4 contract ("cleanupToStep MUST throw StreamTruncatedError on attached readers when chunks past the reader's cursor are deleted") was wired into InMemoryStreamManager, RedisStreamManager, and runtime-cloudflare/DOStreamManager in the streaming-hardening MR. The binding-side DurableObjectStreamManager (in packages/store-cloudflare/) delegates via stub.fetch to a SEPARATE in-DO impl — StreamDurableObject — which the original plan conflated with DOStreamManager. As a result, the binding-side path does NOT enforce G4.
The binding-side reader is a pure SSE consumer (createSSEReaderCore in packages/store-cloudflare/src/durable-stream.ts). Closing the gap requires adding a new event type to the StreamEvent discriminated union (currently chunk | end | fail | status) — call it StreamTruncatedEvent — and wiring it across:
packages/core/src/store/stream-event.ts— new event type + schemapackages/store-cloudflare/src/stream-durable-object.ts—handleCleanupToStepwritestruncated_at_stepmarker AND broadcasts the new event to attached SSE connections.packages/store-cloudflare/src/durable-stream.ts—createSSEReaderCore.getNextrecognizes the new event and throwsStreamTruncatedError.packages/ai-sdk/src/cloudflare/sse-parser.ts— recognize the new event so it doesn't drop into the "unknown frame" branch.- Add
.cf.test.tsinfra topackages/store-cloudflare/so the cross-impl contract test from@helix-agents/core/testing.streamManagerContractTestscan run againstDurableObjectStreamManagerand verify the fix.
Effort: ~80-120 LOC across 3 packages + test infra. Real wire- protocol change.
Mitigation in the meantime: The streaming-hardening release notes (and the corresponding changeset) should call out that G4 is NOT enforced on the binding-side DurableObjectStreamManager path in v7.0.1; consumers using that import directly should be aware that cleanupToStep will not surface as StreamTruncatedError on attached SSE readers until this follow-up lands.
FU-G4-BINDING-AFTER-ENDSTREAM-RETURNS-DONE: push-SSE binding reader can't satisfy the poll-reader "first next() after post-endStream cleanup is done" mechanic — ⏳ accepted divergence
Status: ⏳ accepted divergence (documented + opted out, portable safety property pinned by a dedicated test). Severity: low (best-effort push-SSE timing; the REAL G4 safety guarantee — never throw after terminal — holds). Surfaced by: running the binding-side Miniflare contract suite during the Wave 3 four-battery fix-round (the original Wave 3 commit had never actually run this suite green — G4.truncation-throws was hanging 60s and G4.truncation-after-endStream-returns-done was failing; both were wrongly dismissed as Miniflare flakes).
Also the CF DO client (WE MR C1): DOStreamManagerClient over the DO's /sse is a push-SSE reader too, with the same skip and reason; its live case G4.truncation-throws passes since the DO's rewound frame.
The divergence: the shared contract case G4.truncation-after-endStream-returns-done asserts the FIRST iterator.next() after a post-endStream cleanupToStep returns done. This is a POLL-reader mechanic — InMemoryStreamManager, RedisStreamManager, and the in-process DOStreamManager re-read the store / marker on each next(), so a just-deleted chunk vanishes immediately. The binding-side DurableObjectStreamManager is a push-SSE consumer: when a reader attaches, the DO eagerly replays ALL surviving chunks into the reader's local buffer BEFORE a later cleanupToStep deletes them server-side. The reader therefore delivers the already-pushed chunks (which genuinely existed and were streamed) and then terminates with done — it satisfies the real guarantee (it never THROWS StreamTruncatedError after a terminal status) but not the poll-reader "first next() is done" timing.
Resolution (Wave 3):
- Opted the case out for this manager via
skipKnownFailures['G4.truncation-after-endStream-returns-done']with an in-line push-SSE justification (durable-stream-contract.integ.test.ts). NOT faked — the active-stream throw case (G4.truncation-throws) remains enforced and is NOT opted out. - Pinned the portable safety property (drain-to-done after
endStream+cleanupToStepwith an attached reader NEVER throwsStreamTruncatedError) with a dedicated binding-side test inintegration/durable-stream-late-attach-g4.integ.test.ts.
Also fixed in the same fix-round (the real bug behind the 60s hang): the binding-side reader (durable-stream.ts) set only state.truncationThrown before throwing StreamTruncatedError, not state.closed. A consumer's catch-then-retry (G4.truncation-throws asserts the next next() returns done) on a still-active stream therefore blocked indefinitely instead of returning done. Now sets state.closed = true before both throws, mirroring do-stream-manager.ts / redis-stream.ts.
If we ever want full parity: the binding reader would need to NOT eagerly buffer (i.e., re-fetch survivors lazily per next()), which defeats the push-SSE design. Not worth it — the divergence is delivery timing of already-valid chunks, not a missing guarantee.
FU-G4-PERF-POLISH: G4 truncation marker check is O(N) per next() + not cleared on resetStream
Status: ✅ closed Severity: low (perf polish; no correctness issue) Surfaced by: Code-quality review of Batch 3 streaming-hardening commit (32dbdea76).
Original description (predates the fixes below): the G4 truncation-mark mechanism stores a per-stream marker the reader iterator consults on every next(). Two improvements were proposed: (1) cache the marker on the reader closure to avoid re-scanning the chunk list on every yield; (2) clear the marker on resetStream.
As-scoped reframe (Wave 3):
Sub-issue (2) — clear the marker on reset — was already DONE before this FU was actioned. All three impls clear the marker on the transition back to active:
- memory:
in-memory-stream.tsresetStream+reactivateStream - redis:
RESET_STREAM_SCRIPTHDEL __truncated_at_step(redis-stream.ts) + the CAS-to-active script used byresumeStream/reactivateStream - DO:
resetStream+reactivateStreamDELETE ... WHERE key = 'truncated_at_step'(do-stream-manager.ts)
Guarded by the cross-store contract test G4.resetStream-clears-truncation-marker (core/src/testing/stream-manager-contract.ts). Nothing to do here.
Sub-issue (1) — cache the marker / avoid O(N) per next() — was partly stale, now bounded on redis + DO:
- Memory needed NO change. Its marker check is already O(1) (a field read on the in-memory stream object) and its chunk lookup is amortized-O(1) via the
[R3C-C1]hint index (findNextChunk). There is no per-next()list scan to eliminate. - Redis + DO had the real cost. While the truncation marker is SET, the resumable reader re-ran a FULL buffer rebuild on every
next()(redis:LRANGE chunksKey 0 -1; DO:SELECT ... WHERE sequence > cursor). This is wasted work in the steady state where a long-lived reader survived a cleanup and keeps draining surviving chunks below the marker.
Fix (closing sub-issue 1 on redis + DO): added a closure-local lastObservedTruncStep to both resumable readers. The expensive rebuild is now SKIPPED when the marker value is unchanged AND the local buffer still has undrained survivors past the cursor — the exact steady state the FU flagged. The rebuild deliberately STILL runs (a) once the buffer drains and (b) when a new cleanup advances the marker, because the on-drain rebuild is what surfaces any chunks the writer appended after the cleanup while the marker stayed set, and confirms emptiness for the G4 throw gate. This bounds the perf win to the safe path and cannot trade G4 correctness for it. The cache is reset when the marker is cleared (reset/reactivate).
Verification: the full G4 contract suite stays green on redis (redis-stream-contract.integ.test.ts, requires Redis) and DO (do-stream-manager-contract.cf.test.ts, requires workerd). A focused regression test (redis-stream.integ.test.ts → FU-G4-PERF-POLISH: marker-cache skip still delivers post-cleanup writes) pins the correctness-critical path: it FAILS with a blanket marker-unchanged skip (spurious StreamTruncatedError) and PASSES with the buffer-non-empty guard. The optional store-read-count micro-test was skipped as brittle per the FU.
Touched:
packages/store-redis/src/redis-stream.ts(+ integ test)packages/runtime-cloudflare/src/do/do-stream-manager.ts- (
packages/store-memory/src/in-memory-stream.ts— no change needed)
A.1 (Stateless purity cleanup) follow-ups
FU-A1-01: runId generator pattern duplicated in executor.ts
Status: ✅ done — see "Done items" section. Closed by: commit 869787e5c.
O5 (workerd parity harness) follow-ups
FU-O5-CF-DO-APPROVAL-STREAM-READ: approval-gate approvalId not readable via env.streamManager on cf-do-d1
Status: open — Scenario 3 (approval-deny) is gated OFF on cf-do-d1 in the shared lifecycle-hooks-parity suite via ParityOpts.gateApprovalStreamRead.
What's gated. The shared lifecycle-hooks parity scenario module's Scenario 3 (approval-gate deny path) recovers the approvalId synchronously from env.streamManager.createReader(sessionId) (it needs it to POST the approval-response submit). On the cf-do-d1 backend this is NOT presentable:
- The DO base persists its event stream to its OWN INTERNAL
DOStreamManager(overctx.storage, wired indurable-object-agent-base.ts > ensureInitialized), NOT the binding-sideDurableObjectStreamManageroverenv.STREAMSthat thecf-do-d1BackendEnv.streamManagerexposes. Soenv.streamManager.createReader(sessionId)reads a DIFFERENT (empty) stream than where the DO emitted thetool_approval_requestchunk — it returns no reader and the scenario throwsstream reader missing. - The
approvalIdis generated at approval time and lives ONLY on thetool_approval_requestchunk; it is NOT persisted onstate.pendingClientToolCalls(seePendingClientToolCallincore/src/types/state.ts— noapprovalIdfield), so there is no portable way to recover it from the harnessBackendEnv(state store / status route).
Why it's a gate, not a fake. Scenarios 1 + 2 (the portable client-tool suspend/resume + previousRunId linkage + tracingContext round-trip contracts) DO EXECUTE on cf-do-d1 and pass — the approval path is the only genuinely runtime-divergent leg. Per the spec's gating discipline (refinement #5: "synchronous mid-suspend handle internals" are CF-non-portable), Scenario 3 is skipped on cf-do-d1 rather than weakened. The EXECUTING CF-DO approval-gate coverage continues to live in the dedicated standalone DO tests (which read the DO's /sse route directly).
To close. Either (a) surface the approvalId on the DO /status (or a new route) / on pendingClientToolCalls so the scenario can recover it portably, or (b) give the cf-do-d1 BackendEnv.streamManager a reader that proxies the DO's internal /sse stream so createReader returns the real approval chunk. Then drop gateApprovalStreamRead: true from the cf-do entrypoint and confirm Scenario 3 EXECUTES.
Where: gate in e2e/src/parity/scenarios/lifecycle-hooks.ts (ParityOpts.gateApprovalStreamRead + itOrSkipApproval); applied in e2e/src/__tests__/lifecycle-hooks-parity.cf.test.ts.
FU-O5-CFW-TRACING-CONTEXT-PERSISTENCE: tracingContext set in onAgentSuspended does not survive resume → completion on cfw-workflows-d1
Status: open — Scenario 2 (tracingContext round-trip) is gated OFF on cfw-workflows-d1 in the shared lifecycle-hooks-parity suite via ParityOpts.gateTracingContextPersistence. (Since plan 0c MR-1b the workerd row this suite runs on is named cfw-workflows-d1-pool; cfw-workflows-d1 is now the Node-driven conformance lane.)
What's gated. The shared lifecycle-hooks parity scenario module's Scenario 2 writes state.tracingContext inside onAgentSuspended (via env.stateStore.saveState, mirroring the langfuse-hooks pattern) and asserts it round-trips through loadState() AFTER the resumed run COMPLETES. On JS / Temporal / DBOS this holds — tracingContext is a durable top-level SessionState field that survives a completion save. On cfw-workflows-d1 it does NOT:
- The CFW D1 store (
store-cloudflare/src/d1-state.ts) packstracingContextinto thesuspension_contextJSON column (V8/V9 schema), alongsidesuspendedAwaitingChildren/suspendedStepId/expiresAt. The suspend-leg hook'ssaveStatewrites it correctly (confirmed — the column round-trips on load). - BUT the resume-leg completion's
saveStateAndPromoteStagingrebuildssuspension_contextfrom the now-COMPLETED (non-suspended) in-memory state, which carries notracingContext, so it writessuspension_context = NULL— clobbering the suspend-time write. The post-resumeloadState().tracingContextis thereforeundefined.
This is a genuine CFW-runtime (D1-store data-model) divergence, NOT a harness adapter artifact: the setupCfwWorkflowsD1 adapter only does AGENT_WORKFLOW.create() + poll + the durable submit; it never touches state persistence. The same workflow body + D1 store are exercised by the standalone hitl-cycle.workflow.test.ts.
Why it's a gate, not a fake. Scenario 1 (the portable client-tool suspend/resume + exactly-once cardinality + previousRunId linkage contract) and Scenario 3 (approval-deny) BOTH EXECUTE and pass on cfw-workflows-d1 — the tracingContext persistence assertion is the only genuinely runtime-divergent leg. Per the spec's gating discipline (refinement #5), Scenario 2 is skipped on cfw rather than weakened.
To close. Promote tracingContext out of the suspension-scoped suspension_context column into a durable top-level column so it survives non-suspended saves like the JS/Temporal/DBOS stores — then drop gateTracingContextPersistence: true from the cfw entrypoint and confirm Scenario 2 EXECUTES.
The fix must cover BOTH CF stores. packages/runtime-cloudflare/src/do/do-state-store.ts (CF-DO — note: this DO store lives in runtime-cloudflare, NOT store-cloudflare) uses the identical buildSuspensionContext packing + NULL-on-non-suspended-save as packages/store-cloudflare/src/d1-state.ts (CFW), so at the store level CF-DO clobbers tracingContext on a completion save exactly the same way. CF-DO's Scenario 2 passes today NOT because its store preserves the field, but because of resume topology: CF-DO resumes IN-PLACE inside the live DO (the in-memory SessionState still carries tracingContext through to the completion save), whereas CFW resumes via a COLD-LOADED fresh workflow instance whose completion save rebuilds suspension_context from a state that already dropped it. So the top-level-column migration must be applied to BOTH d1-state.ts and do-state-store.ts — and when CF-DO later joins a scenario that resumes via a fresh load (not in-place), it would hit the same clobber without it.
Where: gate in e2e/src/parity/scenarios/lifecycle-hooks.ts (ParityOpts.gateTracingContextPersistence + itOrSkipTracing); applied in runtime-cloudflare/src/__tests__/lifecycle-hooks-parity.wf-noiso.test.ts.
D (Test infrastructure overhaul) follow-ups
See test-infrastructure-roadmap.md for the full list. Key items still open at A.2 closure:
- F6 (workspaces on Temporal/CFW Workflows) — explicit fail-fast, needs design work for cross-runtime workspace support.
- F7 (DBOS resume() contract bug fixes) — being addressed in sub-project A.3 (next).
- O5 (workerd-context CF DO + CFW Workflows harness setup helpers) — blocked on sub-project B. Scope sharpened by FU-A2-16 / FU-A2-LIFECYCLE-HOOKS-E2E (closed): real-workerd CFW Workflows workflow-body execution + HITL suspend/resume + lifecycle hooks are now directly covered at the package level (
runtime-cloudflare/src/__tests__/workflows/*.workflow.test.ts, drivingenv.AGENT_WORKFLOW.create(...)+introspectWorkflowInstance). What O5 still owes is the harness-matrix leg: asetupCfwWorkflowsD1(andsetupCfDoD1) that returns a realBackendEnvwhoseexecutoris a workerd-residentCloudflareAgentExecutorreachable over an HTTP/RPC surface, so the sharedgetViableBackends-driven tests (lifecycle-hooks-parity,cross-runtime-resume-matrix, etc.) can run thecfw-workflows-d1/cf-do-d1rows instead of hard-filtering them underIMPL_PENDING_O5.
Cross-service remote agents (2026-07-09)
Deferred items from the cross-service remote-agents effort (spec: docs/superpowers/specs/2026-07-09-cross-service-remote-agents-design.md; guide: docs/guide/cross-service-remote-agents.md). Each item below was verified against the source at the time of writing — file references are the place to start.
FU-REMOTE-01: worker-handler workspace validation + /workspace bridging — open
createRemoteAgentWorkerHandler (runtime-cloudflare/src/do/worker-handler.ts) performs no construction-time workspace wiring validation — the file does not mention workspaces at all — so a DO producer whose agent declares workspaces surfaces the misconfiguration as a runtime WorkspaceFailedError instead of a startup error, and the handler does not bridge /workspace introspection. Parity target: AgentServer's construction-time validation. Spec §4.3 non-goals.
FU-REMOTE-02: migrate legacy snapshot surfaces to checkpoint-pinning — open
Three snapshot surfaces still use the live two-read pattern that round-2 review proved unsound (state and sequence read non-atomically → skipped or duplicated in-flight-step patches for local frontends):
createServerSnapshot(core/src/stream/resumable.ts:62-73) —loadState()thengetSnapshotSequence().- ai-sdk
buildSnapshot(ai-sdk/src/handler/snapshot.ts:163-199) —loadState()thengetStreamInfo().latestSequence; itsgetLatestCheckpointread only suppliescheckpointId/stepCountmetadata, it does not pin state. - DO-native
handleSnapshot(runtime-cloudflare/src/do/durable-object-agent-base.ts:3650-3683) —loadState()thenstreamManager.getLatestSequence().
The remote-protocol handlers already checkpoint-pin; port the same discipline to these three. Spec §3.1.
FU-REMOTE-03: toolCallId stamping on state_patch chunks — open
Optional §7 in-step re-ordering aid: state_patch chunks carry no toolCallId today (verified — no such field on the chunk type). Stamping the originating toolCallId would let consumers attribute same-step parallel-sibling patches. Additive wire field; v2. Spec §7 item 4.
FU-REMOTE-04: oldestRetainedSequence on RemoteStatusResponse — resolved (decided no)
Spec §11 open item 7 asked whether /status should also carry the retention floor. Decided no: the floor is already carried by RemoteSnapshotResponse.oldestRetainedSequence (core/src/types/remote-protocol.ts:208, §3.4) and by the error{code:'TRUNCATED', floor} frame on /sse, so RemoteStatusResponse stays a cheap liveness/recovery poll without duplicating it — a consumer that needs the floor calls getSnapshot. Closes spec §11 item 7.
FU-REMOTE-05: remote completion fold degrades on a bare catch — no typed-error branching — open
processRemoteSubAgentCompletion (core/src/orchestration/remote-sub-agent-completion.ts:187-218) wraps getSnapshot/getUsage in bare try/catch blocks with ZERO instanceof branching, so a genuine 500, a schema-drift error, a missing route on an old peer (RemoteAgentFeatureUnavailableError), a RemoteAgentNotFoundError, and a raceWithTimeout timeout all emit the IDENTICAL warn. The typed errors are constructed by the transport (core/src/transport/http-transport.ts:330-344) but the fold never looks at them. Cheap improvement: a reason classification field on the warn — separates the cases in logs without changing degrade semantics. Open design question: should the CODE match the documented contract (branch on typed errors, e.g. treat feature-unavailable as expected version skew and a 500 as an incident), or should the docs keep matching the code (all read failures degrade identically)? Decide before adding the field.
FU-REMOTE-06: fold read timeouts bound the parent, not the request (no AbortSignal on the transport) — open
raceWithTimeout in the completion fold bounds how long the PARENT waits; it does not cancel the underlying request. RemoteAgentTransport (core/src/types/remote-protocol.ts:366-397) only accepts an AbortSignal on stream() — getStatus/getSnapshot/getUsage/interrupt take none — so a timed-out fold read keeps running in the background. Relatedly, HttpRemoteAgentTransport.fetchWithRetry (core/src/transport/http-transport.ts:357-396) passes NO signal and NO timeout to fetchFn, so those three reads are unbounded at the fetch layer and a 5xx retry loop can outlive the fold. The complete fix widens the transport interface (a breaking change for third-party transports) — hence deferred.
FU-REMOTE-07: fold timeouts are not configurable — open
GET_STATUS_TIMEOUT_MS (5s) / GET_SNAPSHOT_TIMEOUT_MS (10s) / GET_USAGE_TIMEOUT_MS (10s) are module constants (core/src/orchestration/remote-sub-agent-completion.ts:62-64). The helper's only per-call knob (maxFoldStateBytes) originates on RemoteSubAgentConfig and is hand-threaded through four runtime DTOs, so adding a second knob is a cross-runtime change, not a local edit. Too-tight a bound degrades a healthy-but-slow producer (silently dropping its state/usage fold), so the knob matters if operators ever hit it. Deliberately deferred; the rationale is documented inline at the constants.
FU-REMOTE-08: remote stream-proxy dispatch loop is implemented four times — open
The core helper executeRemoteSubAgentDispatch (core/src/orchestration/remote-sub-agent-dispatch.ts:222) is used ONLY by Temporal (runtime-temporal/src/remote-sub-agent-activity.ts:43). Three runtimes hand-roll their own for await (transport.stream(...)) loop with the same concerns (remoteSource stamping, REMOTE_REF_PERSIST_CHUNK_INTERVAL persist cadence, dead-producer observations, ALREADY_RUNNING → attach, terminal fold via processRemoteSubAgentCompletion):
runtime-js/src/execution/remote-subagent-executor.ts:483runtime-dbos/src/steps/execute-remote-subagent.ts:460runtime-cloudflare/src/steps.ts:2745
They are behaviorally equivalent today and pinned by the cross-runtime remote parity battery (packages/e2e/src/__tests__/remote-agent-*.integ.test.ts), but a future fix to one can silently miss the other three. Either converge the three onto the core helper (each has a real durability reason not to — DBOS step boundaries, CFW step boundaries, JS in-process retry/timeout) or document that per-runtime rationale inline at each loop so the duplication is a decision rather than drift.
FU-REMOTE-09: sub-agent CALL-COUNT usage stats are at-least-once under durable retry — open
The TOKEN fold is idempotent at read — selectEmbeddedRemoteRollup (core/src/usage/rollup-utils.ts:260) picks ONE entry per subSessionId by the §6.2 total order. The CALL-COUNT stats are not: processSubAgentEntry (core/src/usage/rollup-utils.ts:129-131) increments subAgentStats.totalCalls/byType[...] for EVERY entry, and the D2 rule records recordSubAgent on both anchor branches, so a durable retry or the dual fast/recovery recording path inflates the call counts. The comment at runtime-cloudflare/src/steps.ts:2257-2258 ("re-recording is idempotent-at-read") overstates this. Fix: either narrow the comment ("token fold is idempotent at read; call-count stats are at-least-once") or dedup the stats by subSessionId at rollup time (groupSubAgentEntries already groups them, so the plumbing exists).
FU-REMOTE-10: computeTerminalUsage is a permanent stub — JS producers never attach end.usage — open
computeTerminalUsage (runtime-js/src/run-loop.ts:3955-3961) returns {} unconditionally: RunLoopInput carries usageContext (record-only), not a rollup-capable UsageStore handle. Consequence: a JS-runtime producer's end frame never carries usage, so every consumer of a JS producer makes a getUsage() recovery round-trip per remote-child completion. Correctness-neutral (the fold still happens via the recovery path, tagged remoteRollupOrigin: 'recovery'), but the end.usage fast path never materializes for JS producers. Fix: thread a UsageStore (or a getRollup callback) through RunLoopInput.
FU-REMOTE-11: EndFrameSchema strictly validates end.state / end.usage — open
EndFrameSchema (core/src/transport/http-transport.ts:43-48) validates state as z.record(...) and usage as UsageRollupSchema, so ONE malformed field fails the WHOLE end frame → malformedFrameEvent → a recoverable: true error event (:473-479). This is asymmetric with the deliberately-opaque chunk passthrough (:424-430), where the payload is not validated at all.
Corrected impact (the original write-up claimed the child's output is discarded — it is not): every consumer treats the recoverable error as a stream drop and falls through to its post-drop getStatus recovery branch (e.g. runtime-js/src/execution/remote-subagent-executor.ts:628-641, core/src/orchestration/remote-sub-agent-dispatch.ts:505-518), which harvests a completed child's output and folds via the recovery path. So the real cost is a wasted stream re-attach plus a degrade from the end fast path to getStatus+getSnapshot+getUsage — output is lost only if that recovery ALSO fails. Suggested fix: parse state/usage as z.unknown().optional() in the envelope and validate them at the fold step, so a bad optional field can't invalidate a valid terminal frame.
FU-REMOTE-12: e2e cross-service workers hand-redeclare SubAgentNamespace with a stale (wrong) rationale — open
packages/e2e/src/cross-service/workers/producer-worker.ts:17-31 and consumer-worker.ts:19-27 locally re-declare the SubAgentNamespace interface and blame "a rollup-plugin-dts quirk [that] drops the export keyword". That diagnosis is wrong: the root re-export simply did not exist, and it was ADDED in commit d93a2a773e (runtime-cloudflare/src/index.ts:187 now exports type SubAgentNamespace). Both workers should import the real type from @helix-agents/runtime-cloudflare and delete the misleading comments.
FU-REMOTE-13: examples/research-assistant-temporal is BROKEN — missing two activities — open
Its hand-written worker registration list (examples/research-assistant-temporal/src/activities.ts, the createActivities return object) omits agentDeclaresPersistentChildren and injectPendingCompletionNotifications. agentWorkflow calls activities.agentDeclaresPersistentChildren UNCONDITIONALLY on every workflow entry (runtime-temporal/src/workflow.ts:670), so any run of this example dies at the first activity call with Activity function agentDeclaresPersistentChildren is not registered on this Worker — the exact failure that hit examples/remote-agents-temporal before it moved to prototype-derived dynamic binding. Fix the example (see FU-REMOTE-14 for the systemic fix).
FU-REMOTE-14: no production-facing canonical Temporal activity binder — open
Root cause behind FU-REMOTE-13: the SDK ships a complete, canonical activity list ONLY as createTestActivities() — a public but TEST-named subpath export (@helix-agents/runtime-temporal/testing → runtime-temporal/src/__tests__/testing/test-activities.ts). Production/example worker authors therefore hand-roll the registration object, and it ROTS silently the moment the framework adds an activity: agentWorkflow proxies the whole GenericActivities surface, so there is no compile-time link between the proxied methods and the registered names. examples/remote-agents-temporal/src/activities.ts works around this by binding the prototype's own methods dynamically (see its bindActivities doc comment). Ship a production-facing binder (e.g. createAgentActivities(activities) from the package root) so no consumer has to choose between a test-named import and a hand-rolled list.
FU-REMOTE-15: the demo frontend's §7 reconstruction logic has NO runtime test — open (REQUIRED)
examples/remote-agents-cloudflare-do/consumer/public/app.js (1511 lines: three reconstruction modes, the heldThrough rewind guard, boundary re-snapshot, late-join, provenance dedup) is the reference implementation of the spec §7 consumer contract and is currently CODE-REVIEWED ONLY — six review rounds each found a real bug in it. Task 6's smoke test (examples/remote-agents-cloudflare-do/tests/smoke.cf.test.ts) runs in workerd via cloudflare:test and cannot execute browser DOM code, so it does not cover any of this. The repo already ships the right harness pattern for exactly this — real wrangler dev + Playwright + a record/replay OpenAI mock, used by examples/opennext-cloudflare-do and examples/research-assistant-cloudflare-do (see example-app-testing.md). Recording a fixture requires an OPENAI_API_KEY — hand-crafting fixtures is explicitly forbidden by that doc, which is why this is deferred rather than done. Treat as REQUIRED, not nice-to-have: this is the only executable check the §7 contract can get.
FU-REMOTE-16: unify createRemoteAgentWorkerHandler (DO) and AgentServer routing behind one protocol router — open
Both the Cloudflare DO worker handler and the Node AgentServer are "a fetch(Request) → Response that speaks the remote-agent protocol" — the same eight-route table (start / resume / sse / status / snapshot / usage / interrupt / abort), auth gate, agentType allowlist, projection, and error envelopes. Today they are SEPARATE standalone impls: createRemoteAgentWorkerHandler imports nothing from agent-server, the projection logic is duplicated (worker-handler-projection.ts vs AgentServer's inline mapper), and the auth posture diverges (worker handler = warn-and-serve; AgentServer = fail-closed-throw).
Better design: a shared RemoteAgentProtocolRouter core — the route table + auth + allowlist + projection + envelope handling — parameterized by a resolveTarget(agentType, sessionId) seam. The only irreducibly DO-specific piece is "resolve sessionId → DO stub + forward"; the Node side resolves to its in-process executor. Everything else is shareable.
Now unblocked by the "no backwards compatibility" directive (free to reshape both public surfaces). Nontrivial refactor; deferred. NOT a correctness issue — both handlers work and are independently tested; this is a duplication/consistency cleanup.
(For the record: FU-REMOTE-17 — the de-backcompat CODE sweep that was the sibling of this item — is DONE; see the remote-agents-de-backcompat.md changeset and the required status/chunk/awaitingClientTool tightening + REMOTE_PROTOCOL_VERSION removal.)
FU-REMOTE-18: consumer-side TRUNCATED → re-snapshot recovery is unimplemented — open (low priority; currently UNREACHABLE)
The spec §3.4 consumer contract — "on a TRUNCATED fail frame, refetch the snapshot and resume from its checkpoint-pinned streamSequence" — is documented and half-built but NOT wired on the consumer:
- Producers emit it correctly:
agent-server/src/http/handler.ts:379andruntime-cloudflare/.../durable-object-agent-base.ts:4087emit afail/errorframe witherrorDetail.code === 'TRUNCATED'+ afloorwhen a consumer requestsfromSequencebelow the retention floor. - The transport surfaces it correctly:
do-stub-transport.tsmaps it to a{type:'error', errorDetail:{code:'TRUNCATED'}, recoverable:false}event. - But no consumer dispatch loop acts on it.
remote-sub-agent-dispatch.ts:504(+ the runtime-js / runtime-dbos / runtime-cloudflare twins) returns{kind:'failed', failureReason:'remote-failed'}on ANY!recoverableerror, and TRUNCATED isrecoverable:false— sogetSnapshot()is never called and a TRUNCATED fails the sub-agent even though its output/state is recoverable. (A non-recoverable TRUNCATED even bypasses the existing post-dropgetStatusharvest.)
Reproduction test (committed, teeth-having): packages/e2e/src/__tests__/cross-service-do-do-truncated.integ.test.ts — a describe.skip known-gap guard that drives the real shared dispatch through a mid-stream TRUNCATED and asserts the consumer re-snapshots + completes gapless. It flips green the instant a consumer implements the branch.
Why low priority / currently UNREACHABLE in production: the retention floor (oldestRetainedSequence) is effectively always 0 today — nothing advances it. resetStream() (the only floor-raiser) is called only in tests; the run-scoped cleanupToStep(minSequence) deletes a partial step's RECENT chunks (above the floor), not old ones, and has no production caller; and there is no automatic retention cap / TTL / max-size trimming on stream chunks (streams retain everything). So a consumer's fromSequence (≥ 0) is never below the floor, and TRUNCATED is never emitted in normal operation. This is a designed-ahead §3.4 resilience signal for a FUTURE with bounded stream retention; implement the consumer recovery when (if) that retention is wired.
The fix (when needed): in each dispatch loop's case 'error', branch on event.errorDetail?.code === 'TRUNCATED' BEFORE the !recoverable fail — call transport.getSnapshot(childSessionId), adopt its state, re-attach stream({ fromSequence: snapshot.streamSequence }), then un-skip the test above.
CI health (MR !272)
MR !272 (branch ci-health) bumps wrangler/Miniflare, passes the test env vars through turbo, fixes the Cloudflare Node-proxy flake family, relaxes two wall-clock bounds and stops .integration_base retrying script_failure. Entries here are either new, or record the resolution of an entry that another MR files (so that entry is not duplicated). When that other MR lands, mark its entry fixed and point it here.
Resolutions of entries filed on other branches
- FU-FLAKE-OPENNEXT-WRANGLER-DEV-CRASH (reported on !269,
redis-create-session-atomic; this is the only entry for it): fixed by !272. wrangler 4.141.0 no longer dies under the refresh-spam tests. The in-workerUncaught Error: Network connection lost/Unable to enqueueerrors are still logged (see FU-OPENNEXT-UNCAUGHT-ON-CLIENT-ABORT below) but are no longer fatal towrangler dev. The example jobs now fail the readiness loop when the server never answers, and after the run print an explicit "wrangler dev exited during the Playwright run" line if the process died. Evidence: over the 45 pipelines before !272, 3 of 36test:examples:opennext-cloudflare-doruns crashed (16755525964 on main, 16756530064 and 16756897246 on !269), all on 4.63.0; locally 0 crashes in 7 full runs on 4.141.0 (2 of them under a 10-processyesCPU stress) and 0 in 3 full runs on 4.63.0 on the same machine (the crash does not reproduce on macOS arm64). The CI runs on !272 are the evidence for the Linux amd64 runner; see the MR description. Correction to the entry: research-assistant job 16756390703 did not crash and has no503. It failedstatus should transition from started/running to endedbecause the replayed mock finished the run before the first status poll; !272 fixes that assertion (completedis a valid first observation). - FU-FLAKE-CF-D1-PROXY-FETCH-FAILED (filed on !264,
cf-multirun-flake-fix): fixed by !272. Root cause: Miniflare 3's Node proxy (undici 5.29) drops loopback connections under concurrent load (ECONNRESET, surfacing asTypeError: fetch failed). Transport A/B, 4 runs of concurrent D1 + DO proxy calls: Miniflare 3 22 of 13,440 calls failed, Miniflare 5 0 of 13,440. The same failure explains the CAS-race losers that sawD1StateErrorinstead ofStaleStateError, the large-stateappendMessagesfailure, theG1.concurrent-writersDO failures and the durable-lockacquirefailure.concurrent-subagent-fold.integ.test.tsalso gets Workflows-style bounded retry-on-throw in its mockstep.do(runStepWithWorkflowRetries) and printsresult.errorin every status assertion. With one injectedfetch failedper test, the old mock fails 3 of 6 tests and the new one passes 6 of 6. No transport-level retry was added: the writes are not idempotent, and the bump removes the transport failure. - FU-FLAKE-DURABLE-LOCK-RACE-RETRY (filed on
b8-dod-catchup): fixed by !272. The race test now uses a fresh resource per attempt and releases infinallyviaPromise.allSettled; its first failure (a Miniflare 3 proxyECONNRESET) is also gone with the bump. - FU-FLAKE-MINIFLARE-SYNC-PROXY-DESYNC (on main): marked fixed in place, in "Known CI flakes" above.
- FU-FLAKE-REMOTE-DUP-START-RACE (on main, fixed by !271): listed in the exposure table below only.
FU-FLAKE-CF-DO-STREAM-WALLCLOCK-BOUND: wake-latency bounds failed under CI load — ✅ fixed (CI health 2; !272's bound raise did not fix it)
Reopened and fixed (CI health 2): !272's 2000ms bound still failed (expected 4146 to be less than 2000, !288 16900882197), and the same class hit six more contract cases (G1 drain 5s, CU.empty-backlog 1s, WL.* / RF.* 1s endsWithin; 7 sightings after !272). The contract now has no positive wall-clock window: each case awaits the event, and DOStreamManager / RedisStreamManager run it with the new internal safetyNetPollMs: Infinity, so only the write's or transition's wake can deliver. Red-first: removing endStream's / failStream's wake makes the DO contract hang (test timeout), and dropping Redis's end pub/sub handling fails 6 Redis cases the same way. The poll itself keeps a test per manager (do-stream-manager-resumable.cf.test.ts "safety-net poll", redis-stream-safety-net.integ.test.ts).
Tests: G3.bounded-staleness in packages/core/src/testing/stream-manager-contract.ts (< 400ms; runs for store-memory, store-redis, store-cloudflare and the DO stream manager) and iterator returns done on endStream well before any safety-net tick in packages/runtime-cloudflare/src/__tests__/do-stream-manager-resumable.cf.test.ts (< 500ms).
Symptom: expected 404..588 to be less than 400/500 with the wake working. Jobs: main 16755525962 (588 vs 500) and 16754640436 (407 vs 400, 531 vs 500); 16756029846 (404); 16755485929 (434); 16749367148 (556 vs 500); 16756390700 (411); store-memory test:unit 16756727163 (579). On !272's first pipeline, before this fix landed, test:cloudflare 16758170964 failed the same way (822).
Fix: both bounds are now < 2000ms, and G3's nextWithTimeout is 4s. The bounds only need to separate a working wake (~50ms) from the 5s safety-net poll; the old values measured scheduler latency under load. Local: store-memory contract 20/20 green on the new bound; the old bound did not fail locally (0/20 even under CPU stress), so the CI jobs above are the red evidence.
FU-CF-DO-RESUMABLE-READER-DROPS-FRAMES: live DO continuation truncated after the first frame of a read — ✅ fixed (MR !272)
Class: product bug (silent data loss on the live stream), latent on main, exposed deterministically by the wrangler 4.141 / workerd 1.20260925 bump.
Symptom: in the opennext example, the confirmation after a client-tool turn streamed live as "The page" and stopped; a reload showed "The page background has been changed to blue." refresh-exact-messages.spec.ts C failed 3/3 on 4.141 (3/3 green on 4.63). Probably also behind the long-running comprehensive-refresh.spec.ts:472 flake ("turn 2 has 0 text parts after refresh"), which Playwright's in-run retry has been hiding on main.
Root cause: DOStreamManagerClient.createResumableReader (packages/runtime-cloudflare/src/do/do-clients.ts) split each network read into lines and returned from inside the line loop on the first chunk, discarding every later frame of that read. Newer workerd coalesces back-to-back DO SSE writes into one read.
Fix: complete lines are queued across next() calls. Unit test yields every frame when one read() carries several SSE frames fails before (1 of 4 deltas) and passes after. Changeset: @helix-agents/runtime-cloudflare patch. Conformance finding RM-51 (S1), pending the owner's catalogue batch 4. No scenario observes it yet (the owner's pending plan is 0c, a cf-do host cell), so the record's evidence exception or its flip to fixed is the conformance owner's to make.
FU-CI-TURBO-ENV-PASSTHROUGH: turbo's strict env mode dropped test env vars — ✅ fixed (MR !272)
The root pnpm run test:integration dropped RUNTIME_DBOS_INTEG (the DBOS suite collected 0 files and exited 0) and TEMPORAL_ADDRESS (Temporal tests used localhost:7233, which worked in CI only because the Kubernetes executor exposes service containers on localhost). Fixed with globalPassThroughEnv in turbo.json and a CI guard, pnpm run check:turbo-env (lint job), that fails when test code reads a variable turbo would drop. The conformance owner was filing this; no ID existed on main or conformance-catalogue-snapshot-truth when !272 was written, so this is the ID. The DBOS gate itself is removed by !259 (dbos-integ-ci-unpause), which also owns the test:integ:dbos job.
FU-OPENNEXT-UNCAUGHT-ON-CLIENT-ABORT: opennext worker throws on client disconnect mid-stream — open
Owner: the B8 session. Under the refresh-spam tests the worker logs Uncaught Error: Network connection lost and Uncaught TypeError: Unable to enqueue from OpenNextNodeResponse._internalWrite (writing into a stream the browser already closed). Non-fatal on wrangler 4.141, but it is noise in every run and was the lead-in to the 4.63 crashes. Fix direction: cancel the upstream reader when the Node response closes, or catch the enqueue after close in the route's streaming bridge.
Located (CI-health-3, 2026-10-07): the throw is in @opennextjs/aws's cloudflare-node wrapper (dist/overrides/wrappers/cloudflare-node.js:42-81): the platform body is a ReadableStream with no cancel, and its Writable turns controller.enqueue after close into callback(e), which Node's pipe re-emits uncaught. Our own body streams guard enqueue-after-close. Our side's gap (nothing stops reading upstream on a disconnect) was CP-102, now fixed; only the third-party throw remains. 0–5 per local run with or without enable_request_signal; see FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT's re-measure. An upstream OpenNext issue is drafted with CP-102's MR (filing is the user's call).
FU-FLAKE-OPENNEXT-PROXYWORKER-DROPPED-POST: an opennext example spec's send fails with wrangler dev's own 500 after a dev-server stall — open
Owner: example apps (the opennext-cloudflare-do specs). Rated by: the conformance owner, 2026-10-19 (orchestrator: an FU, not a finding; no quarantine, retried on sight while open).
Seen: examples/opennext-cloudflare-do/tests/comprehensive-refresh.spec.ts:610 ("AGGRESSIVE refresh after follow-up completion"), on two unrelated MRs on base f0b3da371e: !446 job 17296523426 and !447 job 17296629295. The spec reports "the follow-up send answered HTTP 500 (turn 2 was never committed …): Error: Network connection lost", then an R34 page.reload 15 s timeout. !446's run also failed mid-text-reload-scoring.spec.ts #3 ("the mock never held the text step") in the same window, very likely the same drop (its send died before any model call; inferred, the session is not tied to the test).
- The 500 is wrangler dev's, not the product's. Each failed send is logged as
Error inside ProxyWorker (the affected request failed; the dev server continues): POST /api/chat/:sessionId (failed after 1 attempt): Network connection lost.The chat route's own failures are typed JSON ({ error, code }); this one never reached it. - Cause: a dev-server stall under runner load, then a dropped ProxyWorker → UserWorker connection. In the same windows static
/_next/static/chunks/*.js304s take 26.9 to 28.7 s andGET /27.7 s; the ProxyWorker then logs "recovered on attempt 2 after a dropped connection to the UserWorker" for every in-flight GET. It retries those, but not a non-idempotent POST, which fails. Production has no ProxyWorker. - Related: FU-OPENNEXT-UNCAUGHT-ON-CLIENT-ABORT (above): its
Uncaught Error: Network connection lostlines run through both logs, passing tests included, and may help destabilise the dev worker under refresh spam, but they are not the failed request. - Fix direction (the specs, not the product): a send whose answer is the dev ProxyWorker's own failure (status 500, plain body
Network connection lost) AND whose turn the snapshot shows NOT committed is re-sent once, logged, under the same request key. Never a blanket retry, and never when the snapshot holds the turn.
FU-FLAKE-POOL-WORKERS-STACKED-STORAGE-LOST: test:cloudflare isolated-storage teardown loses the connection — ✅ fixed (CI health 2)
Fixed (CI health 2): a keep-alive race on Miniflare's Node loopback server, upstream cloudflare/workers-sdk#14848. vitest-pool-workers 0.9.14 pins miniflare 4.20251011.0, whose loopback server keeps Node's keep-alive timeout (5s + Node 24's 1s buffer). workerd pools the /storage push/pop sockets and closes idle ones itself at 5s, so an idle machine never races; on a starved runner its timer fires late, Node closes first, and the pop after a ~6s test goes out on a dead socket. In the sweep the failure rate by test duration was 0% under 2s, 1.3% at 5–6s and 15.1% at 6–8s; 96% of the failed tests ran ≥5s against 28% of passing ones. Fix: upstream's own change (#14850, server.keepAliveTimeout = 0) as patches/miniflare@4.20251011.0.patch; upstream only ships it in pool-workers releases that need vitest 4. Guard: runtime-cloudflare/src/__tests__/pool-workers-loopback-keepalive.test.ts (red on the stock Miniflare: "closed an idle keep-alive socket"). Drop the patch with the pool-workers bump.
Owner: the B8 session. filestore-provider.cf.test.ts (runtime-cloudflare) and CloudflareFileStoreFileSystem grep tests fail with Error: Network connection lost from WorkersTestRunner.updateStackedStorage (@cloudflare/vitest-pool-workers@0.9.14). Jobs: !272 16758170964, !270 16757391364, !264 16754841608. Same family as FU-FLAKE-WORKERD-INTROSPECT-TEARDOWN (pool-workers teardown). pool-workers stays on ^0.9 in !272 because newer releases need vitest 4.
More sightings (2026-09-28), same signature (Error: Network connection lost out of WorkersTestRunner.updateStackedStorage / ensurePoppedActiveTryStorage, workerd stream disconnected prematurely), now spanning more runtime-cloudflare files — atomic-suspend-write, do-stream-manager-paused, do-stream-manager-cleanup-to-step, do-stream-manager-contract (G1), forced-completion-do, and the filestore-filesystem / filestore-provider grep and R2 tests, several failing together in one job: main 16766713719; !281 16766313974; !282 16767233852 and 16767365277 (!282 changes no runtime-cloudflare or store-cloudflare file — only core's ScriptedModel test helper, which no cf test imports; a retry of 16767233852 passed as 16767441949).
FU-FLAKE-CFW-FORCED-COMPLETION-CONNECTION-LOST: forced-completion.workflow.test.ts fails with Network connection lost — ✅ fixed (CI health 2)
Fixed (CI health 2): the same updateStackedStorage loopback race as FU-FLAKE-POOL-WORKERS-STACKED-STORAGE-LOST (vitest.workflows.config.ts uses isolated storage), fixed by the same patch. Its later 120s timeout ("D-fix", main 16882767277) ran while the job rebuilt five Next.js apps (FU-CI-TEST-JOBS-REBUILD).
Test: packages/runtime-cloudflare/src/__tests__/workflows/forced-completion.workflow.test.ts (test:workflows, @cloudflare/vitest-pool-workers@0.9.14). Signature:Error: Network connection lost., with a burst of jsg.Error: Instance dispose (broken.outputGateBroken) in the log. That is the FU-FLAKE-WORKERD-INTROSPECT-TEARDOWN family, but that entry names other files. Jobs: !272 16758970357 ("runs terminal-text → forced finish"); before !272: !264 16750600721, !262 16754754857 and !266 16755705444 ("resumes in forced mode on a cold start"). !272 does not change the pool-workers version, the Workflows runtime or this test.
FU-FLAKE-REMOTE-AGENTS-EXAMPLE-USAGE-FETCH: remote-agents example logs a failed usage fetch after a workerd connection reset — ✅ fixed (WE-2; RE-OPENED 2026-10-24 by jobs 17439386801 and 17440876190, closed by W1's remote-agents flake MR; also filed as FU-EXAMPLE-REMOTE-AGENTS-USAGE-RACE)
RE-OPENED 2026-10-24, closed by W1's remote-agents flake MR: the spec's networkidle wait, not the demo. It failed twice on !496: MR pipeline 2984746656 job 17439386801, and train car 2985027174 job 17440876190.
- Where. Both times the 180 s test timeout hit at
cross-service.spec.ts:124, thepage.waitForLoadState('networkidle')right after "a reload while the demo's usage refresh is in flight" (:118-123). - An idle page that never read as idle. After that reload both job logs show only
GET /andGET /app.js, then no request until the timeout.consumer/public/app.jsstarts no fetch at load (:1565-1574sets text, adds listeners and renders), so the page was idle, yetnetworkidlenever fired. The wait spanned a navigation that aborted the in-flight/api/child-usagefetch. Playwright itself discouragesnetworkidle, and the spec also used it at:103. - No trace.
playwright.config.tshadretries: 0withtrace: 'on-first-retry', so no trace was ever recorded and each job uploaded onlyerror-context.md. - Fix. Both waits now wait for the reloaded page's script to have run:
#session-idshows the pinned sessionId, written in the same task that wires the buttons. The config recordstrace: 'retain-on-failure'. - Not the cause: job 17440876190's workerd
Uncaught Error: Network connection lost.at the reload is absent from the other job (FU-EXAMPLE-REMOTE-AGENTS-WORKER-UNCAUGHT-CONNECTION-LOST).
Root cause (WE-2, 2026-10-20): a test-side race, not the connection reset.
- What the demo does. When the run's stream ends, the demo refreshes its usage panel: the writer's rollup, then each remote child's
/api/child-usage, in turn (consumer/public/app.jsrefreshUsage). - What the spec did. It reloaded the page (
page.reload()) without waiting for that refresh. - The failure. On a loaded runner a
child-usagefetch was still in flight at the reload (wrangler logs it AFTER the reload'sGET /), so the browser aborted it. The demo then logged "[demo] usage fetch failed: TypeError: Failed to fetch". - Green runs finish every usage request before the reload (9–25 ms each, against 38–142 ms on the red main runs 2966843082 and 2968887323). workerd's "Network connection lost" / "Connection reset by peer" lines come after the test, when the browser tears down the open stream, and appear in green runs too.
- Fix.
- The demo marks each finished refresh (
#usage[data-refreshed]), and the spec waits for it before reading the panel and reloading. The panel check was vacuous before: it could read an empty panel. - The demo no longer shows an error for a fetch the browser aborted because the page itself is going away (
pagehide). - The spec now holds every
child-usageanswer 1.5 s and reloads during a refresh, so the race is reproduced on every run.
- The demo marks each finished refresh (
Update (CI health 2): seen on main 16765146699, 16765362858 and 16825373949 (2026-09-27 → 09-30, the same "no console/page errors" assertion after Uncaught Error: Network connection lost in wrangler dev), then 38 green runs in a row with no retry (this job has retries: 0). Not reproduced; left open, owner orchestrator → CI-health-3 (it is an example-app job).
Test: examples/remote-agents-cloudflare-do/tests-e2e/cross-service.spec.ts:22, the assertion at :103 ("no console/page errors across the full flow"). Symptom: the page logs [demo] usage fetch failed: TypeError: Failed to fetch right after workerd reports Connection reset by peer (os error 104) (wrangler 4.141) or Network connection lost (wrangler 4.63). Every earlier assertion passes, and the server stays up. Jobs: !272 16759512532; !268 16757428815 (the old wrangler, so pre-existing on main's tooling). Not this entry: the job's other four failures in the same window (16756390704, 16756192324, 16755700166, 16756092320) are a toBeTruthy state-fold stall. That is CP-77, owned by S5.
FU-EXAMPLE-REMOTE-AGENTS-WORKER-UNCAUGHT-CONNECTION-LOST: a browser abort surfaces as an uncaught error in the demo's worker — open (owner: orchestrator)
Job 17440876190 (!496's train car 2985027174, 2026-10-24) logged workerd ✘ [ERROR] Uncaught Error: Network connection lost. at 19:50:04, the instant the spec reloaded the page over an in-flight /api/child-usage fetch. It is most likely the aborted request's worker-side fetch surfacing as an uncaught error in the consumer worker. A client abort should not be an uncaught worker error. It is absent from the other failure of the same step (job 17439386801), so it did not cause FU-FLAKE-REMOTE-AGENTS-EXAMPLE-USAGE-FETCH's re-opening. Hygiene only: place it from a stack (the next failure's retained trace and worker log) before fixing.
FU-FLAKE-REMOTE-AGENTS-CF-DO-SMOKE-TIMEOUT: the remote-agents cf-do example smoke times out at 60s under a loaded runner — open (owner: B8 session)
Test: examples/remote-agents-cloudflare-do/tests/smoke.cf.test.ts ("cross-service remote agents (DO → DO over a service binding) > completes the tree, folds child state + usage, snapshots the child, and projects the external mount"), in test:unit. Symptom: Error: Test timed out in 60000ms — no assertion failed. Job: !282 16767233838 (2026-09-28 00:15), in the same minutes the same runner's test:cloudflare (16767233852) was losing workerd connections (FU-FLAKE-POOL-WORKERS-STACKED-STORAGE-LOST). !282 changes no example, runtime or store code, and the smoke passed 3/3 locally at the same head. No earlier sighting of this exact signature was found in the recent failed test:unit jobs; if it recurs, check whether it co-occurs with the pool-workers connection losses (a shared workerd/runner load cause) before raising the timeout.
FU-FLAKE-D1-CAS-RACE-LOSER-D1STATEERROR: a D1 CAS-race loser rejects with D1StateError, not StaleStateError — ✅ fixed (guarded-commit outcome MR)
Also tracked as FU-FLAKE-D1-ATOMIC-SUSPEND-WRITE-LOADSTATE-ERROR. Fixed (guarded-commit outcome MR, spec 2026-10-16-guarded-commit-outcome-design.md): the loser's D1StateError was its own pre-read (loadState) failing under contention: isTypedStoreError let a D1StateError through untranslated, and the old version poll ran only after a failed batch. D1 now resolves a failed pre-read inside runGuardedCommit (a moved pinned version → StaleStateError, else CommitNotAppliedError), and the contract case commit.outcome-lost-race-pre-read.{step,state} reproduces the CI shape deterministically (red on the old store). The CAS-race test keeps its 20-writer case and now prints each loser's name, message and cause chain.
Test: packages/store-cloudflare/src/__tests__/integration/atomic-suspend-write.integ.test.ts:515 ("all losers throw StaleStateError when UNIQUE(messages.sequence) fires"), assertion :548. Jobs: main 16758460796 (e0aae15185); earlier !259 16756105477 and 16756470721, and !264 16756272942. Not proven to be the Miniflare proxy transport failure. D1StateError is the wrapper D1StateStore puts around any unexpected error, and the proxy's fetch failed surfaces that way elsewhere. But CI truncates the wrapped cause, and it did not reproduce locally in 0/20 runs on !272 (Miniflare 5, 10 of them under CPU stress) or 0/15 on main (Miniflare 3, under CPU stress). Next step: print r.reason and its cause in the assertion message, so the next CI occurrence says whether it is transport (resolved by the Miniflare 5 bump) or a UNIQUE-error classification gap in saveStateAndPromoteStaging.
FU-FLAKE-INTERRUPT-RESUME-FIXED-SLEEP: interrupt-resume e2e interrupts after a fixed 500 ms sleep — ✅ fixed (CI health 2)
Fixed (CI health 2): "should handle checkpoint restoration on resume" interrupts after the run's second step_committed marker and reads the checkpoints once after result() (no sleep, no poll). Seen again on main 16825373934 (2026-09-30). The file's other sleep-then-interrupt cases assert nothing that depends on how far the run got, so an early interrupt cannot fail them. Since S5 a run-start commit also writes a checkpoint, so the old "any checkpoint" assertion held even for an instant interrupt; the test now also asserts the latest checkpoint's stepCount is at least 2. With the old fixed sleep forced to 0ms it fails on that assertion (expected 0 to be greater than or equal to 2).
Test: packages/e2e/src/__tests__/interrupt-resume-d1-with-js.integ.test.ts, "Interrupt/Resume E2E - JS runtime + D1 store › Edge Cases › should handle checkpoint restoration on resume". It fails at :659 with expected 0 to be greater than 0: listCheckpoints is still empty after the interrupt.
Job: !269 job 16760751939 (test:integ:e2e 3/4, pipeline 2886547356). It passed 5/5 isolated locally.
Root cause: a test defect, the fixed-sleep proxy pattern. The test sleeps 500 ms, then calls interrupt(), and expects at least one checkpoint to exist by then. Under CI load the interrupt can land before step 1 commits. The old script_failure auto-retry hid this until !272 removed it.
Fix direction: wait for a real signal before interrupting, not a fixed 500 ms. Either poll for a persisted checkpoint, or hold a tool step on a gate and interrupt while it is held.
Related: !259's FU-DBOS-E2E-FIXED-SLEEP-PROXIES is the DBOS sibling. The other interrupt-resume-* lanes probably share the pattern and should be fixed together.
FU-CI-WORKERS-TYPES-5-PEER: wrangler 4.141 wants @cloudflare/workers-types@^5 — open
Owner: the B8 session. pnpm peers check reports wrangler 4.141.0's optional peer @cloudflare/workers-types@^5.20260925.1 unmet (the repo is on ^4.2026…, and the published packages declare ^4.0.0 as their own optional peer). Harmless today (optional peer, types only). Moving to v5 changes the published peer range, so it needs its own MR with a typecheck pass.
Exposure: integration shards that were green only through the removed script_failure retry
Sweep of the 45 pipelines before !272 (include_retried=true), integration shards whose first attempt failed with script_failure and whose automatic retry passed. With the retry gone, these show up as red until their fixes land.
| Failing test (first attempt) | Jobs (failed attempt) | Cause / owner |
|---|---|---|
e2e remote-agent-flow "duplicate start throws ALREADY_RUNNING" | 16755525951 (main), 16757763574, 16757391353, 16756192309, 16756128346, 16756105472, 16755700151 | FU-FLAKE-REMOTE-DUP-START-RACE, fixed by !271 (on main) |
runtime-cloudflare concurrent-subagent-fold (append/modify/removal/failed-child) | 16757391357, 16756390693, 16756192313, 16754841601, 16750450972 | FU-FLAKE-CF-D1-PROXY-FETCH-FAILED, fixed by !272 |
e2e ai-sdk-redis-temporal "Non-streaming mode" (InvalidStoredStateError, status undefined) | 16755699318, 16755485917, 16754864347, 16754754845 | RM-50 (Redis createSession not atomic), fixed by !269 |
store-cloudflare durable-lock "only one holder when racing" | 16754754849, 16750857812 | FU-FLAKE-DURABLE-LOCK-RACE-RETRY + Miniflare 3 ECONNRESET, fixed by !272 |
e2e ai-sdk-d1-with-js (every later D1 call (message?.id === id)) | 16755572339 | FU-FLAKE-MINIFLARE-SYNC-PROXY-DESYNC, fixed by !272 |
e2e multi-message-execute-dbos-persistent "idle-timeout exit then new message" | 16754864348 | CP-74 (DBOS persistent send into an exiting workflow), owned by B2-core |
store-cloudflare atomic-suspend-write CAS race "all losers throw StaleStateError" | 16758460796 (main), 16756105477, 16756470721, 16756272942 | FU-FLAKE-D1-CAS-RACE-LOSER-D1STATEERROR: fixed by the guarded-commit outcome MR (the CAS loser's pre-read) |
e2e client-tool-happy-parity dbos-postgres | 16758483177 | FU-FLAKE-DBOS-CLIENT-TOOL-PARITY-TIMING, owner B8, filed on !266 |
Repeats after the sweep window: CP-74 on main 16759539277 and on !266 16759542438 (owner B2-core); the fold failure 16758483181 (fixed here); durable-lock 16758324556 (fixed here); duplicate start 16758324552 (fixed by !271); RM-50 16757995539. RM-50 is fixed by !269, which is not merged yet, so ai-sdk-redis-temporal shards can go red until !269 lands (it merges right after !272).
Not in the table: CP-77 (unowned) and FU-CONF-CP62-DBOS-SETTLE-TIMING (conformance owner) did not appear as auto-retried integration shards in this window. The conformance jobs never auto-retried script_failure; !271's test:conformance:dbos 16756806141 and conformance:verify 16756806142 were retried by hand.
CI health 3
Open items CI-health-3 found while chasing CI reds. They are not fixed in its MRs.
FU-FLAKE-CONF-EXECUTOR-DRIVER-DISPOSE-WALLCLOCK: conformance executor-driver.test.ts dispose bound is a wall-clock upper bound — ✅ fixed (!297)
Fixed (!297, conformance owner batch 10): the dispose bounds are now driven on vi.useFakeTimers() / advanceTimersByTimeAsync, and the test asserts which call hit which bound, the order, and the fake-clock settle instant (first call inside its bound, then drainBoundMs, then the hung after-drain call cut at cellWorkBoundMs) instead of elapsed milliseconds. The sibling wall-clock upper bounds in the same describe (the closed-but-unsettled reap case and the hung-first-call case) got the same treatment. The code under test reads only setTimeout, so no test seam was needed.
The original report follows.
Owner: the conformance owner. Surfaced by: !293 pipeline 2911240084.
- Where: arm64
conformance:verifyjob 16923705133, gate stepunit-tests, on the developer runner host between 19:09 and 19:15 UTC. - Retry: job 16923979153, green, disclosed in !293.
- What was running on the runner:
- !293's other jobs;
- no other helix-agents pipeline;
- no CI-health-3 load runs;
- other projects' jobs and local sessions: unknown.
- Test:
packages/e2e/src/conformance/__tests__/executor-driver.test.ts› "dispose() with env.terminateCellWork — 'stopped' judged from backend truth › I3/N2: the SLOWEST dispose path takes disposeWorstCaseMs — first call just inside its bound…" (from !290, 4a82e0b347). - Failure:
expected 501 to be less than or equal to 490at:3033, i.e.expect(took).toBeLessThanOrEqual(worst + TOLERANCE_MS). A positive wall-clock window that a loaded runner overshoots. - Plan (owner, done): drive the dispose bounds with fake timers and assert which call hit which bound, not elapsed ms.
FU-FLAKE-POSIX-SHELL-SIGKILL-ESCALATION-WALL-CLOCK: workspace-posix-core shell.test.ts times the SIGKILL escalation against a wall-clock bound — open
- Test:
packages/workspace-posix-core/src/__tests__/shell.test.ts:202, "timeout escalates to SIGKILL if the process traps SIGTERM". It assertselapsed < 3000ms (a 100 ms timeout + the 2000 ms grace + overhead). - Seen: !332 pipeline 2932535786,
test:unitjob 17075264721 (2026-10-10 ~01:14Z):expected 3090 to be less than 3000. The host load was about 28 on 14 CPUs (!332's arm64 cloudflare / workflows / cfw-engine jobs, with the conformance owner's cfw matrix shard overlapping). Unrelated to !332's diff. - Fix direction: assert the causality, not elapsed time: the process was killed by SIGKILL after the grace (exit signal / exit code, the "trapped" output showing SIGTERM was delivered first, and that the run ended well before the trap's 100 s sleep). Keep a wall-clock bound only as a generous hang guard.
- Owner: CI-health (parked; non-CF).
FU-FLAKE-CFW-PARALLEL-SUBAGENT-EVENTS-WALL-CLOCK: runtime-cloudflare workflow.test.ts proves parallel sub-agent spawns with a 50 ms start-time window — open
- Test:
packages/runtime-cloudflare/src/__tests__/workflow.test.ts:2545, "should spawn and wait for multiple sub-agents in parallel using events". It asserts the spawn and wait steps of the two sub-agents started within 50 ms of each other (:2693,:2698). The sibling parallel-tool test asserts the same shape at:2539-2540. - Seen: !332 pipeline 2932535786,
test:unitjob 17075264721 (2026-10-10 ~01:13Z):expected 54 to be less than 50, at host load about 28 (same overlap as above). Unrelated to !332's diff. - Fix direction: prove parallelism by ordering or causality, not elapsed time: e.g. both spawn steps start before either completes (a gate that holds the first until the second has started), and both waits are entered before any sub-agent result is delivered.
- Owner: WB.
FU-FLAKE-REDIS-STREAM-PAUSE-PUBSUB-WALL-CLOCK: store-redis proves a pause arrives over pub/sub with a 1 s wall-clock bound — open
- Test:
packages/store-redis/src/__tests__/redis-stream-pause-pubsub.test.ts:94, "reader observes pause via pub/sub within 200ms (not 5s poll tick)". It timespauseStream()to the reader's end and assertselapsed < 1000ms, as a stand-in for "the pause arrived over pub/sub, not on the 5 s status-poll tick". - Seen: main 5ac3ac53b8 (!314,
.gitlab-ci.ymlonly), pipeline 2933885949,test:unitjob 17083046904 on the arm64 Mac runner (2026-10-10 ~17:06Z):expected 1456 to be less than 1000. The host was running main's own arm64 test jobs in parallel (unit, cloudflare, workflows) and another project's vitest, at a 1-minute load average of about 20. Unrelated to any diff; the orchestrator retried it under §A3 as job 17083365904. - Fix direction: assert the causality, not elapsed time: the reader ends because the pause message arrived over pub/sub, before any status-poll tick could have ended it (e.g. make the poll interval injectable and far longer than the test, or observe that the subscriber's handler delivered the pause), and keep a wall-clock bound only as a generous hang guard.
- Owner: CI-health (parked; non-CF).
FU-FLAKE-CFW-SPIKE-ITEM34-REARM: cfw spike.integ.test.ts items 3+4 failed the gate whenever its sleep window broke on a slow runner — ✅ fixed (owner batch 17)
- Seen: !331 pipeline 2931779326,
conformance:verifyjob 17071864020 (its harness-tests step). The FU-FLAKE-CFW-SPIKE-ITEM34-SLEEP-RACE guard threw its ownHarnessError: "control's 4000ms sleep deadline passed 345ms before the recreate finished (step a to evict start 2760ms, evict took 1585ms)". It is 1 of 86 gating runs since 2026-10-05 (1.2%). The guard was right not to report an engine verdict, but a gating cell that fails whenever the runner is slow still fails the gate. - Fix: the window is measured on every attempt. When it broke, the cell terminates that attempt's instances and re-arms with fresh ones and a sleep derived from the measurement: twice the measured create-to-recreate time, at least double the last sleep, capped at 20 s, 3 attempts. Only a window that breaks on every attempt is a
HarnessError. An instance first seencompletebefore step a was observed (a whole sleep between two status reads) also counts as a broken window, instead of timing out the step-a wait as if the engine had failed; one thaterroredor wasterminatedstill fails the test. A first attempt that holds costs what it did (about 9.7 s). - Proof (forced first sleeps, working tree, reverted): at 20 ms the window broke (the deadline passed 253 ms before the recreate), re-armed at 546 ms (2× the measured 273 ms) and held with 283 ms slack. At 1 ms the instance finished before the evict (attempt 1), broke again at 146 ms (88 ms overrun), and held at 468 ms on attempt 3 with 220 ms slack.
- After (×5, under the stress lock, 2026-10-10 00:18-00:44Z): 10 of 10 passed (5 on main, 5 with the fix); with the fix every window held on its first attempt, with 2604-3772 ms of slack (arm 228-1396 ms, evict 202-1238 ms), at host loads up to 40. It held again (3642 ms slack) in the post-rebase
test:conformance-harnessrun.
FU-FLAKE-CFW-SPIKE-ITEM34-SLEEP-RACE: cfw spike.integ.test.ts items 3+4 raced its own 2 s sleep — ✅ fixed (!303)
Fixed (!303): the cell stays gating (its assertions are the evidence that engine state survives dispose and recreate) and now makes its timing window explicit:
- all four instances are created first and awaited together, so they pass step a nearly at once;
- the sleep is 4 s, not 2 s, and the "past the deadline" wait is relative to the latest recorded step-a time (
max(tStepA) + SLEEP_MS + 1_500), not a fixed delay after the recreate; - a window guard records when each instance was first seen
runningwith one step output and whenevict()resolved. If any instance's sleep deadline had already passed by then, the cell throws aHarnessErrornaming the instance and the measured gap. That is a harness timing fault, never a pass and never an engine failure; - the title no longer says "recorded, not gating".
The cell's runtime went from about 5.9 s to about 9.7 s.
The original report follows.
- Test:
packages/e2e/src/harness/cfw/__tests__/spike.integ.test.ts› "cfw T0 spike › items 3+4 (recorded, not gating): engine state across dispose + recreate, and the re-init trigger". - Signature:
control: expected { status: 'complete', stepOutputs: 1 } to deeply equal { status: 'running', stepOutputs: 1 }. - Where: job 16942478435, on !299's pipeline 2914044694.
- Cause: the cell created its four instances one after another, each with a 2 s sleep. On a loaded runner the time from
controlpassing step a to the end of the dispose exceeded 2 s, so control's sleep finished before the dispose and the engine correctly reportedcomplete. The premise (every instance still asleep at the dispose) was never checked.
FU-FLAKE-CF-VITEST-ONTASKUPDATE-RPC-TIMEOUT: store-cloudflare test:cloudflare exits 1 on a vitest worker RPC timeout with every test green — fixed in code (pending landing; see FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT)
Owner: cf-we-do-streams (inherited from CI-health-3).
- Where: !292 pipeline 2910411150, arm64
test:cloudflarejob 16919213090; retried as job 16919266809. - Signature: store-cloudflare 283 passed, 1 skipped, yet vitest exits 1 on the unhandled
Error: [vitest-worker]: Timeout calling "onTaskUpdate". It hit right afterstate-store-contract.cf.test.ts, a 254-test file that ran 67.7 s, while 4 MR pipelines shared the runner. - Lead: vitest 2.1.9's worker → main birpc call (
onTaskUpdate) has a fixed timeout. A long single file on a starved host misses it. - Fix direction: measure, then either split the contract file or fix the starvation. Don't raise the timeout.
- Unrelated to !292's diff: store-cloudflare is untouched.
- Seen again (2026-10-09): !325 pipeline 2931531317,
test:cloudflarejob 17067827202 (load 24-34): store-cloudflared1-guarded-commit-exhaustion.cf.test.tsfailed on the sameTimeout calling "onTaskUpdate". The same job's e2e module-fetch timeout is FU-FLAKE-E2E-STRICT-STRUCTURED-D1-JS-MODULE-FETCH-RPC, the same family. - Seen again (2026-10-09): !331 pipeline 2931779326,
test:cloudflarejob 17069728534: store-cloudflare exited 1 onError: [vitest-worker]: Timeout calling "resolveSnapshotPath"with every test green. It is the same family through a different RPC method. Retried under §A3; disclosed in !331's description. - Seen again (2026-10-10): !332 pipeline 2932439027, arm64
test:cloudflarejob 17074660683, 00:09:49Z: store-cloudflare 309 passed, and the job exited 1 on the sameTimeout calling "onTaskUpdate". Also on the host at the time: WA's implementer subagents. (Corrected in owner batch 17: the job ran 00:05:20-00:14:46Z; the conformance owner's forced-delay proof had ended at about 23:55Z, and its next run started at 00:17:51Z, so neither overlapped.) !332 does not touch store-cloudflare. - Seen again (2026-10-19): !450 (MA-40) pipeline 2965117274,
test:cloudflarejob 17299104049: store-cloudflare 401 passed, 1 skipped, and the job exited 1 on the sameTimeout calling "onTaskUpdate", right after the guarded-commit outcome contract cases. MA-40 touches store-cloudflare (its V26 migration) but no test there failed. - Seen again (2026-11-18, the same family): !929's pipelines 3085375830 and 3085692378,
test:cloudflarejobs 18154337356 and 18156791062: a vitest worker RPC timeout in store-cloudflare after the long contract file, this time at a file's load; recorded under FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT. - Fixed by (pending landing): the same change as FU-CF-D1-SETTLE-FENCE-LOAD-TIMEOUT, which has the cause and the witness counts: vitest's worker RPC times a call out after 10 min inside workerd (60 s elsewhere). The cause was not a slow call or a long file: workerd's clock stands still while a test only awaits D1, so the 60 s timer fired early. That is why the fix direction above ("don't raise the timeout") no longer applies; vitest 4.1 has no worker RPC timeout at all.
FU-STALE-ENTRY-RUNS-ON-COMPLETED-SESSION: a stale resume / retry entry starts a second run after the winner completed (CP-96) — done (!295)
Resolution (!295). Fixed in the store, on every runtime and store: the run-start commit carries two required StartRun fields, expectCurrentRunStatus (the run CAS covers the id AND the status) and entryStatuses (a session gate against the status the store reads inside the commit). A run-status mismatch is the C3 loser (AgentAlreadyRunningError); a session-status mismatch is RunStartRejectedError('entry_status_changed'). The rule and the per-runtime sources are in execution-flow.md §Entry gates; the design is 2026-10-04-cp96-entry-status-gate-design.md. Temporal, DBOS and CF Workflows were probed on the real engines: the interleave was already refused by the executor's owner check or the run-id CAS (green before and after); the new gates are proven there by between-plan-and-commit tests. The text below is the original report.
Catalogue: CP-96 (reserved; owner batch 10). It first surfaced as the e2e redis-cas-races.integ.test.ts › "Concurrent Resume Races › should allow exactly 1 executor to succeed when 5 race with 10ms stagger" (expected 2 to be 1, !290 job 16919048157), but it is not Redis-specific.
Mechanism. The status gate of a resume() / retry() entry is not atomic with its run-start commit, and the winner writes its active entry state only AFTER its commit (R29):
- Entry A's run-start commit makes run R1 current (
running). The session still readsinterrupted/faileduntil A's post-commit entry-state write. - In that window, entry B reads
seenRunId = R1and passes its status gate on the stale status (runtime-js:resumeSession's friendly checks; theassertNoLiveOwnercheck runs only for arunningsession). - A's run completes, so R1 is idle.
- B's run-start commit expects R1. The CAS matches and the owner guard passes (R1 is not live), so B wins and runs on a session A already finished.
Observed. runtime-js + InMemoryStateStore, deterministic: a gate on appendMessages({ startRun }) holds A after its commit until B reaches its commit, then holds B until A completes. The repro and logs are in the reliability dossier, handoffs/ci-health-3-data/cp96-repro/.
resume(continue): runs [interrupted, completed (A), completed (B)]. B appends a second finish turn after the terminal one, and the output is overwritten (a→b). A real agent would run its tools twice.resume(with_message): the same; B's message and turn land on the completed session.retry: worst. B's retry rewinds A's successful turn: the transcript keeps only B's[user, assistant, tool], so A's completed result is LOST.execute: benign. B is a later new turn, whichexecuteallows on a completed session anyway.
Scope.
- runtime-js, every store: confirmed live.
- Temporal (
activities.tsplan gate ~:1084 / :1135; entry state written after the commit at ~:1278), DBOS (run-commit.ts:650 / :706; ~:862) and CF Workflows (steps.ts:948 / :1005; ~:1149): the same structure, plan-time gate plus post-commit entry state. A live probe is pending. - The CF Durable Object is safe: its claim block re-checks
allowedStatusesinside the sametransactionSyncas the commit and writes the entry state there (R31,durable-object-agent-base.ts:630-650,entry_status_changed).
Fix options.
- Runtime-level.
- Each entry takes the current run RECORD with its id. If that run is
running, the entry applies the live-owner check whatever the session status says (JS: the lease; Temporal / DBOS / CFW: their owner probes). - A live winner then refuses the stale entry. An entry that saw an older run already loses on C3's
expected_mismatch_idleno-retry rule. - Small, but it must be repeated on each runtime.
- Each entry takes the current run RECORD with its id. If that run is
- Store-level (recommended).
StartRuncarries the entry's accepted session statuses, and every store's run-start commit checks them atomically with the CAS:RunStartRejectedError('entry_status_changed'), which the DO already does.- Alternatively, the commit writes the entry's
activestatus itself, as the DO's R31 does, so the window closes. - One rule in core's
run-start-operationscontract covers every runtime. It touches memory, Redis, Postgres, D1 and the DO store.
The choice between them is the user's.
FU-D1-MIGRATIONS-CONCURRENT-ADD-COLUMN: concurrent D1 migrators fail with "duplicate column name" — done
Closed by: W1 MR2a (cf-w1-cfw-stores-submit; !334, merged as 137b07d57a). store-cloudflare's D1 migrations now run through core's claim-first runSqlMigrations (the version row is the batch's first statement; ADD/DROP COLUMN statements are guarded; a failed batch re-reads the version, so a lost claim is success and any other failure rethrows the original error). Proven by the workerd concurrency case d1-migrations-concurrent.cf.test.ts and d1-migrations-concurrent.integ.test.ts. The entry below is the pre-fix diagnosis.
Owner: cf-w1-cfw-stores-submit (W1 MR2; routed by the orchestrator to CF-D2, which handed it over). Surfaced by: the !304 audit (FU-PG-MIGRATIONS-TABLE-CREATE-RACE's D1 sibling), verified in source.
packages/store-cloudflare/src/d1-migrations.ts:applyMigrationAtomicbatches a version's DDL and its version row atomically, but two isolates that both read version N-1 both batch it. V2/V3 are plainALTER TABLE … ADD COLUMN, andaddColumnsIfMissingcheckspragma_table_infooutside the batch, so the second batch fails whole withduplicate column name.- Impact: transient. The losing isolate's
runMigrationthrows once; a retry finds the version recorded. No schema damage (the batch is atomic). - Fix direction: treat a duplicate-column failure whose version is now recorded as success (re-read the version), or guard the batch on the version row.
FU-MEMORY-CF-MIGRATIONS-NOT-ATOMIC: memory-cloudflare applies a migration's DDL and its version row separately — done
Closed by: W1 MR2a (cf-w1-cfw-stores-submit; !334, merged as 137b07d57a). memory-cloudflare migrations run through the same core runner: each migration is one batch with its version row, so the DDL and the version can no longer land apart. getMemoryMigrationVersion no longer masks a read error as version 0. The package gains a test:cloudflare workerd pool. The entry below is the pre-fix diagnosis.
Owner: cf-w1-cfw-stores-submit (W1 MR2; routed by the orchestrator to CF-D2, which handed it over). Surfaced by: the !304 audit, verified in source.
packages/memory-cloudflare/src/migrations.tsrunMemoryMigration: a version's DDL runs in one batch and its version row in a second statement, a plainINSERTwith noON CONFLICT. V2'sALTER TABLE … ADD COLUMNs are not idempotent.- Concurrent isolates: the second fails on the version primary key or on
duplicate column name. - Worse, a crash between the two writes: V2's columns exist but V2 is unrecorded, so every later
runMemoryMigrationfails onduplicate column name, permanently. - Fix direction: one atomic batch per version (DDL plus
INSERT … ON CONFLICT(version) DO NOTHING), asstore-cloudflare'sapplyMigrationAtomicdoes, and idempotent ADD COLUMNs.
CI health 2
CI health 2 removed the last genuine-failure retries: test:cloudflare's retry: when: script_failure, the config-level vitest retry in runtime-cloudflare / store-cloudflare / memory-cloudflare vitest.integ.config.ts, and 23 per-test { retry: 2 } options (22 in e2e *.cf.test.ts, 1 in a runtime-cloudflare .wf-noiso file). It fixed the flakes those retries hid. It also stopped every test job rebuilding the repo (see development-workflow.md, "CI test jobs").
The sweep (the evidence base). 164 pipelines updated 2026-09-26 → 2026-10-03 (139 MR, 25 main), every job attempt (include_retried=true); traces of all 120 failed attempts behind a red-then-green job and of 198 example jobs. After !272, test:cloudflare's first attempt was red in 36 of 78 pipelines (7 of 14 on main), every one turned green by its retry:
| Family | Signature | Post-!272 | Cause | Fix |
|---|---|---|---|---|
| Loopback keep-alive race | Network connection lost from WorkersTestRunner.updateStackedStorage | 19 of 36 | Miniflare 4.20251011.0's loopback server kept Node's keep-alive timeout (5s + 1s); on a starved runner Node closed a socket workerd still pooled | pnpm patch (upstream workers-sdk#14850) — FU-FLAKE-POOL-WORKERS-STACKED-STORAGE-LOST |
| StreamManager contract wall clock | expected [] to deeply equal [1,2,3], expected 4146 to be less than 2000, drain: timed out after 5000ms, 'pending' ≠ 'resolved', endsWithin | 7 | positive wall-clock windows on DO proxy reads | event awaits + poll disabled — FU-FLAKE-DO-STREAM-CONTRACT-TRIMMED-AFTER-ATTACH, FU-FLAKE-CF-DO-STREAM-WALLCLOCK-BOUND |
| Fixed sleep before interrupt | 'Filter Run 1 content…' not to contain 'Run 1'; Timeout waiting for stream … 'ended' after 5000ms | 4 (+5 before) | 150ms sleep, then a 5s status deadline | interrupt on the tool's tool_start — FU-FLAKE-CF-D1-GETCHUNKS-AFTER-RESUME, FU-FLAKE-CF-D1-DO-SEQUENCE-PRESERVE |
| Self-inflicted load | [vitest-worker]: Timeout calling "resolveId" / "onTaskUpdate", CFW forced-completion 120s timeout | 2 | each test job rebuilt every package (five next builds) beside its tests | test jobs run turbo --only — FU-CI-TEST-JOBS-REBUILD |
| Shared mock FIFO race (non-blocking child) | parent 'failed' ≠ 'completed' in persistent-subagent-d1-with-js | (e2e shard, !287 + !289) | parent took the child's scripted finish | per-agent scripts — FU-FLAKE-PERSISTENT-NONBLOCKING-SHARED-MOCK |
| CP-52 lifecycle-hooks parity (cf-do-d1) | 'suspended_client_tool' ≠ 'completed' | 3 (last 09-27) | product bug | fixed by S5 (!288) |
Other jobs: test:integ:e2e interrupt-resume-d1-with-js checkpoint restore (FU-FLAKE-INTERRUPT-RESUME-FIXED-SLEEP, fixed here); the remote-agents example late-join console error (FU-FLAKE-REMOTE-AGENTS-EXAMPLE-USAGE-FETCH: last seen 2026-09-30, then 38 green runs with no retry; not reproduced, left open); conformance history.companion-continue|temporal-postgres "failed for a different reason" (main 16853177130; routed to the conformance owner).
Hidden Playwright retries (retries: 1, 2 on CI in resumable-streams-nextjs) are out of this MR's scope; see the two entries below (owner: CI-health-3).
FU-FLAKE-PERSISTENT-NONBLOCKING-SHARED-MOCK: persistent-subagent-d1 non-blocking spawn ends failed — ✅ fixed (CI health 2)
persistent-subagent-d1-with-js.integ.test.ts › "non-blocking spawn persists ref with spawned status to D1" (expected 'failed' to be 'completed'; !287 pipeline 2905237226, !289 job 16916000765). The PARENT run failed: the non-blocking child runs concurrently with it, and on the shared global-FIFO MockLLMAdapter the parent's second model call could take the child's scripted finish({ result }), which its output schema rejects (invalid_type at summary; call order parent, parent, child). 4 of 25 runs failed under CPU load. The test now uses per-agentType RoutingMockLLMAdapter scripts, as the file's other concurrent cases already did; 25 of 25 green under the same load.
FU-E2E-DBOS-LAUNCH-RACE: concurrent first DBOS.launch() on a fresh database — ✅ fixed (CI health 2)
Reproducing FU-E2E-DBOS-LOAD-SENSITIVITY on a fresh Postgres under load, two forks' first launches ran DBOS's schema migration concurrently and failed (duplicate key value violates unique constraint "dbos_migrations_pkey", then deadlock detected exhausting the 8 transient retries); CI's integration shards start on a fresh Postgres too. createDBOSE2EContext now serializes launches with a Postgres advisory lock, taken by polling pg_try_advisory_lock: a blocked pg_advisory_lock call is an open statement, and DBOS's migration runs CREATE INDEX CONCURRENTLY, which waits for every open statement, so the two waited on each other (caught live: virtualxid vs advisory). 15 of 15 fresh-database runs green (under CPU load) after, against 3 of 4 failing before.
FU-CI-TEST-JOBS-REBUILD: every CI test job rebuilt the whole repo beside its tests — ✅ fixed (CI health 2)
turbo's test tasks depend on build, and turbo has no cache in CI, so pnpm run test:cloudflare (and test, test:integration, test:workflows, typecheck, lint) rebuilt every workspace package in each job, concurrently with the tests: about 40 tasks, five next builds and the docs among them. Tests that take ~10ms locally took 1–13s; vitest's own worker RPC timed out (60s) twice. Every test job now runs turbo with --only on the build job's artifacts, and the Next.js example jobs build only their own app. Evidence: the MR's before/after job-duration table.
FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT: opennext refresh tests pass only on their in-run retry — done (:600 is a test-budget defect; fixed)
Resolved (WE-2, 2026-10-16): :600 failed on its own fixed budget, never on a failed reload.
- Evidence, !378's job 17244353739 (the Playwright trace's network log and
stall-sampler.log):- both attempts failed with
Test timeout of 120000ms exceededwhilepage.reloadwaited forload; - in the retry, all 43 page loads answered 200, no request hung or failed, and every per-reload assertion held;
- the 50-reload loop ran at a median 2.03 s per cycle (the
/chat/<id>server render: median 1.13 s, p90 3.2 s, max 7.7 s), so about 41 reloads fit in what the two turns left of 120 s.
- both attempts failed with
- The sampler classifies it by its own rule: CPU pressure (
psi cpu-some55.2 s), no throttling, max sample gap 3.4 s,mem-full0. That is pod CPU contention (candidate (a) below), not a UserWorker hang. - The older sightings are the same mechanism: in !353's 17170359244 and !351's 17170348018, Playwright reports
Test timeout of 120000ms exceededfirst;page.reload: net::ERR_ABORTED; maybe frame was detached?is the in-flight reload torn down by that timeout (as suspected below). - The fix:
:600bounds each reload (page.reload({ timeout: 15_000 }); a stuck reload fails at once, naming its iteration and the earlier reloads' durations), sizes its timeout from its work (AGGRESSIVE_SETUP_BUDGET_MS+ 50 × (15 s + 2 s)) instead of the describe's 120 s, and logs every reload's duration with a median / p90 / max summary. The other refresh tests' earlier in-run retries (below) predate S5 and the later fixes recorded here.
Owner: orchestrator → CI-health-3. Measured: 16 of 48 test:examples:opennext-cloudflare-do jobs reported a flaky test: comprehensive-refresh.spec.ts:691 "rapid refresh during tool transitions" 5× (incl. main), :617 "very early refresh" 2× (incl. main), :600 "AGGRESSIVE refresh after follow-up completion" (flaky on !288 16898206128, red on !287 16913998804), mid-text-reload-scoring #1/#3/#4/#5/#10/#13, refresh-exact-messages A, api-consistency :300/:344 and stream-resume-consistency :273 (several on S5's own !288 before it merged). Needs a post-S5 re-measure at --retries=0.
Post-S5 re-measure (CI-health-3, 2026-10-07):
CI history, 21 jobs on the Intel runners since S5: 9 had a flaky test (passed only on the in-run retry) and 2 failed outright, so at
--retries=0about 11 of 21 (~52%) would have been red. Flaky:brief-flow.spec.ts:302(aggressive refresh spam) 3×,mid-text-reload-scoring#4 3×,comprehensive-refresh:4851×,mid-text-reload-scoring:4291×,refresh-exact-messagesA 1×. Hard failures:comprehensive-refresh:600×2 (!310 16979225616, !307 16980311879), bothpage.reload: net::ERR_ABORTEDinside the 50-reload loop and the 120 s test timeout (the reload is just what was in flight when the page closed;:600takes 23–52 s in some jobs and 1.1–1.8 min in others).No first attempt failed on content (no drift, duplication or snapshot mismatch): every one is a wait timeout or the mock's
the mock never held the text stepharness error.refresh-exact-messagesA's page snapshot (16980311879's artifacts) shows the send itself failed with workerd'sNetwork connection lost.Local repro (
wrangler dev --local, Playwright 3 workers,--retries=0): E1:600alone 5/5; E2 the four refresh specs 42/42, the whole suite 64/64 three times idle and twice with 14yesburners on 14 cores (load 68); E3 the same withenable_request_signal, 64/64 idle and loaded. (The 14 burners of the loaded runs were left running afterwards, 07:28Z to about 15:00Z on 2026-10-07, until the orchestrator killed them: they loaded this host, i.e. the shared local arm64 runner, through that window. Jobs on the DigitalOcean Intel runners, including every opennext job cited here, were unaffected.) 0 failures in every run, so the CI failures need the CI runner's conditions (5.8 min suites, k8s memory pressure): the flake reproduces only on CI, which points at the runner rather than the product. Runner pressure is FU-CI-K8S-QUOTA-SINGLE-PIPELINE-SATURATES's territory, but it is context, not a discriminator: pods loggedInsufficient memoryat scheduling in most opennext jobs, passing ones (9–92 lines) as well as one of the two failures (12; the other 0). Every run still logged 0–5 uncaughtUnable to enqueue/Network connection lost(FU-OPENNEXT-UNCAUGHT-ON-CLIENT-ABORT); the flag did not change the counts.Product side: the abandoned server-side stream reader on a client disconnect was CP-102, fixed by the client-disconnect MR. The uncaught
Unable to enqueueis OpenNext'scloudflare-nodewrapper (its platform body stream has nocancel), third-party. The MR's probe established that underwrangler devrequest.signalfires with the flag (3/3) and never without it, but not promptly after a quiet cut (local workerd's timing is per platform, CP-102; Cloudflare's edge reports it promptly: FU-CF-CLIENT-DISCONNECT-QUIET-RUN-EDGE).Main after !312 (job 17002761997, pipeline 2922619258, 50252a54, on the DigitalOcean Intel runners, so not loaded by the burners noted above; the orchestrator retried it as 17003631736): 1 failed and 5 flaky, an 11.7 min suite against about 5.8 normally:
comprehensive-refresh:600(red:page.reloadinside the 120 s test timeout),:455,client-tool-multiturn:26,brief-flow:231,mid-text-reload-scoring#7 and #13 (the mock never held the text step). Still no content drift, but not all waits: 3 runs FAILED, every one on the LLM call'sNetwork connection lost.(workerd), which ended as a non-retryableframework_internal_erroreven though workerd marked the errorretryable: true.client-tool-multiturn:26's snapshot shows its second turnfailedwith no reply, andbrief-flow:231waited out a BriefAgent run that had failed the same way. That is FU-LLM-WORKERD-CONNECTION-LOST-NOT-RETRIED, a product gap, not runner noise; the runner's load makes the drops more frequent.Recurrence on the new DOKS runner (2026-10-10): !335's pipeline 2933699958, job 17081887194 on runner
do-nyc3(the canary DigitalOcean nyc3 runner).comprehensive-refresh.spec.ts:600failed on both attempts (2.0 min and 2.1 min against the 120 s test timeout), each ending inpage.reload: net::ERR_ABORTED; maybe frame was detached?; 64 of 65 passed. The Playwright phase took 6.7 min (398 s, 15:38:54Z to 15:45:32Z), against about 193 s on the old runner (the orchestrator's figure), with 12 jobs started at once on freshly scaled nodes. On the old runner the spec was flaky 2 of the last 10, and its slow runs were the flaky ones (same source). CI's response is !314'sKUBERNETES_CPU_REQUEST: "4"on the three Playwright example jobs; that eases the runner side, but the test's timing assumption (50 reloads inside one 120 s test timeout) is still this FU's root cause.Recurrence on !337 (2026-10-10): pipeline 2933981207, job 17083529209:
comprehensive-refresh.spec.ts:552("refresh during follow-up") hit its 120 s test timeout on both attempts (page.waitForFunction); the §A3 retry 17084425970 passed. !337 changed nothing underexamples/or any product package.Hard failure on main too (2026-10-10): 137b07d57a, pipeline 2933694046, job 17081851116 on the old Intel runner:
:600over 120 s on both attempts. MR2a is CF Workflows only, and this example runs on the CF DO, so W1 ruled it out by code.:600's duration on the same old runner pool, by main commit (the orchestrator's figures): a1c6655f56 28.1 s (suite 2.2 min); a07e5aa1fb (!332, CP-102) 32.3 s (3.5 min); 7dc07aa1ec (owner batch 17, e2e and docs only) 49.8 s (4.8 min); 137b07d57a over 120 s on both attempts (8.2 min).- !332 as a suspect, from the code (cf-we-do-streams-2): the 50 reloads run after both turns completed (the test waits for
status === 'ended'), so each reload's snapshot is terminal and the client opens no/sseresume (the client resume rule). None of !332's per-disconnect DO-path work runs in the loop: no DO/ssereader is opened, so none is aborted or cancelled. ItshandleChatStreamadditions on a non-streaming request are a once-per-call diagnostic check (a Zod parse ofCloudflare.compatibilityFlags) and signal threading on paths a completed session does not take. - !332's one change that touches every request in that loop is the example's new
enable_request_signalcompatibility flag: each reload aborts its in-flight page, RSC, asset and snapshot requests, and with the flag workerd reports those aborts to the Worker (OpenNext's handler included). Whether that costs time is not knowable from the code. - The step attributable to !332 (+4.2 s) is smaller than the next step (+17.5 s at 7dc07aa1ec, which changes no product code), so runner load dominates the growth. The flag is still worth a measurement:
:600alone at--retries=0, a1c6655f56 against a07e5aa1fb, plus a07e5aa1fb with the flag removed, ×10 each, alternating, with CI's setup (the replay mock andwrangler dev --local); Fisher on failures, Mann-Whitney on duration.
- !332 as a suspect, from the code (cf-we-do-streams-2): the 50 reloads run after both turns completed (the test waits for
Recurrence on !340 (2026-10-11): pipeline 2936215040, job 17095764736 on
do-nyc3(917 s):comprehensive-refresh.spec.ts:600timed out on both attempts (120 s across its 50 reloads,page.reload: net::ERR_ABORTED), andmid-text-reload-scoring:307#5 was flaky (the mock never held the text step). The orchestrator §A3-retried it (17096040655) and reports:600at 65/65 locally on WA MR1's head and on WE MR B's branch, so this is DOKS timing again. It is now the most frequent CI blocker on the Durable Object path; the three-arm:600measurement above ran on 2026-10-11 (next bullet).Three-arm
:600measurement (cf-we-do-streams-2, 2026-10-11, the stress lock held, the Mac runner idle)::600alone,--retries=0, the replay mock (OPENAI_MOCK_STREAM_PACE_MS=8) and CI'swrangler dev --localvars, 10 rounds with the arm order rotated (ABC, BCA, CAB, …):Arm Failures Median IQR A a1c6655f56(before !332)0/10 10.89 s 10.63–11.12 s B a07e5aa1fb(!332, the flag on)0/10 10.64 s 10.57–11.10 s C a07e5aa1fbwithoutenable_request_signal0/10 10.66 s 10.60–11.12 s Fisher's exact test on failures gives p = 1 for every pair; Mann-Whitney U on duration gives A–B p = 0.52, B–C p = 0.97 and A–C p = 0.65. No run logged
Unable to enqueueorNetwork connection lost. So neither the flag nor the rest of !332 changes:600. Two departures from the plan: the mock andwrangler devwere restarted for every run with a fresh.wranglerstate (like CI's fresh pod) instead of three servers kept warm, and load averages were not logged. The box was idle, so this says nothing about a loaded or CPU-limited one.- What it rules out for the fix: locally the test takes about 11 s; in CI it runs past 120 s and logs
Network connection lost/Unable to enqueue. Something specific to CI is over 10× slower and erroring, and load alone does not explain both, so raising the 120 s budget is not the fix (at most a disclosed stopgap if the cause turns out to be the environment). - Next (cf-we-do-streams-2, its own slot after WE MR B; the orchestrator's assignment): reproduce CI's regime locally (the CI Playwright image under the pod's CPU and memory limits, CI's worker count, plus a loaded variant), find what makes each reload slow and what drops the connection (
wrangler devunder contention, the replay mock, a DO/ssecap), and bring the mechanism to the orchestrator before any fix.
- What it rules out for the fix: locally the test takes about 11 s; in CI it runs past 120 s and logs
Recurrence, and the UserWorker stalls (cf-we-do-streams-2, 2026-10-14):
:600failed on both attempts in !353's train pipeline 2946718056 (job 17170359244,do-nyc3), and its first attempt failed in !351's pipeline 2946716363 (job 17170348018,do-nyc3; it passed on the in-run retry). Neither MR changes the example or the DO path.ERR_ABORTEDis a consequence, not the failure. Both attempts end on the 120 s test timeout. Playwright then closes the page, so the in-flightpage.reloadaborts; retry #1 ended inwaitForTimeoutfor the same reason.- The loop isn't uniformly slow; the UserWorker stalls. In 17170359244 the reloads ran about one every 1-2 s from 13:53:58Z to 13:55:00Z. Then all
wrangler devoutput stopped for 28.2 s (to 13:55:28Z), then 15.0 s and 12.0 s more. At the end of each gap every request in flight completed at once, including unrelated ones:- a reasoning snapshot (29308 ms);
- the aggressive session's
GET /chat/…(28317 ms); GET /(9076 ms) and static chunks.
- An LLM call in that window logged
durationMs: 33836against the replay mock. - The stalls end with the proxy's retries. In both jobs, nearly every stall over 8 s in the test phase ends within 3 s of the ProxyWorker's
recovered on attempt 2 after a dropped connection to the UserWorker. That's 9 at once after the 28 s stall; 17170359244 logged 28 drops in all and 17170348018 logged 6, with stalls of 22.6 s and 25.1 s. - Nothing else is logged in the stalls: no
Reloading local server, no exit and no OOM line. The uncaughtNetwork connection lost/Unable to enqueuecluster around them (about 20 per job). - Locally none of this happens. The same suite on WE-2's 2026-10-14 DoD runs:
:600in 10.4 s and 10.5 s, with 0 proxy drops. - So: the UserWorker (workerd) stops answering for 12-28 s at a time on the DOKS runner, and
:600, the test with the most page loads, accumulates the most stall time inside one 120 s budget. - Candidates:
- (a) pod CPU starvation or a node stall (
do-nyc3also wedged a build job at 12:04Z the same day); - (b) memory pressure or GC near the 8Gi limit;
- (c) a UserWorker restart that wrangler doesn't log.
- (a) pod CPU starvation or a node stall (
- Measurement: the CI job now runs
scripts/ci/opennext-stall-sampler.shevery 2 s. It records the pod cgroup's CPU throttling,memory.current/memory.events, CPU / memory / IO pressure, the load average and the workerd PIDs with their CPU and RSS. It uploadsstall-sampler.logas an artifact and prints astall-sampler:summary (scripts/ci/opennext-stall-summary.mjs).- A sample gap well over 2 s means the whole pod froze (a);
- throttling or pressure in the stall intervals means (a) or (b);
- a new workerd PID means (c).
- Remove both scripts and their
.gitlab-ci.ymllines once the mechanism is known. The local CI-regime reproduction waits until then.
A failed follow-up SEND, not lost content (WE-2, 2026-10-19).
:610failed on both attempts on !430's train (pipeline 2961196222, job 17270710506) and on !377 (job 17250891625) withfinal.assistantMessageslength 1 instead of 2, which reads like content loss but is not. The retry attempt's trace (trace.zip, the test's own session) shows the follow-upPOST /api/chat/<sid>answered HTTP 500 with workerd'sNetwork connection lostfrom Miniflare's entry worker, so turn 2 was never committed. Every later/snapshotheld the same 2 messages, and the page showed the error until the reload loop cleared it. The first attempt was the usual reload timeout. So "no first attempt failed on content" still holds. The spec now asserts the follow-up send answered 200 and that the snapshot reached 4 messages andendedbefore its reload loop, failing there with the status and body instead of at the final count.
Sub-project pipeline
| Sub-project | Status | Description |
|---|---|---|
| D | done | Test infrastructure overhaul (shared harness + capability sets) |
| A.1 | done | Stateless purity cleanup (re-port to core helpers) |
| A.2 | done | Temporal HITL rewrite (workflow exits on HITL boundary) |
| A.3 | done | DBOS resume() contract bug fixes (F7) — closed 7d9498b8e |
| B (B3 cleanup) | done | FU-A2-01 + FU-A2-03 closed; F6 → future-work |
| dropped | Originally listed as "generic transport adapter"; the transport surface (RemoteAgentTransport interface + HttpRemoteAgentTransport + DOStubTransport) is already transport-agnostic with no surfaced demand for additional implementations. C had no spec, plan, or concrete deliverables — placeholder only. New transports (WebSocket, gRPC, etc.) get fresh specs when concrete demand surfaces. | |
| E | pending | Doc reconciliation |
Done items
(closed items are moved here with their closing-PR reference)
FU-FLAKE-OPENNEXT-API-CONSISTENCY-SNAP1-BEFORE-COMMIT: "snapshot tool count should not grow with page refreshes" took its baseline before the tool step committed — fixed (test only)
Sighting: !525's merge-train pipeline 2987495611 (2026-10-26), job 17454047713. api-consistency.spec.ts:193 failed on both attempts with Tool "get_current_time" count changed between snap1 and snap2 (expected 0, received 1). It dropped the train car. It was the job's first script_failure among the project's 100 most recent failed jobs; the others were all runner_system_failure.
Cause (test only):
- The test took snap1 a fixed 2 s after the first assistant element appeared.
- A snapshot holds only committed steps (
buildSnapshot,[0, checkpoint.messageCount)). On a loaded runner the tool step had not committed by then, so snap1 had 0 calls. - After the reload, snap2 had the run's one real call. The product held its invariant: the count never grew past the run's own calls.
Fix:
- snap1 now waits for the run to end (
waitForStreamComplete: the snapshot'sstatus === 'ended', bounded at 120 s, with a fast failure on an empty reply). - The no-growth assertions are unchanged.
- The file's other comparisons already compare finished runs, or check mid-stream snapshots only against upper bounds.
Proof (WE-2, slot a, 2026-10-25):
- Setup: the replay mock was paced at 400 ms per SSE event, so the 12-event tool-call step committed seconds after the first assistant element. Wrangler and the mock ran on private ports. Each run was the single test with
--retries=0. - Main's spec: red. The snapshots held
{},{}, then{ get_current_time: 1 }, and the test failed withcount changed between snap2 and snap3(expected 0, received 1). This is the same false growth from a baseline taken before the commit. At this pace the commit landed after snap2, where in CI it landed after snap1. - The fix, same pace: green in 17.6 s. All three snapshots held
{ get_current_time: 1 }. - The fix, CI's 8 ms pace: green in 5.9 s, with the same counts.
FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES: companion__waitForResult answered a timeout in several shapes, and CF Workflows ignored timeout — done
Closed by: this MR (branch reliability-cf-wa-wait-timeout-one-shape). Spec:2026-10-15-companion-wait-timeout-one-shape-design.md. Sighting: the !357 train flake at core companion-tool-dispatch.test.ts (a timing-dependent 'timeout' vs 'timeout (current: …)').
- What it was. Core had three deadline checks per poll (bare
timeoutat the loop top,timeout (current: X)in the interrupted branch and after the read) and an unclamped sleep, so the shape depended on timing; the DO and DBOS answeredtimeout (current: X); CF Workflows ignored the caller'stimeout, parked forsubAgentEventTimeoutand recorded a failed result whose content was{"error":"Unknown error"}. DBOS acceptedtimeout: 0; CF Workflows validated nothing. - The fix. Core
CompanionWaitResultSchema,waitForResultTimeout(name, currentStatus)andWaitForResultArgsSchemaare the one contract. Core and the DO decide the deadline once per poll, after the read, with a clamped sleep and injectednow/sleep; unconfirmed interrupted / orphan observations reach that decision (C1). CF Workflows parks formin(timeout, subAgentEventTimeout)and answers the typed timeout as a successful result; untimed it keepssubAgentEventTimeout(O1). DBOS uses the shared schema and builder. - C-1, a
waitForEventrejection never decides a child's outcome (option B, fix round 3). The CF Workflows parks read a slice timeout from the error MESSAGE, which the engine'sWorkflowTimeoutError("Execution timed out after Nms") does not match, so a slice timeout became the child's failure (a zombie). Every park now treats any rejection as "no event this slice" (the name is logged only); at the bound no park writes a terminal / delivered ref over a possibly live child. The ephemeral sub-agent park's bound (option iii, rounds 4-5) stops the child's run and records its true outcome (framework_subagent_wait_boundwhen the stop ended it), terminates an instance with no run, and an unconfirmed ref is reconciled later by the CFW readers. Every park has ONE persisted deadline and ends exactly when step-recorded time reaches it, never deaf, with a 1 s floor only after repeated fast rejections (an iteration shorter than the floor, consult included; a dropped event, a stale wake-up or a failed stop read counts too, and nothing resets the count within a park, rounds 14-15), a floor that doubles after each further 5 up to 30 s, never sleeping into the deadline before a final listen, and a read of the child's persisted state after every floor and before a timeout answer (rounds 6-9, review I-N1 / I-N2 / N-1 / N-2 / R7-1 / R7-3 / m1).FU-CFW-WAITFORRESULT-STALE-BUFFERED-EVENT(below) is folded in (round 9). - Folded in (fix round 1):
FU-CFW-WAITFORRESULT-NON-TIMEOUT-ANSWERS-AS-ERRORS(below) — done. - Related, still open:
FU-CFW-COMPANION-WAIT-SUSPENDED-CHILD-AS-FAILED(asuspendedcompanion is now a typedfailedanswer on CF Workflows),FU-CFW-WAITFORRESULT-CLOSED-CHILD-TIMEOUT,FU-COMPANION-WAIT-FAILED-MESSAGE.
FU-CFW-WAITFORRESULT-NON-TIMEOUT-ANSWERS-AS-ERRORS: CF Workflows' interrupted / parked failed / terminated waitForResult answers reached the model as tool errors — done
Closed by: this MR (fix round 1 of FU-WAIT-TIMEOUT-STATUS-TWO-SHAPES). CF Workflows recorded its pre-park interrupted answer and its record step's interrupted / non-complete outcomes as failed tool results, so the model read { "error": "<guidance or the child's error>" } ("Unknown error" for a terminated event with no error). Both now answer the typed CompanionWaitResultSchema object as a successful result, with core's and the DO's fields and wording: { name, status: 'interrupted', message } (the CP-68 resume guidance), { name, status: 'failed' | 'terminated' } (no error text, as elsewhere). Witnessed by workflow.test.ts (failed / terminated / interrupted / completed while parked) and cfw-persistent-companion.test.ts (pre-park interrupted).
FU-OPENNEXT-WRANGLER-DEV-STATIC-REVALIDATION-STALL: wrangler dev's local proxy can leave a reload's static-asset revalidations unanswered — open (dev tooling, not product)
Seen (WE-2, 2026-10-17, WE MR C1's lock window): one of C1's 5 opennext runs (mid-text-reload-scoring + comprehensive-refresh, --retries=0, 3 Playwright workers, CI's replay mock, wrangler dev --local) failed comprehensive-refresh "AGGRESSIVE refresh after follow-up completion" on its per-reload bound: R31: page.reload did not reach "load" within 15000 ms. The 30 reloads before it took 47–435 ms.
- Mechanism, from the wrangler log: reload 31's document was served (200, 175 ms), but only 2 of its 9 static-chunk revalidations (
/_next/static/*304s) were ever answered. A normal reload logs all 9, and other workers' sessions carried on, soloadnever fired. - Not C1: an ABA comparison at
--workers=2(main ×2 / C1 ×2 / main ×2, quiet host) was 6/6 green, with reload medians 33–38 ms on both sides and no stall. - Likely also CI's 12–28 s stalls, which ended with the proxy's "recovered on attempt 2 after a dropped connection to the UserWorker" (FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT, done).
Seen again in CI (2026-10-19), on unrelated MRs: comprehensive-refresh.spec.ts:610 failed with the same ProxyWorker signature on !446 and !449 (a reload's static revalidations hanging ~20 s before recovering; a follow-up POST "failed after 1 attempt: Network connection lost") and on !452 (job 17300790896, W1's CP-159, unrelated code). The "Unable to enqueue" / "Network connection lost" proxy lines also appear, at a similar rate, on main's passing runs. The failing runs logged more proxy recoveries (5 and 23) than main's passes (0 and 2), on a busier runner (load1 9.2 vs 6.2), so the incidence tracks runner load, not the code. Fresh pipelines passed the job on both MRs (!446 job 17300812806, !449 job 17300814835) with no code change.
Next: reproduce it under wrangler dev with the 3-worker load, and capture the proxy's connection state when a static revalidation hangs (a pinned Miniflare / wrangler version is the likely fix, or an upstream report). The test's per-reload bound (15 s) makes a recurrence fail fast and name the reload.
FU-CI-OPENNEXT-STALL-SAMPLER-REMOVE: remove the TEMPORARY opennext stall sampler once :600 holds — open (trigger-gated)
test:examples:opennext-cloudflare-do still runs scripts/ci/opennext-stall-sampler.sh and prints scripts/ci/opennext-stall-summary.mjs (both marked TEMPORARY in .gitlab-ci.yml), added to diagnose FU-FLAKE-OPENNEXT-REFRESH-PLAYWRIGHT. That is now explained (a test-budget defect, fixed in :600), but the sampler stays as diagnostics in case the fix does not hold (orchestrator, 2026-10-16).
Trigger: the budget fix has merged, and test:examples:opennext-cloudflare-do has passed on about 5 subsequent pipelines with no comprehensive-refresh.spec.ts:600 failure. Then: delete both scripts and their .gitlab-ci.yml lines (the sampler start, STALL_SAMPLER_PID, the summary call, and the stall-sampler.log artifact path) in one small MR. If :600 fails again first, read the job's stall-sampler: summary and its per-reload duration log before anything else.
FU-SUBAGENT-RESULT-RECORDING-RELOADS-PARENT: recording each sub-agent result re-read the parent's whole history — done
Closed by: this MR (CI-health-3, "sub-agent result recording reads a bounded tail"). Surfaced by: CI-health-3's profile of FU-FLAKE-CF-CONCURRENT-SUBAGENT-FOLD-TIMEOUT. Related: FU-CFW-PER-STEP-COST.
- Cost: for every completing child, CF Workflows read the parent's whole history twice:
recordSubSessionMessage→loadAgentState(state, every message, refs and the interrupt flag, of which it usedagentType,userIdandstepCount), and corerecordSubSessionResult(state, refs and every message again, to look for onetoolCallId). The D1 append commit reads no message payloads (only counts), so the profile's "third read" was the commit'sCOUNT(*)s. Temporal's record activity and its resume drain loaded the parent per child the same way. DBOS, runtime-js and the CF DO record results in memory with the step commit (no per-child re-read). - Fix (no store-interface or contract change):
- Core
recordSubSessionResultnever reads the whole history. Its idempotency check reads from the new optionalhistoryScanFromon (the parent's count when the issuing assistant message was committed), or, without it, pages backwards from the end and stops at the issuing assistant message (a result always follows it). The at-least-once retry still finds its own earlier result. A new optionalparentStatesaves the parentloadState. - CF Workflows:
recordSubSessionMessageandexecuteRemoteSubAgentCallread the state row only; the main wave passes its step-cachedcommittedCountashistoryScanFrom. The respawn, remote and unconfigured paths use the backward scan. - Temporal: the record activity loads the parent once (for the helper and usage); the
applyResultsAndReloaddrain passes the state it loaded for the whole drain.
- Core
- Measured (D1 calls,
concurrent-subagent-fold.integ.test.ts"sub-agent wave D1 cost", asserted): a 16-child wave with a 1-message history went from 404 to 324 calls. A 1 200-message history used to add 164 calls at 16 children and 68 at 4 (about 8 per child); it now adds 36 at both. The 36 are 9 full-history reads per turn (its LLM steps and the wave's fold step), which are per turn, not per child, and stay with FU-CFW-PER-STEP-COST.
FU-PG-MIGRATIONS-TABLE-CREATE-RACE: concurrent PostgresMigrations runs crashed at start on a fresh schema — done
Closed by: !304 ("store-postgres: concurrent migrators are safe").
- Where it failed: main pipeline 2914016750 (41cd3fbbc0),
test:integ:e2e 2/4, job 16942254650.cross-runtime-subagent-resume-matrix-suspendedS6 (js + postgres + redis) died in setup:duplicate key value violates unique constraint "pg_type_typname_nsp_index"atensureMigrationsTable. - Cause (store):
run()/getCurrentVersion()created the migrations table with a bareCREATE TABLE IF NOT EXISTS, outside any lock. Concurrently, both sessions pass the existence check and the loser fails on the catalog's unique index.- Behind it, overlapping migrators (only phases 1 and 3 were serialized):
- a
CREATE INDEX CONCURRENTLYwaits for every older transaction, so it deadlocked (40P01) against another migrator's phase-1 transaction waiting for the build's table lock (a no-opADD COLUMN IF NOT EXISTSstill takes ACCESS EXCLUSIVE), and against a second migrator's build of the same index. Under host load, 8 migrators re-collided on every attempt; - the INVALID-index cleanup dropped a peer's live build (a build in progress also reads
indisvalid = false), and could drop a peer's rebuild under the same name from a stale scan; - a migrator whose
IF NOT EXISTSno-oped on a peer's unfinished build recorded the version anyway.
- a
- A
tablePrefixover 34 characters got truncated identifiers; from 39 on, two index names collided.
- Cause (e2e):
createPostgresTestContext()named its prefixe2e_${testId ?? Date.now()}_. The sixcross-runtime-subagent-resume-matrix-*files build a postgres combo with notestIdin parallel workers, so two that built in the same millisecond shared tables, and either one'scleanup()dropped the other's. - Fix (user decision, option B):
- Migration lease: one row in
<prefix>migration_lease, taken with an atomic upsert, renewed everyleaseTtlMs / 3, released at the end, taken over after a crashed holder's ttl. Phases 1 and 3 check it with a renew-and-check as each phase transaction's last statement (clock_timestamp(), so a long phase never blocks its own renewal;MigrationLeaseLostError, retried). One migrator at a time: neither deadlock can form. - The migrations and lease tables are created under
pg_advisory_xact_lock. peerWaitMs(default 600 s): one run's total wait for the lease and for a build still running on its tables (found viapg_stat_progress_create_index, scoped throughpg_locks, so another role's unrelated build never stalls it). Out of budget:MigrationLeaseTimeoutError/MigrationIndexNotReadyError; the timed-out attempt records nothing.- Cleanup: each INVALID candidate is retired under
SHARE UPDATE EXCLUSIVEon its table (no live build of any role lets it be taken;lock_timeoutmeans busy, skip), re-checked by OID and renamed toidx_<prefix>dead_<oid>; only that name is dropped. - Phase 3 records a version only when every concurrent index it builds is valid (
MigrationIndexNotReadyError). run()re-enters on 40P01,MigrationIndexNotReadyErrororMigrationLeaseLostError(5 attempts, jittered backoff, warnpostgres.migration.retry);appliedaccumulates over attempts. PostgreSQL < 12 is refused.MAX_TABLE_PREFIX_LENGTH = 34, enforced by every store (breaking for longer prefixes; upgrade note in the changeset). The repo's DBOS e2e prefixes andtii_h6_…were shortened.- e2e: the prefix gets a random suffix;
cleanup()lists also dropabandoned_run_startsandmigration_lease.
- Migration lease: one row in
- Red-first (each fails on the code before its fix):
migrations-concurrent-start.integ.test.ts: concurrentgetCurrentVersion()andrun()(the CI signature on main; then 40P01 exhaustion under load until the lease; the test now asserts zero retries); another role's unrelated build (waited out the whole budget); a stalled holder's stale scan (dropped its successor's index: OID changed); the 34 / 35-character prefixes (index shapes vs a short prefix; the old 58-character test passed vacuously on colliding names).- New behaviour: the lease contract (crashed-holder takeover, a live holder never taken, renewal past the ttl, release), an orphan build waited for, and a waiting migrator that completes.
- Regression guard (passes on main too): a cancelled build replaced with a valid index.
postgres-migrations.test.ts: the peer-wait budget spans the whole run (52 s for a 10 s budget when it was per attempt); the prefix bound and a guard that fails any future migration identifier over it.postgres-test-context-isolation.integ.test.ts: same-millisecond contexts sharede2e_1700000000000_, andcleanup()leftabandoned_run_starts.
FU-PG-MIGRATIONS-SESSION-LOCK: serialize a whole PostgresMigrations.run() on one pinned connection — superseded
Closed by: !304. The migration lease serializes a whole run without a pinned connection (a session-scoped pg_advisory_lock needed a pin-a-connection method on PostgresAdapter, an interface change for every custom adapter). Nothing is left to do.
FU-CI-NATIVE-BUILD-CPU-FEATURES-HANG: a CI job hung 90 min inside pnpm install on a native build — done
Closed by: !296.
- Where it failed: main pipeline 2911492581 (7bca6187bd),
test:integ:e2e 2/4, job 16925087186. It hitserver_timeout_runningafter 5400 s. - Signature: the trace ends inside
pnpm installatcpu-features install: node-gyp rebuild … gyp http GET https://nodejs.org/download/release/v24.21.0/node-v24.21.0-headers.tar.gz. Another job'sgyp ERR! not found: makeis the same family. - Cause:
pnpm-workspace.yamlallowBuildsletcpu-featuresandssh2run their install scripts (!252 kept npm's set).- So every CI install compiled them with
node-gyp. With a fresh~/.cache/node-gypper job, that downloads the Node headers from nodejs.org with no timeout of its own. - That GET stalled on the amd64 runner.
- Fix:
- Both are now
falseinallowBuilds. They are optional native accelerators reached only throughworkspace-docker→dockerode→docker-modem's SSH transport, andssh2falls back to pure-JS crypto. - The root
node-gypdevDependency, which served only those builds, is removed. - CI's
.pnpm_installrunstimeout 900 pnpm install --frozen-lockfile.
- Both are now
FU-FLAKE-APPROVAL-GATES-PLAYWRIGHT: approval-gates Playwright tests passed only on their in-run retry — done
Closed by: CI-health-3's second MR.
- Measured: 18 of 50
test:examples:approval-gatesjobs (2026-09-29 → 10-03) had a flaky test. - Signature:
send-button … element is not enabledright afterinput.fill(…). - Cause: the tests typed before React hydrated the page under
next dev. The controlled input's state stayed empty, so Send stayed disabled. - Fix:
- The page exposes a
data-hydratedapp-ready marker (set in auseEffect), and every spec waits for it before typing. tests/hydration-guard.spec.tsmakes hydration deterministically late (JS chunks held back,gotoatdomcontentloaded). Without the wait it fails with the CI signature 2/2; with it, it passes.retriesis now 0.
- The page exposes a
- Proof: the suite passed 15/15 at
--retries=0, at a recorded host load average of 23–121 (from concurrent activity; the light example rebuild run beside it added almost none).
FU-B2-05: CF Workflows cross-turn proofs limited by CP-67 / DI-18 — done
Closed by: CF-D1 MR 1c (!291). B2's CF Workflows history, from_checkpoint and retry cases now run across turns on the real engine: runtime-cloudflare/src/__tests__/history-b2.wf-noiso.test.ts, describe('FU-B2-05 cross-turn') (C1x: a second execute() sees turn 1's user message and tool activity; C3x: retry() of a failed turn 2 re-runs turn 2's trigger once with turn 1 kept; C7x: from_checkpoint to the turn-2 input checkpoint rewinds within turn 2 only). They cover existing behaviour, so they pass on main as well; 5× green on head.
Update (CF-D1 MR 1a): a second execute() turn now runs in its own instance and reactivates the ended stream (CP-88), and handle.resume() reactivates it too (CP-67); the real-engine suite runtime-cloudflare/src/__tests__/cfw-terminal-truth.wf-noiso.test.ts proves cross-turn runs. B2's history / from_checkpoint / retry cases can now be re-run as cross-turn tests.
Update (CF-D1 MR 1b, !287): DI-18 is fixed: handle.result() waits for a real outcome (no 5-minute cap, waits through an operator pause) and never caches a rejected poll, so it no longer limits these assertions. The B2 cross-turn re-run itself was not part of !287; owner: CF-D1 MR 1c.
Originally: CFW never reactivated an ended root stream for a plain follow-up turn (CP-67), so a second turn hung on real workerd. B2's CFW history, from_checkpoint and retry proofs are therefore multi-step single-instance tests, not cross-turn ones. DI-18 (handle.result() after retry()/resume() can memoize a bogus {failed, 'Unknown error'} under local-dev Workflows) also limits what the workerd tests can assert. Re-run the CFW cross-turn cases when CP-67 is fixed.
FU-CFW-SESSION-ID-REUSE-AFTER-DELETE: a reused session id collides with its old instances — done
Status: done. Closed by: CF-D1 MR 1c (!291) for the sub-agent half; MR 1b (!287) closed the root half. Surfaced by: CF-D1 MR 1a DoD audit (D4).
Resolution (CF-D1 MR 1c, !291, design spec C7). Every CF Workflows child instance id is now keyed on the parent run (respawnSuffix(runId)): a sub-agent spawn is …__spawn__<run suffix>, a companion spawn …__spawn__<run suffix>-<callId> (the call id too, because a persistent child's session id repeats within one run when a failed child is re-spawned), a companion continuation / resume …__continue__<run suffix>-<step>-<callId> / …__resume__<run suffix>-<step>-<callId>; respawns already were. A re-executed child-create step whose create() already landed adopts its own instance (already_exists, one shared createChildInstance helper for all five sites). Proof: the real-engine case cfw.child-id-unique-after-session-reuse (C7) in runtime-cloudflare/src/__tests__/cfw-stop-conditions.wf-noiso.test.ts (execute → deleteSession → execute with a repeated tool-call id; red on main: the second run hung on the old child id), plus workflow.test.ts "child create is idempotent across a step re-execution". The child SESSION survived the parent's delete until deleteSession cascaded (RM-64, FU-DELETE-SESSION-ORPHANS-CHILDREN).
- Done (snapshot MR + CF-D1 MR 1b). Root turn / resume / retry instance ids are keyed on a random key minted by the call (
__turn__<key>), never on a counter that restarts with the session, so a reused ROOT session id always builds fresh ids. CF-D1 MR 1b pins it on the real engine:cfw-terminal-truth.wf-noiso.test.ts"FU-CFW-SESSION-ID-REUSE-AFTER-DELETE" runs execute →deleteSession→ execute on the same id three times, each a fresh completed session. The planned per-session-lifetime nonce is therefore not needed for root ids. - Was open (closed by MR 1c). A sub-agent's base instance id is
agent__<type>__<childSessionId>, and the child session id is derived from the parent session id and the tool call id. If a reused parent session's model emits a tool call id it emitted before the delete, the child's base id is already taken. Real providers mint unique tool call ids, so this needs a deterministic model (a scripted test model, or a provider that numbers calls per conversation). Fix: key child instance ids on the parent run's key (as respawns already are), or on a nonce stored on the session row at creation.
FU-FLAKE-CP52-ALARM-SUSPEND-HOOK-WINDOW: runtime-cloudflare CP-52 run-guard tests raced a timed onAgentSuspended window — done
Closed by: !292 (CI-health-3 EVICT-02).
- Where it failed: main went red after !291 merged (pipeline 2911027602,
test:integ:restjob 16922542276):do-run-guards.integ.test.ts› "CP-52 durable wake: the execution loses its pending wake (eviction) — the alarm, not a request, completes the run",waitFor(suspension-settled) timed out, last statecompleted. - Not an !291 regression: an A/B of 28427e5281 against 5bb1eaaf2b passed 5/5 each.
- Not a product bug: the run completed correctly, with 2 runs and the tool result delivered once.
- Cause: a wall-clock test window.
/test/suspend-hook-delay {ms: 500}heldonAgentSuspendedfor 500 ms.- Inside it, the test had to poll, then submit, then evict.
- On a loaded runner the window expired first, so the still-registered pending wake continued the run and the intermediate
suspended_client_toolstate the test waited for never appeared. - A 700 ms pause between submit and evict reproduces the CI signature.
- Fix (test-only):
- The helper worker holds
onAgentSuspendedon a gate (/test/hold-suspend-hook,/test/release-suspend-hook) that the test releases after its submit / evict / wake-hold. - All four timed-window tests in the file use it: the alarm test, the CP-52 race, the submit-during-hook test and the R42 partial batch. Each could otherwise fail, or quietly degrade into its own control case.
- The helper worker holds
FU-FLAKE-JS-LEASE-BLOCKED-RENEWAL: runtime-js lease "condition 1" took the session over under load — done
Closed by: !292 (CI-health-3 EVICT-02).
- Where it failed:
runtime-js/src/__tests__/lease-takeover-fencing.test.ts› D2 "condition 1: a model call blocked for > 3 lease periods keeps the run live" failed in !290's arm64test:unit(job 16919048155):expected { …(12) } to be an instance of AgentAlreadyRunningError. - Cause (test design, not a product bug):
- The test ran a real 150 ms lease with real-time renewals (every 50 ms) across a 600 ms sleep and the intruder's whole entry path.
- An event-loop stall longer than
ttl − ttl/3(~100 ms) on a loaded host lapses the lease on the store clock, so the takeover is legitimately accepted. That is the documented JS lease contract. - A 2-TTL busy-wait before the intruder reproduces the exact CI assertion.
- Fix:
Date(the store's lease clock and the lease's local clock) is frozen.- It advances by
ttl/3only after a renewal is observed landing (a spy onrenewRunLease), across 12 periods (4 TTLs). - The same injected stall now passes.
FU-EVICT-06: injectable poll cadence for companion waits — done (test seam + tests)
Closed by: WA MR2 Task 14 (the stop-as-one-signal MR). Sighting that made it urgent: core companion-tool-dispatch.test.ts › "flapping liveness (not-live, not-live, live) RESETS the confirmation counter" got 'timeout' instead of 'completed' after 39.5 s of real time on main pipeline 2937097066, job 17100610293 (the 2-CPU edge runner): the test flipped the child's ref on a 4.2 s wall-clock timer against the real 1 s poll, with a 20 s wait.
- What it was.
WAIT_FOR_RESULT_POLL_MS(core, 1 s) and the DO companion poll (250 ms) were constants, so the orphan-detection, CP-68 debounce and failure-window tests waited on the real clock: 2-5 s each (corecompanion-tool-dispatch.test.tstook 32 s), with wall-clock timers racing the polls. - The seam (defaults unchanged). Core
CompanionToolDispatchDeps.waitForResultPollMs(default 1 s); the DO's internalDOCompanionDispatchContext.childPollIntervalMs(default 250 ms, never set by DOAB), used bycompanion__waitForResultand the blocking-spawn park. - The tests. Core: the orphan-detection group and the CP-68 / CP-103 waits run on a 1 ms cadence, and the cases that change state mid-wait (the flapping probe, the concurrent-resume debounce) are driven by the wait's own polls (its per-poll ref read), never a timer. The flapping case now has teeth: its old script (
false, false, true, thentrue) never reached 3 not-live observations even with a counter that does not reset (the probe is consulted twice per poll); the new one (false, false, true, false, falseby poll) compensates the child at the 4th poll under a non-resetting counter (checked by mutation). The file runs in under 1 s (was 32 s). DOdo-persistent-agents.test.ts: the failure-window cases use a 20 ms window over 1 ms polls (every poll fails, so the window expires on any host), and the poll cases use 1 ms polls: 0.4 s of tests (was 3.3 s). - Not done: backoff for the cross-DO 250 ms poll (a production cadence change, not a test seam).
FU-EVICT-02: a wake-recovered DO child never ends its stream (the "wake race" e2e) — done
Closed by: !292 (CI-health-3 EVICT-02). Catalogue: MA-22 (reserved; catalogued in owner batch 10), open with fix evidence until a cf-do cell flips it.
What it was. The e2e cross-service-do-do-recovery › "FU-EVICT-02 wake race" failed 4 of 5 runs on main after a heavy local run (expected 'running' to be 'completed', the consumer session) and passed when the machine was idle. The original hypothesis (a 409 on the ephemeral remote /start surfacing as a failed result) was not the cause. There were two problems:
- A product bug (MA-22).
- A DO child (a remote child, or a sub-agent in its own DO) has a
parentSessionId, so the JS run loop never ends, fails or pauses its stream: it assumes the stream is the parent's. DurableObjectAgentBaseended the child's own stream only after a COMPLETED run on/start,/resumeand/retry(the old "CC58" block). The wake auto-resume drive (ensureExecutionContinuesImpl) did nothing, and a run that endedfailedwas never failed on any drive.- So an evicted child recovered to
completed(status, output and every chunk written) with noendframe, and its parent's attached stream stayedrunninguntil the remote timeout (10 min). - Fix: one
finalizeOwnStreamAfterDriveafter every drive, wake included. It ends on completion, fails with the run'serrorDetailon failure and pauses on suspension, each fenced to the run (CP-85).
- A DO child (a remote child, or a sub-agent in its own DO) has a
- A hollow test.
- The wake-race and evict-single-step scripts put
slow_toolin the same step as__finish__. Since RM-29 C2 a call co-emitted with__finish__was never executed (until the 2026-11-03 finish-siblings change, which runs it first), so there was no held-open step, and the child finished in ~15 ms. - The lost-POST re-POST then landed on a COMPLETED child and started a second turn instead of the designed 409 → attach. That is CP-95,
/startnot idempotent: a contract decision for the cross-service bucket. - The eviction landed after everything had finished when the machine was idle (green), and inside that second turn when it was loaded (red).
- Both scripts now hold the step open with a test-released
gated_tool(producer worker/__test/release), and__finish__is its own step. The tests evict and release on observed chunks, never a timer. - The D7 lost-POST test also asserts exactly one
run_startedon the producer.
- The wake-race and evict-single-step scripts put
Evidence.
- runtime-cloudflare
do-eviction-recovery.integ.test.ts: 3 new tests (woken child completes →ended; woken child fails →failed; live child fails →failed) and anendedassertion on the companion child. 4 red on main, 14/14 green with the fix. - e2e wake race: red on main (the consumer
runningat 40 s, CF-D1c's exact signature), green with the fix.
FU-OPENNEXT-EXAMPLE-FLAKY-REFRESH-TESTS: opennext Playwright suite is green only through its in-run retry — done
Closed by: the S5 snapshot MR (branch reliability-example-refresh-flake, design spec docs/superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md).
Resolution (S5, plan Task 24): the snapshot is one checkpoint-anchored read and the example's session pages use the canonical client (snapshotResumeOptions(snapshot): seed without the in-flight message, resume from the cursor with replayPrefix; useHelixChat with resync; the count-unchanged refetch workaround deleted). All 7 test.fixme markers are removed and each test passed --repeat-each=15 --retries=0 (15/15 each, on the paced replay mock the suite now runs with, and again on the unpaced mock). The new mid-text-reload-scoring.spec.ts scores no-loss / no-duplication / cursor-and-id-match per reload. Red on origin/main (a118f8e17e, example built from base, unpaced, 6 workers × 50): 24 of 1450 runs lost the in-flight text permanently (rendered "" vs the final text). The retries: 1 in playwright.config.ts is unchanged (out of scope here).
Re-measured after the rebase onto 5bd30e20da (CF-D1 MR 1a): each of the 7 tests again passed --repeat-each=15 --retries=0 (15/15 each), as did the CP-52 and CP-74 cases (15 separate vitest run --retry=0 processes each). Stress (--workers 6 --repeat-each 50 --retries 0): comprehensive-refresh 650/650 (paced mock); mid-text-reload-scoring 1150/1150 (1000 mid-run reloads scored); the unpaced files of the base red run 1449/1450, the one failure a workerd Network connection lost on the first send POST (no session started; see FU-S5-WORKERD-FIRST-SEND-CONNECTION-LOST), zero content losses. The full opennext suite passed 64/64 with retries 0. Logs: tmp/s5-evidence/un-quarantine-*{,-rebased}.log and green-stress-6x50-branch-*.log (local, not committed).
Quarantine (2026-09-27, user decision): these 7 tests fail on the RM-08 torn snapshot often enough to block unrelated MRs. GitLab merges only on a green pipeline, so they are marked test.fixme. Each has a comment that names RM-08, this entry and the owner:
examples/opennext-cloudflare-do/tests/comprehensive-refresh.spec.ts"refresh mid-stream: resumes and completes correctly" (was:370)comprehensive-refresh.spec.ts"refresh during follow-up: turn 1 preserved, turn 2 completes, no cross-pollination" (was:472; see also FU-OPENNEXT-REFRESH-DURING-FOLLOWUP-LOSES-TURN2-TEXT)interleaved-content.spec.ts"preserves all text content on refresh mid-stream @slow" (was:223)interleaved-content.spec.ts"text content survives refresh after first tool appears @slow" (was:345)api-consistency.spec.ts"snapshot should have consistent tool count after stream completes" (was:161)interleaved-content.spec.ts"aggressive refresh during tool execution preserves content @slow" (was:287; reported as:293; added later on 2026-09-27 after it flaked on !279 and !276)comprehensive-refresh.spec.ts"AGGRESSIVE refresh loop: state converges to stable completion" (was:400; added later on 2026-09-27 after it flaked on !279 and !276)
Exit condition: S5's snapshot MR removes all 7 test.fixme markers. The same MR shows all 7 green at --repeat-each=15 --retries=0 or more. This entry stays open until that MR merges. The other tests named below are not quarantined.
Owner: S5 (reliability-example-refresh-flake). Every test listed below is a refresh-during- or refresh-after-stream consistency test, the same family as :472, whose root cause is RM-08 (torn snapshot); S5's checkpoint-anchored snapshot work covers them. playwright.config.ts sets retries: 1. Over the 36 opennext jobs in the 45 pipelines before !272, 23 had 1-3 tests that failed once and passed on the retry. Most frequent: comprehensive-refresh.spec.ts:472 (refresh during follow-up, 14 jobs; now its own entry, FU-OPENNEXT-REFRESH-DURING-FOLLOWUP-LOSES-TURN2-TEXT), interleaved-content.spec.ts:287, comprehensive-refresh.spec.ts:520 and :400. !272's first pipeline had two: api-consistency.spec.ts:161 ("assistant message has no content") and comprehensive-refresh.spec.ts:348 (120s timeout). FU-CF-DO-RESUMABLE-READER-DROPS-FRAMES is a likely cause of the refresh-during-follow-up family; re-measure after !272 lands before chasing the rest.
FU-OPENNEXT-REFRESH-DURING-FOLLOWUP-LOSES-TURN2-TEXT: refresh mid-stream in turn 2 leaves turn 2 with no text — done
Closed by: the S5 snapshot MR (branch reliability-example-refresh-flake, design spec docs/superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md).
Resolution: the root cause was RM-08 (a torn snapshot): a reload while turn 2 streamed could read a snapshot whose cursor was past turn 2's committed text, or that was ended without its last step, so the resume delivered nothing for that text. The snapshot is now one checkpoint-anchored read (messages = the committed rows up to the checkpoint, streamSequence = that checkpoint's cursor), and the example resumes with the canonical client (snapshotResumeOptions, the C-1 replay prefix). A second, independent cause (FU-CF-DO-RESUMABLE-READER-DROPS-FRAMES) was fixed by !272. Evidence: the test (no longer quarantined) passed 15/15 at --repeat-each=15 --retries=0, before and after the rebase onto 5bd30e20da, and comprehensive-refresh.spec.ts passed 650/650 under 6 workers × 50 repeats with no retries (tmp/s5-evidence/un-quarantine-comprehensive-refresh-refresh-during-follow-up{,-rebased}.log, green-stress-6x50-branch-paced.log). Base a118f8e17e lost text permanently in 24 of 1450 stress runs. See FU-OPENNEXT-EXAMPLE-FLAKY-REFRESH-TESTS.
Owner: S5 (reliability-example-refresh-flake), ruling by the reliability controller, 2026-09-26. Root cause: RM-08 (torn snapshot); S5's checkpoint-anchored snapshot work is the fix. Test: examples/opennext-cloudflare-do/tests/comprehensive-refresh.spec.ts:472 "refresh during follow-up", assertion :517 (getTextParts(turn2Msg).length >= 1, received 0).
Evidence:
- Local, same machine,
--repeat-each=15 --retries=0: 10/15 fail on main's wrangler 4.63.0 and 2/15 on !272 (wrangler 4.141.0 plus the FU-CF-DO-RESUMABLE-READER-DROPS-FRAMES fix). Every failure is at:517. - CI: flaky (failed once, then passed on Playwright's in-run retry) in 14 of the 36 opennext jobs in the 45 pipelines before !272, including main 16757438431 and 16754640438. On !272 it failed both attempts in job 16758515639.
What it is: after a reload while turn 2 is streaming, and after the stream ends, the second assistant message has no text part. Turn 1 is intact. It is a content-loss bug on the resume/refresh path, not a test-timing issue: the test waits for waitForStreamEnd before it asserts. Next step: capture the snapshot plus resume SSE (the zz-diag network capture used for the reader bug works) on a failing iteration.
FU-CP81-CF-DO-LIFECYCLE-HOOKS-QUARANTINE: cf-do-d1 lifecycle-hooks-parity Scenario 2 quarantined on CP-52 (filed as CP-81, withdrawn as its duplicate) — done
Closed by: the S5 snapshot MR (branch reliability-example-refresh-flake, design spec docs/superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md).
Resolution (S5, plan Task 24): the CP-52 fix landed in S5's snapshot MR (a submit landing inside onAgentSuspended is never dropped; durable wake, bounded by the crash-loop budget). The gateTracingContextPersistence: true line and its comment are removed from lifecycle-hooks-parity.cf.test.ts; the cf-do-d1 Scenario 2 case passed 15/15 separate vitest run --retry=0 processes. Removed from the quarantine register.
Owner: S5 (reliability-example-refresh-flake). The user assigned the CP-81 fix to S5's snapshot MR. CP-81 was withdrawn as a duplicate of CP-52 (same cause and fix site), so the fix and the un-quarantine cite CP-52.
Quarantined (2026-09-27, user decision): only the cf-do-d1 case of packages/e2e/src/__tests__/lifecycle-hooks-parity.cf.test.ts › "tracingContext set by onAgentSuspended persists through resume → completion". The entrypoint sets the existing gateTracingContextPersistence: true option, which skips Scenario 2 for that entrypoint only. The JS, Temporal and DBOS cases (lifecycle-hooks-parity.integ.test.ts) and cf-do-d1 Scenarios 1 and 3 are unchanged.
Failure: expected 'suspended_client_tool' to be 'completed'. CP-52 (was CP-81): the CF DO drops a client-tool submit that arrives while the onAgentSuspended hook is running. ensureExecutionContinues returns early on isExecuting (packages/runtime-cloudflare/src/do/durable-object-agent-base.ts:653), and the run exits suspended with nothing to wake it. CI jobs: 16758514869, 16759542448, 16760323616, 16764206430.
Exit condition: S5's snapshot MR fixes CP-52 and removes the gateTracingContextPersistence: true line (with its CP-52 comment) from the cf entrypoint. The same MR shows the case green at 15 or more repeats with no retries.
FU-DBOS-PERSISTENT-SEND-INTO-EXITING-WORKFLOW: DBOS persistent message sent during idle-timeout exit is lost (CP-74); test quarantined — done
Closed by: the S5 snapshot MR (branch reliability-example-refresh-flake, design spec docs/superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md).
Resolution (S5, plan Task 24): the CP-74 fix landed in S5's snapshot MR (R43: the run id is the single dedupe key, and a late inbox read follows the idle finalize). The it.skip and its comment are removed; "idle-timeout exit then new message starts a fresh persistent workflow" passed 15/15 separate vitest run --retry=0 processes on Postgres. Removed from the quarantine register.
Owner: the DBOS persistent-mode session (CP-74 fix). Finding: CP-74 (packages/e2e/src/conformance/catalogue/control-plane/CP-74.ts). A message sent into a persistent workflow that is exiting on its idle timeout is lost. The session already reads completed, but the workflow is still DBOS-PENDING, so execute() sends into an inbox that nothing reads again.
Quarantined (2026-09-27, user decision): only packages/e2e/src/__tests__/multi-message-execute-dbos-persistent.integ.test.ts › "idle-timeout exit then new message starts a fresh persistent workflow", as it.skip (vitest 2 has no fixme) with a CP-74 comment. It failed both attempts on !275. The rest of the file still runs.
Exit condition: the CP-74 fix's MR removes the it.skip and its comment, and shows the test green at 15 or more repeats with no retries.
FU-CONF-STOPWHEN-PARITY: stopWhen honoured only on JS and the CF DO — done
Closed by: conformance catalogue batch 6 (branch conformance-catalogue-batch-6), which verified the lead and catalogued it as DI-29 (S1, open). See DI-29 in docs/conformance/findings.md.
The batch-5 reviewer's F9 source lead held: Temporal and DBOS call shouldStopExecution with maxSteps only, and CF Workflows never calls it. A probe with stopWhen: () => true stopped JS after 1 LLM call, while Temporal and DBOS made 3 and CF Workflows ran to maxSteps 5 and failed. The docs now carry DI-29 caveats (the agents guide, execution-flow, and the Temporal, DBOS and CF Workflows runtime pages). The runtime fix itself is tracked by DI-29, not here.
FU-CONF-TEMPORAL-REAP-LATE-CHILD — a child start the server records only after the reap's lookup bound — done
Closed by: MR !275 (branch conformance-temporal-drain-children), in its Greptile P1 fix.
Gap:
- A parent's final history can record
StartChildWorkflowExecutionInitiatedwith noChildWorkflowExecutionStartedorStartChildWorkflowExecutionFailedafter it. - The reap looked such a child up for 1s, then dropped it and forgot the closed parent.
- A child started later escaped every later reap, dispose included.
Fix:
StartedWorkflows.terminateTreenow KEEPS each start it could not resolve.- Every later call looks every kept start up again first ("pass 0"). That includes the next cell's reap about other sessions, and dispose.
- Once the child exists, pass 0 walks and terminates it, attributed to its own session.
- A start still unresolved at dispose is reported (
CellWorkTermination.unresolvedChildStarts, plus aconsole.warn) and is never dropped.
Temporal semantics checked:
temporalio/temporal'sprocessStartChildExecutionreturns early when the parent is no longer running and the child has not started, for everyParentClosePolicyincludingABANDON. Its comment reads: "ideally for case 3, we should continue to start child … we can't do that".- So a late start is possible only from a start task already in flight when the parent closed.
- The Docker dev server runs this same server code.
Testing:
- There is no live test. The in-flight window is too narrow to force, and the harness has no hook to pause the server's start task.
- It is covered by fake-history unit tests (
temporal-client-adapter.test.ts, "P1: …"). They failed before the fix: the late child was not reaped, andpendingChildStartsdid not exist.
FU-CONF-TEMPORAL-DRAIN-GROUND-TRUTH — judge a conformance cell's Temporal work "stopped" from backend truth, and end the whole workflow tree — done
Closed by: branch conformance-temporal-drain-children, the conformance harness MR "reap a cell's whole Temporal workflow tree at cell end; judge drain from backend truth", plus its fix rounds 1 and 2.
Problem: the executor driver judged "stopped" from whether a product handle's promise settled, and ended only the root workflows its client had started. On the temporal-* lanes, a persistent-companion child workflow could outlive its cell. A parent stop never reaches a blocking companion (CP-23), and a non-blocking one is ABANDONed. That orphan kept calling the model and writing to its torn-down parent stream. history.companion-continue (B2-core) produced 20–21 (test server, across two before-runs) / 24 (Docker) unhandled "Cannot write to stream … in 'failed'/'ended' state" rejections from runtime-temporal activities.ts on the full temporal file. test:conformance exited 1 although every cell passed.
Fix:
- New optional hook
BackendEnv.terminateCellWork, implemented only by the Temporal helpers (StartedWorkflows.terminateTree).- It walks the workflow tree through each parent's history, never through visibility (the Java test server can't list) and never through runtime-temporal's child-id naming.
- It gives running executions a short close grace, terminates what is still running top-down, re-describes each execution, and attributes terminations to sessions.
- New
RunLog.reapSince(mark):runCellreaps at EVERY cell end.- An execution still running after termination is a genuine leak.
- A run whose executions closed but whose handle never settled is reported in two disjoint classes:
terminatedUnsettled(the harness terminated an execution of its session; session-level, may mix causes) orclosedUnsettled(closed on its own; product-side only).
stopAndDraindoes the same at dispose, replacingterminateInFlightthere. A handle-less undrained run still fails dispose, as on JS and DBOS. The worst case is held underDISPOSE_TIMEOUT_MSby the budget test.- JS and DBOS lanes are unchanged.
- The real-infra test is
packages/e2e/src/harness/__tests__/temporal-cell-reap.integ.test.ts. It passes on the test server and on the Docker server.
FU-CONF-TEMPORAL-REAP-LATE-CHILD (above) was closed in the same MR. The candidate product finding it surfaced is FU-CONF-TEMPORAL-RESUME-HANDLE-EXTERNAL-TERMINATE.
FU-A2-03 — Harden persistent sub-agent edge cases under __resume-N
Closed by: commit 3ebed60f7. Summary: Added focused e2e regression test persistent-subagent-suspension-temporal.integ.test.ts exercising the parent-suspend + persistent-blocking-child interaction. Scenario: parent dispatches a persistent blocking child → child completes → parent calls a client tool (workflow exits with 'suspended_client_tool') → consumer submits result + resumes parent → parent finishes via __finish__. Verifies the persistent SubSessionRef survives the __resume-N workflow boundary intact (no duplication, no status reset). Result: A.2's design is correct — no production fix needed. Non-blocking persistent + parent-suspend is covered structurally by the same code path; the e2e test for it is blocked on shared MockLLMAdapter response-queue races between parent and non-blocking child (a pre-existing test infrastructure limitation documented in persistent-subagent-temporal.integ.test.ts).
FU-A2-01 — Per-call agent hooks not plumbed through Temporal runtime
Closed by: commit 2693eb18f (+ changeset da8bd52ea). Summary: Added AgentRegistry.replace(config) API to runtime-temporal and runtime-cloudflare. Replaces an existing static registration with a new agent config, returning whether a prior registration existed. Wired replaceAgent capability through the e2e harness's BackendEnv (no-op on JS/DBOS; calls registry.replace() on Temporal). approval-gate-hook-parity tests un-skipped on Temporal; both approve and deny paths pass. Pattern: tests call await env.replaceAgent(agent) before each executor.execute(agent, ...) to swap the registered reference for one carrying the test's hooks. JS/DBOS continue to honor inline per-call hooks without registry intervention.
FU-A2-02 — SUBMIT_TOOL_RESULT_SIGNAL_NAME no-op signal call
Closed by: commit 869787e5c. Summary: Removed the no-op handle.signal('submitToolResult', ...) call in applySubmitToOwner and the orphan resolveWorkflowId helper. Post-A.2 the workflow exits on every HITL boundary so the signal hit a completed workflow and only generated temporal.signal_failed warn logs. The durable pendingClientToolCalls write IS the canonical signal; consumers drive the next loop iteration via executor.resume(). Updated 6 unit tests; deleted 2 tests covering the now-impossible signal-failure scenarios.
FU-A1-01 — runId generator pattern duplication
Closed by: commit 869787e5c (batched with FU-A2-02). Summary: Extracted duplicated `run-${Date.now()}-${Math.random().toString(36).slice(2, 10)}` into a module-level generateRunId() helper. Two call sites in execute() and resume() now share one source of truth.
FU-A2-06 — AgentWorkflowResult status enum extension changeset
Closed by: commit e221ac4d3. Summary: Added a MINOR changeset for @helix-agents/runtime-temporal documenting the A.2 Temporal HITL rewrite per the project's "MINOR = observable behavior change" versioning policy. Documents three new 'suspended_*' status values, durable-only submitToolResult, __resume-N workflow ID convention, and the five removed signals (submitToolResultSignal, childSuspendedSignal, childWokeSignal, runResumedSignal, RUN_RESUMED_SIGNAL_NAME).
FU-A2-08 — Remote sub-agent SubSessionRef metadata not persisted on Temporal
Closed by: commit 5085756d0. Summary: The workflow body's dispatchRemoteSubAgents called recordSubSessionResult (core helper) which only UPDATES existing SubSessionRefs — it doesn't CREATE them. For ephemeral sub-agents the ref is created via recordSubAgentResult's addSubSessionRefs call (mode='ephemeral'); for remote sub-agents, no ref-creation path existed.
Fix: added addRemoteSubSessionRef activity that calls addSubSessionRefs with mode: 'ephemeral' + remote: { streamId, lastSequence } metadata. Workflow body's dispatchRemoteSubAgents calls it BEFORE recordSubSessionResult so the ref exists for the helper to update on the result.
Result: remote-agent-temporal.integ.test.ts now 4/4 PASS (was 3/4 + 1 skip). Cross-package typecheck 55/55 PASS.
FU-A2-05 — Cross-runtime Temporal e2e migration
Closed by: commits 13075bc45 (Phase 1: persistent sub-agent fix) + 332c0b2a0–646615b2a (Phase 2: 11 file migrations). Spec: docs/superpowers/specs/2026-05-08-temporal-e2e-migration-design.mdSummary: A.2's Temporal workflow body rewrite caused 18 e2e tests to fail with Completed workflow errors (signal-pattern hits exited workflow) plus a real persistent sub-agent regression (expected 'running' to be 'completed').
Phase 1 fixed the persistent sub-agent regression with two complementary changes: (a) Option A — added resolveAndSyncSubSessionRefs lazy-sync helper to packages/core/src/orchestration/companion-tool-dispatch.ts mirroring DBOS's pattern, syncing stale 'running' refs to terminal status when companion tools read them; (b) Option B — added markPersistentChildCompleted activity called from the workflow body after await childHandle.result() resolves on blocking spawn paths.
Phase 2 added a TemporalAgentExecutor to createTemporalTestContext() (with lazy-startup background worker to avoid contention) and migrated 11 test files from handle.signal('submitToolResult', ...) to the v7 stateless pattern (executor.submitToolResult + executor.resume).
Test results: 13/13 migrated tests pass (was 12/12 fail). 2 tests remain skipped pending FU-A2-07 (sub-agent ownership StaleStateError) and FU-A2-08 (remote SubSessionRef metadata) — both real underlying regressions documented for follow-up rather than band-aided.
FU-A2-09 — Temporal sub-agent resume after failed:parent_suspended
Closed by: commit landing FU-A2-09 fix. Summary: After FU-A2-07 fixed the parent's commitSuspendedStep StaleStateError, the parent's resume reached applyResultsAndReload but failed to drive the failed:parent_suspended child to completion. Two interacting issues:
applyResultsAndReloadleftfailed:parent_suspendedchildren inremainingChildren, settinghasMoreSuspended = true. The workflow body's resume path early-exited with'suspended_awaiting_children', skipping the main loop wheredispatchSubAgentscould γ-cascade re-spawn.- The main loop calls
runLLMStep, which doesn't return sub-agent calls on resume (the LLM has already produced the prior step). SodispatchSubAgentswouldn't re-fire even after lifting the early-exit gate.
Fix (variant of option (c) + workflow-body re-dispatch):
applyResultsAndReloadnow returns a separatechildrenToRespawnfield listingfailed:parent_suspendedchildren. These do NOT count towardhasMoreSuspended(the early-exit gate).- The workflow body's resume path re-dispatches them DIRECTLY via
wf.startChild(using__resume-Nworkflow IDs per spec §5.2 mit#1) BEFORE the early-exit gate. Each re-spawn awaits the child workflow's result; on completion,recordSubSessionResultappends the tool- result message to the parent's history (emittingsubagent_end+tool_endchunks exactly once). On re-suspension, the parent re- suspends with the child still insuspendedAwaitingChildren. - New activity
commitChildRespawnResultsperforms the durable bookkeeping: clears re-spawned children fromsuspendedAwaitingChildrenand re-inserts any that re-suspended.
This is option (c) from the original analysis but cleaner — instead of re-entering the main loop (which requires runLLMStep to fire empty and the dispatcher to re-fetch state), the workflow body does the re-dispatch directly and the main loop only runs when there's actual LLM work to do.
The executor.resume(child) short-circuit (issue #1 in the original description) turned out to be moot — the parent re-spawns the child via wf.startChild from inside the workflow, never going through executor.resume(child).
Un-skips packages/e2e/src/__tests__/client-tool-subagent-ownership-temporal.integ.test.ts.
A.3 — DBOS resume() contract fixes (F7)
Closed by: commits 8ec773936, 16e4acb99, 7d9498b8e. Summary: Single truthy-object pitfall on resume.ts:79 caused 3 reproducible DBOS resume() contract failures (resume on completed, active, or concurrent suspended sessions all incorrectly succeeded). Fix: single multi-status CAS with explicit .ok check, mirroring runtime-temporal's post-A.2 pattern. Added 5 regression-guard unit tests so the bug can't return silently.
FU-A2-39 — Lazy-load Node-only harness setup helpers (workerd compat)
Closed by: commit 0d789d40d. Summary: packages/e2e/src/harness/backend-descriptor.ts previously imported every setup helper at module top-level. Workerd's parser rejected ioredis / pg source on module-load, blocking the C2 "re-run the same integ test in workerd via a 1-line companion import" pattern. Two coordinated changes:
backend-descriptor.ts: converted eager imports to a lazySetupLoaderpattern using alazyImport<T>(path: string)helper that defeats the bundler's static analysis. Each backend now exposessetupLoader(lazy) +setup(resolved accessor).get-viable-backends.ts: switched workerd-context detection from the brokenD1Databaseglobal probe (not exposed by@cloudflare/vitest-pool-workers) to aWebSocketPairprobe (verified empirically). Added a hard-filter for cross-context backends rather than just marking them with a skipReason.lifecycle-hooks-parity.cf.test.ts: replaced itsdescribe.skipplaceholder with a directimportof the integ companion so the harness re-run works under workerd.
The descriptors cf-do-d1 and cfw-workflows-d1 carry an IMPL_PENDING_O5 env requirement marking the deferred O5 work (real-DO + CFW Workflows setup helpers under workerd's vitest pool).
FU-A2-17 — CF DO persistent sub-agents real-DO test expansion
Closed by: Wave 4 test-only commit (this branch). Summary: Expanded cloudflare-do-persistent-agents.cf.test.ts beyond the original two FU-A2-38 G1 scenarios (blocking spawn + waitForResult) to exercise the full companion/sub-agent matrix against real workerd DOs. Four new scenarios (all green on workerd):
- Non-blocking spawn — parent does not wait.
companion__spawnAgent(non-blocking) returnsstatus: 'spawned'and the parent finishes on its own. Teeth: reads the parent's persisted spawn tool result via the/messagesroute and asserts{ status: 'spawned' }with no drainedoutput— a blocking-mode regression (silently waiting) would surfacestatus: 'completed'+outputand fail. - Child failure observed by parent (blocking spawn). A
failerchild whose LLM step errors withrecoverable: falselands instatus: 'failed'. Teeth: the parent's blocking spawn tool result is{ status: 'failed' }(the parent observes the failure, NOT a silent success) AND the child DO's/statusreportsfailed. Verified the assertion fails if the child status is read ascompleted. - Parent interrupt during (non-blocking) child run — CF DO divergence. Spawns a non-blocking child (so the parent releases workerd's input gate), then
POST /interrupts the parent. Asserts the v7 contract (200/400, NEVER 503) and a consistent parent terminal state (completed | interrupted, neverfailed), and that the independent child DO is not collaterally corrupted (completed | running | not_found, neverfailed/interrupted). Documented divergence: the CF DO blockingspawnAgentpoll loop runs inside the tool'sexecute(), does NOT break on the parent abort signal, and does NOT write a durable interrupt flag to the child — the parent observes its own flag only at the NEXT step boundary after the tool returns. This matches the per-runtime interrupt table indocs/internals/subagent-execution.md(CF DO: per-stepcheckInterruptFlag). - Cascading nested sub-agents (3 levels). New
cascade-l1→cascade-l2→researcherchain; each level runs in its OWN DO of the self-bound class and the DO base rewritespersistentAgentsper started agent type, so a child DO spawns a grandchild. Teeth: asserts both the root's and the mid-level child's spawn tool results arecompleted(proving the chain drained bottom-up across 3 DO instances).
Fixtures added to packages/e2e/src/test-worker.ts: a failer blocking persistent agent and the cascade-l1/cascade-l2 coordinators (all registered on the existing PersistentAgentTestServer registry — no new DO class or wrangler binding needed; the self-bound class hosts every level).
Harness caveat (not a runtime bug): all DO instances share ONE persistentAgentTestLLM FIFO response queue, and non-blocking children run fire-and-forget, so a racing background child can drain a response the parent expected. The deterministic blocking scenarios serialize via the spawn poll loop; the non-blocking scenarios make BOTH finish responses schema-compatible with both agents' output schemas ({ done, text, summary } — Zod strips extras) so whichever agent grabs whichever finish, completion-retry never fires. This mirrors the documented limitation in the JS persistent-subagent-parity.integ.test.ts. No production code changed; no runtime bug found.
FU-A2-38 — Re-enable Cloudflare DO real-DO end-to-end tests (G1 + G2)
Closed by: commit 30432ad34. Summary: packages/e2e/src/__tests__/cloudflare-do-persistent-agents.cf.test.ts (G1) and cloudflare-do-interrupt-protocol.cf.test.ts (G2) were previously describe.skip because the supporting test-worker.ts fixtures and wrangler.toml durable-object bindings hadn't been wired up. Both files now run end-to-end against real DOs.
Wired up:
packages/e2e/src/test-worker.ts:persistentAgentTestLLM+PersistentAgentTestServerDO class (self-binding viasubAgentNamespace: env.PERSISTENT_AGENT_SERVER, two persistent agent configs — researcher (blocking) + writer (non-blocking) — declared viaparent-with-persistent'spersistentAgentsfield).interruptTestLLM+InterruptTestServerDO class with anaskUserTool(execute: CLIENT_TOOL_EXECUTE) for the HITL paused-state branch of the interrupt-route contract test.
packages/e2e/wrangler.cloudflare.toml:- New
PERSISTENT_AGENT_SERVERbinding (class_name =PersistentAgentTestServer). - New
INTERRUPT_AGENT_SERVERbinding (class_name =InterruptTestServer). - Migrations v6 + v7 for SQLite-backed DO storage.
- New
- Both
.cf.test.tsfiles: removed thedescribe.skipand the inline FU-A2-38 placeholder comments. The G1 tests cover persistent sub-agent companion-spawn under real workerd-DO; G2 pins the v7 interrupt-route contract (200 / 400 / NEVER 503 — the v6INTERRUPT_NOT_LOCAL503 contract is gone).
FU-A2-40 — CFW Workflows γ-cascade re-spawn
Closed by: commit 6cbd78808. Summary: Mirrored runtime-temporal's γ-cascade closure (FU-A2-09) in CFW Workflows. Three coordinated changes plus dedicated coverage:
packages/runtime-cloudflare/src/steps.ts—commitSuspendedStepnow marks eachsuspendedAwaitingChildrenentry's child session asfailed:'parent_suspended'with CAS retry.applyResultsAndReloadgained achildrenToRespawn: Array<...>return field, populated by a privatescanChildrenForRespawnthat filters durable child states forstatus === 'failed' && failureReason === 'parent_suspended'.packages/runtime-cloudflare/src/workflow.ts— added a re-dispatch loop in the resume branch. For each surfaced child:step.do('respawn-...', () => workflowBinding.create({ id: 'agent__<type>__<id>__respawn__<attempt>', params: {...} }))(the format was__respawn-<attempt>until CF-D1 MR 1a), then a bounded poll (max 5 min, exp backoff) on the child's durable state until terminal, thenrecordSubSessionResultto append the synthetic tool-result message to the parent's history.- New drain-clear step removes drained children from
suspendedAwaitingChildrenand resets the suspension discriminators (status: 'active', clearsuspendedStepId, clearfailureReason) when fully resolved. Without this reset, the main loop's commit path would re-route throughcommitSuspendedStepand produce a spurious'suspended_step_partial'final status when the LLM step completes the run.
Coverage:
- New
subagent-respawn-on-resume.integ.test.ts— 3 D1+Miniflare tests pinning the full γ-cascade contract (mark on suspend, respawn on resume + drain + clear awaiting, no-respawn fallback when no marker present). - 220/220 runtime-cloudflare integ + 1048/1048 unit pass.
P3.R3-PERF.2 — D1 promoteStaging single targeted read
Closed by: commit 388691974. Summary: Pre-fix, D1StateStore.promoteStaging issued THREE reads against D1: a minimal SELECT step_count, custom_state from __agents_states, a COUNT(*) against __agents_messages (for messageCount), and a full loadState (SELECT * FROM __agents_states) for the checkpoint payload. The minimal SELECT and the full loadState BOTH read __agents_states for fields that overlap heavily — wasted work on the hot path (every step boundary calls promoteStaging).
Fix: single targeted SELECT against __agents_states fetching exactly the columns needed for the checkpoint payload (session_id, agent_type, stream_id, step_count, status, output, error, custom_state, interrupt_context, version, resume_count). The COUNT(*) against __agents_messages is unchanged. The loadState call is gone.
Coverage: new d1-state-promoteStaging-reads.test.ts pins the read shape with a prepare()-spy decorator over MockD1Database: exactly 1 SELECT against __agents_states (was 2 pre-fix), 0 SELECT * queries against __agents_states (regression smoke), exactly 1 SELECT (COUNT) against __agents_messages, and exactly 1 SELECT against __agents_staging (getStagedChanges, unchanged). Plus a behavioral assertion that checkpointId is still returned. 242/242 store-cloudflare unit + 402/402 integ pass.
P3.R3-PERF.3 — Postgres three-phase migration runner with concurrent DDL
Closed by: commit 1d2bb369c. Summary: PostgresMigrations.run() previously wrapped every migration's statements in a single transaction. CREATE INDEX CONCURRENTLY is forbidden inside transactions, so V7's partial expiresAt index either had to use a regular CREATE INDEX (taking ACCESS EXCLUSIVE on __agents_states for the build duration) or rely on a documented operator manual workaround.
The runner now uses a three-phase model: (1) transactional schema phase (statements — ALTER TABLE / CREATE TABLE), (2) concurrent phase (concurrentStatements — CREATE INDEX CONCURRENTLY etc., runs outside any transaction), (3) version-row commit phase (INSERT ... ON CONFLICT (version) DO NOTHING). Phases are ordered so concurrent indexes can reference columns added in the same migration. INVALID indexes from prior crashed CONCURRENTLY builds are dropped before retry. V7 retagged: partial expiresAt index moved into concurrentStatements.
Concurrency: dropped the session-level pg_advisory_lock (broken under connection pools — different connections can't release it). Replaced with xact-scoped locks inside phases 1/3 + IF NOT EXISTS idempotency for phase 2 + ON CONFLICT for the version-row insert. Safe across multiple migrator processes.
Coverage: postgres-migrations.test.ts gained 9 new tests pinning the contract: lock NOT taken (smoke against the pre-fix pattern), ON CONFLICT DO NOTHING on version insert, concurrent statements run via adapter.query (NOT inside transaction), invalid-index cleanup before CONCURRENTLY rebuild, prefix-aware index scoping, error propagation, logger event surface. The applyUpTo integ helper was updated to mirror the runner's phase model so test schemas match production.
Tests: 197/197 unit + 327/327 integ for store-postgres.
Postgres-LISTRUNS-IDX — V8 composite (session_id, status, turn) runs index
Closed by: commit c596e67ab. Summary: Closes Postgres parity with the Redis closure in 296ad28e7. Pre-fix, PostgresStateStore.listRuns(sessionId, { status }) filtered with only idx___agents_runs_session_id available, doing a per-session run scan + in-memory status filter on every status query. V8 adds idx___agents_runs_session_status_turn ON __agents_runs(session_id, status, turn) so the composite supports both the WHERE and the ORDER BY without a separate sort step. Built via the concurrentStatements phase (P3.R3-PERF.3) so the index build doesn't block writes on the hot-path runs table. Coverage includes a behavioral test (5 runs, mixed statuses, turn ordering) and an EXPLAIN-based plan assertion (SET LOCAL enable_seqscan = off to force the index path on small test data).
P3.R3-PERF.4 — message_count denormalization (Postgres V9 + D1 V10)
Closed by: commit 054194def. Summary: Pre-fix, every listSessions call evaluated (SELECT COUNT(*) FROM messages m WHERE m.session_id = s.session_id) for every row in the WHERE-matched set — O(N × M) cost where N is matched session count and M is per-session message volume. Both Postgres and D1 paid this cost.
Post-fix: a denormalized message_count INTEGER NOT NULL DEFAULT 0 column on __agents_states. V9 (Postgres) + V10 (D1) ALTER TABLE
- idempotent backfill UPDATE (recomputes from the messages table; re-runnable for drift recovery). The mutating code paths (
appendMessages,saveStateAndPromoteStaging,cloneSession) maintain the column in lockstep with the messages table.listSessionsreads it directly — no per-row subquery work.
saveStateAndPromoteStaging's checkpoint payload now reads the message_count from the states row directly (one less SQL round-trip on every step boundary).
Coverage: 5 new tests in each store's list-query integ file covering appendMessages increment, multi-call accumulation, saveStateAndPromoteStaging atomicity, zero-message sessions, and EXPLAIN-based proof that listSessions does NOT touch __agents_messages post-fix. Cursor pagination (the second half of P3.R3-PERF.4) is deferred — denormalization captures the dominant perf win.
D1-DELETESESSION-TOCTOU — orphan-message race in appendMessages
Closed by: commit b4e393670. Summary: Pre-fix, D1StateStore.appendMessages did the existence check and the message INSERT batch in SEPARATE D1 round-trips. A concurrent deleteSession could land between them, leaving orphan messages with no states row.
Post-fix: each message INSERT is gated by WHERE EXISTS (SELECT 1 FROM __agents_states WHERE session_id = ?) in an INSERT...SELECT form (SQLite's INSERT doesn't accept WHERE). If the gate fires, meta.changes = 0; we sum changes across the batch and throw StateNotFoundError if the total is less than the message count. The mock D1 was extended to recognize the EXISTS-gated INSERT...SELECT pattern so unit tests reproduce the contract.
Coverage: 3 new tests in d1-state.integ.test.ts covering the canonical race scenario, the happy path regression guard, and a partial-deleteSession edge case.
CLEANUP-ORPHANED-STAGING — cross-session orphan sweep (Postgres + D1)
Closed by: commit cab06588e. Summary: Mirrors the runtime-redis closure in 296ad28e7. Pre-fix, Postgres and D1 had only the per-session cleanupOrphanedStaging(sessionId) — operators had no way to sweep abandoned staging rows without enumerating every session ID first. Storage on __agents_staging grew unboundedly for paused / abandoned sessions on production deployments.
Post-fix: both stores expose cleanupOrphanedStagingData(maxAgeMs?: number): Promise<number> that DELETEs staging rows older than now - maxAgeMs (default 1 hour) and logs a structured cleanup_orphaned_staging.summary event. NOT on the SessionStateStore interface — per-store operator method, same convention as Redis.
Coverage: 3 new tests in each store's list-query integ file covering the age filter, zero-deletes case, and multi-session sweep.
P3.R3-BC-MISC (D1 chain collapse) — collapsed initial schema for fresh-DB deployments
Closed by: commit b23000542. Summary: Pre-fix, fresh deployments ran the full V1..V10 migration chain — 10 sequential db.batch() round-trips on every fresh-DB worker boot, adding ~100-500ms of dead time.
Post-fix: runMigration() detects fresh databases (no __agents_states table present in sqlite_master) and applies a single SCHEMA_COLLAPSED_INITIAL constant — the post-V10 schema in one batch, plus all incremental version rows for audit-trail parity. Existing deployments at any partial version (1..9) continue down the V1..V_current incremental chain unchanged.
Detection contract uses sqlite_master so a partial schema from a prior crashed run falls through to the safe incremental path (no silent CREATE TABLE IF NOT EXISTS no-ops mis-reporting the version). Audit-trail parity: the collapsed path records EVERY incremental version row, not just CURRENT_SCHEMA_VERSION, so operator queries like SELECT * FROM __agents_migrations WHERE version = 7 behave consistently across deployment paths.
Coverage: new d1-collapsed-schema-parity.integ.test.ts runs both paths in isolation and compares PRAGMA-reported schemas structurally (table column lists + index definitions). 5 tests covering version stamp, per-table column parity (sorted by name to absorb ALTER TABLE column ordering), per-table index parity (name + columns + uniqueness + partial flag), and version-row history parity.
Maintenance contract: when adding V_{n+1}, also append the new statements to SCHEMA_COLLAPSED_INITIAL. The parity test catches missed updates.
Tests: 242 unit + 420 integ for store-cloudflare. Full turbo typecheck 55/55.
FU-TYPE-SAFETY-2026-05 — Eliminate any + forced casts across all packages
Closed by: commits 4fd490327 (Stages A + B.1 + B.3), 47e3faa18 (Stages B.2 + B.4 + C + D + ToolContext.getState docs), c94ca7606 (close-out review findings). Surfaced by: four parallel pr-review-toolkit:type-design-analyzer agents (2026-05-10) run after the BC removal pass landed.
Summary: ~200+ any annotations and forced as Type casts removed across all packages. The fix spanned four stages:
Stage A — Fast wins (~110 violations): Introduced shared aliases AnyTool = Tool<z.ZodType, z.ZodType> and AnyAgentConfig = AgentConfig<z.ZodType, z.ZodType> in core; bulk- replaced the <any, any> epidemic across 40+ runtime files. Tightened HookManager.invoke<K extends keyof AgentHooks> so temporal + cloudflare + client-tool-workflow-helper no longer need triple-casts. Switched helix-to-aisdk-converter.ts to use its own isToolPart guard (now validates state + toolCallId, not just the discriminator). Replaced type DurableObjectState = any test stub with a structural shape. Dropped 11 tracker.getState() as any sites in favor of typed generic bridges. Replaced as any as Redis test mocks with a stubRedis(...) helper. Replaced statements: any[] in d1-mocks with MockD1PreparedStatement[].
Stage B — Zod schema introduction (~80 violations): Authored packages/core/src/types/state-schemas.ts exporting SessionStatusSchema, RunStatusSchema, CompletionReasonSchema, SubSessionStatusSchema, SubSessionRefStatusSchema, SubSessionRefModeSchema, AgentStatusSchema, plus discriminated- union schemas for Message, UsageEntry, Memory, the content- part variants (SystemMessageSchema, UserMessageSchema, AssistantMessageSchema, ToolResultMessageSchema), and parseAgentStateCheckpoint / AgentStateCheckpointView helpers. Every schema ships a _DRIFT_ASSERTIONS satisfies _SchemaTypeDriftAssertions compile-time guard binding the schema to its TypeScript counterpart — the previous bare-type guards were decorative and silently passed under drift; the new value-bound guard catches optional-field drift via StructuralMatches<A, B> + KeysMatch<A, B>. Applied at every JSON.parse(row.x) as Y site across store-postgres, store-cloudflare, store-redis, memory-redis, and the DO state / usage / stream stores. Tightened DO HTTP wire-protocol schemas: StartAgentRequestV2Schema.history is now z.array(MessageSchema); SubmitToolResultResponseSchema validates cross-DO fetch responses; AgentStatusResponseSchema.status is the 8-literal enum instead of z.string().
Stage C — Public API surface (~30 violations): Typed DOFrontendExecutor.execute(_agent: AnyAgentConfig, input: AgentInput<unknown>, ...) and getHandle() with consumer-side docs explaining why they're not AgentExecutor implementations. Typed agent-server/types.ts agents record to Record<string, AnyAgentConfig>. Replaced the metadata?: any HTTP-handler parameter with the same Zod-validated shape used elsewhere. Typed do-clients.ts public-API surfaces via generics over <TState, TOutput> where possible.
Stage D — Test mock helpers (~90 violations): Authored packages/llm-vercel/src/__tests__/helpers/make-chunk.ts with a makeChunk<T> factory; ~60 as unknown as TextStreamPart<ToolSet> casts replaced across chunk-mapper.test.ts and chunk-mapper-edge-cases.test.ts. Authored packages/memory/src/__tests__/helpers.ts with typed mockLanguageModel + mockMemoryStore stubs; ~30 {} as any sites swept across 8 memory test files. Rewrote ai-sdk snapshot stability test's (p: any) => callbacks with a LoosePart structural type. Replaced production any patterns in packages/ai-sdk/src/handler/handler-factory.ts (~7 sites), packages/llm-vercel/src/vercel-adapter.ts (~4 sites), and packages/runtime-js/src/{js-agent-executor,run-loop}.ts (5 sites) with typed structural bridges or generic <T>() => accessors.
ToolContext.getState<T>() — caller-driven contract: the <T>() => T signature is intentional, not a violation. The persisted state is opaque at the runtime boundary; the runtime returns whatever the caller asks for and relies on the caller to validate via Zod Schema.parse(ctx.getState()). JSDoc at the API surface (core/src/types/tool.ts) documents the contract end-to-end with code examples.
Lingering documented casts (intentional, not bugs): a handful of as <Type> bridges remain at runtime/SDK boundaries where cross-package type drift is fundamental (Zod 4 framework schemas vs Zod 3 in the Vercel AI SDK's ai package generic constraints, partyserver's unknown[] template-literal return, the DO ResumeMode vs core ResumeOptions literal-union mismatch). Each cast is documented at the site with the specific rationale and is confined to a single bridge call rather than scattered.
Close-out review findings (commit c94ca7606): 3 critical, 5 high, 4 medium issues caught by two superpowers code reviews of the migration work — all fixed inline. Highlights:
- C1:
continueregression injs-agent-executor.tsdrain loop was dropping telemetry; scoped to skip only theafterTooldispatch. - C2: silent
try { } catch {}in DO checkpoint message-rebuild replaced with per-skip warn + summary log. - C3: missed
JSON.parse(row.chunk) as StreamChunkcasts inpackages/store-cloudflare/src/stream-durable-object.ts:945, 1051→ appliedStreamChunkSchema.parse(...); updated 31 test fixtures with the requiredstep: 0field. - M1:
StructuralMatches<A, B>didn't catch optional-field drift → addedKeysMatch<A, B>enforcingkeyof Aandkeyof Bmatch exactly. Mutation test (introducingintroducedDrift?: stringintoUserMessageSchema) confirmed the guard now rejects what the previous form silently accepted. - M2:
AgentStatusResponseSchema.statuswasz.string()→ tightened toz.enum([...])with the 8 specific literals. - M4: DO
/resumesilently fell through to continue semantics for'retry'/'branch'resume modes → explicit reject with warn log + thrown error at the DO boundary.
FU-DBOS-ERROR-CHUNK — DBOS failStream now emits AI SDK error chunk
Closed by: commit e7506723e. Summary: Pre-fix, DBOS's runAgentLoopOneTurn on the LLM-terminal failure path called dispatcher.failStream({...}) at packages/runtime-dbos/src/workflows/shared.ts:1098 without first emitting an error chunk into the stream. AI SDK transformers running over the SSE bytes saw the stream go from active → ended without an error event, so consumers couldn't render error state. Runtime-js and runtime-temporal already emit an error chunk before failStream. Fix added dispatcher.emitErrorChunk(...) (new EmitErrorChunkArgs + runEmitErrorChunk + emitErrorChunkStep) immediately before failStream on the failed path, mirroring the other runtimes' pattern. Verified by ai-sdk-dbos.integ.test.ts > should handle DBOS workflow errors as AI SDK error events.
FU-A2-42 — Pre-existing runtime-temporal initialState/branch/history failures (12 tests)
Closed by: commit f4a169588. Summary: Pre-fix, packages/runtime-temporal/src/__tests__/integration/temporal.integ.test.ts had 12 deterministic failures because the Temporal workflow body's 'fresh' mode never read AgentWorkflowInput.initialState, branch, or history. When callers started workflows directly (the integration test harness via runner.executeWorkflow(), the agent-server, or any consumer bypassing the executor's pre-claim createSession({initialState}) shortcut), these inputs were silently dropped — customState arrived empty, branched workflows lost source history, etc. Fix routed the three inputs through to the initializeAgentState activity with the same precedence as runtime-js / runtime-cloudflare: load base from branch.fromSessionId (missing source throws 'session not found'), history overrides branch messages (system messages filtered), schema defaults applied to initialState ?? branchState, new-session path prepends resolved conversation history before the user messages. Also deduped the LLM-error chunk: runLLMStep's onError callback writes an error chunk, and then the LLM-terminal failed path was calling persistTerminalState({status:'failed'}) which wrote ANOTHER error chunk. Added skipErrorEmit flag for the LLM-terminal branch. Additionally, top-level catch RETURNS {status:'failed', error} (matching runtime-cloudflare) instead of re-throwing, so callers can await structured failures via handle.result() rather than catching WorkflowFailedError. All 12 tests now green; runtime-temporal unit suite 242/242; runtime-temporal integ 51/51; cross-runtime e2e (temporal combo) 71/71.
FU-AISDK-1: getChunksFromStep does not respect run boundaries
Status: CLOSED 2026-05-17 via Option A. Both getChunksFromStep and getAllChunks fallback branches removed from loadPartialContentAsMessage in packages/ai-sdk/src/handler/snapshot.ts. When startSequence is unavailable the loader now returns null (no partial content) — safe degradation, the snapshot uses persisted messages only.
Background: The whitelist-inversion fix (commit 049a19e21, 2026-05-16) made the step-based loading branch unreachable for the originally-reported Bug 4 scenarios (a run with startSequence defined now always uses sequence-based loading). The follow-up fix (commit 3f098277a, 2026-05-17) ALSO closed the settled-run leak path where failed/superseded runs with status === 'paused' (race scenarios: process crash before stream cleanup, CF DO eviction) would incorrectly load chunks via the step-based fallback. The final remaining problem — the currentRun === null case where getChunksFromStep(N) returned chunks across all runs at the same step number — is now closed by the fallback removal.
A first attempt at the same fix (commit 5c4fbde42, included in MR !166 pipeline 1) was reverted because a Playwright test in examples/opennext-cloudflare-do failed during that CI run. Subsequent investigation showed the failure was NOT caused by the fallback removal: running the same Playwright test locally with the fix re-applied (on top of intervening main commits including round-5 review findings, v0.7 StepWrites unification, and providerExecuted persistence) passes reliably. The intervening main commits resolved a separate state- persistence path that had been masking the leak via consistent duplication on both live and refresh sides.
The deferred runtime-cloudflare cleanups identified during investigation (remove ?? 0 default in DOStateStoreClient.getCurrentRun; return startSequence for all non-null currentRun in DO /snapshot) are NICE TO HAVE for type clarity but no longer required — the ai-sdk-side safe-degradation closes the leak vector. Track those as separate follow-ups if/when someone touches that code.
Conformance suite 0a — follow-ups
Deferred triage items from the plan-0a whole-branch review. Items the 0a.1 plan ("Conformance Suite 0a.1 — Review Fixes and Driver-Contract Reshape", docs/superpowers/plans/2026-09-24-conformance-suite-0a1-review-fixes-and-contract.md) closed — and those the two post-completion review-fix passes (PR-1: harness code; PR-2: catalogue / CI / docs) closed — have been removed, each verified against the current code rather than taken on a plan's word (the plan document above and the fix commits' messages record what closed what). Each surviving line names the file the issue lives in and why it's deferred rather than fixed now.
- T1
packages/core/src/llm/scripted-model.ts— the gate/hold mechanism (whenHeld/release) tracks only one held record pergateId; a second concurrent hold on the same id silently overwrites the first's waiters. - T1
packages/core/src/llm/scripted-model.ts—getCallCountundercounts calls that resolved via a scriptedfailresponse; it's a public core API used outside conformance too, so this needs either a documented contract or a fix plus a semver note, not a quiet behavior change. - T1
packages/core/src/llm/scripted-model.ts— a call that's both pre-aborted (abort signal already set) and scripted tofailis classified'aborted'regardless of the scripted failure, which can mask a scenario bug that scripts a fail but never actually races the abort. - T2
packages/e2e/src/conformance/__tests__/docs.test.ts— the file-level check is stricter than its name suggests (it also asserts cross-file consistency, not just doc-file shape); rename or split it. - T3
packages/e2e/src/conformance/judge.ts—describeVerdictand the empty-ledger-entry rendering path still lack direct unit coverage (0a.1'sdrift.tsnow REJECTS a ledger entry with no assertions before it can ever reachjudge.ts, so the live risk is lower, butdescribeVerdict's own handling of that shape is still only exercised indirectly viasuite-lane.test.ts/cell.test.ts, notjudge.test.tsdirectly). - T4
packages/e2e/src/conformance/observation/persisted.ts—readPersisted's 10k-page pagination guard (for (let guard = 0; guard < 10_000; ...)) still falls through silently on the bound instead of raising a clearly-labeled "truncated" outcome; a session with more than 2,000,000 messages (200/page) would fall out of the loop and get judged by whatever the length-vs-getMessageCountcomparison happens to produce (usually aShortReadError, but with no indication the CAUSE was the guard, not a real corrupt-row skip). - T5
packages/e2e/src/conformance/judge.ts— a single root cause can produce duplicate "problem" lines (once per affected assertion id) instead of being collapsed into one line naming all affected ids. - T6
packages/e2e/src/conformance/drivers/executor-driver.ts(wrap) —outcome()conflates a throw from starting the call (get()rejecting) with a throw from awaiting its result; both collapse to the sameterminal: 'threw'shape, losing which phase actually failed. (0a.1'sRunOutcomereshape did not change this —thrownstill doesn't distinguish "never started" from "started and threw resolving the result".) - T8
packages/core/src/llm/scripted-model.ts(waitForAbort) — accepts and reportstruefor an abort signal that was already set BEFOREstop()was even called, which can make a stop-latency assertion pass vacuously. - T8
packages/e2e/src/conformance/findings.ts(CP-25, CP-26) — evidence provenance issource-only with alive: ...note embedded in the evidence string rather than a first-classlive-reproprovenance entry; worth reconciling once a live probe scenario exists for these. - T10 (generateStep / mock restore) — a mocked step generator is restored via
deleteon the module namespace rather thanvi.restoreAllMocks()/ re-assignment, which is fragile against import-order changes; needs the exact call site identified and normalized. packages/e2e/scripts/conformance-verify.ts/packages/e2e/src/conformance/verify.ts— the printed line index in a mismatch report doesn't correspond to the actual.jsonlline number (off by the header/merge step), and a result line with an unexpected/unknown key is silently ignored rather than surfaced as a schema drift.packages/core/src/llm/scripted-model.ts— two concurrent calls against the SAME unscoped.script()(global) queue can have their responses swapped if the queue is consumed out of call order; needs either per-call correlation or an explicit "no concurrent calls withoutgateIdorscriptFor" invariant. (0a.1'sscriptFor(sessionId, ...)fixed this for SESSION-scoped queues only — two interleaved sessions'scriptForresponses are proven not to swap — the shared global queue this item describes is untouched.)
New follow-ups from the 0a.1 plan (owning plan noted per item)
DONE (Phase B increment 1, 2026-09-26) —
DRIVER_CAPABILITIEShas per-channelobserve-persisted/observe-llm/observe-hooks/observe-stream/observe-client(plusobserve-host-statusandobserve-listed-count), every scenario'sneedslists the channels it reads, and the per-cell driver view (drivers/cell-driver.ts) makes an undeclared read aHarnessErrorand an unobservable channel a declared gap (authoring guide §2, "Observation channels are capabilities too"). Original item: 0bpackages/e2e/src/conformance/drivers/session-driver.ts/observation/observation.ts— observation CHANNELS are not driver capabilities:observe()is one capability, but a host/client driver may be able to observepersistedand notllm/hooks(or vice versa), and today nothing lets it say so. Add per-channelobserve-*capabilities (e.g.observe-persisted,observe-llm,observe-hooks) toDRIVER_CAPABILITIESand to each scenario'sneeds, so an unobservable channel makes the cell a DECLARED GAP instead of a crash or a vacuous pass (0a.1 final review M6).DONE (Phase B increment 1, 2026-09-26) — the host and client drivers (
makeHostDriverCfDo/makeHostDriverJsindrivers/host-driver.ts,makeClientDriverCfDo/makeClientDriverJsindrivers/client-driver.ts) are all built withdefineMakeDriver, anddefineConformanceSuiterefuses a lane whoseMakeDrivercapabilities differ from itslanes.tsentry. The cf-do agent install is a'harness'seam (the bundled agent registry, fingerprint-checked inprepare()); js-chat-host registers the wired agent on its chat-route map. Original item: 0bpackages/e2e/src/conformance/drivers/session-driver.ts— the host and client tier drivers plan 0b adds MUST be built withdefineMakeDriver(capabilities, fn), not a bare function assigned acapabilitiesproperty by hand —buildBackendInstancerejects a driver whose owncapabilitiesdisagree with itsMakeDriver's static set, anddefineMakeDriveris the only way to satisfy that invariant correctly (flagged as a concern in 0a.1's Task 3 fix-round-1 report). Also: follow the executor driver's SPLIT, not a blanket wrap —BackendEnv.wireAgent(pure harness wiring: hook recorder, test seams) fails as aHarnessError(:crash), whileBackendEnv.installAgentpropagates unwrapped and its meaning is declared where it is built —agentPreparation'sInstallKind. Use'product'ONLY for a real product registration API (Temporal'sAgentRegistry.replace(); in 0b e.g. agent-server'sagentsmap), so a typed rejection at registration is judged bycheckRejectionlike any other runtime answer. A harness seam (the CF DO module slot, the CFW test injection registry, a no-op) is'harness', and its failure is aHarnessErrorthat can never match a ledgered.rejected-typed(PR-1 item 4, PR-3 item 3; supersedes the older "don't copy the blanketprepare()wrap" note).0c
packages/e2e/src/harness/setup-helpers/cf-do-d1.ts/cfw-workflows-d1.ts— still silently ignoreSetupOptions(includingllmAdapter) entirely, unlike the JS/Temporal/DBOS helpers, which now all either honor or reject (rejectUnsupportedSetupOptions) every option (0a.1 Task 3). Plan 0c, which is what actually calls these helpers viaruntime-cloudflare's workflow pool, must either honorSetupOptionsor reject it the same way. Partly done (Phase B Task 3): the newcf-dobackend (setup-helpers/cf-do.ts, which replaced thecf-do-d1registry row) honoursllmAdapter(aScriptedModelonly) and rejects everything else viarejectUnsupportedSetupOptions; the legacy workerdcf-do-d1andcfw-workflows-d1helpers still ignoreSetupOptions. Done for every Phase B backend (increment 1, 2026-09-26):cf-dohonours or rejects every field, andjs-chat-hostdelegates tosetupJsMemory, which does too. What stays open is only the two legacy workerd-pool helpers, which no conformance lane uses; plan 0c (CF Workflows lanes) owns them. Plan 0c MR-1b: the CF Workflows conformance lane's new helper,setupCfwMiniflare(cfw-miniflare.ts), honoursllmAdapter(aScriptedModelonly) and rejects every other field viarejectUnsupportedSetupOptions. The legacycfw-workflows-d1.tshelper now backs thecfw-workflows-d1-poolrow. Plan 0c MR-2 kept it, because the runtime-cloudflare pool suites still need it (FU-CONF-CFW-POOL-ROW-RETIREMENT, still open, re-scoped to the missing scenario support). It still ignoresSetupOptions(it takes none), and no conformance lane calls it.0c
docs/superpowers/specs/2026-09-23-runtime-conformance-suite-design.md§11 /packages/core/src/llm/scripted-model.ts— plan 0c must first VERIFY whether a workerd isolate is actually shared and reused across a cell's calls the way the Node-poolinProcessControl()assumption relies on (a DO/Workflow eviction can reset module-level state). The SEAM is ready (0a.1 fix wave):ScriptedModelControlmethods may return promises,record()assigns the index atomically, every post-record mutation (gateId,abortedAt,outcome,endedAt) goes throughupdate()keyed bycallId, and the model has async twins (callsAsync,callsForAsync,remainingScriptedAsync,ready(); the executor driver'sobserve()already usescallsForAsync) — proven by a core unit test with an async, snapshot-storing control. What remains for 0c: implement the store/RPC-backed control itself; its own gate-signalling mechanism (gates are still in-process — a held call and therelease()/reset()that unblocks it may live in different isolates); andreset()isolation for a call whoserecord()/dequeue()is still in flight on an async control (today only safe once runs have settled). The same question applies toharness/recorded-hooks.ts'shooksbridge (0a.1 Task 4), which also currently assumes the agent runs in the SAME process as the harness. Resolved by the Node bridge (Phase B for cf-do; plan 0c MR-1b reuses it for CF Workflows): theScriptedModel, its gates and the hook recorder stay in Node (harness/cf-do/bridge-server.ts). Workerd runs aProxyModelthat sends each model call to the bridge, and hooks reach it ashook.fired, so no control state lives in a workerd isolate (plan 0c spec §3.1).0a.1-final-review
packages/e2e/src/conformance/drivers/executor-driver.ts(stopAndDrain) — skips (does not wait on) a run whoseRunHandlehasn't arrived yet whendispose()is called; the skip is loud (surfaces via poison/teardown failure, not silent), but it's still a gap in the drain guarantee worth closing. PR-1 widened the window slightly:wrapnow starts the call through one extra microtask (Promise.resolve().then(get), so a synchronous throw becomes athrewoutcome), so a handle arrives one tick later than before.0a.1-final-review
packages/e2e/src/harness/recorded-hooks.ts(wireAgentHooks) — the re-wire-detectionWeakSettag is on the hooks BUNDLE object; rebuilding an equivalent hooks bundle with fresh wrapper functions (rather than reusing the tagged one) bypasses the "already wired" check silently. Tag the wrapper FUNCTIONS instead, or the agent itself.0a.1-final-review
packages/e2e/src/harness/recorded-hooks.tsandpackages/e2e/src/harness/setup-options.ts— both importHarnessErrorfrompackages/e2e/src/conformance/types.ts, a layer inversion (harness/is meant to be a lower layer thanconformance/); moveHarnessErrorto a shared module both can import from without crossing that boundary.0a.1-final-review
packages/e2e/src/harness/__tests__/recorded-hooks.test.ts/packages/e2e/src/conformance/__tests__/executor-driver.test.ts— the recorder's "no hooks fired" and "two agents sharing one bundle" cases, andexecutor-driver's "prepare() called twice" case, are only weakly exercised (mockenv.wireAgent/env.installAgent, not the realwireAgentHookspath through a backend); worth stronger, more explicit tests (0a.1 Task 4 fix-round-1, deferred by instruction rather than fixed).0a.1-review-fix (PR-1)
packages/core/src/llm/scripted-model.ts— a failedScriptedModelControlcall is sticky and EPOCH-attributed: itsScriptedModelError.originnames the call, session and reset epoch, andready()reports a failure from an EARLIER epoch as "a leftover call of a previous cell". That keeps it loud and correctly labeled, but the cell that CRASHES on it is whichever cell runs next on that model (the rebuild its crash triggers then clears it), not the cell that owned the leftover call — so that cell's own verdict may already have been recorded as a pass. PR-3 narrows this: a cell whose OWN driver-started runs are still unfinished at its end (CellRun.leftRunning, read throughSessionDriver.runs) now triggers a rebuild, so the next cell gets a fresh model, and a control call made during the leftover grace is caught by that cell's ownready(). What remains is a call from work the driver does not track (e.g. a runtime-spawned child run outliving its parent's result). Attributing that back to the owning cell would need the lane to hold results until the next epoch'sready(); deferred.0a.1-review-fix (PR-1)
packages/e2e/src/conformance/drivers/executor-driver.ts/cell.ts— aninstallAgent(product registration) failure is judged only on the rejection path (checkRejection). On a POSITIVE path it has no named id to land on, so the cell crashes (<scenario>:crash): loud, but UNLEDGERABLE — a runtime whose registration starts failing for a supported agent can't be ledgered as a finding without a registration assertion id. Consider a synthetic-but-ledgerable<scenario>.installedid.0a.1-review-fix (PR-1) — FIXED in PR-2, recorded for context:
heldOrEnded(scenarios/stop.ts) leftwhenHeld's timer (up to 2 ×HELD_BOUND_MS= 60 s) pending after the run's outcome won the race.ScenarioModel.whenHeldnow takes an optionalAbortSignalthat clears the timer and drops the waiter, andheldOrEndedaborts it once the race is decided (corescripted-model.test.ts, e2estop-scenario.test.ts).0a.1-final-review
packages/e2e/scripts/benign-teardown.mjs(STRICT_KNOWN_RACE_HEADLINE_PATTERNS) — neither anchor has been verified against a genuine CI capture of the race firing end-to-end (4 live attempts across 0a.1's Task 7 fix rounds, all clean). Capture the REAL DBOS "pool-after-end" race and the REAL TemporalIllegalStateErrorteardown rendering from CI the first time either fires, add them as strict-forgive samples. (The header comment's old "miniflare D1" attribution was corrected in the 0a.1 fix wave: the literal string is thrown by@temporalio/worker'sconnection.tsas anIllegalStateError, not a plainError; the strict anchor was sourced from an existing hand-built test sample, not a real capture, and may need to move to theIllegalStateError/"...to it"form once a real one is seen.) Until then the predicate fails closed by design, which is the safe direction, not a bug.0a.1-final-review
packages/e2e/src/__tests__/helpers/vitest-teardown.ts— itspkill -f workerdteardown runs UNSCOPED (kills everyworkerdprocess on the machine, not just this run's own children) and is therefore still NOT wired into any config'sglobalSetup(0a.1 Task 7 reverted an earlier, unsafe activation attempt). Scope it to the run's own child processes (tracked PIDs, or a run-specific marker) before enabling it for real.0d (repo-wide cleanup) the dead
forceExit/globalTeardownvitest config keys 0a.1's Task 7 found and removed frompackages/e2e's configs exist, equally silently no-op, in ~13 other packages (grepped forforceExit/globalTeardownacrossvitest*.config.ts):packages/agent-server/vitest.integ.config.ts,packages/ai-sdk/vitest.integ.config.ts,packages/memory-cloudflare/vitest.integ.config.ts,packages/memory-redis/vitest.integ.config.ts,packages/runtime-cloudflare/vitest.integ.config.ts,packages/runtime-dbos/vitest.integ.config.ts,packages/runtime-temporal/vitest.integ.config.ts,packages/store-cloudflare/vitest.config.tsandvitest.integ.config.ts,packages/store-postgres/vitest.integ.config.ts,packages/store-redis/vitest.integ.config.ts,packages/tracing-langfuse/vitest.integ.config.ts, and the rootvitest.config.ts. Same fix shape aspackages/e2egot: dropforceExit; rename aglobalTeardownfile's export fromdefaultto a namedteardownand wire it viaglobalSetupwhere a real teardown file exists.(untriaged — environmental, unverified)
examples/research-assistant-cloudflare-dohas at least one red Playwright/workerd test attributable to the same Miniflare temp-dir path-length limit documented indocs/dev/development-workflow.md("Sandbox / environment gotchas":mkdirat: File name too long, triggered by a deeply-nested checkout path) rather than a real product defect. Not yet confirmed whether it is purely environmental (this session's sandbox path depth) or reproduces from a normal-depth checkout too; needs a run from a short path to settle it before it's either fixed or written off.
Conformance Phase A — follow-ups (not findings)
Consumer-code / docs observations surfaced while re-verifying the Phase A bucket-prep catalogue (2026-09-24-conformance-phase-a-bucket-prep). None of these are catalogue findings (they're not framework defects reachable through the conformance suite's driver surface) — they're tracked here so they don't get lost.
FU-CONF-OPENNEXT-DO-ID-PREFIX: the opennext-cloudflare-do example uses two DO-id conventions — open
examples/opennext-cloudflare-do/src/lib/agent-client.ts:28 and brief-agent-client.ts:18 (+ README.md:431) derive the DO id via idFromName('session:' + sessionId). The chat handler's own clients derive it via idFromName(sessionId) (no prefix) — packages/runtime-cloudflare/src/do/do-clients.ts:97, 342, 816 — so the example's debug/admin routes built on getDOStub read an empty DO for a session created through the normal chat path. packages/runtime-cloudflare/src/do/do-executor.ts:383, 427, 455, 508, 535 uses the session: prefix convention too — so the runtime-cloudflare package itself carries two conventions, not just the example.
Owner: examples/opennext-cloudflare-do maintainers (align getDOStub's idFromName prefix with do-clients.ts, or document why the two conventions are intentionally different).
FU-CONF-EXAMPLES-RESUME-ONLY-ACTIVE: seven example ChatClients resume only on status === 'active' — done
Done (conformance owner, 2026-10-16): S5 (bd26dc9b57) moved the example ChatClients to the product's snapshotResumeOptions (opennext-cloudflare-do's through src/lib/use-snapshot-chat.ts), which resumes while the snapshot is active or paused; no example ChatClient gates on 'active' alone any more. The conformance client harness, which reproduced the old gate, mirrors the S5 page since the same owner MR.
examples/opennext-cloudflare-do/src/app/{chat,coordinator,brief-flow,reasoning}/[sessionId]/ChatClient.tsx (lines 40, 36, 38, 39 respectively) and examples/nextjs-redis/src/app/{chat,reasoning,brief-flow}/[sessionId]/ChatClient.tsx (lines 131, 91, 94 respectively) all gate shouldResume on initialSnapshot.status === 'active' only, so a paused session (suspended on a client tool / HITL approval) never reattaches its stream on page reload — the user sees a stalled page until they act blind or navigate away and back. examples/resumable-streams-nextjs/src/app/chat/[sessionId]/ChatClient.tsx:111 does it right: status === 'active' || status === 'paused'.
Owner: examples/opennext-cloudflare-do and examples/nextjs-redis maintainers (port the resumable-streams-nextjs pattern to the other seven ChatClients).
FU-DOCS-SUSPENSION-CONTEXT-FIELD: the guides name a SessionState.suspensionContext field that does not exist — open (LOW; docs)
Filed by: the CP-11 cell (WA). Core's SessionState has no suspensionContext field: a suspended call is a pending entry (pendingClientToolCalls), an answer is that entry's submittedResult (an approval's is { approved, reason }, runtime-js applySubmitToOwner), and the suspension is the run's suspension commit. The only code use of the name is the CF DO store's internal SQLite column suspension_context (do-state-store.ts, buildSuspensionContext). Wanted: reword docs/guide/approval-gates.md (the deny semantics' step 1, and the two later mentions), docs/guide/client-executed-tools.md (its overview, the shared-primitives paragraph and the DBOS note) and docs/guide/state-stores.md (its intro and the persistence list) to name the pending entries and the suspension commit instead.
FU-DOCS-CFW-STREAM-MANAGER-NAME: the CF Workflows and storage guides import a store-cloudflare DOStreamManager that does not exist — open (LOW; docs)
Found by: the CP-337 trace (2026-11-19), which followed the guide to the stream manager a CF Workflows deployment uses. @helix-agents/store-cloudflare exports DurableObjectStreamManager, constructed as new DurableObjectStreamManager({ streamNamespace }) (its index.ts example). DOStreamManager is runtime-cloudflare's in-process manager for the CF Durable Object, a different class.
- Where:
docs/runtimes/cloudflare-workflows.mdimportsDOStreamManagerfrom store-cloudflare (:205,:301) and constructsnew DOStreamManager(env.STREAM_MANAGER)(:216,:238,:315,:1410);docs/storage/index.mddoes the same (:401,:404) and names it in the store table (:39). Line numbers at origin/main fae53b26ac. - Wanted: the class name and constructor store-cloudflare exports, in both guides, and the table row naming
DurableObjectStreamManagerfor the Workflows path.
FU-CONF-RESUMABLE-STREAMS-GUIDE: docs/guide/resumable-streams.md contradicts itself — open
- Lines 113 vs 402:
resume: snapshot.status === 'active'vsinitialSnapshot.status === 'active' || initialSnapshot.status === 'paused'for what is presented as the same recipe. - Lines 149 vs 859: content replay is documented as defaulting to
false, then later ascontentReplay: { enabled: true }(default) — a shape that does not exist in the code; the real knobs aredeps.contentReplayEnabled(on the deps bundle) and thehandleChatStreamparamcontentReplay: true(a boolean, not an{ enabled }object). - Lines 450 vs 453-459: content replay is described as opt-in, then a few lines later as something you "disable globally" — inconsistent framing of the same default.
- Lines 352-357: the v6
HelixChatTransportclass really is gone (new HelixChatTransport(...)does throw), so the callout isn't factually wrong about that. The actual problem is the heading and the advice: "was deleted in v7" reads as "the whole thing is gone," but anexport interface HelixChatTransportand anexport function createHelixChatTransportproducing that same shape still ship atpackages/ai-sdk/src/transport/helix-chat-transport.ts:57, 93— and the callout tells every reader to migrate toDefaultChatTransport+prepareHelixChatRequest/prepareHelixReconnectRequestinstead, without mentioning that a much smaller migration (swapnew HelixChatTransport(...)forcreateHelixChatTransport(...), same shape) is also available for whoever was using it as a transport, not as the in-framework chat-handler recipe.
The HelixChatTransport bullet is resolved: !944 removes createHelixChatTransport, and the guide no longer carries the callout (none at f0abcc475b or fc98feff5a). The other three bullets are not re-checked here.
Owner: docs maintainers (docs/guide/resumable-streams.md) — reconcile the two resume-gate recipes, correct the content-replay knob names/defaults, and reword the "HelixChatTransport was deleted" callout so it doesn't read as "there is no replacement" when createHelixChatTransport still ships.
FU-CONF-DBOS-RESUME-REPLAY: hypothesis for bucket B1 — NOT a finding — done (premise removed by W1 MR2b)
W1 MR2b removed the code this hypothesis is about. handleChatStream's auto-resume race branch, its resumedRun?.startSequence guard and Phase C's same-run inference are deleted: the resume path acts on the submit's continuation. DBOS answers new_run (its in-workflow continuation run), and the handler attaches from the stream head read BEFORE the submits (never from 0: an unreadable head is a typed error), so the continuation does not replay the whole run. The CF DO's new_run carries the wake's own startSequence when its record was read back, else the handler attaches from the pre-submit head. If bucket B1 still wants a client-tier cell for "a DBOS continuation renders once", file it against MR2b's head.
packages/ai-sdk/src/handler/handle-chat-stream.ts:772-773 documents that DBOS never reaches the auto-resume race branch (runtimes that auto-resume on submit take a different, earlier success path). But packages/runtime-dbos/src/lifecycle/resume.ts:152-192 (case (a), a CAS failure because the session is already 'active' AND pendingClientToolCalls is non-empty) returns a handle to the EXISTING workflow — still blocked on DBOS.recv, in the SAME run — rather than starting a new one, which is exactly the "continues in the same run" shape the cited comment assumes DBOS doesn't have. (Case (b), same 'active' check but no pending client tools, throws AgentAlreadyRunningError at resume.ts:304; a status that is neither 'active' nor 'interrupted' / 'paused' — e.g. 'completed' — falls through to a THIRD, separate branch outside the 'active' check, resume.ts:307-311, which throws Session "<sessionId>" is not in a resumable state (current: <status>); neither of those two throwing branches is the mechanism this hypothesis is about.) Because DBOS continues after DBOS.recv in the SAME run, startSequence is that run's own start, and streamLiveResponse may replay the whole run into the existing message id. A DO variant of the same risk: packages/runtime-cloudflare/src/do/do-clients.ts:303 defaults startSequence to 0 when the run row is no longer 'running', so the resumedRun?.startSequence !== undefined guard at handle-chat-stream.ts:799 passes trivially with 0. Bucket B1 must probe both of these on the client tier before this becomes a catalogue finding; this entry exists so the reproduction isn't lost in the meantime.
Owner: Phase B bucket B1 (host/client-tier conformance probe).
FU-CONF-DO-WOKEN-RUN-PAUSED-WINDOW: the woken-run ownership exit must not trust paused on the DO — open (Phase B)
The executor responder ends its ownership of a run woken by a submit (clientToolSubmitWakesRun) once that run was seen active and is now any non-active status, including paused with the answered entry still pending (packages/e2e/src/conformance/drivers/executor-driver.ts, awaitOwnedRunSettled). That is sound on DBOS, which never writes paused for a live run. On the CF DO it is not: ensureExecutionContinuesImpl (packages/runtime-cloudflare/src/do/durable-object-agent-base.ts ~736-829) CASes a stranded active session to paused and then resume()s it, so a live, self-recovering run passes through paused. A poll inside that window would end ownership early, and the run could leak into the next cell. No lane exercises this today (the cf-do adapter is not in the Node matrix). Before Phase B wires a cf-do lane with this responder, make the end condition per-backend (e.g. a declared pausedMayBeTransient, or waiting for a terminal status on the DO) and add a test for the reset-to-paused window.
Blocks enabling the executor-tier responder on cf-do (Phase B Task 3). The cf-do backend (packages/e2e/src/harness/setup-helpers/cf-do.ts) declares clientToolContinuation: 'auto' and clientToolSubmitWakesRun: true from source, but it has no executor-tier lane: its executor refuses every call with a HarnessError, so the responder never runs there. Resolve this item before giving cf-do an executor lane (or any lane that uses the woken-run ownership exit).
FU-CONF-DO-MINIFLARE-ALARM-ACROSS-EVICT: under Miniflare dispose+recreate, an evicted DO's durable alarm does not wake it — partly resolved by Miniflare 5 (the alarm now fires at its deadline); the on-demand fire's differences remain (Phase B)
Under Miniflare 3 (history; see the Miniflare 5 update below): a running DO arms a durable alarm about 30s ahead (DOAB interrupt-poll / execution-watchdog subscribers, packages/runtime-cloudflare/src/do/durable-object-agent-base.ts ~237, ~869-883, ~938-952). In production that alarm survives an eviction and wakes a fresh instance, whose onStart runs wake recovery. The cf-do conformance backend evicts with mf.dispose() + a new Miniflare over the same persist dir, and there the alarm does NOT fire on its own. Measured in packages/e2e/src/harness/cf-do/__tests__/persisted-remote.integ.test.ts ("the durable alarm across an eviction"): the alarm was armed about 30.1s after /start; after the eviction, with no request, the DO stayed asleep until 10s past that deadline; the next request woke it. Whether the stored alarm value itself survives is not observable this way, because the waking request's onStart recovery re-arms the alarm before any read. Consequences: (a) on cf-do only a request (or an explicit alarm fire) wakes an evicted DO, so wake scenarios (Task 7 wake(), Task 8 materialize.wake-hooks / MA-09) must wake it explicitly and cannot rely on the alarm; (b) the product's alarm-driven wake path is not reproduced by the harness. To reproduce it, add a /__conformance/fire-alarms route that calls the DO's real onAlarm() (spec §3 "Wake = the next request, or an explicit alarm fire"), and note which wake each scenario uses.
Task 7 update: how the harness covers the alarm path now. driver.wake(sid, { via: 'alarm' }) (BackendEnv.wake, packages/e2e/src/harness/setup-helpers/cf-do.ts wakeDo) POSTs the worker's entry route /__conformance/fire-alarms/:sid, which calls ConformanceAgentDO.fireAlarmsForConformance() by DO RPC (packages/e2e/src/harness/cf-do/worker/conformance-worker.ts). That method reads ctx.storage.getAlarm(), deletes the alarm (workerd consumes an alarm when it invokes the handler), and calls PartyServer's own alarm(): the entry workerd calls, which runs onStart (DOAB's 'wake' recovery) on a fresh instance and then DOAB's onAlarm() ('watchdog' recovery + the scheduler's reschedule). Nothing is reimplemented. It is RPC and not a DO fetch because PartyServer's fetch() runs onStart before any route handler, which would make every alarm wake a request wake first. Measured live (packages/e2e/src/harness/cf-do/__tests__/timing-controls.integ.test.ts, host and client drivers): the stored alarm DOES survive the dispose+recreate — the fire reads exactly the value armed before the eviction, proving nothing re-armed it first — and firing it re-drives the stranded run to completion. With no alarm armed, nothing is fired (fired: false) and the handler does not run. via: 'request' (the default) is a real product request: the chat snapshot route (the page load).
Which recovery an alarm wake exercises (fix round 1). WakeResult.instance reports it. After an eviction the fire meets a fresh instance ('fresh'): PartyServer's alarm() runs onStart → recoverStrandedExecution('wake') — the SAME hook-less recovery a request wake runs — and onAlarm's 'watchdog' recovery then has nothing to do. So an evicted-DO alarm wake is not independent coverage of the 'watchdog' recovery; that path is reached only by firing on a WARM (non-evicted) DO ('warm', proven live in timing-controls.integ.test.ts). Task 8 (materialize.wake-hooks, MA-09) must not claim alarm-path ('watchdog') coverage for an evicted-DO wake.
Miniflare 5 update (2026-09-27, main 186b0c6a48 bumped miniflare 3 → 5.20260925.0-alpha). The persisted alarm now DOES fire on its own after dispose + recreate, at its deadline, as in production. Measured in persisted-remote.integ.test.ts ("the durable alarm across an eviction", now pinning this; 3/3): with no request, the recreated DO stays asleep until 1s before the deadline, then the alarm alone wakes it, onStart's wake recovery completes the stranded run, and the run is completed. So "the DO woke by itself" is now observable on cf-do. It also means a wake scenario must wake an evicted DO well before the ~30s deadline, or the alarm runs the recovery instead. materialize.wake-hooks wakes within seconds, and the evict/wake self-tests check the DO is still asleep first.
What still differs from production (why this stays open): (1) resolved by Miniflare 5 (above); (2) the harness fires it on demand, not at its deadline (time is compressed, so behaviour that depends on the alarm arriving ~30s later is not reproduced); (3) a throwing handler is reported (handlerError), not retried with workerd's backoff; (4) the handler gets no alarmInfo (retryCount / isRetry; PartyServer's alarm() takes none). Closing this needs Miniflare/workerd to deliver persisted alarms after a restart, or a different eviction model (spec §10).
FU-CONF-CFDO-HOLD-DEADLINE-ALARM: no cf-do control holds the DO's client-tool deadline alarm, so the CF DO lane cannot keep a lapsed call pending — open (harness + a small product test seam; owner: the conformance owner)
The JS lanes keep a lapsed client call pending with 'hold-client-tool-deadline' (packages/e2e/src/harness/client-tool-deadline-holds.ts): runtime-js takes an injectable clientToolDeadlineTimer.schedule, and the harness swaps in its hold set. The CF Durable Object has no such seam. Its deadline is the client-tool-deadline subscriber on the DO's PRIVATE alarmScheduler (packages/runtime-cloudflare/src/do/durable-object-agent-base.ts, the field and its registerSubscriber call), and Miniflare fires the stored alarm at its deadline. So the alarm expires a lapsed call by itself before any request can meet it. The only harness-side lever is overriding the conformance DO's whole alarm(), which would also hold stop-deadline, the watchdog and parent-notice, changing product timing beyond the deadline.
Consequence: these cells are declared gaps on cf-do, and the CF DO is the lane SDK-1's incident happened on (the 2026-10-17 chat incident, a DO replacement, then a reload POST carrying the late real answer):
chat.lapsed-call-real-answer-kept(SDK-1);- CP-163's
chat.send-over-lapsed-pending; - the
lapsed-pending-callcells.
WA's !548 has a DO-side test of the incident path meanwhile.
To close: add a DO test seam that holds only the deadline subscriber's fire, the DO's equivalent of runtime-js's schedule hook. For example, a DurableObjectAgentConfig option, or an injectable compute / fire wrapper for that one subscriber. Then give the cf-do host lane 'hold-client-tool-deadline' and 'lapsed-pending-call' over it, with a test:conformance-harness self-test, and run and ledger the cells above on cf-do.
FU-CONF-TRUE-MID-TEXT-HOLD: no scenario can hold a model call in the middle of a text part — open (Phase B; the hold primitive landed 2026-09-27)
The Phase B spec §4 timing controls list driver.reload(sessionId) "mid-text". That is not delivered yet. ScriptedModel's hold (packages/core/src/llm/scripted-model.ts) blocks BEFORE the call streams anything, so no step can stop after some of its text deltas. The Task 7 self-test ("reload mid-message (tool-gated)", packages/e2e/src/harness/cf-do/__tests__/timing-controls.integ.test.ts) streams a step's text and holds the run in a tool gate instead, so the reload lands mid-MESSAGE, between steps. (Until DI-20's fix, !268, the client also errored on the product's finish-step frame before the reload. Since that fix the first instance ends client:unmounted and the reloaded instance resumes an intact stream, but still only between steps.) To close: add a ScriptedModel hold that fires after N text chunks of a response (streaming the first N, then blocking until released), then a true mid-text reload test and scenario.
Progress (2026-09-27). The hold exists: holdAfterDeltas(n, gateId, then) (core ScriptedModel, timing control B in docs/dev/conformance-scenario-authoring.md "Timing controls"). It pauses after exactly n text deltas on every backend that runs the ScriptedModel, cf-do included (the bridge's delivery barrier makes whenHeld mean the deltas reached the DO). The live self-test (harness/cf-do/__tests__/hold-controls.integ.test.ts) shows the caller holding exactly the partial text until release on cf-do and js-chat-host, host and client tiers. What remains is the scenario side: a true mid-text reload (S5's resume.replay-exact mid-text cursor).
Owner: the conformance owner.
FU-CONF-DI20-MASKED-CLIENT-CELLS: re-attribute the client-tier submit/smoke cells when DI-20 is fixed — done (Phase B, 2026-09-27)
At the client tier the product Chat rejected the finish-step chunk (DI-20) before anything else those scenarios judge happened, so smoke.hello-completes|{cf-do,js-chat-host}|client, submit.race-a-client|cf-do|client and submit.no-race-client|{cf-do,js-chat-host}|client were ledgered to DI-20.
Resolution. DI-20's fix (!268, 0e618772ee) landed on main first. The Phase B increment-1 branch rebased onto it, re-ran the cells and re-attributed every id from the observed run (identical 5/5):
smoke.hello-completes(both backends) andsubmit.no-race-client|js-chat-host|clientnow pass. DI-20's record cites them as corroboration; itsprovenByis only the 4stream.ui-chunk-schema{,-client}cells, which require step frames.submit.race-a-client|cf-do|clientnow reaches the host. With.submittedand.window-openpassing, it fails the same five ids as the host cell (the auto-send gets onlydata-resume-rejected, and the run is strandedpaused), so it is ledgered to CP-52.submit.no-race-client|cf-do|clientfails.submit-acceptedand.continuation-reaches-caller(data-resume-rejectedwhile the woken run completes), so it is ledgered to CP-53.
DI-20 is fixed.
FU-E2E-DBOS-PERSISTENT-IDLE-RESTART-FLAKE: duplicate of CP-74 (a product defect, not a test flake) — closed
packages/e2e/src/__tests__/multi-message-execute-dbos-persistent.integ.test.ts › "idle-timeout exit then new message starts a fresh persistent workflow" fails intermittently at :436, which counts USER messages (expected [ { role: 'user', …(1) } ] to have a length of 2 but got 1). The second user message is lost. That is CP-74 (catalogued in MR !267): the idle-timeout exit marks the session completed while the DBOS workflow is still PENDING in its finally, so execute() routes a send into the exiting workflow, and the message stays unconsumed in dbos.notifications. It is not a test race; an earlier version of this item said so, wrongly. Phase B Task 9 saw it on origin/main 59d47f1e0f (1 of 3 isolated runs) and on conformance-phase-b-do-slice (which changes nothing it imports). On MR pipelines the test:integ:e2e job's retry on script_failure (.gitlab-ci.yml, .integration_base) hides it. Track the fix under CP-74 (and FU-DBOS-PERSISTENT-SEND-INTO-EXITING-WORKFLOW).
FU-INFRA-TURBO-STRICT-ENV: turbo run test:integration drops HELIX_TEMPORAL_BACKEND / RUNTIME_DBOS_INTEG — closed (duplicate of FU-CI-TURBO-ENV-PASSTHROUGH, fixed by !272)
Phase B Task 9's DoD matrix (2026-09-26) found that turbo's strict env mode dropped RUNTIME_DBOS_INTEG / HELIX_TEMPORAL_BACKEND / TEMPORAL_ADDRESS from test:integration. Before the fix, a local pnpm run test:integration ran the runtime-dbos integration suite gated off (0 tests) and pointed Temporal at localhost:7233. The same defect was fixed on main as FU-CI-TURBO-ENV-PASSTHROUGH (MR !272, 186b0c6a48): globalPassThroughEnv in turbo.json, guarded by pnpm run check:turbo-env. --env-mode=loose is no longer needed.
FU-CHAT-RESUME-FALLBACK-SINGLE-READ-TIMING: the chat handler's resume fallback judges a submit's continuation from one state read — open
- Where: ai-sdk
handler/handle-chat-stream.ts,handleResumePath's catch (:925-939at WA MR1 dbf1d82406). After the submit, when the explicit/resumeloses to a runtime's own auto-resume, the handler reads the session ONCE: it reattaches to the continuation only if the submitted call is no longer pending, and otherwise answersdata-resume-rejected(or the typed error). - Why it matters: CP-53 was this read happening before the woken run had consumed the call (the CF DO answered the submit before its continuation entered). WA MR1 closed CP-53 on the DO from the runtime side: the DO now answers the submit only after the continuation's entry has started. The handler still depends on that timing, so any runtime or host that answers a submit before its continuation entered reproduces CP-53. CP-104 (CF Workflows, open) is the same read with nothing resuming the run.
- Fix direction: make the handler's decision independent of runtime timing, for example a bounded, fenced wait for the continuation's run (or for the call to leave
pendingClientToolCalls) before choosing reject over attach, or a runtime-reported "continuation entered" signal. Prove it with a conformance cell where a host answers the submit before its continuation enters. - Owner: WC (the chat delivery contract). Filed by the conformance owner in WA MR1 Task 20 (orchestrator ruling).
FU-CONF-RACE-A-RELEASE-ON-HEADERS: race A's hold bracket releases on the submit's response headers — withdrawn (WA MR1 Task 20)
Withdrawn (WA MR1 Task 20, conformance owner): race A (submit.race-a*) is retired. Its window, the DO still executing the paused run inside a held host onComplete, no longer exists: WA MR1 delivers the settle hook (onRunSettled) after the execution release (runtime-cloudflare do/drive/settle.ts), and nothing between the paused frame and the release awaits. With no bracket left, nothing can deadlock against it. The original entry follows.
submit.race-a holds the DO in onComplete and releases it once the auto-send is answered (.submit-returned, packages/e2e/src/conformance/scenarios/submit.ts: responded is true when the response status arrives). A CP-52 or CP-53 fix that withholds the auto-send's response until the DO is idle would deadlock against that hold: the release waits for the response, and the response waits for the release. It fails visibly (a .submit-returned timeout), never silently. If a fix MR hits it, change the bracket, not the fix: release when the request has reached the host (for example a bridge-recorded arrival of the handler's /resume or submit call), and keep the window proof (isInside + isExecuting). Recorded in CP-52's record and the authoring guide's race A section. Also for that MR: race A's caller-side ids must be re-attributed from the observed run before submit.race-a|cf-do|host can appear in CP-52's provenBy (CP-52's conformance: notes).
Owner: the conformance owner (or the CP-52 fix MR, with the owner's review).
FU-CONF-CLIENT-SESSION-REUSE-FENCE: only cf-do enforces single-use session ids — open
The cf-do bridge refuses traffic for a session a previous cell closed (bridge-server.ts, THE CELL CONTRACT step 5). js-chat-host has no such fence; it relies on cellId-prefixed session ids. A scenario that hard-codes a session id would therefore leak state across cells there without a HarnessError. To close: record the session ids each js-chat-host cell used (its composed cellContract, cellContractFor in suite.ts) and refuse a later cell's request for one of them.
Owner: the conformance owner.
FU-CONF-CP62-DBOS-SETTLE-TIMING: stop.mid-tool-honors-abort|dbos-postgres|executor had two outcome shapes — resolved (Phase B final fix wave, 2026-09-26)
The cell was nondeterministic, with both shapes caused by CP-62. Usually stop() landed after both of the step's tools (slow, held on its gate; note, instant) had started, and the outcome never settled within 30 s (.outcome-settles and its dependents failed; the ledgered shape). Sometimes the note tool's DBOS step had not started yet when stop() cancelled the workflow: that step then threw "Workflow … has been cancelled" through its three retries (1 s, 2 s backoff) and the run failed after ~3 s, so a different set failed (.outcome-interrupted, .transcript-paired, .continuation-completes). Seen in CI job 16756806141 (MR !271): https://gitlab.com/helix-ai/open-source/agents/-/jobs/16756806141. It was not the settle bound straddling retry exhaustion (that cell took 3.7 s), so no bound change could fix it.
Fix: .tool-reached (packages/e2e/src/conformance/scenarios/stop.ts, both stop.mid-tool-* scenarios) now waits until slow is running AND note has run before calling stop(), so the stop always lands at the same point of the step. The assertions are unchanged. Every runtime runs a step's tools in parallel, so this is reachable everywhere. CP-62's fragments were re-checked from fresh runs on a PRIVATE Postgres/Redis (no other session's load on the database; host load average ~40–160 from other sessions): with this change stop.mid-tool-honors-abort|dbos-postgres gave the ledgered 6-id set in 16/16 runs and stop.mid-tool-ignores-abort|dbos-postgres its ledgered 3-id set in 15/16; the pre-change commit gave the usual shape 10/10 locally (the retry-exhaustion shape was CI-only). The three Temporal cells were also identical 5/5 (the same timing applies to them; unchanged ledger). The ledger needed no change.
Residual (not the CP-62 shape; loud, never silent): in 1 of 16 private runs, the FIRST DBOS launch on a brand-new database, the lane rebuild after the stranded honors-abort cell could not stop the run within TERMINATE_BOUND_MS (20 s; the DBOS handle's abort() = cancelWorkflow + status CAS), so the backend was poisoned and the next cell reported :crash (GATED). It did not recur in 20 further warm runs (10 on the final head, 10 on the pre-change commit). If it shows up in CI, look at DBOS cold-start latency against that bound before touching the scenario.
FU-E2E-CROSS-SERVICE-BUNDLES-IN-SRC: leftover cross-service bundles make the e2e typecheck run out of memory — open (pre-existing on main)
packages/e2e/src/cross-service/harness.ts (~92) writes each test run's esbuild output to mkdtemp(src/cross-service/dist/bundle-*), and runs leave those directories behind. packages/e2e/tsconfig.json includes src/**/* with allowJs, so a later tsc --noEmit type-checks every leftover consumer.mjs / producer.mjs. After a local test:integration, the e2e typecheck needs ~6.5–8.8 GB and dies at Node's default heap (JavaScript heap out of memory, exit 134). Reproduced on origin/main 59d47f1e0f with the same leftovers (26 bundle dirs: 8.8 GB, rc=134); without them main needs 3.6 GB and this branch 4.0 GB. CI is not affected (fresh checkouts). Found in Phase B Task 9's final fix wave (2026-09-26). To close: write the bundles outside src/ (the OS temp dir, or a gitignored .cross-service-dist/) and/or add src/cross-service/dist/** to the tsconfig exclude, and remove each run's bundle dir in the harness teardown.
Owner: the cross-service remote-agents area owner (unassigned; reported to the reliability orchestrator).
FU-CONF-PHASE-B-SPEC-EXECUTOR-RESPONDER: Phase B spec §4 predates the executor-tier responder — done
Done (Phase B increment 1, 2026-09-26): Phase B spec §4 now opens with "Executor-tier responder already landed" (MR !258), and its §11 "As built" records the host and client implementations of the same option.
The conformance harness-API additions (branch conformance-harness-api, plan docs/superpowers/plans/2026-09-25-conformance-harness-api-additions.md) landed the executor-tier client-tool responder (StartOptions.clientTools, the 'submit-tool-result' driver capability) and the BackendEnv.clientToolSubmitWakesRun declaration (with its wake wait and woken-run wait — CP-64) EARLY, ahead of Phase B. Phase B spec §4 should note that both already exist at the executor tier. The spec lives on branch conformance-phase-b-do-slice; update it there on rebase.
Owner: the conformance owner (Phase B spec, on rebase of conformance-phase-b-do-slice).
FU-CONF-CF-BRANCH-ADAPTER-CRASH: smoke.branch-completes crashes on the Cloudflare executor lanes — ✅ closed (plan 0c MR-2, !307)
Closed (plan 0c MR-2): smoke.branch-completes|cfw-workflows-d1|executor passes in all 5 runs. The lane declares branching, so the branch reaches the product's own path with no harness adapter in between. cf-do: closed by absence — no cf-do executor lane exists; a future one must reach the product's branch path. No other Cloudflare executor lane exists: setupCfDo refuses the executor tier, and no lane runs over the legacy cf-do-d1 or cfw-workflows-d1-pool adapters. So the crash can no longer occur on any lane. The text below is the history.
Plan 0c MR-1b: the CF Workflows executor lane no longer uses a hand-rolled adapter. No branch cell has run on the lane yet: it is not-yet-run until plan 0c MR-2, whose first runs show whether smoke.branch-completes passes. Its cfw-workflows-d1 row is the real CloudflareAgentExecutor (setupCfwMiniflare) and declares branching (CFW_CAPS), so a branch reaches the product's own path. The hand-rolled adapter kept as the cfw-workflows-d1-pool row declares no branching (CFW_POOL_CAPS), and no executor lane runs over it or over cf-do-d1. What follows describes the state before MR-1b.
No Cloudflare backend declares the branching product capability, so on cf-do-d1 / cfw-workflows-d1 the runner substitutes smoke.branch-completes' REJECTION path, which calls driver.branch(...). The hand-rolled CF harness adapters refuse options.branch with a HarnessError (rejectBranchInHarnessAdapter, packages/e2e/src/harness/setup-options.ts — deliberately, so a branch is never silently run unbranched), and a HarnessError is never judged: the cell crashes (smoke.branch-completes:crash) instead of observing the product's typed framework_not_supported. This must be resolved before any CF executor lane runs: Phase B replaces the adapters with real drivers that reach the product's own branch path and surface its typed rejection. The CF lanes do not run in the executor matrix today, so no cell is affected yet.
Owner: the conformance owner (Phase B CF drivers).
FU-CONF-CP64-JUDGE-REASON: the judge matches failed ids, not failure reasons — open
A ledgered expected failure is matched by assertion ID only (judgeCell), so smoke.client-tool-completes.completes on dbos-postgres would stay expected-fail (CP-64) even if it started failing for a DIFFERENT cause. By design the judge compares ids; the cause is carried in the failure message instead (.completes renders the outcome with describeOutcome, incl. thrown.message — e.g. "Agent session … is already running"), so a reviewer can see it in the cell output. Revisit if the burn-down needs reason-level matching (e.g. an optional per-ledger-entry message pattern checked by conformance:verify).
Owner: the conformance owner.
FU-CONF-HOLD-CHECK-HIDES-UNDELIVERED-ABORT: a cell-end hold crash may be a product's undelivered abort — open (triage rule, non-blocking)
Since MR !282, runCell fails any cell that ends with a ScriptedModel call still blocked in a scripted hold (a plain hold or a paused holdAfterDeltas, via ScriptedModel.outstandingHolds()). The failure is a cell-contract <scenario>:crash, a HARNESS verdict, and the cell's observation is discarded.
Some scenarios rely on the PRODUCT to end their hold: lifecycle.ts's pause-for-resume setup waits for the stop's abort, and materialize.ts waits for the eviction to abort the held call. If a runtime fails to propagate an abort to the held model call (the stop never reaches the LLM adapter's abortSignal), the hold stays blocked to cell end. That product defect then shows up only as a harness :crash naming the hold, never as a ledgerable product finding, and the discarded observation hides the ids that would have shown it.
Harness-cut descendant sessions are excluded — nothing else. When the cell-end REAP (Temporal's terminateCellWork) terminates a DESCENDANT execution of the cell's own roots — a sub-agent or companion child workflow — whose session is NOT the session of any run this cell's RunLog tracks, that child's work was cut by the HARNESS alone: a Temporal workflow terminate never delivers an abort to the activity running a held model call, so the hold stays blocked through no fault of the product. runCell does not count such a hold as a leak; it records it in CellRun.reapedHolds (the results line's reapedHolds, informational, never a verdict) and the model reset aborts it. The own session is read from the child's start input (CellWorkTermination.terminatedDescendantSessions, then minus the tracked run sessions → ReapReport.terminatedSessions). This surfaced on !282's first pipeline (temporal-cell-reap.integ.test.ts on Docker Temporal: the reaped companion child's held call crashed the cell).
NOT exempt, and still a cell crash: a hold in the session of a ROOT or any other TRACKED run, even when the reap terminated that run (a leaked hold, or an undelivered product abort, keeps its root running to cell end — exactly what this check exists to catch; re-review 2 of !282); and a late child kept from an EARLIER cell (it descends from that cell's root, not this one's). So the triage rule below applies to every case but the harness-cut child.
Triage rule (until the fix below): a :crash whose message names ScriptedModel hold gate '…' still held at cell end (or a holdAfterDeltas pause) is a possible product bug — abort not propagated — NOT a harness flake. Check whether the scenario's path relied on an abort (the guide's "Mid-text model holds" cell-contract bullet lists which do) and whether the model record has abortedAt, before dismissing it.
To close: make the scenarios whose hold waits on a product abort record that as a product assertion before cell end (e.g. a .abort-reaches-model id from waitForAbort, released in a finally either way), so an undelivered abort fails a named, ledgerable id and the hold check only ever fires on a genuine scenario leak. Kept separate from FU-CONF-REAP-ALLOWLIST: that one is a verdict-neutral cleanup hiding a leak; this one is a verdict that is attributed to the wrong side (harness instead of product).
Owner: the conformance owner.
FU-CONF-REAP-ALLOWLIST: gate the conformance reap's reaped list on an allowlist — open (non-blocking)
The Temporal cell-end reap (RunLog.reapSince, MR !275) records every execution it terminates in CellResult.reaped. The list is informational only, so it never changes a verdict. The trade-off:
- A NEW product leak is now cleaned up silently and the cell passes. Example: a child or companion left running after its parent completes (the CP-50 or CP-23 shape).
- Before the reap, such a leak eventually made
test:conformanceexit red, by accident.
The fix is to gate reaped in conformance:verify against a checked-in allowlist of expected (scenario, backend) reaps. Each entry names the finding that explains it; today that is CP-23 for history.companion-continue|temporal-*. An entry not on the list fails the gate, and a stale allowlist entry (a cell that no longer reaps) is reported like a stale declared gap.
Owner: the conformance owner.
FU-CONF-TEMPORAL-TERMINATED-ACTIVITY-WRITES: an activity still running after its workflow is terminated can still write to the stream — open (non-blocking)
Terminating a Temporal workflow does not stop an activity that is already running on a worker. The activity is not told about the terminate until it heartbeats or completes, and it keeps running until it finishes or times out. So after the conformance reap (or dispose) terminates a cell's executions, a runtime-temporal activity that was mid-flight can still:
emitto the parent stream (activities.tsemit);- write to the store;
- call the shared
ScriptedModel.
That is the "Cannot write to stream … in 'failed'/'ended' state" shape, now bounded by the activity's own duration instead of the workflow's.
The reap does not observe running activities: it reads execution status, not pending activities. The cell-end model reset aborts held model calls, which ends most such activities quickly, and none has been observed after the reap in the matrix or in B2's runs.
Fix direction: after terminating, wait (bounded) until every terminated execution's describe().raw.pendingActivities is empty, or until the harness's background worker reports no in-flight activity for those workflows. Report the ones that don't drain.
Owner: the conformance owner.
FU-CONF-TEMPORAL-RESUME-HANDLE-EXTERNAL-TERMINATE: candidate finding — a Temporal resume handle never settles when its workflow is terminated externally — open
This is a candidate for the catalogue, NOT a finding yet. resume() returns the stream-observed v7 handle (runtime-temporal/src/handle.ts:115-146). It resolves only on a terminal, suspension or interrupt chunk, or when the stream drains. A workflow terminated from outside (an operator, a deploy script, the conformance reap) writes no chunk and does not end the stream, so the handle's result() may wait forever.
The conformance harness sees exactly this shape whenever it terminates a __resume-N workflow. It reports that under terminatedUnsettled (harness-caused), never under a product finding. The shape resembles CP-65, but CP-65 is a different cause: a resume re-suspends without a marker.
Next step: a conformance scenario that terminates a resumed run's workflow through the backend and expects the handle to settle, as failed or terminated, within a named bound.
Owner: the conformance owner (catalogue decision).
B8 (transcript validity, RM-29 / DI-17) follow-ups
FU-B8-01: Phase B host-tier .heal-spares-pending assertion — open
The C5 continuation-heal claim "a call registered as pending (client tool or approval) is left untouched" is proven today only by runtime unit tests. The executor tier cannot seed an interrupted session with a registered pending client tool (seedMessages writes messages only, and a real client-tool suspension leaves the session paused, not interrupted), so the conformance scenario cannot exercise it yet. Needs a host-tier harness capability (spec §2, C5).
Owner: conformance owner (Phase B host tier).
FU-B8-02: RM-10 masks Temporal .llm-input-paired in transcript.* — unblocked (re-run pending)
On Temporal, every transcript.* scenario's .llm-input-has-turn1-calls failed because the model input carried no history at all (RM-10), which also masked whether .llm-input-paired would otherwise pass there. B2-core fixed RM-10 (runLLMStep now loads the complete history in the activity). Re-run these ids on the B2 matrix — .llm-input-paired becomes a live RM-29 proof on Temporal for the first time.
Owner: B2-core (RM-10).
FU-B8-03: transcript.finish-with-client-sibling scenario for RM-29 shape (d) — open
Shape (d) — a finishWith call deferred by a same-step client tool / suspending sub-agent, executed once after resume — is proven today by runtime unit tests only (JS/DO/CFW/Temporal/DBOS). Add a conformance scenario once Phase B's executor tier supports submitting a client tool result mid-scenario.
Owner: B8 follow-up (needs Phase B executor submit-tool-result capability).
FU-B8-04: transcript.finish-cosiblings.cosibling-not-executed — done
Done: transcript.finish-cosiblings now declares .cosibling-not-executed (the co-emitted note tool's in-process execution witness never records the turn-1 cosibling) with its positive control .cosibling-witness-observable (the same tool executed in turn 2 is witnessed), plus .cosibling-state-not-written / .cosibling-marker-observable (its customState marker is absent after turn 1 / persisted after turn 2, read via PersistedObservation.customState). All four passed pre-B8 and post-B8 on all 7 Node backends (the cosibling never ran); RM-29 carries a conformance: note. The Vercel path, where the co-emitted call was never produced, was RM-35 (fixed: llm-vercel returns every call).
Owner: B8 follow-up.
FU-B8-05: DI-17 fixed-flip — done
Done in !257 (rebased onto !258): DI-17 is fixed with provenBydiscipline.failed-tool-marked (JS ×3 + Temporal ×3) and transcript.finish-with-fails-then-retries (JS ×3); DBOS is contrast; CF DO / CFW / abort branch / Temporal companion dispatch are unit-test-only, re-open on Phase B.
FU-B8-06: synthetic not-executed results are invisible to hooks, the stream and AI SDK UI parts — open
Synthetic not-executed tool results (C4/C5) are detectable only on persisted helix Messages (isNotExecutedToolResult). Three parity gaps:
onMessage: synthetic results and the resultsrunPhase2FinishWithappends on Temporal are not routed through theonMessagehook the way regular tool results are, so a hook-based observer misses them.- Live stream:
llm-vercel's chunk mapper emitstool_startfor every model tool call, losers included, but no runtime emits atool_endfor a synthetic result, so auseChatpart stays ininput-available. - AI SDK converter:
helix-to-aisdk-converter.tsrenders a persisted synthetic result as a genericoutput-errorand does not carrymetadata.helixSyntheticonto the part, so a reloaded UI cannot tell "not executed" from a real failure.
Needs a parity decision (fire hooks? emit a tool_end error chunk wherever synthetic results are appended? surface the marker on the converted part?) before implementing. The guides (finishing-agents.md, sub-agents.md) state the current limit. Also covers MR !257 review A m4.
Owner: needs parity decision (core/runtime + ai-sdk maintainers).
FU-B8-07: pendingClientToolCalls handling duplicated across JS/Temporal/CFW — open
loadAgentState/AgentState don't carry pendingClientToolCalls directly, so each of JS, Temporal and CFW re-derives or re-plumbs the same workaround to pass pending ids into the C5 heal's excludeIds. Consolidate into one shared helper or a field on the loaded state so the three runtimes stop duplicating this.
Owner: open (core/runtime maintainers, cross-runtime cleanup).
FU-B8-08: CFW deferred phase-2 loop duplicates the normal phase-2 loop — open
workflow.ts's deferred-finishWith-after-resume path re-implements the same phase-2 (finishWith execution + terminal-result synthesis) logic as the normal in-step phase-2 loop, instead of sharing one helper. Extract a common function so the two loops can't drift.
Owner: open (runtime-cloudflare maintainers).
FU-B8-09: DBOS CAS→startWorkflow crash window widened slightly by the heal write — open
DBOS's continuation path has a pre-existing crash window between the completed → active CAS and startWorkflow; the C5 heal write (pairing trailing unpaired calls before the new turn's input) adds one more durable write inside that window, marginally widening it. The same applies to the heal in retry() (failed → active) and resume() from interrupted (MR !257 review F1). Pre-existing risk, not introduced by B8, but worth folding into whatever fixes the underlying crash window.
Owner: open (runtime-dbos maintainers).
FU-B8-10: DBOS dispatcher.appendMessages is at-least-once across a crash — open
dispatcher.appendMessages runs the store append inside a DBOS step. If the process crashes after the store write lands but before DBOS checkpoints the step's completion, recovery re-runs the step and appends the same messages again. A duplicated tool_result is an invalid transcript (RM-29). This is pre-existing and affects every dispatcher.appendMessages call site in runtime-dbos/src/workflows/shared.ts (phase-1 tool results, phase-2 finishWith results, the per-step terminal heal), including the new RM-29 superseded-results append (shared.ts ~2518). Temporal and CFW were hardened in the B8 final review: their phase-2 / superseded appends only append results whose toolCallId is still an unpaired trailing call in the persisted history (core filterUnpairedTrailingResults over getAllMessages). Apply the same helper inside the DBOS append step for role: 'tool' messages.
Owner: open (runtime-dbos maintainers).
FU-B8-11: Temporal main-loop no-output phase-2 drops the finishWith tools' customState (RM-34) — open
In runtime-temporal/src/workflow.ts's main-loop phase-2 branch, when no finishWith call produces output, phase2.nextState.customState (a failing finishWith's partial writes and its hooks' writes) is dropped; JS persists it via commitStep. Catalogued as RM-34 (pre-existing) and owned by that bucket; the site carries an RM-34 comment. The deferred-finishWith-after-resume path added in MR !257 does NOT repeat it: it commits the deferred phase-2 customState (review F12).
Owner: RM-34 bucket.
FU-B8-12: expectPairedTranscript test helper copied into six files — open
The adjacency-aware pairing assertion is duplicated across the RM-29 runtime tests (JS, Temporal, DBOS, CFW). Move it to core's testing utilities (@helix-agents/core/testing) and import it everywhere.
Owner: open (test-infrastructure cleanup).
FU-B8-13: runDeferredFinishWith bare catch {} and run-loop's double buildIteration() — open
runDeferredFinishWith (core step-iterator.ts) swallows a buildEffectiveTools failure with a bare catch {} (the normal step then surfaces it), and runtime-js/src/run-loop.ts calls buildIteration() twice per iteration (once for the deferred check, once for runStepIteration). Log the swallowed error and build the deps/hooks once when nothing is deferred.
Owner: open (core / runtime-js maintainers).
FU-B8-14: the deferred-finishWith path skips the maxSteps pre-check — open
A finishWith call deferred by a phase-1 suspension runs after resume without the maxSteps pre-check the normal step performs. This matches the pre-fix behaviour (the call belongs to an already-counted step) but is undocumented as a decision. Decide and pin it with a test.
Owner: open (core maintainers).
FU-B8-15: HELIX_SYNTHETIC_METADATA_KEY vs COMMON_METADATA_KEYS — open
The synthetic marker key lives in its own constant (transcript.ts) instead of a COMMON_METADATA_KEYS.SYNTHETIC entry beside TOOL_FAILED (types/state.ts). Consider moving it for consistency (a public-export change).
Owner: open (core maintainers).
FU-B8-16: Temporal main-loop interrupt emits run_interrupted before persisting the status — open
The main-loop interrupt branch (the finishInterrupted closure in runtime-temporal/src/workflow.ts) emits run_interrupted before persistTerminalState('interrupted'), so a handle can resolve interrupted while loadState still reads active for a moment. The post-drain resume branch and the deferred-finishWith interrupt path (persistFirst: true) already persist first. Make persist-first the only order (pre-existing; changing it reorders the main loop's activity sequence).
Owner: open (runtime-temporal maintainers).
B2-core (LLM history completeness) follow-ups
Deferred from the B2-core bucket (spec ../superpowers/specs/2026-09-25-llm-history-completeness-design.md). None of these blocks the B2 contract; each names the finding that owns it where one exists.
FU-B2-01: DBOS checkpoints the full history every step (O(N²) system-DB growth) — open (MED)
getMessagesStep is a @DBOS.step, so the complete history it returns (now the WHOLE log, since RM-11 removed the 10k cap) is written to the DBOS system database once per LLM step. A session of N messages over S steps records O(N·S) bytes, O(N²) over a long session. Fix: move history load, message build, the onMessage hook fires and the LLM call into callLLMStep, so the history never becomes a step output. Accepted cost for B2 (spec §3.3).
FU-B2-02: DBOS from_checkpoint / retry edge cases (B6 minors) — open (LOW)
- M1 A stranded session (workflow terminal or missing, session still
active) cannot be rewound:resume({ mode: 'from_checkpoint' })throwsAgentAlreadyRunningError. - M2 Between the claim CAS and
DBOS.cancelWorkflow, the suspended workflow can still complete a step; the rewind then races that write. - M3 A sub-agent's JSON input that happens to have a
messagekey is still parsed as anAgentInput(only a content-lessmessagefalls back to the JSON form). - M4 A rewind that fails after the suspended workflow was cancelled rolls the status back to
interruptedbut leaves the oldpendingClientToolCallsin place. - Kept pending calls after a DBOS rewind are not re-dispatched; they rely on the next continuation's trailing heal.
FU-B2-03: DBOS checkpoint state is the pre-step snapshot — open (LOW)
DBOS inter-step and terminal checkpoints store state loaded at the top of the iteration, not the state after the step's tool writes (messageCount is live, state is not). A rewind to such a checkpoint restores customState one step early. Other runtimes snapshot the committed post-step state. (B12 review M-1.)
FU-B2-04: CP-68 residue (B13 minors) — open (LOW)
- No dedicated test for the terminate-write race in
recordPersistentChildNonCompletion(the re-assert is idempotent and self-heals). - CF Workflows: no workerd test for a DIRECT child interrupt (parent-driven is covered by unit tests; the workerd harness cannot drive it, CP-23).
- Temporal and DBOS never record
'interrupted'on a companion'sSubSessionRef, so Temporal'scompanion__waitForResulton a stalerunningref can wait until its timeout (pre-existing; sent to the conformance owner). - Temporal
__resume__${stepCount}child workflow ids can collide across turns (pre-existing; sent to the owner). - A re-interrupt that lands before the resumed child's first step commits rolls back the message
sendMessagedelivered (rootresume({ mode: 'with_message' })has the same window; sent to the owner). - JS: resuming a companion from
interrupteddoes not run the trailing unpaired-call heal (RM-29 territory; sent to the owner).
FU-B2-06: Temporal resume with nothing to drain never settles (CP-65) — open
resume() on a session that is still suspended with nothing to drain early-returns without a suspension_marker, so handle.result() hangs. B2's from_checkpoint path does not hit it (it rewinds first), but other resume-on-suspended cells do. Owned by CP-65.
FU-B2-07: CF Workflows minors (B8 / B12 reviews) — open (LOW)
retry()still reads the triggering message with an uncheckedgetAllMessages(executor.ts), notloadLLMHistory.The lenient
loadAgentStatereads the full history even though bookkeeping callers never use it.markAgentFailedno longer refreshesupdatedAt;persistFromCheckpointHistoryFailuredoes a plainsaveStateafter its CAS (same asmarkAgentFailed).run_resumed.stepfor a rewind reports the pre-rewindstepCount.A rewind's truncation is not undone when a later post-claim step throws (only the status is rolled back).
writeTerminalCheckpointdoes extra store calls inside the retriedexecuteAgentStepstep; a step retry writes one more (equally truthful) checkpoint.Temporal and CFW
retry()runtruncateMessagesbefore throwing "No user message found to retry with". Benign: the truncate target is the latest checkpoint, so nothing committed is lost, but the throw should come first.
FU-B2-08: runtime-js minors (B7 review) — open (LOW)
- Cross-turn
from_checkpointstream cleanup is floored at the current run's start sequence, andstream_resyncis emitted only when chunks were removed. loadFromSessionreads the history twice.retry()'s history-failure seam skips the FIX-B rollback when its CAS is lost.- Loading the companion history before the CAS widens the window in which a
sendMessageappend can land between the read and the resume. - No dedicated stream-readback test for the
retry()seam;initializeSessionseam continuation/checkpoint-branch cases are untested.
FU-B2-09: runtime-temporal minors (B4 / B5 reviews) — open (LOW)
resume()leaves the sessionactiveif the rewind or thewith_messageappend throws after the CAS.extractApplicationFailureDetailaccepts ANYApplicationFailurewhosedetails[0]parses as anErrorDetail; a future activity that puts a message-only object there would be misread as classified.- The branch activity wraps only
HelixErrors as non-retryable (the history-load site wraps everything). - A branch with both an invalid source and an invalid checkpoint reports
state_checkpoint_not_foundbeforestate_session_not_found. - No unit tests for branch
messageIndex/checkpointId-over-messageIndexprecedence.
FU-B2-10: core primitive minors (B1 / B2 / B2b reviews) — open (LOW)
loadLLMHistoryreportsread: 0when paging throws midway (the partial count is lost).rewindToCheckpointhas no guard thatcheckpoint.sessionId === sessionId(callers resolve first); a negativebranch.messageIndexis not validated.- A restored pending entry keeps
isPreSubmitStubverbatim from the snapshot. - No direct test that a currently pending entry wins over the snapshot entry.
FU-B2-11: store minors — open (LOW)
RedisResolved by RM-50 (!269):appendMessagesisRPUSHthenHINCRBY(two round trips); a truncate racing an append can skewlistSessions().messageCountuntil the next write.appendMessagesis now one Lua script (RPUSH+HINCRBY messageCount, guarded on the session existing), andtruncateMessagesis one Lua script that rewrites the count fromLLEN, so the two serialize and the hash count cannot skew.- The executor-tier e2e cases for forced-completion checkpoints cannot fail on stores whose
saveStatemints a checkpoint (memory / Redis / Postgres). The non-minting-store family insession-model-branching.integ.test.tscovers Temporal and runtime-js; DBOS has no such cell (it runs on Postgres only), and D1 / CFW catch it on workerd.
FU-B2-12: out-of-scope findings surfaced by B2 — tracked by the conformance owner
- CP-30 DBOS
retry()ignoresRetryOptions(checkpointId,message). - CP-34 The DO host
/resumedropscheckpointId, sofrom_checkpointcannot reach the DO host tier. - CP-41 CFW / Temporal treat the
parent_messageflag as an interrupt, sosendMessageto a RUNNING child can stop it with the message unread. - CP-58
retry()withoutoptions.messageafter a first-step failure throws "No user message found to retry with" on runtime-js. CFW now behaves the same after a forced-completion failure, whose tail writes a terminal checkpoint (RM-33); an exception exit writes no extra one (R28); retry truncates to the latest checkpoint. - CP-69 DBOS blocking spawn waits for the child's idle timeout (FU-DBOS-BLOCKING-SPAWN-SEMANTICS, fixed by the CP-69 MR).
- CP-70 The DO base swallows a typed pre-claim resume failure: the session stays
pausedwith noerrorDetail. - RM-13 A store that skips a corrupt row on read now makes
loadLLMHistoryfail the run (state_history_incomplete) instead of silently dropping the row; corrupt-row paging itself stays with RM-13.
FU-B2-14: CF Workflows whole-log reads that only need scalars — open (LOW)
All pre-existing on main, performance only:
runtime-cloudflare/src/workflow.ts${agentType}-check-existingreturns the fullAgentStatefrom the file-localloadAgentState, which loads ALL messages (uncheckedgetAllMessages) into the step result. A long session risks the Workflows step-result size limit. The step only needs status / stepCount / a few scalar fields.CloudflareAgentExecutor.loadAgentStatereads messages with uncheckedgetAllMessages; it feedsresume()bookkeeping, never the model, so no messages are needed there.
FU-B2-17: a non-owner write can fail a pinned from_checkpoint rewind — done (superseded by S5)
Superseded by the S5 snapshot MR: no runtime claims a rewind with a version-pinned CAS any more. A from_checkpoint rewind (and a retry's rewind) is part of the entry's run-start commit (StartRun.truncateTo, planned by planRewindToCheckpoint), which is guarded by the run CAS (expectCurrentRunId), not by the session version, so a non-owner version bump (a submit on a kept call, a child's root-ownership update) no longer fails it. The text below describes the old model. rollbackLostRewindClaim is unused (FU-S5-CORE-UNUSED-REWIND-CLAIM-HELPERS).
Owner: orchestrator (re-brief).
A claim-pinned rewindToCheckpoint (claimedVersion) fails with AgentAlreadyRunningError on ANY version bump between the caller's claim CAS and its rewind save, and nothing is written or truncated. Some bumps come from writers that do not own the session:
- a
submitToolResulton a call the rewind would keep; - a child session's root-ownership update (
updateRootOwnership) when the rewound session is that child's root.
Resolved: the stranded claim. A claim from a NON-active source (interrupted / paused / completed / failed) that now has no pending calls cannot have a competing owner, because on every runtime:
assertRewindablerejects anactivesession with no pending calls;continue/with_messagereject it as running. runtime-js and DBOS always did this; Temporal and CFW now do too, and before this change they would start a second run;execute()rejects it.
So the lost claim was a non-owner bump. Core rollbackLostRewindClaim moves the session back to its source status with a pinned CAS, on JS, Temporal, CFW and DBOS. Each runtime has a forced test: an interrupted source with a non-owner bump ends interrupted and rewinds again.
What remains:
The caller sees a spurious
AgentAlreadyRunningErrorand must retry.A lost claim is still not rolled back when either:
- the source was
active(a suspended session), or - the session now has pending calls.
A competing claimer may own the session in both cases. The session is left
activewith its pending calls, which reads as suspended, so it stays rewindable on every runtime and continuable on Temporal and CFW. For apaused/interruptedsource with pending calls, the status change toactiveis the only lasting effect.- the source was
A precise fix, one that re-bases across non-owner bumps, needs a claim token that the claim itself writes, which means a store-level field. Owner: B2 follow-up.
FU-B2-16: root test:integration runs two DBOS suites against one system DB — open (LOW)
Locally, RUNTIME_DBOS_INTEG=1 pnpm run test:integration runs the e2e DBOS cells and the runtime-dbos integ suite in two concurrent processes against the same DBOS system database, and DBOS e2e cells then fail (e.g. a turn ending failed, or agent-hooks-parity DBOS "expected [] to include 'beforeLLMCall'"). Pre-existing on main (same family, different test names); CI runs these as separate jobs, so it does not hit this. Suspected cause: the process-local hook registry / workflow ownership across two DBOS processes (distinct executor ids, so unproven). Workaround: run the DBOS suites package-direct. Owner: ci-health (B8).
FU-B2-15: pre-existing observations from the B16 fix round — reported to the conformance owner
- CF Workflows non-checkpoint
resume()truncates before an unpinned claim, so it can race a live run. Pre-existing on main; reported to the conformance owner (CP-35). B2 later narrowed it:executor.resume()continue/with_messagereject a liveactivesession before any write and claim with a version-pinned CAS before any cleanup or truncate, andfrom_checkpointclaims with a version-pinned CAS before it rewinds. CP-35 stays open forhandle.resume()(no guard, truncates before an unpinned claim) andretry()'scleanupToStepwithoutstartSequence; the live-run half is CP-83. - At
maxSteps, an agent withoutoutputSchemaendscompletedon JS and Temporal butfailedon CF Workflows. Pre-existing on main; reported to the conformance owner (candidate). - Temporal
handle.result()can resolve (from the stream) before the workflow's last activities persist the final rows and terminal status. Pre-existing on main; reported to the conformance owner (candidate).
S5 (consistent snapshots + checkpoint truth) follow-ups
Deferred from the S5 snapshot MR (branch reliability-example-refresh-flake; spec ../superpowers/specs/2026-09-26-consistent-snapshot-checkpoint-truth-design.md, §9 "Not in scope" and the review rounds). None blocks the S5 contract. Owners marked "orchestrator" are routed by the reliability orchestrator.
FU-S5-B1-CONFLICT-REACTION: what a client does after a refused entry — open (owner: B1)
A live run-start conflict surfaces as RunStartConflictError (named AgentAlreadyRunningError). A chat send refused that way answers 409 {code: 'state_already_running', retryable: true}; resumes (handleResumePath, the DO's ensureExecutionContinues) and requests with no new user content still attach to the winner's stream. The spec leaves the client-side reaction to B1: whether useHelixChat should retry a refused send once the winner settles, attach to the winner, or keep surfacing the error (today: chat.error + parseHelixChatError, and chat.regenerate() retries).
FU-S5-LIVE-SUPERSEDED-RUN-CHUNKS: a superseded executor's chunks before the new run's claim — open (LOW, owner: orchestrator)
Once a newer run has claimed the session stream, every stream manager drops an own-session chunk stamped with another runId (chunkFenceRunId), and endStream / failStream / pauseStream are owner-fenced (CP-85). Between the new run's run-start commit and its claimStream, the stream still names the old owner, so a superseded executor's chunks pass. On runtime-js the lease gate covers that window (gateWriterOnLease). On the CF DO, Temporal, DBOS and CF Workflows the old executor stops at its next fenced write, which can be a whole LLM step of streamed text. A live client can show that text until it reloads; a snapshot never contains it.
FU-S5-LIVE-ATTEMPT-DUPLICATION: a retried attempt's text stays on a live screen — open (LOW, owner: orchestrator)
A Temporal activity retry or a CF Workflows step.do retry re-streams the step under a new attemptId. The AI SDK protocol cannot retract rendered text, so a live client shows both attempts until it re-anchors. data-attempt-superseded is a resync trigger: a client wired with useHelixChat({ resync }) (or useAutoResync with stream control) re-anchors on it. Remaining gaps: a client without resync wiring keeps the duplicate until reload, and a run that FAILS with a superseded attempt on screen gets no re-anchor (the failed stream ends before a resync lands).
FU-S5-AUTO-SEND-RECHECK: re-check the spurious auto-send after a cursor resume (B1 / B6) — open (owner: B1)
Out of scope for S5 (spec §9). It was expected to disappear because snapshots now anchor at step boundaries and replayed tool calls are never re-dispatched (useHelixChat filters replayed ids; a replayed PENDING client tool runs once after the replay). Re-check on the host and client tiers with the sendAutomaticallyWhen wiring the examples use.
FU-S5-SUBAGENT-INNER-CONTENT-ON-RELOAD: a sub-agent's inner stream content is not in the snapshot — open (owner: orchestrator)
Out of scope for S5 (spec §9). A snapshot holds the parent session's committed rows; a sub-agent's inner-stream content (its own text and tool parts rendered inside the parent's message) is not rebuilt on reload. The child's content is durable in its own session.
FU-S5-WORKERD-FIRST-SEND-CONNECTION-LOST: workerd Network connection lost on a first send under load — open (LOW, owner: B8)
2 of about 3250 opennext stress runs on the S5 branch (6 workers × 50) failed on the very first send POST with workerd Network connection lost inside miniflare's entry worker: the session never started. 0 in about 1900 base runs, so not proven unrelated. Same local-workerd class as FU-OPENNEXT-UNCAUGHT-ON-CLIENT-ABORT and the "worker restarted mid-request" flake (spec §9, out of scope).
FU-TEMPORAL-DBOS-HOST-TIER-REFRESH-PROOF: no Temporal- or DBOS-hosted chat-handler refresh proof — open (owner: the Temporal round / B6)
The char-exact refresh e2e runs on JS ×3, Temporal ×3, DBOS and CF Workflows against the executor, and the host / client conformance cells run on cf-do and js-chat-host. There is no Temporal- or DBOS-hosted handleChatStream refresh spec, and the executor tier has no snapshot route, so resume.replay-exact has no Temporal / DBOS host cell. Add a Temporal-hosted (and DBOS-hosted) chat host plus host cells.
FU-DBOS-DUAL-EXECUTION-TERMINAL-WRITE: a second live DBOS execution may write a terminal state over the recovered one — open (owner: WA)
Surfaced by: tracing child-identity-recovery's stall (WA, 2026-11-05). All of this is read, not run, and no run has shown it.
DBOS can recover a workflow on another executor while the original execution is alive. The ownership guard (workflows/execution-guard.ts) and checkIfCanceled do not cover that case: the original's pool works (drv-cirx saw the in-process K5 attempt fail at checkIfCanceled in all 8 natural reds, the opposite case), and DBOS records a step's output after its body, so DBOSWorkflowConflictError arrives after the body has run. The recovered execution keeps the same runId, so the run fence does not separate the two.
K5 (runCommitTerminalState) commits with expectedVersion read inside the step body, which catches only a race between one read and one commit. Between K5 and the run status write the run is still live, so a second K5 looks able to land over the first; once the status is terminal it is refused. Two executions planning the same terminal do little harm: the case that matters is different plans, e.g. a body that ran before its output was recorded. Whether step commits are exposed the same way is not read.
Becomes a finding on an observed overwrite (a witness needs two live executions of one workflow id with different plans, held at the K5 window), on a run reporting one outcome while the session shows another, or if DBOS recovers a live executor's workflows in ordinary deployment. May be accepted at-least-once behaviour. Related: FU-B8-10.
FU-EXECUTE-HISTORY-ENDS-UNPAIRED: execute({ history }) ending in an unpaired call is not healed — open (owner: WE-2)
Surfaced by: RM-117's trace (WE-2, 2026-11-05), read, not run. A caller's own history: that ends in an unpaired tool_use seeds the session as given, so the same mid-history gap as RM-117 follows. It is out of RM-117's scope: the caller wrote that history, and the fix may be a typed refusal rather than a heal.
FU-DBOS-HARDCRASH-REPLAY-NONDETERMINISM: DBOS restart-recovery replay fails determinism after an in-process hard crash — open (LOW, owner: orchestrator)
An integ restart-recovery test (approval-gate, client-tool-recovery-dbos) failed DBOS's determinism check in about 1 of 2 full-suite runs during S5 Task 13: the orphaned pre-crash body, still running after the harness's in-process hardCrash, recorded a step at the function id the recovered body expected. The S5 ownership guard (an execution that lost its workflow writes no durable step; workflows/execution-guard.ts) addresses that class. Re-measure over 20 or more runs; close if clean, otherwise confirm which executor recorded the step (listWorkflowSteps).
FU-S5-PATH5-FOREIGN-TAB-SEND: a new message from another tab can attach to a live stream — done (CP-13 reject half)
handleChatStream answered a refused SEND with 409 state_already_running, but the active-stream attach probe (path 5) still attached a request whose trailing user message is new (sent from another tab, or while the run streams) instead of refusing it: 200 with the running turn's text, the message dropped. Done: path 5 excludes a send (hasTrailingUserMessage); a send goes to the fresh-turn path and the runtime's refusal is the typed 409. The "compare the message id" direction is D-2's idempotency key (a retried send attaches to its own run), which lands with CP-13's interrupt_previous half.
FU-S5-STREAM-OUTAGE-SWEEPER: a permanent stream-backend outage leaves the stream active — open (LOW, owner: orchestrator)
If the stream backend stays down through a run's terminal path, the terminal claim and the stream transition fail and are logged; the run status is terminal but the stream stays active. The snapshot derives ended / failed from the run, so readers are correct, but nothing sweeps the stream.
FU-S5-DO-BACKLOG-BOUNDARY-LATE: the DO client's caughtUp can resolve late — open (LOW, owner: orchestrator)
DOStreamManagerClient's resumable reader learns the end of its replayed backlog from the server's SSE replay; the boundary can be late (never early). A late boundary only delays the switch from the backlog filter to the live filter. An exact boundary needs a DOAB change.
FU-S5-OPENNEXT-DEBUG-ROUTES-DO-ADDRESSING: opennext debug routes read the wrong DO — open (LOW, owner: orchestrator)
getDOStub (debug routes only) addresses the DO as session:<id>, but the chat handler's DO clients use the bare id, so the gated debug routes read an empty DO. The same split was fixed in research-assistant-cloudflare-do by S5.
FU-CHILD-ROOT-OWNERSHIP-WRITE-LOGGED: a paused child's root ownership write only logs when it does not land
Status: narrowed by CP-253: a child's write now precedes its frame and a thrown op or foreign owner fails the step (runtime-js, the CF DO, Temporal). Open: decideRootOwnershipFence's none arm, which writes nothing and logs at info. Surfaced by: CP-128 MR D (D6), which made a self-rooted suspension's failed op fail the suspension.
Before CP-253, on runtime-js and the CF DO, persistSuspension only logged for a CHILD's write onto its root: a thrown op, a returned foreign owner, and decideRootOwnershipFence answering none. The child's suspension is already committed, so its call is pending with no routing entry on the root, and a submit to the root answers unknown_tool_call. Temporal's writeRootOwnership throws instead. The none arm may be correct by design for a child nothing awaits.
The thrown arm is recorded on CP-234 (the conformance owner's dedupe); the other two are noted there as read, not reproduced. Decide with CP-234 whether a child's write comes before its commit and fails it, as a self-rooted one does now.
FU-TEMPORAL-CHILD-OWNERSHIP-COLLISION-RETRIED: a rooted child's ownership collision is retried, not final — in progress (WA, batch 10)
Surfaced by: CP-253's review (WA).
GenericActivities.recordCallOwnership (runtime-temporal/src/activities.ts:6878 at main 47ceddad1f) routes a rooted child's call through writeRootOwnership (:6928). On a collision (the root's route names another session) that throws a plain Error (:6939), so the activity's retry policy retries it, writing nothing each time, until it gives up. The self-rooted branch throws ApplicationFailure with nonRetryable: true (:6891), final at once. Fix: make the child's collision non-retryable too, and pin it as the self-rooted case is pinned ("a foreign owner is final: no frame, and not retryable", runtime-temporal/src/__tests__/activities-entry-before-frame.test.ts:132).
FU-OWNERSHIP-CONDITIONAL-REMOVE: a run's cleanup of its uncommitted call routes needs a conditional remove — open (owner: WA)
Surfaced by: CP-253's review (WA).
A run's cleanup of its uncommitted call entries reads the session's current run and its pending calls, then removes with the plain updateClientToolCallOwnership op: runtime-js clearUncommittedCallOwnership (runtime-js/src/run-loop.ts:882 at main 47ceddad1f), and the same on Temporal once !644 lands (GenericActivities.clearUncommittedCallOwnership). Two limits follow:
- a superseded run's own uncommitted entries stay, since the cleanup removes nothing once another run is current. They name no call, and a later reuse of the id across sessions fails on the collision;
- a run that starts and adds the same id between the cleanup's reads and its remove loses that entry.
The op should carry the writing attempt's identity, and the conditional remove should match it, so the remove never takes another attempt's entry for the same id. That also closes two orders that !644's newer-attempt check (isLlmAttemptSuperseded, runtime-temporal/src/llm-attempt-fence.ts:47-56 (at !644's head d59d88c52f)) misses. Both need the retry to produce the same call id:
- An older
runLLMStepattempt whose ownstep_startlands AFTER the retry's. It timed out before writing itsstep_start(runtime-temporal/src/activities.ts:3678(at !644's head d59d88c52f): a slow history load, abeforeLLMCallhook, the prefix check), then learned of its cancellation on a heartbeat after the retry had routed. The check counts only a LATERstep_startas newer, so the older attempt reads as current and its throw removes the retry's route. The attempt fence has the same ordering gap. - An attempt fenced before its
step_start. The fenced writer drops the chunk (runtime-temporal/src/llm-attempt-fence.ts:113-115(at !644's head d59d88c52f)). If its adapter ignores the abort, it can still route and throw, and the check finds no ownstep_start, so it removes. The Vercel adapter honours the signal.
Correction owed with the fix: the comment in runLLMStep's catch (runtime-temporal/src/activities.ts:4311-4317 (at !644's head d59d88c52f)) says a newer attempt's step_start is "its first chunk, before it routes anything". That holds for the retry, but it implies the older attempt's step_start is always first on the stream, which order 1 shows it isn't.
Ruling (orchestrator, 2026-11-02):
- The writer is an ordered
(turn, step, attempt)in a store-internal writers map, withremoveIfWrittenBy. An exact identity match alone does not close !644's P2a. - It lands after W1's own-key MR 2 and after !644.
- It is a major: a new store obligation.
- The plan is WA's plan (
plan-conditional-remove.md, kept locally by WA).
FU-OWNERSHIP-EXACT-UNCOMPLETE: an exact undo of a drain's completed markers — open (LOW, owner: orchestrator)
Surfaced by: CP-263 (Temporal's drain). See FU-OWNERSHIP-CONDITIONAL-REMOVE (the same ownership-op rework).
A gated consume writes the tree root's completed markers (updateClientToolCallOwnership complete, core completeBeforeGatedConsume) before the commit that consumes the answered calls (Temporal's drain since CP-263). When that commit is refused (a stop, a newer run), restore() undoes the markers it newly wrote with one add of each call to its owner, which clears the marker and restores the route. The op cannot express an exact undo, so three residuals remain:
- a failed compensation leaves a marker beside a call still pending for the stop, so a late submit answers
already_completedinstead ofrun_stopped; - the "newly written" ids come from a read before
complete, so a consume of the same entry plus a stop landing between that read andcompletecan clear another writer's marker; - the
addwrites a route (to the call's real owner) for an id that had none before.
Fix if any of these ever matters: a field-scoped uncomplete op that removes a marker only at its expected at (and restores the route only when the op removed one), in all five stores.
FU-GATED-CONSUME-REPORT-FAILED-IDS: completeBeforeGatedConsume hides a failed marker write from its caller — open (batch 11, store and ownership work)
Surfaced by: Greptile on !662 (SDK-2 MR 1c, DBOS). See FU-OWNERSHIP-EXACT-UNCOMPLETE (the same helper).
Core completeBeforeGatedConsume logs a failed pre-read or a failed complete op and still returns outcome: 'completed'; its marked list cannot tell a failed op from a marker that already existed. A caller that then consumes or removes the entry leaves the call with neither its route nor its marker, so a later real answer is unknown_tool_call instead of late (or already_completed).
- DBOS (fixed in !662): the lapsed-wait settle re-reads the tree root's marker and removes the entry only over the lapse's marker, else restores and throws a retryable error (the step retries).
- Open: the runtime-js orphan drain, CF Workflows'
applyResultsAndReload, Temporal's drain and the child run-starts (runtime-js, CF Workflows) still proceed after a failed marker write.
Fix: the helper reports the ids whose complete failed (a failed list, or a typed throw on request), and each caller decides: retry, or refuse the consume before it commits.
FU-S5-DBOS-OWNERSHIP-GUARD-SDK-COUPLING: the DBOS ownership guard keys on DBOS.tracer identity — open (LOW, owner: orchestrator)
workflows/execution-guard.ts detects a replaced DBOS executor by the identity of DBOS.tracer (one per DBOSExecutor). That holds on the DBOS SDK version the package pins; an SDK upgrade must re-verify it.
FU-S5-RUN-ERROR-DETAIL: run records carry no errorDetail — done (RM-134, !865)
Closed by: RM-134 (!865). RunMetadata.errorDetail is stored on all five stores (Postgres V34, D1 V36, DO-SQLite 32, the Redis run hash, memory; contract RS-ED1, and RS-ED2 for a run the next entry's run-start commit finishes), written by runtime-js's and core's run-status writes, and carried on the CF DO's GET /runs page. The runtimes that do not write it yet have their own entries below.
A failed run's classified ErrorDetail is on the session (SessionState.errorDetail) and the failed stream chunk, not on its RunMetadata. Run-level detail needs a schema change in all five stores.
FU-TEMPORAL-RUN-ERROR-DETAIL: Temporal's run record carries no errorDetail — open (LOW, owner: orchestrator)
Temporal's run-status writes (runtime-temporal/src/activities.ts:7155, :8453, :8785) pass the failure's message at most (:8785 passes no metadata), so a failed Temporal run's record has no RunMetadata.errorDetail (RM-135; the field is RM-134's). Pass the detail its terminal state carries, as runtime-js does, and remove RM-135's ledger fragments for the Temporal lanes.
FU-DBOS-RUN-ERROR-DETAIL: DBOS's run record carries no errorDetail — open (LOW, owner: orchestrator)
DBOS's run-status writes (runtime-dbos/src/steps/run-lifecycle.ts:225, steps/run-commit.ts:2111, lifecycle/interrupt.ts:549) pass no detail, so a failed DBOS run's record has no RunMetadata.errorDetail (RM-135; the field is RM-134's). Pass the detail its terminal state carries, and remove RM-135's ledger fragments for dbos-postgres.
FU-CFW-RUN-ERROR-DETAIL: CF Workflows' run record carries no errorDetail — open (LOW, owner: orchestrator)
CF Workflows' run-status writes (runtime-cloudflare/src/steps.ts:8709, :8752) pass no detail, so a failed CF Workflows run's record has no RunMetadata.errorDetail (RM-135; the field is RM-134's). Pass the detail its K5 terminal commit carries, and remove RM-135's ledger fragments for cfw-workflows-d1.
FU-S5-CORE-UNUSED-REWIND-CLAIM-HELPERS: core still exports the !279 claim / hold helpers — open (LOW, owner: orchestrator)
Every runtime now rewinds inside its run-start commit (StartRun.truncateTo, planned by planRewindToCheckpoint). No runtime calls rewindToCheckpoint, rewindOutcomeOf, rollbackLostRewindClaim, rollbackFailedResumeClaim, failRewoundSession, rewindHoldOf, isRewindHoldTakenBy or rekeyRewindHold any more. Remove them (no back-compat) or document them as building blocks for a custom loop.
FU-CORE-REWIND-HOLD-JSDOC-STALE: PendingClientToolCall.rewindHeld* JSDoc describes the removed hold machinery — done
Closed by: CF-D2 MR 1 (!316). The JSDoc now says the fields are the legacy !279 hold: written only by core's rewindToCheckpoint({ holdPending }), which no built-in runtime calls; assertRewindable still honours a held call and Temporal / CF Workflows strip the fields (releaseRewindHold). Their removal stays with FU-S5-CORE-UNUSED-REWIND-CLAIM-HELPERS. docs/reference/runtime-js.md's from_checkpoint section, which still described the hold / takeover model, was rewritten in the same MR.
The JSDoc on PendingClientToolCall.rewindHeld, rewindHoldKey and rewindHoldToken (packages/core/src/types/state.ts:155-190) still describes the !279 rewind hold: a crashed rewind's held calls, …__rewind-<key> instance ids and takeover tokens. Rewinds now happen inside the entry's run-start commit (StartRun.truncateTo), and no runtime sets these fields any more; only assertRewindable reads rewindHeld and releaseRewindHold strips them. Rewrite the JSDoc to say so (or remove the fields together with FU-S5-CORE-UNUSED-REWIND-CLAIM-HELPERS). Found while rewriting the llm-history changesets as-built for the first release since 2026-09-23 (RELEASE-1).
FU-S5-CFW-HANDLE-BEFORE-RUN-START: CF Workflows timing edges at the run-start window — open (LOW, owner: CF-D1)
getHandle()can return a reconnect handle before the instance's run-start commit lands ("no live run yet" is reported only when no run record exists at all).- A send that arrives while the previous turn's instance finishes waits up to
priorInstanceSettleTimeoutMs(30 s) before its 409.
FU-RUN-START-PLAN-DURABLE: the run-start plan lives in memory — open (MED, owner: orchestrator)
Temporal, CF Workflows and DBOS keep a committed run-start entry's plan (restore target, entry fields, rewind, discarded calls) in memory, keyed by runId (Temporal GenericActivities.committedEntryPlans, CFW AgentSteps.committedEntryPlans), so a retried activity / step.do after the commit rebuilds the entry instead of re-planning. A retry that lands on ANOTHER worker or invocation, after the commit and before the entry state was written, has no plan: a retry entry then fails non-retryably (framework_internal_error) and its committed run ends failed rather than starting from a wrong state (on CFW the engine retries that error twice more, harmlessly). Fix: record the plan's outputs durably inside the run-start commit (an optional StartRun.entryPlan stored on RunMetadata: the restore target's checkpoint id plus the entry fields), across core and all five stores, and drop the fallback.
FU-S5-APPROVAL-ID-DURABLE-RUNTIMES: Temporal / DBOS / CFW do not record approvalId on pending entries — open (LOW, owner: orchestrator)
Only the core step iterator (runtime-js, the CF DO) writes PendingClientToolCall.approvalId. Temporal, DBOS and the CFW approval helper emit approvals themselves and do not, so the snapshot infers a pending SERVER tool to be an approval; a client-executed tool that is also approval-gated renders input-available in the snapshot on those runtimes until the re-sent tool-approval-request arrives. Set approvalId in runtime-temporal, runtime-dbos and approval-gate-workflow-helper.ts.
FU-CFW-TERMINATE-RUN-STREAM-ACTIVE: executor.terminateRun() leaves the stream active — open (LOW, owner: CF-D1)
After terminateRun(sessionId) the CF Workflows instance is terminated, but the session stream stays active and the run stays live until the next entry supersedes it. A design decision so far; a reader that attaches in between waits until the next entry or its live-wait fail-safe.
FU-LAZY-SCHEMA-IDENTITY-TESTS: the lazySchema identity tests are indirect — open (LOW, SDK startup cost)
The toolSchemaJson identity test only proves it through a derived value, and the lazySchemaFactory keyOf test has no assertion that two distinct keys give two distinct schemas. Assert both directly.
FU-S5-ZOD-ERROR-NAME-CHECKS: name / cast error checks should be Zod schemas — open (LOW, owner: orchestrator)
CLAUDE.md principle 5. Manual name and cast checks remain in Temporal activity-failures.ts, asLostRetryRace and isSupersededFailure; CFW StepMessageSchema (a z.custom duck type), the as Partial<AgentWorkflowResult> engine-output casts and the message-regex error classification; and the resync / tracker / use-helix-chat guards in ai-sdk.
FU-TEMPORAL-PERSIST-TERMINAL-STATE-TEST-ONLY: delete the test-only persistTerminalState activity — open (LOW, owner: the Temporal round)
Production Temporal ends a run through commitTerminalState + finalizeTerminalRun. persistTerminalState is still exported and used by about 10 test files only. Move those tests to the production activities and delete it.
FU-S5-SIBLING-CHECKPOINT-SCANS: CFW and DBOS still scan every checkpoint of a session — done (RM-133)
planRetry now reads only the failed run's checkpoints through the new listRunCheckpoints (review I10). Two sibling scans still read every checkpoint in full: CFW appendFailedStepRows (runtime-cloudflare/src/steps.ts), which runs inside one Worker invocation against D1's per-invocation query budget, and DBOS run-commit.ts. Switch both to listRunCheckpoints.
Done with RM-133: both read only the run's checkpoints, without their state (listRunCheckpointMetas): CFW's replayed-append check (ownAppendLanded) matches an append commit by its kind, run and count, then its rows; DBOS's findStepCommit matches a step commit by its kind, run and step. Pinned by runtime-cloudflare/src/__tests__/append-failed-step-idempotent.test.ts and runtime-dbos/src/__tests__/steps/find-step-commit.test.ts (no session scan, no full read).
FU-DBOS-CLIENT-TOOL-RECV-OUTLIVES-FAILED-TURN: a client-tool recv can outlive a failed DBOS turn — open (LOW, owner: the DBOS round)
When a DBOS turn fails while one of its client-tool calls is waiting in DBOS.recv, the waiting step can outlive the settled turn. A late submit for that call then lands on a run that already ended failed. Bound or cancel the pending recv when the turn settles.
FU-S5-ONAGENTFAIL-WRITES-K5-JS-DO: onAgentFail state writes are dropped on runtime-js and the CF DO — open (LOW, owner: orchestrator)
On every terminal exit the K5 commitState carries the terminal status and error. On completion it also carries the agent's onAgentComplete state writes on every runtime. onAgentFail writes land in K5 only on Temporal and CF Workflows: runtime-js (and the CF DO, which runs its executor) fires onAgentFail after K5 and drops what the hook writes. DBOS hooks cannot write state. Apply onAgentFail writes to K5 on JS / DO (as onAgentComplete does), or document the hook as read-only everywhere. Add a conformance cell for it.
FU-S5-DBOS-SUPERSEDED-HOOK-PAYLOAD: DBOS onAgentSuperseded gets a placeholder payload — open (LOW, owner: the DBOS round)
runTurnMappingStops (runtime-dbos/src/workflows/standard-workflow.ts) fires onAgentSuperseded with { error, finalState: {}, stepCount: 0, durationMs: 0 } and context.stepCount: 0. The other runtimes pass the run's real final state, step count and duration. Load the session's state (as a recorded step) and pass the real values, so Langfuse spans and audit hooks see the same payload on every runtime.
FU-S5-DBOS-ONAGENTRESUMED-THROW-FAILS-RUN: a throwing onAgentResumed fails the run on DBOS only — open (LOW, owner: the DBOS round)
On runtime-js, the CF DO, Temporal and CF Workflows an error from onAgentResumed is logged and the resume goes on. On DBOS onAgentResumed is the run's start hook, so a throw fails the run (with one onAgentFail), like a throwing onAgentStart (documented in docs/guide/hooks.md). Pick one rule for every runtime and add a parity cell.
FU-HANDLE-RESUME-LOSER-UNTYPED: a getHandle(...).resume() loser surfaces as a failed result, not a typed error — open (LOW, owner: reliability orchestrator (route))
CP-96 gave resumeLoop (the reconnect handle's resume) a seen run, so a stale handle resume now loses its run-start commit and writes nothing. But the handle factory's safety net reports that loss as an AgentResult with status 'failed' and the message, not as a typed AgentAlreadyRunningError / RunStartRejectedError. This is the existing handle-factory behaviour, not new. Surface the typed loser to the handle's result() consumers.
FU-CP96-DURABLE-LOSER-CLASS: the durable runtimes add two loser classes to the CP-96 race's C3 outcome — open (LOW, owner: reliability orchestrator)
CP-96 ruling U3 gives the stale-entry race one caller-facing outcome: the C3 AgentAlreadyRunningError (RunStartConflictError('expected_mismatch')). runtime-js and the CF DO always report it. Temporal, DBOS and CF Workflows report it too when the stale entry's commit lands after the winner started (the run-id CAS or the run-status CAS; proven by the real-engine cp96-entry-interleave probes' "B held at its commit" cases), but their entries re-plan from a fresh session read (U6), so they ADD two classes: a stale entry that plans after the winner finished loses at its plan-time gate with RunStartRejectedError('entry_status_changed'), and one that arrives while the winner's workflow / instance is open is refused before its commit by an owner check (Temporal's executor assertNoLiveOwner: a plain AgentAlreadyRunningError, no conflictCause; DBOS's entry step assertDbosOwnerless and CF Workflows' assertCfwOwnerless: RunStartConflictError('run_active'); CFW's executor awaitPriorInstanceSettled times out with a plain AgentAlreadyRunningError). Documented per runtime in docs/internals/execution-flow.md §Entry gates. Unifying them on C3 needs the executor's seen run status carried into the durable entry (which reverses part of U6: an interrupt-then-resume would then lose), or an equivalent signal that tells the entry's plan-time gate "a newer run happened since the caller looked".
FU-D1-RUN-START-OWNER-GUARD-SPLIT-READ: D1's run-start owner guard decides on an earlier lease / owner read than the one its batch pins — done (LOW)
Closed by: W1 MR2a (cf-w1-cfw-stores-submit; !334, merged as 137b07d57a). The claim guard now also pins the current run's lease_until and owner_token, so the owner guard and the batch decide on the same values. Proven by d1-run-start-owner-pin.cf.test.ts and a d1-guarded-commit integration case. The entry below is the pre-fix diagnosis.
D1StateStore.commitRunStart (store-cloudflare/src/d1-state.ts) resolves the current run in readCommitView, then reads the live runs. applyRunStartRules' owner guard reads view.current.leaseUntil / ownerToken from the FIRST read, while the claim guard's runViewGuardSql pins lease_until / owner_token from the SECOND (CP-96's audit fix pins the current run's status, not its lease or owner). A JS lease renewal that lands between the two reads lets the owner guard pass on the stale (expired) lease while the batch's pin matches the renewed one, so the run-start supersedes a run whose lease was just renewed. The outcome is typed (the renewing run stops with RunSupersededError, superseded), never silent corruption, and the window is two consecutive reads. Fix: in the same claim-guard branch that pins view.current's status, also pin its lease_until IS ? and owner_token IS ? (keeping the bind count constant, e.g. two NULL-safe binds in the no-current-run branch too), and add a d1-guarded-commit race test that renews the lease between the two reads.
FU-D1-UNPARSEABLE-STATE-ROW-UNTYPED: D1 reads an unparseable session row as a retryable failure, not corruption — open (LOW, owner: CF backlog)
Surfaced by the guarded-commit outcome MR's audit. D1StateStore.readStateRow runs JSON.parse on custom_state / output and InterruptContextSchema.parse on interrupt_context, so a corrupt column throws a raw SyntaxError / ZodError. loadState's outer catch wraps it in a retryable D1StateError, and a commit whose pre-read meets it answers CommitNotAppliedError: both say "transient", so a caller retries a read that will fail the same way. Fix: throw InvalidStoredStateError (field, received value) at the parse, as the status column already does, so isPermanentStoreError reads it as corruption. Redis was not audited for the same shape.
FU-DO-STAGING-CORRUPT-ROW-SKIP-VS-REFUSE: a corrupt staging row is skipped by promote but refused by the step commit — open (LOW, owner: CF backlog)
Filed by: MA-31 (Task 2 review). DOStateStore.getStagedChangesSync (used by getStagedChanges and promoteStaging) logs "Skipping corrupt staged change in DO" and SKIPS a staging row that fails StepWritesSchema, so a promote silently drops that tool's state write. saveStateAndPromoteStaging (since MA-31) refuses the same row with a permanent InvalidStoredStateError(sessionId, 'staging.writes', …). D1's getStagedChanges skips such a row the same way ("Skipping corrupt staged change in D1"). Wanted: one rule. The refusal is the safe one (a skip loses the write silently), so promoteStaging should throw InvalidStoredStateError too, and getStagedChanges should decide whether a reader sees an error or the skip; a case per path, on both stores.
FU-REJECT-CAUSES-CORE-HELPER: REJECT_CAUSES / isRejectCause are duplicated in two executors — open (LOW, owner: reliability orchestrator (route))
runtime-temporal and runtime-cloudflare each define the allow-list of RunStartRejectedError causes they revive from a crossed workflow / instance boundary (REJECT_CAUSES, isRejectCause). A new cause must be added in both. Move the list and the guard into core, next to RunStartRejectedError, and import it in both executors (DBOS can use it too).
FU-CONF-CFW-REAP-CALL-BOUND-PARITY: the cfw cell-end reap ignores callBoundMs, signal and progress — open (LOW, owner: conformance owner)
packages/e2e/src/harness/cfw/cell-work.ts now gets the scaled overall bound (reapCellWorkBoundMs(runs), !318), but unlike the Temporal reap (StartedWorkflows.terminateTree) it still ignores callBoundMs, signal and progress. A hung engine read is therefore detected only by the overall bound, a timed-out cfw reap is not aborted (a later call can overlap it), and the cell's timeout report carries no calls / pending-executions detail. Honour the three options in the cfw reap and add the matching unit tests; no cfw code changed in !318.
FU-DO-FOREIGN-ID-TRIPWIRE-NOISE: a DO child's local-root probe trips the DO store's foreign-id warn — fixed (CP-128 MR D)
Fixed: DOStateStore.loadState stays silent for the session's own root and warns for any other foreign id (do-state-store-foreign-session.sqlite.test.ts).
Opened by CP-157 MR 1 (the rootSessionId audit). A CF DO child carries its consumer's rootSessionId. runtime-js localTreeRootOf probes it with loadState to find the root of the child's own tree, and DOStateStore.loadState logs its "a foreign session id was read from this DO's store" tripwire (do-state-store.ts, a warn meant to catch addressing bugs). The count per op is unchanged: the resolver's clearOwnership and checkpoint-ops updateRootOwnership read the same foreign id before. Fix: a store "holds this session" capability the probe can use without a read, or downgrade the tripwire for this probe. Not a correctness issue: the read answers null and the child is its own root.