Skip to content

One result per remote call (remote settlement) ​

This guide covers the release in which a remote sub-agent's ref and its call's result are committed together (Invariant S), and in which the parent settles its remote children by one transitions table (RM-98, RM-99, CP-176, CP-177, CP-178, RM-100, RM-101). It is a minor release of @helix-agents/core (a 0.x package: minor marks an observable behaviour change), of every store package (store-memory, store-redis, store-postgres, store-cloudflare, and the CF Durable Object store in runtime-cloudflare) and of @helix-agents/runtime-js, @helix-agents/runtime-cloudflare and @helix-agents/runtime-temporal.

No migration and no deploy drain. No column, workflow input or instance param changed: the settlement uses the ref columns every store already has (status, completion time and outcome generation).

What was wrong ​

  • A call could get two results (RM-98, RM-99). The parent wrote a remote child's result and its ref in two separate writes. A crash or a failed write between them left the call paired while the ref still read running, and the next resume paired the call again. A store error during reconcile was read as "the child is lost", which appended a second, failed result. A duplicate tool_result is a provider 400 on every later model call, and retry() keeps it, so the session was wedged.
  • Children were settled by too few rules. A child still running past every bound stayed running forever (CP-176). A child whose /start was still committing read "not found" and was settled lost at once (CP-177). A lost child was never stopped (CP-178). A transient status-read failure on a new turn orphaned a child that might have finished (RM-100). A ref whose remote tool was renamed or removed was skipped forever, leaving its call unpaired (RM-101).

The new rule ​

A commit that writes a remote call's result can carry a settle list (MessageCommit.settle, SaveAndPromoteOptions.settle): the store settles those refs inside the same commit, and refuses the whole commit if one is no longer live. Entries and reconcile on runtime-js / the CF DO, CF Workflows' entries and result appends, and recordSubSessionResult (CF Workflows, Temporal) use it. runtime-js's in-run dispatch still writes the ref terminal just before its step commit (a crash in between leaves the call unpaired, and the next resume pairs it with the child's real outcome), and DBOS does not settle atomically yet (below).

ErrorCodeRetryableHTTP
SubSessionRefConflictErrorstate_sub_session_ref_conflictno409
SubSessionRefMissingErrorstate_sub_session_ref_missingno409
RunStartRejectedError('sub_session_ref_conflict')state_run_start_conflictre-planned once409
RemoteChildStatusUnavailableErrorframework_subagent_status_unavailableyes503

The parent decides each child's fate from the transport's getStatus answer alone (core remoteChildVerdict). The child's deadline is startedAt + the runtime's watch budget (its attempts and the retry waits between them) + 60 s. The full table is in the remote agents guide.

What your callers see ​

  • A new turn (or a with_message resume, or a retry with a message) can be refused with a retryable 503 (framework_subagent_status_unavailable) while a remote child's status cannot be read inside its deadline. Nothing is written; send the message again later. A stock useChat puts it in chat.error; parseHelixChatError(chat.error) reads the code and retryAt, the epoch ms after which the child is settled lost and the entry proceeds. The window is the runtime's watch budget + 60 s from the child's start, by design (a child is never declared lost while it may still run); shorten it with the remote tool's timeoutMs / streamRetries / streamRetryBaseMs.
  • A lost child is now interrupted (best-effort transport.interrupt()), and its call gets the typed framework_subagent_lost result. cause gains deadline_exceeded and tool_unavailable.
  • An interrupted remote child is settled as a typed failure on resume, instead of being left running.
  • A child whose producer does not know it yet is still awaited until its deadline, unless its start was confirmed (the ref records the child's stream).
  • CF Workflows keeps a replaced (too-large) child's true completed status on its ref at an entry, as runtime-js and the CF DO do.

Removed: core's lostRemoteChildDetail(error, ref, timeoutMs, now). Use remoteChildVerdict(read, ref, config, now), which takes the transport read's outcome (a status or an error) and returns the whole verdict.

If you implement a store ​

Apply settle inside the commit's atomic step with core settleRefusal(sessionId, settle, actual), where actual is the ref's status and outcomeGeneration as the commit reads them, on appendMessages (append and run-start commits) and saveStateAndPromoteStaging. Run the contract suite: subSessionRefOperationTests (part of stateStoreContractTests).

If you write your own loop ​

Pass settle: remoteSettlementsFor(rows, await store.getSubSessionRefs(sessionId), Date.now()) on any commit whose rows carry a remote child's result. Treat a SubSessionRefConflictError as "another writer recorded this call's result": do not append it again. Read a remote child's fate with remoteChildVerdict, never from a store error. assertToolResultPairing(messages) checks a transcript has one result per call.

Not yet on DBOS and Temporal ​

DBOS writes a remote child's terminal ref before its workflow appends the result (FU-DBOS-REMOTE-SETTLE-ATOMIC). Temporal and DBOS heal a still-running remote child's trailing call not_executed at the next entry without settling its ref (FU-TEMPORAL-REMOTE-HEAL-SETTLE). Temporal's recordSubSessionResult path settles atomically.

Released under the MIT License.