Remote Agents

Remote agents let a parent agent delegate tasks to agents running on a separate HTTP service. This enables microservice architectures, cross-runtime orchestration, and security boundaries between agents.

When to Use Remote Agents

Use remote agents when you need:

• Different services — Specialist agents deployed as independent services
• Different runtimes — A Temporal orchestrator delegating to agents on an Express server
• Security boundaries — Isolate agents that handle sensitive data or credentials
• Independent scaling — Scale specialist agents separately from the orchestrator
• Different models — Run each agent with the most appropriate LLM

For agents that run in the same process, use local sub-agents instead.

Architecture Overview

sequenceDiagram
    participant Parent as Parent Agent
    participant Transport as HttpRemoteAgentTransport
    participant Server as AgentServer (Express)
    participant Child as Child Agent

    Parent->>Transport: Tool call (subagent__researcher)
    Transport->>Server: POST /start
    Server->>Child: Execute agent
    Server-->>Transport: { sessionId, streamId }
    Transport->>Server: GET /sse?sessionId=xxx
    loop Streaming
        Child-->>Server: Stream chunks
        Server-->>Transport: SSE events
        Transport-->>Parent: Proxy chunks to parent stream
    end
    Server-->>Transport: SSE end event
    Transport-->>Parent: Emit subagent_end + tool_end on parent stream
    Transport-->>Parent: Tool result (agent output)

The runtime emits a tool_end chunk on the parent's stream after the remote sub-agent completes (immediately following subagent_end). This is critical for AI SDK consumers — without tool_end, the parent's subagent__<name> UI tool part stays stuck in state: 'input-available'. See Sub-agent chunk ordering for the exact event sequence and rationale.

Setting Up the Server

The @helix-agents/agent-server package provides AgentServer — an HTTP server for hosting agents remotely.

Install

npm install @helix-agents/agent-server express

Configure AgentServer

import { AgentServer, createHttpAdapter, createExpressAdapter } from '@helix-agents/agent-server';
import { JSAgentExecutor } from '@helix-agents/runtime-js';
import { VercelAIAdapter } from '@helix-agents/llm-vercel';
import { InMemoryStateStore, InMemoryStreamManager } from '@helix-agents/store-memory';
import { ResearcherAgent } from './agents/researcher.js';
import { SummarizerAgent } from './agents/summarizer.js';

const stateStore = new InMemoryStateStore();
const streamManager = new InMemoryStreamManager();
const executor = new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter());

const agentServer = new AgentServer({
  // Registry of agents this server can run
  agents: {
    researcher: ResearcherAgent,
    summarizer: SummarizerAgent,
  },
  stateStore,
  streamManager,
  executor,
});

Workspace Wiring

If any of the registered agents declare workspaces, the underlying executor MUST be configured with matching workspaceProviders. The AgentServer validates this at construction:

import { CloudflareFileStoreWorkspaceProvider } from '@helix-agents/runtime-cloudflare/filestore';
import { InMemoryWorkspaceProvider } from '@helix-agents/workspace-memory';

const executor = new JSAgentExecutor(stateStore, streamManager, llm, {
  workspaceProviders: new Map([
    ['in-memory', new InMemoryWorkspaceProvider()],
    [
      'cloudflare-filestore',
      new CloudflareFileStoreWorkspaceProvider({
        /* ... */
      }),
    ],
  ]),
});

const agentServer = new AgentServer({
  agents: {
    researcher: ResearcherAgent /* declares a workspace with kind 'cloudflare-filestore' */,
  },
  stateStore,
  streamManager,
  executor,
  // Strict-mode validation: declare which provider kinds are wired.
  // Mismatches between agent.workspace and this list throw at construction.
  workspaceProviderKinds: ['in-memory', 'cloudflare-filestore'],
});

Two validation modes:

• Strict (workspaceProviderKinds set): construction throws an AgentServerError with code 'WORKSPACE_WIRING' if any registered agent's declared workspace provider kinds aren't covered. The error names the agent and the missing kind(s).
• Soft (workspaceProviderKinds unset): construction succeeds, but a logger.warn fires when at least one registered agent declares a workspace. The runtime still catches missing providers at LLM tool-call time via WorkspaceFailedError, but you'll see them mid-execution rather than at startup.

Prefer strict mode for production deployments where the wiring is known statically; soft mode is fine for testing / dynamic agent registration.

Wire Up Express

import express from 'express';

const app = express();
app.use(express.json());

// Health check
app.get('/health', (_req, res) => {
  res.json({ status: 'ok', agents: ['researcher', 'summarizer'] });
});

// Mount the agent endpoints at root
app.use('/', createExpressAdapter(createHttpAdapter(agentServer)));

app.listen(4000, () => {
  console.log('Remote Agent Service listening on http://localhost:4000');
});

This exposes the executor routes a RemoteAgentTransport speaks — /start, /resume, /sse, /status, /snapshot, /usage, /interrupt, /abort — plus /submit-tool-result and /workspace. See the HTTP Protocol Reference below for the full list.

TIP

For non-Express frameworks, use createHttpAdapter(agentServer) directly — it returns a generic AgentHttpHandler function you can adapt to any HTTP framework. See the API reference for details.

Creating Remote Sub-Agent Tools

On the client side, create tools that delegate to the remote server.

Configure the Transport

import { HttpRemoteAgentTransport } from '@helix-agents/core';

const transport = new HttpRemoteAgentTransport({
  url: 'http://localhost:4000',
  // Optional: static or dynamic headers
  headers: { Authorization: 'Bearer my-api-key' },
  // Optional: retry config for reads and stop requests (defaults: 3 retries, 1s base delay).
  // /start and /resume are never re-sent (see "Entries are sent once" below).
  maxRetries: 3,
  retryBaseDelayMs: 1000,
});

Create Remote Sub-Agent Tools

import { defineAgent, createRemoteSubAgentTool } from '@helix-agents/core';
import { z } from 'zod';

const researcherTool = createRemoteSubAgentTool('researcher', {
  description: 'Delegate research to a remote specialist agent',
  inputSchema: z.object({
    query: z.string().describe('The research query'),
  }),
  outputSchema: ResearcherOutputSchema,
  transport,
  remoteAgentType: 'researcher', // Must match key in server's agents registry
  timeoutMs: 120_000, // 2 minute timeout
  streamRetries: 3, // Retry on stream drop (default: 3)
  streamRetryBaseMs: 100, // Base delay between retries (default: 100ms)
  maxFoldStateBytes: 262_144, // Cap on the child state folded into the parent (default)
  resumeDeadProducer: false, // Opt-in blind resume of a dead Node producer (default: off)
});

const summarizerTool = createRemoteSubAgentTool('summarizer', {
  description: 'Delegate summarization to a remote specialist agent',
  inputSchema: z.object({
    text: z.string().describe('The text to summarize'),
  }),
  outputSchema: SummarizerOutputSchema,
  transport,
  remoteAgentType: 'summarizer',
  timeoutMs: 60_000,
});

Two optional fields govern what comes back from a completed child:

• maxFoldStateBytes (default 262_144) — the maximum serialized size of the child's terminal customState folded into the parent (surfaced to hooks as AfterSubAgentPayload.childCustomState). Over the cap, the state fold is skipped with a warning; the output and usage folds are unaffected. The cap exists because the fold rides through Temporal activity results and DBOS step results, which have their own payload limits.
• resumeDeadProducer (default false) — opt in to a single blind transport.resume() when the producer is observed dead on both sides of a zero-progress stream attempt (status: 'running' with isExecuting: false, no chunks received), after which the dispatch retries to attach. Off by default because a blind resume is wrong for multi-replica producers; without it, the dispatch fails fast with the producer-dead failure reason instead of burning the retry budget.

Output re-validation

The child's end.output is re-validated by default against the tool's outputSchema (a safeParse on the consumer side). A child whose output doesn't match the schema surfaces a structured tool error to the parent LLM instead of passing an off-schema object through. There is no signature change and no flag to set — if you rely on a remote child returning something looser than its declared outputSchema, audit that schema before upgrading.

Use in a Parent Agent

const OrchestratorAgent = defineAgent({
  name: 'orchestrator',
  description: 'Orchestrates research via remote specialist agents',
  outputSchema: OrchestratorOutputSchema,
  tools: [researcherTool, summarizerTool],
  systemPrompt: `You are a research orchestrator.
1. Use the researcher to gather information
2. Use the summarizer to distill findings
3. Call __finish__ with your final output`,
  llmConfig: { model: openai('gpt-6-luna') },
  maxSteps: 10,
});

The parent LLM sees subagent__researcher and subagent__summarizer as tools — it doesn't know they're remote.

HTTP Protocol Reference

createHttpAdapter mounts two layers of routes — the executor routes (Layer A, what a RemoteAgentTransport speaks) plus the v7 chat handler routes (Layer B) layered on top:

Layer A — Executor routes

Method	Path	Description
POST	/start	Start a new agent execution
POST	/resume	Resume an interrupted agent
GET	/sse	SSE stream of agent events
GET	/status	Get execution status
GET	/snapshot	Checkpoint-pinned snapshot of the session's customState
GET	/usage	Recursive usage rollup for the session
POST	/interrupt	Durable interrupt (records the stop; 202 / 200 / 409)
POST	/abort	Hard stop
POST	/submit-tool-result	Submit a client-tool result or approval response
GET	/workspace	Operator-facing workspace introspection

Layer B — Chat handler routes (v7, wired via chatHandler config)

Method	Path	Description
POST	/chat	Start or continue a chat turn (always streams)
GET	/chat/{id}/stream	Reattach to an in-progress stream after a refresh
POST	/chat/{id}/submit-tool-result	Submit a client-tool result or approval-response
POST	/chat/{id}/interrupt	Durable interrupt for the chat session
POST	/chat/{id}/abort	Abort the current run

The chatHandler config option plumbs handleChatStream through AgentServer({ chatHandler }). Layer B replaces the INTERRUPT_NOT_LOCAL 503 (now removed): a stop is recorded durably in the state store and answered at once: 202 stopping (the run is executing), 200 interrupted / failed (the call resolved it, for example a paused run), 409 state_not_stoppable (no live run) or 404 state_session_not_found.

POST /start

Start a new agent execution.

Request:

{
  "sessionId": "session-abc-123",
  "agentType": "researcher",
  "message": "{\"query\":\"Research TypeScript benefits\"}",
  "state": { "query": "Research TypeScript benefits" },
  "metadata": {}
}

The message field contains the JSON-serialized tool input from the parent agent's LLM. The remote agent server is responsible for parsing this JSON and constructing the appropriate user message for its agent. The state field carries the same input as a parsed object for initializing the agent's custom state.

Identity forwarding. Three optional request fields carry the parent's identity across the boundary:

Field	Type	Purpose
userId	string	User the child session is attributed to
tags	string[]	Parent tags, copied onto the child session
metadata	Record<string, string>	Free-form key/value metadata

Consumer executors forward the parent's values automatically; the producer seeds the child session with them — parity with local spawn (usage attribution, memory entity-scoping, and metadata queries match local children). metadata is also the escape hatch for threading trace ids across the boundary.

Response (200):

{
  "sessionId": "session-abc-123",
  "streamId": "stream-xyz",
  "runId": "run-456"
}

Starting a session that is already live returns HTTP 409 with { code: 'ALREADY_RUNNING', streamId }. This is a typed, expected response: transports surface it as a typed error (RemoteAgentAlreadyRunningError) and every consumer treats it as attach (proceed to /sse with the returned streamId), not failure.

The transports send /start once per call (below): a lost response rejects with RemoteEntryOutcomeUnknownError, never a silent retry. Every runtime's remote dispatch sends its /start with the delegation's request key (X-Helix-Request-Key: the call's minted dispatch id, see Child identity) and re-sends a lost answer with the same key, which the child answers from the key (D-2, below).

ALREADY_COMPLETED (CP-78)

A keyed /start that repeats a delegation whose run already finished, and is still the child's current run, is answered attached: the dispatch reads that run's stream to its end frame and gets the child's output. 409 ALREADY_COMPLETED with errorCodestate_request_already_completed means a later run took the child session over (only a newer parent run can do that, so the old delegation is moot): the dispatch fails the call typed and non-retryable. A keyless /start that meets a completed or failed session still gets 409 ALREADY_COMPLETED (with the server's errorCode, for example state_not_resumable), thrown as a typed HelixError. See CP-78 in the conformance findings.

POST /resume

Resume an interrupted or paused agent, optionally with a new message.

Request:

{
  "sessionId": "session-abc-123",
  "message": "Continue with more detail"
}

GET /sse

Subscribe to agent events via Server-Sent Events.

Query params:

• sessionId (required) — Session to stream
• fromSequence (optional) — Resume from this sequence number (skips already-seen events)

Event format:

id: 42
event: chunk
data: {"sequence":42,"chunk":{...}}

event: end
data: {"status":"completed","output":{...},"state":{...},"usage":{...}}

event: fail
data: {"error":"something failed","errorDetail":{"message":"something failed","code":"..."},"recoverable":false}

event: truncated
data: {"floor":17,"errorDetail":{"message":"...","code":"TRUNCATED"}}

: heartbeat

/sse speaks stream wire contract v2 (the v1 error frame is fail; see the upgrade guide). Every body ends with exactly one terminal frame — end, fail or truncated — then closes. The Cloudflare DO producer also writes non-terminal markers (caught-up { head } after the replay, rewound after a cleanup) that the transports skip. The server sends a : heartbeat comment every 15 seconds to keep the connection alive.

A body with no terminal frame is a drop

A body that ends with no terminal frame (a proxy cut the connection, the producer crashed or evicted the reader) is never a clean end: both transports throw the retryable StreamDropError (remoteSessionId, lastSequence), unless your own signal aborted the read. Recover by re-attaching with stream(sessionId, { fromSequence: lastSequence }), never by re-sending /start. An agent-server producer whose source ended without a terminal event writes a recoverable fail with errorDetail.code transport_stream_drop instead of a synthesized end.

The end frame

Both producer servers (AgentServer and the Cloudflare DO) emit the same end frame: { status: 'completed' | 'paused' | 'interrupted', output?, state?, usage? }. The DO's legacy finalOutput field is gone from this frame — see the repo-root behavior-changes.md for the migration.

• Only status: 'completed' may be folded as success. paused / interrupted frames must route through /status-based recovery — a paused child that folds as success reports output: null to the parent LLM as if it had finished.
• state and usage are emitted only after the producer's terminal persist committed (persist-before-emit), so either may be absent. Fall back to GET /snapshot / GET /usage when they are.
• status is required: a frame with no status is malformed — the transport emits a recoverable fail event (routed through drop-recovery / getStatus), never a silent 'completed'.

The fail frame

fail carries { error, errorDetail?, recoverable }. A recoverable: true fail is treated as a drop (re-attach from the cursor, or the post-drop /status recovery). A non-recoverable fail fails the sub-agent call with the frame's errorDetail kept verbatim on the failed result (RemoteAgentFailedError carries it as its optional third argument), on every runtime: each dispatch maps terminal events through core remoteTerminalOutcome.

fromSequence is exclusive

fromSequence is exclusive (sequence > fromSequence) — pass snapshot.streamSequence directly, with no +1 adjustment. (Contrast: the store-level getChunksFromStep is inclusive despite the similar name.)

If fromSequence is below the stream's retention floor, the server emits a single terminal truncated { floor, errorDetail } frame (errorDetail.code: 'TRUNCATED') instead of silently serving the survivors. Refetch GET /snapshot and resume from its checkpoint-pinned streamSequence. (The remote dispatch does not re-snapshot yet: it fails the sub-agent call with the TRUNCATED detail.)

Malformed frames

Malformed SSE frames surface as errors. They are no longer defaulted to sequence: 0 — that default made a corrupt frame invisible, because the consumer's <= lastSequence dedup silently dropped it. Both transports decode /sse with core decodeTransportEvents, which yields a recoverable fail event for an unparseable frame, which routes the consumer into its normal drop-recovery path (reconnect from the last good sequence).

GET /status

Get the current execution status.

Response:

{
  "sessionId": "session-abc-123",
  "runId": "run-456",
  "runStatus": "running",
  "status": "running",
  "stepCount": 3,
  "isExecuting": true,
  "streamId": "stream-xyz",
  "latestSequence": 42,
  "awaitingClientTool": false
}

Status values: running, completed, failed, interrupted, paused.

runId and runStatus describe the session's current run record (both absent when the store reports none). runStatus is one of running, completed, failed, interrupted, superseded, suspended_client_tool, suspended_awaiting_children or suspended_step_partial. A superseded run (another entry took the session over) is a typed stop, not a failure: it never changes status or error. The parser (remoteStatusResponseSchema()) maps a runStatus it does not know, such as one a newer peer added, to 'unknown' (RemoteRunStatus) instead of rejecting the response; treat 'unknown' as "not known to be live".

RemoteStatusResponse.state was removed

/status no longer carries a state field. Use GET /snapshot — it is the only route that returns a (state, sequence) pair written atomically, so it is the only sound baseline for state reconstruction. See the repo-root behavior-changes.md for the migration.

isExecuting is honest, not an echo. On the Node AgentServer it is derived from the server's live execution handles — this replica's truth — rather than a status === 'active' echo of persisted state. Per-replica caveat: a session executing on another replica of a multi-replica deployment also reports isExecuting: false here, so consumers must not escalate on a single false. status: 'running' with isExecuting: false across several polls means the producer died; see producer-liveness in the Cross-Service Remote Agents guide.

awaitingClientTool is a required boolean: true when the session is suspended awaiting a client-executed tool result. A parent dispatching this agent as a remote sub-agent uses it to fail fast — see the limitation below.

Limitation: client-executed and approval-gated tools inside remote sub-agents

Client-executed and approval-gated tools inside a remote child (CP-157). On runtime-js, the CF Durable Object and CF Workflows, a remote child that pauses on such a call PARKS its parent when the root can answer it. That means its /status lists the tree's unanswered calls (pendingCalls, not truncated) and the transport implements submitToolResult. The call renders in the root UI and is answered there; the root's submit is forwarded to the producer, and the resumed parent re-attaches the child. See the cross-service guide. Every other case keeps the fail-fast: a parent on Temporal or DBOS (FU-CP157-TEMPORAL-DBOS-REMOTE-PARK), an older producer without pendingCalls, a truncated listing, or a transport without submitToolResult. The parent then fails with RemoteSubAgentClientToolUnsupportedError (failureReason: 'client-tool-unsupported') when the child's own call is client-executed, or with CP-45's typed drop below. GitLab #107.

GET /snapshot

Fetch a checkpoint-pinned snapshot of the session's own customState. This is the baseline every state-reconstruction flow starts from — initial state is never on the wire.

Query params:

• sessionId (required)

Response — RemoteSnapshotResponse:

{
  sessionId: string;
  streamId: string;             // the sequence space this snapshot is aligned to
  state: Record<string, unknown>; // THIS session's own customState (possibly projected)
  streamSequence: number;       // checkpoint-pinned sequence; pass DIRECTLY as
                                // `fromSequence` (exclusive). -1 = no patch
                                // continuation — never compute a fromSequence from it
  oldestRetainedSequence: number; // retention floor; 0 = full log retained
  stepCount: number;
  checkpointId?: string;        // optional
}

A non-terminal session is served from the latest checkpoint row: state and streamSequence come out of the same atomically-written record, which is the only (state, sequence) pairing that is sound (a live two-read pairing either skips the in-flight step's patches forever or double-applies them). A terminal session is served the final state with streamSequence: -1 — the snapshot pins a moment, not a position, and there are no further patches to continue from. -1 is also returned when no checkpoint exists yet or when alignment is otherwise unavailable, so treat it uniformly as "snapshot-only": do not derive a fromSequence from it.

An unknown session returns 404 { code: 'NOT_FOUND' } — the response schema has no null branch.

See the Cross-Service Remote Agents guide for the full reconstruction contract (attach-at-S, patch filtering by agentId, re-snapshot triggers).

GET /usage

Fetch the session's usage rollup.

Query params:

• sessionId (required)

Response — RemoteUsageResponse:

{
  sessionId: string;
  rollup: UsageRollup;
}

The rollup is getRollup(sessionId, { includeSubAgents: true }) computed producer-side — the recursive fold, so the remote agent's own grandchildren are already included by construction. A running session returns the partial rollup as-of-now; once terminal it is at least as of the end event.

The route is served only when the producer has a usage store wired: on Node, set AgentServerConfig.usageStore; on the Cloudflare DO producer it is built in (the DO falls back to its own SQLite-backed DOUsageStore). Without one, /usage responds 404 { code: 'FEATURE_UNAVAILABLE' }. Consumers degrade on this: they skip the usage fold and log a warning rather than failing the tool call.

The store powers /usage and the consumer's getUsage() fallback fold — it does not put usage on the SSE end frame. Only the Cloudflare DO producer attaches end.usage today; the Node/JS, Temporal, DBOS, and Cloudflare Workflows producers never do, regardless of whether a usage store is wired.

POST /interrupt

Soft stop — the agent can be resumed later. It is a parent stopping its child, so it must name the delegation it stops (CP-217): the child's dispatch (dispatchId, checked against the child's stored binding), or a companion's consult (parentToolCallId, checked against its run's recorded consult). A body that names neither, or both, is a 400 INVALID_REQUEST. A user's own stop of a root session is /chat/{id}/interrupt, which takes no target.

Operators (an admin UI, a runaway cleanup) stop any session by id, a producer-side child included, without naming a delegation:

• agent-server: POST /chat/{id}/interrupt / POST /chat/{id}/abort (body { reason? }; the authenticate hook sees the action chat-interrupt / chat-abort; no chatHandler needed), or server.interruptAgent(sessionId) / server.abortAgent(sessionId) in process.
• CF Durable Objects: the session DO's native POST /interrupt / POST /abort (runId optional: absent stops the session's current run), reached through your worker's own route to the DO, the ai-sdk Cloudflare chat handler's stop route, or a DO client handle. The remote mount (createRemoteAgentWorkerHandler, /agents/<type>/…) is the parent's route and requires the delegation.

Request:

{
  "sessionId": "session-abc-123",
  "reason": "User requested pause",
  "dispatchId": "d_4f1c…"
}

Answers (core stopAnswerResponse): 202 { status: 'stopping', runId, kind }; 200 { status: 'interrupted' | 'failed', runId }; 409 state_not_stoppable (no live run); 409 state_stop_target_mismatch (no run of that delegation runs here: nothing written). A parent reads both 409s as a confirmed stop.

POST /abort

Hard stop — the agent cannot be resumed. Same body and answers as /interrupt.

Request:

{
  "sessionId": "session-abc-123",
  "reason": "Timeout exceeded",
  "dispatchId": "d_4f1c…"
}

Error Responses

All endpoints return errors in this format:

{
  "error": "Agent type not found: unknown-agent",
  "code": "NOT_FOUND"
}

Code	HTTP Status	Description
NOT_FOUND	404	Agent type or session not found
ALREADY_RUNNING	409	Session is already executing — treat as attach, not failure
ALREADY_COMPLETED	409	Session has already completed or failed
FEATURE_UNAVAILABLE	404	Route exists but the capability isn't wired (e.g. no usage store)
INVALID_REQUEST	400	Missing required fields
INTERNAL_ERROR	500	Server error
state_run_superseded	409	POST /submit-tool-result for a run another entry has superseded
framework_completion_strategy_unavailable / framework_completion_schema_unsupported	400	outputSchema agent whose model the adapter cannot finish (early refusal); nothing written

ALREADY_RUNNING responses additionally carry streamId so a consumer can attach to the live stream without an extra /status round-trip. A /start that passed the existence check but lost the executor's run-start commit to a concurrent /start answers the same 409, with errorCode: 'state_already_running' added. A 500 caused by a typed framework error (for example a snapshot read's state_history_incomplete) keeps code: 'INTERNAL_ERROR' and adds the framework code as errorCode, plus cause when the error carries one (for example corrupt_row). POST /start is the exception: an entry that failed typed (a HelixError thrown by the executor, or a handle whose entry failed before its run-start commit, preCommitFailure) answers exactly { error, code } with core typedErrorResponse's status (400 refusal, 409 non-retryable state_*, 503 retryable while the entry's run does not exist, else 500), as the CF Durable Object's /start does. HttpRemoteAgentTransport throws that body as a typed HelixError (below).

The transports map these onto typed errors so consumers can discriminate with instanceof instead of string matching:

Answer	Typed error
envelope ALREADY_RUNNING	RemoteAgentAlreadyRunningError
envelope NOT_FOUND	RemoteAgentNotFoundError
envelope FEATURE_UNAVAILABLE	RemoteAgentFeatureUnavailableError
any other envelope with an errorCode, or any envelope answered 503	a HelixError with the errorCode and cause; retryable exactly on a 503
a typed { error, code, cause?, retryable? } body (agent-server /start, /resume)	a HelixError with that code and cause; retryable from the body, else on 503
anything else	a generic Error naming the status and body

Both HttpRemoteAgentTransport and the CF DOStubTransport use this one mapping (core remoteProtocolError) on every route. For start() and resume() it goes through core remoteEntryProtocolError, which turns a 5xx that is not the host's typed retryable answer into RemoteEntryOutcomeUnknownError (Entries are sent once). A typed error keeps only the code the server sent; no code is inferred from the message. A CF Durable Object producer's /subagent/:type/* routes put the typed code in errorCode on every error answer: an unknown agent type is NOT_FOUND with errorCode: 'framework_agent_not_found' (an AgentResolutionError on the producer), a busy session ALREADY_RUNNING, a refused resume ALREADY_COMPLETED (completed session) or INVALID_REQUEST with errorCode: 'state_not_resumable', and a rejected run start carries its reason as cause. See the DO drive pipeline upgrade guide.

A response body that fails schema validation at the transport boundary throws RemoteProtocolValidationError — peer schema drift surfaces as a structured error rather than a downstream undefined field access.

TRUNCATED is not an HTTP error: it arrives as the errorDetail.code of the terminal truncated { floor } frame on /sse (see above), because that is the only place a retention-floor violation can be detected.

How It Works

Step-by-step execution flow when a parent agent calls a remote sub-agent:

1. Parent LLM calls tool — e.g., subagent__researcher({ query: "TypeScript benefits" })
2. Runtime detects remote tool — The tool is marked with _isRemoteSubAgent: true
3. Transport calls POST /start — Sends the message, agent type, and the parent's identity (userId/tags/metadata) to the remote server. A 409 ALREADY_RUNNING means "already live" and is handled as an attach
4. Transport connects GET /sse — Subscribes to the event stream for the session, seeded with fromSequence when resuming
5. Remote agent executes — The AgentServer runs the agent using its configured executor (JSAgentExecutor is the standard choice)
6. Chunks stream back — SSE events flow through the transport back to the parent, stamped with remoteSource provenance
7. End frame arrives — On status: 'completed', the output is re-validated against the tool's outputSchema and returned as the tool result; the child's terminal customState and usage rollup fold into the parent (from end.state / end.usage, else GET /snapshot / GET /usage). paused / interrupted route through /status-based recovery instead
8. Parent continues — The parent LLM receives the result and proceeds

Cross-Runtime Behavior

Remote sub-agent tools are first-class constructs across all runtimes:

Runtime	Behavior
JS	Intercepts remote tool calls with enhanced handling: stream proxying to parent, SubSessionRef tracking, timeout management via AbortSignal, internal stream retry loop on drops, interrupt propagation, reconnection on resume
Temporal	Dedicated executeRemoteSubAgentCall activity with deterministic session IDs, crash recovery via transport.getStatus(), heartbeat-based reconnection, stream proxying, interrupt propagation, StreamDropError for activity retry
DBOS	Dedicated executeRemoteSubAgentCall step (Postgres-backed workflow replay) with deterministic session IDs, crash recovery via transport.getStatus() on step retry, stream proxying, interrupt propagation, StreamDropError for step retry, ref-cadence reconnection (seed from the ref at step entry; persist every N chunks)
Cloudflare	Dedicated executeRemoteSubAgentCall step with deterministic session IDs, crash recovery, stream proxying, interrupt propagation via abort-check interval, timeout enforcement, StreamDropError for step retry

TIP

All runtimes provide stream proxying, SubSessionRef tracking with remote metadata, and sub-agent lifecycle hooks (beforeSubAgent/afterSubAgent). Remote sub-agents are routed through a dedicated execution path separate from regular tool calls, with subagent_start/subagent_end stream events.

Cloudflare DO and Local Sub-Agents

In the Cloudflare DO runtime, createSubAgentTool() (local sub-agents) are transparently converted to remote sub-agent calls using DOStubTransport — a RemoteAgentTransport implementation that routes to sibling DO instances instead of an external HTTP service. This means HttpRemoteAgentTransport is only needed when crossing service boundaries; within a single DO namespace, subAgentNamespace handles everything. See Sub-Agents in the DO Runtime.

Streaming Integration

Remote agent events appear in the parent stream using the same subagent_start/subagent_end pattern as local sub-agents:

for await (const chunk of parentStream) {
  switch (chunk.type) {
    case 'subagent_start':
      console.log(`Remote agent started: ${chunk.subAgentType}`);
      break;

    case 'text_delta':
      // Could be from parent or remote agent
      console.log(`[${chunk.agentType}]`, chunk.delta);
      break;

    case 'tool_start':
      // Tools used by the remote agent
      console.log(`[${chunk.agentType}] Tool: ${chunk.toolName}`);
      break;

    case 'subagent_end':
      console.log(`Remote agent finished: ${chunk.subAgentType}`);
      break;
  }
}

This means frontends don't need to distinguish between local and remote sub-agents — the stream protocol is identical.

Transport Configuration

Static Headers

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: {
    Authorization: 'Bearer my-api-key',
    'X-Tenant-Id': 'tenant-123',
  },
});

Dynamic Headers

For tokens that need to be refreshed:

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await getAccessToken()}`,
  }),
});

Retry Policy

The transport retries the reads and stop requests (/status, /snapshot, /usage, /interrupt, /abort) with exponential backoff:

• 5xx errors — Retried with backoff
• 4xx errors — Not retried (client error)
• Network errors — Retried with backoff

maxRetries does not apply to /start and /resume: they are sent once (next section).

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  maxRetries: 5, // Default: 3
  retryBaseDelayMs: 2000, // Default: 1000ms
});

Delay formula: baseDelay * 2^attempt (1s, 2s, 4s, ...).

Entries are sent once

HttpRemoteAgentTransport and the Cloudflare DOStubTransport send /start and /resumeexactly once (CP-95). A re-sent entry whose first response was lost could start the remote run twice, or append the resume's message twice: a finished Cloudflare DO child would run a whole new turn. So an entry with no definite answer is not retried:

The entry got	The transport throws
a 2xx	nothing: the parsed response
a thrown request (the connection failed or was lost; the request may still have arrived)	RemoteEntryOutcomeUnknownError
a 5xx whose body is the host's typed retryable answer (a Helix host's 503, Zod-parsed)	the host's own typed, retryable HelixError (still not re-sent by the transport)
any other 5xx: an untyped body (a load balancer or proxy 503 page) or a typed non-retryable one	RemoteEntryOutcomeUnknownError, with the HTTP status and the mapped answer as cause
a 4xx	the typed refusal (Error Responses); a definite answer

A typed retryable answer is safe to surface as retryable because a Helix host never answers "retryable" once the entry's input is committed (DI-39): the input was not taken. An untyped 503 proves nothing about the input, so it is an unknown outcome, not a retryable one.

RemoteEntryOutcomeUnknownError (code: 'transport_remote_entry_outcome_unknown', entry: 'start' | 'resume', status?) is non-retryable at the transport. A re-send is safe only with a request key (D-2), so every runtime's remote dispatch keys its entries:

• /start carries the call's dispatch id (d_<uuid>, minted with the parent's step row and persisted on its tool call as dispatchId) and the delegation's binding. After an unknown outcome the dispatch re-sends the same request at most twice (core startRemoteChild, KEYED_START_RESENDS = 2). The child answers a repeat from its key: it attaches to (or replays) the run the first request started, or starts it if the first never arrived. So the child never runs twice, and the delegation gets the child's output. A /start whose answer is still lost after the re-sends fails with the error's detail.
• The dead-producer blind resume (resumeDeadProducer) carries dispatchResumeRequestKey(dispatchId) (hashed, domain-separated from the start key) and the binding, so a repeated resume is one entry.

Both keys come from the call's minted dispatch id (valid request keys, never derived from the parent run or the tool call id).

Custom fetch

HttpTransportConfig.fetch?: typeof fetch replaces globalThis.fetch for every request the transport issues (the POSTs, the /status, /snapshot and /usage GETs, and the SSE stream). Use it for Cloudflare service bindings, Miniflare's dispatchFetch in tests, and Node-side proxies:

const transport = new HttpRemoteAgentTransport({
  url: 'https://producer', // hostname is arbitrary over a binding
  fetch: env.PRODUCER.fetch.bind(env.PRODUCER),
});

For Cloudflare consumers, serviceBindingTransport from @helix-agents/runtime-cloudflare wraps this pattern — see the Cross-Service Remote Agents guide.

Custom Transports

RemoteAgentTransport has eight required methods:

interface RemoteAgentTransport {
  start(request: RemoteStartRequest): Promise<RemoteStartResponse>;
  resume(request: RemoteResumeRequest): Promise<RemoteStartResponse>;
  stream(
    sessionId: string,
    options?: { fromSequence?: number; signal?: AbortSignal }
  ): AsyncIterable<TransportEvent>;
  getStatus(sessionId: string): Promise<RemoteStatusResponse>;
  getSnapshot(sessionId: string): Promise<RemoteSnapshotResponse>;
  getUsage(sessionId: string): Promise<RemoteUsageResponse>;
  /** A parent's stop of its child, naming the delegation (CP-217): { reason, target }. */
  interrupt(sessionId: string, stop: RemoteStopRequest): Promise<void>;
  abort(sessionId: string, stop: RemoteStopRequest): Promise<void>;
  /** CP-157: answer a pending call of the session's tree (optional). */
  submitToolResult?(sessionId: string, submit: SubmitToolResult): Promise<SubmitToolResultResponse>;
}

getSnapshot and getUsage are required on the interface — a custom transport written against the previous six-method interface will not typecheck until it implements them. HttpRemoteAgentTransport (packages/core/src/transport/http-transport.ts) is the reference implementation: a GET against /snapshot / /usage, with the response validated by remoteSnapshotResponseSchema() / remoteUsageResponseSchema() and non-2xx bodies mapped onto the typed errors above. A custom transport should decode its /sse body with core decodeTransportEvents(body, { sessionId, fromSequence, signal }) (it throws StreamDropError for a body with no terminal frame) and map an entry's non-2xx answer with core remoteEntryProtocolError, sending each entry once.

If your transport genuinely cannot serve one of them, throw RemoteAgentFeatureUnavailableError — consumers treat that (and RemoteAgentNotFoundError) as feature-unavailable and degrade gracefully: the corresponding state or usage fold is skipped with a warning rather than failing the tool call.

Stream Recovery

Remote sub-agent calls use SSE for streaming. Network interruptions can drop the SSE connection mid-execution. The framework provides defense-in-depth recovery that varies by runtime:

JS Runtime — Internal Retry Loop

The JS runtime retries stream drops internally without surfacing errors to the caller. When a stream drops (the body ends with no terminal frame, so the transport throws StreamDropError):

1. Checks transport.getStatus() to see if the remote agent completed, failed, or is still running
2. If still running, reconnects with fromSequence to resume from the last received chunk
3. Retries up to streamRetries times with exponential backoff (streamRetryBaseMs * 2^attempt)

Configure retry behavior per tool:

createRemoteSubAgentTool('researcher', {
  // ...
  streamRetries: 5, // Default: 3, max: 50 (0 disables retries)
  streamRetryBaseMs: 200, // Default: 100ms
});

Temporal Runtime — Activity Retry

The Temporal runtime throws StreamDropError on stream drops, which triggers Temporal's built-in activity retry. The heartbeat carries lastSequence so the retried activity can resume from the correct position.

Cloudflare Runtime — Step Retry

The Cloudflare runtime throws StreamDropError on stream drops, which triggers Cloudflare Workflows step retry. The lastSequence is persisted to the SubSessionRef for crash recovery.

Re-attach, never re-start (the D-2 guard)

A retry must not start the child a second time. Every runtime's remote dispatch (core's, used by Temporal; runtime-js; DBOS; Cloudflare Workflows) treats the child as started once either is true:

• transport.getStatus(childSessionId) answers, with any status;
• the sub-session ref carries remote.streamId, which the dispatch writes right after a successful start.

From then on a retry (a Temporal activity retry, a DBOS step retry, a Workflows step retry) re-attaches with transport.stream(childSessionId, { fromSequence }) and never re-POSTs /start. The JS runtime retries a drop inside one tool execution (its stream retry loop always re-attaches), and a re-entered call (e.g. after a parent resume) applies the same rule. Per child status:

• interrupted — a typed failure (failureReason: 'remote-interrupted'), not retried: a stopped child does not continue on its own, and a retry could only re-attach to its ended stream. This holds whatever output the child's /status carries (CP-45): a run writes output only at its own completion and a terminal status is sticky, so a stopped child's output is an earlier turn's, never this call's.
• paused — still a re-attach, and still retried. The D5 contract says a paused child may resume (an end { status: 'paused' } is not success and routes through /status recovery); with the guard a retry only re-attaches, so it can never start the child again. Its output is never read either (CP-45). When the dispatch's retries run out with the child still paused (runtime-js streamRetries, a Workflows step's or a DBOS step's retries, a Temporal activity's), the call fails typed: RemoteChildPausedError, classified transport_stream_drop with cause: 'remote_child_paused' and retryable: true; a retry of the call re-attaches to the same child. Except (CP-157): on runtime-js, the CF DO and CF Workflows, a pause the root can answer parks the parent at once instead (no retries spent), and a failed call interrupts its still-live child first. See the cross-service guide.
• A transport drop (StreamDropError) falls through to the dispatch's post-drop /status recovery, as before.

Reconnect cadence (cross-runtime contract)

Retry only helps if the retried attempt knows where to resume from. Every consumer path therefore obeys the same two-part contract: seed fromSequence from the persisted SubSessionRef.remote.lastSequence at (re)entry, and persist progress back to that ref at a bounded cadence while streaming (every REMOTE_REF_PERSIST_CHUNK_INTERVAL = 25 proxied chunks). Without the seed, a retried attempt re-forwards the child's entire stream; without the bounded persist, the seed is stale by an unbounded amount.

Runtime	Cadence
JS / DO	Seed from the ref; persist between retry attempts and every N chunks
Temporal	Per-chunk activity heartbeat carrying lastSequence, under an explicit heartbeatTimeout (30s). Activity timeouts are derived from the tool's timeoutMs and stream-retry budget, so a healthy child that legitimately runs longer than the default activity timeout is not reported failed. The heartbeat also delivers cancellation and detects a hung SSE connection.
CFW	Per-N-chunk SubSessionRef update; the step retry re-seeds from it
DBOS	Seed from the ref at step entry; persist every N chunks and on every failure path

Every proxied child chunk carries remoteSource { childSessionId, childSequence, streamId } provenance. The parent stream writes each of them once per parent run: every stream manager keeps a high-water per child run ((childSessionId, childRunId)) and drops a re-forwarded chunk it already holds, inside the write (RM-86). That is exact because a forwarder records a chunk as forwarded only after its write resolved, on every runtime (RM-89), so a re-attach never seeds past a chunk the stream lacks. A parent's next run clears the high-water, so specified consumers still dedup on (childSessionId, childSequence) across runs.

Orphan reconciliation and settlement

JS and Cloudflare DO. A parent crash truncates history to the last checkpoint, and the re-rolled LLM makes new tool calls, with new dispatch ids — so the new call names a new child (child_<uuid>), not the one still running producer-side. On resume, the JS executor (which the DO runtime embeds) sweeps remote SubSessionRefs (location: 'remote') that are still running:

One result per call (Invariant S). When an entry, reconcile, or a recorded result (CF Workflows, Temporal) writes a remote call's result, the same store commit settles the call's ref: the commit carries a settle for it (MessageCommit.settle), and the store refuses the whole commit (SubSessionRefConflictError, state_sub_session_ref_conflict) if another writer settled the ref first. So a call never gets a second tool_result. (Two writes remain: runtime-js's in-run dispatch writes the ref terminal just before its step commit, so a crash in between leaves the call unpaired and the next resume pairs it with the child's real outcome; DBOS writes the ref before its workflow appends the result, FU-DBOS-REMOTE-SETTLE-ATOMIC.) That would be a provider 400 on every later model call, and retry() would keep it. A failed store write is never read as a lost child either: only the transport's getStatus answer decides what happened to the child.

The sweep reads each child through the transport and applies one table (core remoteChildVerdict). The child's deadline is the longest the in-run dispatch itself would have watched it, its retry waits included (CP-196), plus 60 s: startedAt + watch budget + 60 s (core remoteChildDeadline). Each runtime states its watch budget from its own retry policy (core RemoteWatchBudget):

• runtime-js and the CF Durable Object (streamRetryWatchBudget): streamRetries + 1 stream attempts of timeoutMs each, each followed by a recovery status read (bounded at 5 s, REMOTE_STATUS_READ_TIMEOUT_MS), the backoff waits between them, min(streamRetryBaseMs × 2^attempt, 5 s), and the pre-check and final status reads (streamRetries defaults to 3, streamRetryBaseMs to 100 ms).
• CF Workflows has no entry-time settle to time: a remote call's result commits with its step (the step waits for the child), so an entry never meets a live remote child whose call committed; a child whose step was LOST is interrupted and settled failed (parent_step_lost) by the next entry, after its run-start commit (see "A step lost with its run" below).

The producer answers	Inside the deadline	Past the deadline
completed	the child's real result	same
failed	the child's failure (with its errorDetail)	same
interrupted	a typed failure ("was interrupted before it completed"), never a stale output	same
running / paused	still owed (see below)	lost: deadline_exceeded
not found (RemoteAgentNotFoundError, a DO's 404)	still owed while the child's /start may still be committing; lost at once (not_found) when its start is confirmed (the ref records the child's stream, or is already terminal)	lost: not_found
a typed, non-retryable refusal	lost: refused:<code>	same
any other read failure (a reset, a timeout, a 5xx)	unreadable: never settled (see below)	lost: status_unreachable

A lost child's call gets a typed failed result (metadata.helixErrorDetail, code framework_subagent_lost, non-retryable, cause as above). Because no run awaits it any more, the child is also interrupted, best-effort (transport.interrupt()). A ref whose remote tool is gone from the agent (renamed or removed) cannot be read or stopped: its call is paired lost (cause: 'tool_unavailable') and the ref is failed.

On a resume (no new message), a child still owed whose call survived truncation is left running. The LLM re-calls the tool with the same tool-call id, and transport.start() re-attaches to the existing child session. A child still owed whose call was truncated away is a genuine orphan: it is interrupted, best-effort, and its ref is marked failed. An unreadable child is left as it is, and the next reconcile reads it again. A call that already has its result is never paired again.

The sweep also takes a ref already recorded completed / failed whose call never got its result (a crash between the ref update and the result's append): the child's real outcome comes from getStatus, never a synthetic not_executed result. (On a new turn, a child whose status read fails inside its deadline is never stopped or healed: the entry is refused, retryably, with nothing written — the RM-100 rule "a child whose status cannot be read inside its deadline refuses any entry carrying a message" under the entry rules below. Past its deadline it is lost.)

The parent never reads a remote child's state from its own store (on the DO, that store holds only the parent); remote refs are reconciled through the transport only. A ref is remote by its location: 'remote' (core isRemoteSubSessionRef), never by its stream identity (ref.remote): the parent registers the ref before the child's /start and records the identity only once the child answers, so a child whose /start never answered (the parent died with the request in flight) is still reconciled, and, gone, settled failed. Its call is not in the parent's history (a step's rows commit after its sub-agents return, so the recovered run re-runs the model), so no result row is written for it.

An entry that carries a new message settles the parent's remote children first, so a child's result always sits right after its call and before the message (a provider rejects a tool_use answered after a later user message). The results and the refs' settlements ride the entry's own run-start commit:

• a child that finished (or is lost) gets its real (or typed framework_subagent_lost) result, ahead of the message;
• resume({ mode: 'with_message' }) / retry({ message }) over a child still owed is refused with AgentAlreadyRunningError (state_already_running, a 409 on the DO) and writes nothing: the run still awaits the child, so send the message again once it finished;
• a new turn (execute() on the same session, the DO's /start) over a parent whose run ended (completed, failed, interrupted) is never refused busy: no run awaits the child any more. A child still owed is stopped: best-effort transport.interrupt(), its ref is marked failed, and its call keeps the not_executed (unpaired_on_continuation) result the continuation heal gives every unfinished call. The child's remaining work ends; if you need its output, finish the child (or resume the parent) before starting a new turn;
• a child whose status cannot be read inside its deadline refuses any entry carrying a message with RemoteChildStatusUnavailableError (framework_subagent_status_unavailable, retryable), and nothing is written. The child may have finished, so the entry neither orphans it nor settles it lost. Send the message again later: once the producer answers, the entry proceeds; past the child's deadline it is settled lost. The refusal carries that deadline as retryAt (epoch ms).

The refusal window is deliberate. The deadline, startedAt + watch budget + 60 s, is the longest the parent's own dispatch could have watched the child (each attempt is bounded by timeoutMs, and the dispatch retries, with backoff waits between attempts), so a child is never declared lost, interrupted and its result discarded while it may still legitimately run. The cost is that, while the producer's status read keeps failing, an entry carrying a message is refused for up to that long: with the defaults (streamRetries: 3) and timeoutMs: 5 min, about 21.5 minutes on runtime-js and the CF DO, and about 38 minutes on CF Workflows (6 engine attempts of 5 min and their status reads, plus 310 s of engine waits). The window scales with the remote tool's own settings: to shorten it, lower that tool's timeoutMs, or on runtime-js / the DO its streamRetries / streamRetryBaseMs (createRemoteSubAgentTool(..., { timeoutMs, streamRetries })).

What a chat user sees. Over HTTP the refusal is the typed body { error, code: 'framework_subagent_status_unavailable' } with status 503 (core httpStatusForHelixError: retryable ⇔ 503), on the DO's /start and on a handleChatStream send alike. A stock useChat puts it in chat.error (its transport throws on a non-2xx answer), and parseHelixChatError(chat.error) reads the code and retryAt, so an app can show "waiting for the remote agent (until 14:32)" instead of a generic failure. Re-sending the message is safe: nothing was committed.

An entry with no new message (resume({ mode: 'continue' }), a retry() that re-appends nothing) is never refused: nothing needs protecting, so the run re-attaches to a child still owed (or unreadable) and waits for its real result before its next model step (on the Cloudflare DO, reconcile after the resume).

Cloudflare Workflows never reaches this rule (RM-69): a CF Workflows step commits ONCE, after its last result, so no committed remote call is ever left without its result while its child runs. A stop in the middle of a step commits the step with its answered rows and pairs the rest run_stopped in that commit; a step suspension commits its rows so far, after the step's remote calls ran (RM-92); every dispatch failure is recorded; and a parent whose Workflows instance dies mid-step loses the step as a whole (the next entry starts from the previous commit with a fresh model step). That entry, once its run-start commit won, interrupts the orphaned remote child (best-effort) and settles its live ref failed (framework_subagent_lost, cause parent_step_lost) with no row, since the model never saw the call; an interrupt that was not confirmed leaves the ref's owed stop in place (D3, core lostStepRemoteRefs), and later entries retry it with backoff (CP-203, CP-217: "Interrupt Propagation" below). A remote child registered but never started (its registration step failed) is the only unanswered remote call a CF Workflows commit can hold, and the continuation heal's not_executed result is then the truth. So there is no entry settle, re-attach or awaiting_remote_child refusal on CF Workflows.

A remote child's result recorded during a run settles a live remote ref in that result's own commit on every runtime (core recordSubSessionResult, used by Temporal and by a CF Workflows resumed run), and a CF Workflows step's returned rows settle theirs in the step's one commit.

A step lost with its run (D3), on runtime-js, the CF Durable Object and CF Workflows. A remote child's ref is registered, and the child started, BEFORE its step commits. A run that ends before that commit (it crashed, a newer entry superseded it, or a stop refused the step's commit) leaves a live ref whose call is in no committed row: no run awaits the child, and the trailing-call settle above never reads it, since the model never saw the call. Every entry handles these after its run-start commit won: a new turn (whatever the prior status, a crashed run's active included), resume(), retry() and a resumed loop. It stops each such child, best-effort, and settles each ref failed (framework_subagent_lost, cause parent_step_lost) in one rows-free append commit of its own run; a stop that was not confirmed stays owed on the ref (stopOwed), and later entries retry it with backoff (CP-203, CP-217: "Interrupt Propagation" below). Which refs count is a pure function of the committed log at that moment (core lostStepRemoteRefs): a ref whose call is in an assistant row is never swept, and companion (mode: 'persistent') refs are out of scope (a truncated companion call is orphaned by reconcile, as before).

• The ref is the parent's record, not the child's. A lost-step ref reads failed / parent_step_lost even when the remote child's own session finished completed: the parent never received that result and never will, so the entry does not read the child's status first. The child's own session still reports its real status (transport.getStatus). getSubSessionRefs shows the lost-step ref; the companion tools (companion__listChildren / companion__getChildStatus) list companion refs only, so they never show it.
• Latency. Stopping a lost child is on the entry's path. The stops run concurrently, each bounded by the executor's remoteChildStopTimeoutMs (default 10 s, core REMOTE_CHILD_STOP_TIMEOUT_MS; CF Workflows uses the constant), so an entry with lost children waits at most about one bound however many there are, and only when a lost child's producer does not answer. An entry with no live remote ref adds one ref read.

Temporal and DBOS do not run this sweep: they heal a still-running remote child's trailing call not_executed at the next entry and do not settle its ref (FU-TEMPORAL-REMOTE-HEAL-SETTLE), and DBOS writes a remote child's terminal ref before the workflow appends its result (FU-DBOS-REMOTE-SETTLE-ATOMIC). A child of a step lost with its run is not settled there either (FU-D3-TEMPORAL-DBOS-LOST-STEP).

Error Types

Three error types support remote agent failure handling across all runtimes:

• StreamDropError — Thrown by both transports' stream() when the /sse body ends with no terminal frame (wire v2), and by the Temporal, DBOS and Cloudflare runtimes as their platform-retry signal. Contains remoteSessionId and lastSequence for retry coordination: re-attach from lastSequence, never re-send /start. The JS runtime catches it inside its stream retry loop and does not surface it.
• RemoteAgentFailedError — Thrown across all runtimes when the remote agent has definitively failed (e.g., getStatus reports a failed state, or a non-recoverable fail frame). Contains remoteSessionId, remoteError and, when the remote sent one, its errorDetail (optional third argument).
• RemoteEntryOutcomeUnknownError — Thrown by both transports when /start or /resume got no definite answer (Entries are sent once). Non-retryable.

All three support instanceof checks for reliable error discrimination in catch blocks.

Interrupt Propagation

A parent's stop of a remote child is at-least-once (CP-217): every stop a parent sends its remote child, from any site, goes through core stopRemoteChild. Each stop is bounded by REMOTE_CHILD_STOP_TIMEOUT_MS (10 s), so a hung child holds neither its siblings nor the stop's caller. A stop the child does not confirm is owed on the parent's ref and retried until the child confirms it or the retry window ends. A child is remote by its location: 'remote' (isRemoteSubSessionRef), so a child whose stream identity was never recorded is still stopped (CP-150).

The stop names the delegation. A stop is transport.interrupt(sessionId, { reason, target }) (or abort). The body on the wire is { sessionId, reason, dispatchId } or { sessionId, reason, parentToolCallId }, and target says what the parent is stopping:

• A remote or ephemeral child is named by its dispatch: { dispatchId }. The child compares it with its own stored ChildBinding (one child per dispatch, so the session IS the delegation). A match stops the session's current run, a CP-157 continuation included.
• A persistent companion is named by its consult: { parentToolCallId }. The child stops its current run only when that run records that consult (RunMetadata.parent). A run of another consult, a later one included, or a direct entry is never stopped.
• A mismatch writes nothing. The child answers 409 state_stop_target_mismatch, and the parent reads it as confirmed: nothing of that delegation runs there. So a delayed retry can never stop a newer delegation.

Where a parent stops a remote child, on runtime-js, the CF Durable Object and CF Workflows:

• the in-flight dispatch, when the parent's run is stopped: interrupt, or abort for an abort stop. The ref stays LIVE (running, with its lastSequence), so a resume re-attaches; the next entry's lost-step sweep settles it if the parent never resumes;
• the in-flight dispatch when the call fails for any other reason (CP-157 §5): the ref is written failed;
• the stop cascades (runtime-js stopChildren, the DO's settle cascade, CF Workflows stopChildrenOfRun);
• the entry's lost-step sweep (D3), and the interrupt of a child settled lost (CP-178) or orphaned (D4).

What confirms a stop. The child answered it, answered state_not_stoppable (it had already ended), state_stop_target_mismatch, or that its session does not exist (state_session_not_found, or the protocol's NOT_FOUND). Every transport throws the child's typed answer (core remoteProtocolError; the HTTP transport did not, before CP-217). Anything else leaves the stop owed: a failure, no answer within the bound, a retryable answer, a typed refusal, or no transport for the child.

The debt is SubSessionRef.stopOwed, written in the same ref write the site makes anyway: { kind, at, target, reason, owedByRunId, attempts?, nextAttemptAt?, abandonedAt?, refusalLogged? }.

• Independent of the ref's status. The status is what the parent knows of the CALL; the debt is what it owes the CHILD. Any status may carry a debt, and a debt write re-states the status it read, so it never ends a ref.
• The kind only goes up. An abort upgrades an owed interrupt, never the reverse.
• One retry. Core retryOwedRemoteStops sends the owed kind again, naming the debt's target, for every remote ref with a debt (live or settled). A confirmed retry clears the debt. An unconfirmed one backs it off: attempts + 1, next due baseMs × 2^attempts, capped at maxMs. A typed refusal is logged at error once (stop.child_stop_refused), then owed like the rest.
• Abandoned past the window. A debt owed longer than windowMs since at is abandoned: one warn (code: 'remote_stop_owed_abandoned'), abandonedAt stamped, the record kept. Check that producer by hand: its child may still be running.
• Serialised. Every debt write is a compare-and-set on the status AND the debt it read (UpdateSubSessionRef.expectStopOwed: the debt's kind, at, target and attempts; every store). Two passes over one debt may each send a stop (harmless), but never put back a debt the other cleared, nor erase or overwrite a debt that was upgraded to an abort or re-aimed at another delegation since they read it. A stored debt the schema rejects reads as none, for the read and the compare-and-set alike.

The schedule is core OwedStopSchedule { baseMs, maxMs, windowMs }, default DEFAULT_OWED_STOP_SCHEDULE: 30 s doubling to 1 h, given up after 24 h. It paces RETRIES only; the first stop is always sent at once. A fresh debt's first retry is baseMs later. Configure it per runtime: JSAgentExecutor's owedStopSchedule option, or DurableObjectAgentConfig.owedStopSchedule. CF Workflows uses the default until its pacer (CP-217 MR 2).

Who retries, without a new entry ("eventually", R4). The stop is durable at once. The child stops at its next step boundary once a retry reaches it. While the parent cannot reach the child, the child keeps working: child-side orphan detection is FU-REMOTE-CHILD-PARENT-LEASE.

• The CF Durable Object: its stop-cascade alarm. It fires at the earliest due debt on the DO's refs, and also retries a companion's debt through the companion's own stop route.
• runtime-js: an in-process timer per session that runs core owedRemoteStopsSource. It is unref'd, so it never keeps a process alive, and it is re-armed after each cascade and each entry. Without a live process a debt waits for the session's next entry, which retries every due debt. A host that paces it itself passes onOwedStopsChanged (the DO passes its alarm's reschedule). The runtime-js scheduler hook (executor.wake) replaces only the trigger, not the work.
• CF Workflows: every entry and stop cascade retries the debts on SETTLED refs; a live owed ref is retried by a later cascade. Its own pacer, a durable step.sleep instance per parent run, is CP-217 MR 2.
• Temporal and DBOS: they send one stop and owe nothing yet (FU-TEMPORAL-REMOTE-CHILD-STOP-OWED, FU-DBOS-REMOTE-CHILD-STOP-OWED).

Reading the parent's ref after a stop. A running ref after a stop does not mean the stop failed. Its stopOwed says whether the stop reached the child: absent once the child confirmed it; present while it is owed. On runtime-js and the CF DO a resume or retry re-attaches to the child and records its REAL outcome (an interrupted child's result says it was interrupted), and a new turn settles it first. A lost-step settle (D3) keeps the debt on the settled ref.

Upgrade notes: docs/upgrade-guides/remote-stop-owed.md.

Production Considerations

State Storage

Use RedisStateStore and RedisStreamManager for production. In-memory stores lose state on restart:

import { RedisStateStore, RedisStreamManager } from '@helix-agents/store-redis';
import { JSAgentExecutor } from '@helix-agents/runtime-js';

const stateStore = new RedisStateStore({ url: process.env.REDIS_URL });
const streamManager = new RedisStreamManager({ url: process.env.REDIS_URL });
const executor = new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter());

const agentServer = new AgentServer({
  agents: { researcher: ResearcherAgent },
  stateStore,
  streamManager,
  executor,
});

Authentication

Protect your agent server with authentication headers:

// Static API key
const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: { Authorization: 'Bearer sk-xxx' },
});

// Dynamic JWT tokens
const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await fetchJWT()}`,
  }),
});

On the server side, add authentication middleware before the agent endpoints:

app.use('/agents', authMiddleware);
app.use('/agents', createExpressAdapter(createHttpAdapter(agentServer)));

Timeouts

Set appropriate timeoutMs on each remote sub-agent tool:

createRemoteSubAgentTool('researcher', {
  // ...
  timeoutMs: 120_000, // 2 minutes for complex research
});

createRemoteSubAgentTool('summarizer', {
  // ...
  timeoutMs: 30_000, // 30 seconds for summarization
});

Local vs Remote Sub-Agents

Feature	Local (createSubAgentTool)	Remote (createRemoteSubAgentTool)
Runs in	Same process	Separate HTTP service
Communication	Direct function calls	HTTP + SSE
State store	Shared instance	Independent instance
Hooks	beforeSubAgent/afterSubAgent fire with agentConfig	beforeSubAgent/afterSubAgent fire with agentConfig: undefined
Latency	Minimal	Network overhead
Scaling	Same process	Independent scaling
Runtime	Must match parent	Can differ from parent
Setup	Just the agent definition	Agent server + transport

Patterns

Microservice Specialization

Run different agents with different models or configurations:

// Server A: Runs expensive reasoning agents
const serverA = new AgentServer({
  agents: { analyzer: AnalyzerAgent }, // Uses Claude Opus
  // ...
});

// Server B: Runs fast utility agents
const serverB = new AgentServer({
  agents: { formatter: FormatterAgent }, // Uses GPT-4o-mini
  // ...
});

// Orchestrator delegates to both
const transportA = new HttpRemoteAgentTransport({ url: 'http://server-a:4000' });
const transportB = new HttpRemoteAgentTransport({ url: 'http://server-b:4001' });

Cross-Runtime Orchestration

Use Temporal for durable orchestration while specialist agents run on a lightweight Express service:

// Temporal worker runs the orchestrator with crash recovery
// Express service runs specialist agents (researcher, summarizer)
// HttpRemoteAgentTransport bridges the two

See the Remote Agents (Temporal) example for a full working implementation.

Multi-Tenant Agent Hosting

Host multiple tenants on a single agent server with dynamic headers:

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await getTenantToken()}`,
    'X-Tenant-Id': getCurrentTenantId(),
  }),
});

Limitations

• No agentConfig in hooks — beforeSubAgent and afterSubAgent hooks fire for remote sub-agents, but payload.agentConfig is undefined (use payload.call.agentType instead)
• No shared state — Remote agents have completely independent state (same as local sub-agents)
• Network latency — HTTP overhead compared to in-process local sub-agents
• Remote agent must have outputSchema — Required for structured tool results
• Client-executed and approval-gated tools inside a remote child park the parent only on runtime-js, the CF DO and CF Workflows, and only when the root can answer the call (the producer lists it in pendingCalls and the transport implements submitToolResult; the cross-service guide). Elsewhere they hit the fail-fast guard (RemoteSubAgentClientToolUnsupportedError) or CP-45's typed drop. A client-executed tool anywhere in the remote tree must be implemented by the ROOT UI (stock onToolCall).
• No persistent / companion remote children — Remote sub-agents are ephemeral only. RemoteSubAgentConfig has no mode, and remote SubSessionRefs are ephemeral by construction; extending the companion protocol across services is future work
• Remote children start unlinked traces — No trace context crosses the wire in v1. Thread your own ids through RemoteStartRequest.metadata (forwarded to the child session) and pick them up in producer-side hooks
• Executor is pluggable — AgentServer accepts any AgentExecutor implementation (JSAgentExecutor is the standard choice)
• Interrupt/abort are single-instance — AgentServer tracks execution handles in memory. Interrupt and abort only work on the server instance that started the execution. After a restart, in-flight handles are lost (sessions can still be resumed via POST /resume since state is persisted). For cross-instance lifecycle control, use a durable runtime (Temporal or Cloudflare) behind the agent server

Next Steps

• Cross-Service Remote Agents — State/usage across the boundary, Cloudflare DO producers, service bindings, the reconstruction contract
• Sub-Agents Guide — Local sub-agent orchestration
• API Reference — Full AgentServer and transport API
• Remote Agents Example — Working Temporal + HTTP example
• Runtimes — How different runtimes handle sub-agents
• Streaming — Handle agent stream events