Remote Agents

Remote agents let a parent agent delegate tasks to agents running on a separate HTTP service. This enables microservice architectures, cross-runtime orchestration, and security boundaries between agents.

When to Use Remote Agents

Use remote agents when you need:

• Different services — Specialist agents deployed as independent services
• Different runtimes — A Temporal orchestrator delegating to agents on an Express server
• Security boundaries — Isolate agents that handle sensitive data or credentials
• Independent scaling — Scale specialist agents separately from the orchestrator
• Different models — Run each agent with the most appropriate LLM

For agents that run in the same process, use local sub-agents instead.

Architecture Overview

sequenceDiagram
    participant Parent as Parent Agent
    participant Transport as HttpRemoteAgentTransport
    participant Server as AgentServer (Express)
    participant Child as Child Agent

    Parent->>Transport: Tool call (subagent__researcher)
    Transport->>Server: POST /start
    Server->>Child: Execute agent
    Server-->>Transport: { sessionId, streamId }
    Transport->>Server: GET /sse?sessionId=xxx
    loop Streaming
        Child-->>Server: Stream chunks
        Server-->>Transport: SSE events
        Transport-->>Parent: Proxy chunks to parent stream
    end
    Server-->>Transport: SSE end event
    Transport-->>Parent: Emit subagent_end + tool_end on parent stream
    Transport-->>Parent: Tool result (agent output)

The runtime emits a tool_end chunk on the parent's stream after the remote sub-agent completes (immediately following subagent_end). This is critical for AI SDK consumers — without tool_end, the parent's subagent__<name> UI tool part stays stuck in state: 'input-available'. See Sub-agent chunk ordering for the exact event sequence and rationale.

Setting Up the Server

The @helix-agents/agent-server package provides AgentServer — an HTTP server for hosting agents remotely.

Install

npm install @helix-agents/agent-server express

Configure AgentServer

import { AgentServer, createHttpAdapter, createExpressAdapter } from '@helix-agents/agent-server';
import { JSAgentExecutor } from '@helix-agents/runtime-js';
import { VercelAIAdapter } from '@helix-agents/llm-vercel';
import { InMemoryStateStore, InMemoryStreamManager } from '@helix-agents/store-memory';
import { ResearcherAgent } from './agents/researcher.js';
import { SummarizerAgent } from './agents/summarizer.js';

const stateStore = new InMemoryStateStore();
const streamManager = new InMemoryStreamManager();
const executor = new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter());

const agentServer = new AgentServer({
  // Registry of agents this server can run
  agents: {
    researcher: ResearcherAgent,
    summarizer: SummarizerAgent,
  },
  stateStore,
  streamManager,
  executor,
});

Workspace Wiring

If any of the registered agents declare workspaces, the underlying executor MUST be configured with matching workspaceProviders. The AgentServer validates this at construction:

import { CloudflareFileStoreWorkspaceProvider } from '@helix-agents/runtime-cloudflare';
import { InMemoryWorkspaceProvider } from '@helix-agents/workspace-memory';

const executor = new JSAgentExecutor(stateStore, streamManager, llm, {
  workspaceProviders: new Map([
    ['in-memory', new InMemoryWorkspaceProvider()],
    [
      'cloudflare-filestore',
      new CloudflareFileStoreWorkspaceProvider({
        /* ... */
      }),
    ],
  ]),
});

const agentServer = new AgentServer({
  agents: {
    researcher: ResearcherAgent /* declares a workspace with kind 'cloudflare-filestore' */,
  },
  stateStore,
  streamManager,
  executor,
  // Strict-mode validation: declare which provider kinds are wired.
  // Mismatches between agent.workspace and this list throw at construction.
  workspaceProviderKinds: ['in-memory', 'cloudflare-filestore'],
});

Two validation modes:

• Strict (workspaceProviderKinds set): construction throws an AgentServerError with code 'WORKSPACE_WIRING' if any registered agent's declared workspace provider kinds aren't covered. The error names the agent and the missing kind(s).
• Soft (workspaceProviderKinds unset): construction succeeds, but a logger.warn fires when at least one registered agent declares a workspace. The runtime still catches missing providers at LLM tool-call time via WorkspaceFailedError, but you'll see them mid-execution rather than at startup.

Prefer strict mode for production deployments where the wiring is known statically; soft mode is fine for testing / dynamic agent registration.

Wire Up Express

import express from 'express';

const app = express();
app.use(express.json());

// Health check
app.get('/health', (_req, res) => {
  res.json({ status: 'ok', agents: ['researcher', 'summarizer'] });
});

// Mount the agent endpoints at root
app.use('/', createExpressAdapter(createHttpAdapter(agentServer)));

app.listen(4000, () => {
  console.log('Remote Agent Service listening on http://localhost:4000');
});

This exposes the executor routes a RemoteAgentTransport speaks — /start, /resume, /sse, /status, /snapshot, /usage, /interrupt, /abort — plus /submit-tool-result and /workspace. See the HTTP Protocol Reference below for the full list.

TIP

For non-Express frameworks, use createHttpAdapter(agentServer) directly — it returns a generic AgentHttpHandler function you can adapt to any HTTP framework. See the API reference for details.

Creating Remote Sub-Agent Tools

On the client side, create tools that delegate to the remote server.

Configure the Transport

import { HttpRemoteAgentTransport } from '@helix-agents/core';

const transport = new HttpRemoteAgentTransport({
  url: 'http://localhost:4000',
  // Optional: static or dynamic headers
  headers: { Authorization: 'Bearer my-api-key' },
  // Optional: retry config (defaults: 3 retries, 1s base delay)
  maxRetries: 3,
  retryBaseDelayMs: 1000,
});

Create Remote Sub-Agent Tools

import { defineAgent, createRemoteSubAgentTool } from '@helix-agents/core';
import { z } from 'zod';

const researcherTool = createRemoteSubAgentTool('researcher', {
  description: 'Delegate research to a remote specialist agent',
  inputSchema: z.object({
    query: z.string().describe('The research query'),
  }),
  outputSchema: ResearcherOutputSchema,
  transport,
  remoteAgentType: 'researcher', // Must match key in server's agents registry
  timeoutMs: 120_000, // 2 minute timeout
  streamRetries: 3, // Retry on stream drop (default: 3)
  streamRetryBaseMs: 100, // Base delay between retries (default: 100ms)
  maxFoldStateBytes: 262_144, // Cap on the child state folded into the parent (default)
  resumeDeadProducer: false, // Opt-in blind resume of a dead Node producer (default: off)
});

const summarizerTool = createRemoteSubAgentTool('summarizer', {
  description: 'Delegate summarization to a remote specialist agent',
  inputSchema: z.object({
    text: z.string().describe('The text to summarize'),
  }),
  outputSchema: SummarizerOutputSchema,
  transport,
  remoteAgentType: 'summarizer',
  timeoutMs: 60_000,
});

Two optional fields govern what comes back from a completed child:

• maxFoldStateBytes (default 262_144) — the maximum serialized size of the child's terminal customState folded into the parent (surfaced to hooks as AfterSubAgentPayload.childCustomState). Over the cap, the state fold is skipped with a warning; the output and usage folds are unaffected. The cap exists because the fold rides through Temporal activity results and DBOS step results, which have their own payload limits.
• resumeDeadProducer (default false) — opt in to a single blind transport.resume() when the producer is observed dead on both sides of a zero-progress stream attempt (status: 'running' with isExecuting: false, no chunks received), after which the dispatch retries to attach. Off by default because a blind resume is wrong for multi-replica producers; without it, the dispatch fails fast with the producer-dead failure reason instead of burning the retry budget.

Output re-validation

The child's end.output is re-validated by default against the tool's outputSchema (a safeParse on the consumer side). A child whose output doesn't match the schema surfaces a structured tool error to the parent LLM instead of passing an off-schema object through. There is no signature change and no flag to set — if you rely on a remote child returning something looser than its declared outputSchema, audit that schema before upgrading.

Use in a Parent Agent

const OrchestratorAgent = defineAgent({
  name: 'orchestrator',
  description: 'Orchestrates research via remote specialist agents',
  outputSchema: OrchestratorOutputSchema,
  tools: [researcherTool, summarizerTool],
  systemPrompt: `You are a research orchestrator.
1. Use the researcher to gather information
2. Use the summarizer to distill findings
3. Call __finish__ with your final output`,
  llmConfig: { model: openai('gpt-4o-mini') },
  maxSteps: 10,
});

The parent LLM sees subagent__researcher and subagent__summarizer as tools — it doesn't know they're remote.

HTTP Protocol Reference

createHttpAdapter mounts two layers of routes — the executor routes (Layer A, what a RemoteAgentTransport speaks) plus the v7 chat handler routes (Layer B) layered on top:

Layer A — Executor routes

Method	Path	Description
POST	/start	Start a new agent execution
POST	/resume	Resume an interrupted agent
GET	/sse	SSE stream of agent events
GET	/status	Get execution status
GET	/snapshot	Checkpoint-pinned snapshot of the session's customState
GET	/usage	Recursive usage rollup for the session
POST	/interrupt	Durable interrupt (returns 202; observed at next checkpoint)
POST	/abort	Hard stop
POST	/submit-tool-result	Submit a client-tool result or approval response
GET	/workspace	Operator-facing workspace introspection

Layer B — Chat handler routes (v7, wired via chatHandler config)

Method	Path	Description
POST	/chat	Start or continue a chat turn (always streams)
GET	/chat/{id}/stream	Reattach to an in-progress stream after a refresh
POST	/chat/{id}/submit-tool-result	Submit a client-tool result or approval-response
POST	/chat/{id}/interrupt	Durable interrupt for the chat session
POST	/chat/{id}/abort	Abort the current run

The chatHandler config option plumbs handleChatStream through AgentServer({ chatHandler }). Layer B replaces the INTERRUPT_NOT_LOCAL 503 (now removed): interrupts are durable via the state-store flag and return 202.

POST /start

Start a new agent execution.

Request:

{
  "sessionId": "session-abc-123",
  "agentType": "researcher",
  "message": "{\"query\":\"Research TypeScript benefits\"}",
  "state": { "query": "Research TypeScript benefits" },
  "metadata": {}
}

The message field contains the JSON-serialized tool input from the parent agent's LLM. The remote agent server is responsible for parsing this JSON and constructing the appropriate user message for its agent. The state field carries the same input as a parsed object for initializing the agent's custom state.

Identity forwarding. Three optional request fields carry the parent's identity across the boundary:

Field	Type	Purpose
userId	string	User the child session is attributed to
tags	string[]	Parent tags, copied onto the child session
metadata	Record<string, string>	Free-form key/value metadata

Consumer executors forward the parent's values automatically; the producer seeds the child session with them — parity with local spawn (usage attribution, memory entity-scoping, and metadata queries match local children). metadata is also the escape hatch for threading trace ids across the boundary.

Response (200):

{
  "sessionId": "session-abc-123",
  "streamId": "stream-xyz",
  "runId": "run-456"
}

Starting a session that is already live returns HTTP 409 with { code: 'ALREADY_RUNNING', streamId }. This is a typed, expected response: transports surface it as a typed error (RemoteAgentAlreadyRunningError) and every consumer treats it as attach (proceed to /sse with the returned streamId), not failure. It is necessary because POSTs are at-least-once — a retried /start whose first response was lost 409s against its own first attempt.

POST /resume

Resume an interrupted or paused agent, optionally with a new message.

Request:

{
  "sessionId": "session-abc-123",
  "message": "Continue with more detail"
}

GET /sse

Subscribe to agent events via Server-Sent Events.

Query params:

• sessionId (required) — Session to stream
• fromSequence (optional) — Resume from this sequence number (skips already-seen events)

Event format:

id: 42
event: chunk
data: {"chunk":{...},"sequence":42}

event: end
data: {"status":"completed","output":{...},"state":{...},"usage":{...}}

event: error
data: {"error":"something failed","recoverable":false}

:heartbeat

The server sends a :heartbeat comment every 15 seconds to keep the connection alive.

The end frame

Both producer servers (AgentServer and the Cloudflare DO) emit the same end frame: { status: 'completed' | 'paused' | 'interrupted', output?, state?, usage? }. The DO's legacy finalOutput field is gone from this frame — see the repo-root behavior-changes.md for the migration.

• Only status: 'completed' may be folded as success. paused / interrupted frames must route through /status-based recovery — a paused child that folds as success reports output: null to the parent LLM as if it had finished.
• state and usage are emitted only after the producer's terminal persist committed (persist-before-emit), so either may be absent. Fall back to GET /snapshot / GET /usage when they are.
• status is required: a frame with no status is malformed — the transport emits a recoverable error event (routed through drop-recovery / getStatus), never a silent 'completed'.

fromSequence is exclusive

fromSequence is exclusive (sequence > fromSequence) — pass snapshot.streamSequence directly, with no +1 adjustment. (Contrast: the store-level getChunksFromStep is inclusive despite the similar name.)

If fromSequence is below the stream's retention floor, the server emits a structured non-recoverable error frame carrying code: 'TRUNCATED' and floor instead of silently serving the survivors. Refetch GET /snapshot and resume from its checkpoint-pinned streamSequence.

Malformed frames

Malformed SSE frames surface as errors. They are no longer defaulted to sequence: 0 — that default made a corrupt frame invisible, because the consumer's <= lastSequence dedup silently dropped it. The core transport now yields a recoverable error event for an unparseable frame, which routes the consumer into its normal drop-recovery path (reconnect from the last good sequence).

GET /status

Get the current execution status.

Response:

{
  "sessionId": "session-abc-123",
  "runId": "run-456",
  "status": "running",
  "stepCount": 3,
  "isExecuting": true,
  "streamId": "stream-xyz",
  "latestSequence": 42,
  "awaitingClientTool": false
}

Status values: running, completed, failed, interrupted, paused.

RemoteStatusResponse.state was removed

/status no longer carries a state field. Use GET /snapshot — it is the only route that returns a (state, sequence) pair written atomically, so it is the only sound baseline for state reconstruction. See the repo-root behavior-changes.md for the migration.

isExecuting is honest, not an echo. On the Node AgentServer it is derived from the server's live execution handles — this replica's truth — rather than a status === 'active' echo of persisted state. Per-replica caveat: a session executing on another replica of a multi-replica deployment also reports isExecuting: false here, so consumers must not escalate on a single false. status: 'running' with isExecuting: false across several polls means the producer died; see producer-liveness in the Cross-Service Remote Agents guide.

awaitingClientTool is a required boolean: true when the session is suspended awaiting a client-executed tool result. A parent dispatching this agent as a remote sub-agent uses it to fail fast — see the limitation below.

Limitation: client-executed and approval-gated tools inside remote sub-agents

A remote sub-agent cannot use client-executed tools (execute: 'client') or approval-gated tools (requireApproval: true). Both suspend through the same pendingClientToolCalls map and so surface identically as awaitingClientTool: true; routing a browser-submitted result (or approval) back across the HTTP boundary to resume the remote agent is not implemented. If a remote sub-agent suspends on either, the parent fails fast with RemoteSubAgentClientToolUnsupportedError (the remote-dispatch result carries failureReason: 'client-tool-unsupported') rather than hanging — the parent detects the condition via the awaitingClientTool flag above. Both tool kinds on the parent agent are fully supported; only their use inside a remote child is not. Tracked for future support in GitLab #107.

GET /snapshot

Fetch a checkpoint-pinned snapshot of the session's own customState. This is the baseline every state-reconstruction flow starts from — initial state is never on the wire.

Query params:

• sessionId (required)

Response — RemoteSnapshotResponse:

{
  sessionId: string;
  streamId: string;             // the sequence space this snapshot is aligned to
  state: Record<string, unknown>; // THIS session's own customState (possibly projected)
  streamSequence: number;       // checkpoint-pinned sequence; pass DIRECTLY as
                                // `fromSequence` (exclusive). -1 = no patch
                                // continuation — never compute a fromSequence from it
  oldestRetainedSequence: number; // retention floor; 0 = full log retained
  stepCount: number;
  checkpointId?: string;        // optional
}

A non-terminal session is served from the latest checkpoint row: state and streamSequence come out of the same atomically-written record, which is the only (state, sequence) pairing that is sound (a live two-read pairing either skips the in-flight step's patches forever or double-applies them). A terminal session is served the final state with streamSequence: -1 — the snapshot pins a moment, not a position, and there are no further patches to continue from. -1 is also returned when no checkpoint exists yet or when alignment is otherwise unavailable, so treat it uniformly as "snapshot-only": do not derive a fromSequence from it.

An unknown session returns 404 { code: 'NOT_FOUND' } — the response schema has no null branch.

See the Cross-Service Remote Agents guide for the full reconstruction contract (attach-at-S, patch filtering by agentId, re-snapshot triggers).

GET /usage

Fetch the session's usage rollup.

Query params:

• sessionId (required)

Response — RemoteUsageResponse:

{
  sessionId: string;
  rollup: UsageRollup;
}

The rollup is getRollup(sessionId, { includeSubAgents: true }) computed producer-side — the recursive fold, so the remote agent's own grandchildren are already included by construction. A running session returns the partial rollup as-of-now; once terminal it is at least as of the end event.

The route is served only when the producer has a usage store wired: on Node, set AgentServerConfig.usageStore; on the Cloudflare DO producer it is built in (the DO falls back to its own SQLite-backed DOUsageStore). Without one, /usage responds 404 { code: 'FEATURE_UNAVAILABLE' }. Consumers degrade on this: they skip the usage fold and log a warning rather than failing the tool call.

The store powers /usage and the consumer's getUsage() fallback fold — it does not put usage on the SSE end frame. Only the Cloudflare DO producer attaches end.usage today; the Node/JS, Temporal, DBOS, and Cloudflare Workflows producers never do, regardless of whether a usage store is wired.

POST /interrupt

Soft stop — the agent can be resumed later.

Request:

{
  "sessionId": "session-abc-123",
  "reason": "User requested pause"
}

POST /abort

Hard stop — the agent cannot be resumed.

Request:

{
  "sessionId": "session-abc-123",
  "reason": "Timeout exceeded"
}

Error Responses

All endpoints return errors in this format:

{
  "error": "Agent type not found: unknown-agent",
  "code": "NOT_FOUND"
}

Code	HTTP Status	Description
NOT_FOUND	404	Agent type or session not found
ALREADY_RUNNING	409	Session is already executing — treat as attach, not failure
ALREADY_COMPLETED	409	Session has already completed or failed
FEATURE_UNAVAILABLE	404	Route exists but the capability isn't wired (e.g. no usage store)
INVALID_REQUEST	400	Missing required fields
INTERNAL_ERROR	500	Server error

ALREADY_RUNNING responses additionally carry streamId so a consumer can attach to the live stream without an extra /status round-trip.

The transports map these onto typed errors so consumers can discriminate with instanceof instead of string matching:

Envelope code	Typed error
ALREADY_RUNNING	RemoteAgentAlreadyRunningError
NOT_FOUND	RemoteAgentNotFoundError
FEATURE_UNAVAILABLE	RemoteAgentFeatureUnavailableError

A response body that fails schema validation at the transport boundary throws RemoteProtocolValidationError — peer schema drift surfaces as a structured error rather than a downstream undefined field access.

TRUNCATED is not an HTTP error: it arrives as an SSE error frame on /sse (see above), because that is the only place a retention-floor violation can be detected.

How It Works

Step-by-step execution flow when a parent agent calls a remote sub-agent:

1. Parent LLM calls tool — e.g., subagent__researcher({ query: "TypeScript benefits" })
2. Runtime detects remote tool — The tool is marked with _isRemoteSubAgent: true
3. Transport calls POST /start — Sends the message, agent type, and the parent's identity (userId/tags/metadata) to the remote server. A 409 ALREADY_RUNNING means "already live" and is handled as an attach
4. Transport connects GET /sse — Subscribes to the event stream for the session, seeded with fromSequence when resuming
5. Remote agent executes — The AgentServer runs the agent using its configured executor (JSAgentExecutor is the standard choice)
6. Chunks stream back — SSE events flow through the transport back to the parent, stamped with remoteSource provenance
7. End frame arrives — On status: 'completed', the output is re-validated against the tool's outputSchema and returned as the tool result; the child's terminal customState and usage rollup fold into the parent (from end.state / end.usage, else GET /snapshot / GET /usage). paused / interrupted route through /status-based recovery instead
8. Parent continues — The parent LLM receives the result and proceeds

Cross-Runtime Behavior

Remote sub-agent tools are first-class constructs across all runtimes:

Runtime	Behavior
JS	Intercepts remote tool calls with enhanced handling: stream proxying to parent, SubSessionRef tracking, timeout management via AbortSignal, internal stream retry loop on drops, interrupt propagation, reconnection on resume
Temporal	Dedicated executeRemoteSubAgentCall activity with deterministic session IDs, crash recovery via transport.getStatus(), heartbeat-based reconnection, stream proxying, interrupt propagation, StreamDropError for activity retry
DBOS	Dedicated executeRemoteSubAgentCall step (Postgres-backed workflow replay) with deterministic session IDs, crash recovery via transport.getStatus() on step retry, stream proxying, interrupt propagation, StreamDropError for step retry, ref-cadence reconnection (seed from the ref at step entry; persist every N chunks)
Cloudflare	Dedicated executeRemoteSubAgentCall step with deterministic session IDs, crash recovery, stream proxying, interrupt propagation via abort-check interval, timeout enforcement, StreamDropError for step retry

TIP

All runtimes provide stream proxying, SubSessionRef tracking with remote metadata, and sub-agent lifecycle hooks (beforeSubAgent/afterSubAgent). Remote sub-agents are routed through a dedicated execution path separate from regular tool calls, with subagent_start/subagent_end stream events.

Cloudflare DO and Local Sub-Agents

In the Cloudflare DO runtime, createSubAgentTool() (local sub-agents) are transparently converted to remote sub-agent calls using DOStubTransport — a RemoteAgentTransport implementation that routes to sibling DO instances instead of an external HTTP service. This means HttpRemoteAgentTransport is only needed when crossing service boundaries; within a single DO namespace, subAgentNamespace handles everything. See Sub-Agents in the DO Runtime.

Streaming Integration

Remote agent events appear in the parent stream using the same subagent_start/subagent_end pattern as local sub-agents:

for await (const chunk of parentStream) {
  switch (chunk.type) {
    case 'subagent_start':
      console.log(`Remote agent started: ${chunk.subAgentType}`);
      break;

    case 'text_delta':
      // Could be from parent or remote agent
      console.log(`[${chunk.agentType}]`, chunk.delta);
      break;

    case 'tool_start':
      // Tools used by the remote agent
      console.log(`[${chunk.agentType}] Tool: ${chunk.toolName}`);
      break;

    case 'subagent_end':
      console.log(`Remote agent finished: ${chunk.subAgentType}`);
      break;
  }
}

This means frontends don't need to distinguish between local and remote sub-agents — the stream protocol is identical.

Transport Configuration

Static Headers

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: {
    Authorization: 'Bearer my-api-key',
    'X-Tenant-Id': 'tenant-123',
  },
});

Dynamic Headers

For tokens that need to be refreshed:

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await getAccessToken()}`,
  }),
});

Retry Policy

The transport retries failed requests with exponential backoff:

• 5xx errors — Retried with backoff
• 4xx errors — Not retried (client error)
• Network errors — Retried with backoff

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  maxRetries: 5, // Default: 3
  retryBaseDelayMs: 2000, // Default: 1000ms
});

Delay formula: baseDelay * 2^attempt (1s, 2s, 4s, ...).

Custom fetch

HttpTransportConfig.fetch?: typeof fetch replaces globalThis.fetch for every request the transport issues (the POSTs, the /status, /snapshot and /usage GETs, and the SSE stream). Use it for Cloudflare service bindings, Miniflare's dispatchFetch in tests, and Node-side proxies:

const transport = new HttpRemoteAgentTransport({
  url: 'https://producer', // hostname is arbitrary over a binding
  fetch: env.PRODUCER.fetch.bind(env.PRODUCER),
});

For Cloudflare consumers, serviceBindingTransport from @helix-agents/runtime-cloudflare wraps this pattern — see the Cross-Service Remote Agents guide.

Custom Transports

RemoteAgentTransport has eight required methods:

interface RemoteAgentTransport {
  start(request: RemoteStartRequest): Promise<RemoteStartResponse>;
  resume(request: RemoteResumeRequest): Promise<RemoteStartResponse>;
  stream(
    sessionId: string,
    options?: { fromSequence?: number; signal?: AbortSignal }
  ): AsyncIterable<TransportEvent>;
  getStatus(sessionId: string): Promise<RemoteStatusResponse>;
  getSnapshot(sessionId: string): Promise<RemoteSnapshotResponse>;
  getUsage(sessionId: string): Promise<RemoteUsageResponse>;
  interrupt(sessionId: string, reason?: string): Promise<void>;
  abort(sessionId: string, reason?: string): Promise<void>;
}

getSnapshot and getUsage are required on the interface — a custom transport written against the previous six-method interface will not typecheck until it implements them. HttpRemoteAgentTransport (packages/core/src/transport/http-transport.ts) is the reference implementation: a GET against /snapshot / /usage, with the response validated by RemoteSnapshotResponseSchema / RemoteUsageResponseSchema and non-2xx bodies mapped onto the typed errors above.

If your transport genuinely cannot serve one of them, throw RemoteAgentFeatureUnavailableError — consumers treat that (and RemoteAgentNotFoundError) as feature-unavailable and degrade gracefully: the corresponding state or usage fold is skipped with a warning rather than failing the tool call.

Stream Recovery

Remote sub-agent calls use SSE for streaming. Network interruptions can drop the SSE connection mid-execution. The framework provides defense-in-depth recovery that varies by runtime:

JS Runtime — Internal Retry Loop

The JS runtime retries stream drops internally without surfacing errors to the caller. When a stream drops (ends without an end event):

1. Checks transport.getStatus() to see if the remote agent completed, failed, or is still running
2. If still running, reconnects with fromSequence to resume from the last received chunk
3. Retries up to streamRetries times with exponential backoff (streamRetryBaseMs * 2^attempt)

Configure retry behavior per tool:

createRemoteSubAgentTool('researcher', {
  // ...
  streamRetries: 5, // Default: 3, max: 50 (0 disables retries)
  streamRetryBaseMs: 200, // Default: 100ms
});

Temporal Runtime — Activity Retry

The Temporal runtime throws StreamDropError on stream drops, which triggers Temporal's built-in activity retry. The heartbeat carries lastSequence so the retried activity can resume from the correct position.

Cloudflare Runtime — Step Retry

The Cloudflare runtime throws StreamDropError on stream drops, which triggers Cloudflare Workflows step retry. The lastSequence is persisted to the SubSessionRef for crash recovery.

Reconnect cadence (cross-runtime contract)

Retry only helps if the retried attempt knows where to resume from. Every consumer path therefore obeys the same two-part contract: seed fromSequence from the persisted SubSessionRef.remote.lastSequence at (re)entry, and persist progress back to that ref at a bounded cadence while streaming (every REMOTE_REF_PERSIST_CHUNK_INTERVAL = 25 proxied chunks). Without the seed, a retried attempt re-forwards the child's entire stream; without the bounded persist, the seed is stale by an unbounded amount.

Runtime	Cadence
JS / DO	Seed from the ref; persist between retry attempts and every N chunks
Temporal	Per-chunk activity heartbeat carrying lastSequence, under an explicit heartbeatTimeout (30s). Activity timeouts are derived from the tool's timeoutMs and stream-retry budget, so a healthy child that legitimately runs longer than the default activity timeout is not reported failed. The heartbeat also delivers cancellation and detects a hung SSE connection.
CFW	Per-N-chunk SubSessionRef update; the step retry re-seeds from it
DBOS	Seed from the ref at step entry; persist every N chunks and on every failure path

Every proxied child chunk carries remoteSource { childSessionId, childSequence, streamId } provenance. Specified consumers dedup on (childSessionId, childSequence), so any residual duplicate window left by the cadence is harmless — a re-forwarded chunk is recognized and dropped rather than double-applied.

Orphan reconciliation (JS and Cloudflare DO). A parent crash truncates history to the last checkpoint, and the re-rolled LLM mints new tool-call ids — so the child session id ({parent}-remote-{toolCallId}) it would compute no longer matches the child that is still running producer-side. On resume, the JS executor (which the DO runtime embeds) sweeps SubSessionRefs that are still running with remote metadata:

• Terminal producer-side — the result is folded in as a tool-result message and the ref is marked completed / failed, so the LLM doesn't waste a turn re-calling.
• Still running, tool call survived truncation — the ref is left running. The LLM re-calls the tool with the same tool-call id, and transport.start() reattaches to the existing child session.
• Still running, tool call truncated away — a genuine orphan: best-effort transport.interrupt() to bound producer-side spend, then the ref is marked failed.

Error Types

Two error types support remote agent failure handling across all runtimes:

• StreamDropError — Thrown by Temporal and Cloudflare runtimes when the SSE stream drops without an end event. Contains remoteSessionId and lastSequence for retry coordination. The JS runtime handles stream drops internally and does not throw this error.
• RemoteAgentFailedError — Thrown across all runtimes when the remote agent has definitively failed (e.g., getStatus reports a failed state). Contains remoteSessionId and remoteError.

Both errors support instanceof checks for reliable error discrimination in catch blocks.

Interrupt Propagation

When a parent agent is interrupted or aborted, all runtimes propagate the interrupt to running remote agents via transport.interrupt(). This is best-effort — if the interrupt call fails (e.g., network error), the failure is logged but does not prevent the parent from completing its interrupt flow.

Production Considerations

State Storage

Use RedisStateStore and RedisStreamManager for production. In-memory stores lose state on restart:

import { RedisStateStore, RedisStreamManager } from '@helix-agents/store-redis';
import { JSAgentExecutor } from '@helix-agents/runtime-js';

const stateStore = new RedisStateStore({ url: process.env.REDIS_URL });
const streamManager = new RedisStreamManager({ url: process.env.REDIS_URL });
const executor = new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter());

const agentServer = new AgentServer({
  agents: { researcher: ResearcherAgent },
  stateStore,
  streamManager,
  executor,
});

Authentication

Protect your agent server with authentication headers:

// Static API key
const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: { Authorization: 'Bearer sk-xxx' },
});

// Dynamic JWT tokens
const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await fetchJWT()}`,
  }),
});

On the server side, add authentication middleware before the agent endpoints:

app.use('/agents', authMiddleware);
app.use('/agents', createExpressAdapter(createHttpAdapter(agentServer)));

Timeouts

Set appropriate timeoutMs on each remote sub-agent tool:

createRemoteSubAgentTool('researcher', {
  // ...
  timeoutMs: 120_000, // 2 minutes for complex research
});

createRemoteSubAgentTool('summarizer', {
  // ...
  timeoutMs: 30_000, // 30 seconds for summarization
});

Local vs Remote Sub-Agents

Feature	Local (createSubAgentTool)	Remote (createRemoteSubAgentTool)
Runs in	Same process	Separate HTTP service
Communication	Direct function calls	HTTP + SSE
State store	Shared instance	Independent instance
Hooks	beforeSubAgent/afterSubAgent fire with agentConfig	beforeSubAgent/afterSubAgent fire with agentConfig: undefined
Latency	Minimal	Network overhead
Scaling	Same process	Independent scaling
Runtime	Must match parent	Can differ from parent
Setup	Just the agent definition	Agent server + transport

Patterns

Microservice Specialization

Run different agents with different models or configurations:

// Server A: Runs expensive reasoning agents
const serverA = new AgentServer({
  agents: { analyzer: AnalyzerAgent }, // Uses Claude Opus
  // ...
});

// Server B: Runs fast utility agents
const serverB = new AgentServer({
  agents: { formatter: FormatterAgent }, // Uses GPT-4o-mini
  // ...
});

// Orchestrator delegates to both
const transportA = new HttpRemoteAgentTransport({ url: 'http://server-a:4000' });
const transportB = new HttpRemoteAgentTransport({ url: 'http://server-b:4001' });

Cross-Runtime Orchestration

Use Temporal for durable orchestration while specialist agents run on a lightweight Express service:

// Temporal worker runs the orchestrator with crash recovery
// Express service runs specialist agents (researcher, summarizer)
// HttpRemoteAgentTransport bridges the two

See the Remote Agents (Temporal) example for a full working implementation.

Multi-Tenant Agent Hosting

Host multiple tenants on a single agent server with dynamic headers:

const transport = new HttpRemoteAgentTransport({
  url: 'https://agents.example.com',
  headers: async () => ({
    Authorization: `Bearer ${await getTenantToken()}`,
    'X-Tenant-Id': getCurrentTenantId(),
  }),
});

Limitations

• No agentConfig in hooks — beforeSubAgent and afterSubAgent hooks fire for remote sub-agents, but payload.agentConfig is undefined (use payload.call.agentType instead)
• No shared state — Remote agents have completely independent state (same as local sub-agents)
• Network latency — HTTP overhead compared to in-process local sub-agents
• Remote agent must have outputSchema — Required for structured tool results
• No client-executed or approval-gated tools inside a remote child — Both suspend through the same pendingClientToolCalls map and surface identically as awaitingClientTool: true, so both hit the same fail-fast guard (RemoteSubAgentClientToolUnsupportedError, failureReason: 'client-tool-unsupported'). Routing a browser-submitted result or approval back across the HTTP boundary is not implemented. Client-executed and approval-gated tools on the parent agent are fully supported (GitLab #107)
• No persistent / companion remote children — Remote sub-agents are ephemeral only. RemoteSubAgentConfig has no mode, and remote SubSessionRefs are ephemeral by construction; extending the companion protocol across services is future work
• Remote children start unlinked traces — No trace context crosses the wire in v1. Thread your own ids through RemoteStartRequest.metadata (forwarded to the child session) and pick them up in producer-side hooks
• Executor is pluggable — AgentServer accepts any AgentExecutor implementation (JSAgentExecutor is the standard choice)
• Interrupt/abort are single-instance — AgentServer tracks execution handles in memory. Interrupt and abort only work on the server instance that started the execution. After a restart, in-flight handles are lost (sessions can still be resumed via POST /resume since state is persisted). For cross-instance lifecycle control, use a durable runtime (Temporal or Cloudflare) behind the agent server

Next Steps

• Cross-Service Remote Agents — State/usage across the boundary, Cloudflare DO producers, service bindings, the reconstruction contract
• Sub-Agents Guide — Local sub-agent orchestration
• API Reference — Full AgentServer and transport API
• Remote Agents Example — Working Temporal + HTTP example
• Runtimes — How different runtimes handle sub-agents
• Streaming — Handle agent stream events