Client-Executed Tools

Guide for declaring tools that execute in the browser (or any non-runtime party), and receiving their results back into the agent loop.

v7 model: stateless suspension

Major change in v7

The client-tool model was redesigned in v7. The legacy in-memory wait (promise + setTimeout) is gone; the runtime now suspends durably and resumes only when an explicit executor.resume() reads SessionState.suspensionContext. If you are upgrading from v6, read the v6 to v7 migration guide end-to-end before deploying.

When the LLM emits a client-tool call, the runtime:

1. Validates the input against inputSchema and emits a tool_start stream chunk.
2. Persists a pending entry in SessionState.pendingClientToolCalls (keyed by toolCallId) and writes the suspension context to the state store atomically with the step's other state changes (saveStateAndPromoteStaging). The entry records the run that suspended on it (runId), which fences later submits (see the HTTP contract).
3. Records the run's suspended_client_tool status and then pauses the stream. The status is always written before the stream is paused, so a stream that reads paused means the suspension is committed. (DBOS and Cloudflare Workflows write the run status before the suspension commit; JS and Temporal after it.)
4. Returns RunOutcome.suspended_client_tool from the run loop. The in-flight JavaScript loop dies; nothing waits in memory.
5. handle.result() resolves with status: 'suspended_client_tool'. The chat handler closes the SSE stream cleanly.

Resumption is explicit:

1. The client submits via submitToolResult({ kind: 'client-tool-result', ... }). The submission writes the result into pendingClientToolCalls[toolCallId].response.
2. The chat handler (or your code) calls executor.resume({ sessionId }) once every call of the step's batch has its answer (see Submit every call of a batch before resuming).
3. The new run reads the suspension context, processes the result as a tool-result message, and continues the loop.

Per design §10.7, this is the same machinery used by approval gates — both primitives share pendingClientToolCalls + suspensionContext, discriminated by the submission's kind field.

Security

The POST /submit-tool-result endpoint accepts a sessionId + toolCallId and submits a tool result into the LLM's context. It MUST be gated by authentication — anyone who can reach the endpoint can inject arbitrary data into an agent's reasoning.

This also applies to the other endpoints exposed by AgentServer (/start, /resume, /sse, /status, /interrupt, /abort) — none of them are safe to expose publicly without auth.

Recommended patterns

• Session-bound tokens: issue a short-lived HMAC or JWT when starting a session. The token encodes the authorized sessionId; on each submit, reject any request whose body's sessionId doesn't match the token.
• Server-issued session IDs: don't let clients choose sessionId values. Generate them server-side, keyed to authenticated user identity.
• Reverse proxy: deploy behind a proxy (Cloudflare Access, Envoy, nginx + OAuth2 proxy) that enforces your auth before requests reach the agent server.

Using the authenticate hook

AgentServer accepts an authenticate hook. The hook is invoked before every endpoint handler runs. Return true to accept, false for a generic 401, or { error, status? } for a custom rejection (e.g., 403 for "authenticated but not authorized for this session").

import { AgentServer } from '@helix-agents/agent-server';

const server = new AgentServer({
  executor,
  agents: {
    /* ... */
  },
  stateStore,
  streamManager,
  authenticate: async (req, operation) => {
    const token = String(req.headers.authorization ?? '').replace(/^Bearer\s+/, '');
    if (!token) return { error: 'missing_auth', status: 401 };

    const session = await verifyToken(token); // your impl
    if (!session) return { error: 'invalid_token', status: 401 };

    // Session-bound check: the body's sessionId must match the token's.
    if (operation === 'submit-tool-result') {
      const body = req.body as { sessionId?: string };
      if (body?.sessionId !== session.sessionId) {
        return { error: 'session_mismatch', status: 403 };
      }
    }

    return true;
  },
});

When the hook is not configured, the server logs a WARN at startup to make misconfigurations visible. Do not rely on the warning alone — gate the endpoints explicitly.

Rate limiting

The framework does not ship rate limiting for /submit-tool-result. Even with authentication, a bad actor (or a buggy client) can flood the endpoint. Gate it upstream:

• Node / Express / Fastify: use express-rate-limit or fastify-rate-limit with a per-session key (derived from the authenticate hook or from the request's sessionId body field). Recommended baseline: 60 submits/min per session, matching ordinary client-tool pacing.
• Cloudflare Workers: Cloudflare Rules Rate Limiting at the Worker level, OR a sub-request to a dedicated rate-limit Durable Object keyed on sessionId. Under CF DO, the per-session fetch path naturally serializes (each sessionId → single DO instance → single alarm/request queue), which provides a coarse rate limit for free but doesn't bound total QPS across shared infrastructure.
• Body size: the framework's Content-Length gate (413 Payload Too Large) defends against full-body allocation DoS, but transport-level body limits (e.g. express.json({ limit: '2mb' })) still provide defense-in-depth and cheaper rejection before the framework runs.

Service identification in logs

If you run multiple AgentServer instances in a fleet (per-service, per- region, etc.), set AgentServerConfig.serviceName to disambiguate log messages. The unauthenticated-mode warning (and future server-level logs) include the service name so on-call can triage which deployment is misconfigured without grepping hostname-per-process.

When to use

Some tool calls can't or shouldn't run on the server:

• Browser-only capabilities — manipulating live editors, the DOM, indexedDB, wallet signing, etc.
• Client-side state — tools that need access to the user's local data.
• Human-in-the-loop (future) — approval dialogs, parameter editing, manual result entry. Today, client-executed tools provide the primitive that future HITL patterns will build on.

Declaring a client tool

Use execute: 'client' on defineTool:

import { defineTool } from '@helix-agents/core';
import { z } from 'zod';

export const editContentTool = defineTool({
  name: 'editContent',
  description: 'Apply structured edits to the current document.',
  inputSchema: z.object({
    edits: z.array(
      z.object({
        selector: z.string(),
        replacement: z.string(),
      })
    ),
  }),
  outputSchema: z.object({
    applied: z.number().int().min(0),
    failed: z.number().int().min(0),
    newVersionId: z.string().optional(),
  }),
  execute: 'client',
});

Mixing execute: 'client' with finishWith: true is rejected at defineTool time.

Client wiring (React)

The recommended setup is useHelixChat — one line that composes useChat + useResumeClientTools + the correct sendAutomaticallyWhen wiring:

import { useHelixChat } from '@helix-agents/ai-sdk/react';

const chat = useHelixChat({
  id: sessionId,
  transport,
  toolHandlers: {
    editContent: async (input, { abortSignal }) => {
      return await applyEdit(input, abortSignal);
    },
  },
});

useHelixChat handles three things you can't safely skip:

1. Stable chat.id comparison in the dispatcher — without this, client-tool handlers re-fire on every parent re-render.
2. Helix-flavored sendAutomaticallyWhen — the AI SDK v6 stock helper lastAssistantMessageIsCompleteWithToolCalls is stateless and causes a runaway POST loop after tool-output submissions.
3. Dev-mode safety warnings that fire when addToolOutput doesn't appear to reach the server.

For finer control, drop down to the building blocks:

import { useHelixSendAutomaticallyWhen, useResumeClientTools } from '@helix-agents/ai-sdk/react';
import { useChat } from '@ai-sdk/react';

const sendAutomaticallyWhen = useHelixSendAutomaticallyWhen(sessionId);
const chat = useChat({ transport, sendAutomaticallyWhen });
useResumeClientTools({ chat, toolHandlers });

Don't use AI SDK's stock lastAssistantMessageIsCompleteWithToolCalls

That helper is stateless: once a terminal tool part appears, every subsequent sendAutomaticallyWhen evaluation returns true, which causes AI SDK to fire an infinite chain of POSTs (490 POSTs in 58s observed in production). Use useHelixSendAutomaticallyWhen or useHelixChat instead.

See the @helix-agents/ai-sdk README for the full composability ladder including non-React (createHelixSendAutomaticallyWhen).

The useResumeClientTools hook replaces ~280 LOC of manual dispatcher scaffolding (the processedToolCallsRef + seededForSessionRef + 500ms forceUpdate pattern v6 customers maintained). Failures are surfaced to AI SDK automatically via chat.addToolOutput({ state: 'output-error', ... }), so the LLM sees a normal failed tool result on the next step.

Approval-gated tools (requireApproval) emit tool-approval-request parts that useResumeClientTools intentionally skips — those route through chat.addToolApprovalResponse(...). See the approval gates guide for the approval UX.

See the canonical reference at examples/client-tools/.

Error handling: onClientToolError

useHelixChat and useResumeClientTools accept an optional onClientToolError callback that fires once per client-tool failure. The callback receives a phase discriminant:

• 'handler' — your handler function threw or rejected. The hook will submit an output-error to the agent automatically.
• 'submit' — the hook tried to submit the result to the server but chat.addToolOutput failed (network error, AI SDK invariant). The on-screen tool ran successfully but the agent will time out via the pending-call deadline. Surface this to the user (retry banner, toast).

const chat = useHelixChat({
  id: sessionId,
  transport,
  toolHandlers: { editContent: applyEdit },
  onClientToolError: (err, { toolName, toolCallId, phase }) => {
    if (phase === 'submit') {
      toast.error(`${toolName} ran but couldn't reach the server. Retrying...`);
    } else {
      console.error(`${toolName} failed:`, err);
    }
  },
});

Returning undefined is treated as a tool error

A handler that resolves with undefined (e.g. a missing return statement) is detected by the hook and submitted to the agent as an output-error:

toolHandlers: {
  editContent: async (input) => {
    await applyEdit(input);
    // FORGOT return — this is now an error, not a successful empty result
  },
}

The agent receives errorText: "Client tool 'editContent' (toolCallId) returned undefined. Handlers must return a value (or throw to signal failure)." and onError fires with phase: 'handler'.

If your tool intentionally has no payload to return, return an explicit value such as { ok: true } or null. Throwing also works and is preferred when the handler can fail in distinguishable ways.

Other falsy values are valid. A handler that returns null, 0, '', or false will forward that value to the agent unchanged — only the explicit absence of a return (i.e., undefined) is treated as the "missing return" bug.

Direct API (without Vercel AI SDK)

import { JSAgentExecutor } from '@helix-agents/runtime-js';

const executor = new JSAgentExecutor(stateStore, streamManager, llmAdapter, {
  logger: myLogger, // optional
  agentRegistry: { [myAgent.name]: myAgent }, // enables output-schema validation on submit
});
const handle = await executor.execute(agent, { message: 'edit it' }, { sessionId });

// Wait for the run to suspend at the client-tool boundary.
const result = await handle.result();
if (result.status === 'suspended_client_tool') {
  // Submit the client-side result (kind discriminator was added in v7).
  const answer = await executor.submitToolResult({
    kind: 'client-tool-result',
    sessionId: handle.sessionId,
    toolCallId: result.suspended.toolCallIds[0],
    result: { applied: 3, failed: 0 },
  });

  // The answer says what the stored result caused (see "Continuation model").
  // On runtime-js the caller continues the turn: `caller_resumes`.
  if (answer.status !== 'unknown_tool_call' && answer.continuation.kind === 'caller_resumes') {
    const resumed = await executor.resume(agent, handle.sessionId);
    await resumed.result();
  }
}

The kind discriminator selects the variant. For client tool results, kind: 'client-tool-result' is optional (it's the default — v6 callers that submit { sessionId, toolCallId, result } without kind still parse correctly). For approval-gate decisions, kind: 'approval-response' is required along with the approvalId from the tool_approval_request chunk.

Wire agentRegistry so the framework validates client-submitted results against each tool's outputSchema. The registry accepts a Record<agentType, AgentConfig>, a Map, or a (agentType) => AgentConfig | undefined callback for dynamic resolution. Without it, submit-side validation is skipped — clients can submit arbitrary shapes and the LLM may receive malformed data.

Submit every call of a batch before resuming

When one step emits several client-tool calls, result.suspended.toolCallIds lists all of them, and the session continues only once every one has its answer: a submitted result or error, or a deadline that has lapsed (the continuation turns that into the call's real client_tool_timeout result). The rule is core's isClientToolBatchPartial (a submit's continuation decides with isSubmittedBatchReadyAtLastSubmit) and holds on every runtime:

• resume({ mode: 'continue' }) and resume({ mode: 'with_message' }) on a session whose batch is still partial are refused before anything is written, with a retryable HelixError whose code is state_not_resumable (the message names the calls still awaited). No model call runs and a with_message message is not stored. (A continue resume of a run whose stop is recorded but not yet resolved is not refused: it ends the run by that stop, interrupted or, for an abort, failed with framework_cancelled, with no model call. On runtime-js, the Cloudflare DO and Cloudflare Workflows.)
• resume({ mode: 'from_checkpoint' }) to a checkpoint whose restored batch is still partial is not refused. The rewind runs, and the new run takes the restored calls over and suspends again at once, without calling the model or inventing a result for the unanswered calls. A rewind that discards the unanswered calls continues normally.
• Every submit before the last one answers continuation: { kind: 'awaiting_batch' } (see Continuation model): nothing continues yet. Readiness is decided at the batch's last submittedAt (core isSubmittedBatchReadyAtLastSubmit), never by a reader's clock. The chat handler's submit path answers an empty stream until then, the Cloudflare DO and CF Workflows start the continuation only on the last answer, and DBOS keeps waiting in the workflow for the missing calls.

So a client may submit the calls of a batch one at a time. To abandon a batch, do it explicitly: send the new message as resume({ mode: 'with_message', message, abandonPending: true }) on CF Workflows, runtime-js or the Cloudflare DO (below), submit an error result for each unanswered call, or call interrupt().

This is a breaking behaviour change (see the upgrade guide). Before it, a resume() after a partial submit (and, on the DO, the submit itself) continued the run: the drain failed each unanswered call as runtime_restarted, and the real result submitted later got already_completed (CP-82). On Temporal the same resume left the session active with no workflow running (CP-86).

Routing invariant

Consumers never submit against a sub-agent's sessionId. Always use the root sessionId. The framework maintains SessionState.clientToolCallOwnership on root sessions and routes the submission to the owning sub-agent transparently.

If your agent tree is:

• Root session parent-abc
• Sub-agent session child-xyz (with rootSessionId: 'parent-abc')

...and the sub-agent's LLM emits a client tool call tc-42, the client still submits with:

executor.submitToolResult({
  kind: 'client-tool-result',
  sessionId: 'parent-abc', // ALWAYS the root
  toolCallId: 'tc-42',
  result: ...,
});

The framework resolves clientToolCallOwnership['tc-42'] → 'child-xyz' and writes the response into the owning sub-agent's pendingClientToolCalls durable map. The next executor.resume({ sessionId: 'parent-abc' }) cascades down to the suspended child via the same ownership table. (For runtime-dbos, which uses workflow-native messaging, the dispatch is DBOS.send to the owner's workflow ID resolved from clientToolCallOwnership — see the runtime-dbos client tools doc.)

Continuation model

submitToolResult is a durable write on every runtime: it records the result into pendingClientToolCalls. What happens next differs by runtime, so the answer says what the stored result caused: accepted and already_completed carry a required continuation (core SubmitContinuation), decided by the runtime from a read of the owning session after the write. A caller acts on it and never infers it by re-reading state.

Runtime	continuation on a complete batch	Who continues the turn
runtime-js	caller_resumes	a follow-up executor.resume()
runtime-temporal	caller_resumes	a follow-up executor.resume() (new workflow instance)
runtime-dbos	new_run { runId }	the parked workflow: DBOS.send wakes its DBOS.recv(), and it continues the turn in that new run
runtime-cloudflare (Workflow)	new_run { runId, startSequence? } (or consumed)	the __submit__ instance the submit started
runtime-cloudflare (DO)	new_run, deferred (or consumed)	the DO's wake, or the drive / run that already carries the turn

The other kinds, on every runtime:

• Partial batch: awaiting_batch. Other calls of the batch are still unanswered (see above); nothing continues yet.
• A re-send of a drained answer: consumed. An earlier continuation already took the answer (already_completed whose pending entry is gone, or the durable completed marker). accepted is consumed only after a stop (next item).
• A stop recorded after the answer was stored: consumed (CF Workflows and the CF DO, WA MR2 §7.1). The run is settled by its stop, nothing continues, and the stop's terminal discards the stored answer (CF Workflows' __submit__ is never created; the CF DO starts no drive). A stop recorded before the answer arrived answers run_stopped instead (below).
• deferred { blockingRunId } (the CF DO): a run of the session carries (or carried) the continuation; attach to the session stream.
• caller_resumes on the CF DO: nothing else will continue it (for example a wake that did nothing), so the caller's resume() continues it or answers typed.

The direct pattern stays valid everywhere. submitToolResult() then resume(agent, sessionId) continues the turn on runtime-js and Temporal, and on CF Workflows resume({ mode: 'continue' }) adopts the same __submit__ instance, whether it is still waiting to start or its run has already started (one instance, no loser, no wait). On DBOS and the CF DO the runtime already continues the turn, so act on the kind instead of calling resume().

CF Workflows: the last answer starts the continuation. The submit creates (or adopts) a deterministic instance agent__<type>__<root session>__submit__<key> (the key is 12 hex characters of SHA-256 of the suspended run id and a generation 1..3), which waits in its body for the suspended instance to close (continuationPredecessorTimeoutMs, default 10 min), then drains the answers and runs the model. Details and the bounds are on the Cloudflare Workflows page. A consequence for direct callers: a later resume({ mode: 'with_message' }) waits for that continuation to settle and then becomes the next turn (two entries, two model calls). For ONE entry and ONE model call with the message, send with_message + abandonPending (below) instead of submitting first, or do not submit the answers first. A wait past continuationPredecessorTimeoutMs + entryOutcomeFailSafeMs throws RunStartOutcomeUnknownError with reason: 'continuation_awaiting_predecessor' (no run yet; nothing is abandoned or terminated, so read the session) or the RETRYABLE 'continuation_running' (the continuation's run is still running; the message was not written, so send it again once it settles). The same retryable reason, with ownerStopped: true, comes earlier when the continuation's instance closed without its terminal commit and 3 settles of its run all failed.

Why the DO and DBOS auto-continue: a Durable Object has no long-running process between requests, so its wake (called inline in /submit-tool-result once every call of the step's batch has its answer) is the only thing that can advance the suspended run; DBOS's workflow is alive, blocked on DBOS.recv(). CF Workflows self-wakes so the turn never depends on a second HTTP request surviving (CP-104).

DO: the submit materializes before it writes. The DO's /submit-tool-result first resolves the owning agent (to validate the result against the owning tool's outputSchema) and materializes the agent the continuation would run (resolver, LLM adapter, usage store, completion check). A failure there answers typed (for example 503framework_agent_resolution_failed, 404 framework_agent_not_found, 400 for a completion refusal) and writes nothing; the call stays pending. A sub-agent-owned call is validated by the owning DO. A submit forwarded to another DO relays that DO's typed refusal verbatim.

DO: a continuation that cannot start is never silent. The continuation is a wake drive: a failure is classified. A non-retryable failure (for example the agent type is no longer registered) ends the run failed with its typed errorDetail; a retryable or unclassified one re-arms the alarm under the crash-loop budget, and exhausting it ends the run failed with framework_auto_resume_exhausted, whose cause is the last failure (CP-70).

DO: the submit answers from the continuation. A submit that completes its batch is answered 200 { "status": "accepted", "continuation": … } only once the continuation's entry has started (or another drive owns the run), so a read right after the answer sees the call consumed (CP-53). The continuation comes from the wake: new_run { runId, startSequence? } when the wake started the run; deferred { blockingRunId } when a registered drive or a live run carries the answer; consumed when another entry already drained it; else caller_resumes. When the continuation's entry fails, the submit answers typed instead: a refused or fatal entry answers the run's typed failure; a transient one (the run is re-armed and continues) answers its code on a non-retryable row. Past concurrentEntryTimeoutMs (30 s) it answers 500 transport_error ("the run's continuation was not confirmed within N ms": it may not have started yet, or may already have consumed the result) while the continuation keeps running (the unconsumed-submit alarm re-checks every second). The result is already stored in every case, so a client re-reads the snapshot (useHelixChat().retry()) and never re-submits. A duplicate or re-submit of a result that is already stored or consumed (a second tab, a client retrying on a 5xx) answers 200 { "status": "already_completed", … } at once, without waiting for the continuation, and never replaces the stored result. already_completed is about this submit's result being recorded, not the run or its continuation having finished: the agent may still be running, so read the snapshot for its status. A submit forwarded to another DO answers a retryable 503 past the bound plus 1 s (a re-submit answers already_completed). The full table is on the Cloudflare Durable Objects page.

When you use the AI SDK (useHelixChat), the sendAutomaticallyWhen-driven follow-up request opens an SSE stream so the client reads the continuation as it happens. handleChatStream acts on the last answered submit's continuation: for new_run and deferred it attaches to the live stream without calling resume(); for caller_resumes the same request drives resume(); for awaiting_batch and consumed it answers a 200 that ends the client's tool step, so a stock useChat auto-send is sent once (CP-180, submit response contract). Either way the consumer code is identical across runtimes.

Fixed: dropped stream frames on re-attach and replay (RM-51)

Before @helix-agents/runtime-cloudflare 5.20.0, DOStreamManagerClient.createResumableReader yielded only the first SSE frame of each network read, so the client-tool re-attach (including a following client-tool call) could arrive truncated when the DO's writes were coalesced. This is fixed in 5.20.0; upgrade if you see truncated attached or replayed messages. See RM-51 in the conformance findings.

This flow does NOT emit data-stream-resync

stream_resync / data-stream-resync is a recovery-only signal (retry(), or a runtime-js / Cloudflare DO resume() that rolled back uncommitted messages or stream chunks past its checkpoint). The client-tool continuation flow — including the DO re-attach — does not emit it. If you saw data-stream-resync on every client-tool turn on an older release, that was a symptom of the getLatestCheckpoint truncation bug (fixed); it is no longer emitted on the happy path. Do not attempt to "fix" DO by removing ensureExecutionContinues() — that would strand suspended DO runs (there is nothing else to advance them).

A new message over pending calls

A user who types a new message while the run waits on client calls abandons those calls. Send it as ONE resume entry:

await executor.resume(agent, sessionId, {
  mode: 'with_message',
  message: 'never mind, do Y instead',
  abandonPending: true,
});

ResumeOptionsWithMessage.abandonPending answers every unanswered awaited call client_tool_abandoned (a result already stored is kept as given), commits the message after the results, and runs ONE model call. No submit runs, so no submit continuation runs a model step on the abandoned results alone. It is supported on CF Workflows, runtime-js and the Cloudflare DO, and only for a session suspended on client-tool calls alone (core planWithMessageEntry; a suspension that also awaits children is the busy refusal (AgentAlreadyRunningError, state_already_running); any other session it cannot honour is refused state_not_resumable, FU-WITH-MESSAGE-MIXED-CHILDREN-ORDER). Temporal and DBOS refuse it with framework_not_supported before any write (FU-TEMPORAL-ABANDON-SEND-STUCK, FU-DBOS-ABANDON-SEND-MODEL-STEP). On CF Workflows a session already owed a submit continuation (the batch's last answer was submitted) has nothing left to abandon: the call is the busy loser (AgentAlreadyRunningError, nothing written).

handleChatStream does this for you (path 4) on an executor that declares MinimalExecutor.supportsAbandonOnResume; on Temporal and DBOS it keeps the per-call abandonment submits.

A with_message resume drains first — for a session PAUSED on client-tool calls alone (core planWithMessageEntry). There a resume({ mode: 'with_message' }) that has client-tool results to drain (with or without abandonPending) puts them, and a deferred finishWith call's not-executed result, in the log BEFORE the message, so every tool_use keeps its adjacent tool_result. The same holds for a crashed run's active session that still holds its calls (the entry's run takes them over and drains them). An interrupted session holds none: a stop's end clears them, and the next entry pairs their tool_use as not executed. A suspension that also awaits children still puts the message first (FU-WITH-MESSAGE-MIXED-CHILDREN-ORDER). On CF Workflows the message is the entry's own append commit after the drain; on runtime-js and the CF DO the results and the message ride the run-start commit, or, when a result needs a tool to run first (an approved approval), the message waits on a pending entry and lands right after that result. Temporal and DBOS still commit the message first (FU-TEMPORAL-WITH-MESSAGE-DRAIN-ORDER, FU-DBOS-WITH-MESSAGE-DRAIN-ORDER).

retry() after abandon-and-send, or after a drain-first with_message resume, works the same as any retry. When the run failed before committing a step, no options.message is needed: core planRetry rule 4 re-runs the entry from its own marked input row. After committed steps it needs options.message, like any other retry (see RetryOptions).

Per-runtime durability (v7)

Property	runtime-js	runtime-cloudflare (DO)	runtime-temporal	runtime-cloudflare (Workflow)	runtime-dbos
HITL support in v7.0	Full	Full	Full	Full	Full (DBOS-native)
Durable across restart	Yes (state store)	Yes (DO SQLite + state store)	Yes (state store; workflow exits on HITL)	Yes (workflow replay + state store)	Yes (workflow replay)
Max practical wait	State-store TTL	State-store TTL	State-store TTL (workflow exits on HITL)	State-store TTL (workflow exits on pause)	Days+
Submit routing	submitToolResult + resume	submitToolResult (the DO wakes)	submitToolResult (durable) + resume (new instance)	submitToolResult (starts __submit__)	DBOS.send (idempotent)
Hibernation guard	n/a	Removed in v7 (DOs free to evict)	n/a (workflow exits on HITL)	n/a (workflow exits on pause)	n/a

In v7, the in-memory wait is gone on every runtime. runtime-js is durable across process restarts (state lives in the state store, not in the JS heap). The Cloudflare DO hibernation guard that v6 used to keep DOs awake during pauses is removed. A call's deadline still fires with no request: the DO's client-tool-deadline alarm subscriber expires the lapsed calls and continues the run (S-A, see Cloudflare Durable Objects). CF Workflows has no deadline timer yet: a lapsed call expires on the next request (FU-CFW-CLIENT-TOOL-DEADLINE-NO-TIMER).

Temporal uses the v7 stateless model: the workflow exits at HITL boundaries with status: 'suspended_*' and durable suspension state via commitSuspendedStep. submitToolResult is a durable write only — no Temporal signal is sent (the workflow has already exited). executor.resume() starts a NEW workflow instance with workflow ID suffix __resume-<runId> (the run id the call minted for its entry); the new workflow's mode='resume' branch calls applyResultsAndReload to drain submitted results into the message log, fire onMessage + afterTool hooks, and proceed.

Cloudflare Workflows uses the v7 stateless model: the workflow exits at HITL boundaries with status: 'suspended_*' and durable suspension state via commitSuspendedStep. The submit that completes the batch starts a new mode: 'resume' instance (__submit__) that waits for the suspended instance to close, drains the queued submissions via applyResultsAndReload and continues; an explicit resume({ mode: 'continue' }) adopts it. This eliminates billable wall-time during HITL waits (~80% reduction on multi-minute approvals).

DBOS specifics

runtime-dbos uses DBOS Transact's native messaging primitives — DBOS.recv for the suspension and DBOS.send for the wake-up — backed by the dbos.workflow_events table in Postgres. There is no separate timer / alarm subsystem: the recv deadline is a workflow-body argument, and the platform handles both the suspend and the timeout. runtime-dbos is the only v7.0 HITL-capable runtime that does not participate in the unified suspensionContext model — it suspends inside the workflow body, not at request boundaries.

• Recv re-arms naturally on replay. When the DBOS process restarts mid-wait, the workflow body re-runs from the top; deterministic @DBOS.step calls return cached results, and the await DBOS.recv(toolCallId, deadlineSec) re-arms exactly where it was. Buffered messages from DBOS.send are consumed by the re-armed recv. No runtime_restarted error path is needed.
• Submit idempotency is platform-level. DBOSAgentExecutor.submitToolResult passes the toolCallId as the idempotencyKey argument to DBOS.send, so duplicate submits are deduplicated by DBOS itself. The framework's pendingClientToolCalls[toolCallId].submittedAt check is defense-in-depth.
• Persistent mode shares the workflow. Persistent agents use DBOS.recv('inbox') to drive turns and DBOS.recv(toolCallId) to wait for client-tool results. Both run in the same long-lived workflow on different topics — they don't interfere, and idle-TTL hibernation works on either.
• Sub-agent ownership routing dispatches to a workflow ID. routeSubmitToolResult reads clientToolCallOwnership[toolCallId] from the root session, resolves the owner's workflow ID via resolveOwnerWorkflowId (deterministic for standard mode; from state.metadata.dbosWorkflowId for persistent mode), and calls DBOS.send to that workflow.
• Interrupt during a wait. interrupt() cancels the workflow; the recv rejects. Cleanup runs in lifecycle/interrupt.ts (clearPendingClientToolsOnInterrupt, called immediately after DBOS.cancelWorkflow), NOT from the workflow body's catch path — once DBOS.cancelWorkflow has fired, every @DBOS.step() call from inside the workflow throws via DBOS's checkIfCanceled rejection, so the workflow body cannot reach clearPendingClientToolCallStep. The cleanup talks directly to the state store (loadState/saveState) and clears both pendingClientToolCalls (on owner) and clientToolCallOwnership (on root). Late submits after cancellation return unknown_tool_call, not already_completed. See packages/runtime-dbos/docs/client-tools.md#limitations for the rationale.

For the full set of races and recovery paths, see packages/runtime-dbos/docs/race-conditions.md and the DBOS-specific guide at packages/runtime-dbos/docs/client-tools.md.

Timeouts

Every suspension has a deadline. Configure per-tool:

defineTool({
  name: 'editContent',
  // ...
  execute: 'client',
  clientToolTimeoutMs: 60_000, // 60 seconds
});

Fallback chain: per-tool → AgentConfig.clientToolTimeoutMs → DEFAULT_CLIENT_TOOL_TIMEOUT_MS (5 minutes).

On timeout, the runtime emits a tool_end chunk with error: 'client_tool_timeout'. The LLM sees a normal failed tool result and can react (retry, give up, report to the user).

HTTP contract

In v7, the canonical chat-handler route is POST /chat/{id}/submit-tool-result (registered by AgentServer's chat handler middleware). The legacy executor-level POST /submit-tool-result route still works but does not emit chat- specific resume cookies.

Content-Type: application/json

{
  "kind": "client-tool-result",
  "sessionId": "root-session-id",
  "toolCallId": "tc_123",
  "result": {...}
}

Or for errors:

{
  "kind": "client-tool-result",
  "sessionId": "root-session-id",
  "toolCallId": "tc_123",
  "error": "user_cancelled"
}

For approval-gate responses:

{
  "kind": "approval-response",
  "sessionId": "root-session-id",
  "toolCallId": "tc_123",
  "approvalId": "<runId>::tc_123",
  "approved": true
}

Responses:

• 200 { "status": "accepted" } — submission recorded. On a Cloudflare Durable Object, a submit that completes its batch is accepted only once the continuation's entry has started; it can instead answer a typed failure or 500transport_error (continuation model), with the result already stored.
• 200 { "status": "already_completed" } — tool already had a result (idempotent). It says this call's result is recorded, not that the run is done.
• 404 { "status": "unknown_tool_call" } — no pending call matches (also returned when a rewind discarded the call).
• 404 { "status": "unknown_tool_call", "reason": "run_stopped" } — the call belongs to a run that was stopped (interrupt / abort). The recorded stop wins (WA MR2 §7.1): the submit reads the stop of the call's run before it stores anything, so nothing is stored and nothing continues. A stop that is recorded but not yet resolved (for example one another process recorded on a paused run) is resolved by this submit, so the session is interrupted (or failed with framework_cancelled for an abort) when the answer arrives. A stop that already ended the run answers the same, as long as the call is still unanswered in the log. Do not re-submit: send a new message to start the next turn (its continuation heal closes the call with a not-executed result). reason is optional and additive; a client that ignores it reads an unknown call. The ai-sdk chat resume path answers it 409 state_not_resumable with cause: 'run_stopped' when no other answer of the request applies, else as a data-resume-rejected rejection { reason: 'unknown-tool-call', toolCallId, cause: 'run_stopped' } beside the applied ones, inside the answer that ends the client's tool step (submit response contract). Every runtime answers it: runtime-js, the Cloudflare Durable Object, CF Workflows (no __submit__ continuation is created), DBOS (the parked workflow is cancelled and nothing is sent to it) and Temporal.
• 409 { "error": "...", "code": "state_run_superseded" } — a late submit to a run that another entry has superseded. Each pending entry records the run that suspended on it, and the submit's write is fenced on that run, so nothing is written. A run-start commit that takes the session's pending calls over (a resume) re-stamps them to the new run, and a submit that raced that commit re-reads once and lands on the new run. The 409 means no run will use this result: re-read the session. createHelixChatTransport().submitToolResult and DOFrontendExecutor.submitToolResult throw a typed RunSupersededError for it, and route handlers can map the error with submitToolResultErrorResponse(error) from @helix-agents/ai-sdk/cloudflare. Direct executor.submitToolResult() callers get the RunSupersededError thrown.
• 400 { "error": "The submit is not a valid client-tool answer or approval", "code": "validation_error" } — the body is not a valid submit (missing sessionId / toolCallId, bad JSON, JSON nested too deeply, …). The message is fixed, so the answer never reveals the schema; with verboseValidationErrors: true it also carries the Zod details.
• 400 { "error": "The submitted result failed the tool output validation", "code": "INVALID_RESULT", "toolName": "...", "toolCallId": "..." } — the submitted result didn't match the owning tool's outputSchema. The error is core's fixed INVALID_RESULT_MESSAGE, so the answer never reveals the tool's outputSchema. With verboseValidationErrors: true (the same gate as validation_error's details) the answer instead carries the detailed message and the Zod issues, so the client can surface them or retry. This holds on agent-server and the Cloudflare Durable Object; the ai-sdk submitToolResultErrorResponse always answers the fixed body. Error-submissions (error set, result undefined) skip this validation — they're always accepted regardless of schema.
• 411 { "error": "...", "code": "validation_error", "cause": "length_required" } — agent-server only: the request had no Content-Length header and a non-empty Transfer-Encoding header (typically chunked). Its host parses the body before the route sees it, so the route can only bound a body that declares its length. The Cloudflare Durable Object reads the body itself under the cap (core readRequestText): it accepts a chunked submit under the cap and counts and refuses one over it with the typed 413 below (cause: 'request_body'). See the Security section above for rate-limit / body-size guidance.
• 413 { "error": "...", "code": "validation_payload_too_large", "retryable": false, "cause": "tool_result" | "request_body", "limitBytes": N, "sizeBytes": M } — the answer is too large (the submit size limit). Nothing was stored: the call stays pending, so answer it again with a smaller result. Typed client-side error: HelixPayloadTooLargeError from @helix-agents/ai-sdk (its cause, limitBytes, sizeBytes).

The submit size limit

One limit governs every submit (DI-51): the UTF-8 byte length of JSON.stringify(result), at most maxResultBytes (default 1 MB, DEFAULT_SUBMIT_MAX_RESULT_BYTES).

• The executor owns it. Configure it on the executor: submitToolResultSchema: createSubmitToolResultSchema({ maxResultBytes }), an option of JSAgentExecutor, CloudflareAgentExecutor, TemporalAgentExecutor, DBOSAgentExecutor and the CF Durable Object's config (which passes it to every executor it builds). The executor exposes it as executor.submitLimits and refuses an answer over it in submitToolResult (the direct API too) before anything is read or stored.
• Hosts read it, never configure it. @helix-agents/agent-server and handleChatStream take the limits from the executor they wrap and pre-check every answer of a request before storing any, so they refuse exactly where the executor would. One answer over the limit refuses the whole request. An executor that declares no limits is held to core's defaults (fail closed): a host never reads an unbounded body or forwards an unbounded result. The CF DOFrontendExecutor declares the limits of its submitToolResultSchema option (default: core's, the DO's own default): a DO that raises its limit passes the same schema to its DOFrontendExecutor. A mismatch fails safe: the edge refuses first, or the DO's own typed 413 propagates with the same body.
• One typed refusal. Core payloadTooLargeError builds it: code validation_payload_too_large (category validation, never retryable), HTTP 413, body { error, code, retryable: false, cause, limitBytes, sizeBytes? } where cause is tool_result (the answer) or request_body (a body over the transport guard). The guard is derived, never a second limit: submitRequestBodyCap = maxResultBytes × 4. A chat request's own maxRequestBytes (10 MB) is a separate whole-request cap and answers the same typed 413 with cause: 'request_body'.
• The store's limit (MA-31). A submitted answer waits in the session's stored pending map until its batch resumes, so the state store's limit applies too. An answer that would push the stored map over the store's maxStateBytes (the CF stores: 1.8 MB by default) is refused with core's state_size_exceeded (HTTP 413, body { error, code, retryable: false, cause, limitBytes, sizeBytes }), nothing stored; the batch's earlier answers stay stored. A maxResultBytes over a finite store limit can never work, so JSAgentExecutor, CloudflareAgentExecutor and the CF Durable Object refuse it when they are constructed (a typed validation_error, core assertSubmitLimitFitsStore); a store with no limit is unaffected. A maxResultBytes equal to the store limit is accepted, but a result of exactly that size still cannot be stored (the pending entry adds bytes), and is refused typed.
• Client side. A chat route's refusal is a non-2xx JSON body, so useChat / useHelixChat put it in chat.error; parseHelixChatError(chat.error) returns { code, retryable, cause, limitBytes, sizeBytes }.

The core helpers: checkSubmitResultSize(result, limits?, toolCallId?) (the one check), assertSubmitWithinLimits, measureSubmitResultBytes, resolveSubmitLimits, submitToolResultSchemaFor(limits) (the full submit schema for given limits) and isPayloadTooLargeError.

Programmatic error classification

Consumers handling tool-result errors should prefer the exported type guards over string-matching on errorDetail.code:

import { isKnownErrorCode, isClientToolErrorCode } from '@helix-agents/core';

if (!result.success && result.errorDetail?.code) {
  if (isClientToolErrorCode(result.errorDetail.code)) {
    // 'client_tool_timeout' | 'aborted' | 'runtime_restarted' | 'disposed' | 'validation_failed'
    handleClientToolFailure(result);
  } else if (isKnownErrorCode(result.errorDetail.code)) {
    // Framework error code (provider_*, tool_*, state_*, etc.)
    handleFrameworkError(result);
  } else {
    // Extension code — custom integration or future addition
    handleCustomError(result);
  }
}

The guards narrow result.errorDetail.code: ErrorCode | (string & {}) to the specific unions, so downstream switch statements get compile-time exhaustiveness checks. KNOWN_ERROR_CODES is derived from the ErrorCode union via a compile-time completeness assertion — new codes added to the framework automatically land in the Set.

Durability of already_completed

After this release, already_completed is a durable signal, not best-effort. The framework persists a completedClientToolCalls marker on the root session whenever ownership is cleared (i.e., when a client tool completes). Duplicate submits within AgentConfig.completedTombstoneRetentionMs (default 24h; the registering agent's value is recorded on the call's pending entry, PendingClientToolCall.completedRetentionMs, when the call is registered, so clearing the call never looks the agent up) return already_completed regardless of:

• The runtime restarting (Node process crash, deploy)
• The Cloudflare Durable Object being evicted
• The Temporal worker being replaced
• The in-memory completed cache (10-min TTL) expiring

Past the retention horizon, duplicates surface as unknown_tool_call (HTTP 404). Both responses should be treated identically by clients — both indicate "the framework does not need this submission; treat as no-op." The framework's canonical helix-chat-transport.ts already gates on !res.ok && res.status !== 404 and applies this no-op pattern; consumers writing custom transports should mirror it.

Configuring retention

const myAgent = defineAgent({
  // ...
  completedTombstoneRetentionMs: 60 * 60_000, // 1 hour
  // Defaults to 24h. Generous for any reasonable browser-retry /
  // service-worker-retry / deploy-rollover window. Lower it if you
  // have unusual storage or forensic constraints.
});

defineAgent throws unless the value is a finite number >= 0. 0 turns retention off: each marker is reaped by the next marker write.

Storage cost

Markers are persisted in the root session's completedClientToolCalls JSON map (Postgres JSONB / D1 + DO SQLite TEXT / Redis hash field). Each marker entry is a {toolCallId, completedAt} pair, ~50-100 bytes serialized. Markers are reaped opportunistically on the next clearOwnership (the next time a client tool completes on the same session). Steady-state size is bounded by retention × concurrent-call-rate.

Migration

Schema migrations run automatically on first startup of the new runtime. They are additive (new column with default NULL); old code reading the new column ignores it, and old rows return completedClientToolCalls: undefined, which falls through to today's behavior. No data backfill required. Rolling deploys are safe in either direction.

• Postgres: migration v5 adds completed_client_tool_calls JSONB.
• Cloudflare D1: migration V8 adds completed_client_tool_calls TEXT.
• Cloudflare DO SQLite: migration v4 adds completed_client_tool_calls TEXT.
• Redis: new hash field completedClientToolCalls. No migration needed — Redis hash fields are dynamic.
• In-memory: passthrough, no schema concept.

Known limitations

• Remote sub-agents cannot use client-executed tools — the submit path has no route to the remote server's pending state across the HTTP boundary. This is enforced: when a remote sub-agent suspends on a client tool, the parent fails fast with RemoteSubAgentClientToolUnsupportedError (the remote-dispatch result carries failureReason: 'client-tool-unsupported') instead of hanging. Detection rides on the awaitingClientTool flag exposed by the remote /status response. Tracked for future support in GitLab #107.
• runtime-js durability is best-effort; orphan calls resolve with runtime_restarted on resume after a process restart. For durable client-tool waits, use runtime-temporal, runtime-dbos, or runtime-cloudflare.
• Mid-execute suspend (ctx.suspend()) is not part of the public API in v1. Tools may opt into client execution only via execute: 'client' (the "whole body is a suspend" shape). The internal plumbing is designed to permit ctx.suspend() as a future addition without breaking changes.
• Error message localization: humanizeClientToolError() emits English text that's delivered verbatim to the LLM. Consumers running agents in non-English contexts will see English error messages in transcripts. There is no injection point for custom humanizers today — if your deployment needs localized error text, intercept the tool-result message in an afterTool / onMessage hook and rewrite the error field before it reaches the next LLM turn. Full humanizer injection is planned for a later release.

Operating in Production

Runbook for diagnosing and remediating client-executed tool issues in production. All three runtimes emit the same structured Logger events, so the diagnostic flow is runtime-agnostic unless noted.

Monitoring

The framework emits structured Logger events (info / warn / error via the configured Logger) and OpenTelemetry spans via tracing-langfuse. Wire a Logger (pino, winston, or any compatible structured logger) into every JSAgentExecutor, TemporalAgentExecutor, DBOSAgentExecutor, Cloudflare executor, and AgentServer so the events below land in your log aggregator.

Log-level configuration. client_tool.suspended and client_tool.submitted are emitted at debug level (they fire on every client-tool call — 2× per call — and would flood at high volume under default info-level loggers). Timeout / abort / ownership-retry events remain at warn / error. To debug a specific "why is this tool stuck?" scenario, raise the logger's level to debug:

// pino
const logger = pino({ level: process.env.HELIX_LOG_LEVEL ?? 'info' });
// Then deploy with HELIX_LOG_LEVEL=debug for a narrow window.

If your deployment can't restart-to-debug, consider adding a wrapper logger that demotes info → debug only for client_tool.* events while leaving the rest at info. Full per-event log-level control is planned for a later release.

Multi-service deployments. Set AgentServerConfig.serviceName to a stable identifier (e.g. "api-v2", "worker-pool-west"). AgentServer wraps the logger via withContext({ service: serviceName }) so every live telemetry event carries the tag. Fleet-wide aggregators can then filter on service to triage which deployment is emitting timeouts.

Key log events to watch:

• client_tool.suspended (debug) — tool call suspended awaiting submit; payload includes sessionId, runId, toolCallId, toolName, deadlineAt.
• client_tool.submitted (debug) — submit landed; payload includes sessionId, toolCallId, hasError (true when the client submitted error instead of result).
• client_tool.timeout (warn) — deadline expired without a submit; payload includes sessionId, toolCallId, deadlineAt.
• client_tool.aborted (warn) — agent interrupted/aborted while waiting.
• client_tool.orphan_recovery_started / client_tool.orphan_resolved (warn) — runtime-js only; emitted when a process restart leaves pending calls that must be drained with runtime_restarted.
• client_tool.ownership_write_failed / client_tool.ownership_clear_failed (error) — OCC (optimistic-concurrency) retry budget exhausted while writing the ownership map on the root session. Usually indicates sustained contention or a state-store outage.
• client_tool.ownership_retry (debug) — per-attempt OCC retry; turn on debug logging if you need to correlate retries with a failure.
• client_tool.send_event_failed / client_tool.signal_failed (error) — Cloudflare Workflow sendEvent, Temporal signal, or DBOS DBOS.send failed after the status check passed. Submit payload was persisted, but the waiting runtime did not wake up.
• client_tool.validation_failed (warn) — client submitted a malformed result that failed the tool's outputSchema. Only emitted when the executor was configured with an agentRegistry (otherwise submit-side validation is skipped).
• client_tool.submit_unknown (warn) — H5: submit couldn't be routed to a pending call. Payload includes sessionId, toolCallId, and a reason discriminator: root_state_not_found (session doesn't exist), ownership_map_absent (no ownership map on root), or ownership_missing_toolcall (ownership exists but this toolCallId has no entry). A trickle is normal during shard rebalancing; persistent volume indicates mis-routing (see completed cache scoping below).
• client_tool.clear_failed_exhausted (error) — N5: OCC retry budget exhausted while clearing a pending entry. Entry is stranded in state; next orphan drain may emit a duplicate tool-result message. Manual cleanup may be needed.
• client_tool.orphan_drain_clear_entry_failed (debug) — V4: per-iteration pending-entry clear failed during orphan drain. Each failure is recorded at debug level; the drain emits a single warn-level summary (orphan_drain_clear_entries_summary, BB7) at the end of the loop listing the total count + first-error for aggregation. Persistent occurrences indicate a state-store outage or OCC thrash.
• client_tool.orphan_drain_clear_entries_summary (warn) — BB7 (round 29): summary line emitted when any per-iteration clears failed during a single drain. Payload carries failedCount, firstToolCallId, firstError. Replaces the previous one-warn-per-failure pattern that flooded logs during sustained state-store outages.
• client_tool.orphan_drain_bulk_clear_failed (error) — Z4 + BB7: the post-loop bulk clear of the pending map failed. The per-iteration clears remain authoritative; the next resume will drain any leftover entries (the BB6 idempotency guard prevents duplicate tool-result messages). Operator action:
  1. Check state-store health — connection pool saturation, CPU, error rate.
  2. If the error is StaleStateError, a concurrent writer is modifying the session. Look for a runaway retry loop or a second runtime attached to the same sessionId.
  3. Cross-reference the orphan_drain_clear_entries_summary event for the same drain — co-occurrence means the store is degraded for this session across both paths.
  4. Sustained >1% of drains over 5 min — page the state-store on-call.
• client_tool.orphan_drain_clear_ownership_failed (warn) — H3: clearing the root-session clientToolCallOwnership entry for a drained tool failed. Ownership entry may leak on the root session; the next submit against the same toolCallId will return unknown_tool_call (or re-associate if a new pending call registers).
• client_tool.orphan_recovery_started / client_tool.orphan_resolved (warn) — S1: emitted when a JS process restart leaves pending calls that must be drained with runtime_restarted. orphan_recovery_started fires once per drain with the count; orphan_resolved fires once per entry with outcome: 'inherited' or the error string.
• submit_tool_result.body_too_large (warn) — N4: Cloudflare DO rejected a submit with HTTP 413 because the declared Content-Length exceeded 4× the active result-byte limit. Indicates either a misbehaving client or a need to raise the schema's maxResultBytes via createSubmitToolResultSchema({ maxResultBytes }).
• submit_tool_result.result_too_large (warn) — Cloudflare DO refused a submit whose result is over the limit (DI-51), whatever its Content-Length; carries toolCallId, sizeBytes, limitBytes.
• submit_tool_result.schema_limits_unreadable (debug, Cloudflare DO) — X1: the DO config's submitToolResultSchema could not expose its SUBMIT_SCHEMA_LIMITS stamp, so the framework fell back to DEFAULT_SUBMIT_MAX_RESULT_BYTES. Fires at most once per Durable Object lifetime (AA2 warn-once guard) to avoid flooding. Action: rebuild the schema via createSubmitToolResultSchema({...}) or re-stamp the symbol after wrapping; or, if default limits are intentional, ignore.

Privacy note: log payloads carry sessionId and toolCallId — stable identifiers that can be joined with user-identifying state. Treat them as PII-adjacent when forwarding to third-party log aggregators; configure field-level scrubbing as appropriate.

AA10 + BB17 (submitted error strings) — the framework does NOT auto-redact: error strings passed via submitToolResult({ error }) are persisted to conversation history as tool-result messages and forwarded to the LLM on the next step so it can react (retry, ask the user, give up). They are also emitted on the stream as the error field of the tool_end chunk.

There is no framework-level scrubber. There is no configuration flag to enable one. Sanitization is entirely the consumer's responsibility. If your tool error path could surface secrets or PII (stack traces, auth tokens, email addresses, internal IDs, database error messages containing row-level data), you MUST sanitize before calling submitToolResult. Consequences of failing to:

• Assume the string will be seen by the LLM. Phrasing the error conversationally ("the user cancelled the confirmation dialog") helps the model recover gracefully; machine-readable codes (confirm_cancelled) are opaque to it. The framework's built-in codes (client_tool_timeout, aborted, runtime_restarted, disposed, validation_failed) are mapped to humanized paragraphs via humanizeClientToolError(); user-supplied errors pass through verbatim.
• The string is persisted to conversation history. Session snapshots, conversation exports, fine-tuning-dataset dumps, and anything reading getAllMessages(sessionId) will carry it.
• The string is written to the stream. Anything listening to the stream (the browser, the caller UI, a relay service) sees it in the clear.
• The string is logged. client_tool.orphan_resolved and tool_end payloads carry the humanized text; the raw machine code (if one of the built-ins) is preserved in the errorCode field for aggregation.

Metrics: CLIENT_TOOL_WAIT_MS_METRIC records the wall-clock duration of each client-tool wait, keyed per tool call at resolve time. The metric stores type: 'client_tool_wait_ms' (→ source_type column) with the owning tool's name in source_name. The data JSON column's value field holds the durationMs.

K2 outcome counters (wired in M2 remediation round): one-per-occurrence counters for the three most common failure modes. Aggregate with SUM(value) (or COUNT(*)) for rate dashboards / alerts.

• client_tool_timeout — deadline elapsed before a submit arrived.
• client_tool_abort — session was interrupted or aborted while suspended.
• client_tool_runtime_restarted — orphan drained after a process restart (runtime-js) or DO eviction (Cloudflare DO).
• schema_limits_fallback (CC5/BB15, Cloudflare DO only) — submit landed with submitToolResultSchema that did not expose its SUBMIT_SCHEMA_LIMITS stamp; the Content-Length gate fell back to DEFAULT_SUBMIT_MAX_RESULT_BYTES (1 MB × 4). Batched in-memory and flushed at thresholds / alarms to avoid per-submit INSERT pressure (CC8). The associated log event submit_tool_result.schema_limits_unreadable fires at most once per DO lifetime; this counter is the continuous fleet signal.

Each counter uses the same source_type / source_name convention as CLIENT_TOOL_WAIT_MS_METRIC: source_name is the tool name, data.value is 1. Example aggregations:

-- Postgres: per-tool timeout rate (timeouts / total waits) over the last hour
WITH waits AS (
  SELECT source_name AS tool_name, COUNT(*) AS total
  FROM __agents_usage
  WHERE source_type = 'client_tool_wait_ms'
    AND timestamp > (extract(epoch from now()) * 1000 - 3600000)
  GROUP BY source_name
),
timeouts AS (
  SELECT source_name AS tool_name, SUM((data->>'value')::int) AS timed_out
  FROM __agents_usage
  WHERE source_type = 'client_tool_timeout'
    AND timestamp > (extract(epoch from now()) * 1000 - 3600000)
  GROUP BY source_name
)
SELECT w.tool_name,
       w.total AS total_waits,
       COALESCE(t.timed_out, 0) AS timeouts,
       ROUND(100.0 * COALESCE(t.timed_out, 0) / w.total, 2) AS timeout_pct
FROM waits w
LEFT JOIN timeouts t USING (tool_name)
ORDER BY timeout_pct DESC;

Cardinality guidance: per-tool breakdowns via source_name are stored row-by-row in the usage table and aggregated by your SQL layer — low cost. If you export these counters to a time-series system (Datadog, Prometheus), aggregate AT TYPE LEVEL (drop source_name) unless your tool set is small (<50 tools) — otherwise per-tool cardinality can cost more than the signal is worth.

Other alert-worthy thresholds (tune to your baseline):

• rate(client_tool_timeout[5m]) > 0.1 per session — sustained timeout rate above 10% usually indicates a tool or client pathology.
• rate(client_tool.submit_unknown[5m]) > 1 per session — above ~1/min per session suggests mis-routed submits.
• rate(client_tool.clear_failed_exhausted[1h]) > 0 — any occurrence is worth investigating (state inconsistency risk).

Example aggregations (assumes the executor is wired with a store-backed UsageStore):

-- Postgres: p50 / p95 wait per tool over the last hour
SELECT source_name AS tool_name,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY (data->>'value')::float) AS p50_ms,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY (data->>'value')::float) AS p95_ms,
       COUNT(*) AS wait_count
FROM __agents_usage
WHERE kind = 'custom'
  AND source_type = 'client_tool_wait_ms'
  AND timestamp > (extract(epoch from now()) * 1000 - 3600000)
GROUP BY source_name
ORDER BY p95_ms DESC;

-- D1 (SQLite): average wait per tool over the last hour
SELECT source_name AS tool_name,
       AVG(CAST(json_extract(data, '$.value') AS REAL)) AS avg_ms,
       COUNT(*) AS wait_count
FROM __agents_usage
WHERE kind = 'custom'
  AND source_type = 'client_tool_wait_ms'
  AND timestamp > (strftime('%s', 'now') * 1000 - 3600000)
GROUP BY source_name
ORDER BY avg_ms DESC;

Traces: when tracing-langfuse is wired up, the client-tool wait is captured as a span nested under the tool-execution span. Use the toolName and toolCallId span attributes to correlate with the Logger events above.

Diagnosing a timeout

Symptom: user reports "the agent said my tool timed out" or the LLM reports an error: 'client_tool_timeout' tool result on the transcript.

1. Look up the session's logs for client_tool.timeout. Capture toolCallId and deadlineAt.
2. Cross-reference with client_tool.suspended for the same toolCallId to see sessionId, runId, and the original deadline. The wall-clock wait is deadlineAt - suspended.timestamp.
3. Check whether the client ever submitted:
   • No client_tool.submitted for that toolCallId before the timeout → the client never submitted. Investigate the browser / client-side error state, authentication, or network path.
   • client_tool.submitted arrived after the timeout → submit landed past the deadline. Either raise clientToolTimeoutMs (per-tool or per-agent) or fix the client-side latency.
4. If submit arrived on time but the wait still timed out, look for:
   • client_tool.send_event_failed / client_tool.signal_failed — the submit reached the state store but the waiting runtime never woke up.
   • client_tool.ownership_write_failed — the pending entry was never fully registered; the submit returned unknown_tool_call.

Remediation:

• Per-tool: bump clientToolTimeoutMs on the defineTool config.
• Per-agent fallback: bump AgentConfig.clientToolTimeoutMs.
• Global fallback: DEFAULT_CLIENT_TOOL_TIMEOUT_MS (5 minutes) is intentional; override per-agent rather than globally.

unexpected tool_use_id after a client tool in a later turn

Symptom: the first chat turn works, but a client-executed tool invoked in a later turn fails on resume. The agent enters a terminal failed state and the LLM provider rejects the request with:

messages.N.content.0: unexpected tool_use_id found in tool_result blocks: <id>. Each tool_result block must have a corresponding tool_use block in the previous message.

The client-side side effect (e.g. the document edit) succeeded, but the agent never produced a confirmation and the session looks unrecoverable.

Root cause (fixed): the runtime cleans up orphaned messages on resume() by truncating to the latest checkpoint's messageCount. On stores that ordered checkpoints by stepCount (Postgres, Redis, D1, Durable Objects), getLatestCheckpoint returned the previous turn's checkpoint — because stepCount is not monotonic across turns, the new turn's suspension checkpoint had a lower stepCount. Truncating to that stale boundary deleted the new turn's user message and the suspending assistant tool_use, orphaning the submitted tool_result. The in-memory store was never affected (it already returned the most-recently-written checkpoint).

Resolution: upgrade to a release where getLatestCheckpoint resolves the session's checkpoint pointer (most-recently-written) on every store. No data migration is required — the fix is read-side only. Sessions already suspended at deploy time recover on their next resume once the new code is running.

runtime_restarted in tool results

What it means: the continuation's orphan drain gave a still-pending call with no submission error: 'runtime_restarted', so the LLM can react rather than stall forever. That happens when:

• A real restart: the JS runtime process restarted while a client-tool call was pending, and the in-memory promise map was lost.
• On an older release, a partial batch (CP-82): the run was resumed (on the Cloudflare DO, woken by a submit) while another call of the same step had not been answered yet. A resume now refuses a partial batch (state_not_resumable) and the DO waits for the whole batch, so this cause is gone. See Submit every call of a batch before resuming.

Operator action:

• A burst of runtime_restarted errors immediately after a deploy or process restart is expected for active runtime-js sessions.
• A runtime_restarted result with no restart behind it, on a release before the batch rule, came from a partial batch (CP-82). Upgrade.
• For short-lived agents that tolerate a restart, no action is needed — the LLM sees a failed tool result and typically retries or gives up gracefully.
• For long-running agents that wait on human approval (seconds to hours), move to runtime-temporal (fully durable via workflow history), runtime-dbos (fully durable via Postgres-backed workflow replay), or runtime-cloudflare (durable via DO state + alarms, or Workflow waitForEvent). These runtimes survive process restarts without losing the pending call.

Inspecting the ownership map

Use case: a submit repeatedly returns unknown_tool_call and you need to tell whether the ownership map is empty, stale, or corrupted.

Pending client-tool calls are persisted under the owning session (sessionId the tool executes in — may be a sub-agent). The routing table lives on the root session as clientToolCallOwnership: Record<toolCallId, sessionId>. Consumers always submit against the root sessionId and the framework looks up the owner.

-- Postgres
SELECT session_id,
       client_tool_call_ownership,
       pending_client_tool_calls
FROM __agents_states
WHERE session_id = 'your-root-session-id';

-- D1 (Cloudflare)
SELECT session_id,
       client_tool_call_ownership,
       pending_client_tool_calls
FROM __agents_states
WHERE session_id = 'your-root-session-id';

client_tool_call_ownership is JSON: keys are toolCallIds, values are the owning sessionId (root itself for top-level tool calls, or a sub-agent session). If a toolCallId your client is submitting for isn't in the map:

• Client submitted too early. The register-pending + ownership writes race with the tool_start chunk on the stream. Retry with small backoff.
• The tool call already resolved. Framework returns already_completed (200). Idempotent — the first submit won.
• State was cleared. The session ended, was branched, or the runtime crashed before persisting ownership. Submit will return unknown_tool_call; no action needed unless the client is stuck looping.

For Cloudflare DOs, inspect the DO's SQLite via the wrangler CLI or a diagnostic endpoint — each DO's state is isolated.

Ownership write retry semantics

When a sub-agent dispatches a client-executed tool call, the framework writes a (toolCallId → owningSessionId) entry into the root session'sclientToolCallOwnership map. This write is performed via compare-and-swap on the SessionState's version — concurrent writes from sibling sub-agents are serialized.

The implementation retries up to 3 attempts on transient CAS conflicts (writeOwnership, writeBothPendingAndOwnership, clearOwnership) and up to 5 attempts on clearPending. On exhaustion:

• The exhaustion event is logged at ERROR level. Event names emitted by packages/runtime-js/src/client-tool-resolver.ts:
  • client_tool.ownership_write_failed — terminal failure of writeOwnership or writeBothPendingAndOwnership.
  • client_tool.ownership_clear_failed — terminal failure of clearOwnership.
  • client_tool.clear_failed_exhausted — terminal failure of the pending-entry clear path that runs inside waitForResult's resolve closure (the stranded entry will be retried by the next orphan drain; manual cleanup may be required if it persists).
  • client_tool.ownership_retry / client_tool.clear_pending_retry — debug-level per-attempt logs that precede the terminal error log when retries are happening.
• The exception is rethrown to the caller. Sub-agent dispatch fails with the original error.

This is the fail-loud path — the framework does not silently drop ownership writes. If you observe the error log without a corresponding caller-visible exception, that's a bug; please file an issue.

Operationally: ownership-write exhaustion almost always indicates a distributed-lock contention issue at the state-store layer (e.g., Redis WATCH/MULTI conflicts under high concurrency, Postgres serialization failures, D1 OCC version churn). Diagnose by looking at the state-store's CAS conflict rate.

When a persistent state-store outage breaks idempotency

Setup: the orphan-drain path has three layers of defense against a mid-drain crash:

1. Per-iteration clearPendingEntry (V4) — delete the pending entry immediately after its tool-result message lands.
2. Post-loop bulk-clear (Z4) — sweep any entry that survived (1).
3. Per-toolCallId idempotency guard on append (BB6, round 29) — if an entry persists to the next drain despite (1) and (2), skip the tool-result append + counter-recording when conversation history already carries a matching tool-result.

Symptom: during a sustained state-store outage (pool exhausted, network partition, replica lag), client_tool.orphan_drain_clear_entries_summary and client_tool.orphan_drain_bulk_clear_failed both fire. The next resume's drain sees leftover entries, the BB6 idempotency guard engages (the skippedAppend: true field on client_tool.orphan_resolved confirms it), and the session resumes cleanly — though the operator sees noisier logs than usual.

When it breaks: if all three layers fail simultaneously (rare, but possible if the store recovers between the broken write and the read that finds the old entry), you can end up with:

• the assistant message carries a tool_call with toolCallId=X
• conversation history has a tool_result with toolCallId=X AND a second tool_result with toolCallId=X (possible only if BB6 somehow failed to see the first — a store-read-inconsistency case).

Providers will reject the session at the next LLM call with an error like "invalid tool_call_id X" or "duplicate tool_call_id X".

Manual recovery:

-- Postgres: find sessions with duplicate tool-result messages by toolCallId.
-- Adjust the messages-join per your schema (__agents_messages table).
SELECT session_id, tool_call_id, COUNT(*)
FROM __agents_messages
WHERE role = 'tool'
  AND tool_call_id IS NOT NULL
GROUP BY session_id, tool_call_id
HAVING COUNT(*) > 1
ORDER BY 1, 2;

Then, for each offending session, delete the duplicate rows (keep one) and reset pending_client_tool_calls to {}:

-- Postgres: delete all but the lowest-id duplicate for a given pair.
DELETE FROM __agents_messages m
WHERE role = 'tool'
  AND session_id = 'YOUR_SESSION'
  AND tool_call_id = 'YOUR_TOOLCALL_ID'
  AND message_id > (
    SELECT MIN(message_id) FROM __agents_messages
    WHERE role = 'tool'
      AND session_id = 'YOUR_SESSION'
      AND tool_call_id = 'YOUR_TOOLCALL_ID'
  );

UPDATE __agents_states
SET pending_client_tool_calls = '{}'::jsonb
WHERE session_id = 'YOUR_SESSION';

For Cloudflare DO sessions, use the DO's SQLite directly via wrangler d1 execute or a diagnostic endpoint.

Prevention: the BB6 idempotency guard closes the common failure path. If you see duplicate tool-results in practice AFTER the BB6 fix shipped, it means the state store returned stale reads — escalate to the store's on-call.

Force-failing a stuck call

Use case: a pending client-tool call will never receive a submit (browser closed, device offline, client bug) and the deadline is too far out to wait.

Submit an error directly — the framework treats it the same as a client-reported error:

if (!executor.submitToolResult) {
  throw new Error('Executor does not support client-executed tools');
}
await executor.submitToolResult({
  sessionId: rootSessionId, // ALWAYS the root
  toolCallId: stuckToolCallId,
  error: 'manually_failed_by_operator',
});

The pending call resolves with the provided error string, the LLM sees a normal failed tool result on the next step, and the agent can react or fail gracefully. Error-submissions skip outputSchema validation, so any string works.

Over HTTP (same effect):

POST /submit-tool-result
Content-Type: application/json

{ "sessionId": "root-session-id", "toolCallId": "tc_123", "error": "manually_failed_by_operator" }

Upgrading from v6

For the v6 → v7 client-tool model rewrite (durable suspension, the kind discriminator on SubmitToolResult, the deletion of HelixChatTransport, the new useResumeClientTools hook), see the canonical v6 to v7 migration guide. The notes below cover earlier (pre-v6) breaking changes that still apply.

Upgrading from pre-client-tool versions

If you're upgrading an existing Helix Agents deployment, most of the changes are additive and no migration is required. Two items MAY require code changes:

1. AgentServer fail-closed auth (BREAKING)

AgentServer now refuses to start at construction time if neither an authenticate hook nor allowUnauthenticated: true is configured. This was introduced to prevent an accidentally-unauthenticated /submit-tool-result endpoint — which is an LLM-context data-injection surface.

Before (silently insecure):

new AgentServer({
  agents: { agent1 },
  // no authenticate, no allowUnauthenticated — endpoints were open
});

After — pick one:

// Option A: wire an authenticate hook.
new AgentServer({
  agents: { agent1 },
  authenticate: async ({ request, operation }) => {
    // return { ok: true, principal } or { ok: false, error, status }
  },
});

// Option B: explicit opt-in if you gate upstream (reverse proxy, API gateway).
new AgentServer({
  agents: { agent1 },
  allowUnauthenticated: true, // emits a warn log + console.warn fallback on startup
});

If you're gating upstream (reverse proxy, API gateway, service mesh) and intentionally running AgentServer unauthenticated, pass allowUnauthenticated: true to acknowledge the posture. On startup you'll see a warn log + a console.warn fallback (the latter catches operators running the default noopLogger).

See the Security section above for the three recommended auth patterns.

2. Reserved tool name prefixes (BREAKING in edge cases)

defineTool() now rejects tool names starting with subagent__ or companion__. These prefixes are reserved for framework-internal dispatch (sub-agent routing, companion tool protocol). A user-defined tool with a reserved prefix would be misrouted and fail at dispatch time — the check moves the failure to defineTool time so it's caught in tests.

If your code has defineTool({ name: 'subagent__foo', ... }) or defineTool({ name: 'companion__bar', ... }), rename it. The reserved list is exported as RESERVED_TOOL_NAME_PREFIXES from @helix-agents/core for consumer linter rules.

Framework-internal constructors (createSubAgentTool, createRemoteSubAgentTool, companion-tool factories) bypass defineTool and continue to produce tools with these prefixes as designed.

3. withContext return shape (BEHAVIOR CHANGE — narrow impact)

withContext(logger, bindings) previously returned the native .child() result directly when wrapping a pino/winston/bunyan logger. It now returns a thin adapter that translates the helix (message, data) call signature to pino's (obj, msg) convention so structured data isn't silently dropped by pino's printf path.

Consumer assertions like expect(wrapped).toBe(pinoChild) will fail. Consumer code that inspected pino-specific fields on the wrapper (wrapped.bindings, wrapped.level) no longer finds them — access the underlying pino logger directly for those needs.

Log output through the wrapper is unchanged (modulo the fix — structured data now reaches the final log record correctly).

4. Consumer usageStore factory on Cloudflare DO (BEHAVIOR CHANGE)

On the Cloudflare DO runtime, DurableObjectAgentConfig.usageStore(ctx) is called once per drive (each run entry or wake materializes its agent, adapter and usage store together) and once per usage read that no drive of this instance materialized (ctx.purpose: 'usage_read'). It may be async. Keep expensive resources (a DB pool) outside the factory and return a cheap handle.

The factory is strict: a throw is framework_usage_store_unavailable (retryable), never a fallback to the internal DOUsageStore. There is no negative cache. See the DO drive pipeline upgrade guide.

---

Known Limitations

The framework's client-tool path is designed for correctness first; a few known scaling cliffs exist today. If your workload approaches any of these limits, please file an issue — several are straightforward to lift with additional indexing or pagination, but haven't been exercised yet.

clientToolCallOwnership grows unbounded per root session (R3.S1)

Each pending client-tool call writes an entry into the root session's clientToolCallOwnership map. Entries are removed on submit / timeout / abort via clearOwnership, so the map size tracks concurrent pending calls — not historical call volume. A session that issues thousands of serial client-tool calls stays at O(1) owned entries, but a session that holds thousands simultaneously pending client-tool calls (e.g. a fan- out to many human approvers at once) will grow the map linearly. The map is serialized as JSON in a single row/column, so at ~10K+ simultaneous pending calls the JSON read/write becomes the dominant cost. Practical threshold: under 1K concurrent pending calls per root, well-supported; 10K+ will work but with degraded write latency.

pendingClientToolCalls per owning session (R3.S2)

Same shape as above but on the owning session (the one whose runtime is suspended on the wait). In JS/Temporal/DBOS this is typically the root; in the DO path sub-agent DOs own their own pending entries. Same scaling envelope applies per DO / per session. If you expect deep fan-outs, batch client-tool calls at the agent layer so any single session holds only O(few) pending calls at a time.

runtime-js in-memory promise map (R3.S3)

JsClientToolResolver holds a process-local Map<toolCallId, resolver> tracking pending waits on the current process. This map is bounded by the process's concurrent pending calls across all sessions it owns. At process restart the map is lost and the runtime drains waits with runtime_restarted. For JS-runtime deployments this is the primary scale-out limit: a single process can comfortably hold tens of thousands of pending waits, but if you need multi-process durability, prefer runtime-temporal, runtime-dbos, or runtime-cloudflare.

completed cache is per-resolver-instance, not per-process (K3)

The JsClientToolResolver.completed LRU/TTL cache (default 10k entries / 10 min) is scoped to a single resolver instance, which today means a single JSAgentExecutor. Processes that run multiple executors (per- tenant isolation, sharded worker pools) get a separate cache per executor. The fast-path coherence guarantee — "a submit for session S finds the live waiter armed for S::toolCallId" — applies ONLY when the submitToolResult() call lands on the same executor instance that armed the waiter.

Routing constraint: if you shard JSAgentExecutor instances, route all submits for a given sessionId to the same shard. Cross-shard submits fall through to the state-store slow path, which is correct but slower and can produce unknown_tool_call during the submit- before-register window (the pre-submit-stub mechanism handles sub- agent cases, but not the primary race on root sessions).

Monitor: client_tool.submit_unknown events with reason: 'ownership_missing_toolcall' indicate submits landing on the wrong shard. A steady trickle is expected during normal shard rebalances; persistent volume suggests mis-routing.

Alarm-scheduler deadline recomputation is O(n) per wake (R3.S4)

AlarmScheduler.reschedule() and fireExpired() walk every subscriber (heartbeat, stop-deadline, client-tool-deadline, parent-notice, child-verify, stop-cascade, execution-watchdog) and, in client-tool-deadline's compute (nextClientToolDeadlineAt), every pending client-tool entry of the current run. For a DO with hundreds of simultaneously-pending client-tool calls each alarm wake performs that many durable reads. Practical threshold is unlikely to matter (clients rarely have hundreds of concurrent deadlines), but it is something to watch if you push a single DO toward fan-out at that scale. The fix, when needed, is to maintain a durable secondary index keyed by deadline so compute() can fetch MIN(deadline) in O(1).