Helix Agents

Appearance

Session Model ​

This document is the canonical reference for Helix Agents' session- centric storage model. The framework uses sessionId as the primary key for all state operations.

For framework-level concepts (Run, Agent, Tool, Sub-Agents, etc.) see ./concepts.md. For the actual step-by-step execution loop see ./execution-flow.md.

The framework uses a session-centric storage model where sessionId is the primary key for all state operations.

Breaking for custom stores (pre-1.0)

The store-enforced commit model below changed the SessionStateStore interface: appendMessages and truncateMessages take a required MessageCommit, commitState / renewRunLease / setRunOwnerUnknownSince / abandonRunStart / listRunCheckpoints are new, the run methods are mandatory, and the run-start rules are part of the store contract. Migration detail (schema migrations, deploy notes, legacy sessions) is in the snapshot and checkpoint upgrade guide.

Session vs Run ​

  • Session: A conversation container. Identified by sessionId. Contains all messages, custom state, and checkpoints.

  • Run: A single execution within a session. Every entry into running creates a new run: execute() (a fresh session or a new turn), every resume() mode, retry(), a sub-agent spawn and a persistent-companion continuation. DBOS also starts a new run inside the same workflow when a client-tool submit wakes a suspended run (DBOS.recv). Recovery of an in-flight run is the one exception: it keeps the run (see Recovery keeps the run).

    Every run is created inside a run-start commit (see Runs): the run record, its input rows and an input checkpoint are written in one atomic store operation, so there is no window where the input is visible and the run is not. listRuns() therefore grows by one per entry on all runtimes (JS, Temporal, DBOS, Cloudflare DO, Cloudflare Workflows), including retries and child sessions.

    RunStatus has four live statuses (running, suspended_client_tool, suspended_awaiting_children, suspended_step_partial) and four terminal ones (completed, failed, interrupted, superseded). Terminal statuses are sticky: updateRunStatus refuses to change a terminal run (see The superseded-run fence). The three suspension variants mirror RunOutcome.kind and AgentResult.status. See ./concepts.md for the full HITL model.

    On Temporal, DBOS and Cloudflare Workflows, every resume() / retry() entry runs in its own workflow / instance, whose id derives from a key minted by that call (Temporal …__resume-<runId> / …__retry__<runId>, DBOS …__resume__<runId> for both, CF Workflows buildWorkflowInstanceId with a per-call key), never from a counter, so a losing caller writes nothing. SessionState.resumeCount counts those entries: the run-start winner bumps it once in its entry state (Temporal, DBOS, CF Workflows; runtime-js and the CF Durable Object never count). Temporal still names a re-spawned child's workflow …__resume-<n> from the child's own resumeCount + 1, and a child re-spawn is not counted. See ./concepts.md §Client-Executed Tools for per-runtime resume mechanics.

    Continuing a completed structured-output session works. stepCount is per turn (each turn restarts its own counter), so a continued turn resets stepCount → 0 and gets a fresh per-turn maxSteps budget. A structured-output agent that completed via __finish__ used to leave a dangling tool_use that broke the next LLM call on continuation; the __finish__ history invariant + heal makes that transcript valid, so a follow-up execute(sessionId) continues cleanly (preserving memory, returning a fresh typed result). The same per-turn stepCount → 0 reset applies to persistent-companion re-consult (continuing a completed child) — uniform across all runtimes (JS, Temporal, DBOS, Cloudflare DO, Cloudflare Workflows). See ./concepts.md §Persistent-companion continuation + the __finish__ heal.

Key Benefits ​

  1. Efficient Message Storage: Messages are stored once per session, not duplicated per run. This is O(n) storage vs O(n²) for run-centric models.
  2. Natural Conversation Continuity: Reusing the same sessionId automatically continues the conversation with full history.
  3. Clean Sub-Agent Isolation: Each sub-agent gets its own sessionId, preventing state conflicts.

Usage ​

typescript
// Start a new session (sessionId is required)
const sessionId = `session-${Date.now()}`;
const handle = await executor.execute(agent, { message: 'Hello' }, { sessionId });

// Continue the same session (pass the same sessionId)
const handle2 = await executor.execute(
  agent,
  { message: 'Follow up' },
  {
    sessionId: handle.sessionId,
  }
);

// Branch from a checkpoint (creates a new session from existing state)
const newSessionId = `session-${Date.now()}`;
const handle3 = await executor.execute(
  agent,
  { message: 'What if...' },
  {
    sessionId: newSessionId,
    branch: { fromSessionId: handle.sessionId, checkpointId: 'cp_123' },
  }
);

v7 SessionState Shape ​

The full SessionState<TState, TOutput> interface lives at packages/core/src/types/session.ts:86-321. The canonical shape is:

typescript
interface SessionState<TState, TOutput> {
  // Identity
  sessionId: string;
  agentType: string;
  streamId?: string;

  // Custom application state + status
  customState: TState;
  status: SessionStatus; // 'active' | 'completed' | 'failed' | 'interrupted' | 'paused'
  stepCount: number;
  output?: TOutput;
  error?: string;
  errorDetail?: ErrorDetail; // structured terminal-failure detail

  // v7: γ-cascade discriminator. Currently 'parent_suspended' marks a
  // child that was failed because its parent suspended; the cascade in
  // applyResultsAndReload re-spawns these on parent resume.
  failureReason?: string;

  // Interrupt context (set when status === 'interrupted' or 'paused')
  interruptContext?: InterruptContext;

  // v7 HITL suspension state
  pendingClientToolCalls?: Record<string, PendingClientToolCall>;
  suspendedAwaitingChildren?: Record<string, SuspendedChildWait>;
  suspendedStepId?: string;
  completedClientToolCalls?: Record<string, number>; // root-only
  clientToolCallOwnership?: ClientToolCallOwnership; // root-only

  // Forced-completion phase (durable, replay-safe)
  forcedCompletion?: ForcedCompletionState;

  // v7 tracing continuity (sessionId-seeded)
  tracingContext?: { traceId: string; rootSpanId: string };

  // v7 session GC + cross-session links
  expiresAt?: number;
  parentSessionId?: string;
  rootSessionId?: string;

  // v7 DBOS write-once mode binding
  mode?: 'standard' | 'persistent';

  // v7 distributed coordination
  version: number; // monotonic; incremented on every modification
  resumeCount: number; // resume / retry entries, counted by each entry's run-start winner

  // Checkpoint tracking (written by commits only; saveState preserves them)
  checkpointId?: string; // the latest-checkpoint pointer
  checkpointedAt?: number;
  latestTruncateCheckpointId?: string | null; // the latest `truncate` commit

  // User context
  userId?: string;
  tags?: string[];
  metadata?: Record<string, string>;

  // v7 persisted workspace ref (so it survives interrupt/resume)
  workspaceRef?: WorkspaceRef;

  // Timestamps
  createdAt: number;
  updatedAt: number;
}

The session's staging-checkpoint marker was removed: saveState no longer writes checkpoints, so every checkpoint the pointer names comes from a commit and carries its own kind.

v7-NEW field summary ​

FieldPurpose
failureReasonγ-cascade discriminator (e.g. 'parent_suspended'); used by applyResultsAndReload to decide re-spawn vs. drain.
pendingClientToolCallsMap of toolCallId → pending entry; canonical signal for "awaiting client submission".
suspendedAwaitingChildrenMap of parentToolCallId → child wait info; populated when parent paused awaiting sub-agents.
suspendedStepIdMid-step suspension marker for mixed server+client tool batches.
completedClientToolCallsRoot-only timestamp map; makes 'already_completed' durable across runtime restarts.
clientToolCallOwnershipRoot-only toolCallId → owningSessionId; routes submissions to the owning sub-agent.
tracingContextsessionId-seeded traceId + rootSpanId; one trace per session across runs.
expiresAtOperator GC hint for abandoned sessions.
modeWrite-once 'standard' / 'persistent' binding (DBOS-enforced).
version / resumeCountOptimistic concurrency + the number of resume / retry entries (counted by the winner).
latestTruncateCheckpointIdThe id of the latest truncate commit; part of the snapshot reader's fence.
workspaceRefPersisted workspace ref (so it survives interrupt/resume cycles and DO evictions): every durable store (RM-19).
parentSessionId / rootSessionIdSub-agent cross-session linkage; rootSessionId enables O(1) ownership writes.

Commits and checkpoints ​

A commit is a store operation that mutates the message log or the snapshot-visible state. It atomically writes a checkpoint { id, kind, runId, messageCount, streamSequence, stepCount, state, keepCount } and moves the session's latest-checkpoint pointer (SessionState.checkpointId) to it. The rows and the checkpoint land together or not at all (commit.all-or-nothing). The store enforces this: a runtime cannot append or truncate without writing the checkpoint.

Commit methodCheckpoint.kindWritten by
appendMessages(s, rows, { …, startRun }) (rows may be empty)inputthe run-start commit
appendMessages(s, rows, commit) without startRunappendan injection, a client-tool / approval result, a continuation heal, a phase-2 row
saveStateAndPromoteStaging / promoteStagingstepa step commit (tool step, suspension, text-only terminal step)
commitState(s, state, commit, opts)stateK5, the terminal state commit; also entry-state writes
truncateMessages(s, keepCount, commit), or startRun.truncateTotruncatea rewind (from_checkpoint, retry); sets latestTruncateCheckpointId
cloneSessionclonea branch (runId: null, streamSequence: -1)

Checkpoint.kind is CommitKind | null, runId is string | null and keepCount is number | null (set only on a truncate checkpoint). null marks a pre-upgrade row or a createCheckpoint row. parseStoredCommitKind reads the stored column and throws UnknownCommitKindError for a value it does not know, rather than treating it as legacy.

Two store methods are not commits:

  • saveState never anchors (commit.save-state-never-anchors). It writes no checkpoint and never moves the pointer, and it preserves the stored checkpointId / checkpointedAt / latestTruncateCheckpointId whatever the state it is given carries. Snapshot-visible state changes go through commitState.
  • createCheckpoint never moves the pointer. It writes a checkpoint row with kind, runId and keepCount all null, and getLatestCheckpoint keeps naming the last commit's checkpoint.

MessageCommit and the commit sequence ​

typescript
interface MessageCommit {
  streamSequence: number; // the LIVE stream head at the commit
  runId: string; // the run the commit is attributed to, and fenced on
  startRun?: StartRun; // present iff this append starts a run
}

MessageCommit has no kind: the method (and whether startRun is present) decides it. promoteStaging takes the same { streamSequence, runId } in its options, and saveStateAndPromoteStaging in its checkpointMeta.

Runtimes resolve streamSequence with core resolveCommitSequence(streamManager, streamId):

  • no stream manager, or no stream yet → NO_STREAM_SEQUENCE (-1), never 0 (checkpoint.no-stream-sentinel);
  • otherwise the stream's live latestSequence;
  • a head read that throws → HelixError state_checkpoint_sequence_unavailable (retryable). The runtime's native retries apply, then the run fails with that code. There is no silent 0 fallback any more (DI-21).

A run-start commit uses resolveRunStartSequence, which records 0 for a stream the winner has not created yet. The sequence is read after every chunk that describes the committed rows was written, so a checkpoint's cursor covers its own content: every committed content chunk with seq ≤ cp.streamSequence is represented in [0, cp.messageCount) (P1), and no row in that prefix belongs to a chunk above the cursor (P2). This is what lets the snapshot be taken from one checkpoint (see ./stream-protocol.md §Cursor semantics).

The checkpoint-truth rules (K1–K5) ​

  • K1 — every mutation commits. appendMessages, truncateMessages, promoteStaging / saveStateAndPromoteStaging, commitState and cloneSession each write a checkpoint of their kind with the live row count and move the pointer.
  • K2 — live sequence. Every commit records the stream's live head, read after the commit's content chunks were written (above).
  • K3 — saveState never anchors. No store mints a checkpoint in saveState any more. This fixes the sequence-0 checkpoints that saveState used to write on memory, Postgres, Redis and the DO (RM-37), and the Redis saveState block that pointed the session at a checkpoint id that was never written (RM-38). The findings records are closed by the conformance owner.
  • K4 — every boundary commits. Content is committed, with a checkpoint, at every step boundary: a tool step, a text-only terminal step, a __finish__ / finishWith step (deferred included), the forced-completion tails, a client-tool / approval / sub-agent suspension, and an interrupt. Inputs commit when they enter the log: a run's user message (inside its run-start commit, RM-42), injections and client-tool results (append). DBOS commits a tool step's results in the step's own promote, with the live sequence (RM-39, RM-40), and every Cloudflare Workflows message-log mutation is a commit (RM-41).
  • K5 — the terminal state commit. Every terminal exit (completion, failure and stop / interrupt) ends with one commitState (kind: 'state') that carries the run's terminal status, its output / error / errorDetail, and the terminal hooks' state writes (user ruling D1). It is written after the agent's terminal hooks and before the terminal run status and the stream end. A K5 that still fails after the runtime's bounded retries ends the run failed with the commit's error code (commit.terminal-state-failure), recorded through a fenced saveState. The Cloudflare DO host's onRunSettled hook runs from the settle path, after endStream, and does not write session state.

The ordering rules that go with them:

  • O1 — commit before terminal. The commit that covers a run's final content lands before the stream is ended, failed or paused (ordering.content-before-ended / -failed / -paused).
  • Terminal order: K5 → terminal run status → stream end / fail (ordering.state-before-run-status, ordering.run-status-before-end), on every runtime and every terminal exit.
  • O2 — run-start order: run-start commit → stream claim → stream live → run_started → stream_resync → content (ordering.run-started-after-commit; it replaced ordering.stream-live-before-input). See ./execution-flow.md §Run-start order.

getLatestCheckpoint is the pointer ​

getLatestCheckpoint(sessionId) returns the checkpoint the session's checkpointId pointer names: the checkpoint of the most recent commit. It never orders by stepCount and never falls back to a highest-stepCount row; a session with no pointer has no latest checkpoint (null). stepCount is per turn and not monotonic across runs, so an ordering by it returned an earlier turn's checkpoint and made resume / retry truncate away the current turn's committed messages.

Because only commits move the pointer, the latest checkpoint is always a truthful anchor: its messageCount is the live row count at the commit and its streamSequence the live head. The cross-store suites in @helix-agents/core/testing enforce it (commit-checkpoint-operations, checkpoint-operations).

Message counts must be truthful ​

Three different numbers describe a session's message log, and all of them must equal the live row count at the moment they are read:

  • getMessageCount(sessionId) — a live count on every in-tree store (COUNT(*) on Postgres / D1 / DO-SQLite, LLEN on Redis, array length in memory). loadLLMHistory compares every history read against it, so it must never come from a denormalized column.
  • Checkpoint.messageCount — the log length a checkpoint was taken at. A rewind, retry or branch truncates or copies to exactly this many messages, so a wrong value deletes or duplicates real messages.
  • listSessions().sessions[].messageCount — the denormalized per-session column (states.message_count on Postgres / D1, the messageCount hash field on Redis).

truncateMessages must update the denormalized count in the same transaction (Postgres), batch (D1) or Lua script (Redis) as the delete (RM-16, RM-22, RM-30). Postgres also feeds that column into the messageCount of later checkpoints written by saveStateAndPromoteStaging, so a stale value there inflated every checkpoint after a rewind. The Durable Object store's saveState used to mint a checkpoint with message_count left at its column default of 0, so every DO turn's latest checkpoint claimed an empty log (RM-36); saveState no longer mints checkpoints at all.

The cross-store contract tests enforce this: core/src/testing/message-count-operations.ts (truncate-then-checkpoint-count, truncate-then-list-count) and the commit.*-checkpoints cases of core/src/testing/commit-checkpoint-operations.ts (each commit records the live count).

Every terminal exit leaves a truthful latest checkpoint ​

Every terminal exit writes K5 before the session becomes observably terminal (terminal run status, closed stream). This covers completion (text-only, structured-output __finish__ / finishWith, forced completion), the framework-forced tails (maxSteps), and failure and stop / interrupt (D1). So after any run ends, the latest checkpoint is that run's state commit, whose messageCount is the live row count.

Two consequences:

  • retry() never targets K5. It targets the failed run's last step boundary or its run-start commit (see Retry target).
  • resume({ mode: 'from_checkpoint' }) / branch with an explicit id act on the checkpoint given.

The exception-exit rule applies where the runtime classifies the finish as a failure: Temporal and DBOS currently treat a content_filter / refusal / max_tokens / error / unknown finish of an agent without outputSchema as completed, not failed (DI-26).

Runs ​

Run records ​

RunMetadata (packages/core/src/types/session.ts) is written by the run-start commit and updated by updateRunStatus / renewRunLease / setRunOwnerUnknownSince:

FieldMeaning
runId, turnThe run's id and its turn number (1, 2, 3, …).
statusA live or terminal RunStatus (sticky once terminal).
startSequenceThe run-start commit's streamSequence: the stream head at the commit. The run's own run_started is above it (run.start-sequence).
startMessageCountThe input checkpoint's messageCount, input rows included (run.start-message-count).
startUIMessageCountThe first-row UI index of the UI message that will hold the run's response: the open assistant group's first row on a continuation, else the next row (run.start-ui-index). The snapshot's resumeMessageId derives from it.
entryStepCountThe step count the run entered with (StartRun.entryStepCount). retry() restores a first-step failure to exactly this count (a max_steps cut-off restarts at 0 instead, rule 3).
inputRowCountHow many rows the run-start commit appended. The run's own input rows are [startMessageCount - inputRowCount, startMessageCount).
startDigestHash of the commit's rows and startRun (the idempotency key).
ownerTokenThe ONE execution that started the run (absent on the DO).
leaseUntilJS only: the lease expiry on the store clock; null once released.
ownerUnknownSinceCFW only: when an entry first saw the run's instance in status unknown.
parentA child run: { sessionId, toolCallId } of the parent consult it serves.
requestKeyD-2: the request key a keyed entry recorded (StartRun.requestKey). At most one run per session holds a key; getRunByRequestKey finds it.
requestDigestD-2: computeRequestDigest of that entry's own input. A repeat of the key with another digest is a different request (state_request_key_conflict).

The run methods (createRun, updateRunStatus, getCurrentRun, listRuns, getRun) are mandatory. A runtime built on a store that lacks one is rejected at construction (runtime.rejects-store-without-runs), never silently skipped. Once a run-start commit returned, getRun(runId) must return the run to every caller, even where getCurrentRun may lag on a replicated store: the stream claim and failCommittedRun decide on the run's own record.

The run-start commit ​

A run-start commit is appendMessages(sessionId, rows, { streamSequence, runId, startRun }). rows may be empty (a resume with no new input). StartRun is:

typescript
interface StartRun {
  turn: number;
  startUIMessageCount: number;
  expectCurrentRunId: string | null; // CAS: the run the caller saw as current
  expectCurrentRunStatus: RunStatus | null; // CAS: its status (null iff the id is null)
  entryStatuses: readonly SessionStatus[]; // session gate: the status(es) the plan read; non-empty
  ownerToken: string | null; // null on the DO
  ownerless: { kind: 'lease'; ttlMs: number } | { kind: 'asserted' };
  truncateTo?: number; // atomic rewind (from_checkpoint, retry)
  finishCurrent?: { runId: string; status: 'completed' | 'failed' | 'interrupted' };
  pendingRestamp?: { adopt: readonly string[]; drop: readonly string[] };
  entryStepCount?: number;
  entryState?: { forcedCompletion: ForcedCompletionState | null; clearOutput?: boolean }; // DI-40
  parent?: { sessionId: string; toolCallId: string };
  resumedCompanionRef?: { parentSessionId: string }; // CP-103 A1: a resumed companion's parent ref
  parentRunFence?: { parentSessionId: string; parentRunId: string }; // CP-161: a companion spawn / respawn child's parent run
  requestKey?: string; // D-2: the entry's request key, recorded on the run
  requestDigest?: string; // D-2: the digest of the entry's own input
}

The input checkpoint records the state the run entered with (DI-40). entryState is written onto the session row (its version kept) before the store cuts the input checkpoint from that row, in the same atomic step (core applyRunEntryState):

  • a new turn passes { forcedCompletion: null, clearOutput: true };
  • a retry() passes the phase planRetry restored;
  • a from_checkpoint resume passes the rewind target's phase (and no output).

A continue resume passes nothing: it enters with the stored row's values, so the row IS its entry. A rewind or retry to an input checkpoint therefore restores the run's real entry, never the previous turn's forced phase or output. Wired on runtime-js, the CF DO and CF Workflows; Temporal and DBOS pass nothing yet (FU-RETRY-RESTORES-PREVIOUS-TURN-PHASE).

Inside one transaction, on the store's own clock, the store applies core applyRunStartRules (packages/core/src/store/run-start.ts) and then writes:

  1. Validation. truncateTo must be a non-negative integer; entryStatuses must be a non-empty list of session statuses; expectCurrentRunStatus must be null exactly when expectCurrentRunId is (validation_error otherwise; assertValidEntryGate).
  2. Abandoned entry. A run id recorded by abandonRunStart (and never committed) throws RunStartRejectedError('entry_abandoned') and writes nothing, so a retried call never delivers its input twice.
  3. Idempotency. A run with this runId already exists: an equal startDigest(rows, startRun) writes nothing and returns that run (run.start-idempotent); a different digest throws RunStartRejectedError('idempotency_mismatch'). The digest covers the rows and startRun (with truncateTo), but not pendingRestamp, entryStepCount, entryState, parent, entryStatuses, expectCurrentRunStatus or resumedCompanionRef, so a replay from an older worker stays a no-op (and the session is legitimately active by the time a replay arrives). requestKey / requestDigest are part of it. Request key (D-2). With startRun.requestKey, the store reads the run of this session that holds the key inside the same atomic step and passes it to applyRunStartRules as requestKeyHolder. Another run holding it throws RunStartRejectedError('request_key_taken', { existingRunId }) and writes nothing. This race guard sits after the same-runId replay (a run holds its own key) and before the CAS, so two retries of one request that both found the key free cannot both commit, and the loser always learns the holder rather than a stale-plan refusal. The keyed entry then looks the key up again and re-plans (core planKeyedEntry: it attaches). The lookup and the plan run in the runtime BEFORE any concurrency policy (see Execution Flow §Run-start order).
  4. CAS. The current run is not expectCurrentRunId: a live current run throws RunStartConflictError (expected_mismatch), an idle one RunStartRejectedError('expected_mismatch_idle'). Run-status CAS (CP-96). The current run's status is not expectCurrentRunStatus: RunStartRejectedError('expected_run_status_changed'). Terminal statuses are sticky, so the run the entry saw was live and has moved on.
  5. Owner guard. A live current run may be superseded only by the same ownerToken (the owner continuing its own suspension), a lapsed or absent lease (ownerless.kind === 'lease': leaseUntil absent or < storeNow), or an asserted ownerless predicate (ownerless.kind === 'asserted', proved by the runtime first). Otherwise RunStartConflictError('run_active'). The guard does not apply to a current run whose finishCurrent the store verified (step 5, CP-84 §3.15): its K5 landed, so it is ended in this commit even with an unexpired lease. Session gate (CP-96). The session's status, read inside the same atomic step, is not in entryStatuses: RunStartRejectedError('entry_status_changed'). It follows the owner guard, so a live winner reports run_active.
  6. Supersede. Every other live run becomes superseded, lease cleared. A current run named by finishCurrent is instead ended with that terminal status, but only when the store's own reads confirm its K5 landed (the session is terminal with that status and the latest checkpoint is that run's state commit); otherwise it is superseded.
  7. Rewind. truncateTo below the live count deletes the rows from truncateTo on and writes a truncate checkpoint attributed to the NEW run (keepCount = truncateTo), setting latestTruncateCheckpointId (commit.start-truncate-atomic). A separate truncate could not be fenced by the old, terminal run, and a truncate after the run-start would leave a window with a stale suffix.
  8. Pending re-stamp. pendingRestamp (computed with core planPendingRestamp) drops pending client-tool calls whose tool_use the rewind removed and re-stamps the others to the new run, in the same transaction, so no submit sees a call still stamped with the superseded run. Companion ref (CP-103 A1). resumedCompanionRef: { parentSessionId } (a run that resumes an interrupted persistent companion) turns the parent's sub-session ref for this session status: 'running', completionDelivered: false in the same transaction. A missing parent or ref throws RunStartRejectedError('resumed_companion_ref_missing') and writes nothing; a ref that is terminated (a terminate landed first, WA MR2 Ruling 43) throws RunStartRejectedError('resumed_companion_ref_terminated') and writes nothing (contract scenario commit.start-companion-ref-terminated). A store that cannot hold both in one atomic step refuses the field typed (DO-SQLite: UnsupportedOperationError); it never writes the run without the ref. A matched replay (step 2) applies nothing. Contract scenario commit.start-companion-ref-atomic. Parent-run fence (CP-161). parentRunFence: { parentSessionId, parentRunId } (a first-spawn or respawn companion child on CF Workflows, whose instance the parent creates after its companion step) requires parentRunId to be the parent session's current live run, read in the same atomic step; else it throws RunStartRejectedError('parent_run_superseded') and writes nothing, so a child created for a parent run superseded since never runs. Not part of startDigest. DO-SQLite refuses the field (UnsupportedOperationError): the parent's runs live in another DO. Contract scenario commit.start-parent-run-fence.
  9. Create. The run (status: 'running', startSequence = cp.streamSequence, startMessageCount = cp.messageCount, startUIMessageCount, startDigest, ownerToken, entryStepCount, inputRowCount, parent, requestKey, requestDigest, and on JS leaseUntil = storeNow + ttlMs), its rows and the input checkpoint. The run becomes the session's current run.

A rejected run-start writes nothing (commit.all-or-nothing). The errors:

ErrornamecodeRaised for
RunStartConflictErrorAgentAlreadyRunningErrorstate_run_start_conflicta live conflict: run_active, or expected_mismatch while the current run is live
RunStartRejectedErrorRunStartRejectedErrorstate_run_start_conflictidempotency_mismatch, expected_mismatch_idle, expected_run_status_changed, entry_abandoned, entry_status_changed, resumed_companion_ref_missing, resumed_companion_ref_terminated, expected_run_stopped, consumed_pending_changed, parent_run_superseded, request_key_taken (details.existingRunId names the holder)

RunStartConflictError keeps the AgentAlreadyRunningError name, so the existing "the winner is driving" fallbacks match it. entry_status_changed comes from the store's session gate on every store (the session left the statuses the entry planned from, for example an /abort landed in between). An entry that gets expected_mismatch_idle or expected_run_status_changed retries once only when a fresh read shows the run it saw (id and status) is still current (its own read lagged, shouldRetryRunStart); otherwise a newer run started or the seen run moved on, and the entry throws the typed loser error (asRunStartLoserError → RunStartConflictError). Both new fields and the rule order are explained in Execution Flow §Entry gates. A custom store passes the status it read atomically as applyRunStartRules({ sessionStatus }) (upgrade guide).

The store suites are runStartOperationTests (run-start-operations) and fenceOperationTests (fence-operations) in @helix-agents/core/testing. A store that implements the methods without these rules passes the type check but breaks run.single-live.

Run ids, owner tokens and the ownerless predicate ​

RuntimeWho commits the run-startrunIdownerTokenOwnerless (asserted) when
JSthe executor (commitRunStart)minted once per entry call, never re-minted on a retry of that calla random UUID per entryn/a: lease mode, evaluated by the store
CF DODOAB (commitRunStartSync → DOStateStore.runStartCommitSync), in one transactionSyncminted by DOAB per request and handed to the executornone (null)the DO is not executing that run, checked inside the same synchronous transaction
Temporalan activity of the run's workflowexecutor entries: minted by the executor, carried in the workflow input; child runs: wf.uuid4() in workflow code${workflowId}:${workflowRunId}the current run's workflow is closed (describeWorkflow); a running workflow whose workers are all down blocks, conservatively
DBOSa step of the run's workflowexecutor entries: minted and passed in the workflow input; in-workflow continuations: <suspended run>:continuation (dbosContinuationRunId); companion runs: ${childSessionId}-continue-${toolCallId}the DBOS workflowIDthe workflow status is terminal (SUCCESS, ERROR, CANCELLED)
CFWthe instance's run-start step.do${instanceId}:${agentType}-run-start:0 (the executor passes it as params.runId)the exact instance idthe instance status is complete, errored or terminated; or the unknown escape hatch below

On the DO, the in-memory claim (_executionState) is set only after transactionSync returns, in the same tick, so a commit that throws (and rolls back) never leaves a phantom claim (run.do-claim-rollback).

CFW unknown escape hatch. The first entry that sees the current run's instance in status unknown records ownerUnknownSince on the run (setRunOwnerUnknownSince, set-if-null; a later live observation clears it). Once it has persisted for more than cfwUnknownTerminateAfterMs (default 600 000 ms, on the bindings), a later entry terminates the instance, which makes it ownerless, and writes a run_terminated { runId, reason: 'owner_unknown_timeout' } chunk for the killed run before its own run_started. An operator-paused instance needs the admin path executor.terminateRun(sessionId).

Pre-upgrade running rows become supersedable through the same rules: no lease on JS, a closed workflow on Temporal, not executing on the DO.

The JS run lease ​

A JS run starts with ownerless: { kind: 'lease', ttlMs }, so the store sets leaseUntil = storeNow + ttlMs.

  • The TTL is the executor option runLease: { ttlMs } (JSRunLeaseOptions), default JS_RUN_LEASE_MS (runtime-js's export of core DEFAULT_RUN_LEASE_TTL_MS, 30 000 ms).
  • Core RunLease renews it every ttl / 3 on a timer, so renewal continues through long model and tool calls. renewRunLease(sessionId, runId, ttlMs) is a CAS on the store's own clock: it succeeds only for the current running run, and otherwise throws RunSupersededError.
  • A rejected renewal, or a transient failure that would let the lease lapse, stops the executor at once as superseded.
  • Every chunk the run writes is gated on the lease (gateWriterOnLease): while the local deadline holds, with no I/O; past it, the gate first runs a renewal (RunLease.confirmHeld), so a run nobody superseded renews and writes, and a superseded one stops without writing. This covers the takeover window before the new run's stream claim lands. Only a renewal the store rejects with RunSupersededError stops the run; a transient or hung renewal is retried.
  • The executor stops the timer just before its suspension or terminal status write; the store releases the lease (leaseUntil = null) in that updateRunStatus.
  • In-process child runs (sub-agents, companions) hold their own lease from their own run-start commit. On the DO, children have no lease (the DO claim decides).
  • A JS event loop blocked for longer than about ttl − ttl/3 (20 s by default), for example by a synchronous CPU-bound tool, may let a concurrent entry supersede the run (the store sees the lease expired). The old run then stops, typed, as superseded, never failed. With no concurrent entry nothing is lost: the run renews and continues.

The superseded-run fence ​

Every write a run's executor makes is fenced on the run:

  • Commits. Every runId-carrying commit (append, start, step, state, truncate) is a CAS on "runId is the current run and live" (core assertRunIsCurrentLive).
  • saveState. Executors pass { fenceRunId }, and the store applies the same check in the same atomic step as the version CAS. { fenceIdle: { runId } } is the idle fence: the write lands only while the current run is still runId (null = none) and is not live (core failSessionWithoutRun and DBOS use it to record a pre-commit entry failure on a session with no live run). An unfenced saveState is for non-executor callers.
  • updateRunStatus. Only a live run may change status; a terminal run, superseded included, never transitions again (run.superseded-status-sticky, core decideRunStatusTransition). A write of the status a terminal run already has is an idempotent no-op that writes nothing (CP-84 §3.15: the tail of a run an entry already ended with finishCurrent). Any status other than running clears leaseUntil. A write to a live status (running, suspended_*) never sets completedAt, even when the caller passes one; a terminal write sets it (the caller's value, else now). endSequence is stored only as the caller passes it, never derived (contract run.live-status-no-completedAt).
  • Chunks. Every chunk the executor writes carries its runId; stream managers drop an own-session chunk stamped with another run once a newer run claimed the stream (see ./stream-protocol.md §Chunk fence).
  • Side effects. Core assertRunOwnsSession runs before every tool execution, sub-agent spawn and client-tool / approval registration on every runtime, so a superseded run never executes a tool.

A violation throws RunSupersededError (code: 'state_run_superseded', non-retryable) and writes nothing. The run reports AgentResult.status === 'superseded'; see ./concepts.md §Superseded runs.

Retry target (planRetry) ​

retry() restores a failed session from a target chosen by core planRetry (packages/core/src/store/retry-plan.ts, user ruling D1). It never targets K5:

  1. The failed run's last step boundary: its latest step commit, or a later append commit of that run that carries no assistant row (a phase-2 finishWith result). An append that carries an assistant row holds a failed step's rows, so neither it nor anything after it is a target. A step commit with no assistant row of the run (the checkpoint of a step whose LLM call failed) does not count.

  2. Otherwise, the run's run-start commit (it failed on its first step). The rows rewind to just before the run's own triggering user message (searched only within its input rows, inputRowCount); the retried run re-appends it, followed by options.message when given. The restored step count is the run's entryStepCount.

  3. A maxSteps cut-off continues (C5). When the failed run's record says completionReason: 'max_steps' and the caller gave no checkpointId, rule 1's target is kept (its log, customState and forcedCompletion), but stepCount restarts at 0 (never the run's entryStepCount: a resume enters with its suspension's count), so the retried run gets the full maxSteps budget. The retried input is empty, since the kept log ends in the last step's tool results; options.message is appended when given. A cut-off run that committed no step (a resume that entered at the cap: it drained its submitted client-tool results, then ran zero steps) takes rule 2's target instead, also with step count 0 and no message needed: a run without its own trigger keeps its whole entry, drained results included (they pair the earlier tool_use, RM-29), and continues from it; a run with its own trigger re-appends it as rule 2 says. The reason comes from the run record, not the session's errorDetail, so a runtime gets C5 once it fails a cut-off run and writes the reason. Temporal writes none; DBOS writes its own reasons on a completed run (DI-27, including 'max_steps' for a cut-off), and retry() refuses a completed run, so C5 does not apply there until CF-D1 MR 2 / MR 3.

customState, stepCount and forcedCompletion come from the target, never from K5. An explicit checkpointId is restored as given (resolved first with resolveCheckpoint). A session that failed BEFORE its last entry's run-start commit landed (its current run is an earlier, completed or interrupted run) keeps its whole log and needs options.message. A session with no checkpoint attributed to the failed run (legacy rows) falls back to the latest checkpoint; one with no checkpoint at all restarts fresh.

The retried run's input is options.message when given; otherwise the triggering user message found after the kept prefix. When there is neither (for example a step-boundary target after committed steps, with no new message), planRetry throws validation_error ("No user message found to retry with"). So a first-step failure retries without a message (it re-runs the trigger, CP-58), a maxSteps cut-off retries without one (rule 3), and any other later-step failure needs options.message.

Every runtime runs the retry as a run-start commit with truncateTo (DBOS included: it honours checkpointId and message, CP-30). See ./execution-flow.md §Retry.

Legacy sessions ​

A session whose latest checkpoint predates the commit model (kind: null), or has no checkpoint at all, is in legacy mode: the snapshot serves the complete live log with the stream head as its cursor, whatever the status, because a legacy checkpoint can claim too few rows. Running legacy sessions are excluded by draining before the upgrade; paused ones are covered. The session's first new commit (its next run-start commit) re-anchors it. On Redis, clean the session-status index once (delete the idx:sessions:status:* sets, then run rebuildIndexes(), which only adds entries). See the upgrade guide.

State Store Interface ​

All state stores implement SessionStateStore (defined at packages/core/src/store/state-store.ts). isSessionStateStore requires SESSION_STATE_STORE_REQUIRED_METHODS, which include the run methods, renewRunLease and commitState.

Lifecycle ​

  • createSession(sessionId, options) — Atomically create a session (throws if already exists). All implementations guarantee exactly-one-wins semantics for concurrent calls with the same sessionId, and a concurrent loadState never observes a partially-created session (RM-50).
  • sessionExists(sessionId) / deleteSession(sessionId) — standard lifecycle helpers. deleteSession cascades to every descendant (local refs ∪ parentSessionId, minus remote-ref children; core sessionTreeChildren / collectSessionTree) and is undefined with a live run; the DO-SQLite store refuses with framework_not_supported. See Deleting a session.
  • cloneSession(from, to, options?) — a clone commit: the target's rows, state and checkpoint are written atomically (a failed clone leaves no target, RM-27). fromCheckpointId is validated through resolveCheckpoint first.
    • Where the clone's execution state comes from (RM-72). Never from the source's live status, output, error, errorDetail, interruptContext or failureReason, nor its live forcedCompletion. A checkpoint clone follows the rewind rule (cloneStateFromCheckpoint): customState, stepCount and forcedCompletion from the checkpoint (absent means no phase), status the checkpoint's (active when it records none; a recorded value that is no status rejects the clone with InvalidStoredStateError naming the checkpoint, and no target is written), interruptContext and failureReason cleared. At a terminal (completed / failed) checkpoint the clone also takes its output, error and errorDetail, so it is a coherent copy of that moment; at any other checkpoint they are cleared (a rewind always clears them). A message-index or whole-log clone (cloneStateWithoutCheckpoint) keeps the source's live customState and stepCount, starts active and clears the rest; its forcedCompletion is the latest committed checkpoint's for a whole-log clone (never the live row; user ruling 2026-10-08) and none for a message-index clone (pending FU-CLONE-INDEX-BOUNDARY-FORCED-PHASE). Temporal / CF Workflows branches (loadBranchSource) carry no forced phase (FU-BRANCH-SOURCE-FORCED-PHASE-PARITY). The clone checkpoint carries the same state, on every store.
    • Per runtime. runtime-js and DBOS branch through cloneSession(fromCheckpointId | fromMessageIndex). Temporal and CF Workflows branch through loadBranchSource (a fresh session holding the prefix and the checkpoint's customState) and never call it. The CF Durable Object runs runtime-js, but its DOStateStore refuses cloneSession (UnsupportedOperationError), so DO branching is unsupported and unaffected.

Messages ​

  • appendMessages(sessionId, messages, commit) — an append commit, or the run-start commit with commit.startRun.
  • truncateMessages(sessionId, keepCount, commit) — a truncate commit, fenced on commit.runId.
  • getMessages / getMessageCount / getUIMessageCount.

State ​

  • loadState(sessionId) / saveState(sessionId, state, opts?) — load / save session state. opts is { fenceRunId } or { fenceIdle }; saveState never writes a checkpoint.
  • commitState(sessionId, state, commit, opts?) — a state commit: fenced, version-pinned (opts.expectedVersion, else state.version; StaleStateError on mismatch), returns { checkpointId, newVersion }.
  • mergeCustomState(sessionId, writes) — Atomically merge StepWrites from ImmerStateTracker into custom state.
  • updateStatus(sessionId, status, context?) — Atomic status update (no CAS).
  • compareAndSetStatus(sessionId, expectedStatuses, newStatus, options?) — Atomic CAS on session status (and optional expectedVersion). Returns { ok: true; newVersion } or { ok: false; currentStatus; currentVersion }; options accepts interruptContext, error and expectedVersion.
  • incrementStepCount(sessionId) / incrementResumeCount(sessionId) — Atomic counters.

Runs ​

  • createRun / updateRunStatus (sticky terminal statuses) / getCurrentRun / listRuns / getRun (read-your-writes).
  • getRunByRequestKey(sessionId, requestKey) (required, D-2) — the run of the session that holds a request key, or null; an indexed read, never across sessions. Each store needs the request-key migration (Postgres V18, D1 V22, DO-SQLite 16) or, on memory and Redis, a per-session index. Contract cases request-key.*.
  • renewRunLease(sessionId, runId, ttlMs) — the JS lease CAS, on the store clock.
  • setRunOwnerUnknownSince(sessionId, runId, since | null) — the CFW escape hatch (set-if-null, or clear).
  • listRunCheckpoints(sessionId, runId, page) — one run's checkpoints, full, ordered by messageCount, from an indexed read by run (never a session scan); legacy runId: null rows are never returned. planRetry reads the failed run through it.
  • abandonRunStart(sessionId, runId) — atomic with every run-start commit of the session: returns 'committed' (writes nothing) when the run exists, else records a tombstone and returns 'abandoned'. The tombstone is invisible to every run read; a later run-start commit of that id throws RunStartRejectedError('entry_abandoned'). Used by the Temporal, DBOS and CFW run-start fail-safes.

Checkpoints ​

  • createCheckpoint (a row only, never the pointer) / getLatestCheckpoint (the pointer) / getCheckpoint / listCheckpoints.

Staging and suspension ​

  • stageChanges / getStagedChanges / discardStaging / hasStagedChanges / cleanupOrphanedStaging.
  • promoteStaging(sessionId, stepId, { streamSequence, runId }) — a step commit.
  • saveStateAndPromoteStaging(sessionId, state, appendMessages, { stepId, stepCount, streamSequence, runId }, options?) — a step commit of state + appended messages + staging promotion. Honours expectedVersion (StaleStateError) and refuseIfStopped (a stop record for the commit's run refuses it with CommitStoppedError, state_commit_stopped, checked in the same atomic step after the superseded-run fence). Cross-runtime invariant C-1: when a runtime suspends, this is the single primitive that persists pending tool calls, ownership, completed phase-1 messages, the checkpoint, and suspendedStepId atomically.

Sub-sessions and stop records ​

  • addSubSessionRefs / updateSubSessionRef / getSubSessionRefs.
  • updateClientToolCallOwnership(sessionId, { add?, remove?, complete? }) — CP-128 (MR D): the field-scoped change of the session's client-tool call ownership map (clientToolCallOwnership) and its completed markers (completedClientToolCalls), in ONE atomic step that touches only those two fields (no version bump, no checkpoint), so a child writing its call's ownership onto its ROOT never fails the root's open step commit. The semantics are core applyClientToolCallOwnershipChange (every store uses it): add idempotent per owner, a foreign owner a returned conflict and never overwritten, an added id's marker cleared; remove; complete = remove + marker at the store's clock, markers past the retention reaped. A same-owner add also clears a stale marker; call ids that collide with Object.prototype names are handled as ordinary ids. opts.fenceRunId applies the change only while that run is the session's current, live run, checked in the same atomic step (saveState's fence; else RunSupersededError, nothing written): every write a run makes to its own session passes it, a child's write onto its root does not. A store that bounds its state size (D1, DO-SQLite: maxStateBytes) measures the op's columns like any state write (StateSizeExceededError, nothing written). A missing session is state_session_not_found; a store whose compare-and-swap keeps losing gives up after CLIENT_TOOL_CALL_OWNERSHIP_ATTEMPTS with a retryable state_concurrency_conflict. Writers: runtime-js and the CF DO (which runs it) write every ownership change through it — the resolver's registration (add) and clear (complete, with the retention recorded at registration), a paused child's root write after its suspension commit (persistSuspension; core decideRootOwnershipFence's none writes nothing; a failure is only logged, FU-CHILD-ROOT-OWNERSHIP-WRITE-LOGGED), a self-rooted suspension's own entries BEFORE its commit (a failed op or a foreign owner fails the suspension: the commit carries no fallback), and a rewind's own-map and root-map changes (the own-map change is core ownershipMapChange(from, to)). Temporal does too (its activities): a sub-agent's root write from commitSuspendedStep, a drain's clear (complete, with each call's recorded retention), a self-rooted suspension's own entries (before its commit, failing it like runtime-js) and a rewind's own-map change. DBOS does too: a sub-agent's root write (after its own pending entry), a self-rooted registration's own entry (before its pending save, so nothing can find the pending entry without its route; a submit in that gap answers unknown_tool_call, and a root gets no stub), the clear (complete), a stop settle's drop of the stopped calls' entries (remove) and a rewind's own-map change. CF Workflows does too: a sub-agent's root write (a client call's before its pending entry, an approval's after it; a foreign owner throws, and an approval that lost removes the entry it saved), a self-rooted registration's own entry (before its pending save, as on DBOS), and a rewind's own-map change. Its drain removes the pending entry only: it writes no marker and leaves the ownership entry. Full-state writes keep both fields: saveState, commitState, saveStateAndPromoteStaging, promoteStaging and a run-start commit ignore whatever map or markers the state they are given carries, and keep the stored values. Apart from a run-start commit's markCompleted, this op is the only way either field changes (a source-scan guard, core/src/__tests__/store/ownership-full-state-guard.test.ts, refuses a new assignment outside the store files). A rewind interrupted between its commit and its op leaves an entry with no pending call, which answers unknown_tool_call; so does a submit between a suspension's op and its commit. Contract: core/src/testing/ownership-operations.ts (OWN-*).
  • holdPriorResult(sessionId, { subSessionId, toolCallId, result, expectGeneration, runId? }) / releasePriorResult(sessionId, { subSessionId, toolCallId }) — the held prior (DO companion lifecycle MR B, deliver-then-continue): a CAS that moves a persistent child's undelivered terminal outcome into SubSessionRef.heldPrior for the tool call that will deliver it ('held' / 'stale', never a throw; completionDelivered untouched; expectGeneration is the outcomeGeneration of the ref read the result came from, so a ref reopened since that read is 'stale'), and the delivered mark after the parent's step commit (a CAS on the holder's call id: the slot clears; 'released' / 'stale'). The delivered write is conditional: the hold records the ref's OUTCOME GENERATION (SubSessionRef.outcomeGeneration, 0 until the first reopen, returned by every read, written only by the stores), which every reopen (a non-live ref written live: the resumedCompanionRef run-start commit, a live updateSubSessionRef, the DO's recordChildRunStarted) increments, and the release writes completionDelivered = true in the same atomic step only while they still match — a child reopened (and possibly ended again) since the hold keeps its new outcome undelivered. Never a status / timestamp comparison (two terminations can share a clock tick). A call RE-ENTERED after a crash between its hold and its answer's commit (CP-106: the CF DO's recovery, A4; runtime-js's resume, B2) does not hold again: it reads its own slot back with core ownHeldPrior (ruling (ii)), so the re-entered answer still carries the prior and the release lands against it as usual. Contract: core/src/testing/sub-session-operations.ts and run-start-operations.ts.
  • requestStop(sessionId, { kind, reason, target, deadlineMs }) — records a run-scoped StopRequest atomically: target: { kind: 'current' } resolves to the session's current run only while it is live (else { recorded: false, reason: 'no_live_run' }, nothing written); { kind: 'run', runId } is written as given. An abort upgrades an interrupt, never the reverse. A missing session throws state_session_not_found.
  • readStop(sessionId, runId) — the run's stop record, or null. Nothing clears a record; deleteSession removes it.
  • refuseIfStopped on commitState, appendMessages without startRun, saveStateAndPromoteStaging and promoteStaging refuses a commit of a stopped run (CommitStoppedError); a run-start that would supersede a live, stopped run is refused expected_run_stopped. See Stop as one signal.

Listing and optional extensions ​

  • listSessions(options?) — paginated summaries.
  • patchMetadata?(sessionId, patch) — Used by runtime-dbos persistent mode to record the active DBOS workflow ID.

Third-party stores: atomic implementation required ​

There is no non-atomic fallback. Every commit method must write its rows, state and checkpoint, and move the pointer, in one atomic operation (a transaction on Postgres, a batch on D1 whose version bump is the last statement, a Lua script on Redis with every touched key passed in KEYS, transactionSync on the DO), and the run-start rules must run inside that operation on the store's own clock. A previously-exported defaultSaveStateAndPromoteStaging(store, ...) helper (sequential appendMessages → saveState → promoteStaging) was removed in P3.R3-BC-FALLBACK because the crash-between-calls window it created is exactly the corruption the atomic primitive was added to prevent. All five in-tree stores (memory, redis, postgres, D1, DO) implement the atomic versions; custom stores must do the same and should run the @helix-agents/core/testing suites (stateStoreContractTests, commitCheckpointOperationTests, runStartOperationTests, fenceOperationTests). See the upgrade guide for the schema migrations (Postgres V14 / V15, D1 V17 / V18, DO-SQLite 11 / 12, Redis hash fields).

Released under the MIT License.

Copyright 2024-2026 Helix AI