Remote Agents (Temporal + HTTP)

This example demonstrates a Temporal orchestrator agent delegating to remote specialist agents running on a separate HTTP service. It shows:

• Cross-runtime orchestration (Temporal orchestrator + JS runtime service)
• HttpRemoteAgentTransport for HTTP/SSE communication
• AgentServer with Express for hosting remote agents
• createRemoteSubAgentTool() for transparent remote delegation

Source Code

The full example is in examples/remote-agents-temporal/.

Architecture

graph LR
    Client["Client<br/>(starts workflow)"]
    Worker["Temporal Worker<br/>Orchestrator Agent"]
    Service["Express Service<br/>AgentServer"]
    Researcher["Researcher Agent"]
    Summarizer["Summarizer Agent"]

    Client -->|"Temporal workflow"| Worker
    Worker -->|"HTTP + SSE"| Service
    Service --> Researcher
    Service --> Summarizer

The orchestrator runs on Temporal for durable execution and crash recovery. The researcher and summarizer run on a lightweight Express service using AgentServer. Communication uses HttpRemoteAgentTransport (HTTP for requests, SSE for streaming).

Prerequisites

• Node.js 18+
• Docker (for Temporal and Redis)
• OpenAI API key

Project Structure

examples/remote-agents-temporal/
├── src/
│   ├── agents/
│   │   ├── orchestrator.ts  # Parent agent with remote sub-agent tools
│   │   ├── researcher.ts    # Specialist agent (web search + notes)
│   │   └── summarizer.ts    # Specialist agent (pure LLM, no tools)
│   ├── types.ts             # Shared Zod schemas
│   ├── server.ts            # Express server hosting agents
│   ├── runtime.ts           # Shared Redis stores (state / stream / usage) + Temporal client adapter
│   ├── workflows.ts         # Temporal workflow
│   ├── activities.ts        # Temporal activities (usageStore + logger wired here)
│   ├── worker.ts            # Temporal worker entry point
│   ├── client.ts            # Client that starts the workflow; prints the folded usage rollup
│   ├── approve.ts           # CLI: submit the client-executed `slowApproval` result
│   └── inspect.ts           # CLI: snapshot/usage reads over the remote protocol
├── docker-compose.yml
└── package.json

Running the Example

1. Install Dependencies

cd examples/remote-agents-temporal
npm install

2. Set Up Environment

cp .env.example .env
# Edit .env and add your OPENAI_API_KEY

3. Start Infrastructure

npm run docker:up

This starts Temporal on localhost:7233 and Redis on localhost:6379. Redis is the orchestrator's shared state / stream / usage store — see Shared Redis Stores.

4. Start the Remote Agent Service

# Terminal 1
npm run server

The service starts on http://localhost:4000 with two agents:

• researcher — Searches for information and takes notes
• summarizer — Summarizes text into key points

5. Start the Temporal Worker

# Terminal 2
npm run worker

6. Run the Client

# Terminal 3
npm run client "benefits of TypeScript"

Key Components

Remote Agent Service

The server uses AgentServer to host specialist agents:

// src/server.ts
import express from 'express';
import { AgentServer, createHttpAdapter, createExpressAdapter } from '@helix-agents/agent-server';
import { JSAgentExecutor } from '@helix-agents/runtime-js';
import { VercelAIAdapter } from '@helix-agents/llm-vercel';
import {
  InMemoryStateStore,
  InMemoryStreamManager,
  InMemoryUsageStore,
} from '@helix-agents/store-memory';

const stateStore = new InMemoryStateStore();
const streamManager = new InMemoryStreamManager();
const usageStore = new InMemoryUsageStore();
const executor = new JSAgentExecutor(stateStore, streamManager, new VercelAIAdapter());

const agentServer = new AgentServer({
  allowUnauthenticated: true, // dev only — production must wire `authenticate`
  agents: {
    researcher: ResearcherAgent,
    summarizer: SummarizerAgent,
  },
  stateStore,
  streamManager,
  // Enables `GET /usage` (and backs the consumer's `getUsage()` fallback fold).
  // This Node/JS `AgentServer` never attaches `usage` to the SSE `end` frame —
  // only the Cloudflare DO producer does that.
  // `AgentServer` forwards this into the executor as `ExecuteOptions.usageStore`
  // on every /start and /resume — the JS runtime's only usage opt-in.
  usageStore,
  executor,
});

const app = express();
app.use(express.json());
app.use('/', createExpressAdapter(createHttpAdapter(agentServer)));
app.listen(4000);

This exposes the remote-agent protocol endpoints: /start, /resume, /sse, /status, /snapshot, /usage, /interrupt, /abort, and /submit-tool-result.

Orchestrator Agent

The orchestrator uses createRemoteSubAgentTool to delegate to remote agents:

// src/agents/orchestrator.ts
import {
  defineAgent,
  createRemoteSubAgentTool,
  HttpRemoteAgentTransport,
} from '@helix-agents/core';

const transport = new HttpRemoteAgentTransport({
  url: process.env.REMOTE_AGENT_URL || 'http://localhost:4000',
});

const researcherTool = createRemoteSubAgentTool('researcher', {
  description: 'Delegate research to a remote specialist agent',
  inputSchema: z.object({
    query: z.string().describe('The research query'),
  }),
  outputSchema: ResearcherOutputSchema,
  transport,
  remoteAgentType: 'researcher',
  timeoutMs: 120_000,
});

const summarizerTool = createRemoteSubAgentTool('summarizer', {
  description: 'Delegate summarization to a remote specialist agent',
  inputSchema: z.object({
    text: z.string().describe('The text to summarize'),
  }),
  outputSchema: SummarizerOutputSchema,
  transport,
  remoteAgentType: 'summarizer',
  timeoutMs: 60_000,
});

export const OrchestratorAgent = defineAgent({
  name: 'orchestrator',
  outputSchema: OrchestratorOutputSchema,
  tools: [researcherTool, summarizerTool],
  systemPrompt: `You are a research orchestrator.
1. Use the researcher to gather information
2. Use the summarizer to distill findings
3. Call __finish__ with your final output`,
  llmConfig: { model: openai('gpt-4o-mini') },
  maxSteps: 10,
});

Specialist Agents

The researcher agent uses tools (web search, note-taking):

// src/agents/researcher.ts
export const ResearcherAgent = defineAgent({
  name: 'researcher',
  stateSchema: ResearcherStateSchema,
  outputSchema: ResearcherOutputSchema,
  tools: [webSearchTool, takeNotesTool],
  systemPrompt: (state) => `You are a research specialist...`,
  llmConfig: { model: openai('gpt-4o-mini') },
  maxSteps: 10,
});

The summarizer is a pure LLM agent (no tools):

// src/agents/summarizer.ts
export const SummarizerAgent = defineAgent({
  name: 'summarizer',
  outputSchema: SummarizerOutputSchema,
  tools: [],
  systemPrompt: `You are a summarization expert...`,
  llmConfig: { model: openai('gpt-4o-mini') },
  maxSteps: 5,
});

Shared Schemas

Output schemas are shared between the orchestrator and the remote service:

// src/types.ts
export const ResearcherOutputSchema = z.object({
  findings: z.array(
    z.object({
      title: z.string(),
      snippet: z.string(),
      url: z.string(),
    })
  ),
  rawNotes: z.array(z.string()),
});

export const SummarizerOutputSchema = z.object({
  keyPoints: z.array(z.string()),
  summary: z.string(),
});

export const OrchestratorOutputSchema = z.object({
  topic: z.string(),
  researchFindings: z.array(z.string()),
  summary: z.string(),
  sources: z.array(z.string()),
});

Execution Flow

1. The client starts a Temporal workflow for the orchestrator
2. The orchestrator LLM calls subagent__researcher with a query
3. The Temporal workflow routes the call to a dedicated executeRemoteSubAgentCall activity that calls POST /start on the remote service, then consumes GET /sse with crash recovery and stream proxying
4. The researcher runs independently (web search, note-taking), returns structured output
5. The orchestrator LLM calls subagent__summarizer with the findings
6. The summarizer returns key points and a summary
7. Before __finish__, the orchestrator calls its slowApproval tool (execute: 'client'), which suspends the workflow until a human submits a result (see below)
8. The orchestrator calls __finish__ with the final structured output

Client-Executed Approval (HITL)

The orchestrator ships a slowApproval tool marked execute: 'client' (in src/agents/orchestrator.ts): after the researcher + summarizer pipeline runs, the orchestrator must get human approval before calling __finish__. Because it runs on the Temporal runtime, the wait is durable — and in v7 the durability comes from the state store, not a long-lived workflow holding an in-memory promise:

1. When the LLM invokes slowApproval, the workflow exits early with status: 'suspended_client_tool' and persists the pending call to SessionState.pendingClientToolCalls in the shared Redis store. The Temporal workflow terminates — nothing is held in memory.
2. A human runs npm run approve -- <sessionId> <toolCallId> approve|reject [note] (src/approve.ts), which calls submitToolResult.
3. TemporalAgentExecutor.resume() spawns a fresh __resume-N workflow that drains the submitted result into messages and continues the run to __finish__.

The canonical "awaiting client submission" signal is the pendingClientToolCalls map on the persisted session — the session-level status stays active, so the run remains resumable across the suspension. This demonstrates that a remote-agent orchestrator and a durable human-approval gate compose: the researcher/summarizer children run cross-service while the parent suspends on a client tool. See the client-executed tools guide for the full contract.

Snapshot, Usage & Output Re-validation

The agent service serves two protocol reads that a consumer executor uses to fold a remote child's state and usage back into the parent. src/inspect.ts calls them directly over HttpRemoteAgentTransport:

npm run inspect -- <sessionId>

• GET /snapshot?sessionId= — the session's own customState, plus a streamSequence pinning that state to a position in the event stream. As long as the session is not terminal — AgentServer.getSnapshot short-circuits only on completed / failed, so active, interrupted and paused all run the pin loop — the value can come back checkpoint-pinned (always >= 1) and can be passed straight to GET /sse?fromSequence= (exclusive) for exact late-join reconstruction. A terminal session returns streamSequence: -1 — it serves the final live state row, which pins a moment, not a position, so there is no patch continuation. -1 is also returned when no checkpoint can be aligned (a fresh session before its first promote, a <= 0 checkpoint sequence, or a concurrent-promote pointer race).
• GET /usage?sessionId= — the session's recursive usage rollup. The route always folds with includeSubAgents: true, and it is served only because the server wires AgentServerConfig.usageStore; that same store also backs the consumer's getUsage() fallback fold. This Node/JS-hosted AgentServer never attaches usage to the SSE end frame — wired or not, only the Cloudflare DO producer does that. Without a usage store, the route responds 404 FEATURE_UNAVAILABLE and consumers skip the usage fold with a warning.

inspect talks to the remote agent service (researcher / summarizer), not the Temporal-hosted orchestrator. To get a remote child's sessionId, read it off the worker log: src/activities.ts passes consoleLogger into GenericActivities (whose default is the silent noopLogger), so the core remote-sub-agent dispatch prints [remote-subagent] Stream connected { remoteSessionId: 'session-…-remote-call_r1' } for every remote call. The id is also derivable — <orchestratorSessionId>-remote-<toolCallId> — and rides on the parent's subagent_start stream chunk as subSessionId. Because the service uses InMemoryStateStore, remote sessions are lost when the server restarts; passing a stale one makes inspect degrade on the transport's typed RemoteAgentNotFoundError with a "session not found" message.

Two consumer-side behaviors come for free once the orchestrator's activity surface is wired with a usageStore (src/activities.ts — without one, GenericActivities records no usage at all and the rollup recovered via getUsage() has nowhere to fold):

• Output re-validation — the orchestrator's remote tools declare an outputSchema; the consumer runtime safeParses the child's end.output against it and surfaces a structured tool error on mismatch, so a remote agent that drifts from its contract fails loudly instead of poisoning the orchestrator's context.

• Usage fold — the orchestrator's rollup includes the researcher's and summarizer's tokens via the embedded remote rollup, even though they ran in a different process against a different store. npm run client prints it when the workflow returns, reading getRollup(sessionId, { includeSubAgents: true }) from the same Redis usage store the worker writes to:

  Usage rollup (orchestrator session, includeSubAgents: true)
  
    orchestrator's own tokens:        prompt=6812 completion=524 total=7336
    including sub-agents:             prompt=13715 completion=1136 total=14851
    => folded in from remote children: 7515 total tokens
    sub-agent calls:                 2 (researcher, summarizer)

  Illustrative magnitudes, not a transcript — token counts differ on every run, and nobody should expect to reproduce these digits. The shape is the point, and the invariant it demonstrates is exact:

  tokensIncludingSubAgents − tokens  ==  what the remote children spent
                    14851 − 7336     ==  7515

  The orchestrator's own LLM calls never saw those 7,515 tokens — the researcher and the summarizer burned them in another process, against another store. They show up here only because the consumer's completion fold recovered each child's recursive rollup via getUsage() — this Node/JS producer does not put usage on the SSE end frame — and recordSubSessionResult embedded it. Cross-check the folded number against npm run inspect -- <childSessionId>, whose usage.rollup is the producer side of the very same fold.

Shared Redis Stores (orchestrator side)

The orchestrator's state store, stream manager and usage store are all Redis-backed (src/runtime.ts), because three separate processes have to agree on them:

Process	Needs
Temporal worker	writes session state, streams, and the usage rollup (src/activities.ts)
npm run approve	writes the client-tool result into SessionState.pendingClientToolCalls, then resumes
npm run client	reads the folded usage rollup after the workflow returns

The remote agent service keeps its own in-memory stores — it is a different service, and the protocol (/snapshot, /usage, SSE end) is what carries its state and usage back to the consumer.

Production Considerations

• Replace InMemoryStateStore with RedisStateStore on the agent service
• Replace InMemoryStreamManager with RedisStreamManager on the agent service
• Add authentication headers to the transport
• Set appropriate timeoutMs values based on expected agent execution times
• Use Temporal Cloud for production workflow execution

Next Steps

• Remote Agents Guide — Full guide with patterns and configuration
• API Reference — AgentServer and transport API
• Temporal Runtime — Temporal runtime reference
• Sub-Agents Guide — Local sub-agent orchestration