Agent Communication

CMDOP agents on different machines can call each other directly. From a single chat turn your laptop’s agent can ask the prod-1 agent to scan logs, then ask db-1 to validate a schema, then aggregate the answers. This is the server-to-server feature.

Why server-to-server matters

The relay sits between every pair of agents in a workspace, so machine-to-machine calls work without inbound ports, port forwarding, or VPNs. The caller’s chat turn is preserved — the answer comes back into the same conversation, with the same audit trail. See report 05 §2.1 for the wire-level story.

The single funnel: `remoteagent`

Every cross-machine agent call goes through one client (internal/connect/remoteagent/client.go). The flow:

Resolve the workspace (CLI flag → env → named workspace → active → legacy → OAuth).
Resolve the target machine (UUID → hostname → name → fuzzy prefix).
Check Online and abort fast if not.
Dial the relay, set the target machine ID on the connection.
Call AgentService.Run (unary) or AgentService.RunStream (token stream).

Timeouts are clamped to [1 ms, 600 s], default 120 s. The same funnel powers the three agent-facing tools below.

Three agent tools that use it

Tool	Shape	When to use
`ask_agent(hostname, prompt)`	Unary, returns final reply	One target, fire-and-forget
`ask_agent_stream(hostname, prompt)`	Stream, emits tokens + tool events	One target, you want UI updates
`ask_agents(hostnames, prompt, timeout_ms?)`	Fan-out, parallel goroutines	Many targets, compare answers

See report 04 §2 for the registered tool catalogue and report 05 §2.2 for the fan-out implementation.

How a single call flows

Fan-out (`ask_agents`) semantics

Per-host timeout. [1, 300] s, default 120 s.
Total deadline. [1, 600] s, default 240 s.
Dedup. Hostnames are deduplicated while preserving insertion order.
Result map. Keyed by hostname, deterministic order. Each entry is one of:
- Response — success.
- RemoteError — the target agent ran but reported an error.
- Error — our side could not reach the target (resolve, dial, offline, timeout).
Cancellation. Cancelling the parent context drops all in-flight workers; cancelled hosts appear with TimedOut: true.

Source: internal/agent/builtin/tools/connecttool/ask_agents.go:79–237.

Error taxonomy

Class	Cause	Where to look
`resolve_error`	unknown or ambiguous hostname	Machine Identity
`offline`	target `is_online=false`	`cmdop agent status` on the target
`dial_error`	network / TLS to relay	local relay logs, `cmdop agent logs -f`
`auth_error`	no API key, OAuth expired	`cmdop login`
`remote_error`	target agent ran but failed	target machine’s logs
`timeout`	per-host or total deadline fired	tighten `timeout_ms` or scope hostnames

Permission gate fires on the receiver

The caller’s outgoing ask_agent is not gated locally — the caller is the operator. The receiver’s permissions.yaml decides whether the inbound tool can execute. Self-to-self calls (same OAuth identity, verified via CallerHostname server-side) bypass the gate by design.

The receiver decides what tools the caller may invoke. See Permissions.

Self-to-self calls (same OAuth user) skip the permission gate. If you want to hard-gate every inbound call, run the receiver under a different account.

Streaming

ask_agent_stream emits typed events:

Event	Meaning
`TOKEN`	Next text fragment from the LLM.
`TOOL_START`	A tool call is about to run on the target.
`TOOL_END`	The tool finished; payload includes result snippet.
`THINKING`	Provider thinking marker (when supported).
`ERROR`	A non-fatal error during the run.
`HANDOFF`	The agent delegated to a subagent.
`CANCELLED`	The stream was cancelled by the caller.

As of 2026-04-26 the daemon ships per-token events; the desktop direct-pipe path was on unary Ask during the gap and is being flipped back. See report 05 open-question 1.

Example calls


// ask_agent — single target, unary
{
  "tool": "ask_agent",
  "args": {
    "hostname": "prod-1",
    "prompt": "Show last 50 lines of /var/log/nginx/error.log"
  }
}


// ask_agents — fan-out across three hosts
{
  "tool": "ask_agents",
  "args": {
    "hostnames": ["prod-1", "prod-2", "prod-3"],
    "prompt": "uptime",
    "timeout_ms": 30000
  }
}

A successful fan-out result map looks like:


{
  "prod-1": { "ok": true, "response": "up 12 days, load 0.41" },
  "prod-2": { "ok": false, "remote_error": "ssh_session: read timeout" },
  "prod-3": { "ok": false, "error": "timeout", "timed_out": true }
}

When to use which tool

One target, fire-and-forget → ask_agent.
One target, want UI tokens → ask_agent_stream.
Many targets, want to compare answers → ask_agents.
Many targets but you need a sequential pattern (rolling deploy) → loop over ask_agent, not ask_agents. Fan-out is parallel by design.

TAGS: agent-communication, ask_agent, ask_agents, fan-out, server-to-server DEPENDS_ON: [machine-identity, permissions, daemon]