Skip to main content
Execution broker protocol v1. Status: normative for the broker execution backend (internal/execution/broker). This document defines the interface between:
  • the client: Formation’s broker execution backend, running inside a Formation instance whose administrator may be untrusted by the broker; and
  • the broker: any HTTPS service that runs agent workspaces on its own sandbox infrastructure, admits work by its own policy, enforces every limit, and cleans workspaces up independently of Formation.
Formation does not know or care who operates a broker. Anyone may implement one: a hosting provider offering managed execution, a platform team fronting its own cluster, or a self-hosted service in front of a sandbox vendor. A Formation install is pointed at a broker with two settings, its base URL and an opaque bearer token (see Sandboxed execution). Where this document and the Formation client disagree, this document wins and the client is the bug. A broker that conforms to this document works with Formation without any Formation change.

Contents

Trust model

  • The client is untrusted. The person who administers a Formation instance can read or replace anything it stores, including the token. Every limit in this document is enforced by the broker; nothing may depend on the client behaving. The client’s own checks are conveniences that fail earlier, never controls.
  • A token authenticates exactly one client instance. The broker calls the grant a token carries a lease. Resource ownership is the client instance, not the token: rotating the token keeps the instance’s workspaces and snapshots.
  • The client never receives a credential for the broker’s sandbox provider, nor that provider’s sandbox, image or account identifiers, nor another instance’s identifiers. Every id the client sees is minted by the broker and is meaningless anywhere else.
  • An id that does not exist and an id owned by another instance are indistinguishable: both answer 404 not_found.
  • The broker must not log, persist or emit in metrics: argv, environment values, stdin, stdout/stderr, file paths, file contents or the token. The one path it may persist is a snapshot’s directory, which restore needs; it is never logged or returned.

Conventions

  • Transport: HTTPS only (TLS 1.2+). The broker refuses plaintext. The client does not follow redirects; a 3xx is an error. A redirect could otherwise carry the Authorization header to a host the operator never configured.
  • Base URL: the backend’s configured endpoint, for example https://broker.example.com. It is an https URL with a host and no userinfo, query or fragment; it may carry a path prefix. All paths below are relative to it and begin with the API version /v1. A breaking change ships as /v2. /v1 adds only optional response fields and new error codes, which clients must tolerate.
  • Bodies: application/json; charset=utf-8. The broker rejects unknown request fields (400 invalid_request), so a client typo fails loudly. Clients ignore unknown response fields; a broker may return additional implementation-defined fields (for example pricing or diagnostics).
  • Attribution fields: the request fields this document marks attribution (agent and run_id on Acquire) are optional identifiers a broker may use to account usage per agent; they are never a control and never content. A broker that implements this revision accepts them and ignores the ones it does not use. A broker written against an earlier revision answers 400 invalid_request for them; because a 400 consumes no idempotency key, the client then re-sends the same acquisition without them, so attribution degrades to absence rather than failing a workspace.
  • Binary data (stdin, stdout, stderr, file contents) is standard base64 with padding (RFC 4648 §4) in fields suffixed _b64.
  • Times are RFC 3339 UTC strings with second precision, for example 2026-10-13T21:04:05Z. Durations are integer seconds in fields suffixed _seconds.
  • Headers:
    • Authorization: Bearer <token> on every /v1 request.
    • Idempotency-Key where stated (see Idempotency).
    • X-Request-Id: optional client value (at most 64 characters of [A-Za-z0-9._-]). The broker echoes it, or a generated one, on every response. Both sides include it in their logs.
    • Mesh-Version: optional client build version, for broker diagnostics only.
  • Body limit: 12 MiB per request after base64 (413 payload_too_large).
  • Every response carries cache-control: no-store.

Identifiers

Workspace, snapshot and lease ids are minted by the broker. Each is 1 to 64 characters of [A-Za-z0-9._-] and is never . or .., so it is safe as a URL path segment. Clients store ids verbatim and never parse them. A broker may use a recognizable shape (for example ws_ + random characters), but clients must not rely on one.

Credential

The client authenticates with one opaque bearer token, issued by the broker (or by whoever provisions the instance on the broker’s behalf).
  • Format: defined by the broker. The client trims surrounding whitespace, requires a non-empty value and otherwise never parses, validates or transforms it. Brokers are encouraged to use a fixed, recognizable prefix so secret scanners can find leaked tokens, and to store only a hash of the secret part.
  • Client storage: Formation stores the token in its write-only, encrypted secret slot for the backend (secret kind execution_broker_token), or reads it from MESH_EXECUTION_BROKER_TOKEN. It is sent only in the Authorization header, never in a URL, log line, sandbox environment, tool output or error message.
  • Scope: fixed by the broker when it issues the token and readable with GET /v1/lease: allowed sizes and default size, maximum concurrent workspaces, maximum workspace lifetime, idle timeout, maximum exec timeout, snapshot TTL and count quota, allowed images, egress ceiling. The client cannot change scope.
  • Expiry: a token may expire. After expiry every request answers 401 lease_expired. Running workspaces continue to their own deadline and are reaped by the broker; a new token for the same instance can still control them.
  • Rotation: issuing a new token for an instance may supersede the previous one. A superseded token should keep working for a short grace period so the new token can be installed in Formation; after it, it answers 401 lease_superseded. Workspaces and snapshots belong to the instance and are unaffected.
  • Revocation: immediate. Every subsequent request with that token answers 401 lease_revoked. A broker may also terminate the instance’s running workspaces on revocation.

Mapping to the Formation backend contract

Contract source: internal/execution/execution.go (execution.Backend), internal/execution/command.go (execution.DurableBackend) and internal/execution/provider.go. execution.Size maps to the broker’s size verbatim (small, standard, large, xlarge); SizeDefault (empty) is sent as an omitted size. WorkspaceSpec.Name is sent as label, WorkspaceSpec.Env as env, WorkspaceSpec.Egress as egress, and WorkspaceSpec.Image as image (omitted when empty).

Operations

GET /v1/lease

Read-only grant, limits and admission status for the presented token. Safe to poll (rate limited like everything else). Creates nothing, reserves nothing. 200:
  • api_version is informational. The protocol version is the /v1 path prefix; clients do not branch on this field.
  • lease.state: active | superseded (still inside its grace period). Expired, revoked or post-grace tokens never reach a 200.
  • admission.state: open | paused | emergency_stopped. paused and emergency_stopped add a broker-defined "reason" string (for example payment, instance_paused, operator, or unavailable when the broker cannot read its own stop state and is failing closed). Admission conditions the broker evaluates only at create time, such as funds or budgets, are not necessarily reflected here.
  • sizes lists only sizes this token may use, smallest first. The values shown are examples; the broker defines what each tier means.
  • egress.ceiling: null (any valid domain allowed, up to max_egress_domains) or a list of entries; each requested domain must equal an entry or be matched by a *. entry.
An operator may set Formation’s install ceiling (max_size) from the largest listed size; the broker rejects anything larger either way.

POST /v1/workspaces (Acquire)

Headers: Idempotency-Key required. Request:
Each failed check creates nothing and leaves no reservation or charge behind. Recommended order: authenticate → idempotency replay → validate → stop and pause state → instance concurrency → admission (for example funds or budget) → provider create → restore. A failure after provider create terminates what was created before answering. 201 (and on idempotent replay 200 with the same body shape and current state):
expires_at is the hard deadline. After it every operation on the workspace answers 410 workspace_gone. The client treats a workspace as unusable after expires_at and acquires a new one. Errors: 400, 401, 402 insufficient_funds, 402 budget_exhausted, 403 paused, 403 emergency_stopped, 404 not_found (snapshot), 409 idempotency_conflict, 409 idempotency_in_progress, 410 snapshot_gone, 422 size_not_allowed|image_not_allowed|egress_required|egress_not_allowed, 429 concurrency_limit, 429 risk_limit, 429 rate_limited, 502 upstream_error, 503 admission_unavailable.

GET /v1/workspaces/{workspace_id}

200:
state: starting | running | terminating | terminated. termination_reason (when terminated): released, expired, idle, lease_revoked, paused, emergency_stopped, orphaned, failed. A terminated workspace’s status object may also carry metered_seconds, an optional response field: the integer seconds the broker metered the sandbox for, by its own clock and rounding. The client records terminated_at, termination_reason and metered_seconds beside its own release time when the release answers with them (below), and reports a release the broker has not confirmed as such. Formation’s usage reporting reads these columns; the broker’s own settlement remains the authority on what the sandbox cost.

GET /v1/workspaces?state=live&cursor=…&limit=…

Lists this instance’s workspaces. state: live (starting, running, terminating; default) or all (adds recently terminated ones). limit 1–100, default 50. 200: {"workspaces":[<status objects>], "next_cursor": string|null}.

POST /v1/workspaces/{workspace_id}/exec (Exec)

No idempotency key: exec is not replayable. On a transport error or timeout the outcome is unknown and the client reports it, never retries. Request:
  • argv: required, 1–4096 strings, total at most 256 KiB. Executed without a shell.
  • dir: optional absolute path; empty = the image’s default working directory.
  • stdin_b64: optional, decoded at most max_file_bytes.
  • timeout_seconds: optional; 0/omitted = default_exec_timeout_seconds. Values above max_exec_timeout_seconds, or beyond the workspace’s expires_at, are clamped, and the response says so (timeout_clamped: true).
200:
A non-zero exit_code is a result, not an error. Signals are folded into 128 + signal; the client classifies exit_code > 128 as execution.KilledSignal. Each stream is truncated at max_output_bytes and flagged; the client must surface truncation rather than present partial output as complete. Client HTTP timeout: timeout_seconds + 30. Errors: 404 not_found, 410 workspace_gone, 409 workspace_not_ready (starting), 413, 502 upstream_error.

POST /v1/workspaces/{workspace_id}/files/read (ReadFile)

Request {"path": "/abs/path"} (absolute, at most 4096 bytes, no NUL; carried in the body so it never appears in access logs). 200: {"contents_b64": "…", "size_bytes": 123}. Errors: 404 file_not_found (client wraps fs.ErrNotExist), 404 not_found (workspace), 410 workspace_gone, 413 file_too_large (file over max_file_bytes), 422 not_a_file. file_not_found and not_found are distinguished only by error.code; the client checks the code, not just the status.

POST /v1/workspaces/{workspace_id}/files/write (WriteFile)

Request {"path": "/abs/path", "contents_b64": "…"} (decoded at most max_file_bytes). Creates parent directories and replaces the file. Naturally idempotent. 204. Errors: 404 not_found, 410 workspace_gone, 413, 422 not_a_file.

POST /v1/workspaces/{workspace_id}/snapshots (Snapshot)

Headers: Idempotency-Key required. Request {"dir": "/abs/project/dir"}. 201 (replay 200):
expires_at = created_at + snapshot_ttl_seconds. Every snapshot has a finite TTL, and the broker deletes it at expiry even if the client never calls DELETE. Errors: 404, 410 workspace_gone, 422 snapshot_quota_exceeded (live snapshots at max_snapshots; delete some first), 502 upstream_error.

DELETE /v1/snapshots/{snapshot_id} (DeleteSnapshot)

204, including when already deleted or expired. 404 not_found for ids this instance never owned; the client treats 404 as success too, because there is nothing for it to reclaim.

DELETE /v1/workspaces/{workspace_id} (Release)

Terminates the sandbox and stops its usage. Idempotent. 200 with the status object (state: "terminated", or "terminating" when the provider did not confirm in time and the broker keeps retrying). 404 not_found is success for the client. Release never fails because of funds, budgets, a pause, an emergency stop or a superseded token; a revoked or expired token is refused like any other request. The client reads the answered status object: terminated confirms the end and the client keeps terminated_at, termination_reason and, when present, metered_seconds; terminating and 404 are successful releases whose end the broker did not confirm, which the client records as unconfirmed.

Durable commands

Identity: the client’s durable tool-call identity, 1–128 characters of [A-Za-z0-9._:-], unique per workspace. Command bodies are the same as exec.
  • PUT /v1/workspaces/{id}/commands/{identity} (StartCommand). 202 {"state": "starting"|"running"|…}. Claiming the identity is atomic: a repeated PUT with the same identity never launches a second process and returns the current status (200) whatever the body. 422 command_not_started means preflight failed before dispatch; the client wraps execution.ErrCommandNotStarted. A 4xx the broker answers before it claims the identity (authentication, token state, admission, an unknown or ended workspace, a malformed request) also proves nothing started. Every other failure, including timeouts and 5xx, means the outcome is unknown: the client calls GET, and never relies on a later PUT to learn the outcome.
  • GET /v1/workspaces/{id}/commands/{identity} (ReadCommand). 200:
    state: starting | running | completed | canceled | unknown. Exit fields are meaningful only for completed/canceled. An unknown identity answers 200 {"state":"unknown"}; a gone workspace answers 410 workspace_gone, which the client also maps to CommandUnknown. A state the client does not recognize is never read as finished or not started.
  • POST /v1/workspaces/{id}/commands/{identity}/cancel (CancelCommand). 202; idempotent; fences a delayed launch. Completed results survive.

Errors

Every non-2xx response:
message is safe to show an operator and contains no secrets. Clients branch on code only. retryable is a hint; Retry-After (seconds) is set on 429 and 503 when known. details is optional; details.reason carries a broker-defined cause where noted. A broker decides its own admission policy. The codes below let it say why it refused in a way every client can act on; a broker that has no notion of funds, budgets or risk limits simply never sends those codes. A body that is not this envelope (for example a load balancer’s HTML 502) is still an error; the client keys it by status and fails closed.

Idempotency and retries

  • Idempotency-Key: 1–200 characters of [A-Za-z0-9._:-], required on POST /v1/workspaces and POST …/snapshots. Scope: (instance, operation, key). Retained at least 7 days.
  • The broker hashes the canonical request (env values included only as a hash). A replay with the same key and hash returns the original resource (same workspace_id/snapshot_id) with 200 and its current state; it never creates, reserves or charges twice, even across token rotation. A different hash answers 409 idempotency_conflict. A replay while the original is still executing answers 409 idempotency_in_progress.
  • A request rejected before any side effect (4xx other than conflicts) does not consume the key: retrying with the same key re-evaluates, so a client that got 402 can retry after the account is funded.
  • Client keys are deterministic per logical operation and reused verbatim by every retry of it. Formation keys an acquisition by its workspace and snapshot, and a snapshot capture by a per-call nonce.
  • Safe to retry automatically (same key where applicable): GET requests, POST /v1/workspaces, POST …/snapshots, DELETE requests, files/write, files/read and …/cancel. Never retried automatically: exec. An uncertain PUT …/commands/{identity} (StartCommand) outcome is recovered through GET …/commands/{identity}; because claiming is atomic, re-sending the same PUT cannot launch twice, but it is never how a client decides a command did not run.
  • Retries use exponential backoff starting at 1 s with jitter, at most 5 attempts, and only for transport errors, 409 idempotency_in_progress, 429 rate_limited, 502 and 503. 429 concurrency_limit and 429 risk_limit clear on the scale of a run, so a client may surface them immediately instead.

Broker obligations

The client is not trusted to clean up, and its database may be lost. A conforming broker therefore:
  • enforces every limit in the token’s scope itself, and never silently clamps a size, image or egress request (it refuses with the codes above);
  • bounds every workspace by a hard lifetime and an idle timeout that hold even if the broker’s own control loop and the client are both down (for example a provider-side sandbox timeout);
  • reaps workspaces past expires_at, workspaces of revoked, paused or stopped scope, and creates whose outcome it could not confirm;
  • sweeps provider-side sandboxes that have no live broker record;
  • deletes snapshots at their TTL, retrying until the provider confirms;
  • fails closed: when it cannot evaluate admission or stop state it refuses new work (503 admission_unavailable) rather than admitting it.
How a broker meters, prices or bills usage, and how it provisions tokens, is outside this protocol.

Client checklist

  1. Configuration: an https endpoint (reject other schemes at the form) and an opaque token in a write-only secret slot. The connection test is GET /v1/lease.
  2. HTTP client: TLS 1.2+ required, no redirects, per-call timeouts (exec: timeout_seconds + 30, where an omitted timeout_seconds means the lease’s default_exec_timeout_seconds; others 60 s), token in the Authorization header only. Never log the token, request bodies, output or paths. Log X-Request-Id and error.code.
  3. Map every operation as in the mapping table, including 404-as-success for Release and DeleteSnapshot, file_not_found → fs.ErrNotExist, command_not_started → ErrCommandNotStarted, and 410 workspace_gone → CommandUnknown on read.
  4. Derive idempotency keys before Acquire and Snapshot and reuse them on every retry of that call.
  5. Never send a nil egress policy: the broker backend requires the install’s workspace egress mode to be allowlist. Surface egress_required as an operator-actionable error.
  6. Treat 401, 402 and 403 codes as actionable tool errors naming the cause (token, funds, budget, paused, stopped) and who fixes it, not as transient failures. Never fall back to another backend.
  7. Reaper: Release stale workspaces as for any backend; optionally list GET /v1/workspaces?state=live and release ids with no live client record.
  8. The client is a convenience, not a control: do not re-implement limits, pricing or admission in the client.