broker execution
backend (internal/execution/broker).
This document defines the interface between:
- the client: Formation’s
brokerexecution backend, running inside a Formation instance whose administrator may be untrusted by the broker; and - the broker: any HTTPS service that runs agent workspaces on its own sandbox infrastructure, admits work by its own policy, enforces every limit, and cleans workspaces up independently of Formation.
Contents
- Trust model
- Conventions
- Credential
- Mapping to the Formation backend contract
- Operations
- Durable commands
- Errors
- Idempotency and retries
- Broker obligations
- Client checklist
Trust model
- The client is untrusted. The person who administers a Formation instance can read or replace anything it stores, including the token. Every limit in this document is enforced by the broker; nothing may depend on the client behaving. The client’s own checks are conveniences that fail earlier, never controls.
- A token authenticates exactly one client instance. The broker calls the grant a token carries a lease. Resource ownership is the client instance, not the token: rotating the token keeps the instance’s workspaces and snapshots.
- The client never receives a credential for the broker’s sandbox provider, nor that provider’s sandbox, image or account identifiers, nor another instance’s identifiers. Every id the client sees is minted by the broker and is meaningless anywhere else.
- An id that does not exist and an id owned by another instance are
indistinguishable: both answer
404 not_found. - The broker must not log, persist or emit in metrics: argv, environment values, stdin, stdout/stderr, file paths, file contents or the token. The one path it may persist is a snapshot’s directory, which restore needs; it is never logged or returned.
Conventions
- Transport: HTTPS only (TLS 1.2+). The broker refuses plaintext. The client
does not follow redirects; a 3xx is an error. A redirect could otherwise carry
the
Authorizationheader to a host the operator never configured. - Base URL: the backend’s configured
endpoint, for examplehttps://broker.example.com. It is anhttpsURL with a host and no userinfo, query or fragment; it may carry a path prefix. All paths below are relative to it and begin with the API version/v1. A breaking change ships as/v2./v1adds only optional response fields and new error codes, which clients must tolerate. - Bodies:
application/json; charset=utf-8. The broker rejects unknown request fields (400 invalid_request), so a client typo fails loudly. Clients ignore unknown response fields; a broker may return additional implementation-defined fields (for example pricing or diagnostics). - Attribution fields: the request fields this document marks attribution
(
agentandrun_idon Acquire) are optional identifiers a broker may use to account usage per agent; they are never a control and never content. A broker that implements this revision accepts them and ignores the ones it does not use. A broker written against an earlier revision answers400 invalid_requestfor them; because a400consumes no idempotency key, the client then re-sends the same acquisition without them, so attribution degrades to absence rather than failing a workspace. - Binary data (stdin, stdout, stderr, file contents) is standard base64
with padding (RFC 4648 §4) in fields suffixed
_b64. - Times are RFC 3339 UTC strings with second precision, for example
2026-10-13T21:04:05Z. Durations are integer seconds in fields suffixed_seconds. - Headers:
Authorization: Bearer <token>on every/v1request.Idempotency-Keywhere stated (see Idempotency).X-Request-Id: optional client value (at most 64 characters of[A-Za-z0-9._-]). The broker echoes it, or a generated one, on every response. Both sides include it in their logs.Mesh-Version: optional client build version, for broker diagnostics only.
- Body limit: 12 MiB per request after base64 (
413 payload_too_large). - Every response carries
cache-control: no-store.
Identifiers
Workspace, snapshot and lease ids are minted by the broker. Each is 1 to 64 characters of[A-Za-z0-9._-] and is never . or .., so it is safe as a URL
path segment. Clients store ids verbatim and never parse them. A broker may use
a recognizable shape (for example ws_ + random characters), but clients must
not rely on one.
Credential
The client authenticates with one opaque bearer token, issued by the broker (or by whoever provisions the instance on the broker’s behalf).- Format: defined by the broker. The client trims surrounding whitespace, requires a non-empty value and otherwise never parses, validates or transforms it. Brokers are encouraged to use a fixed, recognizable prefix so secret scanners can find leaked tokens, and to store only a hash of the secret part.
- Client storage: Formation stores the token in its write-only, encrypted secret
slot for the backend (secret kind
execution_broker_token), or reads it fromMESH_EXECUTION_BROKER_TOKEN. It is sent only in theAuthorizationheader, never in a URL, log line, sandbox environment, tool output or error message. - Scope: fixed by the broker when it issues the token and readable with
GET /v1/lease: allowed sizes and default size, maximum concurrent workspaces, maximum workspace lifetime, idle timeout, maximum exec timeout, snapshot TTL and count quota, allowed images, egress ceiling. The client cannot change scope. - Expiry: a token may expire. After expiry every request answers
401 lease_expired. Running workspaces continue to their own deadline and are reaped by the broker; a new token for the same instance can still control them. - Rotation: issuing a new token for an instance may supersede the previous
one. A superseded token should keep working for a short grace period so the
new token can be installed in Formation; after it, it answers
401 lease_superseded. Workspaces and snapshots belong to the instance and are unaffected. - Revocation: immediate. Every subsequent request with that token answers
401 lease_revoked. A broker may also terminate the instance’s running workspaces on revocation.
Mapping to the Formation backend contract
Contract source:internal/execution/execution.go (execution.Backend),
internal/execution/command.go (execution.DurableBackend) and
internal/execution/provider.go.
execution.Size maps to the broker’s size verbatim (small, standard,
large, xlarge); SizeDefault (empty) is sent as an omitted size.
WorkspaceSpec.Name is sent as label, WorkspaceSpec.Env as env,
WorkspaceSpec.Egress as egress, and WorkspaceSpec.Image as image
(omitted when empty).
Operations
GET /v1/lease
Read-only grant, limits and admission status for the presented token. Safe to
poll (rate limited like everything else). Creates nothing, reserves nothing.
200:
api_versionis informational. The protocol version is the/v1path prefix; clients do not branch on this field.lease.state:active|superseded(still inside its grace period). Expired, revoked or post-grace tokens never reach a200.admission.state:open|paused|emergency_stopped.pausedandemergency_stoppedadd a broker-defined"reason"string (for examplepayment,instance_paused,operator, orunavailablewhen the broker cannot read its own stop state and is failing closed). Admission conditions the broker evaluates only at create time, such as funds or budgets, are not necessarily reflected here.sizeslists only sizes this token may use, smallest first. The values shown are examples; the broker defines what each tier means.egress.ceiling:null(any valid domain allowed, up tomax_egress_domains) or a list of entries; each requested domain must equal an entry or be matched by a*.entry.
max_size) from the largest listed
size; the broker rejects anything larger either way.
POST /v1/workspaces (Acquire)
Headers: Idempotency-Key required.
Request:
Each failed check creates nothing and leaves no reservation or charge behind.
Recommended order: authenticate → idempotency replay → validate → stop and
pause state → instance concurrency → admission (for example funds or budget) →
provider create → restore. A failure after provider create terminates what was
created before answering.
201 (and on idempotent replay 200 with the same body shape and current
state):
expires_at is the hard deadline. After it every operation on the workspace
answers 410 workspace_gone. The client treats a workspace as unusable after
expires_at and acquires a new one.
Errors: 400, 401, 402 insufficient_funds, 402 budget_exhausted,
403 paused, 403 emergency_stopped, 404 not_found (snapshot),
409 idempotency_conflict, 409 idempotency_in_progress, 410 snapshot_gone,
422 size_not_allowed|image_not_allowed|egress_required|egress_not_allowed,
429 concurrency_limit, 429 risk_limit, 429 rate_limited,
502 upstream_error, 503 admission_unavailable.
GET /v1/workspaces/{workspace_id}
200:
state: starting | running | terminating | terminated.
termination_reason (when terminated): released, expired, idle,
lease_revoked, paused, emergency_stopped, orphaned, failed.
A terminated workspace’s status object may also carry metered_seconds, an
optional response field: the integer seconds the broker metered the sandbox
for, by its own clock and rounding. The client records terminated_at,
termination_reason and metered_seconds beside its own release time when
the release answers with them (below), and reports a release the broker has
not confirmed as such. Formation’s usage reporting reads these columns; the
broker’s own settlement remains the authority on what the sandbox cost.
GET /v1/workspaces?state=live&cursor=…&limit=…
Lists this instance’s workspaces. state: live (starting, running,
terminating; default) or all (adds recently terminated ones). limit 1–100,
default 50. 200: {"workspaces":[<status objects>], "next_cursor": string|null}.
POST /v1/workspaces/{workspace_id}/exec (Exec)
No idempotency key: exec is not replayable. On a transport error or
timeout the outcome is unknown and the client reports it, never retries.
Request:
argv: required, 1–4096 strings, total at most 256 KiB. Executed without a shell.dir: optional absolute path; empty = the image’s default working directory.stdin_b64: optional, decoded at mostmax_file_bytes.timeout_seconds: optional;0/omitted =default_exec_timeout_seconds. Values abovemax_exec_timeout_seconds, or beyond the workspace’sexpires_at, are clamped, and the response says so (timeout_clamped: true).
200:
exit_code is a result, not an error. Signals are folded into
128 + signal; the client classifies exit_code > 128 as
execution.KilledSignal. Each stream is truncated at max_output_bytes and
flagged; the client must surface truncation rather than present partial output
as complete. Client HTTP timeout: timeout_seconds + 30.
Errors: 404 not_found, 410 workspace_gone, 409 workspace_not_ready
(starting), 413, 502 upstream_error.
POST /v1/workspaces/{workspace_id}/files/read (ReadFile)
Request {"path": "/abs/path"} (absolute, at most 4096 bytes, no NUL; carried
in the body so it never appears in access logs). 200:
{"contents_b64": "…", "size_bytes": 123}. Errors: 404 file_not_found
(client wraps fs.ErrNotExist), 404 not_found (workspace),
410 workspace_gone, 413 file_too_large (file over max_file_bytes),
422 not_a_file.
file_not_found and not_found are distinguished only by error.code; the
client checks the code, not just the status.
POST /v1/workspaces/{workspace_id}/files/write (WriteFile)
Request {"path": "/abs/path", "contents_b64": "…"} (decoded at most
max_file_bytes). Creates parent directories and replaces the file. Naturally
idempotent. 204. Errors: 404 not_found, 410 workspace_gone, 413,
422 not_a_file.
POST /v1/workspaces/{workspace_id}/snapshots (Snapshot)
Headers: Idempotency-Key required. Request {"dir": "/abs/project/dir"}.
201 (replay 200):
expires_at = created_at + snapshot_ttl_seconds. Every snapshot has a finite
TTL, and the broker deletes it at expiry even if the client never calls
DELETE. Errors: 404, 410 workspace_gone, 422 snapshot_quota_exceeded
(live snapshots at max_snapshots; delete some first), 502 upstream_error.
DELETE /v1/snapshots/{snapshot_id} (DeleteSnapshot)
204, including when already deleted or expired. 404 not_found for ids this
instance never owned; the client treats 404 as success too, because there is
nothing for it to reclaim.
DELETE /v1/workspaces/{workspace_id} (Release)
Terminates the sandbox and stops its usage. Idempotent. 200 with the status
object (state: "terminated", or "terminating" when the provider did not
confirm in time and the broker keeps retrying). 404 not_found is success for
the client. Release never fails because of funds, budgets, a pause, an
emergency stop or a superseded token; a revoked or expired token is refused
like any other request.
The client reads the answered status object: terminated confirms the end
and the client keeps terminated_at, termination_reason and, when present,
metered_seconds; terminating and 404 are successful releases whose end
the broker did not confirm, which the client records as unconfirmed.
Durable commands
Identity: the client’s durable tool-call identity, 1–128 characters of[A-Za-z0-9._:-], unique per workspace. Command bodies are the same as exec.
-
PUT /v1/workspaces/{id}/commands/{identity}(StartCommand).202{"state": "starting"|"running"|…}. Claiming the identity is atomic: a repeatedPUTwith the same identity never launches a second process and returns the current status (200) whatever the body.422 command_not_startedmeans preflight failed before dispatch; the client wrapsexecution.ErrCommandNotStarted. A4xxthe broker answers before it claims the identity (authentication, token state, admission, an unknown or ended workspace, a malformed request) also proves nothing started. Every other failure, including timeouts and5xx, means the outcome is unknown: the client callsGET, and never relies on a laterPUTto learn the outcome. -
GET /v1/workspaces/{id}/commands/{identity}(ReadCommand).200:state:starting|running|completed|canceled|unknown. Exit fields are meaningful only forcompleted/canceled. An unknown identity answers200 {"state":"unknown"}; a gone workspace answers410 workspace_gone, which the client also maps toCommandUnknown. A state the client does not recognize is never read as finished or not started. -
POST /v1/workspaces/{id}/commands/{identity}/cancel(CancelCommand).202; idempotent; fences a delayed launch. Completed results survive.
Errors
Every non-2xx response:message is safe to show an operator and contains no secrets. Clients branch
on code only. retryable is a hint; Retry-After (seconds) is set on 429
and 503 when known. details is optional; details.reason carries a
broker-defined cause where noted.
A broker decides its own admission policy. The codes below let it say why it
refused in a way every client can act on; a broker that has no notion of funds,
budgets or risk limits simply never sends those codes.
A body that is not this envelope (for example a load balancer’s HTML
502) is
still an error; the client keys it by status and fails closed.
Idempotency and retries
Idempotency-Key: 1–200 characters of[A-Za-z0-9._:-], required onPOST /v1/workspacesandPOST …/snapshots. Scope: (instance, operation, key). Retained at least 7 days.- The broker hashes the canonical request (env values included only as a
hash). A replay with the same key and hash returns the original resource
(same
workspace_id/snapshot_id) with200and its current state; it never creates, reserves or charges twice, even across token rotation. A different hash answers409 idempotency_conflict. A replay while the original is still executing answers409 idempotency_in_progress. - A request rejected before any side effect (
4xxother than conflicts) does not consume the key: retrying with the same key re-evaluates, so a client that got402can retry after the account is funded. - Client keys are deterministic per logical operation and reused verbatim by every retry of it. Formation keys an acquisition by its workspace and snapshot, and a snapshot capture by a per-call nonce.
- Safe to retry automatically (same key where applicable):
GETrequests,POST /v1/workspaces,POST …/snapshots,DELETErequests,files/write,files/readand…/cancel. Never retried automatically:exec. An uncertainPUT …/commands/{identity}(StartCommand) outcome is recovered throughGET …/commands/{identity}; because claiming is atomic, re-sending the samePUTcannot launch twice, but it is never how a client decides a command did not run. - Retries use exponential backoff starting at 1 s with jitter, at most 5
attempts, and only for transport errors,
409 idempotency_in_progress,429 rate_limited,502and503.429 concurrency_limitand429 risk_limitclear on the scale of a run, so a client may surface them immediately instead.
Broker obligations
The client is not trusted to clean up, and its database may be lost. A conforming broker therefore:- enforces every limit in the token’s scope itself, and never silently clamps a size, image or egress request (it refuses with the codes above);
- bounds every workspace by a hard lifetime and an idle timeout that hold even if the broker’s own control loop and the client are both down (for example a provider-side sandbox timeout);
- reaps workspaces past
expires_at, workspaces of revoked, paused or stopped scope, and creates whose outcome it could not confirm; - sweeps provider-side sandboxes that have no live broker record;
- deletes snapshots at their TTL, retrying until the provider confirms;
- fails closed: when it cannot evaluate admission or stop state it refuses new
work (
503 admission_unavailable) rather than admitting it.
Client checklist
- Configuration: an
httpsendpoint (reject other schemes at the form) and an opaque token in a write-only secret slot. The connection test isGET /v1/lease. - HTTP client: TLS 1.2+ required, no redirects, per-call timeouts (exec:
timeout_seconds + 30, where an omittedtimeout_secondsmeans the lease’sdefault_exec_timeout_seconds; others 60 s), token in theAuthorizationheader only. Never log the token, request bodies, output or paths. LogX-Request-Idanderror.code. - Map every operation as in the mapping table,
including
404-as-success forReleaseandDeleteSnapshot,file_not_found→fs.ErrNotExist,command_not_started→ErrCommandNotStarted, and410 workspace_gone→CommandUnknownon read. - Derive idempotency keys before
AcquireandSnapshotand reuse them on every retry of that call. - Never send a nil egress policy: the
brokerbackend requires the install’s workspace egress mode to be allowlist. Surfaceegress_requiredas an operator-actionable error. - Treat
401,402and403codes as actionable tool errors naming the cause (token, funds, budget, paused, stopped) and who fixes it, not as transient failures. Never fall back to another backend. - Reaper:
Releasestale workspaces as for any backend; optionally listGET /v1/workspaces?state=liveand release ids with no live client record. - The client is a convenience, not a control: do not re-implement limits, pricing or admission in the client.