> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Execution broker protocol

> The open HTTPS protocol between Formation's broker execution backend and any remote execution broker.

Execution broker protocol v1. Status: normative for the `broker` execution
backend ([`internal/execution/broker`](https://github.com/TextureHQ/formation/tree/main/internal/execution/broker)).

This document defines the interface between:

* **the client**: Formation's `broker` execution backend, running inside a Formation
  instance whose administrator may be untrusted by the broker; and
* **the broker**: any HTTPS service that runs agent workspaces on its own
  sandbox infrastructure, admits work by its own policy, enforces every limit,
  and cleans workspaces up independently of Formation.

Formation does not know or care who operates a broker. Anyone may implement one: a
hosting provider offering managed execution, a platform team fronting its own
cluster, or a self-hosted service in front of a sandbox vendor. A Formation install
is pointed at a broker with two settings, its base URL and an opaque bearer
token (see [Sandboxed execution](https://docs.formation.texturehq.com/runtime/sandboxed-execution.md#execution-broker-broker)).

Where this document and the Formation client disagree, this document wins and the
client is the bug. A broker that conforms to this document works with Formation
without any Formation change.

## Contents

* [Trust model](#trust-model)
* [Conventions](#conventions)
* [Credential](#credential)
* [Mapping to the Formation backend contract](#mapping-to-the-mesh-backend-contract)
* [Operations](#operations)
* [Durable commands](#durable-commands)
* [Errors](#errors)
* [Idempotency and retries](#idempotency-and-retries)
* [Broker obligations](#broker-obligations)
* [Client checklist](#client-checklist)

## Trust model

* The client is **untrusted**. The person who administers a Formation instance can
  read or replace anything it stores, including the token. Every limit in this
  document is enforced by the broker; nothing may depend on the client
  behaving. The client's own checks are conveniences that fail earlier, never
  controls.
* A token authenticates exactly one client instance. The broker calls the
  grant a token carries a **lease**. Resource ownership is the client
  instance, not the token: rotating the token keeps the instance's workspaces
  and snapshots.
* The client never receives a credential for the broker's sandbox provider,
  nor that provider's sandbox, image or account identifiers, nor another
  instance's identifiers. Every id the client sees is minted by the broker and
  is meaningless anywhere else.
* An id that does not exist and an id owned by another instance are
  indistinguishable: both answer `404 not_found`.
* The broker must not log, persist or emit in metrics: argv, environment
  values, stdin, stdout/stderr, file paths, file contents or the token. The one
  path it may persist is a snapshot's directory, which restore needs; it is
  never logged or returned.

## Conventions

* **Transport**: HTTPS only (TLS 1.2+). The broker refuses plaintext. The client
  does not follow redirects; a 3xx is an error. A redirect could otherwise carry
  the `Authorization` header to a host the operator never configured.
* **Base URL**: the backend's configured `endpoint`, for example
  `https://broker.example.com`. It is an `https` URL with a host and no
  userinfo, query or fragment; it may carry a path prefix. All paths below are
  relative to it and begin with the API version `/v1`. A breaking change ships
  as `/v2`. `/v1` adds only optional response fields and new error codes, which
  clients must tolerate.
* **Bodies**: `application/json; charset=utf-8`. The broker rejects unknown
  request fields (`400 invalid_request`), so a client typo fails loudly.
  Clients ignore unknown response fields; a broker may return additional
  implementation-defined fields (for example pricing or diagnostics).
* **Attribution fields**: the request fields this document marks *attribution*
  (`agent` and `run_id` on Acquire) are optional identifiers a broker may use
  to account usage per agent; they are never a control and never content. A
  broker that implements this revision accepts them and ignores the ones it
  does not use. A broker written against an earlier revision answers `400
  invalid_request` for them; because a `400` consumes no idempotency key, the
  client then re-sends the same acquisition without them, so attribution
  degrades to absence rather than failing a workspace.
* **Binary data** (stdin, stdout, stderr, file contents) is standard base64
  with padding (RFC 4648 §4) in fields suffixed `_b64`.
* **Times** are RFC 3339 UTC strings with second precision, for example
  `2026-10-13T21:04:05Z`. **Durations** are integer seconds in fields suffixed
  `_seconds`.
* **Headers**:
  * `Authorization: Bearer <token>` on every `/v1` request.
  * `Idempotency-Key` where stated (see [Idempotency](#idempotency-and-retries)).
  * `X-Request-Id`: optional client value (at most 64 characters of
    `[A-Za-z0-9._-]`). The broker echoes it, or a generated one, on every
    response. Both sides include it in their logs.
  * `Mesh-Version`: optional client build version, for broker diagnostics only.
* **Body limit**: 12 MiB per request after base64 (`413 payload_too_large`).
* Every response carries `cache-control: no-store`.

### Identifiers

Workspace, snapshot and lease ids are minted by the broker. Each is 1 to 64
characters of `[A-Za-z0-9._-]` and is never `.` or `..`, so it is safe as a URL
path segment. Clients store ids verbatim and never parse them. A broker may use
a recognizable shape (for example `ws_` + random characters), but clients must
not rely on one.

## Credential

The client authenticates with one **opaque bearer token**, issued by the broker
(or by whoever provisions the instance on the broker's behalf).

* **Format**: defined by the broker. The client trims surrounding whitespace,
  requires a non-empty value and otherwise never parses, validates or
  transforms it. Brokers are encouraged to use a fixed, recognizable prefix so
  secret scanners can find leaked tokens, and to store only a hash of the
  secret part.
* **Client storage**: Formation stores the token in its write-only, encrypted secret
  slot for the backend (secret kind `execution_broker_token`), or reads it from
  `MESH_EXECUTION_BROKER_TOKEN`. It is sent only in the `Authorization` header,
  never in a URL, log line, sandbox environment, tool output or error message.
* **Scope**: fixed by the broker when it issues the token and readable with
  `GET /v1/lease`: allowed sizes and default size, maximum concurrent
  workspaces, maximum workspace lifetime, idle timeout, maximum exec timeout,
  snapshot TTL and count quota, allowed images, egress ceiling. The client
  cannot change scope.
* **Expiry**: a token may expire. After expiry every request answers
  `401 lease_expired`. Running workspaces continue to their own deadline and
  are reaped by the broker; a new token for the same instance can still
  control them.
* **Rotation**: issuing a new token for an instance may supersede the previous
  one. A superseded token should keep working for a short grace period so the
  new token can be installed in Formation; after it, it answers
  `401 lease_superseded`. Workspaces and snapshots belong to the instance and
  are unaffected.
* **Revocation**: immediate. Every subsequent request with that token answers
  `401 lease_revoked`. A broker may also terminate the instance's running
  workspaces on revocation.

## Mapping to the Formation backend contract

Contract source: `internal/execution/execution.go` (`execution.Backend`),
`internal/execution/command.go` (`execution.DurableBackend`) and
`internal/execution/provider.go`.

| Formation call | Broker request | Notes |
| - | - | - |
| `Provider.Test(ctx, token, config)` | `GET /v1/lease` | Read-only; creates nothing. Passes iff `200`, `lease.state == "active"` and `admission.state == "open"`. |
| `Capabilities()` | static | v1 values are fixed: `Shell=true`, `EgressControl=true`, `Resumable=false`, `IdlePersistence=false`, `Snapshots=true`. `GET /v1/lease` → `capabilities` reports the same set. |
| `Acquire(spec)` | `POST /v1/workspaces` | `Workspace.ExternalRef = workspace_id`. Warm start = `snapshot_id` in the same request. |
| `Exec(w, cmd)` | `POST /v1/workspaces/{id}/exec` | Never retried automatically. |
| `ReadFile(w, path)` | `POST /v1/workspaces/{id}/files/read` | `404 file_not_found` → error wrapping `fs.ErrNotExist`. |
| `WriteFile(w, path, b)` | `POST /v1/workspaces/{id}/files/write` | Creates parents. |
| `Snapshot(w, dir)` | `POST /v1/workspaces/{id}/snapshots` | `SnapshotRef = snapshot_id`. |
| `DeleteSnapshot(ref)` | `DELETE /v1/snapshots/{id}` | `204` and `404 not_found` are both success. |
| `Release(w)` | `DELETE /v1/workspaces/{id}` | `200` and `404 not_found` are both success. |
| `StartCommand(w, identity, cmd)` | `PUT /v1/workspaces/{id}/commands/{identity}` | `422 command_not_started` → wrap `execution.ErrCommandNotStarted`. See [Durable commands](#durable-commands). |
| `ReadCommand(w, identity)` | `GET /v1/workspaces/{id}/commands/{identity}` | `404 not_found` / `410 workspace_gone` → `CommandUnknown`. |
| `CancelCommand(w, identity)` | `POST /v1/workspaces/{id}/commands/{identity}/cancel` | Fences a delayed launch; idempotent. |
| (reaper, orphan check) | `GET /v1/workspaces?state=live` | Lets a client release broker workspaces it no longer tracks. Optional for v1 clients. |
| (status) | `GET /v1/workspaces/{id}` | Diagnostics and reconciliation. |

`execution.Size` maps to the broker's `size` verbatim (`small`, `standard`,
`large`, `xlarge`); `SizeDefault` (empty) is sent as an omitted `size`.
`WorkspaceSpec.Name` is sent as `label`, `WorkspaceSpec.Env` as `env`,
`WorkspaceSpec.Egress` as `egress`, and `WorkspaceSpec.Image` as `image`
(omitted when empty).

## Operations

### `GET /v1/lease`

Read-only grant, limits and admission status for the presented token. Safe to
poll (rate limited like everything else). Creates nothing, reserves nothing.

`200`:

```json theme={null}
{
  "api_version": "execution-broker/v1",
  "lease": {
    "id": "lse_5x2kq7m3c9a1v0b8n4d6r2t5wz",
    "instance_id": "8a1b…",
    "state": "active",
    "expires_at": "2026-10-27T21:04:05Z",
    "superseded_at": null
  },
  "admission": { "state": "open" },
  "capabilities": {
    "shell": true, "egress_control": true, "resumable": false,
    "idle_persistence": false, "snapshots": true, "durable_commands": true
  },
  "limits": {
    "max_concurrent_workspaces": 2,
    "max_lifetime_seconds": 3600,
    "idle_timeout_seconds": 900,
    "max_exec_timeout_seconds": 900,
    "default_exec_timeout_seconds": 30,
    "max_snapshots": 20,
    "snapshot_ttl_seconds": 1209600,
    "max_file_bytes": 8388608,
    "max_output_bytes": 4194304,
    "max_env_entries": 64,
    "max_egress_domains": 100
  },
  "sizes": [
    { "name": "small", "cpu_cores": 1, "memory_mib": 512 },
    { "name": "standard", "cpu_cores": 2, "memory_mib": 2048 }
  ],
  "default_size": "standard",
  "images": { "default": "buildpack-deps:bookworm", "allowed": ["buildpack-deps:bookworm"] },
  "egress": { "ceiling": null }
}
```

* `api_version` is informational. The protocol version is the `/v1` path
  prefix; clients do not branch on this field.
* `lease.state`: `active` | `superseded` (still inside its grace period).
  Expired, revoked or post-grace tokens never reach a `200`.
* `admission.state`: `open` | `paused` | `emergency_stopped`. `paused` and
  `emergency_stopped` add a broker-defined `"reason"` string (for example
  `payment`, `instance_paused`, `operator`, or `unavailable` when the broker
  cannot read its own stop state and is failing closed). Admission conditions
  the broker evaluates only at create time, such as funds or budgets, are not
  necessarily reflected here.
* `sizes` lists only sizes this token may use, smallest first. The values
  shown are examples; the broker defines what each tier means.
* `egress.ceiling`: `null` (any valid domain allowed, up to
  `max_egress_domains`) or a list of entries; each requested domain must equal
  an entry or be matched by a `*.` entry.

An operator may set Formation's install ceiling (`max_size`) from the largest listed
size; the broker rejects anything larger either way.

### `POST /v1/workspaces` (Acquire)

Headers: `Idempotency-Key` **required**.

Request:

```json theme={null}
{
  "label": "mesh-ws-6f1e…",
  "size": "standard",
  "image": "buildpack-deps:bookworm",
  "env": { "HOME": "/root" },
  "egress": { "allowed_domains": ["github.com", "*.githubusercontent.com"] },
  "snapshot_id": "snap_9f2kq7m3c9a1v0b8n4d6r2t5wx",
  "agent": "ops-bot",
  "run_id": "6f1e0d4c-2b7a-4e3f-9c1d-8a5b6c7d8e9f"
}
```

| Field | Rule |
| - | - |
| `label` | Required. At most 128 characters of `[A-Za-z0-9._:-]`. Diagnostic only; not unique; never used as the provider-side name. |
| `agent` | Optional, *attribution*: the slug of the agent the workspace serves (`^[a-z0-9][a-z0-9-]{0,63}$`). For a broker that accounts usage per agent; omitted when the client has none. |
| `run_id` | Optional, *attribution*: the UUID of the Formation run the workspace serves, the same value the client labels the vendor object with. Omitted when the client has none. |
| `size` | Optional; omitted = `default_size`. Must be in the token's `sizes`, else `422 size_not_allowed` (never silently clamped). |
| `image` | Optional; omitted = `images.default`. Must be in `images.allowed`, else `422 image_not_allowed`. |
| `env` | Optional object, at most `max_env_entries` entries; keys `^[A-Za-z_][A-Za-z0-9_]{0,127}$`; values strings; total at most 32 KiB. Passed to the sandbox at create; never persisted or logged by the broker. Must not carry credentials. |
| `egress` | **Required**. `allowed_domains` has at most `max_egress_domains` entries in Formation's `execution.ParseDomain` grammar (lowercase hostname or one leading `*.`; no IPs, ports, paths or bare `*`). `[]` means no outbound network. Missing or `null` → `422 egress_required`: a broker never runs a workspace with open egress. Entries outside `egress.ceiling` → `422 egress_not_allowed`. |
| `snapshot_id` | Optional warm start. Must be an unexpired snapshot of the same instance, else `404 not_found` / `410 snapshot_gone`. The client falls back to a cold start on either. |

Each failed check creates nothing and leaves no reservation or charge behind.
Recommended order: authenticate → idempotency replay → validate → stop and
pause state → instance concurrency → admission (for example funds or budget) →
provider create → restore. A failure after provider create terminates what was
created before answering.

`201` (and on idempotent replay `200` with the same body shape and current
`state`):

```json theme={null}
{
  "workspace_id": "ws_0kd3m9q1x7c2v5b8n4a6r2t5yz",
  "state": "running",
  "size": "standard",
  "image": "buildpack-deps:bookworm",
  "created_at": "2026-10-13T21:04:05Z",
  "expires_at": "2026-10-13T22:04:05Z",
  "idle_timeout_seconds": 900
}
```

`expires_at` is the hard deadline. After it every operation on the workspace
answers `410 workspace_gone`. The client treats a workspace as unusable after
`expires_at` and acquires a new one.

Errors: `400`, `401`, `402 insufficient_funds`, `402 budget_exhausted`,
`403 paused`, `403 emergency_stopped`, `404 not_found` (snapshot),
`409 idempotency_conflict`, `409 idempotency_in_progress`, `410 snapshot_gone`,
`422 size_not_allowed|image_not_allowed|egress_required|egress_not_allowed`,
`429 concurrency_limit`, `429 risk_limit`, `429 rate_limited`,
`502 upstream_error`, `503 admission_unavailable`.

### `GET /v1/workspaces/{workspace_id}`

`200`:

```json theme={null}
{
  "workspace_id": "ws_…", "state": "running", "size": "standard",
  "created_at": "…", "expires_at": "…", "terminated_at": null,
  "termination_reason": null
}
```

`state`: `starting` | `running` | `terminating` | `terminated`.
`termination_reason` (when terminated): `released`, `expired`, `idle`,
`lease_revoked`, `paused`, `emergency_stopped`, `orphaned`, `failed`.

A terminated workspace's status object may also carry `metered_seconds`, an
optional response field: the integer seconds the broker metered the sandbox
for, by its own clock and rounding. The client records `terminated_at`,
`termination_reason` and `metered_seconds` beside its own release time when
the release answers with them (below), and reports a release the broker has
not confirmed as such. Formation's usage reporting reads these columns; the
broker's own settlement remains the authority on what the sandbox cost.

### `GET /v1/workspaces?state=live&cursor=…&limit=…`

Lists this instance's workspaces. `state`: `live` (starting, running,
terminating; default) or `all` (adds recently terminated ones). `limit` 1–100,
default 50. `200`: `{"workspaces":[<status objects>], "next_cursor": string|null}`.

### `POST /v1/workspaces/{workspace_id}/exec` (Exec)

No idempotency key: exec is **not replayable**. On a transport error or
timeout the outcome is unknown and the client reports it, never retries.

Request:

```json theme={null}
{ "argv": ["git", "status"], "dir": "/workspace/repo", "stdin_b64": "", "timeout_seconds": 30 }
```

* `argv`: required, 1–4096 strings, total at most 256 KiB. Executed without a
  shell.
* `dir`: optional absolute path; empty = the image's default working directory.
* `stdin_b64`: optional, decoded at most `max_file_bytes`.
* `timeout_seconds`: optional; `0`/omitted = `default_exec_timeout_seconds`.
  Values above `max_exec_timeout_seconds`, or beyond the workspace's
  `expires_at`, are clamped, and the response says so (`timeout_clamped: true`).

`200`:

```json theme={null}
{
  "exit_code": 0, "stdout_b64": "…", "stderr_b64": "",
  "stdout_truncated": false, "stderr_truncated": false, "timeout_clamped": false
}
```

A non-zero `exit_code` is a result, not an error. Signals are folded into
`128 + signal`; the client classifies `exit_code > 128` as
`execution.KilledSignal`. Each stream is truncated at `max_output_bytes` and
flagged; the client must surface truncation rather than present partial output
as complete. Client HTTP timeout: `timeout_seconds + 30`.

Errors: `404 not_found`, `410 workspace_gone`, `409 workspace_not_ready`
(`starting`), `413`, `502 upstream_error`.

### `POST /v1/workspaces/{workspace_id}/files/read` (ReadFile)

Request `{"path": "/abs/path"}` (absolute, at most 4096 bytes, no NUL; carried
in the body so it never appears in access logs). `200`:
`{"contents_b64": "…", "size_bytes": 123}`. Errors: `404 file_not_found`
(client wraps `fs.ErrNotExist`), `404 not_found` (workspace),
`410 workspace_gone`, `413 file_too_large` (file over `max_file_bytes`),
`422 not_a_file`.

`file_not_found` and `not_found` are distinguished only by `error.code`; the
client checks the code, not just the status.

### `POST /v1/workspaces/{workspace_id}/files/write` (WriteFile)

Request `{"path": "/abs/path", "contents_b64": "…"}` (decoded at most
`max_file_bytes`). Creates parent directories and replaces the file. Naturally
idempotent. `204`. Errors: `404 not_found`, `410 workspace_gone`, `413`,
`422 not_a_file`.

### `POST /v1/workspaces/{workspace_id}/snapshots` (Snapshot)

Headers: `Idempotency-Key` **required**. Request `{"dir": "/abs/project/dir"}`.

`201` (replay `200`):

```json theme={null}
{ "snapshot_id": "snap_…", "created_at": "…", "expires_at": "2026-10-27T21:04:05Z" }
```

`expires_at = created_at + snapshot_ttl_seconds`. Every snapshot has a finite
TTL, and the broker deletes it at expiry even if the client never calls
`DELETE`. Errors: `404`, `410 workspace_gone`, `422 snapshot_quota_exceeded`
(live snapshots at `max_snapshots`; delete some first), `502 upstream_error`.

### `DELETE /v1/snapshots/{snapshot_id}` (DeleteSnapshot)

`204`, including when already deleted or expired. `404 not_found` for ids this
instance never owned; the client treats `404` as success too, because there is
nothing for it to reclaim.

### `DELETE /v1/workspaces/{workspace_id}` (Release)

Terminates the sandbox and stops its usage. Idempotent. `200` with the status
object (`state: "terminated"`, or `"terminating"` when the provider did not
confirm in time and the broker keeps retrying). `404 not_found` is success for
the client. Release never fails because of funds, budgets, a pause, an
emergency stop or a superseded token; a revoked or expired token is refused
like any other request.

The client reads the answered status object: `terminated` confirms the end
and the client keeps `terminated_at`, `termination_reason` and, when present,
`metered_seconds`; `terminating` and `404` are successful releases whose end
the broker did not confirm, which the client records as unconfirmed.

## Durable commands

Identity: the client's durable tool-call identity, 1–128 characters of
`[A-Za-z0-9._:-]`, unique per workspace. Command bodies are the same as exec.

* `PUT /v1/workspaces/{id}/commands/{identity}` (StartCommand). `202`
  `{"state": "starting"|"running"|…}`. Claiming the identity is atomic: a
  repeated `PUT` with the same identity never launches a second process and
  returns the current status (`200`) whatever the body. `422 command_not_started`
  means preflight failed before dispatch; the client wraps
  `execution.ErrCommandNotStarted`. A `4xx` the broker answers before it claims
  the identity (authentication, token state, admission, an unknown or ended
  workspace, a malformed request) also proves nothing started. Every other
  failure, including timeouts and `5xx`, means the outcome is unknown: the
  client calls `GET`, and never relies on a later `PUT` to learn the outcome.
* `GET /v1/workspaces/{id}/commands/{identity}` (ReadCommand). `200`:

  ```json theme={null}
  {
    "state": "completed", "exit_code": 0, "stdout_b64": "…", "stderr_b64": "…",
    "output_truncated": false, "stdout_bytes": 1234, "stderr_bytes": 0
  }
  ```

  `state`: `starting` | `running` | `completed` | `canceled` | `unknown`. Exit
  fields are meaningful only for `completed`/`canceled`. An unknown identity
  answers `200 {"state":"unknown"}`; a gone workspace answers
  `410 workspace_gone`, which the client also maps to `CommandUnknown`. A state
  the client does not recognize is never read as finished or not started.
* `POST /v1/workspaces/{id}/commands/{identity}/cancel` (CancelCommand). `202`;
  idempotent; fences a delayed launch. Completed results survive.

## Errors

Every non-2xx response:

```json theme={null}
{ "error": { "code": "insufficient_funds", "message": "Add funds to resume execution.", "retryable": false, "details": {} } }
```

`message` is safe to show an operator and contains no secrets. Clients branch
on `code` only. `retryable` is a hint; `Retry-After` (seconds) is set on `429`
and `503` when known. `details` is optional; `details.reason` carries a
broker-defined cause where noted.

A broker decides its own admission policy. The codes below let it say why it
refused in a way every client can act on; a broker that has no notion of funds,
budgets or risk limits simply never sends those codes.

| Status | `code` | Meaning / client action |
| - | - | - |
| 400 | `invalid_request` | Malformed JSON, unknown field, bad id or path. Fix the request. |
| 401 | `unauthenticated` | Missing, malformed or unknown token. |
| 401 | `lease_expired` | The token expired. A new token must be installed. |
| 401 | `lease_superseded` | The token was rotated and is past its grace period. Install the new token. |
| 401 | `lease_revoked` | The token was revoked. The broker is off for this instance until a new token is installed. |
| 402 | `insufficient_funds` | The account the instance runs under cannot cover the workspace. |
| 402 | `budget_exhausted` | A usage budget is reached. |
| 403 | `paused` | Admission is paused for the account or instance. `details.reason` is broker-defined (for example `payment`, `instance_paused`). |
| 403 | `emergency_stopped` | The broker's operator stopped execution. |
| 404 | `not_found` | Unknown id, or owned by another instance. |
| 404 | `file_not_found` | Path does not exist in the workspace. |
| 409 | `idempotency_conflict` | Key reused with a different request. |
| 409 | `idempotency_in_progress` | Original request still running; retry after `Retry-After`. |
| 409 | `workspace_not_ready` | Workspace still `starting`. Retry shortly. |
| 410 | `workspace_gone` | Terminated, expired or idle-reaped. Acquire a new workspace. |
| 410 | `snapshot_gone` | Snapshot expired or deleted. Cold start. |
| 413 | `payload_too_large`, `file_too_large` | Over a size limit. |
| 422 | `size_not_allowed`, `image_not_allowed`, `egress_required`, `egress_not_allowed`, `snapshot_quota_exceeded`, `not_a_file`, `command_not_started` | Valid JSON, but not permitted. |
| 429 | `concurrency_limit` | Instance at `max_concurrent_workspaces`. Release one or wait. |
| 429 | `risk_limit` | The broker is temporarily limiting this account (velocity, open reservations, single-request caps and so on). |
| 429 | `rate_limited` | Request-rate limit. Honour `Retry-After`. |
| 500 | `internal` | Broker bug. Retry only idempotent requests. |
| 502 | `upstream_error` | The sandbox provider failed. Idempotent requests may be retried with the same key. |
| 503 | `admission_unavailable` | The broker cannot evaluate admission or stop state and fails closed. |
| 503 | `unavailable` | Broker overloaded or draining. |

A body that is not this envelope (for example a load balancer's HTML `502`) is
still an error; the client keys it by status and fails closed.

## Idempotency and retries

* `Idempotency-Key`: 1–200 characters of `[A-Za-z0-9._:-]`, required on
  `POST /v1/workspaces` and `POST …/snapshots`. Scope: (instance, operation,
  key). Retained at least 7 days.
* The broker hashes the canonical request (env values included only as a
  hash). A replay with the same key and hash returns the original resource
  (same `workspace_id`/`snapshot_id`) with `200` and its current state; it never
  creates, reserves or charges twice, even across token rotation. A different
  hash answers `409 idempotency_conflict`. A replay while the original is still
  executing answers `409 idempotency_in_progress`.
* A request rejected before any side effect (`4xx` other than conflicts) does
  not consume the key: retrying with the same key re-evaluates, so a client that
  got `402` can retry after the account is funded.
* Client keys are deterministic per logical operation and reused verbatim by
  every retry of it. Formation keys an acquisition by its workspace and snapshot, and
  a snapshot capture by a per-call nonce.
* Safe to retry automatically (same key where applicable): `GET` requests,
  `POST /v1/workspaces`, `POST …/snapshots`, `DELETE` requests, `files/write`,
  `files/read` and `…/cancel`. Never retried automatically: `exec`. An uncertain
  `PUT …/commands/{identity}` (StartCommand) outcome is recovered through
  `GET …/commands/{identity}`; because claiming is atomic, re-sending the same
  `PUT` cannot launch twice, but it is never how a client decides a command did
  not run.
* Retries use exponential backoff starting at 1 s with jitter, at most 5
  attempts, and only for transport errors, `409 idempotency_in_progress`,
  `429 rate_limited`, `502` and `503`. `429 concurrency_limit` and
  `429 risk_limit` clear on the scale of a run, so a client may surface them
  immediately instead.

## Broker obligations

The client is not trusted to clean up, and its database may be lost. A
conforming broker therefore:

* enforces every limit in the token's scope itself, and never silently clamps
  a size, image or egress request (it refuses with the codes above);
* bounds every workspace by a hard lifetime and an idle timeout that hold even
  if the broker's own control loop and the client are both down (for example a
  provider-side sandbox timeout);
* reaps workspaces past `expires_at`, workspaces of revoked, paused or stopped
  scope, and creates whose outcome it could not confirm;
* sweeps provider-side sandboxes that have no live broker record;
* deletes snapshots at their TTL, retrying until the provider confirms;
* fails closed: when it cannot evaluate admission or stop state it refuses new
  work (`503 admission_unavailable`) rather than admitting it.

How a broker meters, prices or bills usage, and how it provisions tokens, is
outside this protocol.

## Client checklist

1. Configuration: an `https` endpoint (reject other schemes at the form) and an
   opaque token in a write-only secret slot. The connection test is
   `GET /v1/lease`.
2. HTTP client: TLS 1.2+ required, no redirects, per-call timeouts (exec:
   `timeout_seconds + 30`, where an omitted `timeout_seconds` means the
   lease's `default_exec_timeout_seconds`; others 60 s), token in the `Authorization` header
   only. Never log the token, request bodies, output or paths. Log
   `X-Request-Id` and `error.code`.
3. Map every operation as in [the mapping table](#mapping-to-the-mesh-backend-contract),
   including `404`-as-success for `Release` and `DeleteSnapshot`,
   `file_not_found` → `fs.ErrNotExist`, `command_not_started` →
   `ErrCommandNotStarted`, and `410 workspace_gone` → `CommandUnknown` on read.
4. Derive idempotency keys before `Acquire` and `Snapshot` and reuse them on
   every retry of that call.
5. Never send a nil egress policy: the `broker` backend requires the install's
   workspace egress mode to be allowlist. Surface `egress_required` as an
   operator-actionable error.
6. Treat `401`, `402` and `403` codes as actionable tool errors naming the
   cause (token, funds, budget, paused, stopped) and who fixes it, not as
   transient failures. Never fall back to another backend.
7. Reaper: `Release` stale workspaces as for any backend; optionally list
   `GET /v1/workspaces?state=live` and release ids with no live client record.
8. The client is a convenience, not a control: do not re-implement limits,
   pricing or admission in the client.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.