> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sandboxed execution

> Configure isolated execution providers and manage workspace lifecycle.

Formation runs agent shell commands and file operations in a remote workspace. The
model loop, credentials policy and run ledger remain in the Formation control plane.
A workspace belongs to one run at a time and is acquired lazily when a tool first
needs it. Runs that never execute workspace tools do not create a sandbox.

## Configure execution

1. An owner opens **Settings → Agent runtimes**, configures a supported provider
   and tests the connection. The test checks provider access; it does not prove
   a particular image can build your repository.
2. Enable the backend and set its workspace size ceiling if needed.
3. Open the agent's **Tools** page and configure the coding capabilities it needs.
   The runtime card shows the effective backend. If several backends are enabled,
   an agent can pin one; otherwise it uses the first enabled backend in catalog
   order.
4. Give the agent a small task that checks out a disposable repository, runs a
   command and reports its result. Check the run trace for acquisition or
   credential errors before assigning larger work.

| Backend | Current support | Network and persistence |
| - | - | - |
| Fly Sprites (`fly`) | Remote commands and files | Open egress; live filesystem retention for paused continuations. Project snapshot capture/restore is not implemented by this adapter. |
| Modal (`modal`) | Remote commands, files and project snapshots | Supports restricted outbound policy and immutable snapshots. |
| Execution broker (`broker`) | Remote commands, files, durable commands and project snapshots, through any service implementing the [execution broker protocol](https://docs.formation.texturehq.com/runtime/execution-broker-protocol.md) | Requires the allowlist network policy; snapshots expire on the broker's schedule. See [Execution broker](#execution-broker-broker). |

### Modal settings and bounds

Modal is configured under **Settings → Agent runtimes** (token, environment,
image) and through the operator API's agent-runtime settings, which also accept
the bounded fields below. Each field is a closed set; an empty value uses the
default. The dashboard form keeps values it does not show when it saves.

| Field | Values | Default | Effect |
| - | - | - | - |
| `max_size` | `small`, `standard`, `large`, `xlarge` | no ceiling | Largest workspace tier this install permits. |
| `sandbox_timeout_minutes` | 30, 60, 120, 240 | 60 | Hard lifetime of every sandbox. Modal ends it at this bound even if Formation is down. |
| `idle_timeout_minutes` | 10, 20, 30, 60 | 30 | Modal ends a sandbox that has run no command for this long. Never longer than the sandbox timeout. |
| `snapshot_ttl_days` | 30, 60, 90 | 30 | Modal deletes each project snapshot image this long after capture. There is no "never" value. |

Workspace tiers on Modal. Modal measures CPU in physical cores, and one core is
two vCPUs, so a Modal tier has twice the vCPUs of the same tier on Fly:

| Tier | Memory | Modal CPU | vCPU |
| - | - | - | - |
| `small` | 512 MiB | 1 core | 2 |
| `standard` | 2 GiB | 2 cores | 4 |
| `large` (current default request) | 4 GiB | 4 cores | 8 |
| `xlarge` | 8 GiB | 8 cores | 16 |

Every sandbox is created with its CPU and memory limits equal to its request.
Modal bills a running sandbox for the larger of its request and its actual use,
so a sandbox without limits can burst above its tier and be billed for it.
With limits equal to the request, the tier sets the per-second cost for the
sandbox's whole life. A build that needs more memory than its tier is
OOM-killed rather than allowed to burst; raise `max_size` if your builds need
more. The bill runs while the sandbox exists, not only while a command runs.
The idle timeout and the workspace release at the end of each run end that
bill early.

Sandboxes are tagged with identifiers only: `mesh_instance` (the deployment's
`MESH_INSTANCE_ID`, when set), `mesh_agent`, `mesh_run` and `mesh_workspace`.
The tags never carry repository names, prompts, messages or credentials. Use
them to filter Modal's sandbox list and billing reports.

Before Formation reuses a sandbox by its stored id (commands, files, snapshots), it
checks that the sandbox is running under the Formation App (`mesh-workspaces`) in
the configured environment. A sandbox that is not, or a check that fails, is
refused. Release is the exception: Formation terminates a workspace's sandbox by
its stored id without the check, so teardown (including credential
revocation) never depends on the environment setting. If you change
`environment` while a workspace is running, Formation treats that workspace as gone
for further work and still terminates it when the workspace is released. A
Modal token scoped to one environment is still the stronger boundary. This
check does not replace it.

These are the implemented backend catalog entries. Kubernetes and a local/Docker
worker are design directions, not available execution backends. The OpenAI
compatible inference adapter does not provide local shell execution.

Execution-provider credentials are instance infrastructure and can serve multiple
agents. The workspace and the agent's acting credentials remain separate.
Disabling a backend clears agents' pins to it; the save reports which agents
returned to the instance default. An otherwise stale or invalid pin fails rather
than silently selecting another backend.

## Execution broker (`broker`)

An install can run workspaces on a remote **execution broker** instead of a
Modal or Fly account the operator holds. Formation then talks only to the broker,
which owns the sandbox infrastructure, admits each workspace by its own policy,
enforces every limit and cleans up independently of Formation. The wire contract is
the open [execution broker protocol](https://docs.formation.texturehq.com/runtime/execution-broker-protocol.md) (v1);
any service that implements it works, whoever operates it. The client is
[`internal/execution/broker`](https://github.com/TextureHQ/formation/tree/main/internal/execution/broker).

Configuration is a broker **token** and the broker **endpoint**:

| Setting | Stored as | Environment (wins when set) |
| - | - | - |
| Token (opaque bearer credential) | Write-only install secret, kind `execution_broker_token` | `MESH_EXECUTION_BROKER_TOKEN` |
| Endpoint (https only) | `agent_runtimes.broker.config.endpoint` | `MESH_EXECUTION_BROKER_URL` |
| Size ceiling | `agent_runtimes.broker.config.max_size` | — |

The token authenticates exactly this instance to the broker. Its format is the
broker's business: Formation never parses it. It is sent only in the `Authorization`
header and is never logged, echoed or put in a sandbox. Whoever provisions the
instance's broker access issues and rotates it; a rotated token may keep working
for a short grace period, and because the backend is resolved again for every
tool call, the next call after the new token is installed uses it. Workspaces
and snapshots belong to the instance, so they survive rotation.

No setting here can raise a limit. Sizes, lifetime, idle timeout, command
timeout ceiling, concurrency, images, the egress ceiling, snapshot quota and any
admission policy are enforced by the broker. The size ceiling can only lower
the tier Formation asks for; a tier the token does not allow is refused, never
silently changed. A broker refuses open egress, so the install's workspace
network policy must be the allowlist mode.

When the broker refuses work, the agent's tool observation names the cause and
who fixes it:

| Broker code | Meaning for the operator |
| - | - |
| `insufficient_funds`, `budget_exhausted` | The account the instance runs under must be funded or its budget raised with the broker's operator. Retrying does not help until then. |
| `paused`, `emergency_stopped` | Execution is paused for the account or instance, or stopped by the broker's operator. |
| `lease_superseded` | The token was rotated; install the new one and the next tool call picks it up. |
| `lease_revoked`, `lease_expired`, `unauthenticated` | The broker is off for this instance until a valid token is installed. |
| `concurrency_limit`, `risk_limit` | Too many workspaces, or temporarily limited; retry later. |
| `admission_unavailable`, `unavailable`, `upstream_error` | The broker, or something it depends on, is unavailable and fails closed. |

A broker outage fails closed. Formation never falls back to another backend or
another payer. An agent pinned to `broker` gets the error, and `broker` is
last in the catalog, so enabling it does not change an unpinned agent's
default backend. Creating a workspace or snapshot carries an `Idempotency-Key`
and is retried within bounds, so a lost response never creates or reserves
twice. A command (`exec`) is never retried. Unless the broker's refusal proves
the command was never dispatched, a failed `exec` (including a 5xx a proxy
returns after dispatch) is reported as outcome unknown: the command may have run.

The connection test calls `GET /v1/lease`, which creates and reserves nothing.
It passes only for the instance's active token with open admission. With a
valid token and paused admission, the test fails with a message saying the
token itself is valid.

A control plane that provisions instances can install the token through the
generic management runtime route proposed with the management contract
(`PUT /api/management/runtimes/{backend}`, here
`PUT /api/management/runtimes/broker`) once that change merges. Until then,
the token is installed with the environment variables above or the settings
API. The dashboard has no dedicated tile for this backend yet.

## Workspace tools

`workspace_exec`, `workspace_read_file` and `workspace_write_file` operate inside
the owned workspace. GitHub and [GitLab coding](https://github.com/TextureHQ/formation/blob/main/docs/runtime/gitlab-coding.md) share the
checkout engine. Checkout configures the repository's commit identity from the
agent's provider account and prepares its work branch.

Commands use the run's limits and their own timeouts. Workspace size is selected
by runtime policy, bounded by the backend's configured ceiling; it is not an
agent-selected amount of CPU or memory. The current default requests `large`
subject to that ceiling. Automatic resizing after an out-of-memory failure is
not implemented.

Output is bounded; large eligible results can also use [scratch
handles](https://docs.formation.texturehq.com/runtime/scratch-store.md). A successful command is an observation, not
proof that later publication or delivery succeeded.

## Credentials

Coding credentials belong to the agent. Its own GitHub/GitLab PAT is eligible for
sandbox delivery by default; an operator can turn delivery off under **Coding
account access**. Existing explicit opt-outs are preserved. The grant is recorded
for the run and checked during use. Changing permissions or rotating a credential
invalidates affected active and held sandboxes and retained snapshots.

Authorized credentials reach the sandbox through a protected file, not the
provider's exec URL. Git and command-line clients can use the PAT's actual
provider permissions. A default repository is a convenience, not a security
restriction on shell commands. Long-lived PATs can be copied by code that can
use them; revoking delivery in Formation does not revoke copies at the provider.

Vault entries can be requested for an individual `workspace_exec` call by name.
Their temporary delivery and rotation behavior differs from coding-account
grants. See [workspace credentials](https://docs.formation.texturehq.com/runtime/workspace-credentials.md) for both
contracts and [agent vault](https://docs.formation.texturehq.com/runtime/agent-vault.md) for secret management.

## Network access

Workspaces have open egress by default. Owners can configure restricted egress
through the workspace-network settings. Modal and execution brokers can
enforce that policy, and the `broker` backend requires it. Fly
Sprites cannot enforce it, so a restricted deployment must reject that backend
or explicitly permit its open egress.

Filesystem/process isolation does not imply network isolation. Requested policy
must match declared backend capabilities. See the [Operator API](https://docs.formation.texturehq.com/operator-api.md#workspace-network-policy)
for the configuration surface.

## Continuing work and snapshots

Work can outlive a single conversation turn. A paused continuation may hold its
workspace for a bounded period; a resumed run can adopt that workspace after
ownership and credential checks. Fly's idle-persistent filesystem can be held
for fourteen days; other backends use the shorter one-hour handoff window.
A held workspace is not a shared working directory for unrelated tasks.

Where the backend supports snapshots, acquisition tries the same work's
checkpoint, then a validated clean repository baseline, then a cold start.
Checkpoints preserve that work's changes. Baselines speed up new work without
copying another task's branch. Snapshot capture does not publish code or replace
pushing a branch. See [snapshot isolation and lineage](https://docs.formation.texturehq.com/runtime/snapshot-lineage.md).

Completed workspaces are released. A background reaper cleans up terminal or
expired workspaces and retired snapshot artifacts using the backend recorded on
the resource, even if new acquisition through that backend is disabled.

On Modal, every snapshot image has a finite lifetime (`snapshot_ttl_days`), and
Formation records each image's expiry with the snapshot. Eight days before an image
expires, the reaper retires its snapshot so new work stops restoring it. Seven
days later the reaper deletes the image, so Formation deletes the images it created
before Modal's expiry does. The expiry is the backstop for an image whose
record Formation lost, because Modal cannot list images. A warm start never restores
an image that could expire before the sandbox ends. It falls back to the next
snapshot or a cold start instead. A paused task's checkpoint older than about
22 days (with the default 30-day lifetime) can be retired this way, and its
resume then starts from the repository baseline. Snapshots captured before
this change have no recorded expiry and keep the earlier no-expiry behavior
until Formation's normal lifecycle deletes them. Removing a backend's credential can
prevent cleanup; an unreleased resource remains recorded for operator repair. Provider outages can delay physical teardown.

## Current limits

* There is no separate compute-seconds budget; runs, commands and (on Modal)
  the sandbox and idle timeouts provide time bounds.
* The Modal bounds above are settings the install's own owners can change.
  They limit what a correctly operating Formation spends. They do not limit what
  someone holding the Modal token can spend directly.
* Snapshot support varies by backend. Fly live-workspace retention is not a
  project snapshot implementation.
* The workspace broker code exists but no coding provider uses it. Current
  GitHub/GitLab workflows deliver the agent's PAT.
* A saved or passing provider configuration is not a live workload acceptance
  test. Validate your image, repository, credentials and network policy together.

Implementation: [`internal/execution`](https://github.com/TextureHQ/formation/tree/main/internal/execution)
and [`internal/workspace`](https://github.com/TextureHQ/formation/tree/main/internal/workspace).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.