> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Retries and delivery failures

> Understand model retries, fallback selection, and connector delivery outcomes.

Formation distinguishes a failed model request from a failed or uncertain external
action. Retrying inference can produce an answer; repeating a tool effect or a
partly delivered reply can create a duplicate. Recovery therefore happens at the
specific operation boundary, with budgets and ownership checks.

## Model requests

The model connector's retry layer classifies failures, retries transient ones
with bounded backoff, and can move to a configured fallback model. Authentication
and ordinary invalid requests fail without retry. Context cancellation, ownership
loss and the overall deadline stop recovery.

| Process setting | Default | Meaning |
| - | - | - |
| `MESH_MODEL_RETRY_MAX_ATTEMPTS` | `3` | Attempts per model target; `1` disables repeat attempts, not deadlines. |
| `MESH_MODEL_RETRY_ATTEMPT_TIMEOUT_MS` | `300000` | Bound for an in-flight completion. `0` disables this bound; other deadlines still apply. Sized so a strong reasoning model's single slow completion on a large context is not cut off into a re-drive loop. |
| `MESH_MODEL_RETRY_TOTAL_DEADLINE_MS` | `600000` | Total recovery time, including backoff. Fits a slow completion plus a retry or fallback route. |
| `MESH_MODEL_RETRY_MAX_DELAY_MS` | `20000` | Maximum retry wait, including provider hints. |
| `MESH_MODEL_FALLBACK_MODELS` | Empty | Ordered fallback model IDs on the agent's existing connector and credentials. |

These model-retry settings do not switch to another provider account. A custom
OpenAI endpoint keeps its destination for all fallback models. Automatic
[model routing](https://docs.formation.texturehq.com/runtime/adaptive-model-routing.md) can also select a model and
record a fallback for the run; inspect the recorded route for that run. A
routed run's attempts stay within its recorded route, so its only fallback is
the one recorded there (usually the primary); `MESH_MODEL_FALLBACK_MODELS`
applies to runs without a captured route. Automatic selection is on by default for new agents.

When a request times out and another target is available, recovery moves to the
next target instead of spending another full wait on the stalled one. On the
last target, bounded retries can continue. Other transient failures retry the
same target first. The total budget is shared across configured routes, so adding
fallbacks without sufficient time does not guarantee a full attempt for each.

Only the model request is retried. Completed tool calls are not rerun by this
layer. Provider adapters must honor cancellation. Failed attempts and fallback
changes are logged with bounded labels; raw provider error bodies are not copied
into user replies or normal diagnostic strings.

The run ledger records the model step, primary prepared request and aggregate
result. It does not provide a separate durable wire-body record for every
internal retry/fallback attempt. Logs help explain that finer sequence.

## Tool effects and replies

Calls that may act outside Formation cross a durable effect boundary before execution.
An uncertain result is not equivalent to rejection: a service may have accepted
an operation even if its response was lost. Formation does not automatically replay
an uncertain effect merely because a run resumed.

MCP effects, browser actions and connector sends have their own failure contracts.
Enabling a tool does not create idempotency support at its remote provider.
A tool's query classification affects how a failure is presented to the model;
it is not permission to retry a write disguised as a read.

Slack has one deliberately narrow live-send retry: an explicitly rejected HTTP
429 with a usable retry hint can retry the same rendered chunk within bounded
attempts and time. Previously delivered chunks are not resent. Ambiguous
transport/5xx outcomes stop delivery. There is no Slack durable chunk outbox or
automatic partial-send resumption after restart. See [Slack delivery
recovery](https://github.com/TextureHQ/formation/blob/main/docs/runtime/slack-delivery-recovery.md).

## What a person sees

For a failed interactive turn, Formation can send a bounded failure notice, using a
contextual draft when appropriate and a fixed fallback otherwise. It does not
paste the raw exception or claim that nothing happened. A timeout, exhausted
budget, undelivered reply or provider quota failure can produce a classified,
actionable explanation. Intentional cancellation and supersession do not need
an apology notice.

Failure notices recheck the current audience before exposing task context.
Sending a notice is itself best-effort: the connector can be unavailable too.
A notice is not a promise that the task has been retried or committed to future
work. [Commitments](https://docs.formation.texturehq.com/runtime/commitments.md) have their own durable lifecycle.

## Current limits

The earlier egress design proposed an `AgentMessage`/`EgressError` type system and
an optional general-purpose semantic quality judge. Those design examples are
not implemented public contracts. Current protection comes from typed provider
failures, bounded retry, durable run/effect state, audience checks and controlled
communication paths. Do not assume every outgoing sentence passes a model judge
or that provider failures automatically create follow-up commitments.

Use [Observability](https://docs.formation.texturehq.com/runtime/observability.md) to distinguish model failure, tool
uncertainty and delivery failure. Implementation:
[`internal/model/retry.go`](https://github.com/TextureHQ/formation/blob/main/internal/model/retry.go),
[`internal/app/failure_message.go`](https://github.com/TextureHQ/formation/blob/main/internal/app/failure_message.go),
and [`internal/connector/slack/send_recovery.go`](https://github.com/TextureHQ/formation/blob/main/internal/connector/slack/send_recovery.go).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.