Skip to main content
Formation distinguishes a failed model request from a failed or uncertain external action. Retrying inference can produce an answer; repeating a tool effect or a partly delivered reply can create a duplicate. Recovery therefore happens at the specific operation boundary, with budgets and ownership checks.

​
Model requests

The model connector’s retry layer classifies failures, retries transient ones with bounded backoff, and can move to a configured fallback model. Authentication and ordinary invalid requests fail without retry. Context cancellation, ownership loss and the overall deadline stop recovery. These model-retry settings do not switch to another provider account. A custom OpenAI endpoint keeps its destination for all fallback models. Automatic model routing can also select a model and record a fallback for the run; inspect the recorded route for that run. A routed run’s attempts stay within its recorded route, so its only fallback is the one recorded there (usually the primary); MESH_MODEL_FALLBACK_MODELS applies to runs without a captured route. Automatic selection is on by default for new agents. When a request times out and another target is available, recovery moves to the next target instead of spending another full wait on the stalled one. On the last target, bounded retries can continue. Other transient failures retry the same target first. The total budget is shared across configured routes, so adding fallbacks without sufficient time does not guarantee a full attempt for each. Only the model request is retried. Completed tool calls are not rerun by this layer. Provider adapters must honor cancellation. Failed attempts and fallback changes are logged with bounded labels; raw provider error bodies are not copied into user replies or normal diagnostic strings. The run ledger records the model step, primary prepared request and aggregate result. It does not provide a separate durable wire-body record for every internal retry/fallback attempt. Logs help explain that finer sequence.

​
Tool effects and replies

Calls that may act outside Formation cross a durable effect boundary before execution. An uncertain result is not equivalent to rejection: a service may have accepted an operation even if its response was lost. Formation does not automatically replay an uncertain effect merely because a run resumed. MCP effects, browser actions and connector sends have their own failure contracts. Enabling a tool does not create idempotency support at its remote provider. A tool’s query classification affects how a failure is presented to the model; it is not permission to retry a write disguised as a read. Slack has one deliberately narrow live-send retry: an explicitly rejected HTTP 429 with a usable retry hint can retry the same rendered chunk within bounded attempts and time. Previously delivered chunks are not resent. Ambiguous transport/5xx outcomes stop delivery. There is no Slack durable chunk outbox or automatic partial-send resumption after restart. See Slack delivery recovery.

​
What a person sees

For a failed interactive turn, Formation can send a bounded failure notice, using a contextual draft when appropriate and a fixed fallback otherwise. It does not paste the raw exception or claim that nothing happened. A timeout, exhausted budget, undelivered reply or provider quota failure can produce a classified, actionable explanation. Intentional cancellation and supersession do not need an apology notice. Failure notices recheck the current audience before exposing task context. Sending a notice is itself best-effort: the connector can be unavailable too. A notice is not a promise that the task has been retried or committed to future work. Commitments have their own durable lifecycle.

​
Current limits

The earlier egress design proposed an AgentMessage/EgressError type system and an optional general-purpose semantic quality judge. Those design examples are not implemented public contracts. Current protection comes from typed provider failures, bounded retry, durable run/effect state, audience checks and controlled communication paths. Do not assume every outgoing sentence passes a model judge or that provider failures automatically create follow-up commitments. Use Observability to distinguish model failure, tool uncertainty and delivery failure. Implementation: internal/model/retry.go, internal/app/failure_message.go, and internal/connector/slack/send_recovery.go.