> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Sentry alert routing

> Route new and regressed runtime errors to Slack, and triage them on a beta SLA.

[Error tracking](https://docs.formation.texturehq.com/error-tracking.md) explains how Formation *reports* failures to Sentry.
This runbook explains what to do once they are there: which issues page a human,
where that notification lands, and how fast someone is expected to respond during
beta.

Formation does not ship alert configuration as code. There is no Sentry Terraform
provider, dashboard-as-code file, or in-repo rule definition in this repository —
the reporting adapter lives in
[`internal/errtrack`](https://github.com/TextureHQ/formation/tree/main/internal/errtrack)
and stops at delivering events. The alert rule below is applied once, by hand, by
someone with Sentry project-admin access. This document is the source of truth for
*what* that rule says so it can be audited and recreated without guesswork.

## What alerting adds

The runtime already has three operational lenses, and alerting is deliberately a
fourth that sits on top of them rather than replacing any:

* [Run inspection](https://docs.formation.texturehq.com/runtime/observability.md) is the system of record — every run,
  model call, tool call and outcome in PostgreSQL.
* [Telemetry](https://docs.formation.texturehq.com/telemetry.md) is the latency lens — OTLP traces and logs.
* [Error tracking](https://docs.formation.texturehq.com/error-tracking.md) is the grouping lens — failures deduplicated
  into issues in Sentry.

Alerting is the *push* on top of the grouping lens: it decides which of those
grouped issues is worth interrupting a human for, and sends that interruption to
a place humans are already looking. Everything an alert links to is answered by
the three lenses above; the alert's only job is to get someone there in time.

## Sentry project and environments

One Sentry project receives runtime errors. Refer to it by the name your
organization chose for it — this document uses **`mesh-runtime`** as the project
slug. Events arrive tagged with the name of the server component (`mesh`) and with
an environment.

The environment tag is the value of `MESH_ENV`, threaded straight through from the
process configuration to the Sentry event — the same value telemetry uses. It is
not a separate Sentry setting and it must not be configured independently, or the
two lenses will disagree about which deployment an error came from.

| `MESH_ENV` | Sentry `environment` | What it is |
| - | - | - |
| `production` | `production` | Live installation serving real conversations. Alerts page. |
| `staging` | `staging` | Pre-production verification. Alerts are informational, not paging. |
| `development` | `development` | Local and CI. Off the alerting path entirely. |

`MESH_ENV` defaults to `development` when unset, so an installation that forgot to
set it will report as `development` and correctly stay out of the paging path.
Set it explicitly on every deployed installation. See
[Installation](https://docs.formation.texturehq.com/operations/installation.md) for where it is set.

Scope the paging alert rule to `environment:production`. Create the same rule a
second time scoped to `staging` only if you want a separate, non-paging channel
for pre-production noise; do not let staging share the production channel, because
the whole point of the production channel is that a message in it means "a real
conversation just broke."

## Release tracking and regression detection

A "regression" alert fires when an issue Sentry had marked resolved receives a
new event and flips back to unresolved. A plain **Resolve** regresses on *any*
later event — no release metadata required. The release machinery below matters
only for **release-aware** resolution (resolving *in the next release* or *in a
specific release*), where Sentry regresses the issue only when an event from a
newer release arrives, and for identifying which build an event came from.

Formation stamps the release automatically from the binary's embedded VCS revision
(`vcs.revision`), with no deploy-time argument to thread through. A clean `go
build` or the repository's image workflow stamps it; a `go run`, a dirty tree, or
a container stage with no `.git` leaves it empty, which degrades grouping rather
than failing anything. For reproducible deployments, pin an image digest or a
commit tag as described in [Installation](https://docs.formation.texturehq.com/operations/installation.md) so each deploy
maps to a distinct release.

Confirm, before relying on release-aware resolution or on build identification
from an event, that:

1. Issues in the `mesh-runtime` project show a non-empty **Release** on recent
   events. If they do not, the running binary was built without a VCS stamp —
   fix the build before trusting release-based resolution or release-based triage.
2. The release value advances when you deploy a new build. When you resolve an
   issue *in a release*, Sentry regresses it only on an event from a newer
   release; a release that never changes makes every recurrence look like the
   same unresolved issue, so the release-aware regression never fires. (A plain
   Resolve has no such constraint — it regresses on any later event.)

Do not set `SENTRY_RELEASE` to pin or override this. Formation resolves the release
itself and passes it to the SDK; leaving the SDK to derive its own would let two
sources disagree. See
[`internal/errtrack`](https://github.com/TextureHQ/formation/tree/main/internal/errtrack)
for the resolution logic.

## Sample rate

Regression detection and frequency thresholds are only trustworthy if production
is not silently dropping error events. Keep the **error** sample rate at `1.0` in
production so every reported incident reaches Sentry.

* `MESH_ERROR_TRACKER_SAMPLE_RATE` controls the fraction of *error* reports. Its
  default is `1` (send everything). Leave it unset or set it to `1` in production.
  A value below `1` means Sentry sees a fraction of incidents, which makes a
  first-seen or frequency condition undercount and makes "is this fixed?" unanswerable.
* `MESH_ERROR_TRACKER_TRACES_SAMPLE_RATE` is a **separate** knob for the adapter's
  optional APM *tracing*, inert unless `MESH_ERROR_TRACKER_TRACING` is on. It has
  no effect on whether errors are delivered. Do not conflate the two: you can turn
  trace sampling down for cost without ever touching error delivery.

Both variables are documented in [Error tracking](https://docs.formation.texturehq.com/error-tracking.md). When the error
tracker section is environment-managed (`MESH_ERROR_TRACKER` set), these values
come from the environment and the dashboard refuses to change them.

## Alert rule specification

Create one **issue alert** on the `mesh-runtime` project, scoped to the production
environment. Apply it by hand in **Alerts → Create Alert → Issues** and keep this
spec as its definition of record.

**Name:** `mesh-runtime: new or regressed error (production)`

**Environment:** `production`

**Trigger — fire when any of these is true (`OR`):**

| Condition | Sentry condition |
| - | - |
| A brand-new issue appears | *A new issue is created* (first-seen) |
| A resolved issue comes back | *The issue changes state from resolved to unresolved* (regression) |
| An existing issue spikes | *The issue is seen more than* **`25`** *times in* **`1 hour`** |

The first two conditions are the core of the rule: a genuinely new failure mode,
or a bug you thought you fixed reappearing in a new release. The frequency
condition is the backstop for an issue that was already known and un-resolved but
suddenly starts happening far more often — tune the `25 / 1 hour` threshold to
your traffic once you have a baseline; start conservative so it does not drown the
new/regressed signal.

> **First-seen production errors — mind cross-environment grouping.** Sentry
> groups events into a single issue *across* environments; `environment` is a tag
> on the events, not a separate grouping dimension. *A new issue is created*
> (first-seen) fires once per issue, at the moment the issue is first created in
> **any** environment. So if a failure first appears in staging, the issue already
> exists, and the first *production* occurrence just adds an event to it — it does
> **not** re-trigger first-seen, and a production-scoped new-issue alert will miss
> it. To keep first-seen production errors eligible, do one of the following:
>
> * **Separate the signal by project (recommended).** Send production to its own
>   Sentry project (e.g. `mesh-runtime` for production, `mesh-runtime-staging` for
>   staging). An error's first production occurrence is then genuinely a new issue
>   in the production project and first-seen fires. This is the only way to get a
>   true per-environment first-seen signal, because grouping is per project.
> * **Add a production-frequency trigger as a catch.** If you keep one project for
>   all environments, add a fourth `OR` condition — *the issue is seen more than*
>   **`0`** *times in* **`1 minute`** with the filter `environment equals
>   production` — so an issue's first production event pages even when the issue
>   was created earlier in staging. Use **`0`** (not `1`): *more than 1* needs two
>   matching events and would page only on the **second** production event,
>   missing the first. A `more than 0` threshold effectively fires on any
>   production event, so it is noisier and can re-fire on later production
>   activity — keep the `30 minute` action interval below to damp repeats. This
>   remains a backstop; project separation (above) is the cleaner way to get a
>   true per-environment first-seen signal.

**Filters (optional, recommended):**

* `environment equals production` (redundant with the scope but explicit).
* Set **Action interval** to `30 minutes` so one noisy issue cannot repeat-page
  within the window; the new/regressed conditions still fire immediately for
  distinct issues.

**Action:** Send a Slack notification to the chosen channel (below).

### Slack destination

Route to a dedicated, low-volume channel so a message there is always meaningful.
This document uses **`#mesh-alerts`**; substitute your workspace's channel. The
Sentry Slack integration must be installed and the channel selected in the alert
action — Formation does not post to Slack itself, Sentry does.

Keep a separate `#mesh-alerts-staging` (or similar) for the optional staging rule.
Production and staging must not share a channel.

### Message template

Sentry's Slack action renders the issue card automatically (title, culprit,
environment, release, event count, and a link). Keep the human-authored portion
short and action-oriented:

```
:rotating_light: *mesh-runtime* — {{ issue.title }}
env: {{ environment }} · release: {{ release.version }}
{{ issue.url }}
```

If your Sentry plan does not support templated action text, the default issue card
is sufficient — it already carries environment, release and the link, which is
everything the triage step below needs.

## Ownership and triage SLAs (beta)

Every issue in the `mesh-runtime` project has one on-call owner at a time. During
beta the owner is whoever is on the runtime on-call rotation; if no formal rotation
exists yet, it defaults to the runtime maintainer. The owner acknowledges and
triages — they do not have to fix it themselves, but they own routing it until it
is resolved or reassigned.

Severity is assigned by the owner at acknowledgement, from the issue and the run
it links to:

| Severity | Definition | Acknowledge within | Mitigate or route within |
| - | - | - | - |
| **P1** | Conversations are broken for users: turns failing, deliveries failing, or boot failing in production. | **15 minutes** (business hours), 30 min off-hours | **1 hour** |
| **P2** | A real failure that does not block most conversations: one connector or model path degraded, elevated but non-total error rate. | **1 business hour** | **1 business day** |
| **P3** | Low-impact or self-recovering: a rare edge case, a noisy-but-harmless report, an expected transient. | **1 business day** | Backlog / next planned work |

Business hours and the exact rotation are an operational detail for your team to
set; the table is the contract for what each severity promises.

**Triage steps for any alert:**

1. **Acknowledge in the Slack thread** so the rest of the team knows it is owned.
2. **Open the production event that caused the alert.** Do not just read the
   issue's first event — if the issue began in staging, its first event carries
   staging context and release, which would route you to the wrong build. Open the
   specific event the alert fired on (the alert/Slack card links straight to it),
   or filter the issue's events to `environment:production` and read the latest
   one. Use *that* event's environment and release when identifying the build to
   investigate.
3. **Follow it to the run.** The error carries bounded diagnostics and routing
   identifiers — conversation key, event id, agent slug — not conversation bodies.
   Use those identifiers to open the run in [Observability](https://docs.formation.texturehq.com/runtime/observability.md),
   which is the system of record for what actually happened.
4. **Classify** execution vs delivery: a failed model call is execution; a failed
   send can mean an answer exists but did not arrive. See
   [egress reliability](https://docs.formation.texturehq.com/runtime/egress-reliability.md). They route to different fixes.
5. **Resolve in Sentry** when fixed. A plain Resolve will regress the issue on any
   later event. If you resolve *in a release* (release-aware), do it in a build
   whose release advances — otherwise a recurrence in a non-advancing release is
   not flagged as a regression.

### Backstop

A lightweight external poller may also watch the `mesh-runtime` project as a
belt-and-suspenders signal in case the Sentry Slack integration itself is down.
It is a backstop, not the primary path: the Sentry-native rule above is the
documented, auditable source of alerts. If both fire, dedupe on the issue link.

## What a human still has to do

This repository cannot create the alert. A person with Sentry project-admin access
must, once:

1. Confirm the `mesh-runtime` project exists and is receiving production events
   with a non-empty release.
2. Install the Sentry ↔ Slack integration and pick the `#mesh-alerts` channel.
3. Create the issue alert exactly as specified under
   [Alert rule specification](#alert-rule-specification), scoped to `production`.
4. Optionally create the staging-scoped, non-paging twin on its own channel.
5. Record the rotation that fills the **owner** role in the SLA table.

Re-run steps 1–3 against this document whenever the rule is changed, so the live
configuration and the runbook never drift.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.