Sentry alert routing

​

What alerting adds

• Run inspection is the system of record — every run, model call, tool call and outcome in PostgreSQL.
• Telemetry is the latency lens — OTLP traces and logs.
• Error tracking is the grouping lens — failures deduplicated into issues in Sentry.

​

Sentry project and environments

​

Release tracking and regression detection

1. Issues in the mesh-runtime project show a non-empty Release on recent events. If they do not, the running binary was built without a VCS stamp — fix the build before trusting release-based resolution or release-based triage.
2. The release value advances when you deploy a new build. When you resolve an issue in a release, Sentry regresses it only on an event from a newer release; a release that never changes makes every recurrence look like the same unresolved issue, so the release-aware regression never fires. (A plain Resolve has no such constraint — it regresses on any later event.)

​

Sample rate

• MESH_ERROR_TRACKER_SAMPLE_RATE controls the fraction of error reports. Its default is 1 (send everything). Leave it unset or set it to 1 in production. A value below 1 means Sentry sees a fraction of incidents, which makes a first-seen or frequency condition undercount and makes “is this fixed?” unanswerable.
• MESH_ERROR_TRACKER_TRACES_SAMPLE_RATE is a separate knob for the adapter’s optional APM tracing, inert unless MESH_ERROR_TRACKER_TRACING is on. It has no effect on whether errors are delivered. Do not conflate the two: you can turn trace sampling down for cost without ever touching error delivery.

​

Alert rule specification

First-seen production errors — mind cross-environment grouping. Sentry groups events into a single issue across environments; environment is a tag on the events, not a separate grouping dimension. A new issue is created (first-seen) fires once per issue, at the moment the issue is first created in any environment. So if a failure first appears in staging, the issue already exists, and the first production occurrence just adds an event to it — it does not re-trigger first-seen, and a production-scoped new-issue alert will miss it. To keep first-seen production errors eligible, do one of the following:

• Separate the signal by project (recommended). Send production to its own Sentry project (e.g. mesh-runtime for production, mesh-runtime-staging for staging). An error’s first production occurrence is then genuinely a new issue in the production project and first-seen fires. This is the only way to get a true per-environment first-seen signal, because grouping is per project.
• Add a production-frequency trigger as a catch. If you keep one project for all environments, add a fourth OR condition — the issue is seen more than 0 times in 1 minute with the filter environment equals production — so an issue’s first production event pages even when the issue was created earlier in staging. Use 0 (not 1): more than 1 needs two matching events and would page only on the second production event, missing the first. A more than 0 threshold effectively fires on any production event, so it is noisier and can re-fire on later production activity — keep the 30 minute action interval below to damp repeats. This remains a backstop; project separation (above) is the cleaner way to get a true per-environment first-seen signal.

• environment equals production (redundant with the scope but explicit).
• Set Action interval to 30 minutes so one noisy issue cannot repeat-page within the window; the new/regressed conditions still fire immediately for distinct issues.

​

Slack destination

​

Message template

:rotating_light: *mesh-runtime* — {{ issue.title }}
env: {{ environment }} · release: {{ release.version }}
{{ issue.url }}

​

Ownership and triage SLAs (beta)

1. Acknowledge in the Slack thread so the rest of the team knows it is owned.
2. Open the production event that caused the alert. Do not just read the issue’s first event — if the issue began in staging, its first event carries staging context and release, which would route you to the wrong build. Open the specific event the alert fired on (the alert/Slack card links straight to it), or filter the issue’s events to environment:production and read the latest one. Use that event’s environment and release when identifying the build to investigate.
3. Follow it to the run. The error carries bounded diagnostics and routing identifiers — conversation key, event id, agent slug — not conversation bodies. Use those identifiers to open the run in Observability, which is the system of record for what actually happened.
4. Classify execution vs delivery: a failed model call is execution; a failed send can mean an answer exists but did not arrive. See egress reliability. They route to different fixes.
5. Resolve in Sentry when fixed. A plain Resolve will regress the issue on any later event. If you resolve in a release (release-aware), do it in a build whose release advances — otherwise a recurrence in a non-advancing release is not flagged as a regression.

​

Backstop

​

What a human still has to do

1. Confirm the mesh-runtime project exists and is receiving production events with a non-empty release.
2. Install the Sentry ↔ Slack integration and pick the #mesh-alerts channel.
3. Create the issue alert exactly as specified under Alert rule specification, scoped to production.
4. Optionally create the staging-scoped, non-paging twin on its own channel.
5. Record the rotation that fills the owner role in the SLA table.