> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Health checks and the boot contract

> Liveness vs readiness, what each endpoint promises, and the boot ordering a deploy relies on.

# Health checks and the boot contract

Formation exposes two kinds of health endpoint, and the difference between them is
load-bearing. Point your platform's checks at the right one and a bad deploy
rolls back on its own; point them at the wrong one and you get crash loops, or
worse, a replica that passes every probe while serving 500s.

## The endpoints

| Path | Kind | Question it answers | Dependencies | Use it for |
| - | - | - | - | - |
| `/livez` | Liveness | Is the process up and responsive? | None | Liveness signal / restart decisions |
| `/health` | Liveness | (same as `/livez`) | None | Liveness signal / legacy probes |
| `/healthz` | Liveness | (same as `/livez`) | None | Liveness signal / legacy probes |
| `/readyz` | Readiness | Should this replica take traffic? | Database + boot gate | Routing / load-balancer rotation |

`/health` and `/healthz` predate `/livez` and answer identically; `/livez` is
the conventional alias so a platform's liveness probe can name it without
ambiguity. (`/healthz` is kept because some managed front ends reserve it and
answer it themselves before the request reaches the container — see the note in
`newMux`. `/health` is the one guaranteed to reach Formation.)

### Liveness — `/livez`, `/health`, `/healthz`

Always `200 ok` while the process is up, with **no dependency checks at all**.
That is deliberate. Liveness is the *is-this-process-healthy* signal an
orchestrator uses to decide whether to restart, and a liveness check that pinged
the database would mark a replica unhealthy whenever the database had a blip —
on a platform that restarts on liveness failure, that turns a dependency outage,
the one failure a restart cannot fix, into a crash loop. Liveness answers
exactly one question: is this process's HTTP loop responsive? If yes, it is
healthy.

> **Who acts on a failed liveness check is platform-specific.** A Kubernetes
> `livenessProbe` restarts the container. Fly.io, by contrast, treats a failed
> top-level `[checks]` entry as *monitoring* — it surfaces in `fly checks list`
> and alerting but does **not** restart the machine on its own. On Fly, if you
> need restart-on-failure, wire an explicit supervisor or alert that acts on the
> signal; the check alone will not do it. Either way the rule is the same:
> liveness is dependency-free, and routing is never gated on it.

### Readiness — `/readyz`

`200` only when **both** of these hold, and `503` otherwise:

1. **Boot gate open.** Boot has reached the point where this replica can answer
   a turn: migrations applied, agent registry loaded, listener bound, and the
   steady-state queue/worker loops started. The gate opens once, late in boot,
   and closes again the instant shutdown begins.
2. **Live dependency probe passes.** The database answers a bounded ping (2s),
   re-evaluated on **every** request. The ping runs on a **dedicated small probe
   pool** (a couple of connections, separate from the application pool), so
   `/readyz` answers "is PostgreSQL reachable?" rather than "does the application
   pool have a spare connection right now?". Binding the probe to the application
   pool was rejected: under load it would report a healthy database as
   `database: unreachable` and shed a merely-busy replica from rotation, the
   opposite of what readiness is for. Pool saturation is an application-capacity
   concern to watch via pool-exhaustion metrics, not a database-reachability
   signal, and keeping the two distinct is the point of the separate probe pool.

Neither fact alone is readiness:

* The boot gate without the live probe would keep advertising a replica that
  booted healthy and then *lost* its database in rotation.
* The live probe without the boot gate would advertise a replica whose database
  is reachable but whose schema has not been applied and whose registry has not
  loaded — a process that passes the ping and then 401s every real request.

Readiness is the **AND** of the two. The response body names which check failed,
so a `503` is a diagnosis, not a mystery:

```json theme={null}
{ "status": "unavailable", "checks": { "boot": "ok", "database": "unreachable" } }
```

```json theme={null}
{ "status": "ok", "checks": { "boot": "ok", "database": "ok" } }
```

## The boot contract

Boot runs in a fixed order, and each step has a defined failure posture:

```
telemetry (advisory) → logging → error tracking (advisory) → open pool
  → migrate (HARD FAIL) → load agent registry (HARD FAIL) → bind listener
  → start queue/worker loops → OPEN READINESS GATE → serve
```

**Hard failures** (the process exits and never reports ready):

* **Unreachable database** — the pool cannot open, or its ping fails.
* **Failed schema/migration step** — Formation's first write on any request is a
  conversation-graph insert, so a boot against a database without the current
  schema does not degrade, it fails *every* request. Serving 500s from a process
  whose liveness says "ok" is strictly worse than not starting, so migrations
  run *before* the listener and a failure returns instead of continuing.
* **Agent registry load failure** — the ingress routes against the registry, so
  a process that cannot read the agents table would answer 401 to every Slack
  event. Fatal, and paged.

**Advisory failures** (boot continues, degraded, and says so):

* **Error tracking (Sentry et al.)** — brought up *before* the pool so the first
  thing that can fail already has somewhere to report, but a missing DSN or a
  client that will not construct is logged and boot proceeds. Formation is serving
  correctly; it just is not reporting.
* **Telemetry/tracing** — same contract. A malformed or unavailable exporter
  never fails a boot.

The readiness gate opens only *after* every hard-fail step has passed and the
listener is bound, which is why `/readyz` answers `503` for the entire
pre-serving window and `200` only once the replica can actually serve a turn.
On shutdown the gate closes *first*, before the drain, so the load balancer
routes away while in-flight turns finish rather than racing new requests into a
draining replica.

## Wiring it on your platform

Route **traffic** on `/readyz` and the **liveness signal** on `/livez`. On
Fly.io (see `fly.toml` in the repo root):

```toml theme={null}
[http_service]
  internal_port = 8080

  # Rotation gates on readiness: a booting or draining replica (503) stays out.
  [[http_service.checks]]
    method = "get"
    path = "/readyz"
    grace_period = "30s"   # covers the cold-start migration window
    interval = "10s"
    timeout = "5s"

[checks]
  # Liveness is a monitoring signal (shown in `fly checks list`), not a router
  # input. Fly does NOT auto-restart a machine on a failed top-level check — add
  # an explicit supervisor/alert if you need restart-on-failure. Touches no
  # dependency.
  [checks.livez]
    type = "http"
    port = 8080
    method = "get"
    path = "/livez"
    grace_period = "10s"
    interval = "15s"
    timeout = "2s"
```

The same split applies to any orchestrator: a Kubernetes `readinessProbe` points
at `/readyz` and a `livenessProbe` at `/livez`; a plain load balancer health
check points at `/readyz`. Set the readiness `grace_period` generously enough to
cover a cold-start migration — a slow migration must read as "not ready yet",
never as "dead".


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.