​

Health checks and the boot contract

​

The endpoints

​

Liveness — /livez, /health, /healthz

Who acts on a failed liveness check is platform-specific. A Kubernetes livenessProbe restarts the container. Fly.io, by contrast, treats a failed top-level [checks] entry as monitoring — it surfaces in fly checks list and alerting but does not restart the machine on its own. On Fly, if you need restart-on-failure, wire an explicit supervisor or alert that acts on the signal; the check alone will not do it. Either way the rule is the same: liveness is dependency-free, and routing is never gated on it.

​

Readiness — /readyz

1. Boot gate open. Boot has reached the point where this replica can answer a turn: migrations applied, agent registry loaded, listener bound, and the steady-state queue/worker loops started. The gate opens once, late in boot, and closes again the instant shutdown begins.
2. Live dependency probe passes. The database answers a bounded ping (2s), re-evaluated on every request. The ping runs on a dedicated small probe pool (a couple of connections, separate from the application pool), so /readyz answers “is PostgreSQL reachable?” rather than “does the application pool have a spare connection right now?”. Binding the probe to the application pool was rejected: under load it would report a healthy database as database: unreachable and shed a merely-busy replica from rotation, the opposite of what readiness is for. Pool saturation is an application-capacity concern to watch via pool-exhaustion metrics, not a database-reachability signal, and keeping the two distinct is the point of the separate probe pool.

• The boot gate without the live probe would keep advertising a replica that booted healthy and then lost its database in rotation.
• The live probe without the boot gate would advertise a replica whose database is reachable but whose schema has not been applied and whose registry has not loaded — a process that passes the ping and then 401s every real request.

{ "status": "unavailable", "checks": { "boot": "ok", "database": "unreachable" } }

{ "status": "ok", "checks": { "boot": "ok", "database": "ok" } }

​

The boot contract

telemetry (advisory) → logging → error tracking (advisory) → open pool
  → migrate (HARD FAIL) → load agent registry (HARD FAIL) → bind listener
  → start queue/worker loops → OPEN READINESS GATE → serve

• Unreachable database — the pool cannot open, or its ping fails.
• Failed schema/migration step — Formation’s first write on any request is a conversation-graph insert, so a boot against a database without the current schema does not degrade, it fails every request. Serving 500s from a process whose liveness says “ok” is strictly worse than not starting, so migrations run before the listener and a failure returns instead of continuing.
• Agent registry load failure — the ingress routes against the registry, so a process that cannot read the agents table would answer 401 to every Slack event. Fatal, and paged.

• Error tracking (Sentry et al.) — brought up before the pool so the first thing that can fail already has somewhere to report, but a missing DSN or a client that will not construct is logged and boot proceeds. Formation is serving correctly; it just is not reporting.
• Telemetry/tracing — same contract. A malformed or unavailable exporter never fails a boot.

​

Wiring it on your platform

[http_service]
  internal_port = 8080

  # Rotation gates on readiness: a booting or draining replica (503) stays out.
  [[http_service.checks]]
    method = "get"
    path = "/readyz"
    grace_period = "30s"   # covers the cold-start migration window
    interval = "10s"
    timeout = "5s"

[checks]
  # Liveness is a monitoring signal (shown in `fly checks list`), not a router
  # input. Fly does NOT auto-restart a machine on a failed top-level check — add
  # an explicit supervisor/alert if you need restart-on-failure. Touches no
  # dependency.
  [checks.livez]
    type = "http"
    port = 8080
    method = "get"
    path = "/livez"
    grace_period = "10s"
    interval = "15s"
    timeout = "2s"