Skip to main content
The master key, its custody, its rotation, and the drill that proves a backup restores. This is the runbook for the two failure narratives roadmap/tracks.md §E4 (repository) names: the key leaks and there is no rotation path, or the key is lost and every ciphertext is permanently opaque.

​
What is encrypted, and with what

Integration credentials Formation holds — Slack bot tokens and signing secrets, Telegram tokens, model-provider keys, GitHub and Linear tokens, the error tracker’s DSN, memory-provider keys, agent-runtime tokens — is a row in the secrets table, sealed with AES-256-GCM under one master key. The key comes from MESH_AGENT_MASTER_KEY in the process environment and is never written to the database; ciphertext and key must have different blast radii, or a database dump is plaintext. Agent provisioning explains the setup boundary. The agent vault (agent_vault_entries, runtime/agent-vault.md) is a second table under the same master key: the secrets people hand an agent for its own work, as opposed to the infrastructure credentials above. Its rows are sealed under a subkey derived from the master key with HKDF (internal/secrets.Derive), so the two tables’ ciphertexts are mutually unreadable even though one key protects both, and they are tagged with the master key’s fingerprint so everything below — the boot check, rotate-key, verify-secrets — treats the vault as one more table under one key. The reports list it as owner type agent_vault. Encrypted renewal tokens in agent_vault_drop_tokens use the same vault subkey with agent/drop identity as AAD, and appear as kind drop_token in those reports. Rotating the key preserves the URLs that interrupted renewal runs will recover. Each row carries a key version tag. Rows written before rotation existed carry the legacy tag v1, which says nothing about which key wrote them. Every row written since carries the writing key’s fingerprint: sha256: plus the first 16 hex characters of the SHA-256 of the key material. A fingerprint names a key without revealing it. The secret_keys row named agent:credentials — the descriptor — carries the fingerprint of the key that last wrote or rotated the table, and it is what boot reads first. Pending MCP OAuth authorizations (mcp_oauth_authorizations) also contain encrypted state, including the PKCE verifier and client details, which may include a client secret. Authorization attempts expire after 10 minutes; rows are deleted when consumed or swept on a later authorization start. These rows are excluded from rotate-key, verify-secrets, and their reports. Restart pending authorization flows after rotating the key. A clean verification result does not validate pending OAuth flows.

​
Key custody

  • Where the key lives: in the deployment platform’s secret store, injected as MESH_AGENT_MASTER_KEY into the process environment. Nowhere in the repository, nowhere in the database, nowhere in a backup of the database.
  • Who can read it: whoever can read the deployment’s secrets. That set is the set of people who can read every credential Formation holds; keep it small and written down for your deployment.
  • Where it is backed up: the key must be held in at least one place that is not the deployment platform — a password manager entry or an offline record — because a lost key makes every backup worthless. The database backup and the key backup must never be the same artifact.
  • What the process does about a wrong key: refuses to boot. If the descriptor names a fingerprint and the process was handed a different key, mesh exits with one line naming both fingerprints and the remedy, before the agent loader touches a single credential. A legacy v1 descriptor cannot be judged that way; the loader’s per-credential decrypt is the only test for it.

​
The commands

Both are subcommands of the mesh binary, run against DATABASE_URL with no other serving configuration, like mesh migrate. Neither prints a key or a plaintext; fingerprints and counts are the whole output.
Decrypts every row in secrets, agent_vault_entries, and agent_vault_drop_tokens with the key and prints, per owner type and kind, how many opened and how many did not, plus whether the descriptor names this key. Exits non-zero if anything failed to open or the descriptor names another key. It writes nothing and takes no locks.
Re-encrypts every row in those three stores from the current key to the next in one transaction, then moves the descriptor. Properties worth knowing:
  • Atomic. The table is never half-rotated. Any row that cannot be opened with the current key (or, see below, the next) stops the run and rolls everything back; a row it cannot read is a row it must not overwrite.
  • Locked. The transaction takes an EXCLUSIVE lock on secrets, secret_keys, agent_vault_entries, and agent_vault_drop_tokens for its duration: readers proceed (a live process keeps decrypting), writers wait. A concurrent dashboard save cannot insert a row the rotation never saw; it lands after the commit, under whichever key that process holds, where the boot check will see it.
  • Idempotent, without trusting tags. A row tagged with the next key is skipped only after it actually opens with the next key; a mis-tagged or corrupt row is refused, not skipped. So a run interrupted by a lost connection is simply run again.
  • Handles the restart race. A row written under the next key by a process that predates the rotation but tagged v1 is restamped, not refused.
  • Refuses the no-op. The same key as current and next is an error.
  • --dry-run computes the full report and commits nothing.

​
Runbook: rotating the master key

The serving process reads with one key and does not accept two at once, so a rotation is a restart. The clean sequence is stop, rotate, start with the next key: downtime is the stop-to-start interval, and the rotation itself is one transaction over a small table, so that interval is dominated by the restart. Rotating while the process is still serving is possible — the table lock makes writers wait rather than slip past — but any credential saved after the rotation commits and before the restart is encrypted under the old key, and the process will then refuse to boot with the next key until rotate-key is run again. Stopping first removes that step; the boot check makes forgetting it safe rather than silent. The master key also keys the query digests in two audit tables, memory_retrieval_events and conversation_search_events (internal/auditdigest), so a query cannot be recovered from them by hashing guesses. Those digests are not re-encrypted by rotate-key, because they are one-way: rows written before a rotation stay comparable only with each other, and rows after it start a new digest space. Nothing in Formation reads them to make a decision.
  1. Generate the next key. Any high-entropy string works; 32 random bytes, base64-encoded, is the shape the code normalizes to.
  2. Store the next key in the secret store before rotating, under a name that distinguishes it from the current one. A rotation whose next key is lost is the worst outcome in this document.
  3. Stop the serving process (scale to zero, or stop the container). Skipping this is allowed; see above for what it costs.
  4. Dry-run from an operator shell against the production database:
    Read the per-kind counts. They are the inventory of what will move.
  5. Rotate:
  6. Update the deployment’s MESH_AGENT_MASTER_KEY to the next key and start the process. Do not set MESH_AGENT_MASTER_KEY_NEXT on the serving process; the service never reads it.
  7. Verify from the operator shell with the next key:
  8. Remove the old key from active deployment configuration once verification is clean. Retain a protected recovery copy while any retained backup still contains ciphertext encrypted under it. Label key backups by fingerprint so a restore can select the matching key.
If boot fails at step 6 with “MESH_AGENT_MASTER_KEY is not the key the secrets table is encrypted under”: step 3 was skipped and a credential was saved under the old key between steps 5 and 6. Run step 5 again with the same current and next keys — it rotates the straggler and skips the rest — then start again. If the old key has leaked, rotation is step one of the compromise runbook below, not the whole of it.

​
Runbook: backup and restore, with the drill

The database is the system of record for every credential, every memory, and the whole conversation graph. The key is not in it. A backup therefore restores only if the key that encrypted it is still held; the drill proves both halves together. Backup. Use the platform’s point-in-time recovery where it exists, and take a logical dump on the same schedule as a belt-and-braces artifact that can be restored anywhere:
Credential ciphertext in secrets, agent_vault_entries, agent_vault_drop_tokens, and mcp_oauth_authorizations remains encrypted; the dump does not contain the master key. Conversation bodies, memories, prompt snapshots and model responses are stored as plaintext. Write the dump with restrictive permissions, keep it in encrypted backup storage, and give it the same access controls as the production database; a dump is the database. Restore drill. Quarterly, and after any change to the secrets layer:
  1. Create a scratch database and restore the most recent dump into it.
  2. Run mesh verify-secrets against the scratch database with the production key. Every checked row must open and the descriptor must match.
  3. Run it again with a deliberately wrong key. Every checked row must fail and the descriptor must be reported as not matching. This proves the check is real.
  4. Optionally rehearse rotation on the scratch copy (rotate-key to a throwaway key, verify-secrets with it, then confirm a boot with the old key against the scratch copy is refused).
  5. Drop the scratch database. Record the date below.
The next drill is due by 2026-12-10. The most recent date is also written in the README so a reader who never opens this page still sees it.

​
Runbook: credential compromise

What to do, in order, when a credential Formation holds has leaked. If the master key leaked: everything it protected is exposed. Rotate the master key first (runbook above), because until then every credential you re-issue is stored under a key the attacker holds. Then treat every credential in the table as compromised and work down the inventory rotate-key printed:
  1. Slack: regenerate each connector’s signing secret and reinstall the app to rotate its bot token; paste both through the connector settings.
  2. Telegram: revoke and re-issue each bot token with BotFather; paste it.
  3. Model providers: revoke each key in the provider console; paste the new one per agent, then the deployment fallback if one is set.
  4. GitHub and Linear: revoke the agent accounts’ tokens in those systems and capture new ones on the agent’s tools tab.
  5. Error tracker and memory providers: rotate at the vendor, paste in settings.
If one credential leaked: revoke it at the issuer and paste the replacement. The master key is unaffected and need not rotate. If the master key is lost: first recover the matching key from its separate backup. A new key cannot decrypt existing integration credentials, vault entries, or renewal tokens. Preserve an encrypted database backup and inventory before attempting recovery. Formation has no automatic lost-key reset command: simply deleting secrets rows leaves the key descriptor and vault state inconsistent. If no matching key survives, provision a fresh credential store/installation and re-enter credentials and vault values from their original sources, with a reviewed migration of any recoverable non-secret state. Do not destroy the only copy of the old database while establishing that recovery.

​
The content KEK (separate key, not yet in use)

Customer content — message bodies, memory, prompts, tool results — is not sealed under the master key and, today, is not sealed at all (see the content protection design, backlog L03/L14). The foundation for sealing it ships in internal/contentcrypt, wired to no writer yet, under a second, independent key:
  • MESH_CONTENT_KEK is a key-encryption key: 32 random bytes, base64 or hex. It wraps per-instance data keys, which are stored only wrapped in content_keys; content_key_descriptor pins the database to its instance (MESH_INSTANCE_ID, bound into every envelope) and to the KEK’s fingerprint.
  • It is never MESH_AGENT_MASTER_KEY (the commands refuse that), is rotated by its own command, and needs its own escrow copy, labelled by fingerprint. Losing it makes every sealed value unreadable; recovering it is restoring the escrow copy. Formation never generates a KEK: content keys with no KEK is a recovery error, never a fresh install.
Between the rewrap’s commit and a replica’s restart, that replica keeps sealing and opening with the data keys it already unwrapped (the data keys themselves do not change), but it cannot mint one: a new key is stored only when the descriptor names the KEK that wrapped it, so a stale replica’s mint fails with the wrong-KEK error instead of writing a key the restarted replicas cannot unwrap. A restore drill for content is the database plus its matching KEK, under the original instance ID, followed by verify-content-keys. Custody (Fly secret plus control-plane escrow, with a later move to a cloud KMS by rewrapping), the seal budget and key age are proposals awaiting owner decisions (Q09, Q11).

​
What is not built

  • Dual-key read. The serving process reads with one key; rotation is a restart. The tags are load-bearing enough that a process accepting a MESH_AGENT_MASTER_KEY_PREVIOUS for reads is a bounded follow-on once zero-restart rotation matters (more than one replica).
  • KMS or Vault as the key provider. secret_keys.provider admits kms and vault; only env is implemented.
  • Automated drill scheduling. The quarterly cadence is a calendar entry.