> ## Documentation Index
> Fetch the complete documentation index at: https://docs.formation.texturehq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Content protection design

> Inventory of stored customer content, threat models, and the proposed encryption, key custody, and retrieval design for hosted Formation (backlog L03).

> **Status: proposal.** Nothing in this document is implemented, and nothing
> here changes runtime behavior. Every recommendation is a proposal that needs
> the owner decisions listed in [Decisions needed](#decisions-needed): Q01
> (alpha scope), Q09 (promise scope, key custody, and search leakage), Q10
> (support consent), and Q11 (retention, backups, and RPO/RTO). The questions
> are defined in the control plane's `docs/hosted-launch-strategy.md`. This
> document answers backlog item L03 in the control plane's
> `docs/hosted-launch-backlog.md`. It is the input to L14, L15, and L16.

[Access control](https://docs.formation.texturehq.com/access-control.md) states the current position: credentials
are encrypted, while conversation bodies, memory, prompts, and traces are
readable by anyone who can read the database. This document maps every place
that content lives today and proposes how to change that for hosted
instances. The goal is a promise that a **database-only reader** cannot
recover customer content. Runtime and deploy authority remain trusted.

All file references are `path:line` at commit `49219a71`, relative to the
repository root. The schema was checked by applying every migration to a
scratch Postgres and listing `information_schema.columns`. Writers and
readers were traced to the sqlc query and its Go call site, or to the raw SQL
that bypasses sqlc.

## Summary

* **Content has many copies, not one.** One inbound message is persisted as
  `message_events.body` and `canonical_body`, then as stemmed lexemes in
  `message_search_documents.tsv`, as JSON in `work_items.payload`, and as an
  intake request. That request can be copied into `commitments.objective`,
  `commitment_origins.original_request`, and `commitment_inbox.payload`. The
  model request built from it (persona, memory, history, and tool results) is
  written to `model_calls.request_body`. It is also written to
  `prompt_snapshots.prompt`, `run_checkpoints.payload`,
  `task_continuations.checkpoint`, and, when model routing is on,
  `decision_calls.request_body`. Encrypting one column protects nothing.
* **Retention is mostly unbounded.** Body pruning is opt-in
  (`retention.model_call_body_days`, default 0 = keep forever) and partial.
  Many content columns have no pruning at all; see
  [Cross-cutting findings](#cross-cutting-findings).
* **Search is part of the boundary.** Message and memory lexemes sit in GIN
  indexes. Memory embeddings sit in `memory_embeddings` as `real[]`. Encrypting
  the bodies without moving search leaves a near-verbatim copy behind.
* **Proposed design:**
  * AES-256-GCM envelopes, each carrying a version and a key ID. Additional
    authenticated data (AAD) binds each envelope to its instance, purpose, and
    record.
  * Per-instance data keys stored wrapped in Postgres. The key that wraps them
    is held outside Postgres, separate from `MESH_AGENT_MASTER_KEY`.
  * **Search moves into a runtime-owned protected index**, persisted only as
    sealed segments. Authorization stays in SQL, unchanged.
  * The prototype measured the index for an agent with 10k–300k messages:

    * **0.02–0.2 ms at the median** for distinctive-word queries;
    * **up to 2 ms median and 8 ms p95** when queries hit very common words.

    Today's Postgres full-text search on the same corpus measured 0.3–9 ms
    and up to 256 ms median respectively. Details are in
    [Measured prototype](#measured-prototype).
* **Fix these authorization issues before encryption** (details in
  [Authorization review](#authorization-review)):
  * The control plane's management token can read memory and persona text.
  * Content reads are not audited.
  * Hosted (external-auth) users are all owners.

## Threat models

### Database reader

The database reader is anyone or anything that gets the database without the
running Formation runtime. Examples:

* a leaked dump, branch, or clone;
* a point-in-time restore or backup copy;
* a stolen database credential, or a mistakenly granted one;
* a Neon support or infrastructure path;
* a logical replication target;
* a misdirected `pg_dump`.

**Promise (proposed).** Once L14–L16 are complete for an instance, this
reader cannot read:

* covered content columns, and their derived copies (search terms,
  embeddings, digests);
* the keys that would decrypt them.

The reader can still see the residual metadata listed in
[Coverage and exceptions](#coverage-and-exceptions). That includes:

* IDs and relationships, timestamps, and lifecycle states;
* token counts and costs;
* audience membership lists;
* external identifiers needed for routing;
* ciphertext sizes and update timing.

The reader **can tamper**, for example by deleting rows, replaying older
ciphertext of the same record, or corrupting envelopes. AEAD with
record-bound AAD turns substitution and corruption into decryption failures.
It does not prevent deletion or rollback of a whole record to an older valid
envelope of the same record and purpose. This design does not try to give
the database integrity against a writer.

### Operator and deploy authority (trusted)

**Runtime and deploy authority remain trusted.** That includes:

* any principal of the hosting operator who can deploy code to the instance;
* anyone who can read its Fly secrets or exec into the machine;
* anyone who can change its image;
* anyone who holds an IAM grant that lets the runtime identity unwrap keys.

The runtime must decrypt content to build model requests. Code that
authority deploys can capture plaintext and keys. Neither a container secret
nor a KMS grant prevents that. Fly documents that deploy access can extract
application secrets.

What the design does provide against operators:

* **Separation of duties.** Database access alone (support querying Postgres,
  a database admin, a Neon console session) no longer yields content.
* **Evidence.** KMS unwrap logs record key use. A content-read audit records
  each decrypt-to-API read (see F2 in [Authorization review](#authorization-review)).
  KMS logs do not replace the per-read audit, because keys are cached.
* **Customer-key revocation path.** A later customer-managed wrapping key can
  stop *future* unwraps. It cannot retract plaintext already processed,
  cached, or sent to providers.

The authenticated operator API is not a database reader. It is a runtime
read, so it decrypts. Today every owner, every agent operator granted to the
agent, every `read`-scope API token, and (see F1) the control plane's
management token can read content through it. Encryption does not change
that. Authorization does.

### Out of scope

The following are separate boundaries. They need their own statements in
the customer promise:

* **Providers that receive content by design:** Slack, Telegram, the phone
  provider, model and decision providers, embedding providers, web search,
  read and browser providers, MCP servers, effect tools, and execution
  sandboxes. See [Outside Postgres](#outside-postgres).
* **Customer repositories.**
* **A stronger operator-exclusion claim,** which would need
  customer-controlled runtimes or attested confidential execution (Q15).

## Data-flow inventory

**Classes**

* **CONTENT**: customer or model text, prompts, tool input and output, files,
  persona.
* **DERIVED**: computed from content and possibly invertible. Examples:
  tsvectors, embeddings, unkeyed hashes, previews.
* **PII-META**: names, handles, emails, phone numbers, URLs, filenames,
  external IDs, audience member lists.
* **SECRET**: credentials, or values derived from them.
* **OPS**: IDs, timestamps, states, counts.

**Protection column.** The "Proposed" column says what L14–L16 should do with
each item:

* **E**: encrypt as an envelope.
* **I**: move into the protected index.
* **H**: replace with a keyed HMAC (a blind key).
* **K**: keep plaintext as declared residual metadata.
* **R**: reduce or stop retaining.
* **X**: drop.

Rows are grouped by area. Columns that are only IDs, states, or timestamps
are omitted unless they matter to the design.

### Conversation graph and intake

All three ingress paths go through `graph.Store.RecordMessage`
(`internal/graph/graph.go:110`):

* inbound messages via `Runtime.RecordInbound` (`internal/app/runtime.go:310`);
* outbound replies via `Runtime.executeTurn` (`internal/app/runtime.go:711`);
* the no-runtime Slack path via `internal/app/slack_ingress.go:882`.

| Store | Class | Writer | Readers | Retention | Proposed |
| - | - | - | - | - | - |
| `message_events.body` | CONTENT | `InsertMessageEvent` → `internal/graph/graph.go:191` | History `internal/graph/history.go:140`; search `internal/graph/search.go:371`, `:425`; resume `internal/app/run_worker.go:434`; steering `internal/steering/repository.go:114`; dreaming `internal/dreaming/snapshot.go:87`; names `internal/actorprofile/names.go:44`; raw SQL `internal/identity/identity.go:50`, `:85`, `internal/commitment/waits.go:231`, `internal/commitment/completion_verifier.go:47`; admin `internal/admin/inspection.go:388`, `:467`, `:622`, `internal/admin/runs.go:235` (280-char `trigger_summary`) | None. Append-only, never pruned | E + I |
| `dashboard_deliveries.body` | CONTENT (native delivery mailbox) | `internal/dashboard/chat.go` durable outbound acceptance | Native chat GET scoped to operator and agent; full database exports | None; retained until instance database deletion | E |
| `message_events.canonical_body` | CONTENT (Markdown re-render) | same, `internal/graph/graph.go:204` | admin `internal/admin/inspection.go:467`, `:622` only | None | E |
| `message_events.metadata` | PII-META + OPS: `memory_audience` (`readers[]` = Slack IDs, Telegram IDs, phone numbers), `memory_provenance` (IDs), the turn stamp (model, tokens, `prompt_digest`) | `internal/graph/graph.go:187`, `:212`; stamp `internal/app/runtime.go:1822` | Evaluated in SQL: `mesh_identity_audience_allows`, `mesh_messages_allowed`, evidence triggers (0038, 0039); `db/queries/conversation_search.sql`, `db/queries/graph.sql:425`, `db/queries/memory_retrieval.sql:76`, `db/queries/dreaming.sql:78`, `db/queries/continuations.sql:67`, `db/queries/run_lease.sql:155` | None | K (audience is used by SQL authorization); H `prompt_digest` |
| `message_search_documents.tsv` | DERIVED: stemmed lexemes **with positions**, close to a verbatim copy | Trigger `mesh_index_message_search` (`db/migrations/0090_conversation_search.sql:59`) on insert or update of `body` | `SearchConversationHistory` via `internal/graph/search.go:371`, `:270` | Cascades with the message | X → I |
| `message_attachments.name`, `.source_url`, `.mime_type` | PII-META. File bytes are **never stored**: they are fetched on demand, at most 8 MiB, in memory (`internal/connector/slack/files.go:35`) | `InsertMessageAttachment` → `internal/graph/graph.go:223` | History aggregation (`db/queries/graph.sql:408`); `internal/turn/documents.go:52`, `:92`, `:99`; admin `internal/admin/inspection.go:513`, `:524` (`source_url`) | None | E (name, URL); K (mime, size) |
| `images.data` | CONTENT: a drawing a person submitted through a sketch link (PNG, at most 2 MiB) | `InsertImage` → `internal/sketch/sketch.go:331` | `GetImage` at `internal/sketch/sketch.go:497` (the wake turn's input, through `PrepareWake`); `GetSketchImageForToolCall` at `internal/sketch/wake.go:150` (`view_sketch`) | With its conversation (`ON DELETE CASCADE`) | E |
| `sketch_submissions.scene`, `.note` | CONTENT: the drawing's Excalidraw scene, including its text, and the person's note | `InsertSketchSubmission` → `internal/sketch/sketch.go:344` | `internal/sketch/sketch.go:467` (wake), `:540`, `:580` | With its conversation | E |
| `sketch_submissions.submitted_from` | PII-META (submitter IP address) | same | none | With its conversation | K |
| `sketch_requests.prompt`, `.title` | CONTENT: what the agent asked the person to draw | `CreateSketchRequest` → `internal/sketch/sketch.go:236` | `internal/sketch/sketch.go:271` (public link page), `:467`, `:540`, `:580`, `:611` | With its conversation | E |
| `conversations.title` | PII-META (Telegram chat title) | `UpsertConversation` → `internal/graph/graph.go:135` | `internal/graph/search.go:403` (sent to the model); admin `internal/admin/inspection.go:411` | None | E |
| `conversations.external_thread_id`, `.external_channel_id` | PII-META. For **phone** these hold the sender's E.164 number (`internal/connector/phone/phone.go:98`) | `internal/graph/graph.go:138`, `:142` | Routing, dedupe, search, admin | None | K. Phone numbers: H plus an E copy (see [exceptions](#coverage-and-exceptions)) |
| `actors.display_name`, `.preferred_name` | PII-META. A phone actor's display name is the raw number | `internal/graph/graph.go:295`; `internal/graph/names.go:106`; `internal/actorprofile/names.go:74` | History and search (sent to the model); admin `internal/admin/inspection.go:454`, `:569`, `:609` | None | E |
| `connector_identities.external_id`, `.external_handle` | PII-META (Slack U-IDs, phone numbers, names) | `internal/graph/graph.go:303`; `internal/graph/names.go:115`; `internal/identity/commands.go:21` | `internal/graph/graph.go:272` (lookup by external ID); `internal/app/audience_policy.go:88`; admin `internal/admin/identity_linking.go:105` | None | handle: E. `external_id`: K (Slack/Telegram IDs) or H + E (phone) |
| `agent_connectors.metadata` | PII-META: phone `allowed_senders[]` (third-party numbers), Telegram bot username | `internal/admin/phone_connector.go:381`, `:468`; `internal/admin/telegram_connector.go:245` | `internal/agent/loader.go:113`; admin `internal/admin/agent_profile.go:523`, `:680` | None | E (`allowed_senders`) |
| `conversation_search_events.query_digest` | DERIVED: HMAC-SHA256 under a subkey of `MESH_AGENT_MASTER_KEY` | `internal/graph/search.go:347` | admin `internal/admin/trace.go:439`, `:626` | None | K (already keyed) |
| `conversation_search_events.audience`, `.selections` | PII-META / OPS | same | same | None | K |
| `work_items.payload` | CONTENT + PII-META. The normalized message (`Body`, `CanonicalBody`, names, `SourceURL`, audience readers) | `EnqueueWork` → `internal/workqueue/workqueue.go:204` (from `internal/app/slack_ingress.go:657`, `internal/app/telegram_ingress.go:214`, `internal/app/phone_ingress.go:235`); rewritten at `internal/workqueue/workqueue.go:300` | `internal/app/durable_queue.go:348`; `internal/steering/repository.go:154` | Scrubbed to `{}` seven days after completion (`db/queries/workqueue.sql:200`, `internal/workqueue/workqueue.go:537`) | E (whole payload), plus R |
| `work_items.conversation_key`, `conversation_turn_claims.conversation_key`, `turn_disposals.conversation_key` | PII-META (phone number for phone installs) | `internal/workqueue/workqueue.go:204`; `internal/claim/claim.go:218`; `internal/app/wire.go:436` | Queue and claim logic | Claims are swept; work items are kept | H |
| `engagement_classifications.signal`, `turn_disposals.detail` | OPS, possibly carrying error text or a name candidate (`internal/addressing/addressing.go:327`–`:404`) | `internal/app/wire.go:394`, `:436` | admin `internal/admin/inspection.go:717` | None | R (fixed codes only) |
| `identity_challenges.code_hash` | DERIVED: unkeyed SHA-256 of an 80-bit code. The code itself is plaintext in the message body | raw SQL `internal/identity/identity.go:150` | `internal/identity/identity.go:141`–`:275` | None | K (high entropy) |

### Run ledger, model, and tools

The model request is assembled by `completeModelOnce`
(`internal/turn/turn.go:2459`). `ledgerRequestBody` (`internal/turn/turn.go:2482`)
returns the exact wire bytes, with images replaced by a name, size, and hash
prefix (`internal/model/audit.go:10`). Those bytes are persisted **before**
the provider call.

| Store | Class | Writer | Readers | Retention | Proposed |
| - | - | - | - | - | - |
| `model_calls.request_body` | CONTENT: persona, memory, history, current message, tool results | `CreateModelCall` → `internal/run/run.go:270`, `internal/run/checkpoint.go:155` | admin trace `internal/admin/trace.go:430` (returned verbatim) | Nulled after `retention.model_call_body_days` when set; default 0 = forever (`internal/app/retention_maintenance.go:51`) | E, plus R by default |
| `model_calls.response_body`, `.response_content` | CONTENT (the raw provider JSON; the reply text) | `CompleteModelCall` → `internal/run/run.go:296` | trace; resume `internal/run/attempt.go:154` (`response_content`) | `response_body` is pruned like the request. `response_content` is **never pruned** | E |
| `model_calls.error`, `runs.error`, `run_steps.error`, `tool_calls.error` | OPS, but `err.Error()` can embed provider or tool text | `internal/run/run.go:330`, `:651`, `:509`, `:492` | admin `internal/admin/runs.go:235`, trace | Never | R (bounded codes plus E for detail) |
| `prompt_snapshots.prompt` | CONTENT: the full assembled system prompt | `CreatePromptSnapshot` → `internal/app/wire.go:1211` ← `internal/prompt/repository.go:71` ← `internal/turn/context_budget.go:165` | admin `internal/admin/trace.go:394` (full body) | **None.** `model_calls.prompt_snapshot_id` is `ON DELETE RESTRICT` (`db/migrations/0019_run_ledger.sql:116`) | E, plus R |
| `prompt_snapshots.digest`, `message_events.metadata.prompt_digest` | DERIVED: unkeyed SHA-256 (`internal/state/state.go:10`). Lets a reader confirm a guessed prompt | same | `internal/admin/agent_profile.go:606` | None | H |
| `tool_calls.raw_arguments`, `.validated_arguments` | CONTENT: search queries, message text, file contents, commands | `CreateToolCall` → `internal/run/run.go:370` | trace `internal/admin/trace.go:434`; `internal/run/attempt.go:157`; `internal/run/checkpoint.go:300`; raw SQL `internal/commitment/ownership_effects.go:133` | Never | E |
| `tool_observations.content`, `.metadata` | CONTENT: the full tool result, stored before any spill to scratch | `CreateToolObservation` → `internal/run/run.go:430`, `:474` | trace; attempt and checkpoint resume; `internal/webbrowser/artifacts.go:54` | Never | E |
| `tool_source_evidence.audience` | PII-META | `internal/tool/mcp/service.go:736`; `internal/webcap/runtime.go:211` | SQL predicates; raw SQL `internal/commitment/completion_disclosure.go:17` | Never | K |
| `run_checkpoints.payload` | CONTENT: messages, the assembled prompt, continuation background, model response (`internal/turn/checkpoint.go:25`) | `PutRunCheckpoint` → `internal/run/checkpoint.go:158`, `:257`; `internal/run/run.go:308` | `internal/run/checkpoint.go:109` ← `internal/turn/checkpoint.go:210`, `:496`; `internal/app/commitments.go:88` | Deleted for terminal runs only when the body retention window is set (`internal/app/retention_maintenance.go:62`) | E |
| `task_continuations.checkpoint` | CONTENT: objective, transcript, background, summary (`internal/continuation/context.go:24`) | `internal/continuation/service.go:327` ← `internal/app/continuations.go:54` | `internal/continuation/service.go:183` | **Never deleted** | E, plus R |
| `decision_calls.request_body`, `.response_body` | CONTENT. Under model routing, the **full main-model request** (`internal/modelrouting/connector.go:181`) | `internal/decision/audit.go:44`, `:84`; `internal/decision/observed_connector.go:188`; `internal/modelrouting/connector.go:213` | admin `internal/admin/decision_calls.go:66` (whole row) | A separate setting, `decisions.body_retention_days`, default 0 (`internal/app/decision_maintenance.go:24`) | E |
| `decision_calls.answer` | DERIVED (structured verdict) | `internal/decision/audit.go:84` | admin list and trace | Never | E |
| `progress_events.body`, `.snapshot`, `.model_output`; `progress_state.last_text` | CONTENT (progress text, objective, raw model output) | `internal/progress/repository.go:123`, `:160`, `:175`, `:211` | `internal/progress/repository.go:273` → admin `internal/admin/trace.go:690`; `:221`, `:284` | None | E |
| `worker_tasks.objective`, `.result` | CONTENT. **Dormant:** no non-test importer of `internal/worker` | `internal/worker/task.go:40`, `:78` | `internal/worker/task.go:44` | None | E when enabled |
| `scratch_entries.content`, `.preview` | CONTENT: a second copy of spilled tool output, plus notes | `internal/scratch/scratch.go:210` | `internal/scratch/scratch.go:262`, `:294` | One-hour TTL, hard-deleted (`internal/scratch/scratch.go:343`) | E |
| `model_routes.route` | DERIVED/OPS (classifier answers, no raw text) | raw SQL `internal/modelrouting/store.go:198` | `internal/modelrouting/store.go:184`, `:211` → admin `internal/admin/model_routing.go:286` | None | K |
| Browser screenshots | CONTENT. JPEG bytes as a `secrets` row (`browser:artifact:<id>`), already AES-GCM under the master key | `internal/webbrowser/artifacts.go:41`, `internal/webbrowser/storage.go:84` | `internal/webbrowser/artifacts.go:90` → model; admin `browser-artifacts/{artifact}` | 24 hours (`internal/webbrowser/cleanup.go:66`) | Move to content envelope (E) |
| `project_snapshots.repo`, `.validation`; `workspaces.external_ref` | PII-META (repo `owner/name`). Snapshot **contents** are Modal images | `internal/workspace/workspace.go:1325`, `:1427`, `:860` | `internal/workspace/workspace.go:999`, `:989` | Backend artifact deleted 7 days after supersession; rows kept | K (declared) |

### Memory, retrieval, and dreaming

| Store | Class | Writer | Readers | Retention | Proposed |
| - | - | - | - | - | - |
| `memory_entries.body`, `.kind` | CONTENT | `CreateMemoryEntry` → `internal/memory/memory.go:433`; `AmendMemoryEntry` → `:506`; `internal/dreaming/apply.go:320`, `:343`; admin `internal/admin/agent_profile.go:1104`, `:1256`; import `internal/admin/memory_import.go:274` | Retrieval `internal/memory/retrieval.go:104`, `:230`; sharing `internal/memory/sharing.go:52`; dreaming `internal/dreaming/snapshot.go:157`; admin `internal/admin/agent_profile.go:528` (**agent detail**), `:1184`, `:1359`; export `internal/admin/agent_export.go:200` | Forget only sets `deleted_at`; the body stays | E + I |
| GIN `to_tsvector('english', body)` on `memory_entries`, twice: `idx_memory_entries_body_fts` (0018) and `memory_entries_lexical_idx` (0038) | DERIVED lexemes. **The same index exists twice** | Postgres | `SearchEligibleMemory` (`db/queries/memory_retrieval.sql:1`), `HasUnknownAudienceWithheld`, `DreamRelevantMemories` (`db/queries/dreaming.sql:100`) | Partial on `deleted_at IS NULL` | X → I |
| `memory_entry_revisions.body`, `.notes` | CONTENT (every historical body, including forgotten ones; model rationale) | `internal/memory/memory.go:644`; `internal/dreaming/apply.go:349`, `:487`; admin `internal/admin/agent_profile.go:1117`, `:1264`; `internal/admin/memory_import.go:282` | Indexer `internal/memory/indexer.go:76` (body sent to the embedder); admin `internal/admin/agent_profile.go:1372` | Immutable, never pruned | E |
| `memory_embeddings.embedding` | DERIVED, **treat as content** (text embeddings are substantially invertible). `real[]`, not pgvector | `internal/memory/indexer.go:122`; raw `internal/memory/pgvector/pgvector.go:59` | `internal/memory/indexer.go:109`; pgvector search in SQL `internal/memory/pgvector/pgvector.go:29` | Old generations are never pruned; vectors of forgotten memories remain | E (search moves to the runtime) |
| `memory_retrieval_events.query_digest` | DERIVED, keyed HMAC (`internal/auditdigest`, `internal/memory/memory.go:333`) | `internal/memory/retrieval.go:143` | none (write-only) | None | K |
| `memory_revision_evidence.audience`, `memory_sharing_grants.audience`, `memory_share_requests.*` | PII-META (the authorization graph; readers lists). `code` is about 38 bits, plaintext, 15-minute TTL | Triggers (0038); `internal/memory/sharing.go:67`, `:154`, `:223`; `internal/dreaming/apply.go:353`, `:360` | `mesh_memory_allowed` (SQL); admin `internal/admin/memory_sharing.go:33`, `:38` | Grants are revocable | K (audience); H (`code`) |
| `memory_import_sources.system`, `.ref` | PII-META | `internal/admin/memory_import.go:305` | `internal/admin/memory_import.go:372`; `internal/admin/agent_export.go:207` | Kept | K |
| `dream_jobs.snapshot` | CONTENT: **verbatim transcripts** (message bodies, author names), memory bodies, public source text, up to 24 KB | `internal/dreaming/service.go:207`; `internal/dreaming/apply.go:167` | `internal/dreaming/service.go:73` → admin `internal/admin/dreaming.go:41` (raw JSON); `internal/dreaming/apply.go:427`; `internal/dreaming/questions.go:27` | Kept | E, plus R |
| `dream_proposals.proposal`, `.before_state`, `.verification` | CONTENT (model-proposed memory text and a pre-change memory copy) | `internal/dreaming/apply.go:210` | admin `internal/admin/dreaming.go:41`; `internal/dreaming/apply.go:399`; `internal/dreaming/questions.go:27` | Kept | E |
| `dream_public_sources.url`, `.body` | CONTENT (operator-provided). Unique index on `md5(url), md5(body)` | `internal/dreaming/service.go:93` ← `internal/admin/dreaming.go:115` | `internal/dreaming/snapshot.go:26` | Kept | E; H for the dedupe key |
| `dream_memory_lifecycle.event_end`, `.event_status` | DERIVED (dates of personal events) | `internal/dreaming/apply.go:292`, `:381` | `internal/dreaming/apply.go:107` | Kept | K (declared) or E |

### Commitments and schedules

Most of these writes use hand-written SQL in `internal/commitment`,
`internal/worksource`, and `internal/app`, not sqlc. Encryption at the
`internal/store` layer alone would miss them.

Retention for this area runs inside `maintainRetention`
(`internal/app/retention_maintenance.go:72`). It covers five prune queries
and uses the same opt-in window. Rewriting a body clears `bodies_pruned_at`
again (`internal/commitment/criteria.go:216`,
`internal/commitment/worksource_scope.go:51`, `internal/worksource/reconcile.go:415`).

| Store | Class | Writer | Readers | Retention | Proposed |
| - | - | - | - | - | - |
| `commitments.objective`, `.attempt_reply`, `.evidence_contract`, `.last_reason` | CONTENT | `CreateCommitment` ← `internal/commitment/ownership_origins.go:207`; `internal/commitment/proposal.go:536`; raw `internal/commitment/ownership_external.go:55`, `internal/commitment/worksource_scope.go:71`; `TransitionCommitment` ← `internal/commitment/commitment.go:624` | admin `internal/admin/commitments.go:63`, `:255` (**whole row**); prompts `internal/commitment/runtime.go:180`; classifiers `internal/commitment/capture.go:125`, `internal/commitment/proposal.go:602`, `internal/commitment/intake_resolution.go:106` | **Never pruned** | E |
| `commitments.objective_hash`, `commitment_proposals.objective_hash`, `commitment_origins.source_key`, `commitment_effects.operation_hash`, `commitment_completion_evidence.evidence_key`, `.fingerprint`, `commitment_inbox.event_key`, `schedules.creation_key` | DERIVED: **unkeyed SHA-256** of objectives, tool and argument pairs, or instruction and cadence pairs, used in unique and primary keys | `internal/commitment/commitment.go:653`; `internal/commitment/ownership_effects.go:39`; `internal/commitment/completion_verifier.go:426`; `internal/commitment/worksource.go:140`; `internal/schedule/service.go:110` | Equality lookups | Never | H (keyed, same equality semantics) |
| `commitment_proposals.objective`, `.evidence_contract` | CONTENT | `internal/commitment/proposal.go:199` | `internal/commitment/proposal.go:434`, `:527`; `internal/commitment/capture.go:128` | Short-lived (deleted on promotion or expiry) | E |
| `commitment_intakes.request`, `commitment_intake_associations.request` | CONTENT: raw inbound message body, up to 64 KB | raw `internal/commitment/intake.go:224`; `internal/commitment/intake_resolution.go:161`, `:297` | `internal/commitment/intake_resolution.go:550`, `:433`; `internal/app/commitment_intake.go:145` | Placeholder after the window | E |
| `commitment_origins.original_request`, `.authority` (holds a copy of `objective`) | CONTENT | raw `internal/commitment/ownership_origins.go:273`, `:345`; `internal/commitment/worksource_scope.go:51`; `internal/commitment/proposal.go:575` | `internal/commitment/completion_verifier.go:40` (verifier model); `internal/commitment/intake_resolution.go:106` | Pruned to `''`; `authority.principal` and `scope_hash` stay | E |
| `commitment_inbox.payload` | CONTENT: request text, a human's **answer text** (`internal/commitment/waits.go:309`), work-source items | raw `internal/commitment/ownership_inbox.go:69` | `internal/commitment/inbox_context.go:32` (to the model) | Pruned to `{}` | E |
| `commitment_origin_notices.body` | CONTENT (the reply to be delivered) | raw `internal/commitment/ownership_delivery.go:83` | `internal/commitment/ownership_delivery.go:149`, `:253` → connector | Pruned (terminal notices only) | E |
| `commitment_completion.criteria`, `.reply_artifact`, `.wait_detail`; `commitment_completion_evidence.arguments`, `.observation`; `commitment_human_requests.question` | CONTENT | raw `internal/commitment/criteria.go:216`, `:339`; `internal/commitment/completion_verifier.go:129`; `internal/commitment/waits.go:79`, `:165`; `internal/commitment/completion_observation.go:96` | admin `internal/commitment/criteria.go:110` ← `internal/admin/commitments.go:278`; verifier model | Pruned | E |
| `commitment_completion.workspace_checkpoint` | PII-META (repo, directory) | raw `internal/commitment/workspace_checkpoint.go:97` | JSON key filters (`db/queries/continuations.sql:34`, `db/queries/workspaces.sql:45`) | Not pruned | K (`run_id` is used by SQL); E for the rest |
| `commitment_effects.observation` | CONTENT: the full result of an external-write tool | `internal/commitment/ownership_effects.go:63`; `internal/commitment/runtime.go:438` | `internal/commitment/runtime.go:410`; `internal/commitment/ownership_checkpoint.go:126` (to the model) | **Never pruned** | E |
| `commitment_references.url`, `.object_id` | PII-META (external issue URL and ID) | raw `internal/commitment/ownership_origins.go:284` | `internal/commitment/ownership_tools.go:107`; unique index on `object_id` | Never | E (`url`); H (`object_id`) |
| `commitment_source_plans.config`, `commitment_source_observations.query`, `.result`, `commitment_source_items.item`, `commitment_source_sweeps.items` | CONTENT + PII-META (query arguments, source results, item objective and URL) | raw `internal/worksource/service.go:125`; `internal/worksource/reconcile.go:78`, `:415`, `:426`, `:466` | admin `internal/admin/commitment_sources.go:79`, `:103`; `internal/tool/mcp/observation.go:146` (**equality on `query`**) | Partly pruned | E; H for the `query` equality |
| `commitment_events.reason`, `commitment_effect_resolutions.result` | OPS, sometimes operator free text | many (see `AddCommitmentEvent` callers); `internal/commitment/commitment.go:559` | admin `internal/admin/commitments.go:255` | Never | E |
| `schedules.instruction`, `schedule_occurrences.instruction` | CONTENT (recurring task text, copied into every occurrence) | `internal/schedule/service.go:123`, `:245`; `internal/schedule/scheduler.go:74`, `:212` | admin `internal/admin/schedules.go:35`, `:57`; `internal/commitment/capture.go:135`; `internal/schedule/runtime.go:31`; `internal/turn/participant_context.go:26` | **Never** | E |
| `schedules.cadence`, `.audience`, `.provenance` | PII-META (timezone, times, readers) | same | same | Never | K |

### Configuration and credentials

| Store | Class | Writer | Readers | Proposed |
| - | - | - | - | - |
| `agent_persona_revisions.body` | CONTENT (operator-authored prompt) | `internal/admin/persona.go:131` | `internal/agent/loader.go:342` (system prompt); admin `internal/admin/persona.go:112`, `:171`, `:230`; export `internal/admin/agent_export.go:189`; **management token** (F1) | E |
| `agent_tool_config.value`, `web_connections.config`, `tool_providers.endpoint`, `agent_tool_providers.name` | PII-META (default repos and teams, account IDs, MCP endpoints) | `internal/toolconfig/toolconfig.go:315`; `internal/webcap/settings.go:299`; `internal/tool/mcp/service.go:226`, `:259` | `internal/toolconfig/toolconfig.go:175`; admin `internal/admin/mcp.go:78` | K (declared) |
| `mcp_catalog_snapshots.tools` | DERIVED (third-party tool schemas) | `internal/tool/mcp/snapshot.go:186` | `internal/tool/mcp/snapshot.go:78` | K |
| `model_routing_profiles.binding` | SECRET-derived: unkeyed `sha256(provider, endpoint, apiKey)` (`internal/modelrouting/service.go:64`) | `internal/modelrouting/store.go:141` | `internal/modelrouting/store.go:57` | H |
| `install_settings.value` | OPS (memory provider endpoints; secrets rejected) | `internal/admin/memory_settings.go:133` | `internal/settings/memory.go:17` | K |
| `secrets.ciphertext` (+ `redacted_hint`) | SECRET. AES-GCM under `MESH_AGENT_MASTER_KEY`. **The AAD is only `redacted_hint`** (`internal/admin/handlers.go:873`–`:879`) or a constant (`"MCP credential"` `internal/tool/mcp/service.go:351`; `"configured"` `internal/slackprovision/store.go:74`, `internal/webbrowser/storage.go:89`, `internal/webcap/settings.go:314`). A row can move to another owner and still open; the code does this on purpose at `internal/admin/model_credentials.go:44`. `redacted_hint` is 48 bits of an unsalted SHA-256 of the plaintext: an offline guessing oracle for low-entropy secrets | many (see `upsertEncryptedSecret`) | `internal/agent/loader.go:125`, `:152` (all agent secrets are decrypted at load) | Unchanged by L14. Separate hardening: bind owner and kind in AAD |
| `agent_vault_entries`, `agent_vault_drop_tokens` | SECRET. HKDF subkey `mesh/vault/v1`; AAD `vault:<agent>:<name>` and `vault:<agent>:drop-token:<drop>` (`internal/vault/vault.go:386`; `internal/vault/drops.go:691`) | `internal/vault` | `internal/vault` | Already bound. The plaintext `name`/`description` are PII-META; K, or E for `description` |
| `mcp_oauth_authorizations.ciphertext` | SECRET. Constant AAD `"MCP authorization"` (`internal/tool/mcp/oauth.go:552`). **Not covered by `rotate-key`/`verify-secrets`** (`internal/rekey/rekey.go:173`) | `internal/tool/mcp/oauth.go:565` | `internal/tool/mcp/oauth.go:643` | Separate hardening |
| `agent_avatars.png` | PII-META (low) | `internal/admin/agent_identity.go:66`, `:285`; `internal/slackprovision/service.go:220` | `internal/admin/agent_identity.go:34` | K |

### Identity, operators, and audit

| Store | Class | Writer | Notes | Proposed |
| - | - | - | - | - |
| `users.email`, `user_external_identities.subject` | PII-META | `db/queries/users.sql`; `management.sql` | Operator identity | K (declared) |
| `audit_events.actor_email`, `.source_ip`, `.target`, `.details` | PII-META. `details` can carry an install-stop `reason` (operator free text, `internal/admin/security.go:183`) and attempted sign-in emails (`internal/admin/handlers.go:230`) | `internal/admin/audit.go:119`–`:191` (non-GET only) | Append-only, never pruned | K. Install-stop reason: E |
| `agent_vault_drops.consumed_from` | PII-META (submitter IP) | `internal/vault/drops.go` | | K (declared) or R |
| `sessions`, `api_tokens`, `user_invites` | SECRET-derived (token hashes) and OPS | | Hashes of high-entropy tokens | K |

### Outside Postgres

| Location | What can contain content | Reference | Proposed |
| - | - | - | - |
| Process logs (stderr JSON; OTLP when log export is on) | No call site logs bodies directly. `"error", err` sites can embed upstream response bodies: Telegram (`internal/connector/telegram/send.go:179`), Slack `users.info`/`auth.test`/reactions (`internal/connector/slack/users.go:142`, `internal/connector/slack/auth.go:79`, `internal/connector/slack/react.go:171`), Linear and GitHub up to 256 bytes (`internal/tool/effects/linear/client.go:377`, `internal/tool/effects/github/client.go:345`), Fly (`internal/execution/fly/fly.go:654`), panic values (`internal/app/errtrack.go:152`). Operator emails and phone numbers are logged (`internal/admin/operators.go:358`, `internal/admin/phone_shared.go:506`) | `internal/app/logging.go:35` | R: bounded error classes at these boundaries; a redaction test like errtrack's |
| OTLP traces | Labels by design (`docs/telemetry.md`). **But** `span.RecordError(err)` exports full error text (`internal/app/ingress_dispatch.go:91`, `:133`, `:415`), contradicting `docs/telemetry.md`. `mesh.engagement.signal` can include a name candidate or classifier error | `internal/telemetry/spans.go:84` | R: replace with `telemetry.End` |
| Error tracker | Scrubbed of credentials, but content is **truncated to 1 KiB, not removed** (`internal/errtrack/redact.go:48`–`:162`). Tool failures carry upstream text | `internal/app/ingress_dispatch.go:538`; `internal/turn/turn.go:2436` | R: send error class and fingerprint only |
| In-memory | Registry holds all decrypted agent secrets and persona text (`internal/agent/loader.go:125`, `:341`); Slack channel-name cache (`internal/app/conversation_names.go:28`). Under this design the protected index also lives here | | Trusted runtime boundary |
| Local disk | None in production: document extraction runs in-process over stdin and stdout (`internal/document/bundled.go:84`), with no cache (`internal/document/reader.go:75`) | | none |
| Execution sandboxes | Fly Sprites receive exec argv **in the WebSocket URL query string**, stdin, written files, repo checkouts, and the grant file with vault values (`internal/execution/fly/fly.go:291`, `:613`; `internal/workspace/workspace.go:700`). Modal also keeps **snapshot images** of the project directory until reaped (`internal/execution/modal/snapshot.go`; `internal/app/workspace_runtime.go:504`) | | Separate provider boundary; declare it |
| Providers by design | Model and decision providers (full prompts, screenshots); embedding endpoint (memory bodies and every retrieval query, `internal/memory/indexer.go:111`, `internal/memory/retrieval.go:351`; the default onboarding config sends workspace-audience memory to OpenRouter, `internal/admin/onboarding_memory.go:31`); web search, read, and browser providers (queries, URLs, and vault values through `fill_secret`); Slack, Telegram, Twilio, Telnyx; MCP servers; GitHub, Linear, and GitLab; Qdrant when configured (vectors leave the instance) | | Separate provider boundary; declare it |
| Control plane | The provisioner generates `MESH_AGENT_MASTER_KEY` and writes it **only** as a Fly app secret (`apps/provisioner/src/provision.ts:216`–`:235` in the control plane repo). No code path writes a separate recovery copy. The CP holds the derivation key for every instance's management token (`apps/provisioner/src/management.ts:70`), so it can read what F1 exposes. The CP database holds no Formation conversation content today | | Key escrow (see [Key custody](#key-custody)) |

### Cross-cutting findings

1. **Copies multiply.** Count the stores per message, memory, and model
   request above before claiming coverage. L14 needs a coverage check that
   seeds a canary string through every writer and greps a database-only dump.
2. **Retention is opt-in and partial.** With the default
   `retention.model_call_body_days = 0`, nothing is pruned. Even with a
   window set, the following stay forever:

   * `response_content`, `prompt_snapshots`, tool arguments and observations,
     progress text, `task_continuations`, `commitments.objective` and
     `attempt_reply`, `commitment_effects.observation`, schedules, and every
     error column;
   * message and memory bodies, which have no retention at all.

   Encryption does not reduce exposure to runtime readers; less retention
   does. Propose defaults under Q11.
3. **Unkeyed hashes act as equality oracles.** Short or predictable plaintext
   can be confirmed by hashing guesses. Affected columns:

   * `objective_hash`, `operation_hash`, `evidence_key`, `creation_key`, and
     `source_key`;
   * `prompt_digest` and the `prompt_snapshots.digest` column;
   * the md5 dream dedupe index;
   * `model_routing_profiles.binding`;
   * `secrets.redacted_hint`.

   Every such column that guards uniqueness must become a keyed HMAC with the
   same equality semantics.
4. **SQL evaluates JSON and text that look like content.** Authorization reads
   `message_events.metadata->'memory_audience'`, the evidence tables, and
   `audience` columns. Some predicates compare audience JSON by equality.
   `commitment_source_observations.query` is matched by equality, and
   `workspace_checkpoint->>'run_id'` is indexed. These must stay plaintext,
   or become keyed equality values. That is why the audience membership lists
   are declared residual metadata rather than encrypted.
5. **Raw SQL writers are common.** These areas have writers outside sqlc:

   * identity (`internal/identity`);
   * commitments and work sources (most commitment tables);
   * web browser (`internal/webbrowser`);
   * model routing (`internal/modelrouting/store.go`);
   * three Postgres triggers that copy `message_events.body` and metadata
     (0038, 0039, 0090).

   The envelope must live in a shared package that each writer calls. The
   triggers that derive from `body` must be dropped or rewritten.
6. **The dormant `worker_tasks` table and the two dead query paths**
   (`ListScratchEntriesForRun`, `ListTurnDisposalsForMessage`) should be
   covered or removed before a coverage claim, so they do not become unseen
   writers later.

## Proposed envelope and keys

### Format

Store each covered value as a `bytea` envelope:

```
byte 0       version (0x01)
bytes 1..16  data-key ID (UUID of the content_keys row)
bytes 17..28 96-bit random nonce
bytes 29..   AES-256-GCM ciphertext || 16-byte tag
```

* **AEAD:** AES-256-GCM from the Go standard library, the primitive
  `internal/secrets` already uses. It is hardware-accelerated on arm64 and
  amd64. The prototype measured 1 KiB at 0.61 µs to seal and 0.46 µs to
  open (Apple M3 Max).
* **Nonce limit.** Random 96-bit nonces are limited to about 2³² seals per
  key. Each data key therefore carries a seal budget well under that (for
  example 2³⁰), and a time limit (for example 90 days). A new data key is
  minted when either is reached. XChaCha20-Poly1305 would remove the budget,
  but it adds a dependency and a second primitive. Prefer the budget.
* **AAD** binds the envelope to where it may be opened, and to its own header:

  ```
  "mesh.content.v1" 0x00 header(version || key ID || nonce)
                    instance_id 0x00 purpose 0x00 record_key
  ```

  * `instance_id` is the hosted instance's stable ID. External-auth installs
    already receive it as `MESH_INSTANCE_ID` (`internal/config/config.go:480`).
    L14 records it in a descriptor row checked at boot, so a database restored
    under another instance ID refuses to open rather than misbinding. An
    isolated restore drill therefore runs under the original instance ID.
  * `purpose` names the table and column, for example
    `message_events.body` or `model_calls.request_body`.
  * `record_key` is the row's immutable primary key, plus the revision number
    for versioned rows.

  With this binding, an envelope copied to another row, another column, or
  another instance's database fails authentication. It does not open in the
  wrong place. The prototype tests each substitution case:
  `TestEnvelopeBindsInstancePurposeAndRecord` in
  `internal/experimental/contentsearch/envelope_test.go`.
* **Mutable fields stay out of the AAD**, so state transitions never
  re-encrypt.
* **Readers fail closed.** A row that fails to open is reported as corrupt.
  It is never shown as empty or silently skipped. This matches the
  `verify-secrets` posture in [Secrets and recovery](https://docs.formation.texturehq.com/secrets-and-recovery.md).
* **JSON columns:** encrypt the whole value, unless SQL evaluates part of it
  (finding 4 above). In that case, split the column: keep the operational keys
  in a plaintext JSON column and put the rest in an envelope column.

### Key hierarchy

```
wrapping key (KEK)  ── held outside Postgres (custody below), one per instance
   └─ data keys (DEK) ── random 256-bit, stored only WRAPPED in content_keys
        └─ envelopes ── every covered column
```

* **`content_keys` table (new).** Columns: `id`, `purpose_class`,
  `wrapped_key`, `kek_id`, `created_at`, `retire_after`, `seal_count`, and
  `status`.
  * `kek_id` is a fingerprint, never key material.
  * `seal_count` is the durable, cross-replica account of the key's seal
    budget. A replica never seals against a local count alone: it reserves a
    block of seals with one atomic conditional `UPDATE` on the row (matching
    only while the key is active, within its age limit, and the block fits
    the budget) and seals from that block. A crash forfeits the unused rest
    of a block, so the count can over-state seals but never under-state them,
    and replicas together cannot exceed the budget. When a reservation
    matches nothing, the class rotates to a new key.
  * A database-only dump holds wrapped data keys and ciphertext. Without the
    KEK it holds neither plaintext nor usable keys.
* **Separate from credentials.** The KEK is not `MESH_AGENT_MASTER_KEY` and
  is not derived from it. Credential rotation (`mesh rotate-key`) and content
  rotation are independent operations with independent recovery copies, as
  the strategy requires.
* **Purpose classes** (proposal: `conversation`, `run_ledger`, `memory`,
  `work`, `index`) allow later per-class rotation or retention without
  changing the format. Per-audience or per-actor keys from the earlier
  access-control proposal are **not** required for tenant storage protection,
  and are not proposed here.
* **Data keys in memory.** The runtime unwraps data keys at boot and on first
  use, and caches them in memory. Each covered value is opened only when
  needed. Keys never appear in logs, telemetry, `install_settings`, or API
  responses. Key IDs and status may (L14).

### Key custody

| | A. Per-instance KEK as an environment secret | B. Cloud KMS key per instance | C. Customer-managed key (later) |
| - | - | - | - |
| Where the KEK lives | A new Fly secret, `MESH_CONTENT_KEK`, generated by the provisioner beside today's master key, plus an escrow copy | A KMS key that never leaves the KMS. The runtime calls Decrypt on wrapped data keys | Customer KMS key wrapping the instance's data keys, or wrapping B's key |
| Database-reader protection | Yes | Yes | Yes |
| Protection from deploy authority | None (deploy can read Fly secrets) | None (deploy runs code holding the grant) | None for live plaintext; revocation stops future unwraps |
| Audit | None per unwrap; only Fly secret change history | Per-unwrap KMS log, independent of Postgres | Customer-visible log |
| Rotation | Rewrap the `content_keys` rows under the new KEK. No content rewrite | KMS rotation plus rewrap, both cheap | Rewrap under the customer key; no content rewrite |
| Recovery | **Needs an escrow copy.** Today the master key exists only as a Fly secret (`provision.ts:216`–`:235`). Proposed escrow: a CP-held copy sealed to an offline or KMS recovery key, labeled by fingerprint | The KMS key persists independently of Fly. Deletion is scheduled and cancellable | The customer controls recovery; losing the key is data loss |
| Runtime identity | Fly secret injected into the machine | Needs workload identity from Fly machines to the KMS (for example OIDC federation), **not yet verified for our Fly setup** | Same as B, plus cross-account grants |
| Availability dependency | None at runtime | KMS reachable at boot and on new data keys only (data keys are cached) | Same as B, and the customer can break the instance |
| Cost and complexity | Lowest | Moderate (IAM per instance, KMS cost per key) | High; enterprise only (Q15) |

**Proposal (needs Q09 and Q11):**

1. Ship L14 with **A plus escrow**.
2. Design the `kek_id` and rewrap path so that **B** can replace A without
   touching content.
3. Treat **C** as a later enterprise option.

Both A and B satisfy the database-reader promise. Neither supports a claim
that the hosting operator cannot access content. Before broad launch, B adds independent
unwrap audit, a deletion schedule, and recovery outside Fly. Whichever is
chosen, the provisioner must **never regenerate a missing KEK**. A missing
key is a recovery incident, not a fresh install. This matches the existing
`RecoveryRequiredError` posture for the master key.

### Rotation, recovery, and retention

* **KEK rotation:** rewrap every `content_keys` row in one transaction, then
  retire the old KEK only after every retained backup that references it has
  expired. Label KEK escrow copies by fingerprint, as
  [Secrets and recovery](https://docs.formation.texturehq.com/secrets-and-recovery.md) already requires for the
  master key.
* **Data-key rotation:**
  * New writes use the current data key. Old envelopes stay readable through
    their key ID.
  * Re-encrypting old content is a bounded background job, needed only after
    a suspected data-key exposure or to retire a key.
  * It uses the same resumable machinery as the L16 backfill.
* **Compromise.** A suspected KEK exposure means rotating the KEK, minting new
  data keys, and re-encrypting. It cannot recall what the holder already
  decrypted.
* **Recovery is a restored pair: the database plus its matching KEK.** The
  backup drill in [Secrets and recovery](https://docs.formation.texturehq.com/secrets-and-recovery.md) extends to
  restoring both into an isolated instance, then running a content check
  (`verify-content`, opening a sample of every purpose) and search.
* **Instance deletion and crypto-shredding.** Destroying every KEK copy
  renders all retained backups of that instance unreadable once they expire.
  This is the one place encryption shortens retention in practice. State it
  only after the escrow and backup schedules are confirmed (Q11).
* **Record deletion** still deletes rows. Backups keep the envelope until they
  expire. That is unreadable without the KEK, and readable by anyone who holds
  both.

## Retrieval

### What breaks

Encrypting the bodies breaks every lexical and vector path that Postgres
evaluates today:

* `SearchConversationHistory` and its context and exclusion queries
  (`db/queries/conversation_search.sql`);
* `SearchEligibleMemory`, `HasUnknownAudienceWithheld`, and
  `DreamRelevantMemories`;
* the pgvector query (`internal/memory/pgvector/pgvector.go:29`);
* the `mesh_index_message_search` trigger, which would index ciphertext.

Authorization does **not** depend on the bodies. It is evaluated over audience
JSON, evidence, lineage, and IDs. Search and memory retrieval already treat
the index as a source of candidates and re-authorize in SQL
(`db/migrations/0090_conversation_search.sql` header,
`internal/memory/retrieval.go:223`). Every option below keeps that property:
the index proposes IDs, and SQL decides.

### Options

| Option | How it works | Database-reader leakage | Measured cost |
| - | - | - | - |
| 0. Today: Postgres FTS | GIN `tsvector` sidecar and `ts_rank_cd` | Every stemmed term, with positions: a near-verbatim copy | 9 / 256 ms median at 300k (distinctive / common words) |
| 1. Constrained alpha: decrypt and scan | No index. Open the agent's recent envelopes and match in the runtime, over a bounded window | None beyond row counts | Linear, about 2 ms per 1k messages scanned: 20 ms at 10k, about 600 ms at 300k |
| 2. Searchable representation: HMAC blind index | Keyed HMAC token per distinct term in a GIN `text[]`. Postgres matches and ranks by overlap | Term **equality and frequency** across rows. On Zipfian text, frequency analysis recovers common words and links rows that share rare words (names, tickets). No stemming unless terms are pre-stemmed | About the same as option 0: 10 / 212 ms median at 300k |
| **3. Runtime-owned protected index (recommended)** | A per-agent in-memory inverted index (BM25), persisted only as sealed segments. It returns candidate IDs to the existing SQL authorization | Segment count, sizes, and update timing (roughly message volume, already visible from row counts) | 0.2 / 2.0 ms median at 300k |

Embeddings need their own decision. Vectors can be inverted to approximate
text, so plaintext `memory_embeddings` is not an acceptable leakage. Under
option 3:

* **Vectors become envelopes.** Semantic search runs in the runtime over the
  SQL allowlist (`ListEligibleMemoryRevisionIDs` is already capped at 10,000).
* **That search is a brute-force cosine pass** over at most 10,000 vectors.
* **Cost (estimated, not measured):** 10,000 × 1,536 multiply-adds is about
  15.4M floating-point operations. Expect low single-digit milliseconds plus
  the decrypt of about 60 MiB of vectors, which can be cached per agent. L15
  must measure it.
* **The remote Qdrant backend sits outside the boundary.** It is excluded from
  the hosted protection scope or declared an exception.

### Measured prototype

The prototype lives in `internal/experimental/contentsearch`. Every line is in
`_test.go` files, so nothing imports it and it never ships in the binary. It
compares the options over the same deterministic synthetic corpus:

* a Zipfian 40k-word vocabulary;
* 38% stopwords;
* a mean of 24 words per message, with a 2% tail of long pastes.

It ranks 200 candidates per query, the same pool as `searchCandidatePool`
(`internal/graph/search.go:52`), and uses two query mixes:

* **distinctive:** two or three words from one message, each occurring in at
  most 1% of messages. This is what people search for.
* **common:** any words from one message, so very common words appear often.
  This is the stress case.

Each mix runs 200 queries per size, after one warm-up pass. Postgres is 14,
local, over TCP, using TEMP tables on one connection. The machine is an Apple
M3 Max.

Option 3's time includes opening the 200 candidate envelopes. Option 2's time
includes opening the returned envelopes. Option 0 reads plaintext bodies.

Latency per query in milliseconds, shown as p50 / p95:

| Messages in the agent | Query mix | Messages matched (p50 / p95) | 0. Postgres FTS | 2. Blind index | 1. Decrypt and scan | 3. Protected index |
| - | - | - | - | - | - | - |
| 10,000 | distinctive | 42 / 140 | 0.27 / 0.84 | 0.35 / 1.07 | 20.3 / 27.0 | **0.02 / 0.08** |
| 10,000 | common | 2,162 / 5,408 | 7.9 / 21.8 | 8.4 / 15.5 | 20.3 / 21.2 | **0.21 / 0.39** |
| 100,000 | distinctive | 452 / 1,198 | 4.1 / 10.2 | 4.2 / 10.1 | 197 / 203 | **0.14 / 0.20** |
| 100,000 | common | 17,280 / 53,868 | 175 / 648 | 91 / 278 | 199 / 223 | **1.0 / 2.6** |
| 300,000 | distinctive | 1,121 / 3,796 | 8.8 / 53.5 | 10.1 / 31.3 | 594 / 635 | **0.21 / 0.39** |
| 300,000 | common | 45,274 / 162,767 | 256 / 1,828 | 212 / 467 | 646 / 1,012 | **2.0 / 8.2** |

On-disk size (table plus indexes) for the two Postgres options:

| Option | 10k | 100k | 300k |
| - | - | - | - |
| 0. Postgres FTS | 6.9 MiB | 58 MiB | 169 MiB |
| 2. Blind index | 8.5 MiB | 70 MiB | 201 MiB |

The host was a shared developer machine with other work running. Postgres
latencies moved by up to 3× between runs: an earlier run with a 50-candidate
pool measured FTS at 240 / 619 ms for 300k common-word queries. Read the
ratios, not the absolute values.

Rebuild and persistence for option 3, one agent:

| Messages in the agent | Plaintext corpus | Sealed snapshot | Resident heap after load | Seal a full snapshot | Load (open + decode) | Rebuild from message envelopes |
| - | - | - | - | - | - | - |
| 10,000 | 1.8 MiB | 0.58 MiB | 3.0 MiB | 4.7 ms | 2.0 ms | 32 ms |
| 100,000 | 18.3 MiB | 4.2 MiB | 16.1 MiB | 22 ms | 11 ms | 320 ms |
| 300,000 | 55.0 MiB | 11.8 MiB | 41.1 MiB | 48 ms | 28 ms | 1.05 s |

Appending one message to an in-memory index costs 2.6 µs. Sealing a 1 KiB
envelope costs 0.61 µs, and opening one costs 0.46 µs.

**Reading the numbers:**

* **Search speed.** At these sizes the protected index is faster than
  Postgres FTS. `ts_rank_cd` has to score every matching row, while the
  in-memory BM25 walks postings. The synthetic corpus's most common words may
  be more frequent than in real conversation, so treat the "common" rows as a
  stress case.
* **Tokenizer.** The prototype's tokenizer is not Postgres's `english`
  configuration, so this is a **cost** comparison, not a quality comparison.
  Ranking quality, stemming parity, and real-data latency are L15 acceptance
  tests.
* **Option 1 grows linearly.** It is acceptable only with a bounded window
  (for example the most recent 10–20k messages per agent), and only as an
  explicitly disclosed capability reduction (Q01/Q09).
* **Option 2 buys little.** Its cost is about the same as option 0, and it
  leaks term frequency. Use keyed equality only where SQL needs exact
  equality on an identifier (finding 3), not for full text.
* **What is not measured:**
  * memory overhead at real scale across many agents;
  * multi-replica catch-up;
  * real English text;
  * the authorization join over IDs (unchanged from today, minus the text
    predicate);
  * vector search.

To rerun:

```sh theme={null}
MESH_TEST_DATABASE_URL=postgres://… \
  go test ./internal/experimental/contentsearch -run '^$' \
  -bench 'DecryptAndScan|ProtectedIndexQuery|Postgres' -benchtime 200x -benchmem
go test ./internal/experimental/contentsearch -run '^$' -bench 'Envelope|Snapshot|Append' -benchtime 10x -benchmem
```

Set `CONTENTSEARCH_EXPLAIN=1` to log the Postgres plans.

### Recommended design (L15)

1. **Scope and cost.** One protected index per agent and corpus (`messages`,
   `memory`), held by the runtime process that serves search. Resident cost
   is listed above. Agents are loaded lazily and evicted when idle.
2. **Persistence.** The index is persisted as **sealed segments** in a new
   table:

   * columns: `agent_id`, `corpus`, `segment_seq`, `through_watermark`,
     `envelope`;
   * the envelope uses the `index` purpose class.

   New messages are appended in memory and flushed as small delta segments.
   Compaction merges segments and drops tombstones.
3. **Startup and catch-up.** On start, or on another replica, the runtime
   loads the segments, then catches up. It decrypts and scans the messages
   after the newest watermark, which is a short tail (option 1's cost over
   that tail only).
4. **Restore.** If segments are missing or unreadable, the runtime rebuilds
   from the envelopes (cost listed above). It never falls back to plaintext.
5. **Deletion.** Message deletion, cascades, retention purges, and memory
   forget or soft-delete write tombstones. Compaction removes them. The SQL
   authorization join still rejects a deleted ID that remains in an
   unflushed segment.
6. **Query path.** The order must match today's: `SearchConversationHistory`
   applies exclusions, time range, speaker, evidence, and audience **before**
   its `LIMIT` (`db/queries/conversation_search.sql`), and only the lineage
   walk after it. Truncating the index's ranking first would let ineligible
   high-ranking matches (old, private, excluded) fill the pool, return
   nothing when eligible matches sit below the cutoff, and let withheld rows
   shape the count and order.
   * None of those pre-limit predicates reads a body. SQL evaluates them
     first and yields the eligible ID set (perspective, exclusions, time
     range, speaker, evidence, audience).
   * The index ranks only within that set and returns the top 200, today's
     `searchCandidatePool`.
   * SQL runs the lineage walk over `me.id = ANY($ids)`, as today after the
     limit.
   * The runtime opens only the authorized rows and combines the lexical
     score with recency and `requester_spoke` as today.
   * With that order the candidate-pool recall tradeoff is unchanged: only
     the lineage walk can still shrink a full pool. The cost of the
     eligible-set query at the largest agent size is an L15 measurement; if
     it is too large, the index may carry the body-free filter fields
     (conversation, time, actor, direction) and evaluate them itself, with
     audience and evidence still decided in SQL, or keep fetching ranked
     pages until 200 eligible candidates are found. It must never truncate
     before filtering.
7. **Memory retrieval** uses the same index for lexical candidates. Semantic
   candidates come from encrypted vectors searched in the runtime over the SQL
   allowlist. `HydrateEligibleMemory` re-authorization is unchanged.
8. **Dreaming relevance** uses the memory index instead of
   `ts_rank_cd(to_tsvector(body))`.
9. **Drop the plaintext derivatives** after the index is verified:
   * `message_search_documents` and its trigger;
   * both GIN indexes on `memory_entries`;
   * plaintext `memory_embeddings` rows.

## Coverage and exceptions

**Proposed covered scope** for the database-reader claim, once L14–L16 are
complete. This is the "E" and "I" rows above.

* Message bodies, canonical bodies, attachment names and URLs, conversation
  titles, and actor and identity display names.
* Persona revisions.
* Memory entries, revisions, notes, and embeddings.
* Dreaming snapshots, proposals, and public sources.
* Model call request and response bodies, response content, and error
  detail; prompt snapshots.
* Tool arguments and observations; run checkpoints, task continuations,
  decision bodies and answers, progress text, and scratch.
* Browser screenshots.
* Work-queue payloads.
* Every commitment, intake, origin, inbox, notice, completion, evidence,
  effect, and source body.
* Schedule instructions.
* The search index and the vectors.

**Declared residual metadata.** This stays plaintext and must be stated in the
promise:

* IDs, foreign keys, timestamps, states, counts, token usage, costs, model
  names, and durations.
* Audience and evidence JSON, including **member lists of external user IDs**,
  which authorization evaluates in SQL.
* Slack, Telegram, and team IDs used for routing and dedupe.
* Phone numbers used as routing keys, until they move to keyed values plus an
  envelope copy (proposed in L14; Q09 decides whether that is in scope).
* Repository names; MCP endpoints; tool configuration defaults; operator
  emails; audit trail metadata; vault entry names; vault drop IPs; avatars.
* Ciphertext sizes, row counts, and write timing.
* Keyed HMAC equality values. These reveal which rows share a value, never
  the value.

**Separately stated boundaries:**

* Model, decision, and embedding providers; web, search, and browser
  providers; Slack, Telegram, and phone providers; MCP servers; effect tools.
* Execution sandboxes, including Modal snapshot images.
* Qdrant, if enabled.
* Logs, OTLP, and the error tracker, after the reductions in the inventory
  table.
* The control plane's management path.

**Not claimed:**

* protection from runtime or deploy authority;
* end-to-end or zero-knowledge encryption;
* "the hosting operator cannot access your messages";
* protection of backups or logs taken before migration, until those expire.

## Migration and old backups (L16)

1. **Before conversion.**
   * Capture a recovery point: a database snapshot or PITR mark, plus the
     escrowed KEK.
   * Verify the restore into an isolated instance.
   * Record every running Formation version that can write content.
2. **Compatible readers first.**
   * Release N reads both plaintext and envelope columns, and still writes
     plaintext.
   * Release N+1 writes envelopes only. Plaintext writers are gone, enforced
     by a coverage test and a column `CHECK` once the backfill is complete.
   * Rollback from N+1 is to N, which reads both. A binary older than N is
     **not** a safe rollback target, and the release tooling must refuse it.
3. **Backfill in resumable, bounded batches.**
   * Order by primary key, with a persisted cursor per table.
   * Each row update is guarded: `WHERE id = $1 AND <plaintext column> IS NOT
     NULL AND envelope IS NULL`, so a stale batch never overwrites a newer
     write.
   * A content-read check opens each batch after writing it.
   * Progress is reported per purpose.
4. **Indexes.**
   * Build the protected index from the envelopes.
   * Verify search parity on a sample of real queries (L15 acceptance).
   * Only then drop the tsvector sidecar, the memory GIN indexes, and the
     plaintext vectors.
5. **Remove plaintext.** Null the legacy columns, then drop them. Run
   `VACUUM` so old tuples leave the live heap.
   * **Old copies remain** in WAL, PITR history, branches, and backups until
     those expire. The plaintext lives on in them, not in the live database.
   * The claim is qualified, with a date, until the retention period has
     passed (Q11).
   * Exports already downloaded, logs already shipped, and provider-held
     copies are outside reach and are stated as such.
6. **Exercise failure modes:**
   * an interrupted or backlogged backfill;
   * a wrong or missing KEK (the instance refuses to start, with no
     regeneration);
   * rotation during backfill;
   * an isolated restore of the database plus KEK, with reads and search;
   * the same hostname, agents, Slack installation, and history throughout.

## Implementation slices

### L14: Content envelopes and key custody

* **Scope.**
  * A shared `internal/contentcrypt` package. It owns the envelope format,
    AAD construction, `content_keys`, the data-key cache, the seal budget,
    and `verify-content`.
  * KEK custody (A plus escrow; the control plane generates the KEK and
    escrows it).
  * Every E row in the inventory, through the shared package, including the
    raw-SQL writers.
  * Keyed replacements for the unkeyed hashes (H rows).
  * Error-text reductions in logs, spans, and the error tracker.
* **Acceptance tests.**
  * A canary string written through **every** writer type, including ingress
    for each connector, outbound replies, imports, dreaming, commitments,
    schedules, tool results, checkpoints, and continuations, never appears in
    a database-only `pg_dump`.
  * Neither does any unwrapped key.
  * A wrong, missing, or tampered key, nonce, or tag fails closed. An envelope
    moved to another row, purpose, or instance fails.
  * KEK rewrap and data-key rotation work across a restart.
  * A missing KEK at boot stops the instance with a recovery error and never
    regenerates.
  * Key access is logged separately from content reads, and keys never appear
    in telemetry.

### L15: Protected retrieval

* **Scope.**
  * The per-agent protected index (messages and memory), with sealed segments,
    tombstones, compaction, catch-up, and rebuild.
  * Encrypted vectors with runtime cosine search over the SQL allowlist.
  * Remove the plaintext tsvector, the GIN indexes, and the vectors.
  * Dreaming relevance moves onto the index.
* **Acceptance tests.**
  * Search quality on representative data against today's FTS: recall\@k on
    a labeled query set, and an agreed tolerance.
  * p95 latency at the largest agent size.
  * A private DM hit never appears as a hit or neighbor in a channel search,
    even when it ranks first.
  * Results do not differ in count or order in a way that reveals a withheld
    row.
  * With more than 200 ineligible matches outranking an eligible one, the
    eligible match is still returned (filters before the candidate limit).
  * No cross-agent candidates.
  * Deletion, forget, and retention purge remove results.
  * Rebuild after segment loss, and restart.
  * A database-only dump contains no lexeme, stem, or vector derived from
    seeded private text.

### L16: Migrate existing instances

* **Scope.**
  * Compatible readers (release N), then the envelope-only release (N+1).
  * Resumable guarded backfills with progress reporting.
  * Index build and cut-over, and plaintext removal.
  * Control-plane rollout and rollback gating on versions.
  * A historical-copy schedule.
* **Acceptance tests.**
  * Interrupt and resume, and a concurrent writer during backfill: no loss and
    no stale overwrite.
  * Rollback N+1 → N works, while rollback past N is refused.
  * The same hostname, agents, Slack installation, and history survive.
  * Restore the database plus KEK into an isolated instance, then read and
    search.
  * A missing key never triggers regeneration.
  * A dated statement of when historical plaintext copies expire.

## Authorization review

Encryption decrypts for every permitted runtime read, so these gaps become
plaintext exposure. This review was read-only and traced code at `49219a71`.

* **Confirmed gaps:**
  * **F1. Medium–High.** The management-scope API token (the control plane)
    can read memory and persona text.
  * **F2. Medium.** Content reads are never audited.
  * **F3. High relevance.** Every user on a hosted install is an owner.
* **No IDOR found.** Every caller-supplied conversation, message, snapshot,
  decision, memory, commitment, schedule, source, or artifact ID is bound to
  the slug's agent in SQL. A foreign ID answers 404.
* **Retrieval checks happen before ranking.** Conversation search and memory
  retrieval apply perspective, evidence, and audience before rank and limit,
  and lineage after the candidate limit. Recall can drop; nothing leaks.
* **No unauthenticated route returns stored content.** This covers vault
  drops, OAuth callbacks, and webhooks.

**F1. The management token can read memory and persona.**

* `managementProvisioningRoutes` (`internal/admin/session.go:302`) allows
  `GET`/`PATCH /api/agents/{slug}`, enable, and disable.
* Each of those returns `agentDetail`, which embeds the 10 most recently
  updated memory bodies (`internal/admin/agent_profile.go:528`, `:603`).
* The list also allows `GET /api/agents/{slug}/persona`, which returns every
  revision's text.
* The service user defaults to owner (`db/migrations/0050_operator_roles.sql`),
  so this allowlist is its only limit.
* The control plane holds the derivation key for every instance's token. Its
  current client calls only four operations
  (`apps/provisioner/src/management.ts:70`), but the token's reach is what
  counts.
* The allowlist's own comment says it exists so the control plane cannot
  reach conversation history. Memory reached through `agentDetail` defeats
  that intent.
* A `dry_run` memory import also reports an exact-body duplicate
  (`internal/admin/memory_import.go:362`), which confirms a guessed memory
  text.
* **Fix before L14:**
  * Return a content-free agent view to management callers.
  * Make persona write-only for that scope.
  * Report import duplicates as counts only.

**F2. Content reads are not audited.** `audited()` skips `GET`
(`internal/admin/audit.go:106`). Fix: add a content-read audit record (actor,
token, route, IDs; never bodies) emitted where content is decrypted for an API
response. This covers the trace, prompt snapshot, decision call,
conversation, actor, export, memory, revision, dreaming, commitment, and
schedule routes. This is also the per-read audit the support grant design
(L17, Q10) needs.

**F3. Hosted users are all owners.** `CreateExternalUser` (`db/queries/users.sql:32`)
takes the default role `owner`. External sign-in checks only the
`mesh:admin` capability, and there is no role claim. Per-agent operator
isolation therefore does not exist on hosted installs. Fix: map control-plane
membership and role to Formation role and agent grants (L04/L12) before relying on
it for consent (Q10).

**Lower severity:**

* **F4.** `ListScratchEntriesForRun` has no agent filter
  (`db/queries/scratch.sql:69`). It has no caller today, so either add the
  filter or delete it.
* **F5.** A `read`-scope API token reaches every GET its user's role allows,
  including traces and export. Document this, or add a content-free read
  scope.

**Hardening found in passing** (not authorization, but in scope for the same
release):

* `secrets` AAD does not bind owner, kind, or row, so a credential row can be
  moved between agents.
* `mcp_oauth_authorizations` is omitted from `rotate-key` and
  `verify-secrets`.
* `span.RecordError` exports error text (`internal/app/ingress_dispatch.go:91`).
* Error tracker events carry up to 1 KiB of content-bearing error text.

**Tests L14/L15 should add:**

1. An agent\_operator receives 404 on an ungranted agent for the trace, prompt
   snapshot, decision calls, runs, actor, conversation, memory revisions and
   grants, commitments, schedules, dreaming, commitment sources, and browser
   artifacts. Today only persona, memory, export, avatar, and Slack setup are
   covered (`internal/admin/operators_roles_test.go:90`).
2. A route-table test that walks every `/api/agents/{slug}/` pattern and
   asserts `requireAgentAccess` or `requireOwner`.
3. Foreign-ID 404s for the snapshot, decision, conversation, message, memory
   revisions, and grants.
4. The management token receives no memory body from any allowlisted route
   and cannot read persona text. This **fails today**.
5. A content-read audit row is written for trace, prompt-snapshot, and export
   GETs. This **fails today**.
6. A hosted user without a grant cannot read agent content. This **fails
   today**, by design, until F3.

## Decisions needed

| Question | What this document proposes | Blocks |
| - | - | - |
| **Q01** | Invite a small, disclosed alpha on today's storage, which is readable by privileged operators. Ship L14–L16 before broad launch. If encryption is required before the alpha, use option 1 (bounded decrypt-and-scan) as the temporary search, and disclose the reduction | Alpha scope, privacy copy |
| **Q09** | Covered scope and residual metadata as listed above. Custody: A plus escrow now, with the rewrap path to KMS (B) before broad launch. Search: protected index (option 3) and encrypted vectors. Reject the blind index for full text. Exclude Qdrant from hosted protection, or declare it | L14/L15 scope and estimates |
| **Q10** | Content-read audit (F2) and a per-conversation grant model, built on the decrypt-point audit hook | L17 |
| **Q11** | Default body retention windows, PITR and backup retention, KEK escrow location and fingerprint labeling, RPO and RTO, and the date by which historical plaintext copies expire | L16, the claim wording, crypto-shred statements |

## Related

* [Access control](https://docs.formation.texturehq.com/access-control.md)
* [Secrets and recovery](https://docs.formation.texturehq.com/secrets-and-recovery.md)
* [Data model](https://docs.formation.texturehq.com/data-model.md)
* [Conversation search](https://docs.formation.texturehq.com/runtime/conversation-search.md)
* [Memory retrieval](https://docs.formation.texturehq.com/runtime/memory-retrieval.md)
* [Memory privacy](https://docs.formation.texturehq.com/runtime/memory-privacy.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.