Skip to main content
Status: proposal. Nothing in this document is implemented, and nothing here changes runtime behavior. Every recommendation is a proposal that needs the owner decisions listed in Decisions needed: Q01 (alpha scope), Q09 (promise scope, key custody, and search leakage), Q10 (support consent), and Q11 (retention, backups, and RPO/RTO). The questions are defined in the control plane’s docs/hosted-launch-strategy.md. This document answers backlog item L03 in the control plane’s docs/hosted-launch-backlog.md. It is the input to L14, L15, and L16.
Access control states the current position: credentials are encrypted, while conversation bodies, memory, prompts, and traces are readable by anyone who can read the database. This document maps every place that content lives today and proposes how to change that for hosted instances. The goal is a promise that a database-only reader cannot recover customer content. Runtime and deploy authority remain trusted. All file references are path:line at commit 49219a71, relative to the repository root. The schema was checked by applying every migration to a scratch Postgres and listing information_schema.columns. Writers and readers were traced to the sqlc query and its Go call site, or to the raw SQL that bypasses sqlc.

Summary

  • Content has many copies, not one. One inbound message is persisted as message_events.body and canonical_body, then as stemmed lexemes in message_search_documents.tsv, as JSON in work_items.payload, and as an intake request. That request can be copied into commitments.objective, commitment_origins.original_request, and commitment_inbox.payload. The model request built from it (persona, memory, history, and tool results) is written to model_calls.request_body. It is also written to prompt_snapshots.prompt, run_checkpoints.payload, task_continuations.checkpoint, and, when model routing is on, decision_calls.request_body. Encrypting one column protects nothing.
  • Retention is mostly unbounded. Body pruning is opt-in (retention.model_call_body_days, default 0 = keep forever) and partial. Many content columns have no pruning at all; see Cross-cutting findings.
  • Search is part of the boundary. Message and memory lexemes sit in GIN indexes. Memory embeddings sit in memory_embeddings as real[]. Encrypting the bodies without moving search leaves a near-verbatim copy behind.
  • Proposed design:
    • AES-256-GCM envelopes, each carrying a version and a key ID. Additional authenticated data (AAD) binds each envelope to its instance, purpose, and record.
    • Per-instance data keys stored wrapped in Postgres. The key that wraps them is held outside Postgres, separate from MESH_AGENT_MASTER_KEY.
    • Search moves into a runtime-owned protected index, persisted only as sealed segments. Authorization stays in SQL, unchanged.
    • The prototype measured the index for an agent with 10k–300k messages:
      • 0.02–0.2 ms at the median for distinctive-word queries;
      • up to 2 ms median and 8 ms p95 when queries hit very common words.
      Today’s Postgres full-text search on the same corpus measured 0.3–9 ms and up to 256 ms median respectively. Details are in Measured prototype.
  • Fix these authorization issues before encryption (details in Authorization review):
    • The control plane’s management token can read memory and persona text.
    • Content reads are not audited.
    • Hosted (external-auth) users are all owners.

Threat models

Database reader

The database reader is anyone or anything that gets the database without the running Formation runtime. Examples:
  • a leaked dump, branch, or clone;
  • a point-in-time restore or backup copy;
  • a stolen database credential, or a mistakenly granted one;
  • a Neon support or infrastructure path;
  • a logical replication target;
  • a misdirected pg_dump.
Promise (proposed). Once L14–L16 are complete for an instance, this reader cannot read:
  • covered content columns, and their derived copies (search terms, embeddings, digests);
  • the keys that would decrypt them.
The reader can still see the residual metadata listed in Coverage and exceptions. That includes:
  • IDs and relationships, timestamps, and lifecycle states;
  • token counts and costs;
  • audience membership lists;
  • external identifiers needed for routing;
  • ciphertext sizes and update timing.
The reader can tamper, for example by deleting rows, replaying older ciphertext of the same record, or corrupting envelopes. AEAD with record-bound AAD turns substitution and corruption into decryption failures. It does not prevent deletion or rollback of a whole record to an older valid envelope of the same record and purpose. This design does not try to give the database integrity against a writer.

Operator and deploy authority (trusted)

Runtime and deploy authority remain trusted. That includes:
  • any principal of the hosting operator who can deploy code to the instance;
  • anyone who can read its Fly secrets or exec into the machine;
  • anyone who can change its image;
  • anyone who holds an IAM grant that lets the runtime identity unwrap keys.
The runtime must decrypt content to build model requests. Code that authority deploys can capture plaintext and keys. Neither a container secret nor a KMS grant prevents that. Fly documents that deploy access can extract application secrets. What the design does provide against operators:
  • Separation of duties. Database access alone (support querying Postgres, a database admin, a Neon console session) no longer yields content.
  • Evidence. KMS unwrap logs record key use. A content-read audit records each decrypt-to-API read (see F2 in Authorization review). KMS logs do not replace the per-read audit, because keys are cached.
  • Customer-key revocation path. A later customer-managed wrapping key can stop future unwraps. It cannot retract plaintext already processed, cached, or sent to providers.
The authenticated operator API is not a database reader. It is a runtime read, so it decrypts. Today every owner, every agent operator granted to the agent, every read-scope API token, and (see F1) the control plane’s management token can read content through it. Encryption does not change that. Authorization does.

Out of scope

The following are separate boundaries. They need their own statements in the customer promise:
  • Providers that receive content by design: Slack, Telegram, the phone provider, model and decision providers, embedding providers, web search, read and browser providers, MCP servers, effect tools, and execution sandboxes. See Outside Postgres.
  • Customer repositories.
  • A stronger operator-exclusion claim, which would need customer-controlled runtimes or attested confidential execution (Q15).

Data-flow inventory

Classes
  • CONTENT: customer or model text, prompts, tool input and output, files, persona.
  • DERIVED: computed from content and possibly invertible. Examples: tsvectors, embeddings, unkeyed hashes, previews.
  • PII-META: names, handles, emails, phone numbers, URLs, filenames, external IDs, audience member lists.
  • SECRET: credentials, or values derived from them.
  • OPS: IDs, timestamps, states, counts.
Protection column. The “Proposed” column says what L14–L16 should do with each item:
  • E: encrypt as an envelope.
  • I: move into the protected index.
  • H: replace with a keyed HMAC (a blind key).
  • K: keep plaintext as declared residual metadata.
  • R: reduce or stop retaining.
  • X: drop.
Rows are grouped by area. Columns that are only IDs, states, or timestamps are omitted unless they matter to the design.

Conversation graph and intake

All three ingress paths go through graph.Store.RecordMessage (internal/graph/graph.go:110):
  • inbound messages via Runtime.RecordInbound (internal/app/runtime.go:310);
  • outbound replies via Runtime.executeTurn (internal/app/runtime.go:711);
  • the no-runtime Slack path via internal/app/slack_ingress.go:882.

Run ledger, model, and tools

The model request is assembled by completeModelOnce (internal/turn/turn.go:2459). ledgerRequestBody (internal/turn/turn.go:2482) returns the exact wire bytes, with images replaced by a name, size, and hash prefix (internal/model/audit.go:10). Those bytes are persisted before the provider call.

Memory, retrieval, and dreaming

Commitments and schedules

Most of these writes use hand-written SQL in internal/commitment, internal/worksource, and internal/app, not sqlc. Encryption at the internal/store layer alone would miss them. Retention for this area runs inside maintainRetention (internal/app/retention_maintenance.go:72). It covers five prune queries and uses the same opt-in window. Rewriting a body clears bodies_pruned_at again (internal/commitment/criteria.go:216, internal/commitment/worksource_scope.go:51, internal/worksource/reconcile.go:415).

Configuration and credentials

Identity, operators, and audit

Outside Postgres

Cross-cutting findings

  1. Copies multiply. Count the stores per message, memory, and model request above before claiming coverage. L14 needs a coverage check that seeds a canary string through every writer and greps a database-only dump.
  2. Retention is opt-in and partial. With the default retention.model_call_body_days = 0, nothing is pruned. Even with a window set, the following stay forever:
    • response_content, prompt_snapshots, tool arguments and observations, progress text, task_continuations, commitments.objective and attempt_reply, commitment_effects.observation, schedules, and every error column;
    • message and memory bodies, which have no retention at all.
    Encryption does not reduce exposure to runtime readers; less retention does. Propose defaults under Q11.
  3. Unkeyed hashes act as equality oracles. Short or predictable plaintext can be confirmed by hashing guesses. Affected columns:
    • objective_hash, operation_hash, evidence_key, creation_key, and source_key;
    • prompt_digest and the prompt_snapshots.digest column;
    • the md5 dream dedupe index;
    • model_routing_profiles.binding;
    • secrets.redacted_hint.
    Every such column that guards uniqueness must become a keyed HMAC with the same equality semantics.
  4. SQL evaluates JSON and text that look like content. Authorization reads message_events.metadata->'memory_audience', the evidence tables, and audience columns. Some predicates compare audience JSON by equality. commitment_source_observations.query is matched by equality, and workspace_checkpoint->>'run_id' is indexed. These must stay plaintext, or become keyed equality values. That is why the audience membership lists are declared residual metadata rather than encrypted.
  5. Raw SQL writers are common. These areas have writers outside sqlc:
    • identity (internal/identity);
    • commitments and work sources (most commitment tables);
    • web browser (internal/webbrowser);
    • model routing (internal/modelrouting/store.go);
    • three Postgres triggers that copy message_events.body and metadata (0038, 0039, 0090).
    The envelope must live in a shared package that each writer calls. The triggers that derive from body must be dropped or rewritten.
  6. The dormant worker_tasks table and the two dead query paths (ListScratchEntriesForRun, ListTurnDisposalsForMessage) should be covered or removed before a coverage claim, so they do not become unseen writers later.

Proposed envelope and keys

Format

Store each covered value as a bytea envelope:
  • AEAD: AES-256-GCM from the Go standard library, the primitive internal/secrets already uses. It is hardware-accelerated on arm64 and amd64. The prototype measured 1 KiB at 0.61 µs to seal and 0.46 µs to open (Apple M3 Max).
  • Nonce limit. Random 96-bit nonces are limited to about 2³² seals per key. Each data key therefore carries a seal budget well under that (for example 2³⁰), and a time limit (for example 90 days). A new data key is minted when either is reached. XChaCha20-Poly1305 would remove the budget, but it adds a dependency and a second primitive. Prefer the budget.
  • AAD binds the envelope to where it may be opened, and to its own header:
    • instance_id is the hosted instance’s stable ID. External-auth installs already receive it as MESH_INSTANCE_ID (internal/config/config.go:480). L14 records it in a descriptor row checked at boot, so a database restored under another instance ID refuses to open rather than misbinding. An isolated restore drill therefore runs under the original instance ID.
    • purpose names the table and column, for example message_events.body or model_calls.request_body.
    • record_key is the row’s immutable primary key, plus the revision number for versioned rows.
    With this binding, an envelope copied to another row, another column, or another instance’s database fails authentication. It does not open in the wrong place. The prototype tests each substitution case: TestEnvelopeBindsInstancePurposeAndRecord in internal/experimental/contentsearch/envelope_test.go.
  • Mutable fields stay out of the AAD, so state transitions never re-encrypt.
  • Readers fail closed. A row that fails to open is reported as corrupt. It is never shown as empty or silently skipped. This matches the verify-secrets posture in Secrets and recovery.
  • JSON columns: encrypt the whole value, unless SQL evaluates part of it (finding 4 above). In that case, split the column: keep the operational keys in a plaintext JSON column and put the rest in an envelope column.

Key hierarchy

  • content_keys table (new). Columns: id, purpose_class, wrapped_key, kek_id, created_at, retire_after, seal_count, and status.
    • kek_id is a fingerprint, never key material.
    • seal_count is the durable, cross-replica account of the key’s seal budget. A replica never seals against a local count alone: it reserves a block of seals with one atomic conditional UPDATE on the row (matching only while the key is active, within its age limit, and the block fits the budget) and seals from that block. A crash forfeits the unused rest of a block, so the count can over-state seals but never under-state them, and replicas together cannot exceed the budget. When a reservation matches nothing, the class rotates to a new key.
    • A database-only dump holds wrapped data keys and ciphertext. Without the KEK it holds neither plaintext nor usable keys.
  • Separate from credentials. The KEK is not MESH_AGENT_MASTER_KEY and is not derived from it. Credential rotation (mesh rotate-key) and content rotation are independent operations with independent recovery copies, as the strategy requires.
  • Purpose classes (proposal: conversation, run_ledger, memory, work, index) allow later per-class rotation or retention without changing the format. Per-audience or per-actor keys from the earlier access-control proposal are not required for tenant storage protection, and are not proposed here.
  • Data keys in memory. The runtime unwraps data keys at boot and on first use, and caches them in memory. Each covered value is opened only when needed. Keys never appear in logs, telemetry, install_settings, or API responses. Key IDs and status may (L14).

Key custody

Proposal (needs Q09 and Q11):
  1. Ship L14 with A plus escrow.
  2. Design the kek_id and rewrap path so that B can replace A without touching content.
  3. Treat C as a later enterprise option.
Both A and B satisfy the database-reader promise. Neither supports a claim that the hosting operator cannot access content. Before broad launch, B adds independent unwrap audit, a deletion schedule, and recovery outside Fly. Whichever is chosen, the provisioner must never regenerate a missing KEK. A missing key is a recovery incident, not a fresh install. This matches the existing RecoveryRequiredError posture for the master key.

Rotation, recovery, and retention

  • KEK rotation: rewrap every content_keys row in one transaction, then retire the old KEK only after every retained backup that references it has expired. Label KEK escrow copies by fingerprint, as Secrets and recovery already requires for the master key.
  • Data-key rotation:
    • New writes use the current data key. Old envelopes stay readable through their key ID.
    • Re-encrypting old content is a bounded background job, needed only after a suspected data-key exposure or to retire a key.
    • It uses the same resumable machinery as the L16 backfill.
  • Compromise. A suspected KEK exposure means rotating the KEK, minting new data keys, and re-encrypting. It cannot recall what the holder already decrypted.
  • Recovery is a restored pair: the database plus its matching KEK. The backup drill in Secrets and recovery extends to restoring both into an isolated instance, then running a content check (verify-content, opening a sample of every purpose) and search.
  • Instance deletion and crypto-shredding. Destroying every KEK copy renders all retained backups of that instance unreadable once they expire. This is the one place encryption shortens retention in practice. State it only after the escrow and backup schedules are confirmed (Q11).
  • Record deletion still deletes rows. Backups keep the envelope until they expire. That is unreadable without the KEK, and readable by anyone who holds both.

Retrieval

What breaks

Encrypting the bodies breaks every lexical and vector path that Postgres evaluates today:
  • SearchConversationHistory and its context and exclusion queries (db/queries/conversation_search.sql);
  • SearchEligibleMemory, HasUnknownAudienceWithheld, and DreamRelevantMemories;
  • the pgvector query (internal/memory/pgvector/pgvector.go:29);
  • the mesh_index_message_search trigger, which would index ciphertext.
Authorization does not depend on the bodies. It is evaluated over audience JSON, evidence, lineage, and IDs. Search and memory retrieval already treat the index as a source of candidates and re-authorize in SQL (db/migrations/0090_conversation_search.sql header, internal/memory/retrieval.go:223). Every option below keeps that property: the index proposes IDs, and SQL decides.

Options

Embeddings need their own decision. Vectors can be inverted to approximate text, so plaintext memory_embeddings is not an acceptable leakage. Under option 3:
  • Vectors become envelopes. Semantic search runs in the runtime over the SQL allowlist (ListEligibleMemoryRevisionIDs is already capped at 10,000).
  • That search is a brute-force cosine pass over at most 10,000 vectors.
  • Cost (estimated, not measured): 10,000 × 1,536 multiply-adds is about 15.4M floating-point operations. Expect low single-digit milliseconds plus the decrypt of about 60 MiB of vectors, which can be cached per agent. L15 must measure it.
  • The remote Qdrant backend sits outside the boundary. It is excluded from the hosted protection scope or declared an exception.

Measured prototype

The prototype lives in internal/experimental/contentsearch. Every line is in _test.go files, so nothing imports it and it never ships in the binary. It compares the options over the same deterministic synthetic corpus:
  • a Zipfian 40k-word vocabulary;
  • 38% stopwords;
  • a mean of 24 words per message, with a 2% tail of long pastes.
It ranks 200 candidates per query, the same pool as searchCandidatePool (internal/graph/search.go:52), and uses two query mixes:
  • distinctive: two or three words from one message, each occurring in at most 1% of messages. This is what people search for.
  • common: any words from one message, so very common words appear often. This is the stress case.
Each mix runs 200 queries per size, after one warm-up pass. Postgres is 14, local, over TCP, using TEMP tables on one connection. The machine is an Apple M3 Max. Option 3’s time includes opening the 200 candidate envelopes. Option 2’s time includes opening the returned envelopes. Option 0 reads plaintext bodies. Latency per query in milliseconds, shown as p50 / p95: On-disk size (table plus indexes) for the two Postgres options: The host was a shared developer machine with other work running. Postgres latencies moved by up to 3× between runs: an earlier run with a 50-candidate pool measured FTS at 240 / 619 ms for 300k common-word queries. Read the ratios, not the absolute values. Rebuild and persistence for option 3, one agent: Appending one message to an in-memory index costs 2.6 µs. Sealing a 1 KiB envelope costs 0.61 µs, and opening one costs 0.46 µs. Reading the numbers:
  • Search speed. At these sizes the protected index is faster than Postgres FTS. ts_rank_cd has to score every matching row, while the in-memory BM25 walks postings. The synthetic corpus’s most common words may be more frequent than in real conversation, so treat the “common” rows as a stress case.
  • Tokenizer. The prototype’s tokenizer is not Postgres’s english configuration, so this is a cost comparison, not a quality comparison. Ranking quality, stemming parity, and real-data latency are L15 acceptance tests.
  • Option 1 grows linearly. It is acceptable only with a bounded window (for example the most recent 10–20k messages per agent), and only as an explicitly disclosed capability reduction (Q01/Q09).
  • Option 2 buys little. Its cost is about the same as option 0, and it leaks term frequency. Use keyed equality only where SQL needs exact equality on an identifier (finding 3), not for full text.
  • What is not measured:
    • memory overhead at real scale across many agents;
    • multi-replica catch-up;
    • real English text;
    • the authorization join over IDs (unchanged from today, minus the text predicate);
    • vector search.
To rerun:
Set CONTENTSEARCH_EXPLAIN=1 to log the Postgres plans.
  1. Scope and cost. One protected index per agent and corpus (messages, memory), held by the runtime process that serves search. Resident cost is listed above. Agents are loaded lazily and evicted when idle.
  2. Persistence. The index is persisted as sealed segments in a new table:
    • columns: agent_id, corpus, segment_seq, through_watermark, envelope;
    • the envelope uses the index purpose class.
    New messages are appended in memory and flushed as small delta segments. Compaction merges segments and drops tombstones.
  3. Startup and catch-up. On start, or on another replica, the runtime loads the segments, then catches up. It decrypts and scans the messages after the newest watermark, which is a short tail (option 1’s cost over that tail only).
  4. Restore. If segments are missing or unreadable, the runtime rebuilds from the envelopes (cost listed above). It never falls back to plaintext.
  5. Deletion. Message deletion, cascades, retention purges, and memory forget or soft-delete write tombstones. Compaction removes them. The SQL authorization join still rejects a deleted ID that remains in an unflushed segment.
  6. Query path. The order must match today’s: SearchConversationHistory applies exclusions, time range, speaker, evidence, and audience before its LIMIT (db/queries/conversation_search.sql), and only the lineage walk after it. Truncating the index’s ranking first would let ineligible high-ranking matches (old, private, excluded) fill the pool, return nothing when eligible matches sit below the cutoff, and let withheld rows shape the count and order.
    • None of those pre-limit predicates reads a body. SQL evaluates them first and yields the eligible ID set (perspective, exclusions, time range, speaker, evidence, audience).
    • The index ranks only within that set and returns the top 200, today’s searchCandidatePool.
    • SQL runs the lineage walk over me.id = ANY($ids), as today after the limit.
    • The runtime opens only the authorized rows and combines the lexical score with recency and requester_spoke as today.
    • With that order the candidate-pool recall tradeoff is unchanged: only the lineage walk can still shrink a full pool. The cost of the eligible-set query at the largest agent size is an L15 measurement; if it is too large, the index may carry the body-free filter fields (conversation, time, actor, direction) and evaluate them itself, with audience and evidence still decided in SQL, or keep fetching ranked pages until 200 eligible candidates are found. It must never truncate before filtering.
  7. Memory retrieval uses the same index for lexical candidates. Semantic candidates come from encrypted vectors searched in the runtime over the SQL allowlist. HydrateEligibleMemory re-authorization is unchanged.
  8. Dreaming relevance uses the memory index instead of ts_rank_cd(to_tsvector(body)).
  9. Drop the plaintext derivatives after the index is verified:
    • message_search_documents and its trigger;
    • both GIN indexes on memory_entries;
    • plaintext memory_embeddings rows.

Coverage and exceptions

Proposed covered scope for the database-reader claim, once L14–L16 are complete. This is the “E” and “I” rows above.
  • Message bodies, canonical bodies, attachment names and URLs, conversation titles, and actor and identity display names.
  • Persona revisions.
  • Memory entries, revisions, notes, and embeddings.
  • Dreaming snapshots, proposals, and public sources.
  • Model call request and response bodies, response content, and error detail; prompt snapshots.
  • Tool arguments and observations; run checkpoints, task continuations, decision bodies and answers, progress text, and scratch.
  • Browser screenshots.
  • Work-queue payloads.
  • Every commitment, intake, origin, inbox, notice, completion, evidence, effect, and source body.
  • Schedule instructions.
  • The search index and the vectors.
Declared residual metadata. This stays plaintext and must be stated in the promise:
  • IDs, foreign keys, timestamps, states, counts, token usage, costs, model names, and durations.
  • Audience and evidence JSON, including member lists of external user IDs, which authorization evaluates in SQL.
  • Slack, Telegram, and team IDs used for routing and dedupe.
  • Phone numbers used as routing keys, until they move to keyed values plus an envelope copy (proposed in L14; Q09 decides whether that is in scope).
  • Repository names; MCP endpoints; tool configuration defaults; operator emails; audit trail metadata; vault entry names; vault drop IPs; avatars.
  • Ciphertext sizes, row counts, and write timing.
  • Keyed HMAC equality values. These reveal which rows share a value, never the value.
Separately stated boundaries:
  • Model, decision, and embedding providers; web, search, and browser providers; Slack, Telegram, and phone providers; MCP servers; effect tools.
  • Execution sandboxes, including Modal snapshot images.
  • Qdrant, if enabled.
  • Logs, OTLP, and the error tracker, after the reductions in the inventory table.
  • The control plane’s management path.
Not claimed:
  • protection from runtime or deploy authority;
  • end-to-end or zero-knowledge encryption;
  • “the hosting operator cannot access your messages”;
  • protection of backups or logs taken before migration, until those expire.

Migration and old backups (L16)

  1. Before conversion.
    • Capture a recovery point: a database snapshot or PITR mark, plus the escrowed KEK.
    • Verify the restore into an isolated instance.
    • Record every running Formation version that can write content.
  2. Compatible readers first.
    • Release N reads both plaintext and envelope columns, and still writes plaintext.
    • Release N+1 writes envelopes only. Plaintext writers are gone, enforced by a coverage test and a column CHECK once the backfill is complete.
    • Rollback from N+1 is to N, which reads both. A binary older than N is not a safe rollback target, and the release tooling must refuse it.
  3. Backfill in resumable, bounded batches.
    • Order by primary key, with a persisted cursor per table.
    • Each row update is guarded: WHERE id = $1 AND <plaintext column> IS NOT NULL AND envelope IS NULL, so a stale batch never overwrites a newer write.
    • A content-read check opens each batch after writing it.
    • Progress is reported per purpose.
  4. Indexes.
    • Build the protected index from the envelopes.
    • Verify search parity on a sample of real queries (L15 acceptance).
    • Only then drop the tsvector sidecar, the memory GIN indexes, and the plaintext vectors.
  5. Remove plaintext. Null the legacy columns, then drop them. Run VACUUM so old tuples leave the live heap.
    • Old copies remain in WAL, PITR history, branches, and backups until those expire. The plaintext lives on in them, not in the live database.
    • The claim is qualified, with a date, until the retention period has passed (Q11).
    • Exports already downloaded, logs already shipped, and provider-held copies are outside reach and are stated as such.
  6. Exercise failure modes:
    • an interrupted or backlogged backfill;
    • a wrong or missing KEK (the instance refuses to start, with no regeneration);
    • rotation during backfill;
    • an isolated restore of the database plus KEK, with reads and search;
    • the same hostname, agents, Slack installation, and history throughout.

Implementation slices

L14: Content envelopes and key custody

  • Scope.
    • A shared internal/contentcrypt package. It owns the envelope format, AAD construction, content_keys, the data-key cache, the seal budget, and verify-content.
    • KEK custody (A plus escrow; the control plane generates the KEK and escrows it).
    • Every E row in the inventory, through the shared package, including the raw-SQL writers.
    • Keyed replacements for the unkeyed hashes (H rows).
    • Error-text reductions in logs, spans, and the error tracker.
  • Acceptance tests.
    • A canary string written through every writer type, including ingress for each connector, outbound replies, imports, dreaming, commitments, schedules, tool results, checkpoints, and continuations, never appears in a database-only pg_dump.
    • Neither does any unwrapped key.
    • A wrong, missing, or tampered key, nonce, or tag fails closed. An envelope moved to another row, purpose, or instance fails.
    • KEK rewrap and data-key rotation work across a restart.
    • A missing KEK at boot stops the instance with a recovery error and never regenerates.
    • Key access is logged separately from content reads, and keys never appear in telemetry.

L15: Protected retrieval

  • Scope.
    • The per-agent protected index (messages and memory), with sealed segments, tombstones, compaction, catch-up, and rebuild.
    • Encrypted vectors with runtime cosine search over the SQL allowlist.
    • Remove the plaintext tsvector, the GIN indexes, and the vectors.
    • Dreaming relevance moves onto the index.
  • Acceptance tests.
    • Search quality on representative data against today’s FTS: recall@k on a labeled query set, and an agreed tolerance.
    • p95 latency at the largest agent size.
    • A private DM hit never appears as a hit or neighbor in a channel search, even when it ranks first.
    • Results do not differ in count or order in a way that reveals a withheld row.
    • With more than 200 ineligible matches outranking an eligible one, the eligible match is still returned (filters before the candidate limit).
    • No cross-agent candidates.
    • Deletion, forget, and retention purge remove results.
    • Rebuild after segment loss, and restart.
    • A database-only dump contains no lexeme, stem, or vector derived from seeded private text.

L16: Migrate existing instances

  • Scope.
    • Compatible readers (release N), then the envelope-only release (N+1).
    • Resumable guarded backfills with progress reporting.
    • Index build and cut-over, and plaintext removal.
    • Control-plane rollout and rollback gating on versions.
    • A historical-copy schedule.
  • Acceptance tests.
    • Interrupt and resume, and a concurrent writer during backfill: no loss and no stale overwrite.
    • Rollback N+1 → N works, while rollback past N is refused.
    • The same hostname, agents, Slack installation, and history survive.
    • Restore the database plus KEK into an isolated instance, then read and search.
    • A missing key never triggers regeneration.
    • A dated statement of when historical plaintext copies expire.

Authorization review

Encryption decrypts for every permitted runtime read, so these gaps become plaintext exposure. This review was read-only and traced code at 49219a71.
  • Confirmed gaps:
    • F1. Medium–High. The management-scope API token (the control plane) can read memory and persona text.
    • F2. Medium. Content reads are never audited.
    • F3. High relevance. Every user on a hosted install is an owner.
  • No IDOR found. Every caller-supplied conversation, message, snapshot, decision, memory, commitment, schedule, source, or artifact ID is bound to the slug’s agent in SQL. A foreign ID answers 404.
  • Retrieval checks happen before ranking. Conversation search and memory retrieval apply perspective, evidence, and audience before rank and limit, and lineage after the candidate limit. Recall can drop; nothing leaks.
  • No unauthenticated route returns stored content. This covers vault drops, OAuth callbacks, and webhooks.
F1. The management token can read memory and persona.
  • managementProvisioningRoutes (internal/admin/session.go:302) allows GET/PATCH /api/agents/{slug}, enable, and disable.
  • Each of those returns agentDetail, which embeds the 10 most recently updated memory bodies (internal/admin/agent_profile.go:528, :603).
  • The list also allows GET /api/agents/{slug}/persona, which returns every revision’s text.
  • The service user defaults to owner (db/migrations/0050_operator_roles.sql), so this allowlist is its only limit.
  • The control plane holds the derivation key for every instance’s token. Its current client calls only four operations (apps/provisioner/src/management.ts:70), but the token’s reach is what counts.
  • The allowlist’s own comment says it exists so the control plane cannot reach conversation history. Memory reached through agentDetail defeats that intent.
  • A dry_run memory import also reports an exact-body duplicate (internal/admin/memory_import.go:362), which confirms a guessed memory text.
  • Fix before L14:
    • Return a content-free agent view to management callers.
    • Make persona write-only for that scope.
    • Report import duplicates as counts only.
F2. Content reads are not audited. audited() skips GET (internal/admin/audit.go:106). Fix: add a content-read audit record (actor, token, route, IDs; never bodies) emitted where content is decrypted for an API response. This covers the trace, prompt snapshot, decision call, conversation, actor, export, memory, revision, dreaming, commitment, and schedule routes. This is also the per-read audit the support grant design (L17, Q10) needs. F3. Hosted users are all owners. CreateExternalUser (db/queries/users.sql:32) takes the default role owner. External sign-in checks only the mesh:admin capability, and there is no role claim. Per-agent operator isolation therefore does not exist on hosted installs. Fix: map control-plane membership and role to Formation role and agent grants (L04/L12) before relying on it for consent (Q10). Lower severity:
  • F4. ListScratchEntriesForRun has no agent filter (db/queries/scratch.sql:69). It has no caller today, so either add the filter or delete it.
  • F5. A read-scope API token reaches every GET its user’s role allows, including traces and export. Document this, or add a content-free read scope.
Hardening found in passing (not authorization, but in scope for the same release):
  • secrets AAD does not bind owner, kind, or row, so a credential row can be moved between agents.
  • mcp_oauth_authorizations is omitted from rotate-key and verify-secrets.
  • span.RecordError exports error text (internal/app/ingress_dispatch.go:91).
  • Error tracker events carry up to 1 KiB of content-bearing error text.
Tests L14/L15 should add:
  1. An agent_operator receives 404 on an ungranted agent for the trace, prompt snapshot, decision calls, runs, actor, conversation, memory revisions and grants, commitments, schedules, dreaming, commitment sources, and browser artifacts. Today only persona, memory, export, avatar, and Slack setup are covered (internal/admin/operators_roles_test.go:90).
  2. A route-table test that walks every /api/agents/{slug}/ pattern and asserts requireAgentAccess or requireOwner.
  3. Foreign-ID 404s for the snapshot, decision, conversation, message, memory revisions, and grants.
  4. The management token receives no memory body from any allowlisted route and cannot read persona text. This fails today.
  5. A content-read audit row is written for trace, prompt-snapshot, and export GETs. This fails today.
  6. A hosted user without a grant cannot read agent content. This fails today, by design, until F3.

Decisions needed