Skip to main content

​
Live launch checks

Run the same acceptance scenarios through each communication connector, execution backend, and browser provider before inviting testers. The live runner is separate from the deterministic reliability suite: scripted models and local transports prove recovery invariants, while this matrix checks the actual deployment and external delivery. A live failure is a failed check, including timeouts, missing cost/usage configuration, incomplete history windows, or a model declining a task.

​
Prepare an isolated matrix

  1. Use a disposable Formation deployment and agents whose slugs start with launch-check-. Do not attach production conversations, memories, website profiles or acting credentials. The runner refuses ordinary agent slugs and requires isolated: true plus --live. These guards do not independently prove a deployment is isolated; verify the URL and restart command yourself.
  2. Create a distinct external bot identity and connector for each agent. Complete normal setup, then copy its connector UUID from the operator API. Pin each agent to the execution backend named in its profile. The default required matrix includes Slack, Telegram, and Modal; add profiles for other backends and add their names to required_backends. Execution runs through Formation’s vendor-neutral workspace tools, so the same check covers future backends.
  3. Give each agent workspace_exec, browser navigation/read/action tools, and its own browser profile. Configure a live model and a sufficient allowance. Serve scripts/launch-browser-fixture.html at a URL reachable by the browser provider; allow that domain in the agent’s web settings. It is a synthetic input/button/output page with no authentication or private data.
  4. Create dedicated human test accounts. Slack needs a user token permitted to post and read threads in the test DM and shared room. Keep the room private to the test participants and add the agent. Telegram needs a user session, not a bot token: bot-to-bot messages do not exercise human ingress. Add a second test participant to the shared room so it has a wider audience than the direct conversation. Configure the bot’s group permissions normally.
  5. Copy scripts/launch-check.example.json outside the repository and fill its non-secret identifiers. restart_command is an argv array that restarts only the disposable Formation process. It runs without a shell. The test waits until the command tool starts, then allows a short configurable restart_delay_seconds (default 3) before invoking it. The durable trace proves tool dispatch, not the exact remote process start instant. Keep task deadlines long enough for the 30-second command and the restart. Provider resources and messages cost money; use the existing provider-specific cleanup probes if an isolated deployment cannot clean up after a crash.
Credentials are environment-only:
  • MESH_LAUNCH_API_TOKEN: an operator token scoped to the dedicated agents.
  • MESH_LAUNCH_SLACK_USER_TOKEN: the dedicated Slack participant’s user token.
  • MESH_LAUNCH_TELEGRAM_API_ID, MESH_LAUNCH_TELEGRAM_API_HASH, and MESH_LAUNCH_TELEGRAM_SESSION: the dedicated Telegram participant’s application credentials and preauthorized Telethon StringSession. The adapter never logs in interactively or changes the account’s session. Treat the session string as a credential. Install the optional adapter dependency in a virtual environment: python3 -m pip install -r scripts/launch-check-requirements.txt.
The Slack adapter uses the official thread history API. Some app distributions have a one-request-per-minute history limit; use poll_seconds: 60, duplicate_window_seconds: 60, and longer timeouts for those apps. Rate limiting fails the check; it is not treated as an empty history. The Telegram adapter uses Telethon’s user client.

​
What passes

Each profile must demonstrate:
  • A real participant message receives one provider-visible reply, backed by a successful durable run and first-conversation readiness timestamps.
  • A private synthetic marker is acknowledged privately and never appears in the shared-room response. This is a useful disclosure regression, not a proof against every prompt injection or every privacy failure.
  • A real workspace_exec task succeeds through the configured execution backend.
  • A real browser navigates the fixture, reads controls, types a marker, clicks, and returns the result through the communication connector.
  • Disabling the dedicated agent stops a synthetic reply. Enablement is restored in a finally block, including failures; abrupt host termination may still require the operator to restore it manually.
  • A restart during a command attempt recovers the task with one execution, one command-side file append, and one final provider-visible delivery. The short duplicate observation window is explicit, not an indefinite guarantee.
The report is checkpointed after each exchange. It contains case statuses, provider reply IDs, and run IDs, with owner-only permissions. No prompts, messages, tokens, session strings, tool output or provider error bodies are retained in it. The isolated deployment and the provider still retain their normal test conversation and tool evidence under their own policies. Keep the redacted report private and remove the dedicated test deployment/accounts when finished; the runner does not delete external conversations or configured agents.

​
Extend to another connector

The runner does not switch on channel names. A profile supplies a driver argv array; the bundled adapter implements Slack and Telegram. Another adapter can exercise phone, a generic connector, or a future medium without changing the scenarios. It receives one JSON object on stdin:
It returns {"id":"provider-request-id"}. For action: "history", body is replaced by reference (the request ID); return all agent messages belonging to that request as {"messages":[{"id":"reply-id","body":"reply"}]}. An empty history is valid only after a successful complete provider read. Include every page and fail on truncation, errors or uncertain reads. Do not echo credentials or vendor error bodies. The adapter inherits credentials from its environment; the profile itself must contain no secrets. If the medium has no shared surface, set supports_shared: false; the report explicitly records that scenario as not applicable rather than claiming shared-surface coverage.

​
Deeper service probes

The matrix complements existing isolated probes, which exercise additional provider failure and cleanup behavior:
  • scripts/check-agent-reliability.sh acceptance: deterministic durable runtime and execution acceptance with real PostgreSQL.
  • scripts/check-agent-reliability.sh modal: opt-in real Modal reliability and cleanup probe; follow the isolation gates in docs/runtime/modal-reliability-probe.md in the repository.
  • TestLiveModalDurableCommandRecovery: command start/read/cancel and replacement client recovery, gated by MESH_TEST_MODAL_DURABLE_COMMANDS=1 and a dedicated MESH_TEST_MODAL_API_TOKEN.
  • TestLiveBrowserEditableControls: disposable live browser control semantics, gated by MESH_LIVE_BROWSER_TEST=1; compile with go test -c and pass its credential JSON on stdin as documented in the test. Never put credentials in command arguments or evidence reports.
Local runner tests need no service credentials: python3 -m unittest discover -s scripts -p 'test_launch_check.py'.