/self-verify-beacon-in-sandbox

name: self-verify-beacon-in-sandbox
description: Verify a Beacon change end to end by running a real Claude Code session inside a disposable Linux cloud sandbox and checking that Beacon captured what the agent actually did. Use when asked to verify, validate, test, or prove that a Beacon change works for real rather than just compiling; when asked whether telemetry, event capture, commands, file paths, prompts, tokens, or approvals are still recorded correctly; when investigating a suspected capture gap; or when preparing a Beacon pull request that touches the CLI, hooks, or the collector exporter.
metadata:
  internal: true

Verify Beacon in a sandbox

This repository ships `beacon-sandbox`, which rents a disposable Linux sandbox from Modal, installs the Beacon under test, runs a real Claude Code session inside it, and checks whether Beacon recorded what the agent did.

**Read `beacon-sandbox/AGENTS.md` for the full operating manual** — scenario selection, how to interpret a verdict, and the failure modes that look like Beacon bugs but are not. The commands below are enough to start; that file is what stops you misreading the result.

Step 1: check prerequisites

Always start here. It is free, needs no sandbox, and prints the exact fix for anything missing:

cd beacon-sandbox
go run ./cmd/beacon-sandbox doctor

Add `--json` for machine-readable output with a top-level `ready` boolean.

Step 2: resolve what doctor reports

Apply the `fix:` line it prints, then rerun `doctor`. The two build artifacts you can resolve yourself:

cd cli/beacon && make build-linux-amd64      # if beacon_binary FAILs
cd beacon-sandbox && go run ./cmd/beacon-sandbox doctor --fix   # downloads the collector

**Two prerequisites you must NOT try to resolve yourself:**

1. **Modal authentication.** `modal token new` completes through an authenticated *web session*, so it opens a browser and blocks — running it yourself will hang until timeout. If `modal_auth` FAILs, ask the user to run it, suggesting they type `! pip install modal && modal token new` so the output lands in the conversation. In CI, `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` work non-interactively instead. 2. **The Anthropic credential.** Never invent, echo, or write a key. If `anthropic_credential` FAILs, ask the user which of the three paths they want: `ANTHROPIC_API_KEY`, `--api-key-command CMD`, or `--modal-secret NAME`.

Do not proceed to step 3 while `doctor` reports any FAIL — the run will fail later and more confusingly.

Step 3: run it

go run ./cmd/beacon-sandbox run --scenario s02-bash-command   # one scenario -- do this while iterating
go run ./cmd/beacon-sandbox run                               # the whole suite, ~30 min

Pick the scenario matching what changed:

| You changed | Scenario | |---|---| | Command capture / exporter tool handling | `s02-bash-command` | | File read or write signals | `s03-file-write` or `s04-file-read` | | Prompt, session, token, or cost capture | `s01-hello` | | Approval or permission handling | `s07-denied-tool` | | `endpoint install`, config paths, service startup | `i01-install-supervised` | | The systemd backend, unit files, Linux system mode | `i02-install-systemd` | | Something broad, or preparing a PR | the whole suite (bare `run`) |

The `s0*` scenarios collect through `beacon ci exec`, a temporary collector. The `i0*` scenarios install Beacon first, so they are the only ones that cover installation and service management. `i02` runs inside a nested privileged container, because systemd will not start unless it is PID 1 and the sandbox provider's own init holds that slot — expect it to take several minutes longer.

Other flags: `--repeat N` to tell flaky from broken, `--keep-sandbox` to leave the instance up for debugging.

Four things not to get wrong

1. **A run costs real money** — about $0.06 of sandbox plus a few cents of API per scenario, on the user's account. Say what you are about to run and roughly what it costs before spending it, and prefer a single `--scenario` while iterating. If you only changed *what counts as correct*, use `verify <run-dir>` instead — it re-judges collected artifacts offline and free. 2. **A change under `collector-builder/` needs the collector rebuilt.** The telemetry normalization compiles into `beacon-otelcol`, not the `beacon` CLI, so otherwise you verify the wrong binary and get a meaningless pass. `doctor` warns about this as `collector_freshness`. 3. **INCONCLUSIVE means the model never did the work** — retry the scenario; do not investigate Beacon. 4. **A FAIL may be a stale assertion, not a bug.** Read the failing expectation's `why` field against the verdict's action histogram before concluding anything.

`beacon-sandbox/AGENTS.md` explains each of these properly, plus `diff`, scenario authoring, and how to self-test that the checks still have teeth.