/pilot-aws

name: pilot-aws
description: Operate the AWS-hosted Pilot daemon (founder box) — status, dashboard, logs, queue queries, start/stop/restart, rebuild/upgrade, metrics tunnel, troubleshooting. Auto-invoke when user says "pilot aws", "check the box", "aws daemon", "pilot on aws", "box status", "restart pilot on aws", "box logs", or any operation against the hosted daemon.
allowed-tools: Bash, Read
version: 1.0.0

Pilot AWS Operations (founder box)

Since the S6-lite cutover (2026-07-16, TASK-409) the Pilot daemon runs on an EC2 box, NOT locally. **Local `pgrep pilot` returning 0 is correct.**

Constants

INSTANCE   i-0e0c1ca34e7b561f9        # "pilot-founder-box", t3.xlarge, eu-central-1a
PROFILE    quantflow                   # export AWS_PROFILE=quantflow for every aws call
REGION     eu-central-1
RUNNER     i-0147f5c24d234cdbb         # mgmt runner (AdministratorAccess) — IAM/privileged ops only
DAEMON     tmux session "pilot" as ec2-user, wrapper /home/ec2-user/start-pilot.sh
BINARY     /var/lib/pilot/bin/pilot    # ec2-user-owned (self-upgrade works, #4470); /usr/local/bin/pilot is a SYMLINK to it — never `sudo install` over the symlink; rollback at pilot.prev
STATE      /home/ec2-user/.pilot → /var/lib/pilot/pilot-home (200GB data volume)
REPOS      /Users/aleks.petrov/Projects → /var/lib/pilot/repos (path shim — ledger keys
           on the macOS-era absolute paths; NEVER "fix" these symlinks)
DB         /home/ec2-user/.pilot/data/pilot.db
LOG        /home/ec2-user/.pilot/logs/daemon.log (+ daemon-stderr.log)

The SSM command pattern (used by everything below)

export AWS_PROFILE=quantflow AWS_DEFAULT_REGION=eu-central-1
CMD=$(aws ssm send-command --instance-ids i-0e0c1ca34e7b561f9 \
  --document-name AWS-RunShellScript \
  --parameters 'commands=["<shell here>"]' \
  --query Command.CommandId --output text)
sleep 10
aws ssm get-command-invocation --command-id $CMD \
  --instance-id i-0e0c1ca34e7b561f9 --query StandardOutputContent --output text

For multi-line/quote-heavy payloads write a JSON file and use `--parameters file:///tmp/x.json`; for shipping FILES to the box, base64 the content into the command (`echo <b64> | base64 -d > target`) — inline heredocs with `\n` escapes DO NOT survive SSM JSON (verified failure mode).

Operations

Status (first move, zero GitHub quota)

~/bin/pilot-board            # daemon health, queue, autopilot, log tail
~/bin/pilot-board --gh       # + GitHub issues/PRs (costs shared user quota — sparingly)

Remote half lives at `/usr/local/bin/pilot-board-remote` on the box (edit there).

Live TUI dashboard

~/bin/pilot-dash             # SSM interactive → tmux attach -t pilot

Detach: `Ctrl-B` then `D`. **NEVER press `q` or `Ctrl-C` inside the TUI** — that stops the daemon. Fallback: plain ssm session → `sudo su - ec2-user` → `tmux attach -t pilot`.

Logs

# tail (adjust -n / add grep):
SSM: tail -50 /home/ec2-user/.pilot/logs/daemon.log
# common filters: 'rate limit', 'claim lost', 'approval', 'ERROR', a task id

Queue / ledger queries (read-only)

SSM: sqlite3 -column /home/ec2-user/.pilot/data/pilot.db \
  "SELECT task_id,status,datetime(created_at) FROM executions \
   WHERE status IN ('running','queued') ORDER BY created_at;"

Key tables: `executions`, `execution_claims` (task_id, project_path, generation), `autopilot_pr_state`, `autopilot_scope_release`, `instance_events`. ⚠️ Timestamp trap: pre-2026-07-16 rows may carry legacy `…+02:00` string format — never filter by string time ranges across eras; use rowid or exact ids.

Stop / start / restart ⚠️ OPERATOR-CONSENT ACTIONS

Never do these on your own judgment during watch/autonomous modes; in interactive sessions get explicit user go-ahead. In-flight executions die (they retry via generation claims — proven safe, but wasteful).

# STOP    (graceful; wait, then verify 0):
SSM: sudo -iu ec2-user tmux send-keys -t pilot C-c   # TUI quit = graceful shutdown
     sleep 15; ps -eo comm | grep -c '^pilot$'        # must be 0
# START:
SSM: sudo -iu ec2-user tmux new-session -d -s pilot -x 220 -y 50 /home/ec2-user/start-pilot.sh
     sleep 20; ps -eo comm | grep -c '^pilot$'        # must be 1; then check pilot-board
# RESTART = STOP, verify, START, verify (banner version + no Telegram 409 + poller ticks).

After ANY start: verify version (`/usr/local/bin/pilot version`), exactly 1 process, `curl -s localhost:9091/metrics | grep pilot_queue_depth` (via SSM).

Rebuild / upgrade the box binary

SSM (as ec2-user, NO sudo — dir is ec2-user-owned since 2026-07-19):
  cd /Users/aleks.petrov/Projects/startups/pilot && \
  git fetch -q --tags origin main && git checkout -q <tag-or-origin/main> && \
  make build && install -m 0755 bin/pilot /var/lib/pilot/bin/pilot && \
  git checkout -q main && /var/lib/pilot/bin/pilot version

Then RESTART (above) to activate. Prefer building from a released tag. For releases, daemon self-upgrade also works now (#4470 preflight + writable dir) — letting it upgrade itself on the next train is the default path. Expect ~1 quiet hour post-restart until #4391 ships: startup rescans can burn the GitHub user-aggregate rate pool (see Troubleshooting).

Metrics tunnel (grafterm / local Prometheus tools)

~/bin/pilot-tunnel     # box:9091 → localhost:9091, keep running; then `pilot-tui` (grafterm alias)

Box shell (interactive)

aws ssm start-session --target i-0e0c1ca34e7b561f9 --profile quantflow --region eu-central-1
sudo su - ec2-user

Privileged infra ops (IAM, volumes, instance-level)

User `aleks` lacks iam:PassRole/ec2:CreateVolume etc. Route through the mgmt runner via SSM RunShellScript on `i-0147f5c24d234cdbb` (it has admin). Scope every grant minimally; never print policy docs containing secrets.

Hard rules

1. **One daemon, ever.** Never start pilot locally while the box serves the repos (dual-serve = cross-machi