/cli-setup-verify

name: cli-setup-verify
description: Verify the Omnigent CLI's setup/onboarding flow, terminal UI/UX, and critical user journeys in a completely isolated, reproducible loop. Drives the real `omnigent` binary through a PTY (pexpect) inside a throwaway OMNIGENT_CONFIG_HOME / OMNIGENT_DATA_DIR sandbox that never touches the user's real ~/.omnigent, captures ANSI-stripped frames for UX inspection, and proves a change is verifiable via a before→fix→after baseline diff. Load when developing or reviewing a CLI setup/onboarding/REPL/picker change (omnigent/cli.py, omnigent/onboarding/*, omnigent/repl/*, scripts/install_oss.sh), reproducing a cold-start/first-run UX bug, or confirming a fix actually lands. Several agents can run it concurrently on separate worktrees.

Verifying the Omnigent CLI setup & UX in a closed loop

The Omnigent CLI's first impression is: `curl | sh` → run `omnigent` → pick a model credential → start a session. This skill lets an agent **enter that flow, examine the UI/UX, and prove whether a change is verifiable** — without a browser, without real credentials, and **without ever touching the developer's real `~/.omnigent`**.

The engine is `verify_cli.py` (next to this file). It drives the real `omnigent` binary through a pseudo-terminal (`pexpect`) inside a throwaway sandbox, captures what renders, runs assertions, and prints one machine-readable `SUMMARY {json}` line.

> **The whole point is a verifiable loop**, not a one-shot check: > 1. Run a scenario on the **unfixed** code → baseline (`--label before`). > 2. Make the change. > 3. Run the **same** scenario → `--label after`. > 4. Diff the two `SUMMARY` lines. A fix is "verifiable" only if a concrete > check or note **flips** between the two runs. If it doesn't flip, you > can't prove the fix did anything — go back to step 2.

Why this is safe (read first)

The real `~/.omnigent` here can be **many GB** (chat DB, runner logs, native harness state). The sandbox isolates every write three ways:

• **`HOME` is redirected into the sandbox by default.** This is the load-bearing

one. `OMNIGENT_CONFIG_HOME` / `OMNIGENT_DATA_DIR` (`omnigent/cli.py` `_CONFIG_HOME_ENV_VAR` / `_DATA_DIR_ENV_VAR`) redirect config + data — but the CLI's **diagnostics logger ignores them**: it writes a per-invocation `cli-*.log` under `state_dir()`, hardcoded to `Path.home()/.omnigent/logs` (`omnigent_ui_sdk/terminal/_config.py`). So only redirecting `HOME` keeps a non-help command (`config list`, the setup PTY spawns, `server stop`) from writing into the real home. The driver does this for you.

• `--strip-path` reduces `PATH` so `node`/`tmux`/`claude`/`codex` read as "not

installed" → the true fresh-machine cold start.

• Ambient model keys (`ANTHROPIC_API_KEY`, …) are stripped from the child env

unless you pass `--keep-env-creds`.

**`--inherit-home` opts out** of `HOME` isolation — use it only to reach a real credentialed REPL via ambient `~/.claude` / `~/.databrickscfg` auth. It is **less safe**: a non-help command then writes `cli-*.log` into the real `~/.omnigent/logs`.

Every run **fingerprints the real `~/.omnigent` before and after** (stat-only, no content reads): the top-level config files **and** the set of `logs/cli-*.log` diagnostic files. A new config file/mtime *or* a new `cli-*.log` basename trips `real_config_untouched: false`. With the default isolation that never happens; under `--inherit-home` it correctly does — which is exactly the violation the guard is meant to catch. If that check is ever `false`, stop and investigate. Run `check-isolation` first to confirm the loop is safe on your machine.

Prerequisites

• You're in the **worktree whose code you want to test** (each parallel agent

on its own worktree). The driver runs `omnigent` from `--repo`'s checkout.

• A Python with `pexpect` — the project's `.venv/bin/python` bundles it

(`pexpect>=4.9` in `pyproject.toml`). Run the driver with that interpreter.

• An `omnigent` binary: the driver auto-finds `<repo>/.venv/bin/omnigent`, or

pass `--omnigent <path>`.

• The setup / picker / help / cold-start scenarios need **no credentials and no

harness**. Only `repl-commands` needs a working harness + credential: pass `--inherit-home` (ambient `~/.claude` auth) and/or `--keep-env-creds` (env API key) with `--agent`. It reports `skipped`, never a false pass, when the prompt isn't reachable.

Quick start

REPO=/path/to/your/worktree
PY=$REPO/.venv/bin/python
DRV=$REPO/.claude/skills/cli-setup-verify/verify_cli.py

# 0. Prove the sandbox is safe on this machine (do this once).
#    HOME is isolated by default — no flag needed.
$PY $DRV --scenario check-isolation --repo "$REPO"

# 1. See exactly what a brand-new user sees on a fresh machine.
$PY $DRV --scenario cold-start --strip-path --keep-sandbox --repo "$REPO"
#    → reads the printed `artifacts` path, then `cat <that>/cold_start.txt`

# 2. Lint the top-level help (and any subcommand's).
$PY $DRV --scenario help-snapshot --repo "$REPO"
$PY $DRV --scenario help-snapshot --subcommand server --repo "$REPO"

Each run prints `SUMMARY {…}` and exits non-zero if any check failed (a `skipped` scenario exits 0). Pipe to `… | grep '^SUMMARY' | python -m json.tool` to read it.

Scenario catalog

| Scenario | What it drives | Key checks / notes | Maps to findings | |---|---|---|---| | `check-isolation` | `omnigent config list` in the sandbox (no PTY) | `config_list_ran`, `sandbox_config_home_used`, `real_config_untouched` | safety gate for everything | | `cold-start` | `omnigent setup` via PTY on a simulated fresh machine | `onboarding_rendered`, `harness_menu_present`; note `guided_default_affordance` | cold-start dead-end; missing "recommended start here" | | `setup-snapshot` | `omnigent setup`, optional `--nav-down N` arrow steps | `menu_rendered`; saves a frame per step | picker markers/footer/alignment; narrow-terminal at 80×24 | | `help-snapshot` | `omnigent [--subcommand]