/polly-e2e-dev

name: polly-e2e-dev
description: End-to-end test the polly multi-agent coding orchestrator's critical user journeys (CUJs). Two halves — a deterministic mock-LLM driver (polly_cuj.py) that boots a throwaway local server + mock LLM and asserts the substrate (boot, bridged sys_* tool dispatch, the blast_radius / spawn_bounds / headless_subagent_purpose_guard guardrails, fan-out delegation), and a live real-CLI recipe (real claude/codex/pi, real worktrees/PRs) for polly's actual judgment. Load when developing, testing, or debugging examples/polly — its config.yaml, the claude_code/codex/pi sub-agents, the investigate/fanout/cross-review skills, or the omnigent.inner.nessie.policies guardrails — or reproducing a polly orchestration bug.

polly orchestrator: end-to-end CUJ dev & testing

`polly` (`examples/polly/`) is a multi-agent **coding orchestrator**: a `claude-sdk` "brain" that writes no code itself and delegates everything to three coding sub-agents — `claude_code` (claude-native), `codex` (codex-native), and `pi` (headless, multi-model). Its critical user journeys are orchestration behaviors, not single-turn answers:

• **roster preflight** — first turn runs `command -v claude codex pi`, routes

only to workers whose CLI resolved.

• **investigate** — read-only work fanned to `explore`/`search` sub-agents;

synthesize from their reports.

• **fanout** — independent tasks, each in its own git worktree + sub-agent, each

opening its own PR.

• **cross-review** — an implementer's diff is verified by a **different-vendor**

sub-agent (diff + contract only); blocking issues become fix-tasks.

• **plan gate / inbox** — pull the human in at the plan gate; supervise via the

inbox + autowake, never busy-poll.

• **guardrails** (`omnigent.inner.nessie.policies`) — `blast_radius` (deny

force-push / `rm -rf /`), `spawn_bounds` (cap dispatches per turn), `headless_subagent_purpose_guard` (every dispatch needs `args.purpose`).

This skill tests those CUJs two ways. Use **both** — they cover different things:

| Half | What it proves | Needs | |------|----------------|-------| | **Mock loop** (`polly_cuj.py`) | The **substrate/mechanics** — the brain is *scripted*, so this proves bundle load, server-side policy resolution, bridged `sys_*` tool dispatch, the guardrail DENYs, and fan-out — deterministically, with no creds | nothing (mock LLM) | | **Live recipe** | polly's **judgment** — does the real brain preflight, decompose, delegate, cross-review, and pull in the human correctly | real `claude`/`codex`/`pi` + model creds + network |

> Like the sibling harness skills, turns run from your **current checkout** > (`omni run <bundle> --server <url>` = local runner + remote server), so testing > exercises exactly the code you're on.

Interpreter

The driver and CLI need the repo's Python ≥3.12 env. If `.venv/` is missing, create it once from the checkout:

uv run --frozen python -c "import omnigent; print('ok')"   # builds .venv

Then use `.venv/bin/python` / `.venv/bin/omni` below.

---

Part A — the deterministic mock loop (`polly_cuj.py`)

The driver boots a throwaway local Omnigent server (which carries `omnigent.inner.nessie.policies` — the module polly's guardrails resolve) plus the repo's mock-LLM server, rewrites the polly bundle to the `openai-agents` harness wired to the mock, then runs `omnigent run` turns where the brain is *scripted* (text or tool calls). It prints one `SUMMARY {json}` per scenario and exits non-zero if any check failed.

.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --list-scenarios
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario all
.venv/bin/python .claude/skills/polly-e2e-dev/polly_cuj.py --scenario guardrail_purpose --keep

Read the result with `… | grep '^SUMMARY' | python -m json.tool`. Each run takes ~45–55s for all five scenarios; no credentials or egress are required.

Scenario catalog

| Scenario | Scripts the brain to… | Hard check | |---|---|---| | `boot` | reply with text | exit 0 + non-trivial reply (bundle load, server-side policy resolve, turn completes) | | `tool_dispatch` | call `sys_os_shell` to write a sentinel | the file appears on disk (bridged `sys_*` dispatch works; `blast_radius` ALLOWs benign shell) | | `guardrail_purpose` | `sys_session_send` with **no** `args.purpose` | tool output carries `Denied by policy: … must declare what kind of work it is` (`headless_subagent_purpose_guard`) | | `guardrail_blast_radius` | `sys_os_shell("git push --force …")` | tool output carries `Denied by policy: … blast-radius policy` | | `fanout_dispatch` | emit 6 `sys_session_send` in one turn | ≥2 sub-agent dispatch handles created (fan-out substrate). **Finding:** reports whether the `spawn_bounds` cap fired (see Known sharp edges) |

The verifiable before→after loop

The driver exists for a *loop*, not a one-shot. To prove a fix:

1. On the **unfixed** code, run the scenario → a check is `false` (baseline). 2. Make the change. 3. Run the **same** scenario → the check **flips** to `true`.

A fix is "verifiable" only if a check flips. If it doesn't flip, you can't prove the change did anything — keep working. To cover a new mechanism, add a `scenario_*` function + a row in `_SCENARIOS` (each builds a bundle, scripts the mock, runs a turn, and asserts an **observable effect** — a session item, a deny sentinel, a file on disk).

What the mock loop can and can't prove

It tests **mechanics** because the brain is scripted: tool dispatch, the guardrail gate, session persistence, fan-out plumbing. It does **not** test polly's judgment (whether the *real* brain preflights, decomposes, picks the right vendor, cross-reviews). That is the live recipe.

---

Part B — the live recipe (real claude/codex/pi)

Prereqs (check first)

1. **You're on the branch you want to test.** 2. **A Claude provider for the brain** (`omni setup`, or `ANTHROPIC_API_KEY`, or a Databricks default)