/ComputerUse

name: ComputerUse
metadata:
  short-description: "Route Mac control to Interceptor for DOM/AX/CLI/API handles or a bounded vision agent for pixel-only GUI work; verify with Interceptor or CLI."
version: 1.1.4
evals:
  manual:
    reason: "a pixel-only mission must be checked in the operated app because task completion depends on the requested visible state"
    falsifier: "the reviewer finds the requested state absent, a forbidden action attempted, or the recorded read-back inconsistent with the app"
description: Front door for driving this Mac: anything with a DOM, AX, CLI, or API handle routes to Interceptor; pixel-only GUI work goes to a vision agent, OpenAI Astra via codex exec or a tool-less Claude acting through Interceptor's bridge, picked per task class by a measured bake-off, under a deny list, step caps, and a read-back that decides pass/fail. USE WHEN computer use, drive the computer, control the mac, operate this app for me, do it in the GUI, pixel-only app, no API for this, canvas UI, computer-use bake-off, Claude vs OpenAI computer use. NOT FOR DOM/AX handles or web verification (Interceptor), scraping (a scraping skill, BrightData).

ComputerUse

**Contents:** Ideal State · Intent Routing · The routing table · The supervised run · Data & Config · Gotchas · Examples

One skill routes every computer-use task by a rule you can read, then supervises the run so the vision leg never grades itself, never sees what it must not, and never runs away.

Ideal State

• Every task is routed by the readable rule before anything runs: verification and anything with a DOM, AX, CLI or API handle go to Interceptor, and only pixel-only work goes to the measured vision pick.
• A vision run's verdict comes only from its declared verifier, never from the agent's own report, and no run starts without one.
• The vision leg never sees a running credential app, a visible communications app or a denied host, never acts outside its allowlisted app, and stops at its step and wall-clock caps.
• Send, buy, delete, publish or confirm happens only when the principal named that exact action, and every refusal is reported rather than worked around.

Intent Routing

| Intent | Trigger | File | |--------|---------|------| | **Drive** | "do this in the GUI", "drive <app>", "computer use for <task>" | `Intents/Drive.md` | | **Probe** | "is Astra computer use working", "probe the vision leg" | `Intents/Probe.md` | | **BakeOff** | "bake-off", "which vision agent is better", "Claude vs OpenAI computer use", "re-score the vision legs" | `Intents/BakeOff.md` |

| File | Read when | |------|-----------| | `Instructions.md` | changing what the Astra leg is told; the runner passes it as `model_instructions_file` on every spawn | | `Config.example.md` | setting up the deny lists, caps and Focus shortcuts in the USER `Config.md` | | `Tests/` | changing routing, deny rules, caps or spawn flags (fixtures under `Tests/fixtures/` are synthetic; source tree only, not in the release payload) | | `Setup.md` | first-time setup: the Config.md fields, the Codex app and plugin, permissions, and a dry-run proof | | `References/VisionLegs.md` | changing or debugging how the Astra or Claude leg is spawned and confined: codex flags, the Claude leg step loop, its tool restrictions, env drops, pixel-to-point mapping |

The routing table

| Intent | Handle? | Backend | Why | |--------|---------|---------|-----| | verify / evidence | any | **Interceptor**, always | Constitutional №2: Interceptor is the only sanctioned evidence path. No failure count or backend state changes this row. | | act | yes (DOM, AX element, CLI, API) | **Interceptor** | Deterministic is cheaper, faster, and auditable. | | act | no | **the vision pick** for the target app: Astra or Claude | Pixel-only authoring, canvas UIs, unscripted cross-app GUI flows. | | act | yes, but 2 Interceptor failures on this step | **the vision pick** for that step | Bounded escalation, logged once. Never the reverse direction. | | any | Interceptor preflight failed | **refused** | A broken evidence path is "tool broken", never "no handle". | | act, vision | — | **Claude** only when `vision_pick` names it for that app (or as default) | The pick is written from `Tools/BakeOff.ts` results, never by hand-waving. Unpicked or unmeasured classes stay Astra. |

Code: `Tools/Route.ts` (pure, tested). Runner: `Tools/Run.ts`.

The supervised run

Both vision legs run under the same `supervise()` in `Tools/Run.ts` (`--backend astra|claude`; without it, the `vision_pick` for the target app). The deny scan, watcher, caps, headless refusal and verifier below are shared.

• Refuses on a headless channel (exit 3), on a failed Interceptor preflight (exit 4), without a declared verifier (exit 5), and when the deny list fires (exit 2, naming the app or host).
• Deny list (your `Config.md`): credential-class apps may not be **running**; communications apps may not be **visible** (hidden with Cmd+H is fine); browser tabs on denied hosts block. Enforced before spawn and by a watcher every poll interval during the run.
• Per-run app allowlist (default: the target app alone); the watcher kills the child when the frontmost app leaves it.
• Step cap and wall clock, enforced by the runner. Defaults 60 steps / 10 minutes; override per run. A step is one completed computer-use tool call on the Astra leg (one call can bundle several clicks, so the wall clock is the hard bound) and one executed action on the Claude leg.
• A missing `Config.md` is refused (exit 2), never run with empty deny lists.
• Optional Focus (Do Not Disturb) on/off Shortcuts wrap the run so banners cannot enter a frame.
• Every exit path, aborts included, runs the declared verifier. The run record's `verdict` is written only from the verifier's exit code; the leg's own report is stored as text and never counts.
• Voice line before the first action and at finish. No approval gate for the task you named; send, buy, delete, pub