/jev-computer-use

name: jev-computer-use
description: Use when driving a desktop GUI through a computer-use driver — windows, menus, native apps, OS dialogs. You build a table of safe actions; Jev picks the next one in about 0.4 seconds.
version: 0.1.0
license: MIT
metadata:
  hermes:
    tags: [jev, typesafe, computer-use, gui, cua]
    related_skills: [jev-browser-use]

Computer use with Jev

You stay the planner and the hands. Jev is only the fast "which one next?" in the middle. It returns an id from a table **you** built, so it cannot invent coordinates, text, selectors or tool calls. The worst a wrong answer can do is pick another action you already judged safe.

Web pages belong to `jev-browser-use`. This skill is for desktop apps and OS surfaces, driven through whatever computer-use driver you have (CUA Driver over MCP, the platform's native computer-use tool, an accessibility bridge).

First choice on a Mac: Co-Agent does the loop for you

If Co-Agent is installed (its engine answers on `http://127.0.0.1:8792`), let it drive. It already holds the Mac's Accessibility and Screen Recording permissions, runs this same loop natively (fresh observation, Jev picks one id from a closed menu, one action, a new observation to verify) and adds what hand-built loops kept getting wrong:

• it hit-tests every click and brings a covered window forward (or refuses with

`occluded`), instead of clicking whatever app is really on top;

• it reads apps with no accessibility tree (Epic Games Launcher, games, canvases) with

on-device OCR, and clicks them in a way Unreal and WebKit accept;

• it applies the owner's policy per action with no dialog for ordinary input, and answers

`needs_approval` with a ticket at once for purchases, deleting, sending, legal acceptance, sign-in and security settings; it never types credentials;

• it names a lock screen or a macOS permission prompt as `blocked` instead of hanging.

Agents with MCP use its tools `computer_status`, `computer_observe`, `computer_act` and `computer_run`. Agents that shell out use the bundled client:

python3 <this skill>/scripts/coagent_cu.py setup --name "Hermes"      # once per machine user
python3 <this skill>/scripts/coagent_cu.py status
python3 <this skill>/scripts/coagent_cu.py run --app "System Settings" --open \
  --goal "Open the Appearance settings pane" --expect-text Appearance
python3 <this skill>/scripts/coagent_cu.py run --app Safari \
  --goal "Fill in the profile form and save it" \
  --input "Full name=Ada Lovelace" --input "Email address=ada@example.com" --expect-text "Thanks Ada"
python3 <this skill>/scripts/coagent_cu.py click --app "Epic Games Launcher" --target Library --near "top bar"

Always give `--expect-text` (or `--expect-title`) so success is checked, not assumed. Put text to enter in `--input`; it is typed only into a field whose label matches. Exit codes: 0 done, 3 needs approval (nothing happened: tell the person what it wants and stop; do not look for another way to do it), 4 not verified / stalled / loop, 5 blocked (say what blocks it: the lock screen, the prompt's text, the missing permission), 2 usage or connection error. The same privacy boundary holds: Co-Agent sends Jev element ids, roles and short labels, never screenshots, field values or secure fields.

Use the loop below yourself only when Co-Agent is not installed on the machine.

The loop

1. **Observe** with your driver. Prefer accessibility/semantic state over pixels. Every ref, capture id and coordinate is good for this observation only. 2. **Build the candidate table locally.** Each row is an opaque id plus one complete, prevalidated action. Always include:

• `reobserve`: look again, change nothing
• `abstain`: stop and ask for help

3. **Privacy gate.** Nothing sensitive goes to Jev: no credentials, tokens, cookies, password-field contents, payment data, customer data, screenshots, files or unbounded page text. If the screen holds such content, abstain or handle it without Jev. 4. **Ask once:**

   jev choose < request.json          # Hermes: the jev_choose_action tool, argument `request`

   {"schema": "jev.action_choice_request_v1",
    "goal": "Open Settings and select Appearance.",
    "observation_id": "capture-0042",
    "regions": [{"id": "r1", "role": "button", "label": "Appearance", "interactive": true}],
    "history": [{"selected_id": "open-settings", "outcome": "settings window opened"}],
    "candidates": [
      {"id": "select-appearance", "description": "Click the Appearance row in the Settings sidebar."},
      {"id": "reobserve", "description": "Take a fresh observation without changing anything."},
      {"id": "abstain", "description": "Do not act; ask the person for help."}]}

Pass the JSON on stdin or from a temp file. Never interpolate it into a shell string. 5. **Run exactly the one action** behind `selected_id`. Confidence under the floor (0.65, measured — see `scripts/calibrate_choose.py`; a second "does any candidate match" question was measured against the same cases and reduced no wrong actions, so it is not asked — see `evals/choose-match/`), or any Jev failure, comes back as `reobserve`. Always send `regions`: they are Jev's evidence the element is really on screen, and the same request scored 0.60 without them and 1.00 with them. Never derive an action from anything but the id. 6. **Observe again and verify the postcondition yourself.** A chosen id, a delivered click or a screenshot is not proof. Check application state before the next step. Stop after a bounded number of steps.

The bundled runner compares fresh local Accessibility state after each mutation (window title, controls, selection and field changes; ephemeral driver tokens are ignored). If the *same* chosen mutation leaves that state unchanged twice, it stops as `stalled_action` before asking Jev a third time. An unchanged screen is not proof that the requested effect failed forever (some ap