/computer-use

name: computer-use
description: |
  Drive the user's desktop in the background — clicking, typing,
  scrolling, dragging — without stealing the cursor, keyboard focus,
  or switching virtual desktops / Spaces. Cross-platform: macOS,
  Windows, Linux. Works with any tool-capable model. Load this skill
  whenever the `computer_use` tool is available.
version: 2.0.0
platforms: [macos, windows, linux]
metadata:
  hermes:
    tags: [computer-use, desktop, automation, gui, cross-platform]
    category: desktop
    related_skills: [browser]

Computer Use (universal, any-model, cross-platform)

You have a `computer_use` tool that drives the user's desktop in the **background** — your actions do NOT move the user's cursor, steal keyboard focus, or switch virtual desktops / Spaces. The user can keep typing in their editor while you click around in a browser in another window. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini, or an open model on a local OpenAI-compatible endpoint. There is no Anthropic-native schema to learn.

Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood for the platform plumbing. The Hermes-side `computer_use` tool exposed in this skill is a higher-level Hermes vocabulary; the raw cua-driver MCP tools (which a different agent harness would see) are NOT what you call — call the `computer_use` actions documented below.

The canonical workflow

**Step 1 — Capture first.** Almost every task starts with:

computer_use(action="capture", mode="som", app="<the app you're driving>")

Returns a screenshot with numbered overlays on every interactable element AND an AX-tree index like:

#1  AXButton 'Back' @ (12, 80, 28, 28) [Chrome]
#2  AXTextField 'Address bar' @ (80, 80, 900, 32) [Chrome]
#7  Link 'Sign In' @ (900, 420, 80, 24) [Chrome]
...

The role names match the host platform's accessibility framework (`AXButton` on macOS, `Button` on Windows UIA, `push button` on Linux AT-SPI) — treat them as labels, not as strict types.

**Step 2 — Click by element index.** This is the single most important habit:

computer_use(action="click", element=7)

Much more reliable than pixel coordinates for every model. Claude was trained on both; other models are often only reliable with indices.

**Step 3 — Verify.** After any state-changing action, re-capture. You can save a round-trip by asking for the post-action capture inline:

computer_use(action="click", element=7, capture_after=True)

Capture modes

| `mode` | Returns | Best for | |---|---|---| | `som` (default) | Screenshot + numbered overlays + AX index | Vision models; preferred default | | `vision` | Plain screenshot | When SOM overlay interferes with what you want to verify | | `ax` | AX tree only, no image | Text-only models, or when you don't need to see pixels |

Actions

capture           mode=som|vision|ax   app=…  (default: current app)
click             element=N     OR     coordinate=[x, y]    button=left|right|middle
double_click      element=N     OR     coordinate=[x, y]
right_click       element=N     OR     coordinate=[x, y]
middle_click      element=N     OR     coordinate=[x, y]
drag              from_element=N, to_element=M        (or from/to_coordinate)
scroll            direction=up|down|left|right   amount=3 (ticks)
type              text="…"
key               keys="<save shortcut>" | "return" | "escape" | "<modifier>+t"
wait              seconds=0.5
list_apps
focus_app         app="<app name>"   raise_window=false   (default: don't raise)

All actions accept optional `capture_after=True` to get a follow-up screenshot in the same tool call. All actions that target an element accept `modifiers=[…]` for held keys.

Key shortcuts vary per platform

Use the host's idiomatic modifier:

| Common action | macOS | Windows / Linux | |---|---|---| | Save | `cmd+s` | `ctrl+s` | | New tab | `cmd+t` | `ctrl+t` | | Close tab / window | `cmd+w` | `ctrl+w` | | Copy / paste | `cmd+c` / `cmd+v` | `ctrl+c` / `ctrl+v` | | Address bar | `cmd+l` | `ctrl+l` | | App switcher | `cmd+tab` | `alt+tab` |

When in doubt, capture and look for menu hints, or ask the user which shortcut to use.

Background rules (the whole point)

1. **Never `raise_window=True`** unless the user explicitly asked you to bring a window to front. Input routing works without raising. 2. **Scope captures to an app** (`app="Chrome"`) — less noisy, fewer elements, doesn't leak other windows the user has open. 3. **Don't switch virtual desktops / Spaces.** cua-driver drives elements on any virtual desktop / Space regardless of which one is visible. 4. **The user can be on the same machine.** They might be typing in another window. Don't grab focus. Don't pop modals to the front.

Drag & drop

Prefer element indices:

computer_use(action="drag", from_element=3, to_element=17)

For a rubber-band selection on empty canvas, use coordinates:

computer_use(action="drag",
             from_coordinate=[100, 200],
             to_coordinate=[400, 500])

Scroll

Scroll the viewport under an element (most common):

computer_use(action="scroll", direction="down", amount=5, element=12)

Or at a specific point:

computer_use(action="scroll", direction="down", amount=3, coordinate=[500, 400])

Managing what's focused

`list_apps` returns running apps with bundle IDs / process names, PIDs, and window counts. `focus_app` routes input to an app without raising it. You rarely need to focus explicitly — passing `app=...` to `capture` / `click` / `type` will target that app's frontmost window automatically.

Delivering screenshots to the user

When the user is on a messaging platform (Telegram, Discord, etc.) and you took a screenshot they should see, save it somewhere durable and use `MEDIA:/absolute/path.png` in your reply. cua-driver's screenshots are PNG or JPEG bytes (mi