/agent-desktop

allowed-tools: Bash(agent-desktop:*), Bash(npx agent-desktop:*)
description: Operate native desktop applications on macOS, Windows and Linux through accessibility trees with the agent-desktop CLI. Use when an AI agent must see or act on a desktop GUI, such as clicking buttons, filling forms, reading UI state, navigating menus, toggling checkboxes, scrolling, dragging, typing text, pressing shortcuts, taking screenshots, managing windows, using the clipboard, launching or closing apps, handling OS dialogs and permission prompts, or reading notifications. Triggers include requests to "operate the desktop", "computer use", "click a button in an app", "fill a native form", "read the UI", "open an app", "snapshot an app" or any native desktop GUI task. For websites and web apps use agent-browser instead.
hidden: true
metadata:
    github-path: skills/agent-desktop
    github-ref: refs/tags/v1.2.0
    github-repo: https://github.com/vercel-labs/agent-desktop
    github-tree-sha: 4647b3cc9b929688002a9ae0f3bdd983898e049b
name: agent-desktop

agent-desktop

agent-desktop is a CLI that reads and operates desktop apps through the OS accessibility tree. Every command prints one JSON envelope (`ok`, `data` or `error`). It is a tool, not an agent: the observe, act and verify loop runs in you.

Install and permissions

If `agent-desktop` is not on `PATH`, make one bounded attempt to install it: `npm install -g agent-desktop` (Node 18 or later; macOS 13+, Windows 10 1809+ or Ubuntu 24.04+, on x64 or arm64), then `agent-desktop version`. If npm blocks install scripts, approve them (`npm install-scripts approve agent-desktop` on npm 12, `npm install -g agent-desktop --allow-scripts=agent-desktop` on npm 11) so each call does not start Node first. Report what the install did. If it fails, report the command and the error instead of switching to another tool. Atomic bundles this skill, so do not run `npx skills add` for it.

`agent-desktop status` is the readiness check. On macOS, `permissions.accessibility` must be `granted`, and `screen_recording` too for `screenshot`: run `agent-desktop permissions --guide` and let the user approve it. Windows needs no grant for apps at the terminal's integrity level. On Linux, run inside the user's graphical session; an empty `supported_surfaces` means the AT-SPI bus did not answer. Never script around a consent dialog.

For web pages use `agent-browser`. For the web contents of a Chromium or Electron app you launch, `agent-desktop launch <app> --cdp` opens a debugging port that `agent-browser connect <port>` drives (see `commands-system`).

Shell syntax

Examples use a POSIX shell. In PowerShell, quote every ref (`'@s8f3k2p9:e1'`), because a bare `@token` is the splatting operator. Use `$env:NAME = 'value'` instead of `export`, and `| ConvertFrom-Json` instead of `$(...)` and `jq`. The Windows skill has the PowerShell forms.

First run

1. Run `agent-desktop status`. It reports the platform, permissions and `supported_surfaces`. 2. Run `agent-desktop skills get platform`. It loads the skill for the OS the binary runs on (macOS, Windows or Linux). 3. For flags and syntax, run `agent-desktop <command> --help`. This skill does not repeat flag lists.

The loop

Use `find` when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region

• A small app needs no skeleton. Take a full `snapshot -i`.
• `-i` keeps interactive nodes only, so a row's static-text label is dropped. To tell which row a repeated button belongs to, drill that row's container with `--root` and leave out `-i`. On Windows and Linux that reads the row but does not make its button actionable: a button repeated in other rows is a twin in the window and its ref is refused (see Refs).
• A `--root` drill adds its refs to the same snapshot id. Refs outside the drilled subtree, and the drilled root itself, stay valid; refs that were already inside it are replaced, so an old ref there returns `STALE_REF`.
• Never skeleton a surface (`--surface menu`, `sheet`, `alert`). A surface is already focused.
• An action that opens a menu, sheet or alert lists it in `data.surfaces`. Target that overlay next.
• After an async step, use `wait` (see `commands-system.md`). A fixed sleep is a race.

Refs

• Output refs are qualified: `@<snapshot_id>:e<N>`, for example `@s8f3k2p9:e1`. Pass them to later commands as they are.
• Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
• Only addressable elements get refs. Static text and inert containers do not.
• A command re-identifies the element before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, counted even below a skeleton or depth cut, and exactly one live element still matches. Identical twins there have no ref path: only `--headed mouse-click --xy X,Y` can choose one, it takes the foreground, and a Linux Wayland session refuses it.
• `STALE_REF`: the element is gone or no longer matches. `details.kind` is `saved_ref_absent`, or `retained_object_stale` on macOS. Take a fresh snapshot and use the new ref.
• `AMBIGUOUS_TARGET`: an identical element appeared (`saved_ref_ambiguous`), or the ref was not proven unique at capture (`identity_not_unique`). Repeating the ref or re-snapshotting identical twins gives the same refusal; act on a control that is unique in the window. `identity_not_u