/agent-desktop-windows

name: agent-desktop-windows
tags: windows-automation, accessibility, uia, ai-agent, gui-automation, cli
requirements:
  - agent-desktop
description: >
  Windows platform guide for the agent-desktop CLI. Lists per command group
  what works on Windows and what refuses, the first-contact gotchas
  (PowerShell ref quoting, minimized windows, headed-only input), and where
  to find shell surfaces, notifications, the cursor overlay, UIPI elevation,
  Chromium/Electron settle behavior and troubleshooting.
  Triggers on: "windows desktop automation", "UIA tree", "automate Windows app",
  "agent-desktop on Windows", "elevated window input blocked", "E_ACCESSDENIED",
  or any Windows GUI interaction task.

agent-desktop-windows

The observe-act loop, ref system, JSON envelope, sessions and tracing are the same on every platform. This file covers only what differs on Windows.

Requires Windows 10 1809+ / Windows Server 2019+ (x64 or ARM64). Run it in the interactive desktop session of the signed-in user. Session 0 (services and SYSTEM tasks), Server Core, the secure desktop (UAC prompts, the sign-in screen), a locked desktop and another user's session are not supported: observation, actions and capture there fail or return nothing.

Capability Table

| Group | Commands | Status on Windows | |-------|----------|-------------------| | Observation | `snapshot`, `find`, `get`, `is`, `screenshot` | Works. Surfaces are `window`, `focused`, a Chromium modal as `sheet`, and an open application menu as `menu` | | Surfaces | `list-surfaces` | Works. Lists `window`, `focused` and `sheet` per process, plus `menu` with `item_count` while a menu is open | | Shell surfaces | `open-system-surface` | Works for `start-menu`, `taskbar`, `system-tray`, `system-tray-overflow` and `action-center`, and returns the window `snapshot --surface <kind>` reads. `quick-settings` returns `PLATFORM_NOT_SUPPORTED` naming `action-center` | | Ref interaction | `click`, `right-click`, `type`, `set-value`, `clear`, `select`, `toggle`, `check`, `uncheck`, `expand`, `collapse`, `scroll`, `scroll-to` | Works, semantic in headless and `--headed` mode. `right-click` and `type` need `--headed` and return `POLICY_DENIED` headless | | Multi-click and focus | `double-click`, `triple-click`, `focus` | Works with global `--headed` only | | Keyboard and mouse | `press`, `hover`, `drag`, `mouse-click`, `mouse-move`, `mouse-wheel` | Works. Cursor-moving input needs `--headed`. First Contact covers `press --app` | | Held input | `key-down`, `key-up`, `mouse-down`, `mouse-up` | Fails closed on every platform with `ACTION_NOT_SUPPORTED`. A headless `mouse-down` or `mouse-up` returns `POLICY_DENIED` first | | App and window | `launch`, `close-app`, `list-windows`, `list-apps`, `focus-window`, `resize-window`, `move-window`, `minimize`, `maximize`, `restore` | Works. First Contact covers the name `launch` takes | | Displays | `list-displays` | Works | | Clipboard | `clipboard-get`, `clipboard-set`, `clipboard-clear` | Works; typed text and image content | | Wait | `wait` | Works for ms, `--element`, `--window`, `--text`, `--menu`, `--menu-closed`, `--event` and `--notification` | | Notifications | `list-notifications`, `dismiss-notification`, `dismiss-all-notifications`, `notification-action` | Works over the Action Center. Pass `--headed` when the center is closed | | Cursor overlay | `cursor-overlay` | Works. `enable` returns `data.rendered: true` only after the renderer acknowledges over its pipe. See `references/shell-and-overlay.md` | | System | `status`, `permissions`, `version`, `batch`, `skills`, `session`, `trace` | Works |

First Contact

• **Quote every ref in PowerShell.** PowerShell reads a bare `@token` as its

splatting operator and deletes the argument before the binary sees it, so `set-value @s8f3k2p9:e1 hi` fails `INVALID_ARGS`. Write `set-value '@s8f3k2p9:e1' hi`. cmd.exe and bash need no quoting.

• **A window behind another window is drivable. A minimized one is not.**

Every element of a minimized window reports `offscreen` and ref actions fail. Run `restore --app <image>` and snapshot again. Do not use `focus-window` by reflex: it steals the user's foreground.

• **No permission dialog exists.** `permissions` reports `automation` as

`not_required`. Elevation boundaries are the only access control.

• **Chromium and Electron apps read thin before they settle.** Pass a longer

`--timeout-ms` to `snapshot` instead of concluding the tree is empty.

• **Blocked shortcuts.** `alt+f4`, `win+l`, `win+d`, `alt+tab`, `win+r`,

`win+x`, `win+e`, `ctrl+shift+esc` and their modifier supersets are refused without `--force`. In combos, `cmd` and `meta` are the Windows key, so `meta+c` is a shell shortcut. Write `ctrl+c` to copy.

• **`type` needs `--headed`.** It is keystroke synthesis through `SendInput`,

so a strict-headless `type` returns `POLICY_DENIED`. Use `set-value`, or pass `--headed`. It ends `delivered_unverified` (`verification_scope: "unavailable"`) when the field exposes no readable Text selection, and can verify when one is readable: confirm with a fresh `get`.

• **`press --app` writes to the foreground queue** with no per-process

targeting. A headless press to a non-frontmost app fails closed. A non-interactive caller (service, scheduled task, CI job) gets `delivered_unverified`.

• **`launch` takes an absolute path or a bare name** under System32 or the

Windows directory, with or without `.exe` (`notepad` or `notepad.exe`), not a display name. Use `list-apps` to find a running app.

• **Notifications:** `list-notifications` with the Action Center closed needs

`--headed`. Single-item `dismiss-notification` and `notification-action` need `--headed` and `--expected-app` or `--expected-title` for the entry at INDEX.

• **Surfaces:** an open or context menu is the `menu` surface. Windows does

not advertise a `menubar` surface.

• **Commands that raise shell chrome take the foreground.