/cua

Computer use (Cua Driver)

Drive native desktop apps (read the screen, click, type) through [Cua Driver](https://cua.ai). The user installs Cua Driver and turns on **Let agents use this computer** in **Settings > Computer use agents**. Chat sessions started while it is on get these tools from AO's `ao` tool server:

• `list_windows`: the open windows, with each one's `pid` and `window_id`.
• `launch_app`: open an app in the background.
• `bring_to_front`: bring a window to the front so it draws again. It takes

keyboard focus from the user, so ask first (see below).

• `get_window_state`: a window's accessibility tree and screenshot.
• `click`, `type_text`, `press_key`: act on one element or window.
• `run_steps`: several actions in one call. `hotkey`, `scroll`, `set_value`,

and `drag` are `run_steps` steps; its description lists their arguments.

Each tool's description and input schema come from the installed Cua Driver. Read them before the first call; do not guess argument names. Window reads and actions return text plus the window's screenshot as an image. This example is from macOS, whose accessibility roles start with `AX`:

✅ Performed AXPress on [5] AXButton "".

window_id=37589 pid=44788 size=460x816 elements=33 snapshot_id=s0000002f

since s0000002e: 0 added, 1 changed, 0 removed. Use element_tokens s0000002f:N ...
~ AXStaticText = "7" (Edit field)
Invalidated snapshots s0000002e: their element_tokens are stale.

The tree's roles and actions come from the platform's accessibility API, so they differ by platform; the workflow does not. On macOS a row reads `- [12] AXButton (7) [id=Seven actions=[press]]`. On Windows (UI Automation) it reads `- [4] Button "Close Tab" [id=CloseButton actions=[invoke]]` or `- [0] Document "Text editor" [actions=[set_value,text,scroll]]`. Match the role and label you see, not `AX` names.

Workflow

1. `list_windows` and pick the `pid` and `window_id` of the app. 2. `get_window_state` with that `pid` and `window_id` returns the window's accessibility tree, a `snapshot_id`, and a screenshot. Each actionable row starts with a number in brackets, such as `[12]` in `- [12] AXButton (7) [id=Seven actions=[press]]` (macOS) or `[4]` in `- [4] Button "Close Tab" [id=CloseButton actions=[invoke]]` (Windows). Look at the screenshot when the tree is not enough. 3. Act by element with an `element_token`: the snapshot's `snapshot_id`, a colon, and the row number. Row `[12]` of snapshot `s00000019` is `{"element_token":"s00000019:12"}`; the token carries the pid and window, so `click`, `type_text`, `press_key`, and the `run_steps` actions need nothing else to find the element. Acting by element does not move the user's mouse and usually works on background windows (see below for when it does not). Use `x`/`y` pixels only for canvases and other surfaces missing from the tree; they are in the screenshot's pixels, so pass that window's `pid` and `window_id` with them. 4. Check the result before the next step. AO reads the window after every action and returns it under the action's summary: the screenshot, the window line with a new `snapshot_id`, and the rows that changed since your last read (the full tree when there is no earlier read with the same `query`). Look at the screenshot; the driver often cannot confirm an action's effect. The read makes earlier tokens stale (`stale_element_token`), and live apps reorder their lists, so act next with the new `snapshot_id`, as the diff says. If the result says `observation skipped`, AO could not tell the window: read it with `get_window_state`. 5. If an action by token did something other than what the screenshot led you to expect (it hovered or opened a neighbouring element, say), the page has likely moved since your read: it was still loading, or the window was not painted, so the tree and the real layout disagree. Read the window again and act with a fresh token before falling back to `x`/`y` pixels.

Background windows do not always render. Chromium-based browsers and apps (Chrome, Brave, Edge, Electron), on Windows especially, throttle or stop painting windows that are hidden behind others, so `get_window_state` can return a blank page, or a lazily loaded dialog that never gets past its loading skeleton; AO marks such a read with a `warning:` line. If a window reads blank or stuck loading twice, stop retrying and ask the user whether you may bring it to the front. If they agree, call `bring_to_front` with its `pid` and `window_id`, which sends no keys or clicks, and act on the window it returns. If they say no, or are working in another app, leave the window alone and tell them what you could not do. Skip asking only when the user said they are away from the computer. The window stays drawn only while it is in front: if the user switches away, it can go blank again.

AO refuses foreground input (`delivery_mode: "foreground"` and `run_steps`' `foreground_fallback`): it would raise the window and send real keys and clicks to whatever has focus, which may be the user's own window. When the driver suggests it, bring the window to the front instead, as above.

`run_steps` runs several actions in one call and returns one line per step (`2. click failed (find): ...` names the step that stopped the batch), then the window after the last step with one screenshot. Pass `"observe":{"include_screenshot":false}` to get the tree diff only.

Type into a field by its token: `type_text` with the field's `element_token`. Never type into "whatever has focus": a click that missed, or a shortcut such as Cmd+F that did not move focus, leaves the cursor in another field. In messaging apps, confirm in the screenshot that you are in the right field and chat before typing, and again before pressing Return to send.

AO refuses actions that would land wherever the user is working: `scope: "desktop"` (or a desktop `target`), pixel coordinates without th