/computer-use

name: computer-use
description: "Use Cua Driver through MCP to inspect and operate the user's native desktop apps on Windows, macOS, or Linux. Trigger when a task requires a desktop application, native file dialog, OS window, signed-in browser UI, screenshot-grounded interaction, or a result that must be verified in the application. Keep browser page work in browser-use when the page bridge is sufficient."
fold_cue: "instead_of=blind_pixel_clicking use=list_windows_then=get_window_state — keep an exact window target and refresh state after navigation or UI changes"

Computer Use — operate native desktop apps with Cua Driver

Wisp uses the installed Cua Driver as an MCP server. Cua Driver owns the platform-specific desktop integration; this Skill owns the agent workflow and the safety boundary. It can operate native apps, native file dialogs, and browser windows that are visible to the host desktop.

Before the first action

Use this Skill only when the Cua Driver tools are advertised in the current conversation. If they are absent, do not invent tool names or fall back to blind shell input. Tell the user to install Cua Driver, add a stdio MCP connection with:

command: cua-driver
args: mcp

Then ask them to reconnect the MCP service. The driver must run in the interactive user session. An SSH or service-session process cannot see the user's desktop. On macOS, Accessibility and Screen Recording permission must be granted to the Cua Driver app identity. On Windows and Linux, report any interactive-session or display-server refusal as a capability boundary.

If the available Cua Driver tool exposes a health, doctor, or permission status call, use it first. Otherwise the first call the task needs anyway is the connection check: `launch_app` for an app to open (it returns the pid and window ids), `list_windows` for one that is already open. `list_apps` enumerates every installed application; call it only to find an app whose name you do not know. A process starting successfully is not evidence that the desktop is controllable.

Tool selection

Use the exact tool schemas advertised by the connected Cua Driver server. The common names are:

• `list_apps` and `launch_app` for application discovery and startup;
• `list_windows` for exact process/window identity;
• `get_window_state` for the accessibility tree plus a window screenshot;
• `get_desktop_state` for the primary desktop screenshot and desktop identity;
• `click`, `type_text`, `press_key`, `hotkey`, `scroll`, and `drag` for input;
• window or session cleanup tools when the driver advertises them.

Cua Driver's schemas arrive through `search_mcp_tools`, and each result is large. Ask once for everything the task will plausibly need (observation, click, text, keys, scroll) instead of searching again before each new kind of action.

Do not guess a selector, process ID, window ID, element token, or coordinate. Read the current state first. For input, prefer a window target with an exact `pid` and `window_id`; use the returned accessibility `element_token` when the control exposes a semantic action. Use window-local pixel coordinates only when the element is not actionable semantically. Use a desktop target only for deliberate foreground screen actions.

Observe → act → verify

Work in steps. A step is one observation, the actions that observation already justifies, and one verification:

1. Discover the app and select one exact window. If several candidates match, stop and resolve the ambiguity instead of choosing by title alone. 2. Call `get_window_state` and keep the resulting window identity and fresh element references together. Treat element tokens as stale after a page navigation, dialog transition, window recreation, or material UI change. 3. Act on what that observation shows. Prefer background delivery when the target and platform support it. Request foreground delivery only for that action when the application requires focus and interrupting the user's desktop is acceptable. 4. Read the same target again and verify the application state or external artifact. A successful input dispatch is not proof that the application handled it. 5. If the result is stale, ambiguous, refused, or unverifiable, follow the returned refusal code and re-observe. Do not retry the same blind action.

One observation, several actions

An observation stays valid until the window's layout changes. When the next inputs all go to controls it already shows, and none of them moves, replaces, or removes those controls (a keypad, a toolbar, a row of checkboxes, the fields of one form), send them back to back, as consecutive tool calls in one turn, then observe once and verify the outcome. Wisp runs a turn's tool calls in the order written, and element tokens stay valid until the next observation of that window. Paying a model round trip and a fresh tree for every key press turns a ten-key entry into minutes.

Observe again before acting when an action navigates, opens or closes a dialog or menu, reloads a list, or recreates the window: the next action depends on a layout you have not seen. Keep an action that commits something (send, submit, delete, overwrite) out of a batch: verify what precedes it, get the confirmation it needs, then issue it alone. If a call in a batch returns an error or a refusal, the calls after it still ran; observe and reconcile before sending anything else.

Each observation is large. Do not repeat one to confirm what the previous result already shows. When the schema lets you skip the screenshot or bound the tree, do so whenever the accessibility tree alone answers the question.

Results come from the application

Report what the final observation shows or what reached the disk, never what you expected to see. Do not work out a value yourself (arithmetic, a count, a name the application will choose) and then look for it: read the real value first, and compute a cross-check