/computer-use-linux

name: computer-use-linux
description: "Linux desktop observation and control via native Pi tools or the computer-use-linux MCP server: accessibility trees, screenshots, window targeting, and input synthesis (click, type, scroll)."
author: agent-sh
license: MIT
platforms: [linux]
compatibility: "Native Pi tools require Pi 0.84.4+ and Node.js 22.19+; the standalone CLI/MCP server supports Node.js 18+."

computer-use-linux

Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.

On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified. Window-targeted screenshots, including `screenshot_after: "window"`, fail for the same reason; capture the full screen instead.

Native X11 clicks, scrolls and drags use xdotool/XTEST on `DISPLAY`, ahead of uinput and portal input. Install xdotool for this route. Missing or failed commands return an error without a system-wide fallback. Drags send paced intermediate motion and always attempt release after a failed press or motion; verify the drop target afterward.

When to Use

Use this skill when:

• The user wants the agent to control a Linux GUI app.
• You need desktop state from AT-SPI, screenshots, or compositor window metadata.
• You are configuring the `computer-use-linux` MCP server for your agent.
• A desktop action needs target-aware input instead of blind shell commands.

Do not use this for remote browsers, websites, or headless automation when a browser-specific tool is available. Do not assume desktop actions are safe just because the MCP connection works.

Install

Pick the install that matches how you will run this skill. You can use both.

Claude Code or Codex plugin

claude plugin marketplace add agent-sh/computer-use-linux
claude plugin install computer-use-linux@computer-use-linux
# or, in Codex:
codex plugin marketplace add agent-sh/computer-use-linux
codex plugin add computer-use-linux@computer-use-linux

The plugin registers the MCP server and downloads the pinned, sha256-verified release binaries on first start. It does not put `computer-use-linux` on `PATH`; shell commands in this skill need the CLI install below. Set `COMPUTER_USE_LINUX_BIN` to run a local build instead.

Pi native tools

pi install npm:@agent-sh/computer-use-linux

This enables Pi's `computer_use_linux_*` tools. It does not put `computer-use-linux` on `PATH`. Shell commands in this skill (`doctor`, `setup`, `setup-window-targeting`, `guard-accessibility`, the MCP `command` config, and Verification) need the CLI install below.

Shell CLI / MCP server

Use this when you need `computer-use-linux` on `PATH` for the commands in this skill.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

Rust users can install from crates.io:

cargo install computer-use-linux
computer-use-linux doctor | jq .readiness

If `doctor` reports missing input or accessibility support, run:

computer-use-linux setup
computer-use-linux setup-window-targeting
computer-use-linux doctor | jq .readiness

If `doctor` selects ydotool as the input backend, also enable its per-user daemon. `./install.sh` creates `ydotoold.service`; Ubuntu's ydotool package ships `ydotool.service` instead (`systemctl --user enable --now ydotool`). Direct uinput, X11 xdotool, and RemoteDesktop portal input do not require `ydotoold`. On Wayland, ydotool sends only untargeted scrolls; a scroll at a point goes through uinput or the portal, and without the daemon an untargeted scroll goes through the portal too.

On niri, the server reads `NIRI_SOCKET` from its parent processes or the systemd user environment when the host does not pass it; without it, discovery needs an unambiguous socket matching `WAYLAND_DISPLAY`. Window listing and exact focus use `niri msg` with a direct IPC fallback. Missing positions or unknown/mixed output scaling leave window-relative coordinates unavailable; focus the target and inspect a fresh full-screen screenshot before using desktop coordinates.

On GNOME Wayland, log out and back in after `setup-window-targeting` if the GNOME Shell extension was newly installed.

For MCP hosts with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`, call the optional `complete_interaction` tool once after finishing desktop interaction. A skipped cue is not a task failure. This notification does not guarantee exclusive desktop ownership or that other clients have stopped sending input. This applies only to directly spawned MCP hosts, not the native Pi extension.

`setup_accessibility` verifies the saved GNOME `toolkit-accessibility` key separately from runtime AT-SPI. Inspect its warning and readback before assuming new apps can expose trees. Other accessibility tools may change the key later; setup does not hold it enabled continuously.

Optional foreground accessibility guard

Skip unless: the user explicitly wants GNOME's saved `toolkit-accessibility` setting kept enabled while desktop automation runs.

Run `computer-use-linux guard-accessibility` in a foreground terminal. It registers a passive AT-SPI window-activation listener and watches/reasserts the saved key with readback. The setting affects all apps for the current user. `mcp`, setup, and `get_app_state` never start this guard automatically. Stop with Ctrl-C or SIGTERM before intentionally disabling accessibility. Stopping ends writes and removes its l