computer-use-linux

⚡ Running this agent 24/7? tiyuvta inference — hosted LLM inference built for always-on agents, OpenAI/Anthropic-compatible APIs.

computer-use-linux reads accessibility trees, takes screenshots, and drives clicks, scrolls, and keystrokes across GNOME, KDE/KWin, Hyprland, i3, and COSMIC — Wayland-first, X11 best-effort.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

The Rust crate is published as computer-use-linux and the npm wrapper as @agent-sh/computer-use-linux. Prebuilt binaries ship with the latest release.

What this is

computer-use-linux is a Rust MCP server and CLI for Linux desktop control. The crate ships the main computer-use-linux binary plus a small computer-use-linux-cosmic helper used only for COSMIC Wayland window management. Any MCP host — Codex Desktop's Linux build, Claude Desktop, Hermes Agent, or your own client — can spawn it and gain full control of the local Linux desktop: read accessibility trees, list and focus windows, take screenshots, click, drag, scroll, type, and invoke semantic accessibility actions.

Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive xdotool against an X11 root window or shell out to OCR over screenshots. Four things set this one apart:

• Wayland actually works. Pointer actions can use the org.freedesktop.portal.RemoteDesktop interface on Wayland, with ydotool / ydotoold (uinput) as the deterministic fallback. Literal text prefers wtype on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, org.freedesktop.portal.Screenshot otherwise, and fall back to spawning gnome-screenshot for background/systemd contexts where both DBus paths are denied.
• Window targeting is compositor-aware. The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, i3 IPC, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed.
• Semantic selectors, not pixel coordinates. Tools like click, perform_action, and set_value accept role / name / text / states selectors backed by AT-SPI. Pixel coordinates remain available as a fallback for rendering-only surfaces (canvas, games, X clients without ATK).
• One JSON readiness report. computer-use-linux doctor returns a structured document covering platform, portals, AT-SPI, windowing, input, and a readiness summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.

The crate was extracted from codex-desktop-linux (the Linux distribution of Codex Desktop), which still bundles this binary as a built-in plugin. This standalone repo is the upstream.

Features

MCP tools exposed by the server:

Diagnostics

• doctor — single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and a capability map of available backends)
• setup_accessibility — enables GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees
• setup_window_targeting — installs and enables the bundled GNOME Shell extension when org.gnome.Shell.Introspect is locked down

Discovery

• list_apps — running desktop apps visible to the AT-SPI registry
• list_windows — compositor windows with title, app id, wm_class, focus state, client type (Wayland/X11), and bounds
• focused_window — the window currently holding keyboard focus
• get_app_state — combined screenshot + accessibility tree for a chosen app, with element indices that the input tools accept. Scope it with app_name_or_bundle_identifier or a window target; an unscoped call returns the whole desktop tree, reports tree_scoped: false, and warns
• screenshot — capture the screen as a bounded PNG or JPEG image; can target a window, which is raised to the front and cropped to just that window

Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass max_width, max_height, max_bytes, scale, format: "jpeg", or quality, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes coordinate_width, coordinate_height, scale, format, and quality so callers can convert from a downscaled preview to desktop coordinate pixels.

Input

• click — by element index, semantic selector, or desktop coordinate pixels
• drag — desktop coordinate drag (start / end)
• scroll — page-based scroll on an element or at a pixel location
• press_key — keys / chords; can focus a window or terminal first
• type_text — literal text input, optionally targeted at a window or terminal

For a plain left click by element index or selector, a recognized native AT-SPI click, press, or toggle action takes precedence over bounds. Entry activate and slider jump must be requested explicitly with perform_action; they are never a substitute for a pointer click, even when bounds are missing. This avoids pointer conversion for GTK3 HiDPI extents and GTK4 zero-origin bounds when the element exposes such an action. The preference does not substitute an arbitrary action name for a coordinate click. Explicit x/y, right clicks, and double/multiple clicks retain pointer semantics. Re-check application state after either kind of activation.

For coordinate click or scroll with relative: true, select a target window and measure from its clipped screenshot crop origin. Divide preview x and y by the returned screenshot scale before passing them. These are not raw GDK surface or widget-local coordinates; decorations and clipping can change the origin. A missing window target is rejected.

Targeted press_key/type_text results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. get_app_state returns a compact readiness block by default; pass verbose: true for the full diagnostics report. It also reports tree_scoped (false when no app target narrowed the AT-SPI tree, with a warning in message) and accessibility_tree_truncated (true when the node, depth, or read budget stopped traversal with unread elements left).

Semantic actions

• perform_action — invoke any AT-SPI action exposed by an element (Press, Activate, Toggle, …); defaults to the primary action
• set_value — write to a settable accessibility element (text fields, sliders, spinners)

Navigation

• activate_window — focus a window by window_id, pid, app_id, wm_class, title, or terminal selectors
• move_window / resize_window — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen

Conditional host execution

• complete_interaction - optional desktop completion notification, registered only with COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1. Repeated calls can create repeated notifications; it does not provide desktop exclusivity.

• run_shell — same-user /bin/sh -c execution without login-profile loading, registered only when the server operator starts the MCP process with COMPUTER_USE_LINUX_ENABLE_SHELL=1. It is deliberately absent by default and is not a sandbox.

MCP safety contract

computer-use-linux is not a read-only data source. It can observe the local desktop and, when a mutating tool is called, can change real application state. The tools/list response includes MCP ToolAnnotations so hosts can surface this distinction before invocation:

Annotations are safety hints, not an authorization system. MCP hosts should still ask the user before calls that could submit, delete, send, purchase, overwrite, or otherwise commit state.

run_shell is an explicit trust-boundary opt-in, not a restricted command runner. Enabling it grants an approved MCP call the same file and network authority as the user running the server. The tool clears the ambient environment and inherits only a small desktop/runtime allowlist (PATH, home/user/locale fields, display/session-bus fields); additional variables must be supplied in the visible call payload. Commands use a fixed non-login /bin/sh, an existing canonical working directory, a 30-second default / 120-second hard timeout, process-group cleanup, and stderr audit records keyed by the command SHA-256 rather than command text. Collected streams up to 8 MiB are returned with a 512 KiB per-stream response cap and truncation flag; exceeding 8 MiB on either stream fails the call without partial output. These controls bound accidental leakage and runaway work; they do not make arbitrary shell code safe.

The binary also exposes the same capabilities from the CLI for scripting and debugging: