clawdcursor

---

What it is

Clawd Cursor is a local MCP server that gives any tool-calling agent — Claude Code, Cursor, Windsurf, OpenClaw, the Claude Agent SDK, or your own loop — safe control of the real desktop. It clicks, types, reads the screen, opens apps, and drives any GUI the way a human would: native apps, the browser, even a canvas.

Most "let an agent use the computer" tools take a screenshot and feed it to a vision model — slow, expensive, and brittle. Clawd Cursor compiles the screen into one UI map: it fuses the accessibility tree and OCR into a confidence-scored set of elements, each tagged with a stable el_NN id, and acts on elements by id — not pixel coordinates. Coordinates appear only in the last-resort screenshot/vision tier (live pixels off the current frame), for canvas-only apps or tasks that genuinely need spatial reasoning. The result is cheaper, faster, private, and — uniquely — it checks that each action actually did what it claimed.

If a human can do it on a screen, your agent can too. No API, no integration, no problem — only the right sequence of reads, clicks, keys, and waits. Use it as the last-mile fallback: native API exists? Use it. CLI? Use it. Clawd Cursor is for the click, the legacy app, the GUI with no public surface.

---

Why it's different

The desktop-agent space is crowded. The closest install-and-go peers are Windows-MCP and Terminator (desktop MCP servers); browser-only tools (browser-use, Playwright MCP) are adjacent; and OmniParser / UI-TARS are vision-centric parsing approaches you'd build an agent around, not products you install. Here's the honest comparison across those approaches — what Clawd Cursor does that the popular options don't:

	Clawd Cursor	browser-use	Playwright MCP	OmniParser / UI-TARS	computer-use
Any desktop app, not just the web	✅	web only	web only	✅	✅
Cross-OS (Windows + macOS + Linux)	✅	—	—	varies	sandbox
Perception without a vision model	✅ compiled a11y + OCR map	DOM	a11y tree	❌ vision-centric	❌ vision
Verifies its own actions (deviation)	✅	—	—	—	—
Single safety chokepoint (allow/confirm/block)	✅	—	—	—	—
Any model / vendor	✅	✅	not an agent	model-specific	Claude only
MCP-native (one config, any host)	✅	library	test framework	—	tool-use API
Local-only, no cloud required	✅	✅	✅	needs a model	screens → cloud

Three things here are genuinely rare:

1. Cheapest-tier-first perception, fully local. Accessibility tree (free) → OCR (cheap) → screenshot (expensive — the only tier that puts pixels in the model's context; "screenshot" and "vision" are the same step). The agent climbs only when it must, so token cost tracks task difficulty — and with a local model, nothing leaves the machine. Vision-centric agents (OmniParser, UI-TARS) need a screenshot in the model for every observation.
2. It verifies. Pass expect on a consequential action and Clawd Cursor re-checks the live screen (with a short settle window for async UIs) and reports a DEVIATION instead of a hollow "success." A completed task can't be marked done on evidence that was already true before it acted.
3. One safety gate. Every call — from an editor over stdio, an external agent over HTTP, or the built-in loop — routes through a single safety.evaluate() chokepoint (allow / confirm / block) before it touches the desktop. The agent cannot bypass it.

Plus: an on-screen "desktop control in progress" banner with a blinking red dot whenever an agent is driving — double-click it to stop. A human at the machine always knows, and always has a kill switch.

---

Install

clawdcursor is an MCP server published to npm — install it into any MCP-capable agent (Claude Code, Claude Desktop, Cursor, Windsurf, Zed, OpenAI Codex, or your own loop) the same way you install any other MCP server.

1 — Install the engine + grant consent (once)

npm i -g clawdcursor
clawdcursor consent --accept    # one-time desktop-control consent (required)
clawdcursor grant               # macOS only — approve Accessibility + Screen Recording

Zero-install also works — swap clawdcursor for npx -y clawdcursor in any snippet below and npx fetches it on demand. A global install is recommended anyway: it's pinnable and inspectable on disk (safer for a tool with full desktop control than auto-fetching latest every run), and it's the path on which the macOS native helper builds at install time. Requires Node.js 20+.

Per-OS prerequisites. Windows installs clean — sharp and @nut-tree-fork/nut-js ship prebuilt binaries, so no C++/Python build tools are needed. macOS needs Xcode Command Line Tools (xcode-select --install) for screenshots / vision; core accessibility-driven control still works without them. Linux needs a few system packages npm can't install: tesseract-ocr (OCR), python3-gi + gir1.2-atspi-2.0 (accessibility tree), and — on Wayland — ydotool (synthetic input).

2 — Add it to your agent (pick your host)

Claude Code

claude mcp add clawdcursor -s user -- clawdcursor mcp --compact

OpenAI Codex — add to ~/.codex/config.toml:

[mcp_servers.clawdcursor]
command = "clawdcursor"
args = ["mcp", "--compact"]

Cursor / Windsurf / Claude Desktop — add to the host's MCP config:

{
  "mcpServers": {
    "clawdcursor": { "command": "clawdcursor", "args": ["mcp", "--compact"] }
  }
}

Zed — Zed uses context_servers (not mcpServers) in settings.json:

{
  "context_servers": {
    "clawdcursor": { "command": { "path": "clawdcursor", "args": ["mcp", "--compact"] } }
  }
}

That's the whole setup. Ask your agent: "open Outlook and reply to the latest email from Sarah."

Or: one-command plugin (Claude Code)

Skip the manual config — this repo ships a plugin that registers the tools and bundles the usage skill in one step. It resolves the package's bin (never a hard-coded dist/ path), so an upgrade can't break it:

claude plugin marketplace add AmrDab/clawdcursor
claude plugin install clawdcursor@clawdcursor

# Windows (PowerShell)
powershell -c "irm https://clawdcursor.com/install.ps1 | iex"

# macOS / Linux
curl -fsSL https://clawdcursor.com/install.sh | bash

Notes. You never run clawdcursor mcp yourself — the host spawns it over stdio on demand. clawdcursor doctor is not part of MCP setup; it only configures the built-in LLM for the autonomous agent daemon. On macOS, Accessibility is required (primary control path); Screen Recording is optional (vision fallback only). For editor permission allowlists, use the server-level wildcard mcp__clawdcursor rather than per-tool entries — it survives tool renames.

---

The engine

The perception + verification core (the UI State Compiler, since v1.5.0):

• compile_ui fuses the accessibility tree and OCR into one confidence-scored map of the screen, every element tagged with a stable el_NN id. Act on an element by {element_id, snapshot_id} instead of pixels — near-free in tokens, and it survives DPI, resize, and layout shifts. find_button / find_field locate a target by meaning and hand you the id.
• Reactive verification. expect on an action → Clawd Cursor confirms the outcome on the live screen and returns a DEVIATION when the UI didn't obey.
• Cross-platform parity. The compiler, secure-field redaction, and coordinate handling run on Windows, macOS, and Linux; the external-agent (MCP) surface resolves el_NN refs through the safety gate and discloses when it attached to your existing browser.

Set-of-Mark-style element IDs and a11y/OCR fusion aren't new ideas on their own — what's rare is doing them locally, a11y-first (no vision model required), with a built-in verification gate and one safety chokepoint, across three operating systems, behind a single MCP config.

See the changelog for the full release history, or the latest release.

---

How it works

Where the brain lives decides how you run it. Both modes can run side-by-side.

Brain lives…	Mode	Command	What you call