/open-gui-remote-control

name: open-gui-remote-control
description: Control an Android phone through OpenGUI from Codex or Claude Code. Use when an agent should list online devices, run a natural-language mobile task, check execution status, pause, resume, or cancel through the local OpenGUI backend and CLI.

OpenGUI Remote Control

Use this skill when Codex or Claude Code needs to operate an Android phone through OpenGUI.

The agent should not drive the phone with raw `adb shell input` commands. The supported path is:

Codex / Claude Code
  -> server CLI or REST API
  -> OpenGUI backend task/execution services
  -> standby dispatch
  -> Android client
  -> execution socket action loop

Trigger Guidance

Start this skill when the user asks for any of these:

• "Use OpenGUI to control my phone"
• "让 Codex 操控手机"
• "Run this task on the Android device"
• "Use the OpenGUI CLI"
• "List OpenGUI devices"
• "Check / pause / resume / cancel an OpenGUI execution"
• "让 Claude Code 通过 OpenGUI 跑手机任务"

If the user asks to install or bootstrap OpenGUI from scratch, use `open-gui-bootstrap` first. After backend and Android client are running, return to this skill for task execution.

Core Rules

• First obtain or locate a runnable OpenGUI checkout. The CLI cannot work without the repository.
• Use the repository CLI first: `cd server && pnpm opengui -- ...`.
• Use `--json` whenever the result will be parsed by Codex or Claude Code.
• Do not ask the user to run terminal commands that the agent can run.
• Ask the user only for physical phone actions, Android permissions, or missing secrets.
• Do not change Android socket event names or payloads.
• Do not use IM commands for this workflow.
• Do not bypass OpenGUI with coordinate-only `adb shell input` scripts.
• Treat `devices` as standby presence only; execution can still fail because of model config, Android permissions, app state, or device lifecycle.

Repository Source

OpenGUI's runnable source checkout is:

https://github.com/Core-Mate/open-gui

The checkout must contain both:

• `server/package.json`
• `client/start.sh`

If those paths are missing, the current directory is not the runnable OpenGUI checkout.

Local Or Remote Workspace

Before using this skill, decide where OpenGUI should run.

Use a local workspace when:

• the Android phone is connected to the same machine by USB
• `adb reverse tcp:7777 tcp:7777` should be used
• the agent has terminal access to the developer machine

Use a remote workspace only when:

• the user explicitly asks to run OpenGUI on a remote host
• the remote host can reach the Android device or a device bridge
• the backend URL used by the Android client is reachable from the phone

Do not silently choose a remote host for phone control. USB debugging, `adb reverse`, Android build/install, and phone-side permissions are usually local-machine operations.

Checkout Acquisition

If the current directory already contains the runnable checkout, use it.

If the current directory is a wrapper directory, search one level down for the runnable checkout before cloning:

find . -maxdepth 3 -type f -path '*/server/package.json' -print
find . -maxdepth 3 -type f -path '*/client/start.sh' -print

If no runnable checkout exists and the user wants the agent to set it up locally, clone the public repository:

git clone https://github.com/Core-Mate/open-gui.git
cd open-gui

If the destination already exists, do not overwrite it. Enter the existing directory, inspect `git status`, and pull only when the user asked for the latest code or when the checkout is clean enough to update safely.

If the user wants a remote setup, SSH to the remote host first, then perform the same checkout detection or clone on that host. Keep the backend URL and Android connectivity explicit; a backend running on a remote host will not be reachable through local `adb reverse` unless the user has provided a bridge.

Preconditions

Before sending a task, verify these conditions:

• backend is reachable at `http://localhost:7777` unless the user gave another base URL
• Android client is installed and open
• `adb reverse tcp:7777 tcp:7777` has been applied for a USB-connected phone
• phone-side USB debugging is approved
• Accessibility Service is enabled
• overlay permission is enabled when needed
• at least one standby device is online

If the backend or client is not running, use the repository scripts:

cd server
./start.sh

cd client
./start.sh

CLI Reference

Run commands from the `server/` directory.

List online standby devices:

pnpm opengui -- devices --json

Create and run a new task:

pnpm opengui -- do "观察当前手机屏幕，简要描述你看到了什么，然后结束" --json

Run a task on a specific device:

pnpm opengui -- do "打开设置，检查当前网络状态" --device <deviceId> --json

Run an existing task:

pnpm opengui -- run <taskId> --json

Check execution status:

pnpm opengui -- status <executionId> --json

Pause, resume, or cancel:

pnpm opengui -- pause <executionId> --json
pnpm opengui -- resume <executionId> "继续执行，但不要打开新的 App" --json
pnpm opengui -- cancel <executionId> --json

Use a non-default backend:

pnpm opengui -- devices --base-url http://localhost:7777 --json

Base URL priority:

--base-url > OPENGUI_BASE_URL > http://localhost:7777

Standard Workflow

1. Obtain the runnable checkout

Find or clone `https://github.com/Core-Mate/open-gui`.

Then work from the repository root that contains both:

• `server/package.json`
• `client/start.sh`

If the current directory is a wrapper repo, find the nested runnable checkout before running commands. If no runnable checkout exists, clone it or ask for the intended repository location.

2. Verify backend

Check the backend before task dispatch:

curl -fsS http://localhost:7777/docs >/dev/null

If this fails, start the backend:

cd server
./start.sh