Automation
Plugin
midscene-skills
Vision-driven cross-platform automation Natural-language driven UI control Built on Midscene.js's vision-based automation capabilities — Operates entirely from screenshots, making cross-platform support reliable This repository contains Skills for the
Plugin Signal74/100Solid
maintained98
capable41
documented100
adopted52
Install
$ npx -y skills add web-infra-dev/midscene-skills --agent claude-code
Stats
292
Stars
24
Forks
Active
Maintenance
JavaScript
Language
MIT
License
1d ago
Last commit
7mo ago
Created
Repo: web-infra-dev/midscene-skills
What's inside
Skill(6 skills)
/android-automationVision-driven Android device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control Android devices with natural language commands via ADB./browserVision-driven browser automation using Midscene. Operates from screenshots — no DOM or accessibility labels needed. Runs in headless Puppeteer — does NOT take over the user's mouse or keyboard. Also supports CDP mode and Bridge mode to connect to an existing Chrome. Use this/computer-automationVision-driven desktop automation using Midscene. Control your local desktop (macOS, Windows, Linux) or a remote Windows desktop over RDP with natural language commands. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible/harmony-automationVision-driven HarmonyOS NEXT device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control HarmonyOS devices with natural language commands/ios-automationVision-driven iOS device automation using Midscene CLI. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control iOS devices with natural language commands via/vitest-midscene-e2eEnhances Vitest with Midscene for AI-powered UI testing across Web (Playwright), Android (ADB), and iOS (WDA). Scaffolds new projects, converts existing projects, and creates/updates/debugs/runs E2E tests using natural-language UI interactions. Triggers: write test, add test,
Badges
README
Midscene Skills
Vision-driven cross-platform automation
- Natural-language driven UI control
- Built on Midscene.js's vision-based automation capabilities — Operates entirely from screenshots, making cross-platform support reliable
- This repository contains Skills for the following platforms:
- Browser (Puppeteer, headless Chrome; also supports CDP connection):
skills/browser - Chrome Bridge (user's own Chrome browser):
skills/chrome-bridge - Desktop (macOS, Windows, Linux):
skills/computer-automation - Android (controlled via ADB):
skills/android-automation - iOS (controlled via WebDriverAgent):
skills/ios-automation - HarmonyOS (controlled via HDC):
skills/harmony-automation - Vitest + Midscene E2E (scaffold, convert, and manage AI-powered E2E tests for Web, Android, iOS):
skills/vitest-midscene-e2e
- Browser (Puppeteer, headless Chrome; also supports CDP connection):
Safety Warning
⚠️ AI-driven UI automation may produce unpredictable results since it can control EVERYTHING on the screen. Please evaluate the risks carefully before use.
Installation
Make sure you have Node.js installed.
Then install the skills:
# General installation
npx skills add web-infra-dev/midscene-skills
# Claude Code
npx skills add web-infra-dev/midscene-skills -a claude-code
# OpenClaw
npx skills add web-infra-dev/midscene-skills -a openclaw
Model Setup
Midscene requires models with strong visual grounding capabilities (accurate UI element localization from screenshots).
Because of this, you need to prepare model access and configuration separately from skill installation.
Make sure these environment variables are available in your system. You can also define them in a .env file in the current directory, and Midscene will load them automatically:
MIDSCENE_MODEL_API_KEY="your-api-key"
MIDSCENE_MODEL_NAME="model-name"
MIDSCENE_MODEL_BASE_URL="https://..."
MIDSCENE_MODEL_FAMILY="family-identifier"
Example: Gemini (Gemini-3-Flash)
MIDSCENE_MODEL_API_KEY="your-google-api-key"
MIDSCENE_MODEL_NAME="gemini-3-flash"
MIDSCENE_MODEL_BASE_URL="https://generativelanguage.googleapis.com/v1beta/openai/"
MIDSCENE_MODEL_FAMILY="gemini"
Example: Qwen3-VL
MIDSCENE_MODEL_API_KEY="your-openrouter-api-key"
MIDSCENE_MODEL_NAME="qwen/qwen3-vl-235b-a22b-instruct"
MIDSCENE_MODEL_BASE_URL="https://openrouter.ai/api/v1"
MIDSCENE_MODEL_FAMILY="qwen3-vl"
Example: Doubao Seed 1.6
MIDSCENE_MODEL_API_KEY="your-doubao-api-key"
MIDSCENE_MODEL_NAME="doubao-seed-1-6-250615"
MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
MIDSCENE_MODEL_FAMILY="doubao-vision"
Commonly used models: Doubao Seed 1.6, Qwen3-VL, Zhipu GLM-4.6V, Gemini-3-Pro, Gemini-3-Flash.
Model setup docs:
- Midscene model strategy: https://midscenejs.com/model-strategy
- Qwen3-VL: https://midscenejs.com/model-common-config#qwen3-vl
- Gemini-3-Flash: https://midscenejs.com/model-common-config#gemini-3-pro-and-gemini-3-flash
- Doubao Seed 1.6: https://midscenejs.com/model-common-config
Use
In your chatbot or coding agent, you can say:
Use Midscene computer skill to open the Keynote app and create a new presentation.
Use Midscene browser skill to open the Google search page and search for "Midscene".
Issues
For bug reports, feature requests, and discussions, please visit the main Midscene repository: https://github.com/web-infra-dev/midscene/issues
License
MIT
File tree
web-infra-dev/midscene-skillsView on GitHub
.github/
workflows/
check-skill-description.yml
.gitignore
LICENSE
README.md
scripts/
check-skill-description-length.mjs
skills/
android-automation/
SKILL.md
browser/
SKILL.md
computer-automation/
SKILL.md
harmony-automation/
SKILL.md
ios-automation/
SKILL.md
vitest-midscene-e2e/
references/
troubleshooting.md
scripts/
clone-boilerplate.sh
SKILL.mdFAQ
midscene-skills is a Claude Code plugin with 6 hand-picked skills for automation work, indexed on Flowy. Install it with the command on its page. It includes android-automation, browser, computer-automation. Its skills do not fire on their own yet. Request auto-invocation to have Flowy route them as you prompt. Free and open source.
- What is midscene-skills?
- Vision-driven cross-platform automation Natural-language driven UI control Built on Midscene.js's vision-based automation capabilities — Operates entirely from screenshots, making cross-platform support reliable This repository contains Skills for the
- How do I install midscene-skills?
- Run these in Claude Code: npx -y skills add web-infra-dev/midscene-skills --agent claude-code. Then prompt normally.
- Does midscene-skills auto-invoke its skills?
- Not yet. It is indexed on Flowy as a plain plugin. Request auto-invocation on its page and Flowy will route its skills for you as you prompt.
- Is midscene-skills free?
- Yes. Flowy is free and open source, with nothing gated. You can read every skill in full before you install.