Testing
Plugin
wildclawbench
An in-the-wild benchmark for AI agents in the production harness.
Plugin Signal72/100Solid
maintained80
capable69
documented75
adopted58
Install
$ npx -y skills add internlm/wildclawbench --agent claude-code
Stats
532
Stars
65
Forks
Maintained
Maintenance
Python
Language
MIT
License
1mo ago
Last commit
7mo ago
Created
5h ago
Added
In this plugin
- 03_task1
- 03_task2
- 03_task3
- 03_task4
- 03_task5
- 03_task6
- agentic-paper-digest-skill
- arxiv-summarizer-orchestrator
- calendar-reminders
- architecture
- arxiv-agentic-verifier
- geepers-data
- agenticmail
- ai-meeting-scheduling
- meeting-to-action
- academic-literature-search
- ddgs-search
- openclaw-free-web-search
- bilibili-youtube-watcher
- eachlabs-voice-audio
- loopwind
- aegis-shield
- canary
- skill-trust-auditor
- agent-browser
- self-improving-agent-3.0.5
- video-frames
What's inside
Skill(27 skills)
/03_task1Coordinates meeting scheduling via Gmail and Calendar APIs. Use when: scheduling a meeting, checking participant availability, sending coordination emails, creating calendar events, or notifying organizers of confirmed bookings./03_task2Extracts action items and pending tasks from Slack messages. Use when: reviewing unread messages for todos, finding deadlines or requests directed at the user, or producing a task summary from recent Slack activity./03_task3Analyzes Slack messages to assess deal or project feasibility. Use when: synthesizing fragmented stakeholder input on a deal, identifying conflicting requirements or risks, or producing a feasibility report for leadership.
- /03_task4
- /03_task5
- /03_task6
- /agentic-paper-digest-skill
- /arxiv-summarizer-orchestrator
- /calendar-reminders
- /architecture
- /arxiv-agentic-verifier
- /geepers-data
- /agenticmail
- /ai-meeting-scheduling
- /meeting-to-action
- /academic-literature-search
- /ddgs-search
- /openclaw-free-web-search
- /bilibili-youtube-watcher
- /eachlabs-voice-audio
- /loopwind
- /aegis-shield
- /canary
- /skill-trust-auditor
- /agent-browser
- /self-improving-agent-3.0.5
- /video-frames
README
Hard, practical, end-to-end evaluation for AI agents — in the wild.
WildClawBench is an agent benchmark that tests what actually matters: can an AI agent do real work, end-to-end, without hand-holding?
We drop agents into a live OpenClaw environment — the same open-source personal AI assistant that real users rely on daily — and throw 60 original tasks at them: clipping goal highlights from a football match, negotiating meeting times over multi-round emails, hunting down contradictions in search results, writing inference scripts for undocumented codebases, catching privacy leaks before they happen. Useful things. Hard things.
In the technical report snapshot, the strongest frontier model topped out at 62.2% overall; the latest audited OpenClaw runs have since raised the best score to 67.2%. Most models still land well below that. That makes scores mean something.
Why WildClawBench?
Most agent benchmarks test isolated capabilities — calling a function, parsing JSON, following a single instruction. WildClawBench tests the full picture:
| What We Test | Why It's Hard | |
|---|---|---|
| 🔗 Agency | Multi-step tool orchestration, error recovery, autonomous planning | Agents must chain 10–60+ tool calls, adapt when services fail, and decide what to do — not just how |
| 🎥 Multimodal | Video understanding, image generation, cross-modal synthesis | Track events across a 45-min match video and clip precise highlights; classify 12 clothing photos, assemble 4 styled outfits, and generate full-body model images for each |
| 🧵 Long-Horizon | Complex workflows spanning 10–20 minutes of wall-clock execution | Negotiate meeting times over multiple email rounds; crawl, classify, and summarize 50+ academic papers |
| 💻 Coding | Read undocumented codebases, debug, generate working programs | Read an undocumented codebase, install dependencies, and write working inference from source alone; solve visual puzzles by generating pixel-accurate solutions |
| 🛡️ Safety | Prompt injection defense, credential leak detection, harmful content refusal | Harmful instructions are buried deep inside normal-looking documents; API keys are scattered across a large git history |
What Sets Us Apart
- Real environment, not mocks. Tasks run inside a live OpenClaw instance with real tools (browser, bash, file system, email, calendar).
- 60 original tasks, built by hand. Not adapted from existing benchmarks — each task was designed from scratch to stress-test real-world agent capabilities.
- Four agent harnesses, one task suite. OpenClaw, Claude Code, Codex CLI, and Hermes Agent all execute the same 60 tasks under the same grading. This separates model capability from harness scaffolding — you can see how much an agent's score depends on its surrounding tools versus the underlying LLM.
- Reproducible & isolated. Each task runs in its own Docker container. Same image, same data, same grading code. Ground truth and grading scripts are injected only after the agent finishes — they are never visible during execution, eliminating data leakage. Scores are reproducible across machines.
The WildClawBench Family
The evaluation code lives in this repository; the benchmark data ships as three Hugging Face datasets — pick the one that matches how you want to use it:
| Repository | What's inside | Use it when you want to |
|---|---|---|
| WildClawBench | Task data, Docker images for all four harnesses | Reproduce the paper's evaluation with this repository's pipeline |
| WildClawBench-Harbor | All 60 tasks repackaged in the Harbor format | Evaluate any Harbor-supported agent with a single harbor run — no benchmark-specific setup (details) |
| WildClawBench-Trajectories | Complete agent trajectories for the full 60-task suite, across a growing roster of frontier models, plus raw evaluation outputs | Inspect how models actually behave — or mine real long-horizon traces for analysis and training (details) |
News
- 2026-09 Qwen's Qwen3.8-Omni-Flash release includes WildClawBench's multimodal subset (WildClawBench-MM) in its agent evaluations. Thanks for the recognition!
- 2026-08 Meta's Muse Glimmer release reports WildClawBench evaluation scores. Thanks for the recognition!
- 2026-08 Released WildClawBench-Harbor — the full 60-task suite in Harbor format — and WildClawBench-Trajectories — complete agent trajectories from our frontier-model evaluations, browsable in the HF Agent Trace Viewer and continuously updated as new models are evaluated.
- 2026-07 We expanded the OpenClaw leaderboard with evaluations of the latest frontier models, including GPT-5.6 Sol, Claude Fable 5, Kimi K3 and etc.
- 2026-06 ByteDance Seed's Seed2.1 release includes WildClawBench in its agent evaluations. Thanks for the recognition!
- 2026-05 We released a new version with four agent harnesses — OpenClaw, Claude Code, Codex CLI, and Hermes Agent — so the same 60-task suite can be evaluated under multiple scaffolds.
- 2026-05 We published a technical report PDF.
- 2026-05 Tencent’s Hunyuan3 Preview page reports WildClawBench evaluation scores. Thanks for the recognition!
Leaderboard
WildClawBench reports two complementary leaderboards:
- Model leaderboard (OpenClaw harness) — apples-to-apples comparison of LLMs running inside the same OpenClaw harness.
- Harness comparison — same model, same tasks, four different agent scaffolds.
Full interactive leaderboard at internlm.github.io/WildClawBench.
Model leaderboard (OpenClaw harness)
Overall score follows the weighted Multimodal / Pure-text breakdown in that table. Total time and total cost are the paper’s Overall per-task averages (minutes / USD) multiplied by 60 for the full 60-task suite.
| Rank | Model | Org | Overall Score | Total Time | Total Cost |
|---|---|---|---|---|---|
| 🥇 | GPT-5.6 Sol | OpenAI | 67.2% | 222 min | $56.78 |
| 🥈 | Claude Opus 4.8 | Anthropic | 64.7% | 400 min | $95.95 |
| 🥉 | Claude Opus 4.7 | Anthropic | 62.2% | 328 min | $77.40 |
| 4 | Claude Fable 5 | Anthropic | 62.0% | 324 min | $87.71 |
| 5 | GPT-5.5 | OpenAI | 58.2% | 262 min | $37.80 |
| 6 | Grok 4.5 | xAI | 57.5% | 359 min | $28.43 |
| 7 | Qwen3.8-Max | Alibaba Cloud | 56.2% | 708 min | $24.70 |
| 8 | Muse Spark 1.1 | Meta | 54.8% | 367 min | $23.59 |
| 9 | Kimi K3 | Moonshot AI | 54.5% | 488 min | $40.08 |
| 10 | GLM 5.2 | Zhipu AI | 54.2% | 442 min | $17.10 |
| 11 | Claude Opus 4.6 | Anthropic | 51.6% | 508 min | $81.00 |
| 12 | GPT-5.4 | OpenAI | 50.3% | 350 min | $19.80 |
| 13 | Hy3 | Tencent | 49.7% | 338 min | $2.13 |
| 14 | GLM 5.1 | Zhipu AI | 48.2% | 515 min | $34.80 |
| 15 | Qwen3.8 27B | Alibaba Cloud | 48.0% | 516 min | N/A |
| 16 | Muse Glimmer 30B | Meta | 47.6% | 352 min | $6.06 |
| 17 | Kimi K2.7 Code | Moonshot AI | 46.9% | 674 min | $72.31 |
| 18 | Intern-S2 Preview 397B | InternLM | 44.7% | 541 min | Free |
| 19 | DeepSeek V4 Pro | DeepSeek | 43.7% | 605 min | $12.00 |
| 20 | Qwen3.6 27B | Alibaba Cloud | 43.2% | 421 min | $20.91 |
| 21 | MiMo V2.5 Pro | Xiaomi | 43.0% | 451 min | $12.60 |
| 22 | GLM 5 | Zhipu AI | 42.6% | 373 min | $11.40 |
| 23 | Gemini 3.1 Pro | Google DeepMind | 40.8% | 240 min | $18.00 |
| 24 | MiMo V2 Pro | Xiaomi | 40.2% | 458 min | $26.40 |
| 25 | Gemma 4 31B IT | Google DeepMind | 37.6% | 384 min | $3.46 |
| 26 | Qwen3.5 397B | Alibaba Cloud | 34.5% | 459 min | $22.20 |
| 27 | DeepSeek V3.2 | DeepSeek | 34.0% | 549 min | $11.40 |
| 28 | GLM 5 Turbo | Zhipu AI | 33.9% | 499 min | $15.00 |
| 29 | MiniMax M2.7 | MiniMax | 33.8% | 551 min | $7.20 |
| 30 | Kimi K2.5 | Moonshot AI | 30.8% | 406 min | $6.60 |
| 31 | MiMo V2 Flash | Xiaomi | 30.8% | 433 min | $10.20 |
| 32 | MiniMax M2.5 | MiniMax | 27.1% | 542 min | $9.60 |
| 33 | Step 3.5 Flash | StepFun | 26.7% | 430 min | $6.60 |
| 34 | Grok 4.20 Beta | xAI | 19.3% | 94 min | $9.60 |
Claude Opus 4.8 cost uses the dynamic base-tier rates for this evaluation: $5/M input, $25/M output, $0.5/M cache read, and $6.25/M cache write. Muse Glimmer 30B cost uses the OpenRouter rates for this evaluation: $0.35/M input, $1.50/M output, and $0.04/M cache read. Kimi K2.7 Code cost uses the published rates for this evaluation: $6.5/M input, $27/M output, and $1.3/M cached input. Qwen3.8-Max cost uses the evaluation rates: $1.7535/M input, $5.2605/M output, and $0.219186/M cache read. Qwen3.8 27B was evaluated on a self-hosted vLLM endpoint. Token-level billing data was unavailable, so total cost is reported as N/A.
Harness comparison
Same 60 tasks, same grading, four different agent scaffolds. Time and cost are per-task averages; score is in %. Time is in minutes per task, cost in USD per task. Bold = best harness for that model.
| Model | OpenClaw | Claude Code | Codex | Hermes Agent | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Time | Cost | Score | Time | Cost | Score | Time | Cost | Score | Time | Cost | Score | |
| GPT-5.4 | 5.83 | $0.33 | 50.3 | 9.07 | $0.61 | 48.4 | 7.16 | $0.57 | 56.8 | 8.97 | $0.44 | 50.7 |
| GLM 5 | 6.22 | $0.19 | 42.6 | 10.18 | $0.21 | 31.0 | 7.84 | $0.13 | 38.9 | 6.62 | $0.44 | 46.4 |
| MiMo V2 Pro | 7.63 | $0.44 | 40.2 | 9.90 | $0.15 | 29.9 | 6.44 | $0.15 | 35.3 | 8.30 | $0.26 | 48.1 |
| MiniMax M2.7 | 9.18 | $0.12 | 33.8 | 10.08 | $0.09 | 32.0 | 8.66 | $0.06 | 35.8 | 10.30 | $0.11 | 37.1 |
Tasks
60 tasks across 6 categories, spanning English and Chinese:
| Category | # | Example Tasks | Core Challenges |
|---|
File tree
FAQ
wildclawbench is a Claude Code plugin with 27 hand-picked skills, indexed on Flowy in the Testing category. Install it with the command on its page. It includes 03_task1, 03_task2, 03_task3. Its skills do not fire on their own yet. Request auto-invocation to have Flowy route them as you prompt. Free and open source.
- What is wildclawbench?
- An in-the-wild benchmark for AI agents in the production harness.
- How do I install wildclawbench?
- Run these in Claude Code: npx -y skills add internlm/wildclawbench --agent claude-code. Then prompt normally.
- Does wildclawbench auto-invoke its skills?
- Not yet. It is indexed on Flowy as a plain plugin. Request auto-invocation on its page and Flowy will route its skills for you as you prompt.
- Is wildclawbench free?
- Yes. Flowy is free and open source, with nothing gated. You can read every skill in full before you install.
Is this plugin yours?
Claim it with GitHubSubmit a pluginPromote it