/crawl4ai-skill
Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for
$ npx -y skills add brettdavies/crawl4ai-skill --skill crawl4ai-skill --agent claude-codeHow it fires
How this skill gets triggered: by you, by Claude, or both.
- Fires itselfAuto-invocation. Claude auto-loads it when your prompt matches the work.Auto-invocation is when the right skill fires by itself at the right moment, driven by a FLOW.md router and a hook, instead of you invoking it by name. It is the difference between a skill being installed and a skill actually getting used.Read the full definition →
- You can call itInvoke it directly when you want it.
- Slash command
/crawl4ai-skill
Context preview
The summary Claude sees to decide when to auto-load this skill.
Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for
SKILL.md
crawl4ai-skill.SKILL.mdname: crawl4ai
description: Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.
argument-hint: "[url]"
Crawl4AI
**Verified against `crawl4ai`** [`VERSION`](VERSION). PEP 723 pins in `scripts/*.py` and `tests/*.py` floor at that version.
Overview
Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable schema-based extraction.
This skill exposes both interfaces of the underlying library:
- **CLI** (`crwl`) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)
- **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)
Invoked with a URL argument
When the user runs `/crawl4ai <url>` with a single URL and no further qualifier, treat it as the JS-heavy fetch case and default to:
crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown
`wait_until=networkidle` waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't named a specific element on a JS-rendered page. (Avoid `wait_for=css:body`: `<body>` exists at t=0 on every HTML response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to `wait_for=css:<selector>` if the user named a specific element. Skip the default and route to the relevant section below for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to `/fetch-web` instead per the "When NOT to use" section below.
When NOT to use this skill
- **Static HTML pages** (most documentation sites, blog posts, news articles, tweets) — use `/fetch-web` or `defuddle`
directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.
- **Local file conversion** (`.pdf`, `.docx`, `.pptx`, `.epub`) — use `/markdown-convert`.
- **One-URL agent-context reads** (the agent just needs to read this page) — use `/fetch-web` and let it route to
`defuddle`.
- **Mutating UI flows** (form fills, multi-step clicks, login + navigation) — `/browse` (gstack's persistent headless
Chromium) is built for that.
When stuck
For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see [references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues → ask the user) and worked examples.
---
Quick Start
Installation
pip install crawl4ai
crawl4ai-setup
# Verify installation
crawl4ai-doctor
CLI (Recommended)
# Basic crawling - returns markdown
crwl https://example.com
# Get markdown output
crwl https://example.com -o markdown
# JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache
# See more examples
crwl --example
Python SDK
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com")
print(result.markdown[:500])
asyncio.run(main())For SDK configuration details: [SDK Guide - Configuration](references/sdk-guide.md#configuration).
---
Core Concepts
Configuration Layers
Both CLI and SDK use the same underlying configuration:
| Concept | CLI | SDK | | ---------------- | -------------------------------------- | ------------------------- | | Browser settings | `-B browser.yml` or `-b "param=value"` | `BrowserConfig(...)` | | Crawl settings | `-C crawler.yml` or `-c "param=value"` | `CrawlerRunConfig(...)` | | Extraction | `-e extract.yml -s schema.json` | `extraction_strategy=...` | | Content filter | `-f filter.yml` | `markdown_generator=...` |
Key Parameters
**Browser Configuration:**
- `headless`: Run with/without GUI
- `viewport_width/height`: Browser dimensions
- `user_agent`: Custom user agent
- `proxy_config`: Proxy settings
**Crawler Configuration:**
- `page_timeout`: Max page load time (ms)
- `wait_for`: CSS selector or JS condition to wait for
- `cache_mode`: bypass, enabled, disabled
- `js_code`: JavaScript to execute
- `css_selector`: Focus on specific element
For complete parameters: [CLI Config](references/cli-guide.md#configuration) | [SDK Config](references/sdk-guide.md#configuration)
Output Content
Every crawl returns:
- **markdown** - Clean, formatted markdown
- **html** - Raw HTML
- **links** - Internal and external links discovered
- **media** - Images, videos, audio found
- **extracted_content** - Structured data (if extraction configured)
---
Markdown Generation (Primary Use Case)
Crawl4AI excels at generating clean, well-formatted markdown.
CLI
crwl https://docs.example.com -o markdown # raw markdown
crwl https://docs.example.com -o markdown-fit # filtered (noise removed)
crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit # BM25-relevance filter
crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter
Filter templates: [`templates/filter_b
Read more
name: crawl4ai description: Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser. argument-hint: "[url]"
Crawl4AI
**Verified against `crawl4ai`** [`VERSION`](VERSION). PEP 723 pins in `scripts/*.py` and `tests/*.py` floor at that version.
Overview
Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable schema-based extraction.
This skill exposes both interfaces of the underlying library:
- **CLI** (`crwl`) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)
- **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)
Invoked with a URL argument
When the user runs `/crawl4ai <url>` with a single URL and no further qualifier, treat it as the JS-heavy fetch case and default to:
crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown
`wait_until=networkidle` waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't named a specific element on a JS-rendered page. (Avoid `wait_for=css:body`: `<body>` exists at t=0 on every HTML response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to `wait_for=css:<selector>` if the user named a specific element. Skip the default and route to the relevant section below for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to `/fetch-web` instead per the "When NOT to use" section below.
When NOT to use this skill
- **Static HTML pages** (most documentation sites, blog posts, news articles, tweets) — use `/fetch-web` or `defuddle`
directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.
- **Local file conversion** (`.pdf`, `.docx`, `.pptx`, `.epub`) — use `/markdown-convert`.
- **One-URL agent-context reads** (the agent just needs to read this page) — use `/fetch-web` and let it route to
`defuddle`.
- **Mutating UI flows** (form fills, multi-step clicks, login + navigation) — `/browse` (gstack's persistent headless
Chromium) is built for that.
When stuck
For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see [references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues → ask the user) and worked examples.
---
Quick Start
Installation
pip install crawl4ai crawl4ai-setup # Verify installation crawl4ai-doctor
CLI (Recommended)
# Basic crawling - returns markdown crwl https://example.com # Get markdown output crwl https://example.com -o markdown # JSON output with cache bypass crwl https://example.com -o json -v --bypass-cache # See more examples crwl --example
Python SDK
import asyncio
from crawl4ai import AsyncWebCrawler
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun("https://example.com")
print(result.markdown[:500])
asyncio.run(main())For SDK configuration details: [SDK Guide - Configuration](references/sdk-guide.md#configuration).
---
Core Concepts
Configuration Layers
Both CLI and SDK use the same underlying configuration:
| Concept | CLI | SDK | | ---------------- | -------------------------------------- | ------------------------- | | Browser settings | `-B browser.yml` or `-b "param=value"` | `BrowserConfig(...)` | | Crawl settings | `-C crawler.yml` or `-c "param=value"` | `CrawlerRunConfig(...)` | | Extraction | `-e extract.yml -s schema.json` | `extraction_strategy=...` | | Content filter | `-f filter.yml` | `markdown_generator=...` |
Key Parameters
**Browser Configuration:**
- `headless`: Run with/without GUI
- `viewport_width/height`: Browser dimensions
- `user_agent`: Custom user agent
- `proxy_config`: Proxy settings
**Crawler Configuration:**
- `page_timeout`: Max page load time (ms)
- `wait_for`: CSS selector or JS condition to wait for
- `cache_mode`: bypass, enabled, disabled
- `js_code`: JavaScript to execute
- `css_selector`: Focus on specific element
For complete parameters: [CLI Config](references/cli-guide.md#configuration) | [SDK Config](references/sdk-guide.md#configuration)
Output Content
Every crawl returns:
- **markdown** - Clean, formatted markdown
- **html** - Raw HTML
- **links** - Internal and external links discovered
- **media** - Images, videos, audio found
- **extracted_content** - Structured data (if extraction configured)
---
Markdown Generation (Primary Use Case)
Crawl4AI excels at generating clean, well-formatted markdown.
CLI
crwl https://docs.example.com -o markdown # raw markdown crwl https://docs.example.com -o markdown-fit # filtered (noise removed) crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit # BM25-relevance filter crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter
Filter templates: [`templates/filter_b
Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas.
Repo: brettdavies/crawl4ai-skill