/crawl4ai-skill

name: crawl4ai
description: Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.
argument-hint: "[url]"

Crawl4AI

**Verified against `crawl4ai`** [`VERSION`](VERSION). PEP 723 pins in `scripts/*.py` and `tests/*.py` floor at that version.

Overview

Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable schema-based extraction.

This skill exposes both interfaces of the underlying library:

• **CLI** (`crwl`) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)
• **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)

Invoked with a URL argument

When the user runs `/crawl4ai <url>` with a single URL and no further qualifier, treat it as the JS-heavy fetch case and default to:

crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown

`wait_until=networkidle` waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't named a specific element on a JS-rendered page. (Avoid `wait_for=css:body`: `<body>` exists at t=0 on every HTML response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to `wait_for=css:<selector>` if the user named a specific element. Skip the default and route to the relevant section below for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to `/fetch-web` instead per the "When NOT to use" section below.

When NOT to use this skill

• **Static HTML pages** (most documentation sites, blog posts, news articles, tweets) — use `/fetch-web` or `defuddle`

directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.

• **Local file conversion** (`.pdf`, `.docx`, `.pptx`, `.epub`) — use `/markdown-convert`.
• **One-URL agent-context reads** (the agent just needs to read this page) — use `/fetch-web` and let it route to

`defuddle`.

• **Mutating UI flows** (form fills, multi-step clicks, login + navigation) — `/browse` (gstack's persistent headless

Chromium) is built for that.

When stuck

For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see [references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues → ask the user) and worked examples.

---

Quick Start

Installation

pip install crawl4ai
crawl4ai-setup

# Verify installation
crawl4ai-doctor

CLI (Recommended)

# Basic crawling - returns markdown
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache

# See more examples
crwl --example

Python SDK

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:500])

asyncio.run(main())

For SDK configuration details: [SDK Guide - Configuration](references/sdk-guide.md#configuration).

---

Core Concepts

Configuration Layers

Both CLI and SDK use the same underlying configuration:

| Concept | CLI | SDK | | ---------------- | -------------------------------------- | ------------------------- | | Browser settings | `-B browser.yml` or `-b "param=value"` | `BrowserConfig(...)` | | Crawl settings | `-C crawler.yml` or `-c "param=value"` | `CrawlerRunConfig(...)` | | Extraction | `-e extract.yml -s schema.json` | `extraction_strategy=...` | | Content filter | `-f filter.yml` | `markdown_generator=...` |

Key Parameters

**Browser Configuration:**

• `headless`: Run with/without GUI
• `viewport_width/height`: Browser dimensions
• `user_agent`: Custom user agent
• `proxy_config`: Proxy settings

**Crawler Configuration:**

• `page_timeout`: Max page load time (ms)
• `wait_for`: CSS selector or JS condition to wait for
• `cache_mode`: bypass, enabled, disabled
• `js_code`: JavaScript to execute
• `css_selector`: Focus on specific element

For complete parameters: [CLI Config](references/cli-guide.md#configuration) | [SDK Config](references/sdk-guide.md#configuration)

Output Content

Every crawl returns:

• **markdown** - Clean, formatted markdown
• **html** - Raw HTML
• **links** - Internal and external links discovered
• **media** - Images, videos, audio found
• **extracted_content** - Structured data (if extraction configured)

---

Markdown Generation (Primary Use Case)

Crawl4AI excels at generating clean, well-formatted markdown.

CLI

crwl https://docs.example.com -o markdown                              # raw markdown
crwl https://docs.example.com -o markdown-fit                          # filtered (noise removed)
crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit   # BM25-relevance filter
crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter

Filter templates: [`templates/filter_b