/crawlberg

description: Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out.
license: MIT
metadata:
    author: xberg-io
    version: "0.1.0"
    repository: https://github.com/xberg-io/crawlberg
name: crawlberg

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:027c34d8bc97a85fb89abe9848d849385aa47e7965bf1a770dad8431e89022ec Source-Hash: blake3:c646c26d88f919b6848be2422454cabae94e9cbdfefeef3d4beecb56021295a0 Schema-Version: v1 -->

Crawlberg

Crawlberg is a Rust-native web crawler and scraper. It fetches static HTML with `reqwest`, falls back to headless Chrome when a page needs JS or trips a WAF, and converts every result to clean Markdown via the built-in HTML→Markdown engine.

Use this skill when the user wants to:

• Scrape a single URL to Markdown plus structured metadata.
• Crawl a site following links bounded by depth, page count, and concurrency.
• Enumerate URLs from sitemaps without paying for rendering.
• Drive a real browser (click, type, scroll) and capture the resulting DOM.
• Run the same operations from another agent harness via MCP tools.

Installation

The plugin shells out to a `crawlberg` binary on `PATH`. Install one of:

brew install xberg-io/tap/crawlberg
# or build from source:
cargo install crawlberg-cli --features all

The `serve` and `mcp` subcommands are gated behind non-default cargo features (`api` and `mcp`). The Homebrew tap is built with all features, so both subcommands work out of the box. A from-source build must pass `--features mcp` (and `--features api` for `serve`), or `--features all`, to include them.

Verify:

crawlberg --version

Headless fallback needs Chrome/Chromium reachable locally (`chromiumoxide` launches it on demand). Skip the install if you only plan to use `--browser-mode never`.

Command map

crawlberg scrape <url>          # single page → JSON or Markdown
crawlberg crawl <url...>        # follow links, BFS, depth-bounded
crawlberg map <url>             # enumerate URLs via sitemaps + link extraction
crawlberg interact <url>        # browser actions: click, type, scroll
crawlberg batch-scrape <url...> # scrape many URLs concurrently
crawlberg batch-crawl <url...>  # crawl many seed URLs concurrently
crawlberg download <url>        # download a document, report file metadata
crawlberg citations <input>     # markdown links → numbered citations (text or @file.md)
crawlberg version               # print the crawlberg version as JSON
crawlberg mcp                   # MCP server (stdio) — auto-registered (`mcp` feature)
crawlberg serve                 # REST API server (`api` feature)

`crawl` also handles batching implicitly: pass multiple seed URLs and it fans out via `batch_crawl` internally. The explicit `batch-scrape` and `batch-crawl` subcommands expose the same concurrency for many independent URLs.

Per-subcommand flags:

| Subcommand | Positional | Key flags | | -------------- | ----------------- | ------------------------------------------------------------------------------------------------------------- | | `scrape` | `<url>` | `--proxy`, `--user-agent` (plus shared flags below) | | `crawl` | `<url...>` | `--depth`/`-d` (2), `--max-pages`/`-n`, `--concurrent`/`-c` (10), `--rate-limit` (200), `--stay-on-domain`, `--proxy`, `--user-agent` | | `map` | `<url>` | `--limit`, `--search` | | `interact` | `<url>` | `--actions <json>` (required) | | `batch-scrape` | `<url...>` | `--concurrent`/`-c` (10), `--proxy`, `--user-agent` | | `batch-crawl` | `<url...>` | `--depth`/`-d` (2), `--max-pages`/`-n`, `--concurrent`/`-c` (10), `--rate-limit` (200), `--stay-on-domain`, `--proxy`, `--user-agent` | | `download` | `<url>` | `--max-size` | | `citations` | `<input>` | none (input is markdown text or `@file.md`) | | `version` | — | none | | `serve` | — | `--host` (0.0.0.0), `--port` (3000) | | `mcp` | — | none (stdio transport) |

Shared flags

| Flag | Default | Notes | | ----------------------- | -------- | -------------------------------------------------- | | `--format` | `json` | `json` or `markdown`. | | `--timeout` | `30000` | Request timeout in milliseconds. | | `--browser-mode` | `auto` | `auto`, `always`, or `never`. | | `--browser-endpoint` | — | Optional CDP `ws://` or `wss://` URL. | | `--respect-robots-txt` | off | Pass to obey `robots.txt`. | | `--config <json>`