/crawlberg-mapping-urls

description: Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers `crawlberg map URL` with `--limit`, `--search`, robots, output, and how it differs from a full crawl.
name: crawlberg-mapping-urls

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:26e1d5c5cc2fa54b3f29ebabb1a466df0163c29ca984cb489d8d738e610bd584 Source-Hash: blake3:c646c26d88f919b6848be2422454cabae94e9cbdfefeef3d4beecb56021295a0 Schema-Version: v1 -->

Mapping URLs

`crawlberg map <url>` discovers the URLs a site exposes without rendering or extracting any page content. It reads `sitemap.xml` (including nested sitemaps), then falls back to link extraction from the seed page. Use it to plan a crawl, audit a site's surface, or feed a URL list into another tool.

Quick recipe

crawlberg map https://example.com --limit 500 --search docs --format markdown

Markdown output prints one URL per line — convenient to pipe into a file or a follow-up `crawl`. JSON output (default) returns a structured `MapResult`.

Flag surface

| Flag | Default | Purpose | | ---------------------- | ------- | ------------------------------------------------------------- | | `--limit` | — | Maximum number of URLs to return. Unbounded if unset. | | `--search` | — | Case-insensitive substring filter on discovered URLs. | | `--respect-robots-txt` | off | Honour `robots.txt`. Pass it for any third-party host. | | `--format` | `json` | `json` (full `MapResult`) or `markdown` (one URL per line). | | `--timeout` | `30000` | Per-request timeout in ms. | | `--browser-mode` | `auto` | `auto`, `always`, `never` — see the headless-fallback skill. | | `--browser-endpoint` | — | External CDP `ws://` URL. | | `--config` | — | Inline JSON or `@file.json` for the full `CrawlConfig`. |

`map` takes a single seed URL positionally. There is no `--depth` or `--max-pages` here — those bound a `crawl`, not a map. Scope is the seed host's sitemaps plus links found on the seed page; bound the result with `--limit` and narrow it with `--search`.

How discovery works

1. Fetch and parse `sitemap.xml`, following nested `<sitemapindex>` entries. 2. If no sitemap (or a thin one), extract links from the seed page's HTML. 3. Apply the `--search` substring filter (case-insensitive), then `--limit`.

No page bodies are rendered, so a map of hundreds of URLs returns in seconds — far cheaper than crawling. In `--browser-mode auto` the seed fetch still falls back to headless Chrome if the seed page is a JS shell that hides its links; pass `--browser-mode never` to keep it static-only.

Output

Markdown mode

https://example.com/
https://example.com/docs/
https://example.com/docs/getting-started
https://example.com/blog/post-one

JSON mode

Top-level `MapResult` with a `urls` array; each entry carries the discovered `url`. Read `result.urls[i].url` for each string when scripting.

crawlberg map https://example.com --format json | jq -r '.urls[].url'

Common patterns

Discover then crawl a subsection

crawlberg map https://example.com --search /docs/ --format markdown > urls.txt

Feed the filtered list into a bounded crawl, or scrape individual entries.

Audit a third-party site politely

crawlberg map https://unknown.example --respect-robots-txt --limit 200

When to reach for crawl instead

If the user needs the page *content* (Markdown, metadata, tables) rather than just the URL list, use `crawlberg crawl` — see the `crawlberg-crawling-a-site` skill. Reach for `map` first when the goal is enumeration, planning, or seeding.