/crawlberg-headless-fallback

description: Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
name: crawlberg-headless-fallback

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:96b8fc624830e81d41d770e07bed8f15671042d9d84c5a4cee43604f06b23316 Source-Hash: blake3:c646c26d88f919b6848be2422454cabae94e9cbdfefeef3d4beecb56021295a0 Schema-Version: v1 -->

Headless fallback

Some pages are unscrapable without a real browser — SPA shells, infinite scroll, Cloudflare interstitials, JS-rendered article bodies. Crawlberg ships with an optional headless-Chrome backend driven by chromiumoxide.

Modes

--browser-mode auto    # default — try static first, fall back to browser on JS/WAF
--browser-mode always  # skip static, go straight to browser
--browser-mode never   # static only, fail closed

`auto` (default)

The engine fetches statically, then inspects the response. It launches headless Chrome and re-fetches when it sees:

• WAF responses from one of 8 detected vendor fingerprints (Cloudflare,

Akamai, AWS WAF, Imperva, DataDome, PerimeterX, F5, plus a generic catch-all).

• SPA shells: `<noscript>` warnings, near-empty `<body>` with heavy JS.
• Heuristic JS-render-required signals.

This is the right default. The browser only spins up when needed.

`always`

Skip the static probe entirely. Use when:

• The user already told you the page needs JS.
• You are scraping a site you know is React/Vue/Svelte SPA.
• You need `<script>`-emitted state that never lands in static HTML.

crawlberg scrape https://spa.example.com --browser-mode always --format markdown

`never`

Static only — the browser path is disabled. Use when:

• You are in a hot loop where a stray Chrome launch would blow the budget.
• You are running in a sandbox without a Chrome binary.
• The user explicitly wants only static fetches.

In `never` mode, JS-only pages return empty/stub content. Inspect `markdown.content` and `markdown.warnings` before treating the result as final.

Symptoms that point to headless

In `--browser-mode never` or when you suspect the auto detector missed a signal:

• `markdown.content` is short, nav-only, or just a loading message.
• `status_code` is 200 but `metadata.headings` is empty on a page that

clearly has headings.

• `markdown.warnings` mentions JS-render-required or WAF detection.
• 403/406/503 with WAF response headers (`server: cloudflare`,

`cf-mitigated`, `x-amz-cf-id`, `set-cookie: __cf_bm=…`).

Re-run with `--browser-mode always`. If that succeeds, leave it set for that host.

External CDP endpoint

Point at an already-running Chrome (Browserless, Steel, your own) instead of launching locally:

crawlberg scrape https://example.com \
  --browser-mode always \
  --browser-endpoint ws://browser.internal:9222/devtools/browser/<id> \
  --format markdown

The endpoint must be a WebSocket URL — `ws://` or `wss://`. The CLI rejects anything else with a clear error.

Use external CDP when:

• You are running in containers or CI without a local Chrome.
• You want a shared, warm browser pool across many crawl jobs.
• You need browser-side residential proxies or stealth configuration the

local Chrome cannot provide.

Performance cost

Headless Chrome is expensive relative to a static fetch:

• Cold start: 1-3 seconds the first time it launches.
• Per-page overhead: 500 ms-2 s for `NetworkIdle` wait, plus the page's own

JS load time.

• Memory: each tab takes 100-300 MB; long crawls should bound

`--concurrent`.

Mitigations:

• Stay in `--browser-mode auto` — the engine only pays the cost when it

needs to.

• Use `--browser-endpoint` to share one warm browser across jobs.
• Drop `--concurrent` when you know the crawl will route through Chrome.

Wait strategies

Pass via `--config` JSON when you need control:

crawlberg scrape https://example.com --browser-mode always \
  --config '{"browser":{"wait":"selector","wait_selector":".article-body"}}'

Supported strategies (the `browser.wait` field is a string enum; pair `"selector"` with a sibling `wait_selector`):

• `network_idle` (default) — wait until the network goes quiet.
• `selector` — wait until the CSS selector in `wait_selector` resolves.
• `fixed` — wait a fixed duration.

`extra_wait` adds milliseconds on top of the wait strategy if the page keeps loading content after the primary signal.

Persistent profiles

crawlberg scrape https://app.example.com --browser-mode always \
  --config '{"browser_profile":"prod","save_browser_profile":true}'

Profile names are path-traversal-validated. Use them to keep cookies, localStorage, and login state across runs without re-authenticating.