/scraper-builder

name: scraper-builder
description: "Build production-ready web scrapers for any website using Bright Data infrastructure. Guides you through site analysis, API selection, selector extraction, pagination handling, and complete scraper implementation. Use this skill whenever the user wants to build a scraper, create a crawler, extract data from a website, scrape product pages, handle pagination, build a data pipeline from a web source, or automate data collection from any site — even if they don't explicitly say 'scraper'. Triggers on phrases like 'build a scraper for', 'scrape data from', 'extract products from', 'crawl pages on', 'get data from [website]', or 'I need to pull data from'."

Scraper Builder

You are building a production-ready web scraper for the user. Your job is to guide them from "I want data from site X" to a working, robust scraper that handles real-world challenges like pagination, dynamic content, anti-bot protection, and data parsing.

Critical: Always Validate Your Output

After building the scraper, **always run it** on a small sample (1-3 pages) and show the extracted data to the user before scaling up. If the output is empty, malformed, or missing fields, iterate — fix selectors, switch APIs, or adjust the parsing logic. A scraper that doesn't produce clean data is not done.

Take your time with the reconnaissance phase. Spending 2 minutes analyzing the HTML upfront prevents hours of debugging later. Quality is more important than speed here.

How This Skill Works

This skill orchestrates Bright Data's four APIs to build scrapers intelligently. Rather than writing fragile custom scraping code, you analyze the target site first, then pick the most reliable and cost-effective extraction method. The decision tree is:

1. **Does a pre-built scraper already exist?** → Use Web Scraper API (zero parsing code needed) 2. **Is the page static / no interaction needed?** → Use Web Unlocker API (cheapest, simplest) 3. **Does the page need clicks, scrolls, or JS interaction?** → Use Browser API (full automation) 4. **Need search engine results?** → Use SERP API

The skill produces complete, runnable code — not pseudocode or outlines.

---

Phase 1: Understand the Target

Before writing any code, you need to understand what the user wants and what the site looks like. Ask these questions (skip any the user already answered):

1. **What site?** — The target URL or domain 2. **What data?** — Which fields they need (product names, prices, reviews, etc.) 3. **What scope?** — Single page, category pages, search results, entire site section? 4. **Pagination?** — Do they need to scrape across multiple pages? 5. **Volume?** — Roughly how many items/pages? (affects sync vs async choice and concurrency strategy — see [references/concurrency-guide.md](references/concurrency-guide.md)) 6. **Output format?** — JSON, CSV, database? (default to JSON if unspecified) 7. **Language preference?** — Python or Node.js? (default to Python if unspecified)

Don't over-interview. If the user says "build a scraper for Amazon product pages", you already know: site=Amazon, data=product details, scope=product pages. Jump ahead.

---

Phase 2: Check for Pre-Built Scrapers

Before doing any custom work, check if Bright Data already has a scraper for this domain. This is the fastest, cheapest, and most reliable path.

Read [references/supported-domains.md](references/supported-domains.md) for the curated list of common pre-built scrapers. But the curated list may not be complete — Bright Data supports 100+ domains and adds new scrapers regularly. If you don't see the target domain in the curated list, **query the live Dataset List API** to check:

curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
     https://api.brightdata.com/datasets/list

This returns every available scraper with its `dataset_id` and name. Search the results for the target domain. You can also browse the full documentation index at `https://docs.brightdata.com/llms.txt` to discover scraper-specific docs and supported parameters.

If a pre-built scraper exists

Use the Web Scraper API or Python SDK platform-specific scrapers. This gives you structured JSON with no parsing code needed.

**Python SDK approach (preferred):**

from brightdata import BrightDataClient

async with BrightDataClient() as client:
    result = await client.scrape.amazon.products(url="https://amazon.com/dp/B0CRMZHDG8")
    if result.success:
        print(result.data)  # Structured product data

**REST API approach (shell/curl):**

bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB"

For bulk scraping with pre-built scrapers, use the async trigger/poll/fetch pattern:

async with BrightDataClient() as client:
    # Trigger without waiting
    job = await client.scrape.amazon.products_trigger(url=url)
    # Poll until ready
    await job.wait(timeout=180, poll_interval=10, verbose=True)
    # Fetch results
    data = await job.fetch()

Skip to Phase 5 (pagination/orchestration) if the user needs multi-page scraping with a pre-built scraper.

If no pre-built scraper exists

Continue to Phase 3 — you need to analyze the site and build a custom scraper.

---

Phase 3: Site Reconnaissance

This is the critical step that separates reliable scrapers from brittle ones. You need to understand the site's structure before writing extraction code.

Step 3a: Fetch the page HTML

Use Web Unlocker to get the raw HTML. This tells you whether the content is server-rendered or client-rendered, and gives you the actual DOM to analyze.

import requests
import os

API_KEY = os.environ["BRIGHTDATA_API_KEY"]
ZONE = os.environ["BRIGHTDATA_UNLOCKER_ZONE"]

response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "zone": ZONE,
        "url": "https://target-site.com/page",
        "format