/scraper-studio

name: scraper-studio
description: "Build and run AI-generated Bright Data scrapers from the terminal via `bdata scraper create` and `bdata scraper run`. Use this skill whenever the user wants to generate a scraper from a natural-language description, build a custom scraper without writing code, turn a URL + plain-English description into a reusable scraper, run an existing Bright Data collector against a URL, or batch-scrape a list of URLs through one collector. Triggers on phrases like 'build me a scraper for', 'create a scraper that extracts', 'generate a scraper from a description', 'turn this URL into a scraper', 'run this scraper on', 'run my collector', 'batch scrape', 'scrape these URLs', 'scrape a list of URLs', 'competitive pricing table', 'scraper studio', `scraper create`, `scraper run`, `--urls`, `--input-file`, `collector_id`, `automate_template`, or `/dca/`. Covers the AI flow (template create → trigger AI generation → poll progress), the single-URL run flow (async + poll by default, `--sync` for fast pages), the multi-URL batch flow (`--urls` / `--input-file` → one `/dca/trigger` call with array body), and the silent auto-fallback to the batch endpoint when a URL expands past the realtime page limit. Requires the Bright Data CLI."

Bright Data — Scraper Studio

Build a scraper from natural language, then run it. Two commands live in this skill:

• **`bdata scraper create <url> <description>`** — describe what you want in plain English; Bright Data's AI Flow generates a scraper template and returns a `collector_id`.
• **`bdata scraper run <collector_id> <url>`** — run that collector (or any existing one from the Bright Data web UI) against a URL and get the extracted data back.

**The bridge between the two is `collector_id`.** It is printed by `create` and consumed by `run`. Always save it.

For pre-built scrapers on platforms like Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc., **stop and use the [`data-feeds`](../data-feeds/SKILL.md) skill instead** — those scrapers already exist, are faster, cheaper, and more reliable than building a new one. Use Scraper Studio when no pre-built scraper covers the target site, or when the user wants a custom shape of output for an existing platform.

Setup gate (run first)

if ! command -v bdata >/dev/null 2>&1; then
    echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
    echo "bdata not authenticated — run: bdata login  (or: bdata login --device for SSH)"
fi

Halt and route to setup if either check fails. Both commands require an authenticated CLI.

Pick your path

| Situation | Action | |---|---| | User describes data they want from a URL, no scraper exists yet | `bdata scraper create <url> "<description>"` → save the `collector_id` | | User has a `collector_id` and wants data from one URL | `bdata scraper run <collector_id> <url>` (default async + poll) | | User has a `collector_id` and wants data from many URLs | `bdata scraper run <collector_id> --urls "u1,u2,..."` or `--input-file urls.txt` (single batch call) | | Page is small and you want fast feedback (≤ ~50 s, single URL) | `bdata scraper run … --sync` | | Scraper ran but returned wrong / empty / partial data | inspect the output, then `bdata scraper heal <collector_id> "<what's wrong>"` → review preview → approve → re-run to verify | | Site is a known platform (Amazon, LinkedIn, TikTok, …) | **stop — use `data-feeds` skill** | | You want SERP / discovery, not extraction | **use `search` skill** | | You want a one-off raw page fetch | **use `scrape` skill** |

---

Action 1 — `scraper create`

Generate a scraper from a URL + plain-English description.

bdata scraper create <url> "<description>" [--name <name>] \
    [--deliver-webhook <url>] [--timeout <seconds>] \
    [--json | --pretty] [-o <path>] [--timing] [-k <api-key>]

The description is the most important input. A good description names every field you want and any conditions on how to find them. See [references/prompts.md](references/prompts.md) for examples of strong vs. weak descriptions.

# Minimal
bdata scraper create https://example.com/product/1 \
    "Extract title, price, currency, image URL, and availability \
     from this product page. If the price has a strike-through \
     original price, capture both as price and original_price."

# Save the full AI output for inspection
bdata scraper create https://example.com/product/1 \
    "Extract title, price, and image URL" \
    --name product-scraper-v1 \
    --pretty -o create.json

What happens under the hood

`create` chains three Bright Data API calls — surface this to the user so they can debug from logs:

1. **`POST /dca/collector`** — creates an empty scraper template with a stub webhook delivery target (`https://example.com/webhook` by default). Returns a `collector_id` like `c_mp3tuab31lswoxvpws`. 2. **`POST /dca/collectors/{collector_id}/automate_template`** — triggers Bright Data's AI Flow with the description + URL. 3. **`GET .../automate_template/progress`** (polled) — waits for `status: "done"`. Generation typically takes **5–10 minutes** for moderately complex pages.

Critical: hold the `collector_id`

Every failure path in `create` (AI trigger fails, polling times out, generation finishes with `status: "failed"`) **still leaves a partially-built collector** at the printed `collector_id`. Always tell the user the id is recoverable — they can:

• Open `https://brightdata.com/cp/scrapers/{collector_id}` to finish or inspect it in the web UI.
• Re-trigger generation programmatically against the same id.
• Delete it from the UI if they want a clean slate.

Never claim "create failed, start over" without surfacing the `collector_id` from the response.

`--timeout` — default 600 s

AI generation can run 5–10 min for complex pages. If the page is simple, the default is plenty. For an elaborate