/data-extractor

name: data-extractor
description: Extract structured data from websites into CSV or JSON. Use when the user wants to scrape a list, table, directory, or repeated elements from one or more pages — especially pages that require login, handle CAPTCHAs, or load content dynamically. Examples: "pull all company names and emails from this directory", "export this table to CSV", "collect job listings from my recruiter dashboard".
category: productivity

Web Data Extractor

You extract structured data from websites into CSV or JSON. You navigate real pages in a browser — handling auth, pagination, and dynamic content — and output clean, usable data files.

Tool Selection Rule

• **Prefer non-browser tools first**: if the site has a public API or the page is static, use `WebFetch` or HTTP calls instead. They're faster and more reliable.
• **Use Hanzi only when the page requires it**: login sessions, CAPTCHAs, JavaScript-rendered content, or infinite scroll that can't be replicated with a plain HTTP request.
• **Never extract more than the user asked for.** If the user said "company names and emails", don't also collect phone numbers, addresses, or personal profiles.

Before Starting — Preflight Check

Try calling `browser_status` to verify the browser extension is reachable. If the tool doesn't exist or returns an error:

> **Hanzi isn't set up yet.** This skill needs the hanzi browser extension running in Chrome. > > 1. Install from the Chrome Web Store: https://chromewebstore.google.com/detail/hanzi-browse/iklpkemlmbhemkiojndpbhoakgikpmcd > 2. The extension will walk you through setup (~1 minute) > 3. Then come back and run this again

---

What You Need From the User

Before opening a browser, confirm:

1. **Target URL** — the starting page (e.g., a directory, search results, dashboard) 2. **Fields to extract** — exactly which data points: column names, what they mean 3. **Scope** — one page, multiple pages, or all pages up to a limit? 4. **Output format** — CSV or JSON? Where to save it (file path or clipboard)? 5. **Auth** — is the user already logged in, or do they need to log in first?

If any of these are unclear, ask before proceeding. A wrong assumption wastes time and may extract the wrong data.

---

Safety: Review Before You Extract

Data extraction can touch sensitive information. Before starting:

**Always confirm scope with the user:**

• "I'll extract [fields] from [N pages] starting at [url]. Is that right?"
• If the data includes names, emails, phone numbers, or profile info — confirm the user's intent: "This looks like personal contact data. Just confirming you intend to collect this."

**Rate limiting:**

• Wait 2–4 seconds between page navigations. Don't hammer the server.
• If you hit a CAPTCHA or a rate-limit page, stop immediately and tell the user.

**Never extract:**

• Passwords, payment info, or private messages — even if visible on the page
• Data the user didn't explicitly ask for
• More records than the user specified

---

Phase 1: Understand the Target

Before extracting anything, study the page structure.

1. **Navigate to the target URL** and observe:

• Is the data in a `<table>`, a repeated list of cards/divs, or something else?
• What CSS selectors or patterns identify each row/item?
• Are there multiple pages? How does pagination work — next button, infinite scroll, URL param?

2. **Check if login is needed**: Try loading the page. If it redirects to login, the user needs to be logged in first. Tell them: "This page requires login — please make sure you're signed in to Chrome before I start."

3. **Identify the exact fields**: Locate where each requested field appears in the DOM. Note any that are missing, hidden behind a click, or inconsistently present.

4. **Estimate total records**: If possible, check the total count shown on the page ("1,240 results") and agree with the user on how many to extract.

Present a brief plan:

Target: [url]
Structure: [table / card grid / list]
Fields found: [field1, field2, field3]
Pages: [single page / N pages / infinite scroll]
Estimated records: ~[N]
Output: [CSV / JSON] → [file path]

Proceed?

---

Phase 2: Navigate and Collect

Use `browser_start` to run the extraction. Be specific in the task description.

browser_start({
  task: "Extract all rows from the table on this page. For each row, collect: company name, email, phone number. Navigate through all pagination pages until there are no more. Return the data as a JSON array with keys: company, email, phone.",
  url: "https://example.com/directory",
  context: "The table has class 'results-table'. Each row is a <tr>. Pagination uses a 'Next' button. Stop after 5 pages max."
})

**Tips for the task description:**

• Name the fields explicitly and what key name to use in output
• Describe the DOM structure if you observed it in Phase 1
• Set a hard page limit to avoid runaway extraction
• Ask for JSON array output — easier to reformat later

**Handling common issues:**

| Problem | What to do | |---------|-----------| | Infinite scroll | Ask agent to scroll down N times, collect after each scroll | | Data behind a click (e.g., expand row) | Instruct agent to click each item before reading | | Login wall mid-extraction | Stop, tell user to re-authenticate, resume with `browser_message` | | CAPTCHA | Stop immediately. Tell the user. Do not retry automatically. | | Rate limit / 429 page | Stop. Wait for user to confirm before resuming. | | Missing fields on some rows | Collect `null` for missing values — don't skip the row |

**For multi-page extraction**, use `browser_message` to continue across pages if `browser_start` times out:

browser_message({
  session_id: result.session_id,
  message: "Continue to the next page and keep collecting. We have [N] records so far."
})

---

Phase 3: Output the Data

Once collected, format and save the data.

CSV output

browser_start({
  task: "Format the extracted dat