/web-scraping

name: web-scraping
description: Extract data from websites, including JavaScript-rendered SPAs and dynamic content
triggers:
  - scrape website data
  - extract content from web page
  - get data from JavaScript site
  - parse SPA content
  - headless browser automation
origin: unknown
source_license: see upstream
language: en

Web Scraping

Extract structured data from websites, handling both static HTML and JavaScript-rendered content (React, Next.js, Vue, etc.).

When to Use

• User asks to "scrape", "extract", or "get data from" a website
• Target site uses client-side rendering (SPA frameworks)
• Need to interact with dynamic content (infinite scroll, lazy loading)
• API endpoints are not available or documented

Approach Selection

1. Static HTML (curl + parsing)

**Use when:** Site serves complete HTML without JavaScript rendering.

curl -sL 'https://example.com' | grep -oP 'pattern'
# or with jq for JSON APIs
curl -s 'https://api.example.com/data' | jq '.items[]'

**Pros:** Fast, lightweight, no dependencies **Cons:** Fails on JS-rendered content

2. Headless Browser (Puppeteer/Playwright)

**Use when:** Content is rendered client-side (React, Next.js, Vue, Angular).

**Node.js + Puppeteer** (recommended for WSL2/containers):

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({
  headless: true,
  args: ['--no-sandbox', '--disable-setuid-sandbox']  // Required in WSL2/containers
});

const page = await browser.newPage();
await page.goto('https://example.com', {
  waitUntil: 'networkidle2',
  timeout: 60000
});

// Wait for dynamic content
await new Promise(resolve => setTimeout(resolve, 3000));

// Extract text
const content = await page.evaluate(() => document.body.innerText);

// Extract structured data
const data = await page.evaluate(() => {
  return Array.from(document.querySelectorAll('.item')).map(el => ({
    title: el.querySelector('.title')?.innerText,
    value: el.querySelector('.value')?.innerText
  }));
});

await browser.close();

**Pros:** Handles all JS rendering, can interact with page **Cons:** Slower, heavier resource usage

3. API Inspection (DevTools Network tab)

**Use when:** Site loads data via XHR/fetch calls.

1. Open browser DevTools → Network tab 2. Filter by XHR/Fetch 3. Find API endpoint 4. Replicate with curl/fetch

**Pros:** Fastest, most reliable **Cons:** Requires manual inspection, may need auth tokens

**Alternative: Reverse-engineer from minified JS** (when browser access blocked):

**Method A: Direct curl (if no Cloudflare)**

# Download main JS bundle
curl -s "https://example.com/assets/main-[hash].js" > /tmp/bundle.js

# Search for API patterns
grep -oP '"/[a-z_/-]{3,}"' /tmp/bundle.js | sort -u
strings /tmp/bundle.js | grep -i 'keyword' | head -20

**Method B: TinyFish browser automation (if Cloudflare protected)**

When curl fails due to Cloudflare Turnstile, use TinyFish to bypass protection and download JS via Chrome DevTools Protocol:

# 1. Create TinyFish browser session (bypasses Cloudflare)
# 2. Wait for challenge completion
# 3. Connect to browser via CDP WebSocket
# 4. Use Runtime.evaluate to fetch JS files
# 5. Extract API endpoints from minified code

See `references/tinyfish-js-reverse-engineering.md` for full workflow (tested on rpow2swap.com May 2026).

**Trial-error common paths with size check:**

for path in /api/listings /api/orders /listings /tokens /api/stats; do
  echo "Testing: https://example.com$path"
  timeout 3 curl -s -m 3 -o /dev/null -w "HTTP %{http_code} | Size: %{size_download} bytes\\n" \
    "https://example.com$path" 2>&1 || echo "Timeout/Error"
done

# Look for large responses (>10KB = likely data endpoint, <2KB = likely SPA HTML)

**Success indicators:**

• Response size >10KB → likely JSON data endpoint
• Response size <2KB → likely SPA HTML fallback
• Timeout → endpoint exists but slow/protected

See `references/spa-api-discovery.md` for full technique (tested on rpow2swap.com May 2026).

**Success case (rpow2swap.com, May 2026):**

# 1. Try common API paths with timeout
for path in /api/listings /api/tokens /listings /tokens /api/orderbook; do
  timeout 3 curl -s -m 3 -o /dev/null -w "HTTP %{http_code} | Size: %{size_download}\\n" \
    "https://example.com$path"
done

# Result: /api/listings returned 65KB (200 OK) — found it!

# 2. Fetch and inspect data
curl -s "https://example.com/api/listings" | head -c 2000
# Returns JSON array with full listing data

# 3. Build monitoring bot
# State-based change detection: track seen IDs, alert on new entries

**Key insight:** Many SPAs use predictable REST paths (`/api/<resource>`). Trial-error with timeout is faster than reverse-engineering minified JS.

3.5. Third-Party APIs (Twitter/X)

**Use when:** Scraping Twitter/X content (tweets, profiles, media).

**Primary: vxtwitter API** (no auth, works from terminal)

# Get tweet data
curl -s "https://api.vxtwitter.com/Twitter/status/{tweet_id}" | jq -r '.tweet | {text, author, likes, retweets, replies, media}'

# Get account info
curl -s "https://api.vxtwitter.com/{handle}" | jq -r '.user | {name, description, followers, website}'

# Extract quoted tweet (QRT)
curl -s "https://api.vxtwitter.com/Twitter/status/{tweet_id}" | jq -r '.qrt | {text, author, likes}'

**Fallback: fxtwitter API** (same structure)

curl -s "https://api.fxtwitter.com/{handle}/status/{tweet_id}"

**Pros:** No auth, fast, structured JSON, includes media URLs **Cons:** Rate limited, may lag behind real-time data

**Note:** Twitter's official API requires auth and has strict rate limits. Use vxtwitter/fxtwitter for read-only access.

4. Cloud Browser Services (Cloudflare bypass)

**Use when:** Site has Cloudflare Turnstile, bot detection, or anti-scraping measures.

**Browserbase** (recommended, tested May 2026):

import requests

# Create session
response = requests.pos