/cloud-browser-automation

name: cloud-browser-automation
description: Use cloud browser services (Browserbase) for Cloudflare bypass, JavaScript rendering, and stealth scraping when local tools fail
tags: [browserbase, cloudflare, scraping, automation, cloud]
related_skills: [web-scraping]
origin: unknown
source_license: see upstream
language: en

Cloud Browser Automation

Use cloud browser services (Browserbase, Browser Use Cloud) for web scraping when local tools fail due to Cloudflare protection, JavaScript rendering requirements, or anti-bot measures.

Usage Strategy

**CRITICAL: On-demand only, close immediately**

Always follow this pattern: 1. Create session ONLY when needed 2. Do the task (scrape/navigate/extract) 3. Close session IMMEDIATELY after done 4. Never leave sessions running idle

Why:

• Save credits (pay per minute)
• Avoid hitting concurrent session limits (e.g., Browserbase max 3)
• Clean resource usage
• No waste

When to Use

**Escalation ladder** (always try cheapest/fastest first):

1. Try Hermes web_fetch first
   ↓ (if fails or empty content)
2. Try direct curl with proper headers (Pattern 0 — works for most static sites)
   ↓ (if blocked or needs JS)
3. Try Hermes browser
   ↓ (if blocked by Cloudflare or sandbox issues)
4. Use cloud browser (Browserbase)

**Use cloud browser when:**

• ✅ Cloudflare Turnstile blocking requests
• ✅ JavaScript-heavy sites (SPA, dynamic content)
• ✅ Anti-bot detection (fingerprinting, TLS checks)
• ✅ Complex multi-step automation (login, navigate, extract)

**DON'T use cloud browser when:**

• ❌ Simple HTTP requests (use web_fetch)
• ❌ Static content (use web_fetch)
• ❌ Cost-sensitive tasks (cloud = pay per minute)

Browserbase Setup

API Configuration

# Credentials (saved in memory)
API_KEY="bb_live_O6tVgdzl8B6WquBSj1XpuaC5hMc"
PROJECT_ID="a5008864-bbaa-4966-96e2-2272497b003d"
BASE_URL="https://www.browserbase.com/v1"

# Limits
MAX_CONCURRENT_SESSIONS=3
SESSION_TIMEOUT=5  # minutes (default)

Session Management

**CRITICAL USER PREFERENCE:**

> "gini jadi lo pakai browserbase saat dibutuhkan aja, nah sesi langsung close saat beres"

**Translation:** Use on-demand only, close immediately after done.

**Why:**

• ✅ Save credits (pay per minute)
• ✅ Avoid hitting concurrent limit (max 3)
• ✅ Clean resource usage
• ✅ No waste

**Pattern:**

# 1. Create session ONLY when needed
session = create_session()

# 2. Do the task
data = scrape_page(session)

# 3. Close IMMEDIATELY after done
close_session(session)

**DON'T:**

• ❌ Create session at start of script "just in case"
• ❌ Leave sessions running between tasks
• ❌ Reuse sessions across multiple unrelated tasks
• ❌ Forget to close sessions

Workflow

Step 1: Create Session

curl -X POST "https://www.browserbase.com/v1/sessions" \
  -H "X-BB-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "projectId": "'$PROJECT_ID'",
    "browserSettings": {
      "viewport": {"width": 1920, "height": 1080}
    }
  }'

**Response:**

{
  "id": "session-id-here",
  "status": "RUNNING",
  "connectUrl": "wss://connect-v2.usw2.browserbase.com/?signingKey=...",
  "seleniumRemoteUrl": "http://connect-v2.usw2.browserbase.com/webdriver",
  "expiresAt": "2026-05-03T13:00:00.000Z"
}

Step 2: Automate (Requires SDK)

**Option A: Playwright** (Python)

from playwright.async_api import async_playwright

async with async_playwright() as p:
    browser = await p.chromium.connect_over_cdp(connect_url)
    page = await browser.new_page()
    await page.goto("https://example.com")
    data = await page.evaluate("() => document.body.innerText")
    await browser.close()

**Option B: Selenium** (Python)

from selenium import webdriver

options = webdriver.ChromeOptions()
driver = webdriver.Remote(
    command_executor=selenium_remote_url,
    options=options
)
driver.get("https://example.com")
data = driver.find_element(By.TAG_NAME, "body").text
driver.quit()

**Option C: Manual (Fallback)**

# Get debug URL
curl "https://www.browserbase.com/v1/sessions/$SESSION_ID/debug" \
  -H "X-BB-API-Key: $API_KEY"

# Open debug URL in browser
# Navigate manually
# Screenshot/extract data manually

Step 3: Close Session

curl -X POST "https://www.browserbase.com/v1/sessions/$SESSION_ID/stop" \
  -H "X-BB-API-Key: $API_KEY"

**ALWAYS close, even if task fails:**

try:
    data = scrape_with_browserbase()
finally:
    close_session()  # Always close!

VPS Constraints

**Problem:** Ubuntu 24.04 strict package management blocks `pip install playwright/selenium`.

**Solutions:**

Option 1: Use Docker (Recommended)

# Run Playwright in container
docker run -it mcr.microsoft.com/playwright/python:v1.40.0-jammy \
  python3 /tmp/scraper.py

**Pros:** Isolated, no pip conflicts **Cons:** Need Docker (~500MB)

Option 2: Manual Fallback

# Create session
SESSION_ID=$(curl -X POST ... | jq -r '.id')

# Get debug URL
DEBUG_URL=$(curl ... | jq -r '.debuggerFullscreenUrl')

# User opens URL manually
echo "Open: $DEBUG_URL"
echo "Navigate to target site"
echo "Screenshot and send to me"

# Close session
curl -X POST ".../sessions/$SESSION_ID/stop" ...

Option 3: Delegate to Subagent

# If automation blocked, delegate research
delegate_task(
    goal="Research [topic] via web search",
    toolsets=["web", "terminal"]
)

Common Patterns

Pattern 0: Direct Curl for Static Sites (No Browserbase)

**Challenge:** Site returns HTML but might have Cloudflare or anti-bot checks

**Solution:**

# Direct curl with proper headers (often bypasses basic protection)
curl -s -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.5" \
  -H "Connection: keep-