/ultimate-browsing

name: ultimate-browsing
description: "Reaches web pages the user's browser cannot open: WAF-blocked and bot-scored hosts, platform-native readers (Chinese and social platforms), archive snapshots with provenance, and a browser you own for stealth or throwaway profiles. Not for plain search, an unblocked static fetch, or a page the user's browser can open."

Ultimate Browsing

Web access for the hosts the user's browser cannot open (WAF / 403 / Cloudflare / bot-scored pages) and for platforms with a native reader. Reading and interaction on a page the user's browser can open belong to the `browser` skill's attached engine, not here. Start at the cheapest tier that can reach the blocked host and climb only when it cannot:

**Tier 1 — insane-search** (headless extraction + WAF bypass) -> **Tier 1.5 — agent-reach** (platform-native APIs, esp. Chinese platforms) -> **Tier 2 — a real browser** through omowright from js eval: 2a the owned engine (a browser your code launches, CloakBrowser for stealth), 2b the attached engine (the user's own signed-in browser).

PHASE 0 — ROUTE FIRST (MANDATORY)

User request
  |
  +- a page the user's browser can open (read, click, screenshot, QA, login)  NOT this skill (browser skill, attached engine)
  +- simple search query ------------------------------ NOT this skill (use web search)
  |
  +- URL blocked / 403 / Cloudflare / WAF ------------- TIER 1  insane-search
  +- YouTube/Vimeo/TikTok subtitles or metadata ------- TIER 1  insane-search (yt-dlp)
  +- archive snapshot of a dead or changed page ------- TIER 1  insane-search (surrogates, provenance-tagged)
  |
  +- Chinese platform (xhs/douyin/weibo/bilibili/v2ex/wechat)  TIER 1.5 agent-reach
  +- podcast transcript ------------------------------- TIER 1.5 agent-reach
  +- Twitter feed / LinkedIn profile / GitHub via CLI - TIER 1.5 agent-reach
  |
  +- Tier 1/1.5 returned empty or partial ------------- TIER 2  2a owned engine (stealth)
  +- a blocked host that also needs a click, a scroll or a screenshot  TIER 2  2a owned engine
  +- the user's account on a host the owned engine cannot pass  TIER 2  2b attached engine (their browser)

Read the matching reference before acting: [`references/insane-search/README.md`](references/insane-search/README.md), [`references/agent-reach/README.md`](references/agent-reach/README.md), or [`references/chrome-stealth.md`](references/chrome-stealth.md).

Tier 1 — insane-search (headless extraction)

**When**: blocked-URL bypass, media metadata, archive surrogates — no browser UI needed. **Why first**: ~10x faster than a browser, no process spin-up; handles most "fetch this blocked page" requests via curl_cffi TLS impersonation, yt-dlp (1858 sites), official public APIs, mobile URL transforms, **Phase-2.5 surrogate archives** (Wayback / archive.today snapshots, provenance-tagged — see [`references/insane-search/cache-archive.md`](references/insane-search/cache-archive.md)), a key-gated Jina Reader (`JINA_API_KEY`), and a Playwright real-Chrome fallback. The engine lives **inside this skill** at `engine/` and is invoked as a module. Surrogate results are dated COPIES: a result whose `provenance` is `snapshot` must be reported with its `snapshot_timestamp`, never presented as the live page.

# Core command — auto-detects WAF, runs the full fetch grid (run from the skill dir).
# Keep -B: this skill can live inside a signed app bundle, and bytecode caches written into engine/ break its signature.
python3 -B -m engine "https://example.com/blocked-page"
#   add --selector "<CSS>" for positive-proof validation, --device auto|desktop|mobile,
#   --trace to inspect every attempt, --json for machine-readable output.

# YouTube subtitles / metadata (no browser):
yt-dlp --write-sub --write-auto-sub --sub-lang "en,ko" --skip-download -o "/tmp/%(id)s" "<URL>"

# Reddit / HN / Bluesky / arXiv etc. use official public endpoints — see the Phase 0 index in
# references/insane-search/README.md (Twitter syndication, Reddit .json, HN Firebase, ...).

The full engine harness (rules R1-R7, the Phase 0 official-API index, the no-site-name rule, and the `references/insane-search/*.md` deep-dives for TLS, Playwright routing, Naver, media, etc.) is in [`references/insane-search/README.md`](references/insane-search/README.md). Read it before tuning the engine or adding a WAF profile.

Escalate to Tier 1.5 or Tier 2 when

• The target is a Chinese / social platform with a native reader -> Tier 1.5.
• insane-search returns empty/partial, or the blocked page needs JS interaction, a screenshot, or media playback -> Tier 2.

Tier 1.5 — agent-reach (platform-native readers)

**When**: the target is a platform with a first-class API/CLI that beats generic fetching — especially Chinese platforms that stealth browsers still cannot reach cleanly. Several channels are zero-config (Douyin, V2EX, Reddit, RSS, YouTube); others need a one-time auth you supply via environment variables if you have access (`JINA_API_KEY` for Jina Reader — anonymous access is dead, see `references/insane-search/jina.md`; `TWITTER_*` for X; a transcription key for podcasts).

| Category | Platforms | Entry | |---|---|---| | social | xhs (Xiaohongshu), douyin, weibo, bilibili, V2EX, Reddit, Twitter/X | [references/agent-reach/social.md](references/agent-reach/social.md) | | web | Jina Reader, WeChat articles, RSS | [references/agent-reach/web.md](references/agent-reach/web.md) | | video | YouTube, Bilibili, podcast transcripts, Douyin video | [references/agent-reach/video.md](references/agent-reach/video.md) | | career | LinkedIn | [references/agent-reach/career.md](references/agent-reach/career.md) | | dev | GitHub (gh CLI) | [references/agent-reach/dev.md](references/agent-reach/dev.md) | | search | Exa AI | [references/agent-reach/search.md](references/agent-reach/search.md) |

mcporter call 'douyin.parse_douyin_video_info(share_link: "<URL>")'   # douyin, zero-config
curl -s -H "Aut