/tavily-crawl

name: tavily-crawl
description: |
  Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract an entire docs section, bulk-extract pages, save a site as local markdown files, or says "crawl", "get all the pages", "download the docs", "extract everything under /docs", "bulk extract", or needs content from many pages on the same domain. Supports depth/breadth control, path filtering, semantic instructions, and saving each page as a local markdown file.
allowed-tools: Bash(tvly *)

tavily crawl

Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file.

Before running

Crawl requires authentication. Run the requested command directly when `tvly` is already authenticated; do not add a status check to every invocation.

If `tvly` is missing, follow the [tavily-cli setup](../tavily-cli/SKILL.md#setup). If an installed CLI reports an authentication error, use `tvly login` for authentication only, or `tvly init --skip-skills` when guided verification is also useful. Browser-based OAuth is preferred when an interactive user can complete it. `--no-browser` prints the sign-in link instead of opening it, but still waits for a localhost callback. In an unattended agent or CI environment, leave authentication to the user or use a securely provided `TAVILY_API_KEY`. Do not start a second login immediately after guided setup has completed.

When to use

• You need content from many pages on a site (e.g., all `/docs/`)
• You want to download documentation for offline use
• Step 4 in the [workflow](../tavily-cli/SKILL.md): search → extract → map → **crawl** → research

Quick start

# Basic crawl
tvly crawl "https://docs.example.com" --json

# Save each page as a markdown file
tvly crawl "https://docs.example.com" --output-dir ./docs/

# Deeper crawl with limits
tvly crawl "https://docs.example.com" --max-depth 2 --limit 50 --json

# Filter to specific paths
tvly crawl "https://example.com" --select-paths "/api/.*,/guides/.*" --exclude-paths "/blog/.*" --json

# Semantic focus (returns relevant chunks, not full pages)
tvly crawl "https://docs.example.com" --instructions "Find authentication docs" --chunks-per-source 3 --json

Options

| Option | Description | |--------|-------------| | `--max-depth` | Levels deep (1-5, default: 1) | | `--max-breadth` | Links per page (default: 20) | | `--limit` | Total pages cap (default: 50) | | `--instructions` | Natural language guidance for semantic focus | | `--chunks-per-source` | Chunks per page (1-5, requires `--instructions`) | | `--extract-depth` | `basic` (default) or `advanced` | | `--format` | `markdown` (default) or `text` | | `--select-paths` | Comma-separated regex patterns to include | | `--exclude-paths` | Comma-separated regex patterns to exclude | | `--select-domains` | Comma-separated regex for domains to include | | `--exclude-domains` | Comma-separated regex for domains to exclude | | `--allow-external / --no-external` | Include external links (default: allow) | | `--include-images` | Include images | | `--timeout` | Max wait (10-150 seconds) | | `-o, --output` | Save JSON output to file | | `--output-dir` | Save each page as a .md file in directory | | `--json` | Structured JSON output |

Crawl for context vs. data collection

**For agentic use** (feeding results to an LLM):

Always use `--instructions` + `--chunks-per-source`. Returns only relevant chunks instead of full pages — prevents context explosion.

tvly crawl "https://docs.example.com" --instructions "API authentication" --chunks-per-source 3 --json

**For data collection** (saving to files):

Use `--output-dir` without `--chunks-per-source` to get full pages as markdown files.

tvly crawl "https://docs.example.com" --max-depth 2 --output-dir ./docs/

Tips

• **Start conservative** — `--max-depth 1`, `--limit 20` — and scale up.
• **Use `--select-paths`** to focus on the section you need.
• **Use map first** to understand site structure before a full crawl.
• **Always set `--limit`** to prevent runaway crawls.

See also

• [tavily-map](../tavily-map/SKILL.md) — discover URLs before deciding to crawl
• [tavily-extract](../tavily-extract/SKILL.md) — extract individual pages
• [tavily-search](../tavily-search/SKILL.md) — find pages when you don't have a URL