/robots

name: robots-txt
description: When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.
metadata:
  version: 1.2.0

SEO Technical: robots.txt

Guides configuration and auditing of robots.txt for search engine and AI crawler control.

**When invoking**: On **first use**, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On **subsequent use** or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)

• **Robots.txt**: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
• **Crawler access**: Path-level crawl control; AI crawler allow/block strategy
• **Differentiation**: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See **indexing** for page-level exclusions.

Initial Assessment

**Check for project context first:** If `.claude/project-context.md` or `.cursor/project-context.md` exists, read it for site URL and indexing goals.

Identify: 1. **Site URL**: Base domain (e.g., `https://example.com`) 2. **Indexing scope**: Full site, partial, or specific paths to exclude 3. **AI crawler strategy**: Allow search/indexing vs. block training data crawlers

Best Practices

Purpose and Limitations

| Point | Note | |-------|------| | **Purpose** | Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet) | | **Advisory** | Rules are advisory; malicious crawlers may ignore | | **Public** | robots.txt is publicly readable; use noindex or auth for sensitive content. See **indexing** |

Crawl vs Index vs Link Equity (Quick Reference)

| Tool | Controls | Prevents indexing? | |------|----------|-------------------| | **robots.txt** | Crawl (path-level) | No—blocked URLs may still appear in SERP | | **noindex** (meta / X-Robots-Tag) | Index (page-level) | Yes. See **indexing** | | **nofollow** | Link equity only | No—does not control indexing |

When to Use robots.txt vs noindex

| Use | Tool | Example | |-----|------|---------| | **Path-level** (whole directory) | robots.txt | `Disallow: /admin/`, `Disallow: /api/`, `Disallow: /staging/` | | **Page-level** (specific pages) | noindex meta / X-Robots-Tag | Login, signup, thank-you, 404, legal. See **indexing** for full list | | **Critical** | Do NOT block in robots.txt | Pages that use noindex—crawlers must access the page to read the directive |

**Paths to block in robots.txt**: /admin/, /api/, /staging/, temp files. **Paths to use noindex** (allow crawl): /login/, /signup/, /thank-you/, etc.—see **indexing**.

Location and Format

| Item | Requirement | |------|-------------| | **Path** | Site root: `https://example.com/robots.txt` | | **Encoding** | UTF-8 plain text | | **Standard** | RFC 9309 (Robots Exclusion Protocol) |

Core Directives

| Directive | Purpose | Example | |-----------|---------|---------| | `User-agent:` | Target crawler | `User-agent: Googlebot`, `User-agent: *` | | `Disallow:` | Block path prefix | `Disallow: /admin/` | | `Allow:` | Allow path (can override Disallow) | `Allow: /public/` | | `Sitemap:` | Declare sitemap absolute URL | `Sitemap: https://example.com/sitemap.xml` | | `Clean-param:` | Strip query params (Yandex) | See below |

Path Format Rules (Critical)

Different directives use different path formats — a common source of errors:

| Directive | Format | Correct | Wrong | |-----------|--------|---------|-------| | Disallow / Allow | **Root-relative path only** (starts with `/`) | `Disallow: /admin/` | `Disallow: https://example.com/admin/` | | Sitemap | **Absolute URL only** | `Sitemap: https://example.com/sitemap.xml` | `Sitemap: /sitemap.xml` |

**Wildcards**: `*` matches any character sequence (`Disallow: /tmp/*`). `$` marks exact URL ending (`Allow: /news/.html$`).

**Priority**: More specific paths take precedence. `Allow: /shop/shoes/` overrides `Disallow: /shop/`. Path matching is case-sensitive: `Disallow: /PDF/` does not match `/pdf/`.

Critical: Do Not Block

| Do not block | Reason | |--------------|--------| | CSS, JS, images | Google needs them to render pages; blocking breaks indexing | | `/_next/` (Next.js) | Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See **indexing** | | Pages that use noindex | Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that |

**Only block**: paths that don't need crawling: /admin/, /api/, /staging/, temp files.

AI Crawler Strategy

robots.txt is effective for all measured AI crawlers. Set rules per user-agent; check each vendor's docs for current tokens.

| User-agent | Purpose | Typical | Notes | |------------|---------|---------|-------| | **OAI-SearchBot** | ChatGPT search | Allow | Respects robots.txt | | **GPTBot** | OpenAI training | Disallow | Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed | | **ChatGPT-User** | User-initiated browsing | N/A | **No longer respects robots.txt** (Dec 2025); use server-side controls instead | | **Claude-SearchBot** | Claude search | Allow | Respects robots.txt | | **Claude-User** | Anthropic user-initiated browsing | Allow | Respects robots.txt (unlike ChatGPT-User) | | **ClaudeBot** | Anthropic training | Disallow | Respects robots.txt | > **Deprecated**: ~~anthropic-ai~~ — retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References to `anthropic-ai` in robots.txt have no effect. | **PerplexityBot** | Perplexity search | Allow | Respects robots.txt | | **Google-Extended** | Gemini training | Disallow | Respects robots.txt | | **CCBot** | Common