/seo-technical

name: seo-technical
description: >
  Audit technical SEO across crawlability, indexability, security, URLs, mobile,
  Core Web Vitals, rendering, structured data, and IndexNow. Use when the user
  asks for a technical SEO audit or about crawling, indexing, rendering, or
  site speed. Exclude content strategy and backlinks.
user-invocable: true
argument-hint: "[url]"
license: MIT
metadata:
  author: AgriciDaniel
  version: "2.6.0"
  category: seo

Technical SEO Audit

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: run `"${CLAUDE_PLUGIN_ROOT}/scripts/claude-seo" run sitemap_discovery.py <url> --json`; require a

valid entry in `found`, and report stale or unsafe robots.txt declarations separately from working fallback locations

• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution (method in section 8)
• Crawl budget: for large sites (>10k pages), efficiency matters
• Googlebot **fetch limits**: Googlebot fetches the first **2MB of HTML** and first **64MB of a PDF** (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
• Crawl rate **auto-adjusts** (backs off on 5xx/slow responses); there is **no manual crawl-rate control** (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
• Google's canonical crawling/robots reference moved to **developers.google.com/crawling** (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.
• AMP has no separate ranking advantage. Since 2026-07-01, Google Search sends

users directly to publisher-hosted AMP URLs, so do not recommend AMP Cache, AMP Viewer, or signed exchange maintenance. Audit AMP against the same content, action-parity, and quality requirements as other pages.

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers** (the authoritative table, with robots.txt behaviour per crawler, is in `seo-geo`):

| Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training (NOT ChatGPT Search) | | OAI-SearchBot | OpenAI | `OAI-SearchBot` | ChatGPT Search citability | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing (user-triggered) | | ClaudeBot | Anthropic | `ClaudeBot` | Model training (NOT Claude search citability) | | Claude-SearchBot | Anthropic | `Claude-SearchBot` | Claude search-result citability | | PerplexityBot | Perplexity | `PerplexityBot` | Perplexity search index (not model training) | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training and grounding, and training of the models behind Search gen-AI features (no effect on Search inclusion or ranking) | | Applebot-Extended | Apple | `Applebot-Extended` | Apple Intelligence training opt-out (NOT Siri/Spotlight/Safari) | | CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**

• Blocking `Google-Extended` prevents Gemini training and grounding use (and training of the models behind Search gen-AI features) but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
• Blocking `GPTBot` prevents OpenAI training but does NOT affect ChatGPT Search

citability, which is governed by `OAI-SearchBot`, nor user-triggered browsing (`ChatGPT-User`). Check `OAI-SearchBot` for any citability claim; `GPTBot` status is evidence about training use only

• Blocking `ClaudeBot` prevents Anthropic model training but does NOT affect

citability in Claude's own search features, which is governed by `Claude-SearchBot` (per Anthropic's crawler support article). Check `Claude-SearchBot` for any Claude-search citability claim; `ClaudeBot` status is evidence about training use only

• Blocking `Applebot-Extended` opts out of Apple Intelligence / generative-model

training use but does NOT affect discoverability via Siri, Spotlight, or Safari, which follows `Applebot` (per Apple's support article); `Applebot-Extended` does not itself crawl

**Example, selective AI crawler blocking:**

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

**Recommendation:** Consider your AI visibility strategy before blocking: blocking an AI search crawler removes the site from that engine's answers. Do not promise traffic from allowing one. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.

> **Google's user-triggered fetchers generally ignore robots.txt rules** (other vendors differ: Anthropic's Claude-User honors it). Google now documents **Google-Agent** (user-triggered agentic browsing) plus **Google-GeminiNotebook** (formerly Google-NotebookLM) and **Google Messages** as *user-triggered* fetchers that **cannot be blocked via robots.txt**. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: **Web Bot Auth** (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.

2. Indexability

• Canonical tags: self-referencing, no conflicts with noindex
• Du