/seo-technical

name: seo-technical
description: >
  Technical SEO audit across 9 categories: crawlability, indexability, security,
  URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering,
  and IndexNow protocol. Use when user says "technical SEO", "crawl issues",
  "robots.txt", "Core Web Vitals", "site speed", or "security headers".
user-invocable: true
argument-hint: "[url]"
license: MIT
metadata:
  author: AgriciDaniel
  version: "2.2.4"
  category: seo

Technical SEO Audit

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: run `claude-seo run sitemap_discovery.py <url> --json`; require a

valid entry in `found`, and report stale or unsafe robots.txt declarations separately from working fallback locations

• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters
• Googlebot **fetch limits**: Googlebot fetches the first **2MB of HTML** and first **64MB of a PDF** (uncompressed; 15MB is the broader crawler-infra default). Long-standing, not a 2026 change, but inline base64 images, oversized inline CSS/JS, or bloated nav can push critical content/JSON-LD past the cap and out of the index. Keep key content + structured data within the first 2MB.
• Crawl rate **auto-adjusts** (backs off on 5xx/slow responses); there is **no manual crawl-rate control** (the legacy Search Console setting was removed Jan 2024). Influence crawling via sitemaps, server responsiveness, and robots controls.
• Google's canonical crawling/robots reference moved to **developers.google.com/crawling** (migrated 2025-11-20); IP-range files relocated to `/crawling/ipranges/` and `googlebot.json` was renamed `common-crawlers.json`.

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers:**

| Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing | | ClaudeBot | Anthropic | `ClaudeBot` | Model training | | PerplexityBot | Perplexity | `PerplexityBot` | Search index + training | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) | | CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**

• Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
• Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
• ~3-5% of websites now use AI-specific robots.txt rules

**Example, selective AI crawler blocking:**

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for the full AI crawler/fetcher taxonomy.

> **User-triggered fetchers ignore robots.txt by design.** Google now documents **Google-Agent** (Project Mariner, agentic browsing) plus **Google-NotebookLM** and **Google Messages** as *user-triggered* fetchers that **cannot be blocked via robots.txt**. Use server-side access controls instead. By contrast, `Google-Extended` and `Google-CloudVertexBot` obey robots.txt. Emerging: **Web Bot Auth** (RFC 9421) lets bots authenticate cryptographically via a `Signature-Agent` header + key directory at `agent.bot.goog` (used by Google-Agent); reverse-DNS verification remains the fallback.

2. Indexability

• Canonical tags: self-referencing, no conflicts with noindex
• Duplicate content: near-duplicates, parameter URLs, www vs non-www
• Thin content: pages below minimum word counts per type
• Pagination: rel=next/prev or load-more pattern
• Hreflang: correct for multi-language/multi-region sites
• Index bloat: unnecessary pages consuming crawl budget

3. Security

• HTTPS: enforced, valid SSL certificate, no mixed content
• Security headers:
• Content-Security-Policy (CSP)
• Strict-Transport-Security (HSTS)
• X-Frame-Options
• X-Content-Type-Options
• Referrer-Policy
• HSTS preload: check preload list inclusion for high-security sites
• **Back-button hijacking** (spam-policy violation, malicious practices): flag pages that defeat the Back button via `history.pushState`/`replaceState` (including scripts injected by third-party ad/library platforms). Added to Google's spam policies 2026-04-13; **enforcement live since 2026-06-15** (manual actions + automated demotions): treat as Critical.

4. URL Structure

• Clean URLs: descriptive, hyphenated, no query parameters for content
• Hierarchy: logical folder structure reflecting site architecture
• Redirects: no chains (max 1 hop), 301 for permanent moves
• URL length: flag >100 characters
• Trailing slashes: consistent usage

5. Mobile Optimization & Page Experience

• Responsive design: viewport meta tag, responsive CSS
• Touch targets: minimum 48x48px with 8px spacing
• Font size: minimum 16px base
• No horizontal scroll
• Mobile-first indexing: Googlebot Smartphone is the primary crawler (rollout completed 2024). A mobile version is **not strictly required** (Google says "very strongly recommended"), sites that don't work on mobile can still be indexed, but the real risk is **content/parity loss**, not hard exclusion.
• **Mobile/desktop content parity** (highest-value m