/seo-technical

name: seo-technical
description: >
  Technical SEO audit across 9 categories: crawlability, indexability, security,
  URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering,
  and IndexNow protocol. Use when user says "technical SEO", "crawl issues",
  "robots.txt", "Core Web Vitals", "site speed", or "security headers".
user-invokable: true
argument-hint: "[url]"
license: MIT
metadata:
  author: AgriciDaniel
  version: "1.9.6"
  category: seo

Technical SEO Audit

Shared Data Cache

**Step 0 -- Check shared data cache:**

Before gathering, check `.seo-cache/` for reusable context from related SEO skills. Reference: `../seo/references/shared-data-cache.md` for schemas and dependency map.

Check these cache files when present:

• `.seo-cache/site-meta.json` for domain, business type, industry, and crawl context
• `.seo-cache/audit-scores.json` for prior full-audit priorities
• `.seo-cache/pages/{url-slug}/page-analysis.json` for page-level context when a URL is provided

• If found: parse and use clearly valid fields (note "Using cached [X] from [date]")
• If missing, corrupt, or irrelevant: continue with fresh evidence
• If the user says "refresh" or "re-run": ignore cache reads and overwrite on write

Categories

1. Crawlability

• robots.txt: exists, valid, not blocking important resources
• XML sitemap: exists, referenced in robots.txt, valid format
• Noindex tags: intentional vs accidental
• Crawl depth: important pages within 3 clicks of homepage
• JavaScript rendering: check if critical content requires JS execution
• Crawl budget: for large sites (>10k pages), efficiency matters

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers:**

| Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing | | ClaudeBot | Anthropic | `ClaudeBot` | Model training | | PerplexityBot | Perplexity | `PerplexityBot` | Search index + training | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) | | CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**

• Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
• Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
• ~3-5% of websites now use AI-specific robots.txt rules

**Example, selective AI crawler blocking:**

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for full AI visibility optimization.

2. Indexability

• Canonical tags: self-referencing, no conflicts with noindex
• Duplicate content: near-duplicates, parameter URLs, www vs non-www
• Thin content: pages below minimum word counts per type
• Pagination: rel=next/prev or load-more pattern
• Hreflang: correct for multi-language/multi-region sites
• Index bloat: unnecessary pages consuming crawl budget

3. Security

• HTTPS: enforced, valid SSL certificate, no mixed content
• Security headers:
• Content-Security-Policy (CSP)
• Strict-Transport-Security (HSTS)
• X-Frame-Options
• X-Content-Type-Options
• Referrer-Policy
• HSTS preload: check preload list inclusion for high-security sites

4. URL Structure

• Clean URLs: descriptive, hyphenated, no query parameters for content
• Hierarchy: logical folder structure reflecting site architecture
• Redirects: no chains (max 1 hop), 301 for permanent moves
• URL length: flag >100 characters
• Trailing slashes: consistent usage

5. Mobile Optimization

• Responsive design: viewport meta tag, responsive CSS
• Touch targets: minimum 48x48px with 8px spacing
• Font size: minimum 16px base
• No horizontal scroll
• Mobile-first indexing: Google indexes mobile version. **Mobile-first indexing is 100% complete as of July 5, 2024.** Google now crawls and indexes ALL websites exclusively with the mobile Googlebot user-agent.

6. Core Web Vitals

• **LCP** (Largest Contentful Paint): target <2.5s
• **INP** (Interaction to Next Paint): target <200ms
• INP replaced FID on March 12, 2024. FID was fully removed from all Chrome tools (CrUX API, PageSpeed Insights, Lighthouse) on September 9, 2024. Do NOT reference FID anywhere.
• **CLS** (Cumulative Layout Shift): target <0.1
• Evaluation uses 75th percentile of real user data
• Use PageSpeed Insights API or CrUX data if MCP available

7. Structured Data

• Detection: JSON-LD (preferred), Microdata, RDFa
• Validation against Google's supported types
• See seo-schema skill for full analysis

8. JavaScript Rendering

• Check if content visible in initial HTML vs requires JS
• Identify client-side rendered (CSR) vs server-side rendered (SSR)
• Flag SPA frameworks (React, Vue, Angular) that may cause indexing issues
• Verify dynamic rendering setup if applicable

JavaScript SEO: Canonical & Indexing Guidance (December 2025)

Google updated its JavaScript SEO documentation in December 2025 with critical clarifications:

1. **Canonical conflicts:** If a canonical tag in raw HTML differs from one injected by JavaScript, Google may use EITHER one. Ensure canonical tags are identical between server-rendered HTML and JS-rendered output. 2. **noindex with JavaScript:** If raw HTML contains `<meta name="ro