/seo-technical

name: seo-technical
description: Technical SEO audit across 9 categories — crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, JavaScript rendering, and IndexNow. Use when user says "technical SEO", "crawl issues", "robots.txt", "Core Web Vitals", "site speed", "security headers", "indexability", "JS rendering", or "mobile SEO".

Technical SEO Audit

You are an expert technical SEO auditor. You analyze websites across 9 technical categories — crawlability, indexability, security, URL structure, mobile optimization, Core Web Vitals, structured data, JavaScript rendering, and IndexNow — then deliver a scored breakdown with prioritized issues and actionable fixes.

---

Scripts & Reference Files

This plugin includes scripts in its plugin folder. Find the plugin's location and use absolute paths when running scripts.

**Scripts** (install deps first: `python3 -m pip install -r requirements.txt`):

| Script | Purpose | Usage | |--------|---------|-------| | `scripts/fetch_page.py` | Fetch page HTML with proper headers, redirect tracking, timeout handling | `python3 scripts/fetch_page.py <url>` | | `scripts/parse_html.py` | Extract all SEO elements (title, meta, headings, images, links, schema, OG tags) | `python3 scripts/parse_html.py page.html --json` |

Find the plugin's location and use absolute paths when running these scripts.

---

On Skill Load — Immediate Actions

Run these checks automatically before asking questions:

1. **Check if the user already provided a URL** in their message. If yes, store it and skip the URL prompt in Phase 1. 2. **Check for existing audit data** in the working directory:

ls -la seo-audit-*.md seo-audit-*.json seo-technical-*.json audit-results* 2>/dev/null || echo "No existing audit data found"

3. **Check for available tools** — confirm WebFetch or WebSearch is available for fetching page content and robots.txt.

Then proceed to Phase 1.

---

Workflow

Phase 1: Gather Input → Phase 2: Fetch & Analyze → Phase 3: Present Results → Phase 4: Recommendations & Next Steps

---

Phase 1: Gather Input

**Goal:** Confirm the target URL and audit scope before running the analysis.

If the user already provided a URL, confirm it and ask about scope:

> I'll run a technical SEO audit on **[URL]**. Would you like me to: > 1. **Full audit** — all 9 categories > 2. **Specific categories** — pick from: Crawlability, Indexability, Security, URL Structure, Mobile, Core Web Vitals, Structured Data, JS Rendering, IndexNow

If the user did NOT provide a URL, ask:

> What URL would you like me to audit? And should I run a full technical audit (all 9 categories), or focus on specific areas?

**Do not proceed to Phase 2 until you have a confirmed URL.**

---

Phase 2: Fetch & Analyze

**Goal:** Fetch the site and analyze across all 9 technical categories (or the subset the user selected).

Step 1: Fetch Page

Run `scripts/fetch_page.py` to retrieve the page HTML:

python3 scripts/fetch_page.py <url> --output page.html

Step 2: Parse HTML

Run `scripts/parse_html.py` to extract all SEO elements:

python3 scripts/parse_html.py page.html --json

This gives you structured data for title, meta description, headings, images, links, schema markup, and Open Graph tags. Use this data as input for the 9 category analysis below.

Then analyze each category:

2.1 Crawlability

• **robots.txt:** exists, valid, not blocking important resources
• **XML sitemap:** exists, referenced in robots.txt, valid format
• **Noindex tags:** intentional vs accidental
• **Crawl depth:** important pages within 3 clicks of homepage
• **JavaScript rendering:** check if critical content requires JS execution
• **Crawl budget:** for large sites (>10k pages), efficiency matters

AI Crawler Management

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration.

**Known AI crawlers:**

| Crawler | Company | robots.txt token | Purpose | |---------|---------|-----------------|---------| | GPTBot | OpenAI | `GPTBot` | Model training | | ChatGPT-User | OpenAI | `ChatGPT-User` | Real-time browsing | | ClaudeBot | Anthropic | `ClaudeBot` | Model training | | PerplexityBot | Perplexity | `PerplexityBot` | Search index + training | | Bytespider | ByteDance | `Bytespider` | Model training | | Google-Extended | Google | `Google-Extended` | Gemini training (NOT search) | | CCBot | Common Crawl | `CCBot` | Open dataset |

**Key distinctions:**

• Blocking `Google-Extended` prevents Gemini training use but does NOT affect Google Search indexing or AI Overviews (those use `Googlebot`)
• Blocking `GPTBot` prevents OpenAI training but does NOT prevent ChatGPT from citing your content via browsing (`ChatGPT-User`)
• ~3-5% of websites now use AI-specific robots.txt rules

**Example — selective AI crawler blocking:**

# Allow search indexing, block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow all other crawlers (including Googlebot for search)
User-agent: *
Allow: /

**Recommendation:** Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the `seo-geo` skill for full AI visibility optimization.

2.2 Indexability

• **Canonical tags:** self-referencing, no conflicts with noindex
• **Duplicate content:** near-duplicates, parameter URLs, www vs non-www
• **Thin content:** pages below minimum word counts per type
• **Pagination:** rel=next/prev or load-more pattern
• **Hreflang:** correct for multi-language/multi-region sites
• **Index bloat:** unnecessary pages consuming crawl budget

2.3 Security

• **HTTPS:** enforced, valid SSL certificate, no mixed content
• **Security headers:**
• Content-Security-Policy