/ai-bot-log-audit

name: ai-bot-log-audit
description: Use when analyzing server logs to understand how AI crawlers (GPTBot, ClaudeBot, PerplexityBot) interact with your site. Use when optimizing content placement for LLM retrieval, diagnosing why AI search isn't citing your content, or auditing crawl patterns to find optimization gaps.
license: MIT
metadata:
  author: ClawFu
  version: 1.0.0
  mcp-server: "@clawfu/mcp-skills"

AI Bot Log Audit

> Analyze server logs to understand how AI crawlers retrieve your content, then optimize placement and structure for maximum citation probability. Based on Metehan Yeşilyurt's log file analysis framework.

When to Use This Skill

Use this skill when you need to:

• **Audit AI bot crawl patterns** on your site (what they fetch, how often, what they skip)
• **Diagnose citation gaps** — your content exists but AI search doesn't cite it
• **Optimize content placement** for LLM retrieval mechanics (lost-in-the-middle, embedding similarity)
• **Compare AI bot behavior** across different products (Google, OpenAI, Anthropic, Perplexity)
• **Build a GEO strategy** grounded in actual crawl data, not assumptions
• **Identify which pages AI bots prioritize** and which they ignore

This skill is particularly valuable for:

• SEO professionals adding AI search to their optimization scope
• Publishers monitoring AI traffic and content extraction
• Technical SEOs conducting log file analysis
• Content strategists deciding what to optimize for AI visibility

---

Methodology Foundation

**Source Expert:** Metehan Yeşilyurt (SEO Consultant, speaker at BrightonSEO, The Search Session)

**Core Thesis:** Log file analysis reveals the real mechanics of AI search retrieval. Different AI products (AI Overviews, AI Mode, Perplexity) use fundamentally different retrieval pipelines, so optimizing for "AI search" requires understanding each product's specific crawl and retrieval behavior.

> "AI Overviews, AI Mode, and Web Guide are three completely different products with different retrieval behaviors. You can't optimize for 'AI search' as if it's one thing." — Metehan Yeşilyurt, The Search Session

**Key Insight:** The shift from traditional SEO to AI search optimization is from chasing clicks in deterministic SERPs to chasing citations in machine-generated text. Log files reveal the mechanics behind citation selection.

---

What Claude Does vs What You Decide

| Claude Does | You Decide | |-------------|------------| | Guides log analysis methodology | Which log data to provide | | Identifies patterns in crawl behavior | Strategic priority of AI products | | Recommends content placement optimizations | Content creation/modification scope | | Maps retrieval behavior to optimization actions | Resource allocation for changes | | Creates audit templates and checklists | Which recommendations to implement |

---

What This Skill Does

When invoked, I will guide you through:

1. **Log Collection** — Identify and extract AI bot activity from server logs 2. **Bot Identification** — Map user agents to AI products 3. **Crawl Pattern Analysis** — Understand what AI bots fetch and ignore 4. **Retrieval Mechanics** — Learn how each AI product processes your content 5. **Optimization Actions** — Specific changes to improve AI citation probability

---

Instructions

Phase 1: AI Bot Identification

Known AI Crawlers (2026)

| Bot User Agent | Operator | Purpose | Respects robots.txt? | |---------------|----------|---------|---------------------| | **GPTBot** | OpenAI | Training data + ChatGPT Browse | Yes | | **ChatGPT-User** | OpenAI | Real-time browsing in ChatGPT | Yes | | **OAI-SearchBot** | OpenAI | SearchGPT / ChatGPT Search | Yes | | **ClaudeBot** | Anthropic | Training data collection | Yes | | **PerplexityBot** | Perplexity | Real-time search + answer generation | Yes (mostly) | | **Google-Extended** | Google | Gemini/AI training (deprecated — now part of Googlebot) | N/A | | **Googlebot** | Google | Traditional crawl + AI Overviews + AI Mode | Yes | | **Bytespider** | ByteDance | Training for TikTok AI features | Yes | | **CCBot** | Common Crawl | Open dataset used by many AI models | Yes | | **Applebot-Extended** | Apple | Apple Intelligence features | Yes |

Log Extraction Commands

# Extract all AI bot hits from Apache/Nginx access logs
grep -iE "(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Bytespider|CCBot|Applebot-Extended)" access.log > ai_bots.log

# Count hits by bot
awk -F'"' '{print $6}' ai_bots.log | grep -oE "(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Bytespider|CCBot)" | sort | uniq -c | sort -rn

# Top pages crawled by AI bots
awk '{print $7}' ai_bots.log | sort | uniq -c | sort -rn | head -50

# Crawl frequency by day
awk '{print $4}' ai_bots.log | cut -d: -f1 | tr -d '[' | sort | uniq -c

# Response codes for AI bots (are they getting 200s or errors?)
awk '{print $9}' ai_bots.log | sort | uniq -c | sort -rn

---

Phase 2: Crawl Pattern Analysis

What to Look For

| Pattern | What It Means | Action | |---------|--------------|--------| | **Bot fetches page frequently** | Page is in retrieval index, content matters | Optimize this page first | | **Bot fetches page once then stops** | Page was evaluated and deprioritized | Improve content quality/freshness | | **Bot never fetches a page** | Page not discovered or blocked | Check internal linking, sitemap, robots.txt | | **Bot gets 404/500** | Technical issue blocking retrieval | Fix immediately | | **Bot fetches but doesn't cite** | Content retrieved but not selected for answers | Improve structure, uniqueness, authority |

Crawl Budget Analysis

AI bots have crawl budgets like traditional bots. If your site is large:

QUESTIONS TO ANSWER:
1. What % of pages are being crawled by AI bots?
2. Are high-value pages being fetched?
3. Are AI bots wasting time on low-value pages (tag pages, pagination)?
4. What's the crawl frequency — daily?