/seo-ai-crawlers

name: seo-ai-crawlers
description: Audit AI crawler access and Google AI-feature eligibility for a page — check the snippet controls that actually gate AI Overviews and AI Mode (noindex, nosnippet, max-snippet, data-nosnippet), resolve per-bot effective access from robots.txt for the audited path (training vs retrieval vs user-fetch tokens), read Content-Signal posture, diff what different user-agents are served, confirm non-JS crawlers can read the page, and generate a choice-gated robots.txt preset. Module M14. Feeds the AI Visibility score.
allowed-tools: Read, Grep, Glob, WebFetch, Bash

<!-- Portions Copyright (c) 2026 Enrique Rocha (@soyenriquerocha) - see LICENSE for original authors · tododeia.com · id er-d03758e1 -->

seo-ai-crawlers (M14)

Two separate questions live in this module, and only one of them has vendor documentation behind it:

1. **Google's AI features.** A page can appear in AI Overviews / AI Mode only if it is indexed and eligible to be shown in Google Search **with a snippet**. Those gates are documented, so `M14.ai_eligibility.*` carries the only `established` findings here. 2. **Third-party AI crawlers.** Who may fetch and cite the page — OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Mistral, DuckDuckGo, Cohere. Access is documented per user-agent; the citation *effect* is not, so those findings stay `directional`.

The training-vs-retrieval-vs-user-fetch distinction decides every robots recommendation. Reference: `references/ai-crawlers.md` (UA table with sources, Content-Signal, eligibility gates, presets). AI discovery files (llms.txt, agents.md, `/.well-known/ucp`, agentic sitemap) are module M21 — see `seo-ai-discovery`; agentic-commerce readiness is M18 (`seo-ecommerce`).

Inputs

Work from the PageSnapshot named in your dispatch envelope: read `parsed` from `<run_dir>/pages/<slug>.json` plus `robots_directives`, `headers` (`x-robots-tag`), and `render{needed,used,delta}`; Grep `pages/<slug>.html` for verbatim evidence. `<run_dir>/site/robots.json` carries, for the **audited path**, `verdicts`, `effective_access{[bot]:{class, fetch, url_allowed, via, rule, signals, doc_url, robots_reliability}}`, `content_signals{present, placement, global, by_group[], malformed[], cloudflare_managed_hint}`, `groups`, and `ua_table_version`. When the run included a user-agent diff, `<run_dir>/ua-diff.json` holds per-UA status/title/h1/word_count/canonical/robots/jsonld. Deterministic findings already emitted by `audit.mjs` are listed in `<run_dir>/findings.deterministic.json` — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run `node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs"` and use the printed snapshot path.

Audits

Working from the PageSnapshot (`parsed_rendered` when `render.used` is not `none`, else `parsed`) plus `site/robots.json`:

1. **Google AI-feature eligibility (the established gate).** Collect **every** `<meta name="robots">` / `<meta name="googlebot">` tag *and* every `X-Robots-Tag` header value (header and meta both count; the most restrictive wins). Read `noindex`/`none`, `nosnippet`, `max-snippet:N`, and the `data-nosnippet` attributes — for `data-nosnippet`, measure how much of the main content it covers and specifically whether it wraps the H1 or the lead passage. Also confirm Googlebot itself is not disallowed for this path. Record the **scope**: a directive coming from a template or a site-wide header is `scope: template|site`; one hand-written into a single document is `scope: page`. 2. **Effective per-bot access.** Read `effective_access` for the audited path — not for `/`. A site can allow `/` and disallow `/blog/`, so root-level posture is not an answer. Check the retrieval/citation bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Amazonbot`, `DuckAssistBot`, `Google-CloudVertexBot`), the classic engines (`Googlebot`, `Bingbot`, `Applebot`), the training tokens (`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`, `Meta-ExternalAgent`, `cohere-ai`, `Bytespider`), the user-fetch tokens (`ChatGPT-User`, `Claude-User`, `Claude-Web`, `Perplexity-User`, `Meta-ExternalFetcher`, `MistralAI-User`), and `OAI-AdsBot`. Current OpenAI tokens ship as `GPTBot/1.4` and `OAI-SearchBot/1.4`; match user-agents case-insensitively on the token, never on the full UA string. Anthropic publishes its crawler IP ranges at `claude.com/crawling/bots.json`. Quote `ua_table_version` in the report so a stale table is visible. 3. **Content-Signal posture.** `content_signals` reports the `search` / `ai-input` / `ai-train` keys, whether the line is global (Cloudflare's managed robots.txt appends it after all groups) or inside a `User-agent` group, and any malformed pairs. A group line overrides the global line for that group. **A `Disallow` always wins over a signal**: `Disallow: /` blocks the bot whatever `ai-train` says, and `Allow` + `ai-input=no` means the page is still fetchable while consent is withheld — say exactly that, never "blocked". 4. **User-agent divergence.** Compare what the origin serves to `default`, `googlebot`, `gptbot`, `oai-searchbot`, and `claude-searchbot`: differing status, title, H1, word count, canonical, robots directives, or JSON-LD. Separate a *bot challenge* (403/429/503, interstitial, CAPTCHA markup, near-zero word count) from a genuine content difference. The search-axis cloaking judgment — Googlebot vs a browser being served different content — belongs to **M2** (`M2.cloaking.ua_content_divergence`); M14 owns only the AI-bot access side. 5. **Renderability for non-JS crawlers.** Use the M4 `render` block. If the H1, the primary body, or the JSON-LD exists only in `rendered_html_path` and is absent from `raw_html_path`, the page is not reliably readable by AI crawlers that do not execute JavaScript. 6. **Google-Extended sanity.** If `Google-Extended` is disallowed, state p