/seo-indexability

name: seo-indexability
description: Audit a page's indexability and site health — canonical presence/validity (self vs cross-domain vs chain, canonical to redirect/404, and the genuinely lethal canonical+noindex pair where the canonical points elsewhere), robots meta and X-Robots-Tag noindex/nofollow, duplicate clusters, pagination signals, plus redirect chains/loops, 4xx/5xx and soft-404 internal links, mixed content, HTTP-to-HTTPS enforcement, user-agent content divergence (cloaking), orphan pages and click-depth — and generate self-referential canonical / noindex-removal fixes. Module M2 (covers M3 site health). Feeds the Search SEO score.
allowed-tools: Read, Grep, Glob, WebFetch, Bash

seo-indexability (M2)

If a page can't be crawled, indexed, or canonicalized correctly, every other signal is wasted — this is the floor under the Search score. Schema-side context: `references/schema-tier1.md`. AI-bot access and snippet eligibility are M14 (`seo-ai-crawlers`); this module owns the classic-Search side, including the search-axis cloaking check.

Inputs

Work from the PageSnapshot named in your dispatch envelope: read `parsed` from `<run_dir>/pages/<slug>.json` (`canonicals[]`, `robots_meta[]`) plus `status_chain`, `headers`, `robots_directives.effective`, and the crawl graph in `<run_dir>/crawl.json` for orphans, depth, duplicate clusters, and internal link status; `<run_dir>/profile.json` carries `environment.kind` (`production|preview|staging|local`); when the run included a user-agent diff, `<run_dir>/ua-diff.json` holds the per-UA response fields. Grep `pages/<slug>.html` for verbatim evidence; site artifacts live in `<run_dir>/site/{robots.json,sitemaps.json,discovery.json}`. Deterministic findings already emitted by `audit.mjs` are listed in `<run_dir>/findings.deterministic.json` — do not re-emit those ids; add model-judged findings only. If invoked directly with a URL/path and no snapshot exists, first run `node "${CLAUDE_PLUGIN_ROOT}/scripts/snapshot.mjs" <target> --out "${CLAUDE_PLUGIN_DATA}/runs"` and use the printed snapshot path.

Audits

Working from the PageSnapshot (`parsed_rendered` when `render.used` is not `none`, else `parsed`) plus `headers`, `status_chain`, and `robots_directives`: 1. **Canonical**: exactly one `<link rel="canonical">`; absolute HTTPS URL; classify self-referential vs cross-domain vs chained (canonical points to a URL that itself canonicalizes elsewhere). Flag a canonical that resolves to a redirect or a 4xx. 2. **Canonical + noindex — read it correctly.** `noindex` together with a **self-referential** canonical is *not* the lethal pair: it is a coherent, usually deliberate exclusion (thank-you pages, filtered views, staging). Report it as `M2.robots.noindex_present` and ask before touching it. The pair is lethal only when the canonical points **elsewhere**: the page tells Google "index that other URL instead" while also saying "don't index me", and the contradiction can propagate the `noindex` to the canonical target — that is `M2.canonical.noindex_conflict`. 3. **Robots directives**: `<meta name="robots">` and the `X-Robots-Tag` header for `noindex`/`nofollow`/`none`; reconcile header vs meta (the most restrictive applies; a header directive reaches non-HTML responses too). Record the **scope** — a directive emitted by a template or an edge rule is `template`/`site`, one written into a single document is `page`. 4. **Duplicate clusters**: near-identical title/H1/body across URLs with no consolidating canonical. 5. **Pagination**: paginated series signals (self-canonical per page; do not canonicalize page 2+ to page 1 — that delists deep items). 6. **Site health (covers M3)**: redirect chains (>1 hop) and loops, internal links returning 4xx/5xx, soft-404 (200 status on an empty/"not found" page), mixed content (HTTP subresources on an HTTPS page), HTTP-to-HTTPS enforcement, orphan pages (no internal inlinks inside the crawl frontier) and click-depth from the homepage. 7. **User-agent content divergence (cloaking, search axis)**: from `ua-diff.json`, compare what the origin serves a default browser UA against what it serves `googlebot` — status, title, H1, word count, canonical, robots directives, JSON-LD. A material difference is the search-axis cloaking signal (`M2.cloaking.ua_content_divergence`). The AI-bot side of the same diff (a challenged `GPTBot`, a refused spoofed `Googlebot`) belongs to M14. 8. **Environment**: read `profile.json.environment.kind`. On `preview`/`staging`/`local`, a `noindex` or a blanket robots block is the **expected** configuration, not a defect — record `M2.hosting.preview_environment_audited` so the report says which host was audited.

Fixes

• **AUTO** (`fixable: auto`): inject a single self-referential absolute-HTTPS `<link rel="canonical">` when absent. Deterministic, additive, verifiable diff for `fix`.
• **PROPOSED** (`fixable: proposed`): removing a `noindex` the user has **confirmed** is unintended; duplicate-cluster consolidation (which URL should win is an editorial call); repointing or removing broken internal links.
• **ADVISORY** (`fixable: advisory`): redirect chains/loops, status codes, HTTP-to-HTTPS enforcement, mixed-content origins, and any edge rule behind a UA divergence — these live in server/CDN config, so the tool reports the exact change and writes nothing.
• Never fabricate which URL "should" win, whether a page is intentionally noindexed, or the intent behind a UA-conditional response — ask the user or leave a clearly-marked TODO placeholder.

Verification

• `dom_assert`: parse the DOM for canonical/robots presence and value.
• `header_check`: read `X-Robots-Tag` and follow the redirect chain (status + `Location` per hop, loop detection).
• `link_graph`: internal link status, orphans, and click-depth from `<run_dir>/crawl.json`.
• `ua_diff`: `node "${CLAUDE_PLUGIN_ROOT}/scripts/ua-diff.mjs" --url <final_url> --ua default,googlebot` — the two responses side by side.
• When