/audit-plugin

name: audit-plugin
description: Deep quality audit of all skills, agents, and commands for inconsistencies, gaps, duplication, and token waste
argument-hint: "[optional: specific skill/agent/command name or category to focus on]"

Audit plugin content

Deep analysis of all skills, agents, and commands in the whetstone plugin. Surfaces quality issues that degrade skill effectiveness, waste tokens, or confuse the model.

Scope

PLUGIN_DIR=plugins/whetstone
SYNC_LOG=docs/audit/audit-log.md

If `$ARGUMENTS` specifies a name or category, narrow to that. Otherwise audit everything. Normalize component names to the `ia-` prefix before using them as `--skill`/`--component` filters (e.g. `debugging` → `ia-debugging`). Exit code 2 from those commands means the filter matched nothing — almost always a missing prefix — not a validation failure; re-run with the prefixed name.

**Reactive mode:** If invoked after a skill/agent failed during use, detect the failing component from conversation context and focus the audit on that component first.

Phase 0: Pre-flight — read prior decision log

Read `$SYNC_LOG` in full before any checks run. Build an in-memory set of already-evaluated findings keyed by `(component, issue-signature)` across every run entry. Use it as a filter during Phase 3 presentation:

• **Previously applied, exact match** — drop silently.
• **Previously rejected, exact match** — drop silently unless new evidence contradicts the prior reason; if so, surface with a `RE-EVALUATE` flag and quote the prior rejection reason.
• **Previously deferred, exact match** — surface with a `PREVIOUSLY DEFERRED` tag and the original defer reason so the user can judge whether conditions have changed.
• **No match** — present normally.

While reading, also detect prune triggers and emit a one-line reminder at the end of Phase 3 if any fire:

• Any entry older than 30 days.
• Any entry referencing a component that no longer exists under `$PLUGIN_DIR`.
• Any entry whose reasoning has been superseded by a feedback-memory rule.

Reminder format: "Sync log has N prune candidates — run `/prune-sync-log`."

If `$SYNC_LOG` doesn't exist, note it and continue — the post-apply step in Phase 5b creates it.

Phase 1: Mechanical validation (deterministic, no AI)

Run the full deterministic validation pass first. This replaces manual inventory, validation gates, anti-pattern detection, structural checks, and reference validation with a single script:

python3 distillery/scripts/distiller.py validate-plugin
python3 distillery/scripts/distiller.py test-triggers
python3 distillery/scripts/distiller.py budget --check-all   # advisory

`validate-plugin` checks every skill, agent, and command for:

• Frontmatter: exists, valid YAML, no inert fields, name format, description length (<80 tokens), "Use when" trigger phrase
• Anti-patterns: OVER_CONSTRAINED, BLOATED_SKILL, EMPTY_DESCRIPTION, MISSING_TRIGGER, VAGUE_DESCRIPTION, ORPHAN_REFERENCE, DEAD_CROSS_REF, DUPLICATE_TRIGGER, STALE_VERSION_PIN
• Structural: placeholder text, empty sections, missing headings
• Body size: skills >4K, agents >3K, commands >4K tokens
• Reference integrity: orphaned files in references/scripts dirs, backtick references to nonexistent components
• README and hook pattern count accuracy

The output is a structured JSON report with inventory counts and per-component findings sorted by severity. Include all findings in the Phase 3 presentation alongside AI-generated findings.

`test-triggers` runs the regex regression suite. Include any failing skills as HIGH severity.

`budget --check-all` compares each skill's recorded turn-count/tool-variety baseline against current aggregates and surfaces silent skill bloat. Advisory only — do not block on it, but include any flagged skills as MEDIUM findings in the Phase 3 table (it currently surfaces real findings).

Phase 1b: PluginEval scoring

Score each component against these 10 weighted dimensions. Use the weights to prioritize findings -- issues in high-weight dimensions get higher severity.

| Dimension | Weight | What to measure | |-----------|--------|-----------------| | **Triggering accuracy** | 25% | Does the description cause correct activation? Check for missing synonyms, false-positive triggers, description/content mismatch | | **Orchestration fitness** | 20% | Does the component compose well with others? Check cross-references, handoff clarity, scope boundaries | | **Output quality** | 15% | Does the component define what it produces? Check for output format specs, templates, success criteria | | **Scope calibration** | 12% | Does the component stay in its lane? Check for scope creep, overlap with adjacent components | | **Progressive disclosure** | 10% | Does it load only what's needed? Check body size, references/ split, conditional sections | | **Token efficiency** | 6% | Does it waste tokens? Check for "Claude already knows this" content, redundancy, verbose examples | | **Robustness** | 5% | Does it handle edge cases? Check for missing error paths, ambiguous instructions | | **Structural completeness** | 3% | Frontmatter correct? Required sections present? References linked? | | **Code template quality** | 2% | Do bundled scripts work? Are they referenced correctly? | | **Ecosystem coherence** | 2% | Consistent naming, tone, terminology with the rest of the plugin? |

Note: Many structural completeness checks (frontmatter, references) are already covered by Phase 1's `validate-plugin`. Focus PluginEval scoring on the qualitative dimensions that need AI judgment.

Phase 2: Quality checks

Run these checks against every file. Use parallel subagents (model: sonnet) grouped by category to keep context manageable.

Token efficiency

These checks require AI judgment. Inert frontmatter and body size are already covered by `validate-plugin`.

| Check | Signal | |-------|--------| | "Claude already knows this" | Content explaining what a technology is