/verifying-skill-accuracy

name: verifying-skill-accuracy
description: Fact-check LLM-drafted technical content against primary sources — source hierarchy, programmatic existence probes for tool and plugin names, class-before-instance error triage, the truncated-negative trap, adversarial re-checking, and the verified-stamp discipline. Use when auditing a SKILL.md for factual errors, before stamping a skill verified, when a command or artifact claim needs confirming against upstream documentation, or when reviewing any drafted content whose specifics were written from model memory.

Verifying Skill Accuracy

Skills in this repo are drafted with LLM assistance. The methodology in them is usually sound; the **specifics are not trustworthy until checked**. The measured rate from the first verification pass was **31 factual errors across 11 of 12 skills — about 2.6 per skill**, with a single skill clean on both passes. Assume that rate applies to anything unstamped.

This skill is the procedure for driving that rate down, and for knowing when you are allowed to say a skill has been verified.

When to Use

• Auditing an existing SKILL.md against primary sources
• Before adding or renewing a `verified:` frontmatter stamp
• After drafting new skill content, on the specifics you just wrote
• When a reader reports that a command, ID, or field name does not work
• Reviewing any technical content where the author was a model

When NOT to Use

• Writing a new skill from scratch — use `authoring-security-skills`, then

verify with this skill as the final gate

• Structural or style problems (frontmatter, sections, line count) — that is

`python3 scripts/validate.py --strict`, which checks *form only*

• Deciding whether a technique is a good idea — that is judgment, not fact

The Core Distinction: Form vs Truth

The repo's CI is a closed loop. `validate.py`, `sync_attack.py`, and `run_evals.py` check frontmatter shape, cross-reference integrity, ATT&CK-index consistency, and routing against **self-authored** eval cases. Every one of those can pass on a skill whose commands do not exist.

> Green CI means the skill is well-formed. It says nothing about whether it is > true. Never cite a passing test run as evidence of accuracy.

Sweep by Class Across the Whole Collection

Once you identify an error class, **check it everywhere before moving on** — not just in the skill that surfaced it. Errors of a class cluster, and a per-skill pass leaves the collection internally inconsistent.

Two cases from this repo make the point. The legacy Sigma date format was fixed in `writing-sigma-rules`, then turned up again in `engineering-detections` — in a skill that had already been stamped, because the per-skill inventory did not include a class fixed elsewhere. And `attacking-entra-id` was still calling retired AzureAD PowerShell cmdlets while `investigating-m365-entra` already used the modern Graph equivalents: the collection contradicted itself, and only a cross-cutting grep showed it.

So: fix the instance, then immediately grep every skill for the pattern, and say in the commit which classes you swept. Classes worth sweeping in a security collection — tool renames and archived projects, CVE IDs, cloud metadata endpoints, removed API versions, default ports, spec version claims, and any behaviour gated on a platform version.

Triage: Class Errors Before Instance Errors

Do not start by checking facts one at a time. First ask: **is there a single systematic claim whose failure invalidates everything below it?**

In `analyzing-memory-images`, the skill invoked Volatility as `volatility3`. The real entry point is `vol`. That one error made **all 40 commands in the file uncopyable** — far more damage than the two nonexistent plugins found afterwards, and fixable with one substitution.

Check in this order:

1. **Tool identity** — the binary/entry-point name, and the major version the syntax belongs to. Volatility 2 and 3 share almost no command surface. 2. **Naming scheme** — how the tool names its plugins, modules, or subcommands. Getting the scheme right validates or invalidates dozens of lines at once. 3. **Path and file conventions** — where artifacts actually live on the version in question. 4. **Individual claims** — only now, one by one.

A class error is cheap to fix and expensive to miss. An instance error is the reverse.

Source Hierarchy

Use the highest tier available, and record which tier you used.

| Tier | Source | Use for | | --- | --- | --- | | 1 | Official docs for the **specific version**, upstream source, RFCs, the vendor's own reference | Command syntax, plugin names, API shapes, protocol details | | 2 | Vendor KB, release notes, changelogs, official blog announcing a change | Defaults, retention windows, licence gates, deprecations | | 3 | Maintainer-authored write-ups, conference material by the tool author | Intent, gotchas, why a thing behaves as it does | | 4 | Community posts, tutorials, Stack Overflow | Leads to verify at tier 1 — **never** as the citation | | — | Model memory | Nothing. Not one claim. |

Two rules on top of the table:

• **Match the version.** Docs for 2.x do not establish 3.x behaviour. Prefer

`latest` or `stable` doc branches, and note when the skill targets an older pinned release.

• **Ask when it last changed**, not just whether it is true. Vendor defaults

move: Microsoft's Audit (Standard) retention went 90 → 180 days in October 2023, and material written before that is now wrong without being obviously wrong.

Never Assert These From Memory

Each of these has produced a real error in this repo:

• CLI entry points and subcommand names
• Plugin, module, and package names
• Version numbers, and any "the default is X" claim
• Event IDs and their exact semantics (4778 is session *reconnect*, not connect)
• Log field and column names
• Retention windows and licence gates
• CVE IDs, CVSS vectors, and affected-version ranges
• API paths, parameter