/night-market-diagnostics-toolkit

name: night-market-diagnostics-toolkit
description: Run and interpret repo diagnostic scripts (ratchets, validators, token stats). Use when measuring health. Do not use to run tests; use night-market-operations.

Night Market Diagnostics Toolkit

Measure instead of eyeballing. Every claim about the health of this repo ("the skill graph is clean", "descriptions fit the budget", "no compromised dependencies") has a script that produces a number or a PASS/FAIL. This skill catalogs those scripts, their exact invocations, and how to read their output.

Conventions for everything below:

• Run all commands from the repo root

(`/home/alext/claude-night-market`).

• Plain `python3` is enough for every script in the table except

mutation testing, which needs `uv`.

• All checks here are read-only. None of them mutate the repo.

The ratchet-baseline concept

Several checks are ratchets. A ratchet check counts a category of existing debt (for example, SKILL.md files missing an Exit Criteria section), compares the live count against a frozen number in a baseline JSON file, and:

• passes while the live count is at or below the baseline

(pre-existing debt is tolerated),

• fails the moment a NEW violation pushes the count above the

baseline (debt may not grow),

• nudges you to lower the baseline number when the live count drops

(locking in the win so the debt cannot silently return).

The goal is a baseline that only shrinks, eventually to zero. The two ratchet baselines live at `scripts/skill_graph_baseline.json` (keys `max_dangling_bugs`, `max_uncalled_libraries`) and `scripts/skill_exit_criteria_baseline.json` (key `max_missing_exit_criteria`). When a ratchet prints a "dropped to N (baseline M). Lower ... to lock the win." line, edit the baseline JSON down to N in the same PR.

Tool table

Jargon used in the table: "dangling Skill() ref" means a `Skill(plugin:name)` reference in a skill/command/agent file whose target skill does not exist. "Uncalled library" means a library-role skill no other skill invokes (the risk targeted by `.claude/rules/shared-utility-consumer-rule.md`).

| Tool | Invocation | What it measures | How to interpret | When to run | |------|------------|------------------|------------------|-------------| | Plugin structure validator | `python3 plugins/abstract/scripts/validate_plugin.py plugins/<name>` | plugin.json validity, kebab-case naming, directory layout | Exit 0 with "Plugin validation passed". Any FAIL line names the broken file | Before committing plugin changes. `make validate-all` loops it over every plugin | | Skill-graph drift ratchet | `python3 scripts/check_skill_graph_drift.py` | Dangling Skill() refs and uncalled libraries vs `skill_graph_baseline.json` | Exit 0 at/below baseline. Nonzero exit lists the new dangling refs. "Lower ... to lock the win" means shrink the baseline | After adding, renaming, or deleting skills or Skill() refs | | Exit-criteria drift ratchet | `python3 scripts/check_skill_exit_criteria_drift.py` | SKILL.md files under plugins/ missing an `## Exit Criteria` heading vs `skill_exit_criteria_baseline.json` | Same ratchet semantics as above | After adding or editing any SKILL.md | | Pinned-version checker | `python3 scripts/check_pinned_versions.py` | GitHub-sourced tool pins vs the latest upstream release (needs network) | "N pin(s) current" is healthy. A "holding X at ..." line documents an intentional hold with its reason (for example bandit 1.8.6 for the Python 3.9 hook floor) | In pre-commit, and whenever a CI tool-setup step breaks | | Capabilities sync | `bash scripts/capabilities-sync-check.sh` | plugin.json registrations vs the generated capabilities reference in book/src/ | "PASSED: All capabilities are in sync" plus counts. On drift, run `/sanctum:sync-capabilities --fix` | After changing any skill, command, or agent registration | | Supply-chain scan | `python3 scripts/supply_chain_scan.py` | Lockfiles vs the known-compromised-versions blocklist, plus known malicious artifacts | Two `[OK]` lines is clean. Any hit must be resolved before release | When adding dependencies, during incidents, before releases | | Token stats | `python3 plugins/abstract/scripts/context_optimizer.py stats plugins/ --format json` | Bytes and estimated tokens per SKILL.md, bucketed small/medium/large | Skills over ~5,000 estimated tokens breach the quality-gates file limit and are split candidates | When a skill feels bloated, before modularizing | | Description budget | `python3 plugins/abstract/scripts/validate_budget.py` | Sum of all skill/command description characters vs the 90,000-char ecosystem ceiling (ADR-0004, 160 chars per description) | Prints used/ceiling and headroom. Exit 0 means within budget | After editing any frontmatter description | | skrills validate | `skrills validate --skill-dir plugins --target claude` | Skill frontmatter validity per target framework (Claude, Codex, Copilot) | "Validated N skills: ..." then an `Errors (n):` list with file:line. Fix every listed error | Skill audits. `make validate-skills` falls back to `uv run python scripts/check_plugin_hooks.py` when skrills is absent | | skrills analyze | `skrills analyze --skill-dir plugins` | Skill token usage and dependency structure | Large-token outliers are split candidates | Budget planning. `make analyze-skills` falls back to `scripts/generate_dependency_map.py` | | Mutation testing | `cd plugins/<name> && uv run mutmut run --paths-to-mutate=scripts/,src/ --tests-dir=tests/` | Whether the test suite kills injected code mutations (surviving mutants = untested behavior) | Exit 0 = no survivors. Exit 2 = survivors found (CI treats this as pass-with-report). Any other exit = crash, investigate | Weekly CI (Sunday) or manually before hardening a test suite | | Markdown link checker | `python3 scripts/check-markdown-links.py [file.md ...]` | Broken relative links and anchors. No args = scan the whole repo | Exit 1 prints each broken link as `file: l