/verify-cli-run

name: verify-cli-run
description: Prove that a command-line tool actually did what it said, instead of trusting its exit code and its output. Snapshots the filesystem before and after, names the expected consequence in advance, and returns one of four verdicts with what could not be seen. Use after running a build, a migration, a scaffolder, a formatter, a codegen step, or an AI coding CLI; when a command printed success and you are about to report "done"; when you are about to write "exit code 0, so it worked"; or when a tool claims it edited files. Needs nothing installed.
license: Apache-2.0
metadata:
  version: 3.7.0
  homepage: https://www.reticle.sh
  repository: https://github.com/reticlehq/reticle

Prove the command did what it said

A command exited 0 and printed `✓ done`. You have learned that the tool reached its own success branch. You have not learned that anything happened.

This is the whole problem with verifying a CLI: **the two things everybody checks are both the tool describing itself.** The exit code is chosen by the same code that did the work. The output is written by it. When a tool is wrong about what it did, it is wrong on both, in agreement. That is exactly why their agreement proves nothing.

This skill needs no tools installed. It is four rules and some `git`.

The rule that decides everything

> **Evidence for a consequence must not come from the thing that performed the action.**

Grade every fact before you use it:

| What you have | Grade | Can it prove the command worked? | | --- | --- | --- | | Files on disk, before vs after | **consequence** | **Yes.** The filesystem decided whether the write landed, not the tool | | An API answering when you ask it afterwards | **consequence** | **Yes.** Another party, answering you rather than the tool | | Exit code | presence | No. The tool chose it | | stdout / stderr | context | No. The tool wrote it | | The tool's own `--verbose` report | context | No |

Everything in the bottom three rows is real information and none of it is proof. Use it to explain a verdict, never to reach one.

1. Name the consequence BEFORE you run

This is the method. An expectation written after you see the output can be talked into agreeing with whatever happened; one written in advance can only be met or missed.

Write it down in one line, in the transcript, before the command:

EXPECT: dist/index.js is rewritten, and no file outside dist/ changes.
EXPECT: src/utils.ts gains a function called parseConfig.
EXPECT: the migration creates 3 files under migrations/ and nothing else.

A consequence you cannot state in advance is one you cannot verify. If you cannot name one, say so and stop. That is an honest `no-fault`, not a pass.

2. Snapshot before

**In a git repo.** This is the good case, and it is one line:

git status --porcelain > /tmp/before.txt

`git status --porcelain` is a near-perfect evidence channel: structured, cheap, and independent of the tool. Git's index records what landed on disk, not what the tool meant to do.

**Not in a git repo**, or the paths are outside it:

find <declared-root> -type f -newermt '1970-01-01' -exec shasum -a 256 {} + | sort > /tmp/before.txt

Three rules for the roots you declare:

• **Exclude `node_modules`, `.git/objects` and large build caches** unless they are the subject. Hashing a full `node_modules` is tens of thousands of files.
• **Declare them explicitly.** A tool can write anywhere; you are watching a list you chose.
• **Anything outside that list is a blind spot**, and you will report it in step 6 rather than pretend it did not happen.

3. Check whether it is already true

The commonest false green in CLI work: `dist/index.js` exists, so "the build produced `dist/index.js`" passes, on a no-op rebuild that did nothing at all.

Before running, check whether your expected consequence **already holds**. If it does, the command cannot prove it. Either pick a consequence the command changes, or delete the artifact first and say that you did.

4. Run it, and capture separately

<the command> > /tmp/out.txt 2> /tmp/err.txt; echo "exit=$?"

Keep the exit code. You are not going to use it as proof. You are going to use it to explain the verdict, and to notice when it disagrees with the filesystem.

Two things worth knowing:

• Exit codes wrap: `sh -c 'exit 256'` exits **0**. A large code is not always what you think.
• `130` is a `SIGINT`, `137` is usually a kill. Those are facts about how the run _ended_, not about whether it worked.
• Some tools exit 0 when they declined to do anything. AI coding CLIs do this routinely.

5. Diff, and read the diff as the evidence

git status --porcelain > /tmp/after.txt; diff /tmp/before.txt /tmp/after.txt
git diff -- <the paths you expected to change>

Now answer your step-1 expectation against **this** and nothing else.

Two checks worth making every time, because they cost nothing:

• **Nothing else changed.** A build that also rewrote a lockfile, or a formatter that reformatted a file nobody asked about, is a finding.
• **Content, not just existence.** A file that exists is weaker than a file that contains what you expected. But note the asymmetry: _whether_ a file appeared is decided by the filesystem; _what is inside it_ was written entirely by the tool. Existence is stronger evidence than content.

6. Reach a verdict, one of four

Not two. The two extra values are the point: they are the ones that stop ignorance being rounded towards good news.

| Verdict | When | | --- | --- | | **yes** | The consequence you named in step 1 is visible in the diff, and it was not already true | | **no** | The diff contradicts it, or the tool claimed something the filesystem does not show | | **unknown** | You could not see what you needed. Nothing was watching, the window was wrong, or the effect is somewhere you were not looking | | **no-fault** | Everything was watched, nothing was w