/verifying-review-findings

name: verifying-review-findings
description: "Checks each finding from a code review against the code it cites and keeps only the ones that hold. Drops findings whose quoted code is not in the file, then asks TypeSafe's Jev decision model whether the code shown really has the claimed defect, returning P(confirmed), P(refuted), a severity distribution and whether the proposed fix works. Use for \"verify these review findings\", \"filter false positives from the review\", \"is this bug real\", \"double-check the reviewer's claims\", \"get a second opinion on these findings\", a pre-commit or CI gate that blocks only on confirmed high-severity findings, or a finder-then-verifier review with a different model as the verifier."
metadata:
  version: 0.1.1

Verifying review findings

A model that reviews code for defects also invents them: it misreads a guard ten lines up, a loop bound, or a lock the caller holds. This skill splits the review into two roles, as the open-source reviewer bluesky-social/roast does. A **finder** (you, the `code-review` skill, any reviewer) writes each finding with a verbatim quote of the code it is about. A **verifier** from a different model family then reads that code and confirms or refutes the claim.

The verifier is Jev, a decision model from TypeSafe. It answers typed questions (yes/no, multiple choice, ordered scale) with a probability for every answer in about 0.3 s. Jev is not a Claude model, so when Claude finds and Jev verifies, the verifier is less likely to share the finder's blind spots. Below, P(x) means the probability Jev gives to answer x.

`scripts/verify.py` does the following for each finding:

1. **Grounding.** The quote must occur in the file, compared with runs of whitespace collapsed. A finding with no quote, or a quote that is not in the file, is dropped before Jev sees it. 2. **The window.** The verifier sees a *window* of the file: the top-level block (function, class, statement) that contains the quote, with at least 12 lines on each side, capped at 160 lines. Lines are numbered and the quoted ones are marked `>`. With `--diff`, the file's diff hunks go along too. 3. **Three questions to Jev**, one call per finding, in parallel:

• verdict: `confirmed`, `refuted` or `insufficient_context`
• severity if the defect is real, on the scale `info`, `low`, `medium`,

`high`, `critical`

• when the finding proposes a fix, whether the fix works

4. **Thresholds.** P(refuted) of at least 0.7 drops the finding (`--refute-at`). P(confirmed) of at least 0.7 confirms it (`--confirm-at`). Anything else is `uncertain`. Only a refutation removes a finding; a verifier that is unsure keeps it.

When NOT to use this skill

| Situation | Use instead | |---|---| | Nobody has looked for defects yet | `code-review` (or any reviewer) first; this skill only judges findings that exist | | Checking that a README or docstring still matches the code | `verifying-claims` | | An adversarial read of a whole deliverable (prose, analysis, a PR's design) | `challenging` | | A routing, triage or yes/no question that is not about a code finding | `deciding-with-confidence`, or the Muninn `decide` tool directly | | Building a deterministic check that must be able to go red | `gating` | | The claim is about runtime behaviour the code cannot show (performance, a flaky test, production data) | run it; a verifier reading code can say `insufficient_context` at best |

Stop the run when most findings come back `uncertain` with reason `needs_more_context`: the evidence for those claims lives outside their windows (in a caller, another file, a config). Verify those findings by reading the code yourself.

Procedure

Throughout, `V=/mnt/skills/user/verifying-review-findings/scripts/verify.py`.

1. **Write the findings as a JSON list.** `file` (relative to the repository root) and `code_quote` are required. Copy the quote character for character from the file, one to six lines; a paraphrase is dropped.

   [{"id": "1", "file": "src/util.py", "line": 6,
     "description": "Off-by-one: the loop skips the last element.",
     "code_quote": "for i in range(len(items) - 1):",
     "fix": "for i in range(len(items)):", "severity": "high"}]

The other fields are optional. `id` defaults to the item's position. `line` picks between several matches of the same quote. `description` is the claim Jev judges. `fix` adds the fix question. `severity` is the finder's own estimate and is passed to Jev as a claim. Findings in the `ReportFindings` shape (`summary` and `failure_scenario` instead of `description`) are accepted too. When you are the finder, emit this shape directly.

2. **Pick the path.** `--transport auto`, the default, uses the first of `jev` and `gateway` whose credentials are set.

| Path | Needs | How | |---|---|---| | Jev direct (`jev`) | `TYPESAFE_API_KEY`, a TypeSafe API key | `run` (step 3a) | | Jev via Cloudflare Workers AI (`gateway`) | `CF_ACCOUNT_ID` and `CF_API_TOKEN`, optionally `CF_GATEWAY_ID` | `run`; pass `--transport gateway` when both paths have credentials | | Muninn `decide` tool, which calls Jev | the Muninn MCP connector in the session | `emit`, then `apply` (step 3b) | | Claude Haiku via `deciding-with-confidence` | that skill installed next to this one | `run --transport decider`; the verifier is then also Claude |

3. **Run it.**

a. With credentials in the environment:

   python3 $V run findings.json --repo . --format md      # default output is JSON
   git diff main... > /tmp/d.diff && python3 $V run findings.json --diff /tmp/d.diff
   python3 $V run findings.json --ref HEAD~1               # read files at a git ref

b. Through the Muninn `decide` tool:

   python3 $V emit findings.json --out batch.json

`emit` saves the grounded findings to `batch.json` and prints the Jev requests, one per surviving