/skeptical-triage

name: skeptical-triage
description: Reusable 3-round self-challenge + arbiter pattern for filtering false positives from findings/verdicts. Use when the cost of a false-positive gate block exceeds the cost of ~4 extra LLM turns.
when_to_use: |
  Apply skeptical triage when:
  - A finding could block a gate (gate:code, gate:ship, gate:qa, gate:arch) and flipping it wrongly wastes CTO time
  - A verdict is about to be written to a report that downstream agents will trust (CSO, QA, ADR)
  - Multiple signals disagree (one reviewer says VALID, another says INVALID) — arbiter resolves cleanly
  Do NOT apply to:
  - P2 findings or advisory notes (cost > benefit)
  - Hard findings (secrets in source/git, confirmed CVEs, failing tests) — these are facts, not judgments
  - Quick factual lookups ("does this file exist?", "what version is pinned?")
effort: medium
allowed-tools: Read, Grep, Bash, Glob
paths:
  - "docs/**"
  - "src/**"
  - "lib/**"
  - "app/**"

Skeptical Triage

Filter false positives from multi-angle review, security audit, QA regression flags, or any high-stakes judgment before it turns into a blocker.

Three rounds of skeptical self-review + an impartial arbiter, with a confidence score from the vote.

When to invoke

| Caller | Finding type | Apply triage? | |--------|--------------|---------------| | `/review` | Angle 2/4/7/9 P0/P1 (security, SQL, privacy, concurrency) | Yes | | `/review --deep` | Any angle P0/P1 | Yes | | `security-officer` | CSO audit P0/P1 | Yes | | `security-officer` | Secret in source/git, confirmed CVE | **No** — hard finding | | `qa-engineer` | Flaky-test verdict (is this a regression or flake?) | Yes | | `architect` | ADR trade-off dispute (option A vs. B when both look reasonable) | Yes | | Any | P2/advisory | No |

The 4-step pattern

Run these sequentially. Each round sees prior reasoning. Arbiter sees all rounds.

Round 1 — Reachability / Premise

Question: **is the premise true?**

• For security/reliability: can an external attacker reach this code path with untrusted input? Trace input flow backward from the bug site to its origin. If only trusted internal callers → lean INVALID.
• For regressions: does the failing behavior reproduce from a clean state on the target branch?
• For ADR trade-offs: is the constraint that forces the choice actually binding? (e.g. "we need <10ms p99" — is that real or aspirational?)

Output: `{round: 1, verdict: VALID|INVALID|UNCERTAIN, reasoning: "...", crux: "single key fact"}`

Round 2 — Verify cited defenses / counter-evidence

Question: **are claimed defenses real and sufficient?**

• Every cited defense → use `Grep` to find its actual implementation line.
• Resolve constant names to numeric values. `MAX_BUF_SIZE` is not a verified bound — `#define MAX_BUF_SIZE 64` is.
• For regressions: is the cited "test covers this" actually asserting the right invariant?
• For ADR: is the cited benchmark/precedent real (grep for it, read it), or rumored?

If you cannot point to the line that enforces the defense, **it does not exist.**

Output: same JSON shape, with `grep_used: true/false`.

Round 3 — Missed angles

Question: **what did Rounds 1-2 not consider?**

• Error paths, integer overflow, race windows, different callers, platform differences
• Do NOT rehash prior rounds — add new evidence or concede
• For QA: retry logic masking the failure? Test pollution from another test?
• For ADR: option C that neither reviewer raised?

Output: same JSON shape.

Arbiter

Input: all 3 rounds + original finding/question + source code.

Question: **final call — which side has the stronger evidence?**

• Deliver single `verdict: VALID|INVALID` (no UNCERTAIN — make the call).
• Deliver one-sentence `crux` — the key fact the verdict turns on.
• If 3 prior rounds all said the same thing, only override with overwhelming new evidence and explain why.

Output:

{
  "verdict": "VALID",
  "crux": "memcpy at auth.c:142 copies network-controlled len bytes into 64-byte stack buffer with no bound check",
  "reasoning": "Rounds 1 and 3 verified attacker reach; Round 2 found no size check in 50 LOC radius; arbiter confirms no caller clamps len."
}

Hard rules

Burn these into every round's prompt:

1. **Absence of defense → VALID, not UNCERTAIN.** If you searched for a defense and did not find one, that is the answer. "Other code probably handles this" is not a valid defense. 2. **A constant name is not a verified bound — only its resolved value is.** Grep for the `#define` / `const` declaration. 3. **Name the line or it does not exist.** Vague references to "assumptions in this codebase" do not count. 4. **Do not contradict your own conclusion in the same response.** If you verified a defense is insufficient, that is the verdict. Stop searching for reasons to flip. 5. **Code quality issue ≠ security vulnerability.** Data race on diagnostic state, NULL check on internal-only API, UB only in debug builds → INVALID. 6. **Trust your own reasoning.** If you see the crux on first read, don't manufacture a counter-argument.

Confidence scoring

confidence = valid_rounds_before_arbiter / 3

• `100%` (VVV) — 3/3 rounds VALID. Arbiter rubber-stamps unless it finds something brand-new.
• `67%` (VVI or VIV or IVV) — majority VALID. Arbiter breaks tie with new evidence.
• `33%` (IIV or IVI or VII) — majority INVALID. Arbiter usually confirms INVALID.
• `0%` (III) — 3/3 INVALID. Arbiter rarely overrides.

Arbiter **overrides** the final verdict; confidence reflects the round vote for transparency. Record both in the output so humans can see where the arbiter diverged.

Applying triage results to severity

Once the arbiter returns:

| Arbiter verdict | Confidence | Severity action | |-----------------|------------|-----------------| | `VALID` | ≥ 50% | Keep original severity | | `VALID` | < 50% | Demote: P0→P1, P1→P2 | | `INVALID` | any | Remove from gate tally, record as `[FILTERED]` in report for audit |