/phase-2-metrics

user-invocable: false
install: extras

<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios --> <!-- SPDX-License-Identifier: AGPL-3.0-or-later -->

/pipeline-health — Phase 2: Collect Pipeline Metrics

Phase 2: Collect Pipeline Metrics

Run ALL of these in parallel (note: Phase 2I fetches trajectory comments per issue — run it concurrently with the others but expect it to take longer for large windows; Phase 2K queries persisted audit-agents summaries from Forge's tracking issue and is independent of the target project; Phase 2L queries `audit-finding` labeled issues in the Forge repo and is independent of the target project; Phase 2M scans `~/.claude/history.jsonl` and maps session IDs to JSONL files on disk — it is independent of the target project and completes in <5s for 25K entries; Phase 2N reads local JSONL transcript files using the JSONL Parser Utility and aggregates pipeline flow metrics — it requires Phase 2M to have run first and depends on `SESSION_JSONL_FILES`; Phase 2O extracts per-session structured metrics (phase detection, model distribution, tool call counts, agent spawns) from the same JSONL files — it requires Phase 2N to have run first and reads `TRANSCRIPT_DATA_AVAILABLE` set by Phase 2N; Phase 2P analyzes subagent JSONL files per session, maps each agent to its pipeline role, and computes per-role efficiency metrics — it requires Phase 2O to have run and depends on `TRANSCRIPT_DATA_AVAILABLE` and `SESSION_JSONL_FILES`; Phase 2Q aggregates per-session metrics from Phase 2O into a privacy-safe, report-ready summary — it strips session IDs and content fields, computes distributions, and emits guardrail footnotes — it requires Phase 2O to have run and reads `PER_SESSION_METRICS_RAW`):

2A: Forge changelog (what changed in the pipeline itself)

cd "$FORGE_HOME"
git log --oneline --since="$SINCE" -- commands/ docs/

Record each commit — these are the independent variables. Each prompt change is a hypothesis: "this change should improve X."

2B: Review finding metrics (the primary quality signal)

# Total review findings created in window
gh issue list -R $REPO --state all --label "review-finding" --json number,createdAt,labels,title,state \
  --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"

# Breakdown by severity (from title/labels)
gh issue list -R $REPO --state all --label "review-finding" --limit 500 --json number,title,labels,state,createdAt \
  --jq "[.[] | select(.createdAt > \"$SINCE\")]"

**Categorize each finding** by its agent prefix (SEC, AUTH, BILL, CONC, SCRP, FE, API, DB, INFRA) extracted from the title or finding tags.

2C: Invalidation rate (false positive signal)

# Review findings that were closed as invalid
gh issue list -R $REPO --state closed --label "review-finding,workflow:invalid" --json number,createdAt \
  --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"

# Total review findings closed in window
gh issue list -R $REPO --state closed --label "review-finding" --json number,createdAt \
  --jq "[.[] | select(.createdAt > \"$SINCE\")] | length"

**False positive rate (proxy)** = invalid / total closed. Target: < 10%.

**Note**: This proxy uses `workflow:invalid` as a stand-in for false-positive. Phase 2C.5 reports the actual false-positive rate from the `false-positive` label. Both metrics should be reported; they should converge as validation coverage increases.

2C.5: Validation coverage (finding lifecycle health) <!-- Added: forge#1730 -->

Measures whether the `needs-validation → validated/false-positive` lifecycle is operating. This is the primary health signal for the review system's calibration corpus.

# All finding-issue types: review-finding, staging-review, audit-finding
FINDING_LABELS="review-finding staging-review audit-finding"

TOTAL_FINDINGS=0
TOTAL_VALIDATED=0
TOTAL_FALSE_POSITIVE=0
TOTAL_NEEDS_VALIDATION=0

for LABEL in $FINDING_LABELS; do
  COUNT=$(gh issue list -R $REPO --state all --label "$LABEL" --limit 1000 --json number \
    --jq 'length' 2>/dev/null || echo 0)
  TOTAL_FINDINGS=$((TOTAL_FINDINGS + COUNT))

  VAL=$(gh issue list -R $REPO --state all --label "$LABEL,validated" --limit 1000 --json number \
    --jq 'length' 2>/dev/null || echo 0)
  TOTAL_VALIDATED=$((TOTAL_VALIDATED + VAL))

  FP=$(gh issue list -R $REPO --state all --label "$LABEL,false-positive" --limit 1000 --json number \
    --jq 'length' 2>/dev/null || echo 0)
  TOTAL_FALSE_POSITIVE=$((TOTAL_FALSE_POSITIVE + FP))

  NV=$(gh issue list -R $REPO --state all --label "$LABEL,needs-validation" --limit 1000 --json number \
    --jq 'length' 2>/dev/null || echo 0)
  TOTAL_NEEDS_VALIDATION=$((TOTAL_NEEDS_VALIDATION + NV))
done

TOTAL_RESOLVED=$((TOTAL_VALIDATED + TOTAL_FALSE_POSITIVE))
if [ "$TOTAL_FINDINGS" -gt 0 ]; then
  VALIDATION_COVERAGE_PCT=$(echo "scale=1; $TOTAL_RESOLVED * 100 / $TOTAL_FINDINGS" | bc 2>/dev/null || echo "N/A")
  if [ "$TOTAL_RESOLVED" -gt 0 ]; then
    FALSE_POSITIVE_RATE_PCT=$(echo "scale=1; $TOTAL_FALSE_POSITIVE * 100 / $TOTAL_RESOLVED" | bc 2>/dev/null || echo "N/A")
  else
    FALSE_POSITIVE_RATE_PCT="N/A (no resolved findings)"
  fi
else
  VALIDATION_COVERAGE_PCT="N/A"
  FALSE_POSITIVE_RATE_PCT="N/A"
fi

echo "=== Validation Coverage (2C.5) ==="
echo "Total finding-issues:      $TOTAL_FINDINGS (review-finding + staging-review + audit-finding)"
echo "  validated:               $TOTAL_VALIDATED"
echo "  false-positive:          $TOTAL_FALSE_POSITIVE"
echo "  resolved (total):        $TOTAL_RESOLVED"
echo "  needs-validation (open): $TOTAL_NEEDS_VALIDATION"
echo "Validation coverage:       ${VALIDATION_COVERAGE_PCT}% (target: ≥ 95% for newly closed findings)"
echo "False-positive rate:       ${FALSE_POSITIVE_RATE_PCT}% of resolved findings (actual label-based rate)"

**Metrics to compute**:

• **Validation coverage**: (validated + false-positive) / total finding-issues. Target: ≥ 95% for newly closed findings.
• **False-positive rate