/extract-frames

name: extract-frames
description: >
  Extracts first and/or last frames of every shot from a video using adaptive scene detection.
  Use this skill when the user says "extract frames", "get shot frames", "pull frames", "shot breakdown",
  "scene detect", "first frame of each shot", "last frame of each shot", "extract shots from video",
  or wants to extract key frames at shot cut points from a video file.
allowed-tools:
  - Bash(ffprobe:*)
  - Bash(ffmpeg:*)
  - Bash(awk:*)
  - Bash(tr:*)
  - Bash(mkdir:*)
  - AskUserQuestion
  - Read
argument-hint: "<video-path> [--first] [--last] [--all] [--shots N,N] [--threshold X]"

Extract Frames

Arguments

Parse the user's request for:

• **video path** (required): path to the video file
• **--last**: extract last frame of each shot instead of first
• **--first --last** or **--all**: extract both first and last frames
• **--shots N,N,N**: only process specific shot numbers (1-indexed)
• **--threshold X**: override auto-detected threshold (skip user confirmation)

Default: extract first frame of every shot.

Workflow

Phase 0: Setup

1. Validate the video file exists. 2. Get video metadata via ffprobe:

ffprobe -v error -show_entries format=duration -show_entries stream=r_frame_rate,width,height,codec_name -of default "$INPUT"

3. Parse fps (needed for last-frame calculation) and duration. 4. Create output directory next to the video: `{video_basename}_frames/`. If it already exists, check for `scores.txt` — if present, skip Phase 1.

Phase 1: Score Dump

Dump per-frame scene scores for the entire video:

ffmpeg -i "$INPUT" -vf "select='gte(scene,0)',metadata=print:file=$OUTPUT_DIR/scores.txt" -fps_mode vfr -f null - 2>&1

This is the expensive step. The output file `scores.txt` contains blocks like:

frame:0    pts:0       pts_time:0.000000
lavfi.scene_score=0.000000

Phase 2: Cut Detection

Parse scores.txt with this awk one-liner to get distribution + all candidate frames:

awk '
/pts_time/ { split($0, a, "pts_time:"); ts=a[2]+0 }
/scene_score/ { split($0, a, "="); score=a[2]+0;
  if (score < 0.01) b1++;
  else if (score < 0.05) b2++;
  else if (score < 0.10) b3++;
  else if (score < 0.20) b4++;
  else b5++;
  total++;
  if (score > max) max=score;
  if (score > 0.05) printf "  ts=%.3fs  score=%.6f\n", ts, score;
}
END {
  print "\n--- Distribution ---";
  printf "< 0.01:      %d (%.1f%%)\n", b1, b1/total*100;
  printf "0.01 - 0.05: %d (%.1f%%)\n", b2, b2/total*100;
  printf "0.05 - 0.10: %d (%.1f%%)\n", b3, b3/total*100;
  printf "0.10 - 0.20: %d (%.1f%%)\n", b4, b4/total*100;
  printf "0.20+:       %d (%.1f%%)\n", b5, b5/total*100;
  printf "Max score:   %.6f\n", max;
}' "$OUTPUT_DIR/scores.txt"

**Step 2a — Startup artifact filter:** Discard any frames in the first 0.5s where `score > 0.05` (a fixed preliminary value — the final user-confirmed threshold is not known yet). Fade-ins from black or codec initialization commonly produce score=1.0 spikes at t=0.03-0.08s that are not real cuts. After discarding these, recompute max score from the remaining frames.

**Step 2b — Branch on max score.** If max score (after startup filter) < 0.05, the video has no cuts:

• Report: "No shot boundaries detected — single continuous shot."
• Extract only frame at t=0 (or t=1.0s if t=0 is a black frame — check file size, <50KB indicates black).
• Skip threshold confirmation.

**Step 2c — Gap analysis and threshold.** If max score >= 0.05, use the raw candidates (all frames with score > 0.05, after startup filter) to find the noise ceiling and the lowest-scoring candidate. Place the proposed threshold at the midpoint of the gap. Present to user via AskUserQuestion:

• Score distribution summary
• Gap analysis (noise ceiling → lowest cut, gap width)
• Proposed threshold and resulting shot count
• Options: Accept proposed (Recommended), Lower threshold, Higher threshold, Custom value

If `--threshold` was provided, skip confirmation and set `$THRESHOLD_VALUE` directly. Otherwise set `$THRESHOLD_VALUE` to the user-confirmed value before proceeding.

**Step 2d — Run-based dedup.** Using `$THRESHOLD_VALUE` from Step 2c, identify cut points. A "run" is a sequence of consecutive frames where every frame scores above threshold. The principle: an aftershock immediately follows its parent cut (consecutive frames both above threshold), while a real cut always rises from the noise floor (preceded by at least one below-threshold frame).

awk -v THRESHOLD="$THRESHOLD_VALUE" '
/pts_time/ { split($0, a, "pts_time:"); ts=a[2]+0 }
/scene_score/ { split($0, a, "="); score=a[2]+0;
  if (score > THRESHOLD) {
    if (!in_run) { run_best_ts = ts; run_best_score = score; in_run = 1; }
    else if (score > run_best_score) { run_best_ts = ts; run_best_score = score; }
  } else {
    if (in_run) { printf "%.3f %.6f\n", run_best_ts, run_best_score; in_run = 0; }
  }
}
END { if (in_run) printf "%.3f %.6f\n", run_best_ts, run_best_score }
' "$OUTPUT_DIR/scores.txt"

This keeps the peak frame of each run and discards aftershocks within the same run. It correctly handles:

• **Standard cuts**: isolated spikes → each kept (run length 1)
• **Cuts with aftershocks**: 2-3 consecutive high frames → peak kept, echoes discarded
• **Rapid montages**: each cut separated by noise frames → all kept, even at 0.12s intervals

Phase 3: Frame Extraction

For each detected cut point (plus t=0 for shot 1):

**First frame** (default):

ffmpeg -y -ss $TIMESTAMP -i "$INPUT" -frames:v 1 -update 1 "$OUTPUT_DIR/shot_${NUM}_${TIME}s.png"

**Last frame** (`--last`):

• For shots 1 through N-1: `last_time = next_shot_timestamp - (1/fps)`
• For final shot: `last_time = duration - (2/fps)` (use 2 frames back, not 1 — seeking to `duration - 1/fps` can produce empty files near the end of some videos)

ffmpeg -y -ss $LAST_TIME -i "$INPUT" -frames:v 1 -update 1 "$OUTPUT_DIR/shot_${NUM}_last_${TIME}s.png"

If a last-frame ext