/video-shorts

name: claude-video-shorts
description: >
  Longform-to-shortform video pipeline that automatically extracts viral-ready short clips
  from long videos. V3 pipeline with VLM-powered visual analysis (Gemini 2.5 Flash), auto-detects
  screen recordings vs talking-head, uses "Framed" layout for screen content (padded top/bottom
  with centered captions), face-tracked crop for talking-head. Transcribes with WhisperX, scores
  by visual interest + transcript quality, adds karaoke captions, hook text, normalizes audio.
  Use when user says "shorts", "clips", "highlight", "shortform", "reels", "tiktok clips",
  "extract clips", "viral clips", "repurpose", "longform to shortform", "auto clip",
  "find best moments", or "screen recording shorts".
allowed-tools:
  - Bash
  - Read
  - Write

claude-video-shorts — Longform to Shortform Pipeline (V3)

One-Command Pipeline

V3 Screen-Aware Pipeline (recommended)

For screen recordings, tutorials, and mixed content — uses VLM for visual analysis:

source ~/.video-skill/bin/activate && bash scripts/screen_shorts_pipeline.sh "$INPUT" \
  [--count N] [--min-duration S] [--max-duration S] [--style STYLE] \
  [--platform PLATFORM] [--skip-captions] [--skip-vlm] [--hook-text "Your hook here"] \
  [--mode screen|talking-head|auto]

V1 Audio-Only Pipeline (legacy)

For simple talking-head content where face tracking is sufficient:

source ~/.video-skill/bin/activate && bash scripts/shorts_pipeline.sh "$INPUT" \
  [--count N] [--duration S] [--style STYLE] [--platform PLATFORM]

V3 Pipeline Architecture

INPUT (any video) → Auto-detect mode (screen vs talking-head)
        │
  [1] WhisperX transcription (word-level timestamps)
  [2] PySceneDetect scene boundaries
  [3] Extract keyframes at scene cuts + intervals
  [4] Auto-detect: face rate <60% → screen mode
  [5] VLM analysis (Gemini 2.5 Flash) → visual interest + zoom regions
  [6] Topic segmentation + multi-modal scoring
  [7] Per-topic: extract → reframe → caption → normalize
  [8] Export for platform → manifest JSON

Framed Layout (Screen Mode)

The validated layout for screen recordings (1080x1920):

+--------1080px--------+
|                       |
|  HOOK TEXT (200px)    |  "SEO Health Score: 57/100"
|  Compelling data pt   |  "What Claude found in seconds"
|                       |
+=======================+
|                       |
|  SCREEN CONTENT       |  Cropped from source, scaled to 1080x1372
|  (1372px)             |  VLM-selected crop position
|                       |  ~63% of source resolution
|  Score tables,        |  Readable on phone
|  data, charts...      |
|                       |
+=======================+
|                       |
|  KARAOKE CAPTIONS     |  Impact 72pt, yellow sweep fill
|  (348px)              |  Dark box background (BorderStyle=4)
|  Centered in zone     |  MarginV=240, word-by-word \kf timing
|                       |
+-----------------------+

FFmpeg Filter Chain

crop={W}:{H}:{X}:0 → scale=1080:1372 → pad=1080:1920:0:200:black
→ drawbox (chrome mask) → drawtext (hook) → ass (captions)

Crop Position Logic

VLM analyzes each frame and suggests zoom regions. The crop position (X) varies:

• **Left-aligned content** (tables, lists): x_pct ~0.22
• **Center-aligned content** (title pages, score circles): x_pct ~0.35
• **Right-aligned content** (sidebars, panels): x_pct ~0.55

Multiple zoom regions within a clip create **jump cuts** (not smooth pan) — this matches natural scroll/section transitions in screen recordings.

Multi-Modal Scoring (V3)

| Factor | Weight | Source | Description | |--------|--------|--------|-------------| | Visual interest | 0.30 | VLM | Charts, tables, data = high score | | Content completeness | 0.25 | VLM + transcript | Complete data shown, full thoughts | | Audio engagement | 0.20 | Transcript | Emphasis words, questions, emotion | | Hook potential | 0.15 | Transcript | First sentence strength | | Standalone coherence | 0.10 | Transcript | Works without context |

Caption Style (V3)

The validated caption parameters for the framed layout:

Font: Impact, 72pt, Bold
Primary: White (&H00FFFFFF)
Secondary/Karaoke: Yellow (&H0000FFFF)
Outline: Black, 4px
Background: Semi-transparent black (&HC0000000)
BorderStyle: 4 (opaque box behind text)
Alignment: 2 (bottom-center)
MarginV: 240 (centered in 348px bottom zone)
Letter spacing: 2
Animation: \kf word-by-word karaoke sweep
Words per line: 3

Flags Reference

| Flag | Default | Description | |------|---------|-------------| | `--count N` | 5 | Number of shorts to extract | | `--min-duration S` | 20 | Minimum clip length in seconds | | `--max-duration S` | 55 | Maximum clip length in seconds | | `--style STYLE` | bold | Caption style: default, bold, minimal, neon, shadow | | `--platform PLAT` | shorts | Export: tiktok, reels, shorts, all | | `--skip-captions` | false | Skip caption burn-in | | `--skip-vlm` | false | Skip VLM analysis (audio-only scoring) | | `--hook-text TXT` | auto | Custom hook text for all shorts | | `--language LANG` | en | Transcription language | | `--output-dir DIR` | input dir | Output directory | | `--vlm-model MOD` | gemini-2.5-flash | VLM model for frame analysis | | `--max-frames N` | 40 | Max frames for VLM analysis | | `--mode MODE` | auto | Force: screen, talking-head, auto |

Requirements

Core (always needed)

• FFmpeg with libass support
• WhisperX (`pip install whisperx` in venv, 6GB VRAM)
• PySceneDetect (`pip install scenedetect`)

VLM Analysis (screen mode)

• `GOOGLE_API_KEY` environment variable
• `pip install google-genai` in venv
• Cost: ~$0.01-0.03 per video (~40 frames via Gemini 2.5 Flash)

Face Tracking (talking-head mode)

• MediaPipe (`pip install mediapipe`)
• OpenCV (`pip install opencv-python`)
• CPU-only, no VRAM required

Platform Duration Limits

| Platform | Max Duration | Recommended | Aspect | |----------|-------------|------