/video-caption

name: claude-video-caption
description: >
  Speech-to-text transcription and animated subtitle generation using Whisper and FFmpeg.
  Creates word-by-word animated captions (karaoke-style), SRT/ASS/VTT subtitles,
  burns subtitles into video, extracts existing subtitles, converts between formats,
  and styles captions with custom fonts, colors, and animations. Use when user says
  "caption", "subtitle", "transcribe", "whisper", "SRT", "ASS", "VTT", "burn-in",
  "hardcode subtitles", "word by word", "karaoke", or "add text".
allowed-tools:
  - Bash
  - Read
  - Write

claude-video-caption — Transcription and Animated Subtitles

Pre-Flight

1. Check Whisper is installed: `command -v whisper-ctranslate2 || echo "Run /video setup"` 2. Run `bash scripts/preflight.sh "$INPUT" "$OUTPUT"` 3. Analyze existing subtitle tracks: `ffprobe -v error -select_streams s -show_entries stream=codec_name,codec_type -of json "$INPUT"`

One-Command Caption Pipeline

For the most common use case (transcribe + animated captions + burn-in):

bash scripts/caption_pipeline.sh "$INPUT" "$OUTPUT" [language] [style]

• `language`: auto (default), en, es, fr, de, ja, zh, etc. (99 languages supported)
• `style`: default, minimal, bold, neon, shadow

Step-by-Step Manual Pipeline

Step 1: Extract Audio for Whisper

ffmpeg -y -i "$INPUT" -vn -ar 16000 -ac 1 -f wav /tmp/claude_video_audio.wav

Step 2: Transcribe with faster-whisper

# Auto-detect language, word-level timestamps
whisper-ctranslate2 /tmp/claude_video_audio.wav \
  --model large-v3 \
  --output_format json \
  --word_timestamps True \
  --vad_filter True \
  --compute_type int8 \
  --output_dir /tmp/

# For faster transcription (slight accuracy tradeoff)
whisper-ctranslate2 /tmp/claude_video_audio.wav \
  --model large-v3-turbo \
  --output_format json \
  --word_timestamps True \
  --vad_filter True \
  --compute_type float16 \
  --output_dir /tmp/

**Model selection**: | Model | Size | VRAM | Speed | Accuracy | Use Case | |-------|------|------|-------|----------|----------| | tiny | 75M | ~1 GB | 32x | Low | Quick drafts | | base | 142M | ~1 GB | 16x | Fair | Quick subtitles | | small | 466M | ~2 GB | 6x | Good | General use | | medium | 1.5B | ~5 GB | 2x | Very good | Quality subtitles | | large-v3-turbo | 809M | ~6 GB | 8x | Very good | Best speed/accuracy (recommended) | | large-v3 | 1.5B | ~10 GB | 1x | Best | Professional captioning |

Step 3: Generate ASS Subtitles with Karaoke Timing

Convert Whisper JSON output to ASS format with word-by-word highlighting.

Generate this Python script inline and run it:

#!/usr/bin/env python3
"""Convert Whisper JSON word timestamps to ASS karaoke subtitles."""
import json, sys

def words_to_ass(json_path, output_path, words_per_line=3, style="default"):
    with open(json_path) as f:
        data = json.load(f)

    # Collect all words with timestamps
    words = []
    for segment in data.get("segments", []):
        for w in segment.get("words", []):
            words.append({
                "word": w["word"].strip(),
                "start": w["start"],
                "end": w["end"]
            })

    # Style presets (full 23-field ASS V4+ format)
    # Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour,
    #         BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing,
    #         Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
    styles = {
        "default":  "Default,Arial,20,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,2,1,2,10,10,40,1",
        "bold":     "Default,Impact,24,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,-1,0,0,0,100,100,0,0,1,3,2,2,10,10,40,1",
        "minimal":  "Default,Helvetica,18,&H00FFFFFF,&H000000FF,&H80000000,&H80000000,0,0,0,0,100,100,0,0,1,1,0,2,10,10,30,1",
        "neon":     "Default,Arial Black,22,&H0000FFFF,&H00FF00FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,2,0,2,10,10,40,1",
        "shadow":   "Default,Georgia,20,&H00FFFFFF,&H000000FF,&H00000000,&H80000000,0,0,0,0,100,100,0,0,1,0,3,2,10,10,40,1",
    }
    style_line = styles.get(style, styles["default"])

    # ASS header
    ass = f"""[Script Info]
Title: claude-video captions
ScriptType: v4.00+
PlayResX: 1920
PlayResY: 1080

[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: {style_line}

[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
"""

    def fmt_time(seconds):
        h = int(seconds // 3600)
        m = int((seconds % 3600) // 60)
        s = seconds % 60
        return f"{h}:{m:02d}:{s:05.2f}"

    # Group words into lines
    for i in range(0, len(words), words_per_line):
        chunk = words[i:i + words_per_line]
        if not chunk:
            continue
        start = chunk[0]["start"]
        end = chunk[-1]["end"]

        # Build karaoke text with \kf tags
        text_parts = []
        for w in chunk:
            duration_cs = max(1, int((w["end"] - w["start"]) * 100))
            text_parts.append(f"{{\\kf{duration_cs}}}{w['word']}")

        text = " ".join(text_parts)
        ass += f"Dialogue: 0,{fmt_time(start)},{fmt_time(end)},Default,,0,0,0,,{text}\n"

    with open(output_path, "w") as f:
        f.write(ass)

if __name__ == "__main__":
    words_to_ass(sys.argv[1], sys.argv[2],
                 int(sys.argv[3]) if len(sys.argv) > 3 else 3,
                 sys.argv[4] if len(sys.argv) > 4 else "default")

Step 4: Burn Subtitles into Video

# ASS subtitles (styled, animated)
ffmpeg -n -i "$INPUT" -vf "ass=/tmp/captions.ass" -c:a copy "$OUTPUT"

# SRT subtitles (simple, with basic styling)
ffmpeg -n -i "$INPUT" -vf "subtitles=/tmp/captions.srt:forc