/video-polish

name: video-polish
description: Takes an existing screen recording or demo video and adds professional zoom/pan effects synchronized to the narration. Uses transcript-driven zoom targeting and Remotion for rendering. Optionally replaces audio with a soundtrack.
user-invocable: true
allowed-tools: Bash, Read, Write, Edit, Grep, Glob, WebSearch
argument-hint: [video-file-path]

Video Polish Skill

You take an existing video (screen recording, demo, walkthrough, Loom) and add professional zoom/pan effects that follow the narration. The output looks like a professionally edited video where the camera zooms into whatever the speaker is discussing.

---

What This Skill Does

**Input:** A raw video file (screen recording, Loom, product demo) + optionally a soundtrack **Output:** The same video with smooth zoom/pan effects synchronized to the narration

**What it adds:**

• Zoom-in effects on UI elements, metrics, text, code when the narrator mentions them
• Smooth pan/slide effects across sections (e.g., sliding across table columns)
• Transitions with ease-in-out easing (no jarring jumps)
• Optional audio replacement (background music instead of or mixed with original narration)

**What it does NOT do:**

• Generate new video content
• Add avatars or talking heads
• Edit or cut the video (no trimming, no removing sections)
• Add text overlays or annotations

---

Prerequisites

• **Node.js** (v18+) and **npm** — required for Remotion
• **Remotion** — Video rendering framework. If not already set up, create a project:

  npx create-video@latest --yes --blank --no-tailwind video-polish
  cd video-polish && npm i

• **whisper-cpp** — For audio transcription. Install via `brew install whisper-cpp` on macOS
• **Whisper model** — Download the base English model:

  curl -L -o /tmp/ggml-base.en.bin "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin"

• **Python 3 + Pillow** — For frame extraction and coordinate grid overlays (`pip install Pillow`)

**Before starting:** Verify that Node.js, whisper-cpp, and Python 3 with Pillow are installed. If any are missing, instruct the user to install them before proceeding.

---

How This Skill Works

Step 1: Analyze the Source Video

Get video metadata:

npx remotion ffprobe -v quiet -print_format json -show_format -show_streams <video_path>

Record: duration, resolution (width x height), fps, whether audio exists.

Step 2: Transcribe the Audio

Extract audio and transcribe with word-level timestamps.

**Extract audio:**

npx remotion ffmpeg -y -i <video_path> -vn -acodec pcm_s16le -ar 16000 -ac 1 /tmp/audio.wav

**Transcribe:**

whisper-cli -m /tmp/ggml-base.en.bin -f /tmp/audio.wav --output-json --output-file /tmp/transcript

**Read the transcript** and identify moments where the narrator emphasizes or references specific on-screen elements:

• "look at this", "you can see", "notice", "here", "this is where"
• Naming specific UI elements: "the latency column", "pass rate", "this button"
• Describing what's on screen: "each row represents...", "on the top you can see..."

For each emphasis moment, record:

• Timestamp (when they start talking about it)
• What they're referring to (which UI element, metric, section)

Step 3: Extract Source Frames at Key Timestamps

For each emphasis moment from the transcript, extract the source frame at that timestamp:

npx remotion ffmpeg -y -i <video_path> -vf "select=eq(n\,<frame_number>)" -vframes 1 -update 1 /tmp/frame-<timestamp>.png

Where `frame_number = timestamp_seconds * fps`.

**Important:** The video content may scroll or change over time. Always extract frames at the ACTUAL timestamp, not from a single reference frame. UI element positions change as the user scrolls.

Step 4: Measure Zoom Target Coordinates

For each extracted frame, draw a coordinate grid overlay to precisely identify element positions:

from PIL import Image, ImageDraw
img = Image.open('/tmp/frame-<timestamp>.png')
w, h = img.size
draw = ImageDraw.Draw(img)
for pct in range(0, 100, 5):
    y = int(h * pct / 100)
    x = int(w * pct / 100)
    color = 'red' if pct % 10 == 0 else 'yellow'
    draw.line([(0, y), (w, y)], fill=color, width=1)
    draw.text((5, y+2), f'y{pct}', fill=color)
    draw.line([(x, 0), (x, h)], fill=color, width=1)
    draw.text((x+2, 12), f'x{pct}', fill=color)
img.save('/tmp/frame-<timestamp>-grid.png')

Look at the grid overlay and measure the **center point** (focusX, focusY) of the element the narrator is referring to. Express as normalized 0-1 values (e.g., x=0.47 means 47% from the left edge).

Step 5: Build the Keyframe Timeline

Create a list of keyframes. Each keyframe defines a target zoom state at a specific time. The system smoothly interpolates between consecutive keyframes using ease-in-out.

**Keyframe design rules:**

1. **Anticipate the narration by 0.5 seconds.** If the narrator says "pass rate" at 0:37, start zooming at 0:36.5 so the zoom arrives just as they say it.

2. **Between two zoom-in targets, don't zoom all the way out.** If going from metric A to metric B, reduce zoom to 1.5x briefly while shifting focus, then zoom back in. This is faster and smoother than full-out-then-full-in.

3. **For sliding across adjacent elements** (e.g., table columns), keep the same zoom level and just change focusX. This creates a smooth horizontal pan.

4. **Fast transitions between distant targets** (e.g., jumping from the query column to the latency column) should take 1-1.5 seconds max. Slow slides across distant areas feel boring.

5. **Slow slides across adjacent elements** (e.g., panning from latency → tokens → cost) should take 3-5 seconds. This lets the viewer read each element.

6. **Hold the zoom for at least 2-3 seconds** after arriving at a target. Quick zoom-in → immediate zoom-out is disorienting.

7. **Start and end the video at full view (zoom=1.0).** D