/motion-design-video-analysis

name: motion-design-video-analysis
description: Reverse-engineer a short motion-design / product-update video (animation technique, timing and easing, composition, transitions, likely tooling, vibe, audio) and produce a rebuild spec someone can build from. Default path is one call to google/gemini-3.8-flash over sampled frames, then a mandatory by-eye check of every claim. Use when asked how a marketing or product video was made, what its vibe is, or before replicating one (then continue with motion-design-replication).
metadata:
  type: procedure

Motion-design video analysis

Turns a short (under ~1 min) motion-design clip into two things: an **analysis** (how it was made and how it feels) and a **rebuild spec** (numbers someone can build from without reopening the video). Validated on a 28.6 s product-update clip (Linear's "Loops updates") and reused as the judge on a replica of it.

The model is a fast first reader. You are the ground truth. Both models tested on the Linear clip confidently described a "flat 2D view that becomes 3D at 7.5s"; the frames show a 3D scene from t=0. Plan for that class of error.

Prerequisites

• `ffmpeg` on PATH. Worker images have a full static build at `/usr/local/bin/ffmpeg` (has `fps`, `tile`, `scale`, `crop`, `hstack`; no `drawtext`, no `ffprobe`: read `ffmpeg -i` stderr, or run the community template's `setup.sh` to add ffprobe). The Remotion-bundled ffmpeg under `node_modules/@remotion/compositor-*` is stripped (no `fps`, `tile` or `hstack`); don't use it for analysis.
• Pixel maths: `python3 -m venv /tmp/venv && /tmp/venv/bin/pip install numpy pillow` (system pip is blocked by PEP 668; this venv install was run and works).
• An OpenRouter credential the script can use. In an Agent Swarm, bind it as an egress credential so the key never enters the script (the header `Authorization: Bearer [REDACTED:OPENROUTER_API_KEY]` resolves at egress in **ephemeral `script-run` only**, see Execution notes). Outside a swarm, any way of calling OpenRouter works.
• Getting bytes off agent-fs: `agent-fs download <path> -o <file>`. Viewer URLs serve an SPA shell, not the file; use `agent-fs signed-url <path> --json` when a script needs to fetch it. Upload binaries with `agent-fs write <path> --file <local>` (binary-safe).

Step 1 — probe the file

ffmpeg -hide_banner -i in.mp4 2>&1 | grep -E "Duration|Stream"

Record container duration, resolution, fps, and whether an audio stream exists. These go straight into the rebuild spec.

Step 2 — audio: verify, then exercise transcription for real

1. A stream existing proves nothing. Measure it: `ffmpeg -i in.mp4 -af volumedetect -f null -` and `-af "silencedetect=noise=-30dB:d=0.5"`. 2. "No voice" is a narrower claim than "no audio". A music/SFX bed with no speech is common (the Linear clip: AAC stereo, mean -21 dB, active 0–27.1s, no speech). 3. Always run the transcription call, even when you expect no speech: `ffmpeg -i in.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le out.wav`, stage it, send it to `google/gemini-3.8-flash` as `{type:"input_audio", input_audio:{data:<b64>, format:"wav"}}`. Ask: is there speech; if yes, verbatim transcript with timestamps; if no, describe music (tempo, instrumentation, mood) and each SFX with its timestamp. SFX timestamps matter for the rebuild (they mark the beats).

Step 3 — look at the whole clip yourself first

Before any model call, build contact sheets and read them:

ffmpeg -i in.mp4 -vf "fps=2,scale=480:270,tile=4x5:padding=4:color=red" sheet_%d.png

2 fps catches every cut and caption change in a ~30s clip. For each transition, pull a 10 fps window (`-ss <t-0.4> -t 1.2 -vf "fps=10,scale=320:180,tile=4x4"`) to see how it happens: hard cut, mask roll, blur cross, whip. Write a beat list from this (time, what changes). The model call is a second opinion on your beat list, not the source of it.

Step 4 — default model call: google/gemini-3.8-flash over sampled frames

**Never send raw video.** Every raw-video attempt (base64 `video_url`) timed out on every model, including inside a durable run with `timeoutMs: 180000`; the effective cap stayed ~30s.

1. Sample ~12 frames across the clip: `ffmpeg -i in.mp4 -vf "fps=12/<duration>,scale=480:-1" f_%03d.jpg`. Add frames right before and after each cut you found in Step 3. 2. Base64 them into one JSON (`[{t, b64}]`), write it to agent-fs, get a signed URL. 3. In an ephemeral `script-run`, fetch the JSON, and send one request to `google/gemini-3.8-flash`: a structured prompt, then alternating `{type:"text", text:"t=<n>s"}` / `{type:"image_url", image_url:{url:"data:image/jpeg;base64,..."}}` parts. 4. Prompt for exactly these sections, each claim citing a timestamp: animation technique; timing & easing; composition & layout; transitions; likely tooling; vibe & aesthetic. Ask it to state whether the scene is 2D or 3D **at each timestamp** and what evidence it sees (perspective, foreshortening, blur that varies with depth). The per-timestamp answer makes the miss easy to spot in Step 5; it does **not** prevent it (re-run 2026-09-24 with exactly this question: still "2D" for 0.5–6.6s).

Reference numbers: Linear clip, 12 frames, 10.4s, $0.013; 13 frames with the per-timestamp prompt, 26.6s, $0.022. Keep it under ~14 frames at 480px or you hit the ~30s script cap.

Optional — multi-model comparison

Only when the user asks which model to use, or the default call fails twice. Resolve IDs from the live catalogue (`GET https://openrouter.ai/api/v1/models`, filter `architecture.input_modalities` for `image`/`video`; users' remembered names are often off by a version). Run each candidate once over the same frame JSON. Record completed / latency / `usage.cost` / grounded-or-generic. Known results: `google/gemini-2.5-pro` matched flash at 2.3x cost and 28.9s; `qwen/qwen3.8-max-0902` handled 1 image but timed out on every 3/7/12-frame request (reproduced 3x).

Step 5 — check every model claim against the frame