/music-video

name: guided-music-video
description: "Lyric and karaoke music videos drawn in code: get word- and syllable-level lyric timings (from a cue file, or by forced alignment on a vocal stem), write a treatment and a storyboard by song section, derive the edit from the lyrics and the beat map, and integrate the words into every scene instead of subtitling them. Renders one Helios renderAt(t) page with the song attached. Use for lyric videos, karaoke videos, visualizers with lyrics, and story-driven music videos."

Music video

A music video where the words matter. The picture moves on the beat, the words light up on the voice, and every scene does something different with the lyric. It's one `window.renderAt(t)` page, so the [make-video](../../../plugins/helios/skills/make-video/SKILL.md) contract applies; its [music reference](../../../plugins/helios/skills/make-video/references/music.md) covers `helios analyze`, the beat map and the sync rules. This skill adds what a lyric video needs on top of that.

[karaoke.html](karaoke.html) and [lyrics.example.json](lyrics.example.json) are a tested reference for the mechanics: reading word and syllable timings, lighting each glyph as it's sung, looking lyrics up by content, and making the scene react to a word. Reuse the mechanics; design the look and scenes from the treatment.

Ground rules

• **The motion syncs to the beat; the word syncs to the voice.** A word lights exactly at its sung start and is fully lit by its end. A line may appear dim up to about 0.4 s early. **The highlight never runs ahead of the voice.**
• **Never fake the timing.** A wipe whose speed is computed from the number of letters, or a beat grid from a guessed BPM, drifts off the song within seconds. Get real word times (step 1) and a real beat map (`helios analyze`).
• **Look lyrics up by content, never by hard-coded time.** `lyric('through the night').words[0].start`. Re-timing the lyrics then re-times the whole video.
• **Lyrics are usually copyrighted.** Use the user's own song and lyrics. Don't commit lyric text to public examples.

1. Get the timings

| Tier | When | How | Accuracy | |---|---|---|---| | 0 | The user has an `.srt`/`.lrc` with line times, or nothing but the text | Line times from the file (or from tapping along); split each line's span into words weighted by syllable count. Mark the file `"approx": true`. | Lines ±100 ms, words rough | | 1 | Python is available | Separate the vocal (`demucs --two-stems=vocals`), then align the known lyrics to it with stable-ts or WhisperX | Words ±50–100 ms | | 2 | The words must land on consonants | Forced alignment with torchaudio's MMS_FA on the vocal stem, plus onset refinement and a Whisper cross-check | Word starts ±30–50 ms |

Whatever the tier, **decode the song to WAV once** and use that same file for analysis, alignment and `--audio`. MP3 decoders disagree by 20–50 ms because of encoder delay, which is enough to make every word look late.

Convert the result to one `lyrics.json` that both the page and `helios verify --cues` read:

{ "approx": false,
  "lines": [{ "text": "and carried it through the night", "start": 3.6, "end": 6.6, "first": 5, "last": 10 }],
  "words": [{ "w": "carried", "start": 3.9, "end": 4.5, "syl": [[3.9, 4.2], [4.2, 4.5]] }] }

See [references/lyrics.md](references/lyrics.md) for each tier's commands, the conversion script, and how to check timings without ears.

2. Treatment

Write `TREATMENT.md` before any code. Start it with a random seed, so the concept comes from the song and the seed rather than from whatever music video you'd make by default. If an **apply-randomness** skill is available, use it; otherwise run `LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 256; echo`. Write three different concepts and three motifs, then let verifiable patterns in the seed (a digit run, a repeat, a letter cluster) choose between them, and between the palette and the lyric treatment for each section. Keep the seed and a one-line pattern → choice map in the treatment. The song and lyrics are fixed; the seed only decides what's open.

The treatment, on one page:

• **Concept** in one paragraph, and **one motif** that recurs and changes role across the song.
• **Palette** with one accent owned by one moment or one element. **Type voices:** two or three, each with a job (the sung line, the typed aside, the title).
• **An anti-slop list:** what this video will not contain (glowing brains, code rain, particle nebulae, neon for its own sake, a stock "audio visualizer" bar row).
• **Visual ideas, not illustrations.** Puns, transformations and metaphors for each section. Never draw literally what each line says.
• **Chorus plan:** the chorus returns to the same set or template and escalates each time.
• **How the lyric lives in each scene:** riding a curve, typed as tokens, stamped on a form, engraved, written by a pen, unmasked block by block. Never the same plain subtitle bar all the way through.

3. Storyboard by section

Run `helios analyze song.wav` first, then storyboard against its `sections` and `downbeats`:

| Section  | Time        | Lyric (first words)     | Shot / what happens                         | Lyric treatment       | Out             |
| intro    | 0.0–9.4     | —                       | the motif alone, slow push                  | title writes on       | match cut       |
| verse 1  | 9.4–28.6    | "we built a…"           | city of paper windows lights one by one     | words stamped on sills | windows → stars |
| chorus 1 | 28.6–47.8   | "every window…"         | sky set; motif becomes a constellation      | words as star paths   | iris on the moon |

• **Something happens in every shot.** A camera move in every shot. Calm shots in low-energy passages.
• **Intensity follows the song:** roughly 40 % in the intro, 70 % in verses, 100 % in choruses.
• **Rotate shot types on the beat;** never the same pose or framing on adjacent cuts.
• **The Ou