claude-video-kit

claude-video-kit

English · 中文 · README.en.md

Turn a research brief or JSON script into a vertical explainer — TTS + caption alignment + Remotion — without opening a timeline editor.

Who it's for: people already writing research / technical explainers in Claude Code or Cursor who want a repeatable 9:16 pipeline with a pre-render review gate.
Who it's not for: one-click “topic → AI b-roll Short” factories. We optimize for auditable scripts and reproducible motion graphics, not generative video lottery.

v0.3.0-rc.1 (public release candidate): installable video-explainer Agent Skill. Independent review receipt must bind to the exact script.json before render. Skill never uploads or publishes.

Fastest first success: Agent Skill

git clone https://github.com/runesleo/claude-video-kit.git
cd claude-video-kit
npm ci --prefix remotion
npx skills add runesleo/claude-video-kit --skill video-explainer
npm run doctor -- --output /tmp/video-explainer-first-success
npm run demo -- --output /tmp/video-explainer-first-success

# npm scripts are thin wrappers — the underlying command works the same:
#   node scripts/video-explainer.mjs demo --output /tmp/video-explainer-first-success

The Skill orchestrates this local clone (it does not bundle Remotion). Demo creates a script-bound pass receipt, uses macOS say as an explicitly demo-quality voice, renders 1080×1920, and runs the shorts verifier — no API key, no upload. For real projects: review then render; fix / block / missing / stale receipts cannot start rendering. The eleven review checks are product-defined; pass/fail is judged by the reviewer — the tool enforces process, not truth.

Editing scripts or compositions? Read docs/DESIGN.md first — it's the design system contract.

Final release gate

The canonical video-release-gate implementation now lives in this pipeline: scripts/video_release_gate.py and config/video_quality_profile.json. Keep the pre-render receipt, review-render, and verify-shorts; then evaluate the final master after all edits, with hash-bound QA and independent review. build-distribute-pack requires a current release receipt and verifies it before writing any handoff. Downstream uploaders must verify again immediately before an authorized action. No uploader or automatic publication is added.

See the complete evaluate → verify → handoff commands.

What you get

• One-command pipeline — ./scripts/render.sh <project> runs TTS, alignment, metadata build, and Remotion render end-to-end.
• Metadata-driven timing — slide durations computed from actual WAV length, no manual frame counting.
• Composable slides — cover, text, code compositions out of the box, each drop-in replaceable.
• Dual TTS backends — Fish Audio API (default, excellent Chinese cloning) or local IndexTTS2 (free, GPU required).
• Word-level captions — faster-whisper produces frame-accurate caption tracks that burn into the render.
• Remotion Studio hot reload — iterate on compositions in React with live preview.

How it works

script.json ──▶ TTS ──▶ *.wav ──▶ Whisper align ──▶ captions.json
                                                            │
                                  build-metadata ◀──────────┘
                                         │
                                         ▼
                                  Remotion render ──▶ out/full.mp4

Input (examples/my-first/script.json):

{
  "title": "My first AI video",
  "fps": 30,
  "width": 1080,
  "height": 1920,
  "slides": [
    { "type": "cover", "title": "Hello world", "subtitle": "in 10 seconds" },
    { "type": "text",  "text": "This is my first AI-rendered video.",
      "voice_text": "This is my first AI-rendered video." },
    { "type": "code",  "language": "ts",
      "code": "const video = await kit.render(script);",
      "voice_text": "One function call, one video file." }
  ]
}

Run:

./scripts/render.sh examples/my-first

Output: examples/my-first/out/full.mp4 — a 1080×1920 vertical video with narration and captions, timed exactly to the generated audio.

Setup

Clone and install:

git clone https://github.com/runesleo/claude-video-kit.git
cd claude-video-kit
cd remotion && npm install && cd ..
pip install -r scripts/requirements.txt
cp .env.example .env   # fill FISH_AUDIO_API_KEY, FISH_AUDIO_VOICE_ID

Any AI coding agent: Point your agent at docs/quickstart.md and examples/schoger-demo/. The whole pipeline is four scripts — an agent can read, modify, and run it without framework knowledge.

Requirements

• Node 20+
• Python 3.10+
• ffmpeg (brew install ffmpeg)
• Fish Audio API key (default TTS) — or skip it and use the macOS say fallback for local testing
• (Optional) CUDA GPU if you want to run IndexTTS2 locally

New to Fish Audio? Create an account here — best-in-class Chinese voice cloning with a trivial API.

That's it. No video editor, no recording gear, no timeline.

Supported slide types

voice_text is the narration script — kept separate from on-screen text so your voice-over can be more natural than what the slide shows. voice overrides the default Fish Audio voice ID for that slide.

Add your own slide type by dropping a new composition into remotion/src/compositions/ and registering it in Root.tsx. See docs/quickstart.md.

Shorts pipeline (vertical 9:16)

Phase 1 adds a vertical short-video pipeline: 1080×1920 canvas, dense motion, big text — built for YouTube Shorts / 抖音 / TikTok / 小红书 / B 站 Shorts. Set "preset": "shorts" in metadata to flip canvas + font scale in one place; render normally, then run node scripts/verify-shorts.mjs <out.mp4> for an objective 4-gate acceptance check (canvas, ≤60s, scene rhythm, hook entrance). See docs/SHORTS_PIPELINE.md for slide-type schema, captions config, and rhythm guidance.

Examples in this repo

• examples/my-first/ — 3-slide starter template. Copy this folder, edit script.json, render. The fastest way to get a working video.
• examples/schoger-demo/ — the script behind my first AI-generated short on X. A worked example of a real shipped video, kept for reference (assets not included, only the script).

Verified

This release was end-to-end tested on macOS (Apple Silicon) with examples/my-first. The shipped pipeline produces a 1080×1920 @ 30fps, 8 seconds, 442 KB vertical mp4 — narration via Fish Audio (or say fallback if no API key), captions via Whisper, render via Remotion. See docs/quickstart.md for first-run notes (including China network setup).

Known limitations (v0.1)

• The say fallback is rough — use Fish Audio for anything you'll publish. Without an API key the pipeline degrades to macOS say, which is fine for "does my pipeline run end-to-end" but ships with two real annoyances: (a) it can't disambiguate Chinese homographs, e.g. 行 in 一行代码 gets read as háng instead of xíng; (b) Whisper's base model struggles to align robotic synthetic voices, so captions drift by a beat. Both vanish with a real Fish Audio voice.
• No automatic cover generator — covers are whatever the cover slide renders
• No B-roll / video clip support — slides are still-frame + audio + captions
• Single voice per video (per-slide voice override is in the schema but undertested)
• IndexTTS2 backend is a placeholder — script is documented, CUDA wiring is left to the user
• Chinese is the primary tested language; other languages work but captioning quality scales with Whisper model size (base → medium → large-v3)

What's new in v0.2 — post-render pipeline

The v0.1 pipeline stopped at out/full.mp4. v0.2 adds the "every time you make a video, you end up redoing this" steps into composable scripts you can chain or compose; the release gate is mandatory before handoff:

• scripts/prepend-cover.sh — prepend a still-image cover clip (default 3s) to the rendered video. Probes the main video's width/height/fps and matches them exactly so concat stays lossless. Falls back to re-encode only if codec params differ.
• scripts/shift-subtitles.py — shift every timestamp in an .srt by a fixed offset. Pair with prepend-cover.sh when you need captions to stay aligned after adding an intro.
• scripts/build-distribute-pack.mjs — emit per-platform upload packages (one of Bilibili / Xiaohongshu / Douyin per invocation) after release verification, with chapter timestamps derived from metadata.json. Supports a blacklist-based compliance pass that strips regulator-sensitive words while keeping brand names — stripping brands costs vertical-search discoverability.
• Pre-render review gate (see docs/pre-render-review.md) — a manual convention: before TTS + render, have a second model read the script. TTS + render costs ~30 min per iteration; 5 min of review usually avoids 2 hours of rerun.

Typical chained usage:

# 1. Render as usual (v0.1)
./scripts/render.sh examples/my-first

# 2. Prepend cover
./scripts/prepend-cover.sh \
  --cover my-cover.png \
  --video examples/my-first/out/full.mp4 \
  --duration 3 \
  --out examples/my-first/out/full-with-cover.mp4

# 3. Complete final QA/review and evaluate full-with-cover.mp4 (see docs/RELEASE_GATE.md).
# 4. Verify and build one platform's handoff; repeat with each platform's evidence.
node ./scripts/build-distribute-pack.mjs examples/my-first --intro-offset 3 \
  --platform bilibili --video examples/my-first/out/full-with-cover.mp4 \
  --cover examples/my-first/cover-bilibili.png \
  --release-receipt examples/my-first/VIDEO-RELEASE-GATE-bilibili.json

shift-subtitles.py is an optional utility for the case where you keep a separate .srt file outside the v0.1 pipeline (e.g. exported from Whisper and used as an overlay). The v0.1 pipeline burns captions during Remotion render, so most users don't need it.

./scripts/shift-subtitles.py \
  --input path/to/captions.srt \
  --offset-seconds 3 \
  --output path/to/captions-with-intro.srt

Each script is self-contained and independently runnable — use the ones you need, skip the rest.

Roadmap

Pipeline

• v0.3 — AI-generated B-roll clips (SDXL / video models for visual variety)
• v0.4 — Auto cover image generator (1080×1920, platform-aware)
• v0.5 — Multi-voice conversations (multiple voice IDs per script)

Compositions

• Chart slide (render data as animated SVG)
• Diff slide (before/after comparison, Schoger-style)
• Diagram slide (Mermaid → animated reveal)

Tooling

• Web UI — non-coders can edit scripts in a browser
• kit init CLI — scaffold new video projects without copying examples

API (planned)

• REST endpoint for render-as-a-service — POST a script, receive an MP4

About the author