/capy-video-gen-skill

name: capy-video-gen-skill
description: Multi-shot AI video generation pipeline with face identity consistency. Converts scripts or ideas into complete videos using character extraction, storyboarding, frame generation, and video assembly. 300 experiments validated, 70% face distance improvement. Use when the user asks to create a video from a script, story, idea, or wants multi-shot video with consistent characters.
allowed-tools: Bash, Read, Write, Edit

Capy Video Gen Skill - Script-to-Video Pipeline

Generate complete multi-shot videos from scripts or ideas with consistent character faces across all scenes. Built for HappyCapy AI Gateway. 300 experiments validated, 70% face distance improvement.

Overview

ViMax converts text scripts into full videos through an automated pipeline: 1. Extract characters from script with detailed physical features 2. Generate front/side/back character portraits 3. Design shot-by-shot storyboard 4. Decompose each shot into first_frame, last_frame, and motion descriptions 5. Build camera tree for shot relationships 6. Generate frames with reference image selection (face identity as top priority) 7. Generate video clips from frames 8. Concatenate into final video

Installation Location

The ViMax pipeline code is at: `/home/node/a0/workspace/527fb591-1439-4b5b-ad5d-90f972773f95/workspace/tmp/ViMax/`

All commands must be run from this directory using the venv:

cd /home/node/a0/workspace/527fb591-1439-4b5b-ad5d-90f972773f95/workspace/tmp/ViMax

Prerequisites

• `AI_GATEWAY_API_KEY` environment variable (auto-configured in HappyCapy)
• Python venv at `.venv/` (already set up)

Quick Start

Script-to-Video

Edit the script, requirements, and style in the entry script, then run:

cd /home/node/a0/workspace/527fb591-1439-4b5b-ad5d-90f972773f95/workspace/tmp/ViMax
.venv/bin/python main_happycapy_script2video.py

Idea-to-Video

For generating from a brief idea (auto-generates script first):

cd /home/node/a0/workspace/527fb591-1439-4b5b-ad5d-90f972773f95/workspace/tmp/ViMax
.venv/bin/python main_happycapy_idea2video.py

Programmatic Usage

import asyncio
from langchain.chat_models import init_chat_model
from tools.render_backend import RenderBackend
from utils.config_loader import load_config
from pipelines.script2video_pipeline import Script2VideoPipeline

config = load_config("configs/happycapy_script2video.yaml")
chat_model = init_chat_model(**config["chat_model"]["init_args"])
backend = RenderBackend.from_config(config)

pipeline = Script2VideoPipeline(
    chat_model=chat_model,
    image_generator=backend.image_generator,
    video_generator=backend.video_generator,
    working_dir=config["working_dir"],
)

# Run the pipeline
asyncio.run(pipeline(
    script="Your script here...",
    user_requirement="No more than 8 shots total.",
    style="Cinematic, warm lighting"
))

Pipelines

Script2VideoPipeline

• Input: A formatted screenplay/script with character dialogue and scene descriptions
• Output: Concatenated video at `{working_dir}/final_video.mp4`
• Config: `configs/happycapy_script2video.yaml`

Idea2VideoPipeline

• Input: A brief idea/concept (1-3 paragraphs)
• Output: Auto-generates a script, then produces video
• Config: `configs/happycapy_idea2video.yaml`

Configuration

HappyCapy configs at `configs/happycapy_script2video.yaml`:

chat_model:
  init_args:
    model: gpt-4.1
    model_provider: openai
    api_key: ${AI_GATEWAY_API_KEY}
    base_url: https://ai-gateway.happycapy.ai/api/v1/openai/v1

image_generator:
  class_path: tools.ImageGeneratorHappyCapyAPI
  init_args:
    api_key: ${AI_GATEWAY_API_KEY}
    model: google/gemini-3.1-flash-image-preview

video_generator:
  class_path: tools.VideoGeneratorHappyCapyAPI
  init_args:
    api_key: ${AI_GATEWAY_API_KEY}
    model: google/veo-3.1-generate-preview

working_dir: .working_dir/script2video

Key Components

Agents (AI Processing)

| Agent | File | Purpose | |-------|------|---------| | CharacterExtractor | `agents/character_extractor.py` | Extract characters with static/dynamic features from script | | CharacterPortraitsGenerator | `agents/character_portraits_generator.py` | Generate front/side/back portraits for each character | | StoryboardArtist | `agents/storyboard_artist.py` | Design shot-by-shot storyboard with first/last frames and motion | | ReferenceImageSelector | `agents/reference_image_selector.py` | Select best reference images for each frame (face identity #1 priority) | | CameraImageGenerator | `agents/camera_image_generator.py` | Build camera trees and generate transition videos | | BestImageSelector | `agents/best_image_selector.py` | Select best generated image from candidates | | Screenwriter | `agents/screenwriter.py` | Generate scripts from ideas |

Tools (Generation Backends)

| Tool | File | Purpose | |------|------|---------| | ImageGeneratorHappyCapyAPI | `tools/image_generator_happycapy_api.py` | Image generation via HappyCapy Gateway (Gemini) | | VideoGeneratorHappyCapyAPI | `tools/video_generator_happycapy_api.py` | Video generation via HappyCapy Gateway (Veo) | | RenderBackend | `tools/render_backend.py` | Factory for instantiating generators from config |

Interfaces (Data Models)

• `CharacterInScene` - Character with identifier, static_features, dynamic_features
• `ShotDescription` - Shot with ff_desc, lf_desc, motion_desc, variation_type
• `Camera` - Camera with parent-child relationships
• `Frame` - Frame with shot_idx, frame_type, visible characters
• `ImageOutput` / `VideoOutput` - Generation outputs with save methods

Face Identity Consistency (CRITICAL)

This pipeline includes face identity improvements validated through 257 experiments (70% improvement in face distance, from 0.74 to 0.22):

Built-In Protections

1. **Reference Image Selector**: Face identity is the #1 priority when selecting reference images. The front-view p