/drawing-video

name: drawing-video
description: "Turn a short video into a narrated comic strip. Use when the user uploads a video (.mov/.mp4/etc.) and asks to make a comic from it, narrate it, draw it, Attenborough it, storyboard the clip, or otherwise wants a stills-plus-narration comic strip derived from footage. Covers probing and extracting frames, GROUNDING the actual storyline (native Gemini video parse or frame reading), composing narration in a chosen voice, generating comic panels with Gemini image models anchored to the real stills, QA-ing the panels, and compositing a titled strip. Do NOT use for plain video transcoding or trimming (that is processing-video) or for original illustration from a text prompt (that is invoking-gemini)."
metadata:
  version: 1.0.1

drawing-video

Footage in, narrated comic strip out. Five stages: **probe → ground → narrate → draw → QA/compose**. The hard part is not the drawing; it is not lying about what the footage shows. One stage is a gate.

Requires: `processing-video` (ffmpeg, present), `invoking-gemini` (image + video models), `proxy.env` (CF gateway creds — auto-read by the gemini client).

---

Stage 0 — Probe & extract frames

ffprobe -v quiet -show_entries format=duration -of csv=p=0 in.mov
mkdir -p frames
ffmpeg -v error -i in.mov -vf "fps=1,scale=640:-1" frames/f_%03d.png   # 1/sec, downscaled

Keep full-resolution stills too — you will feed them to the image model as reference. For a subject hidden in clutter, pull a few full-res frames at specific timestamps: `ffmpeg -v error -ss 11 -i in.mov -frames:v 1 hr.png`.

---

Stage 1 — GROUND THE STORYLINE (the gate)

**Never narrate a scene whose subjects and action you have not confirmed from the footage.** Thumbnail-sized frames read poorly: one pass built an entire "indoor cyclist in a garage" storyline for a video that was actually a leashed dog watching a deer — no bicycle in any frame, the leash visible in four — then fed the fiction to the image model, which dutifully drew it. A wrong storyline launders a hallucination into finished art. The absence of a clear read is not license to invent a vivid one.

Two ways to ground. **Prefer Mode B** — it is the more reliable and reads the whole clip, not sampled stills.

Mode B (recommended): native video understanding via gemini-3.5-flash

`gemini-3.5-flash` parses video directly. Compress first, send as inline data.

ffmpeg -v error -i in.mov -vf "scale=480:-2,fps=8" -c:v libx264 -crf 30 -preset veryfast -an small.mp4

import sys, base64; sys.path.append('/mnt/skills/user/invoking-gemini/scripts')
from gemini_client import get_cf_credentials, _cf_request
from pathlib import Path
c = get_cf_credentials()
b = base64.b64encode(Path("small.mp4").read_bytes()).decode()
prompt = ("Watch this video and describe the actual storyline in 3-4 sentences: which "
          "subjects appear, what happens, the arc. Be literal — only what is visibly present.")
contents = [{"parts": [{"inlineData": {"mimeType": "video/mp4", "data": b}}, {"text": prompt}]}]
resp = _cf_request("gemini-3.5-flash", contents,
                   {"temperature": 0.3, "thinkingConfig": {"thinkingLevel": "minimal"},
                    "maxOutputTokens": 600}, c)
print("".join(p.get("text","") for p in resp["candidates"][0]["content"]["parts"]))

Inline works for small files (verified: a 22s 4K clip compressed to 2.3 MB). For long clips, sample frames (below) or use the File API. `gemini-2.5-flash-image` is not a parser — this is `gemini-3.5-flash` (text/multimodal).

Mode A: read the frames yourself

Build a contact sheet (`PIL` grid of `frames/*.png`), then **view enough full-resolution frames to name every subject and the action out loud** before writing a word of narration. If you cannot tell what a shape is, extract a higher-res crop at that timestamp and look again. Do not proceed on a guess.

Either way, end Stage 1 with one plain paragraph: subjects, setting, arc. If the user has already told you the storyline, that paragraph is authority — use it verbatim and skip re-derivation.

---

Stage 2 — Beats & narration

**Let the video decide the panel count.** Break the clip into its natural beats — the distinct moments the story actually has — and make one panel per beat. Do NOT default to four or to a 2×2 grid. A 12-second perch-and-glide is three beats (sentinel → launch → glide); a busier clip may be five or six. Fewer, larger panels for a slow arc; more, tighter panels for rapid action. A hardcoded 2×2 four-panel default is wrong — layout must fit the footage.

Write one short caption per beat, in the requested voice (David Attenborough hushed-drama is the common ask; dial the stakes above what the mundane footage deserves). Captions must be short enough to render legibly in a box — one or two sentences.

Keep a **character/setting bible** in one line: the exact look of each recurring subject (breed/colour, leash colour, the woods, the light). You will paste it into every panel prompt so the panels stay consistent.

---

Stage 3 — Design the page, then draw the panels

**Division of labour (the default):** you are weak at drawing but decent at layout, so *you* design the page and outsource only the drawing. Design a bespoke layout that fits the story — panel count from Stage 2, panel sizes and arrangement chosen to serve the arc (a splash panel for the climax; a tall panel for a vertical action; a downward stack for a descent; growing panels to build drama). Assign each panel a target **aspect ratio**. Then generate each panel with Gemini *at that aspect ratio*, and composite the drawn panels into the layout you designed (PIL). This beats both the rigid uniform grid and the one-shot page.

• Request the aspect in the prompt ("Draw ONE comic panel, 16:7 wide landscape" / "4:3 large" / "3:4 tall"). Gemini 3 honours it closely (measured 2026-07-18: asked 16:7/16:6/4:3, got 2.33/2.72/1.34).
• Composite with **cover-fit** (re