/video-generation

name: video-generation
description: Generate videos from text prompts (and optional reference or frame images) using Letta's asynchronous video generation API. Use when the user asks to create, generate, or make a video or animation from a description, animate an existing image, or turn a prompt into a short video clip.

Video Generation

Generate videos via Letta's async `POST /v1/videos` using `curl` + `jq`. Requires `LETTA_BASE_URL` and `LETTA_API_KEY`, which the runtime provides. If either is unset, stop and ask the user to authenticate with Letta Cloud; do not ask for a video provider key.

This skill is adapted from OpenRouter's `openrouter-video` skill ([OpenRouterTeam/skills](https://github.com/OpenRouterTeam/skills/blob/83827c3f3b1200bc3b609f4ab4ec3241c0cfb7ad/skills/openrouter-video/SKILL.md)). Letta's endpoint forwards the request with Letta's key and charges the organization's Letta credits for the cost of each completed video.

Build every URL relative to `${LETTA_BASE_URL%/}` and send `Authorization: Bearer $LETTA_API_KEY`. Do not hardcode `https://api.letta.com`: Desktop and remote runtimes may provide a proxy base URL, and the credential may only be valid through that URL.

The three steps

Video generation is async: submit → poll → download. A single request can't return the video because generation takes 30s–a few minutes. Tell the user the job was submitted so they know the delay is expected.

1. `POST /v1/videos` → `{ id, polling_url, status: "pending" }` (`polling_url` is the absolute URL of step 2) 2. `GET /v1/videos/{id}` every ~30s until `status` is `completed` (terminal failures: `failed`, `cancelled`, `expired` — surface the `error` field verbatim) 3. `GET /v1/videos/{id}/content?index=0` **with the auth header** → MP4 bytes

A completed job also has `urls`: signed downloads that need no API key until `expires_at` (Unix seconds); anyone holding one can download the video, so prefer step 3.

Pick parameters from the models endpoint, don't guess

`resolution`, `aspect_ratio`, `duration`, and `frame_images[].frame_type` are per-model. Before the first submit for a new model (or whenever the user asks for something specific), fetch the model's capabilities and only send values from the returned sets:

curl -sS "${LETTA_BASE_URL%/}/v1/videos/models" \
  -H "Authorization: Bearer $LETTA_API_KEY" \
  | jq '.data[] | select(.id == "MODEL_ID")'

Fields on each model worth knowing: `supported_resolutions`, `supported_aspect_ratios`, `supported_sizes`, `supported_durations` (often discrete like `[4,6,8]`, not a range), `supported_frame_images` (which `frame_type` values are accepted), `generate_audio` and `seed` (capability bools), `pricing_skus`, and `allowed_passthrough_parameters`. An out-of-set value returns a 400, so validate client-side.

Choosing a model

Recommendation as of 2026-10-29. Video models move fast, so this will go stale; public leaderboards such as [Artificial Analysis](https://artificialanalysis.ai/video/leaderboard/text-to-video) are one rough signal of what's newer, not a verdict. Only pick models that `/v1/videos/models` lists.

• **Default, when quality matters and there's time:** `bytedance/seedance-2.5`. Top tier on the leaderboard, native audio, clips up to 30s. Renders take about 4–6 minutes.
• **Fast:** `minimax/hailuo-3-max` (MiniMax H3). Also top tier; renders in well under a minute. Use it for quick replies in a live thread or for iterating on a prompt, then rerun the final cut on Seedance.
• Other top-tier models on the endpoint include `alibaba/wan-3.0`, `black-forest-labs/flux-3-video` and `heygen/heygen-video-1`. Some providers filter prompts more strictly than others; if one rejects a prompt, try another model rather than rewording around the filter.
• `heygen/avatar-iv` makes a talking head from one photo.

Writing the prompt

For a single clip, read [references/cinematography.md](references/cinematography.md). For multiple shots or a sequence, read [references/storytelling.md](references/storytelling.md).

For worked examples at the right level of detail, see [references/example-prompts.md](references/example-prompts.md).

A timestamped shot list ending on the payoff helps multi-beat clips.

Full workflow (drop-in)

#!/usr/bin/env bash
set -euo pipefail

PROMPT="A golden retriever bounds across wet sand at golden hour to catch a tennis ball mid-air, spray flying off its fur. Low tracking shot at dog height, 35mm, shallow depth of field, warm backlight and lens flare, waves and the ball's thud in the audio. Ends on the dog trotting back toward camera, ball in mouth."
MODEL="bytedance/seedance-2.5"
OUTPUT="video-$(date +%Y%m%d-%H%M%S).mp4"
base_url="${LETTA_BASE_URL%/}"

# Build payload — extend with resolution/aspect_ratio/duration/etc. as needed.
payload=$(jq -n --arg model "$MODEL" --arg prompt "$PROMPT" \
  '{model: $model, prompt: $prompt}')

submit=$(curl -sS -X POST "$base_url/v1/videos" \
  -H "Authorization: Bearer $LETTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$payload")

job_id=$(echo "$submit" | jq -r '.id // empty')
if [ -z "$job_id" ]; then
  echo "Submit failed: $(echo "$submit" | jq -r '.message // .')" >&2
  exit 1
fi
echo "Submitted $job_id" >&2

while :; do
  sleep 30
  resp=$(curl -sS "$base_url/v1/videos/$job_id" -H "Authorization: Bearer $LETTA_API_KEY")
  # Avoid the name `status` — zsh treats it as read-only.
  st=$(echo "$resp" | jq -r '.status')
  echo "Status: $st" >&2
  case "$st" in
    completed) break ;;
    failed|cancelled|expired)
      echo "Generation $st: $(echo "$resp" | jq -r '.error // "unknown"')" >&2
      exit 1 ;;
  esac
done

curl -sS -f "$base_url/v1/videos/$job_id/content?index=0" \
  -H "Authorization: Bearer $LETTA_API_KEY" \
  --output "$OUTPUT"

echo "$resp" | jq --arg out "$(realpath "$OUTPUT")" \
  '{job_id: .id, video_saved: $out, billing}'

Share the saved file the way your current surface supports: attach or upload