/cdf-augmentation

name: cdf-augmentation
description: >-
  Use when authoring or validating Cosmos Data Factory (CDF) augmentation YAML
  configs, or running remote Cosmos Transfer (including Cosmos3 WSM controls),
  Cosmos Predict, image-edit, or image-to-video inference.
license: Apache-2.0
metadata:
  owner: NVIDIA
  service: cosmos-data-factory
  version: 1.2.0
  reviewed: '2026-09-16'
  author: "NVIDIA <opensource@nvidia.com>"
  tags:
    - physical-ai
    - augmentation
    - cosmos
    - wsm
    - image-edit

CDF Augmentation Pipeline Skill

Remote-API pipeline for captioning, generating, and evaluating augmented camera data. Models are configured as HTTP endpoints; no local model weights are included.

Purpose

Use it to select a supported generation mode, author and validate `PipelineConfig` YAML, configure captioning/evaluators, and run the `cdf-augmentation:1.2.0` container. It covers Cosmos Transfer/Predict, Cosmos3 WSM controls, image editing, image-to-video, BYOM endpoints, and quality gates.

Do **not** use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.

Prerequisites

| Requirement | Detail | |-------------|--------| | **Docker** | `docker --version`. The image is **remote-API only** — it bundles no Cosmos/torch weights, so plain remote inference needs **no GPU and no `HF_TOKEN`**. | | **NVIDIA GPU** (conditional) | Only when a configured local stage requires CUDA or decodes **H.264** in the augmentation container. See Limitations. | | **Endpoint URLs** | One reachable URL per role used by the selected config. The examples name local Qwen services (`Qwen/Qwen3.6-27B-FP8` on `vlm`, `Qwen/Qwen2.5-14B-Instruct` on `llm`), but those are not guaranteed to be running. Resolve every required role independently: keep each reachable configured endpoint and ask only for the missing role URLs. | | **API keys** (conditional) | Only for endpoints requiring authentication. Pass the environment variable named by each endpoint's `api_key_env`; never hardcode values in YAML. Local unauthenticated endpoints need none. | | **Input media** | A video/image reachable by `multistorageclient` (`s3://`, `msc://`, `gs://`, `az://`, or HTTPS). Local paths are allowed **only when no external Cosmos evaluator is enabled**; external evaluation requires checker-readable remote URIs. | | **Approved release commit** (execution only) | A 40-character commit hash confirmed by the user as release-owner-approved or retrieved from the verified release manifest. Never invent it or rely on an unverified environment value. |

Inputs

Resolve each value in this precedence order: **state file → explicit prompt arguments → agent context → user prompt.** Ask the user only for what remains unresolved.

| Input | Required | Description | |-------|----------|-------------| | `config_path` | Yes | Path to the pipeline YAML, e.g. `configs/cookbook/video-data-augmentation/config_video_transfer_CT3_omni.yaml`. If absent, pick a starting config from *Supported Models* and confirm with the user. | | `input_media` | Conditional | Source video/image → `data[].inputs.rgb`. Required for every mode except Cosmos Predict `inference_type: text2world`, where `inputs` may be null or `rgb` omitted. Overridable at run time via `data.0.inputs.rgb=...`. | | `control_media` | For precomputed control | Control video → `data[].inputs.controls.<type>`. For Cosmos3 WSM transfer, set `controls.wsm`; the client uploads it rather than exposing its local path to the server. | | `output_paths` | Yes | `data[].output.{video,caption,metadata}`; `evaluation` optional. | | `model_name` | Yes | `augmentation.model.name` — an endpoint `id`, a role, or a known model name. Free-form string, not an enum. | | `endpoint_urls` | Yes | One `endpoints[]` entry per role in use. | | `api_key_env` | If auth | Env-var *name* per endpoint; the value comes from the environment. | | `target_attributes` | No | `captioning.llm.variables` (e.g. `weather_condition`, `lighting_condition`). | | `generation_params` | No | `augmentation.parameters` — pass-through; only set knobs are sent. | | `seed` | No | Under `augmentation.parameters`. Before the first candidate, `null` resolves once to the generator/executor seed when available, otherwise the current Unix time. Retry `n` uses that resolved base seed plus `n`. `pipeline.retry` defaults to `1`, so evaluation permits at most 2 candidates by default. |

BYOM model: endpoints, adapters, roles

Each `endpoints:` list entry declares `role`, `url`, wire `model`, and optional `id`, `adapter`, `api_key_env`, and `timeout`. Roles are `vlm`, `llm`, `image_edit`, `video_transfer`, `video_predict`, `image2video`, and `evaluator`. `augmentation.model.name` resolves by endpoint `id`, then role, then the known model-name mapping. Adapter contracts and full fields are in [configuration-schema.md](references/configuration-schema.md).

Supported Models

When the user hasn't specified a model, choose from their **input type and goal**:

| Input Type → Goal | `model.name` | Role / default adapter | Input → Output | |-------------------|--------------|------------------------|----------------| | Video — change scene attributes (weather, lighting, style) | `cosmos-transfer2.5` | `video_transfer` / `nim` | Video (+ controls) → Video | | Video + precomputed WSM — follow world-state geometry/motion | A `video_transfer` endpoint id, e.g. `cosmos3-transfer-wsm` | `video_transfer` / `openai.video.async` | RGB video + WSM control + prompt → Video | | Video + text — extend or predict continuation | `cosmos-predict` | `video_predict` / `nim` | Video+Text → Video | | Text only — generate video from scratch | `cosmos-predict` (`inference_type: text2world`) | `video_predict` / `nim` | Text → Video | | Image — edit specific attributes | `image-edit` | `image_edit` / `nim` (or `openai.chat.completions`, `openai.images.edits`) | Image → Image | | Image — animate a first