/cdf-autonomous-vehicle-dataset-generation

name: cdf-autonomous-vehicle-dataset-generation
description: Prepare payloads, run and monitor the Autonomous Vehicle Dataset Generation Airflow workflow, and inspect results for driving-clip retrieval and Cosmos3-Nano variants. Existing clips with no retrieval use cdf-video-data-augmentation; seed-image event or anomaly generation uses cdf-event-video-generation.
license: CC-BY-4.0 AND Apache-2.0
metadata:
  author: NVIDIA
  owner: NVIDIA
  service: cosmos-data-factory
  version: "1.1.0"
  tags:
    - physical-ai
    - cdf-orchestration
    - video-curation
    - cosmos

Autonomous Vehicle Dataset Generation

Find driving clips similar to an example video, or generate rain, snow and daylight variations. Prepare a payload (a JSON file of run settings), run `autonomous_vehicle_dataset_generation_dag_k8s`, and inspect the results. The workflow uses Cosmos-Curator, Cosmos Dataset Search (CDS) and Cosmos3-Nano. It starts and stops its own services.

For augmenting existing clips without retrieval, use [cdf-video-data-augmentation](../cdf-video-data-augmentation/SKILL.md). For event or anomaly videos from a seed image, use [cdf-event-video-generation](../cdf-event-video-generation/SKILL.md). For clip-augmentation requests where the user says no search or retrieval is needed, stop and reroute to video data augmentation rather than building an AV search payload or working around the search requirement.

Use a checkout matching the deployed workflow. Keep the repository documentation and payloads available when this skill is copied into an agent sandbox.

Component details

This workflow skill and its references are the primary guide for configuring and running the workflow. Consult a component skill only when the requested configuration or diagnosis requires component-specific details not covered here, and do not invent keys or values.

| Component | Covers | Component skill | |---|---|---| | CDF Curation and Retrieval | Cosmos-Curator clip splitting, embedding and curation; Cosmos Dataset Search collection ingest and similarity search | [`cdf-curation-and-retrieval` skill][curation-skill] | | CDF Augmentation | Cosmos3-Nano variant generation with RGB or WSM controls, and augmentation errors | [`cdf-augmentation` skill][augmentation-skill] |

[curation-skill]: https://github.com/NVIDIA/cdf-curation-and-retrieval/tree/b4d553d00cc00c1293ad997fa8693af809489c71/skills/cdf-curation-and-retrieval [augmentation-skill]: https://github.com/NVIDIA/cdf-augmentation/tree/73c861c43a4952b6060c87265bf39f4abc48199a/skills/cdf-augmentation

If the repository is inaccessible, continue tasks supported by this workflow's guidance. Identify the unavailable reference and the unresolved component-specific detail only when that missing guidance prevents completing the requested work; do not invent component settings or treat component access as a prerequisite for ordinary workflow use.

Examples

Choose a starter from the current [Payload Guide](../../docs/autonomous-vehicle-dataset-generation/payload-guide.md):

| Request | Starter payload | | --- | --- | | Find one similar driving clip | [Retrieval](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-real-retrieval.json) | | Generate one rainy variation | [Rain](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-rain-generation.json) | | Generate one snowy variation | [Snow](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-snow-generation.json) | | Generate one daylight variation | [Daylight](../../payloads/dev/autonomous_vehicle_dataset_generation_dag/cdf-avdg-day-generation.json) |

All starters process one source scene and request one match. Generation starters produce one 93-frame variation at 1280x720. Rain and daylight use RGB controls, 24 frames per second and 35 inference steps. Snow uses WSM controls, 10 frames per second and 50 inference steps.

Example request:

> Prepare the retrieval starter using storage profiles `inputs` and `outputs`. > Keep the example paths and use collection `demo_av_retrieval_001`. Return the > JSON and Airflow steps; do not submit.

Return the edited payload with generation disabled and the linked submission steps. Explain that a successful run writes `search_result.json` and retrieved clips under `results/` in its output folder.

For more examples, use:

• [Workflow requests](evals/evals.json): the `question` fields cover preparing

retrieval, rain, snow and daylight payloads and explaining their results.

• [Generation prompt example](../../docs/autonomous-vehicle-dataset-generation/payload-guide.md#generate-a-variation):

a snow edit for `cosmos.prompt`. The generation starters above contain the complete rain, snow and daylight prompts.

• [Retrieval walkthrough](../../docs/autonomous-vehicle-dataset-generation/getting-started.md#run-the-workflow)

and [RGB-generation walkthrough](../../docs/autonomous-vehicle-dataset-generation/getting-started.md#next-step-generate-a-variation): input layout, run settings, Airflow submission and expected outputs.

Instructions

Prepare the Payload

Copy the selected starter. For S3-backed AV runs, use explicit `s3://<bucket>/<path>` URIs for every storage path, including `input_video_path`, `query_video` and `output_directory`. Replace all `msc://` placeholders in the starter payload; use the actual backing bucket name, which may differ from the MSC profile name. The current AV runtime does not support `msc://` paths throughout the workflow.

For Azure requests, follow [Azure Storage with AV Dataset Generation](../../docs/autonomous-vehicle-dataset-generation/azure-support.md). That guide requires deployment and code changes before running; changing URI prefixes alone does not enable Azure support.

Set the user's paths and choose a unique collection name. Keep other starter settings unless the user requests a documented change. Ask for missing inputs or clarify whether the user wants retrieval or generati