/cdf-orchestration-write-dag

name: cdf-orchestration-write-dag
description: Use when a user describes a custom CDF Orchestration pipeline — a specific ordered combination of stages such as augmentation only, auto-labeling only, detection+captioning only, or image attribute augmentation without full auto-labeling — that no existing DAG in airflow/dags/workflows/ covers, and asks for a new Kubernetes DAG. Also use to check that a generated or existing DAG's model/container/prompt choices match an external spec document (e.g. a CDF `launchable.md`). Not for requests that an existing DAG already covers, including payload-only changes such as different inputs, endpoints, models, or variant counts — route those to the matching workflow skill, unless the user still wants a separate DAG after being pointed to it.
version: "1.0.1"
license: CC-BY-4.0 AND Apache-2.0
metadata:
  owner: NVIDIA
  service: cosmos-data-factory
  version: 1.0.1
  reviewed: '2026-10-08'
  author: NVIDIA
  tags:
    - physical-ai
    - cdf-orchestration
    - airflow
    - dag

Write DAG

What This Skill Does

Composes a new Kubernetes-only Airflow DAG from the **existing task groups and stages** that the shipped DAGs are built from: the shared task groups in `airflow/dags/shared/task_groups/`, plus the workflow-local stages the shipped DAGs define for themselves (for example DIG's simulation and AnomalyGen tasks, AV's curation and retrieval tasks). It does not invent new task-group *logic*. Output is:

1. `airflow/dags/workflows/<dag_name>_dag/configs/<dag_name>_k8s_manifest.yaml` — compute + component manifest 2. `airflow/dags/workflows/<dag_name>_dag/<dag_name>_dag.py` — the DAG builder 3. `airflow/dags/workflows/<dag_name>_dag/models/payload.py` — the payload pydantic model 4. `airflow/dags/workflows/<dag_name>_dag/callables/` — **required workflow-local glue**, not optional. Every `*TaskGroup` used here takes one or more `Callable` constructor args with **no default** (e.g. `InputPreparationTaskGroup(prepare_input_callable=...)`, `CosmosTaskGroup(config_generation_callable=..., output_validation_callable=...)`, `ValidatedOutputTaskGroup(validation_callable=...)`). These callables are genuinely new, per-DAG code — they are what turns generic container-arg building into *this* pipeline's actual input listing / config generation / output validation. Do not skip writing them, and do not treat "don't invent task logic" as license to invent them freely either: for each task group you use, find the closest existing DAG that uses the *same* task group, read its equivalent callable, and adapt it — trim the parts that don't apply (e.g. dropping an `output_group_id`/downstream-XCom concern when there is no next stage) rather than writing from a blank page.

**Kubernetes, and OSMO for workflows that support it.** This repository checks in K8s manifests for every workflow, and additionally checks in OSMO manifests for Event Video Generation and Video Data Augmentation (`event_video_generation_dag_osmo`, `video_data_augmentation_dag_osmo`) — see the `cdf-event-video-generation` skill for the pattern. OSMO DAGs are **external-services-only**: their manifests define OSMO batch tasks but no in-cluster VLM/LLM/image-edit/evaluator service components, so an OSMO variant must reject `external_services: false` payloads and cannot use `service_lifecycle` to deploy anything. A user asking for an OSMO variant of a new custom DAG is in scope — replicate the external-services-only contract from an existing OSMO manifest/payload rather than refusing. NVCF is still unsupported — do not offer it unless the user explicitly says their environment supports it, and if so, treat it as new scope requiring its own investigation rather than a flag on this skill.

**Not scoped to one workflow.** Task groups are shared across the shipped DAGs; a custom DAG can mix stages from any of them (e.g. `CosmosTaskGroup` + `CosmosEvaluatorTaskGroup` with no auto-labeling stages, or `ImageAttributeAugmentationTaskGroup` with no `CosmosTaskGroup` for a labeling-only pass over already-augmented crops).

**These DAGs are not checked in as permanent artifacts.** Unlike the five workflow DAGs that ship with this repository (listed in [Step 0](#step-0-confirm-no-existing-dag-already-covers-this)), a DAG this skill generates is a one-off, user-requested pipeline — it is not added to any workflow skill's DAG-ID table. Because nothing else will ever guide the user through deploying or running it, **this skill owns the full lifecycle, not just authoring** — Step 8 below is not optional follow-through, it's the only place readiness-check/deploy/trigger/monitor for this DAG will ever happen.

---

Making Use of Component Skills

The shared task groups are thin Airflow wrappers around CDF components. Each component owns its config schema, model defaults, prompts, and errors in its own skill; the DAG's `container_args` and generated config files are the contract that skill documents. This skill does not reimplement that contract.

**Open the matching component skill before writing manifest task entries, config templates, prompts, or question banks** — never invent config schema, model defaults, or prompt wording from memory. [`references/component-skills-excerpt.md`](references/component-skills-excerpt.md) links every component skill and keeps an offline summary of each. If the component repo is unreachable, use that summary; if what you need isn't in it, say so and ask rather than extrapolating.

Use this table to pick the component and to see which shipped DAG already runs that stage — the closest shipped DAG is also the one to copy callables and manifest entries from:

| Task group(s) or stage | Component | Component skill | Shipped DAGs that already use it | |---|---|---|---| | `CosmosTaskGroup` | CDF Augmentation | `cdf-augmentation` | IAA (image edit), DIG (image edit), EVG (image-to-video), VDA (Cosmos Transfer); AV subclasses it as workflow-local `AVCos