/cdf-orchestration-setup

name: cdf-orchestration-setup
description: Use when the user asks to deploy, install, reinstall, redeploy, upgrade, or connect to the CDF Orchestration Airflow controller. Owns the end-to-end deploy sequence and runs the cdf-orchestration-preflight audit first. Covers make setup, controller image builds, Helm installation, controller connection, port-forwarding, and post-install verification. Not for standalone prerequisite audits or remediation, K3s environments, authoring DAGs, or running workflow payloads.
license: CC-BY-4.0 AND Apache-2.0
metadata:
  owner: NVIDIA
  service: cosmos-data-factory
  version: 1.1.3
  reviewed: '2026-10-09'
  author: NVIDIA
  tags:
    - physical-ai
    - cdf-orchestration
    - kubernetes
    - airflow

CDF Orchestration Setup

Deploy and verify the Airflow controller after CDF orchestration preflight has established that the checkout and target environment are ready. This skill owns deployment and post-install operations; it does not repeat prerequisite discovery or remediation.

Safety boundary

Before any Helm install, image push, Airflow connection change, or remote mutation, summarize the exact changes and obtain user approval. Never print kubeconfig, NGC keys, Hugging Face tokens, AWS secrets, OSMO tokens, or Kubernetes Secret bodies.

Verify CDF preflight first

Before `make setup`, a controller image build, or any Helm installation or redeployment, invoke and follow `$cdf-orchestration-preflight` for the same checkout and target environment.

Require a fresh preflight report with `install_readiness: READY`. If installation readiness is `BLOCKED` or `UNVERIFIED`, stop and return to preflight to identify and resolve the prerequisite gap. Do not independently rerun prerequisite probes, including local-tool, Kubernetes, GPU, storage/NFS, registry, credential, OSMO, or workflow-mode checks.

A `WARNING` on a tool version means the installed version is outside the tested configuration. It does not block installation: tell the user which tool and version, note that it is untested, and continue once they acknowledge it.

`run_readiness` may still be blocked before the first installation solely because the controller is absent. That is expected and does not block deployment when `install_readiness` is `READY`.

If preflight identifies K3s, stop. This setup skill does not support or invoke K3s. Carry forward the preflight report's repository root, kubeconfig, namespace, storage class, service mode, and image repository and tag rather than rediscovering them.

Preserve the preflight invocation's image and `--probe-pod` arguments, including any exact `--confirmed-push-repository` confirmation, for fresh and post-install audits. Keep bootstrap's temporary probes through those checks; if one has expired, return to preflight to recreate it. Follow its cleanup and single-GPU capacity guidance after the post-install audit.

Select two independent axes

1. Select controller placement: use an existing controller, or deploy Airflow into the current Kubernetes cluster. 2. Select model-service placement separately: use external VLM/LLM/image-edit endpoints, or deploy those services in-cluster. Never infer one choice from the other. 3. Prefer external endpoints on a one-GPU node. The controller and augmentation worker may still run in that node's cluster. **Compute backend: Kubernetes, plus OSMO for whichever DAGs ship an OSMO manifest.** A DAG is registered per platform only when its manifest exists, so determine OSMO support by looking, not by memory:

ls airflow/dags/workflows/*/configs/*osmo*manifest*.yaml

**OSMO workflows accept external endpoints only.** This is structural, not a policy preference: every OSMO manifest defines batch tasks but an empty `components.endpoints`, so there is nothing in-cluster for a service lifecycle to deploy. Any OSMO payload must therefore set `external_services: true` and supply every inference URL the pipeline needs (plus an external evaluator/arbitrator URL when a quality check is enabled). A payload with `external_services: false` is invalid against a deployed OSMO manifest even though the shared payload schema will happily parse it — never submit one, and never reuse a K8s internal-mode starter payload for an OSMO DAG. Internal/in-cluster model services are a Kubernetes-only option.

Read [topologies.md](references/topologies.md) before changing infrastructure.

Deploy a controller on the current cluster

When the user requests setup (not audit-only), choose the command sequence that matches the preflight report, present the exact substituted commands, and obtain explicit approval.

For the build/push path, the image location is `AIRFLOW_IMAGE_REPOSITORY`/`AIRFLOW_IMAGE_TAG` in `make/common.mk`, which preflight already verified. If that file still holds the placeholder repository, return to preflight; do not substitute a path. Name the configured image in the prompt:

> "I will run the following commands in order: > 1. `make setup` — validates secrets from `secrets.env` and generates the Helm values for install > 2. `make build sdg-controller` — builds the Airflow image with DAGs and plugins baked in > (`docker/Dockerfile.airflow` copies `airflow/dags` and `airflow/plugins`), pushes it to > `<AIRFLOW_IMAGE_REPOSITORY>:<AIRFLOW_IMAGE_TAG>` from `make/common.mk`, and writes that > repository/tag into `deploy/values.yaml` > 3. `make install sdg-controller` — installs/upgrades the Helm release using that image > > Installation first uninstalls the release and deletes all cronjobs, jobs, statefulsets, > deployments, replicasets, services, and pods in the target namespace, including unrelated > workloads. Most workload deletions are forced. Uninstall can remove Helm-managed PVCs, > including model-cache and log claims. Preserve needed data before reinstall. > > Proceed?"

For an exact image verified by `--prebuilt-controller-image`:

> "I will run the following commands in order: >