/cdf-orchestration-preflight

name: cdf-orchestration-preflight
description: Set up missing Linux prerequisites and audit whether a Cosmos Data Factory (CDF) Orchestration checkout and its Kubernetes or OSMO environment are ready to install the controller and run every packaged DAG. Use for prerequisite, pre-setup, Kubernetes, GPU, RWX/NFS, credential, controller-readiness, and quick-start checks. Not for deploying the controller (use cdf-orchestration-setup, which calls this audit first), authoring payloads, or running workflows.
license: CC-BY-4.0 AND Apache-2.0
metadata:
  owner: NVIDIA
  service: cosmos-data-factory
  version: 1.1.2
  reviewed: '2026-10-09'
  tags:
    - physical-ai
    - cdf-orchestration
    - preflight
    - kubernetes

Cosmos Data Factory Orchestration Preflight

Determine what the current environment can install and which packaged DAGs it can run. Report partial readiness precisely; do not collapse a mixed environment into a single pass/fail result.

Safety and scope

Start with a read-only audit. Never print kubeconfig content, API keys, tokens, AWS credentials, Kubernetes Secret bodies, or raw command output that may contain them. Do not silently use `~/.kube/config`: use the supplied `KUBECONFIG` path for an existing cluster. For a new cluster, bootstrap creates the kubeconfig.

This skill may offer prerequisite remediation after the audit. Before changing a host, cluster, or tracked configuration, show the exact target and commands and obtain approval. Stop after prerequisites are ready: do not run `make setup`, build or push the Airflow image, install the controller Helm release, or trigger a DAG. Those are deployment or workflow tasks.

Set up missing prerequisites

For a fresh Linux environment, read [bootstrap.md](references/bootstrap.md). It covers local tools, the Python environment, container runtime, Kubernetes, GPU/storage setup and secrets. Reuse working prerequisites and follow the user's execution preference. Installing the Helm client is prerequisite setup; deploying the controller belongs to the setup skill.

Run the audit

Run the bundled script from the repository root:

python3 skills/cdf-orchestration-preflight/scripts/preflight.py audit \
  --repo-root "$PWD" \
  --kubeconfig "$KUBECONFIG"

Use `--format json` when a machine-readable report will help. The script is read-only. If `KUBECONFIG` is unset, ask for its path when a cluster exists; otherwise follow bootstrap. Omit `--kubeconfig` to run local checks before a cluster exists. `AIRFLOW_URL` and `AIRFLOW_TOKEN` may be exported for authenticated post-install checks; never put the token on the command line.

When the user supplies an already-published controller image, pass its exact registry-qualified reference with `--prebuilt-controller-image REGISTRY/REPOSITORY:TAG`. This switches installation readiness from build/push prerequisites to a read-only `docker manifest inspect` check. For a private registry, authenticate Docker before the audit without exposing credentials; the audit does not log in, pull the image, or persist registry credentials. A missing or inaccessible manifest leaves `install_readiness` `UNVERIFIED`.

For a local build, the image location comes from `AIRFLOW_IMAGE_REPOSITORY` and `AIRFLOW_IMAGE_TAG` in `make/common.mk`. The checked-in repository is the placeholder `nvcr.io/<your-org>/<your-team>/sdg-workflow-airflow`, which blocks the audit and `make build sdg-controller` until it is replaced. Follow the [getting-started guide](../../docs/getting-started.md#start-airflow-and-run-workflows): ask for an NGC org/team path that `NGC_API_KEY` can push and a unique tag, then, with approval, set both `?=` values in `make/common.mk`. Tell the user not to commit that edit. Exported variables of the same names override the file for one shell; prefer the file edit because it persists across shells and agents. After the user confirms push permission with the build credentials, pass `--confirmed-push-repository` with the configured repository. This records user confirmation for that exact repository, not an audit-performed push test. Without confirmation, access remains `UNVERIFIED`. The setup skill's build step reads the same values.

For a non-loopback kubeconfig API endpoint, the audit runs a bounded `/busybox/nc` TCP probe from an existing ready NVIDIA GPU Operator toolkit pod. A successful connection is `READY`; a conclusive connection failure is `BLOCKED`. A loopback endpoint is always `BLOCKED`. A missing toolkit pod or probe tool, denied `pods/exec`, or an inconclusive execution remains `UNVERIFIED`. The audit does not create a diagnostic pod.

With a host-installed toolkit, pass `--probe-pod NAMESPACE/POD/CONTAINER` for an existing Running, ready, non-hostNetwork container with Python 3. The first selected container tests API TCP access; selected containers requesting `nvidia.com/gpu` also test loading `libnvcuvid.so.1` on their nodes. Repeat the option to cover each GPU node without a toolkit Pod. Invalid selections, missing Python or denied exec remain `UNVERIFIED`. Bootstrap includes a temporary probe Pod example.

For every node that advertises NVIDIA GPU capacity, the audit verifies `libnvcuvid.so.1` through that node's running NVIDIA GPU Operator toolkit pod. This uses a constant, read-only `kubectl exec` probe and does not create cluster resources. A missing library blocks workflow modes whose readiness profile requires GPU video decode. A missing or non-running toolkit pod, a non-Ready GPU node, or denied `pods/exec` access leaves those modes `UNVERIFIED`; do not report the library as absent without a conclusive probe. Explicit GPU probe containers take precedence over toolkit probes on their nodes; loading failures block decode-dependent modes.

Some modes also require operator-provisioned PVCs in `sdg-workflow` that the Helm chart does not create. The Autonomous Vehicle Dataset Generation modes require `shared-ffmpeg`, the read-only FFmpeg sidecar for Curator