/launch-nemo-rl

name: launch-nemo-rl
license: Apache-2.0
description: Playbook for launching, monitoring, stopping, and debugging NeMo-RL recipes on a Kubernetes cluster via the nrl-k8s CLI. Covers ephemeral vs long-lived RayCluster modes, iterating on runs, and debugging hung or failed training jobs.
when_to_use:
  - "run this recipe on k8s"
  - "launch on the cluster"
  - "submit a training job"
  - "tear down the cluster"
  - "resubmit as rayjob"
  - "why is the run stuck"
  - "how do I get logs for job X"
  - "bring the cluster back up"
allowed-tools: Bash Read Grep Glob Edit Write

launch-nemo-rl — running NeMo-RL recipes on Kubernetes via nrl-k8s

This is the playbook for the `nrl-k8s` CLI at `infra/nrl_k8s/`. Follow it when the user asks to launch / iterate / debug a NeMo-RL recipe on a Kubernetes cluster. Verify current state (`kubectl`, `git log`, the recipe + infra files) before acting — the cluster is shared and the cost of a wrong action is high.

1. One command, two modes

There is a single top-level submission command: **`nrl-k8s run`**. It has two lifecycle modes.

| Mode | Invocation | When to use | Cluster after? | | :----------------- | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- | | Ephemeral (default) | `nrl-k8s run` | One-shot. KubeRay applies a RayJob, runs, tears the cluster down. Best for most runs. | No (auto) | | Long-lived | `nrl-k8s run --raycluster` | Dev loop. Reuses a matching live cluster, applies if absent, warns + reuses on drift (pass `--recreate` to replace). Then submits daemons and training. First-choice for iteration. | Yes |

Ask: *Do I need this cluster after the run?* If yes, use `--raycluster`. Otherwise use the default (ephemeral).

The rest of the CLI is observability / stage-by-stage control:

| Command | Purpose | | :---------------------- | :---------------------------------------------------------------------------------------------- | | `nrl-k8s check` | Validate a recipe + infra pair; optionally write the fully-resolved manifests (`-o`). | | `nrl-k8s status` | Per-role RayCluster state, head pod phase, worker pod phases, daemon job status. | | `nrl-k8s cluster up/down/list/dashboard` | Manage RayClusters independently of a run (e.g. render a manifest with `--dry-run`). | | `nrl-k8s job list/logs/stop` | Observability over Ray Jobs already submitted to a role's cluster. | | `nrl-k8s logs` | Tail a role's pod / daemon logs without needing a submission id. |

2. Recipe + infra pair

Every launch takes two files. Pass the infra with `--infra`, not merged inline:

nrl-k8s run infra/nrl_k8s/examples/<recipe>.yaml \
  --infra infra/nrl_k8s/examples/<recipe>.<profile>.infra.yaml

• **Recipe** (e.g. `qwen3_30b_math_8n_4gpu.yaml`) — NeMo-RL config: model, GRPO/SFT knobs, `cluster.{gpus_per_node,num_nodes}`. Uses `defaults:` to inherit from `examples/configs/recipes/llm/...`.
• **Infra** (e.g. `*.<profile>.infra.yaml`) — K8s/Ray shape: namespace, image, service account, RayCluster spec under `kuberay:`, optional Deployments under `deployments:`, `submit.submitter`, `launch.{mode,codeSource,codePath,entrypoint}`. Pair names follow `<recipe>.<profile>[.prod].infra.yaml` where `<profile>` names the hardware target (e.g. `gb300`).

Example pairs in `infra/nrl_k8s/examples/` — read the neighbouring files to see the current conventions for the target profile.

3. Long-lived mode flags

Three independent dimensions. `--mode` is a macro that picks defaults; individual flags override it.

--mode interactive   → --submitter portForward  --code-source upload  (tails logs)
--mode batch         → --submitter exec         --code-source image   (returns after nohup)

• **Submitter**: `portForward` uses `kubectl port-forward` + Ray Job SDK (gets a `submission_id` the dashboard tracks). `exec` uses `kubectl exec` + `nohup` on the head pod (no submission_id; driver appears as `type=DRIVER` in the dashboard).
• **Code source**: `upload` stages a working_dir from the laptop (Ray 100 MiB cap). `image` / `lustre` expect code on the pod's filesystem — paired with `--code-path` (typically `/opt/nemo-rl`), which is a subPath of the shared-filesystem PVC mount in the standard infra examples.
• **Wait**: `--wait` tails logs until terminal; `--no-wait` returns as soon as the driver is running.

Other long-lived-only flags:

• `--replace` — stop any running training / daemon job before submitting new ones (suffixes daemon submissionIds with a timestamp so Ray accepts the resubmit).
• `--recreate` — delete + re-apply a RayCluster whose live spec has drifted from the rendered manifest (default is warn + reuse).
• `--skip-daemons` — bring up all declared clusters but only submit training. Use on disagg recipes where gym/generation are already healthy.

Gotcha: on infra where the entrypoint does `cd /opt/nemo-rl` (or another in-image / Lustre path) and loads the recipe from there, **`--code-source upload` does NOT override the recipe on the pod** — the uploaded working_dir sits in `/tmp/ray/...` but the entrypoint `cd`s away from it. To actually test a local recipe change, either sync your edits to the shared filesystem mounted into the pods or flip the Hydra overrides in the entrypoint.

4. Ephemeral mode flags (`--rayjob`)

When `--rayjob` is set, `run` branches into the RayJob code