/cortex-training-rl

name: cortex-training-rl
description: >-
  Run reinforcement-learning post-training of a model inside the OpenHands
  agent harness on Snowflake Cortex Training for any Harbor training and eval
  task directories or SWE-smith code localization, and generate the reward
  convergence graph. Use when asked to launch, monitor, debug, stop, or plot an
  OpenHands or Harbor RL run on Cortex Training.
triggers:
- /cortex-training-rl
- cortex training
- cortex-training
- arctic platform
- arctic-platform
- codescout
- harbor

OpenHands RL post-training on Cortex Training

The trainer and the sampler run on Cortex Training GPUs (`4 training + 4 sampling H200s`). Your machine runs a CPU-only driver that executes OpenHands rollouts against a local OpenAI-compatible proxy (`ChatCompletionsProxy`) and sends the exact sampled token IDs (`loss_mask = 1` on assistant turns, `0` on prompts and tool outputs) plus verifier rewards back to Cortex for a GSPO/GRPO update.

Two recipes ship in [Arctic Platform](https://github.com/Snowflake-AI-Research/Arctic-Platform): 1. **Harbor RL (`recipes/rl/harbor` + `arctic_platform/integrations/harbor`, [PR #294](https://github.com/Snowflake-AI-Research/Arctic-Platform/pull/294)):** Takes any Harbor training and eval task directories (`TRAIN_HARBOR_TASKS` and `EVAL_HARBOR_TASKS`, each task containing `instruction.md`, `environment/`, and `tests/test.sh`) and any target OpenHands agent (`HARBOR_SKILL`, `OPENHANDS_TOOLS`, `OPENHANDS_SYSTEM_PROMPT_FILE`, or `HARBOR_AGENT`). Each rollout is orchestrated by Harbor's `Trial` and scored by Harbor's `Verifier` (`tests/test.sh` $\to$ `/logs/verifier/reward.txt` or `reward.json`). 2. **Code localization (`recipes/rl/openhands/code_localization`):** Direct SWE-smith code localization (`file + module + function F1`, max `3.0`).

• Cortex Training guides: [`docs/integrations/openhands.md`](https://github.com/snowflakedb/cortex-training/blob/main/docs/integrations/openhands.md) · [`docs/integrations/bring-your-own-agent.md`](https://github.com/snowflakedb/cortex-training/blob/main/docs/integrations/bring-your-own-agent.md)
• Harbor Snowflake Sandbox provider (`--env snowflake`): [harbor-framework/harbor#3918](https://github.com/harbor-framework/harbor/pull/3918)

Rules

1. **Identify jobs by comment.** Set `ARCTIC_CORTEX_JOB_COMMENT` with your name; never cancel a job whose comment is not yours. 2. **One run per driver machine.** Concurrent runs collide on tmux sessions, ptys, and workspace disk. 3. **Never `kill -9` the driver.** Stop with `SIGTERM` or Ctrl-C so the launcher's exit trap cancels the Cortex GPU job. 4. **Keep credentials private.** Never print, log, or commit PATs or login configs. 5. **Keep recipe hyperparameters intact.** Fix environment issues (`references/troubleshooting.md`) rather than changing batch size, context length, or learning rate.

Workflow

1. Preflight

cortex-training login ~/cortex-training-config.json   # once per machine
cortex-training capacity --hardware H200              # need 8 free
cortex-training list --status running                 # check for live jobs
pgrep -af 'run_qwen35_4b.sh|skyrl.train'              # live driver on this host?
tmux ls 2>/dev/null | grep -c '^openhands-'           # leaked rollout sessions
cat /proc/sys/kernel/pty/nr /proc/sys/kernel/pty/max  # ptys in use / limit
df -h "${OPENHANDS_WORKSPACE:-/tmp/testbed}"          # workspace disk

If no run is live and `openhands-*` tmux sessions remain, clear them before launching:

tmux ls -F '#{session_name}' 2>/dev/null | grep '^openhands-' | xargs -r -n1 tmux kill-session -t

2. Prepare data

Requires `uv`, SkyRL at `skyrl-v0.3.0` (`export SKYRL_HOME=/path/to/SkyRL`), and an `Arctic-Platform` checkout.

• **Any Harbor task set (`recipes/rl/harbor`):**

Point `TRAIN_HARBOR_TASKS` and `EVAL_HARBOR_TASKS` at any directories of Harbor tasks (`instruction.md` + `tests/test.sh`). To export SWE-smith code localization into Harbor task directories (`train/` and `eval/`):

  uv run recipes/rl/harbor/prepare_data.py \
    --export-swe-smith-to ~/data/swe-smith-harbor-tasks \
    --out ~/data/harbor-rl

• **Direct SWE-smith code localization (`recipes/rl/openhands/code_localization`):**

  uv run recipes/rl/openhands/code_localization/prepare_data.py

3. Launch detached

Always launch on a dedicated tmux socket (`-L "$RUN_NAME"`) with `TMUX` and `TMUX_PANE` unset so `libtmux` inside OpenHands does not attach to the launcher's tmux server.

**Harbor RL (`recipes/rl/harbor/run_qwen35_4b.sh`):**

export RUN_NAME=qwen35-4b-harbor-rl-$(date +%m%d-%H%M)
export CKPT_DIR=$HOME/checkpoints/$RUN_NAME
export TRAIN_HARBOR_TASKS=$HOME/data/swe-smith-harbor-tasks/train
export EVAL_HARBOR_TASKS=$HOME/data/swe-smith-harbor-tasks/eval
export HARBOR_SKILL=/path/to/SKILL.md   # optional: ":"-separated SKILL.md paths
export ARCTIC_CORTEX_JOB_COMMENT="<your name> harbor rl - ask before cancelling"
export LOGGER=console OPENHANDS_WORKSPACE=/path/with/space STARTUP_TIMEOUT=14400
mkdir -p "$CKPT_DIR"

tmux -L "$RUN_NAME" new-session -d -s train \
  "echo \$\$ > $CKPT_DIR/launcher.pid; exec env -u TMUX -u TMUX_PANE \
   bash recipes/rl/harbor/run_qwen35_4b.sh trainer.max_training_steps=100"

Optional environment variables to target any OpenHands configuration or Harbor sandbox:

• `OPENHANDS_TOOLS`: comma-separated OpenHands tools (e.g. `terminal,file_editor,task_tracker`).
• `OPENHANDS_SYSTEM_PROMPT_FILE`: custom system prompt template file.
• `HARBOR_AGENT`: Harbor `BaseAgent` (default `harbor_rl.agent:OpenHandsHarborAgent`, or `openhands-sdk`, `openhands`, `module:Class`).
• `HARBOR_ENV`: Harbor `BaseEnvironment` (default `arctic_platform.integrations.harbor.env:HostEnvironment`, or `arctic_platform.integrations.harbor.snowflake_sandbox:SnowflakeSandboxEnvironment` / `snowflake`, `docker`, `modal`, `daytona`).

**Direct code localization (`recipes/rl/openhan