/i4h-workflow-train-rl

name: i4h-workflow-train-rl
description: Use when training, evaluating, or exporting Workflow policies with online RSL-RL or RLinf, including RL checkpoint and Workflow handoff.
license: Apache-2.0
metadata:
  author: "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>"
  version: "0.8.0"
  tags:
    - isaac-for-healthcare
    - i4h
    - reinforcement-learning
    - rsl-rl
    - rlinf

Train a Workflow Policy with RL

Purpose

Resolve a maintained online-RL profile, verify its Scene/objective/model contracts, run the selected vectorized trainer, evaluate and export its artifact, and hand that artifact to normal Workflow policy validation.

Requirements

• Run the Workflow setup skill first so its uv environments and pinned third-party checkouts are available.
• Use a CUDA-capable Isaac Lab/Arena runtime for the RSL-RL workflow.
• For Trocar, provide a local GR00T N1.5 3B base or SFT checkpoint and two visible local GPUs for the isolated controller and simulator runtimes. A compatible checkpoint is a complete local Hugging Face directory that the pinned GR00T N1.5/RLinf loader accepts without conversion; it must retain the N1.5 3B architecture and support the maintained three-camera plus 28-joint observation mapping and 28-D policy action head. Reject another model family, an exported inference-only Task artifact, or a checkpoint whose config changes those interfaces.

Instructions

1. Resolve the checkout and supported profiles. 2. Confirm the Workflow, Scene, observations, actions, rewards, resets, termination, trainer, and runtime Task contracts. 3. Dry-run the exact requested configuration. 4. Preflight the selected trainer runtime and train in the foreground. 5. Evaluate simulator success, export the policy, and validate it through the normal Workflow runner.

Resolve the checkout

export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
[ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
  [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"
cd "$ROOT"

Treat this resolver as part of the skill contract. `I4H_WORKFLOWS_REPO_URL` selects the clone source; `I4H_WORKFLOWS` selects or reuses a checkout. Never replace an existing checkout.

Resolve the profile and contracts

./train.sh rl list
./train.sh rl show <workflow>
./run.sh show <workflow> --mode policy

Read the profile under `./rl/profiles/`, its referenced declarative trainer config under `./rl/config/`, Workflow, Scene manifest and implementation, Arena objective config, embodiment, and runtime policy Task manifest before a long run.

Select the maintained path from the profile:

| Workflow | Trainer | Starting artifact | Training observations/actions | Export and runtime | | --- | --- | --- | --- | --- | | `ultrasound_probe_reach` | RSL-RL PPO | None; train from scratch | 34-D joint/probe/target state → 6-D relative EE pose | TorchScript `policy.pt` → `rsl_rl/ultrasound_probe_reach` in-process Task | | `assemble_trocar` | RLinf PPO actor/critic | Local GR00T N1.5 SFT/base checkpoint | Three cameras + 28 arm/hand joints → 28 policy actions padded to the 43-D Scene action | Native RLinf run bundle → GR00T inference export → existing remote `gr00t_n15/assemble_trocar` Task |

For `ultrasound_probe_reach`, require the target to be sampled from verified upper-torso surface points, the table and phantom to remain fixed, success to require both position and orientation tolerance for consecutive steps, and the exported Task observation order to match training exactly.

For `assemble_trocar`, require Unitree G1 with Dex3 hands, `front` and both wrist cameras, the current 87-value body state (29 positions + 29 velocities + 29 torques), 14 Dex3 joint positions, the 28-D GR00T arm/hand mapping, the 15-value body-action prefix, and the maintained `g1_trocar` reward/termination contract. Do not copy another Trocar environment into the training tree.

Author a new RL-backed Workflow

1. Create and visibly validate the Workflow and Scene before adding training. Keep assets, embodiment, observations, actions, rewards, resets, termination, and success in their normal Arena and Workflow owners. 2. Add `./rl/profiles/<workflow>.yaml` using the schema below. The filename and `workflow` value must match. 3. Add one declarative YAML trainer config under `./rl/config/`; use `<workflow>_<algorithm>_<backend>.yaml` and point `trainer_config` to it as `../config/<file>.yaml`. 4. Reuse a generic backend under `./rl/i4h_rl/backends/`. Add `./rl/i4h_rl/adapters/<workflow>.py` only when the maintained Scene needs workflow-specific observation, action, registration, or evaluation conversion. Do not add workflow branches to `cli.py`, `sim_server.py`, or a package `__init__.py`. 5. Train and evaluate the native checkpoint, export it into a reusable in-process or remote policy Task, add that Task to the Workflow's `policy` TaskGraph, and validate simulator success through the normal Workflow runner. Compare the runtime observation/action values with training, not only their dimensions and ordering: preserve coordinate frames, quaternion convention, normalization, action scaling, previous-action state, and reset semantics.

schema_version: 1
workflow: <workflow>
scene: <scene>
trainer: <rsl_rl-or-rlinf>
algorithm: ppo
adapter_module: i4h_rl.adapters.<workflow>
trainer_config: ../config/<workflow>_ppo_<backend>.yaml
train_task_id: <trainer-environment-id>
ev