/i4h-workflow-validate

name: i4h-workflow-validate
version: "0.6.1"
description: Validate, evaluate, or run i4h envs. Use for policy/checkpoint rollouts and scripted state-machine smoke runs.
license: Apache-2.0
metadata:
  author: "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>"
  tags:
    - isaac-for-healthcare
    - i4h
    - agentic-workflow
    - validation
    - policy-rollout

i4h Workflow — Validate

Purpose

Roll out a policy or scripted state-machine controller against an env and record verification episodes to an HDF5. Use when the user asks to validate, evaluate, run, or rollout a policy/checkpoint, or asks for surgical state-machine smoke runs.

Base Code

These steps drive the i4h-workflows base code (the `workflows/agentic/` tree). To reuse an existing checkout, set `I4H_WORKFLOWS` to its path (no clone happens). Otherwise this resolves the current repo, or clones to `~/i4h-workflows` — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• **Env config (source of truth):** `workflows/agentic/config/environments/<env>.yaml` — read it for the `<env>` defaults: `policy.model_repo`/`model_revision`, `policy.task_description`, `policy.health_port`, and `arena.max_timesteps`.
• Validation runs the policy daemon and Arena together; both processes are required.
• The policy daemon is headless. Arena opens the sim window by default; add `--headless --enable_cameras --rendering_mode performance` only when the user explicitly asks for headless/no-window execution.
• In Claude Code `--print`, Codex `exec`, or any other non-interactive/fresh session, policy evaluation must use **Step 2A** as one foreground bash command. Do not start the policy and Arena in separate tool calls, do not use Claude background tasks for eval, and do not return to the user until Arena exits and the policy cleanup has run.
• In Claude Code specifically, do not use the Bash tool's background mode for `Evaluate ...` prompts, do not launch a command ending in `&`, and do not say "the eval is running in the background." The answer is not complete until the HDF5/log summary has been inspected.
• README quick-run prompts that say "with the state machine" use Arena `--state-machine` and **do not** start a policy daemon.
• Do not run the VLM annotator unless the user asks for success labels.
• `assemble_trocar` is inference-only — validate its YAML default model or a compatible N1.5 checkpoint.

Inputs

• `ENV_ID`: env YAML id.
• `EPISODES`: `1` for sanity, more for real eval.
• `MAX_TIMESTEPS`: use the user-requested cap when the prompt gives one (for example, `300 timesteps` -> `MAX_TIMESTEPS=300`); otherwise read `arena.max_timesteps` from the env YAML for normal evaluation. Use `200` only when the user explicitly asks for a smoke, sanity, or quick check.
• `MODEL_PATH` (optional): path to a `checkpoint-NNNN/` directory containing `model-0000{N}-of-*.safetensors`, `experiment_cfg/`, and `processor/`. Omit to use YAML `policy.model_repo`.
• `USE_LATEST_CHECKPOINT=1`: set this when the prompt says "new checkpoint" or "latest checkpoint" and `MODEL_PATH` is not already known.
• `STATE_MACHINE`: true only when the prompt explicitly says state machine.

Run

Run the steps below in order with the `bash` tool. Script paths like `policy/run.sh`, `arena/run.sh`, and `stop.sh` are commands inside bash, not tool names.

For policy/checkpoint evaluation in Claude Code `--print`, Codex, `codex exec --ephemeral`, or any other non-interactive fresh session, use **Step 2A** after setup. Background policy daemons launched by a finished shell can be cleaned up before Arena connects; the controlled shell keeps policy and Arena in one process lifetime and always stops the daemon afterward. In an interactive local-agent tmux session, the separate Step 2 / Step 3 / Step 4 flow is also acceptable.

Step 1 — setup

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
EPISODES=1
ENV_CONFIG="${REPO_ROOT}/workflows/agentic/config/environments/${ENV_ID}.yaml"
[ -f "${ENV_CONFIG}" ] || { echo "missing env config: ${ENV_CONFIG}" >&2; exit 1; }
PYTHON="${REPO_ROOT}/workflows/agentic/arena/.venv/bin/python"
[ -x "${PYTHON}" ] || PYTHON="${REPO_ROOT}/workflows/agentic/.venv/bin/python"
[ -x "${PYTHON}" ] || { echo "missing workflow python env; run i4h-workflow-setup first" >&2; exit 1; }
MAX_TIMESTEPS="${MAX_TIMESTEPS:-$("${PYTHON}" -c 'import sys, yaml; print(yaml.safe_load(open(sys.argv[1], encoding="utf-8"))["arena"]["max_timesteps"])' "${ENV_CONFIG}")}"
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"

# For prompts such as "Run eval using new checkpoint for 300 timesteps":
#   set MAX_TIMESTEPS=300 and USE_LATEST_CHECKPOINT=1 before this block.
if [ "${USE_LATEST_CHECKPOINT:-0}" = "1" ] && [ -z "${MODEL_PATH:-}" ]; then
  MODEL_PATH="$(find "${RUNS_ROOT}" -path '*/checkpoint/checkpoint-*' -type d -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -1 | cut -d' ' -f2-)"
  [ -n "${MODEL_PATH}" ] || { echo "validate: no checkpoint found under ${RUNS_ROOT}; run finetune first or set MODEL_PATH" >&2; exit 1; }
fi

RUN_DIR="${RUNS_ROOT}/eval_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${RUN_DIR}/data" "${RUN_DIR}/logs"
ln -sfn "${RUN_DIR}" "${RUNS_ROOT}/.latest"

Step 2 — policy daemon

Skip this step when `STATE_MACHINE=true`. For Codex/non-interactive sessions, prefer Step 2A instead of this separate policy-daemon step.

POLICY_ARGS=(--env "${E