/i4h-workflow-dataset-annotate

name: i4h-workflow-dataset-annotate
version: "0.6.0"
description: Use a VLM to verify whether each episode satisfies the env's task description. Use when the user asks to annotate, label episodes, filter demos, or gate finetuning on a success classifier.
license: Apache-2.0
metadata:
  author: "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>"
  tags:
    - isaac-for-healthcare
    - i4h
    - dataset
    - annotation
    - vlm

i4h Workflow — Annotate Dataset

Purpose

Use a VLM to verify whether each episode satisfies the env's task description. Use when the user asks to annotate, label episodes, filter demos, or gate finetuning on a success classifier.

Base Code

These steps drive the i4h-workflows base code (the `workflows/agentic/` tree). To reuse an existing checkout, set `I4H_WORKFLOWS` to its path (no clone happens). Otherwise this resolves the current repo, or clones to `~/i4h-workflows` — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• Annotation is optional. Do not run it during validation unless the user requests labels.
• For natural-language prompts such as "Run Annotation on all recorded episodes", annotate all episodes in one selected HDF5 recording, not every historical HDF5 under `workflows/agentic/runs/`. If the user does not name an HDF5, choose the latest annotatable recording: first look inside `runs/.latest` when it contains an HDF5, otherwise pick the newest non-annotation `.hdf5` under `workflows/agentic/runs/`. Only batch across multiple HDF5 files when the user explicitly asks for all historical recordings, every HDF5 file, or a batch annotation run.
• **Env config (source of truth):** the annotator reads the success criterion (`policy.task_description`) from `workflows/agentic/config/environments/<env>.yaml`. Pass `--task-description` to override.
• Talks to an OpenAI-compatible endpoint via `--base-url` (default `http://localhost:8000/v1`) and `--model` (default `Qwen/Qwen3-VL-8B-Instruct`). Point both at a running vision-model server. Do not use text-only/code models such as `qwen3-coder-next`; offline annotation sends image inputs and requires a VLM.
• Keep every annotation artifact inside `workflows/agentic/runs/<run>/`. Do not create or access `/tmp/annotate_*` or other external temp directories.

Start VLM

> **Skip this section if an OpenAI-compatible endpoint serving a vision model is already running** — just set `VLM_BASE_URL`/`VLM_MODEL` in Run to point at it. A local-agent server running `qwen3-coder-next` does not qualify because it is text-only. `annotator/vllm.sh` defaults to port `8000`, so starting it on top of an existing server collides; don't.

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

For readiness, use `workflows/agentic/annotator/vllm.sh ensure`. Do not replace it with raw `docker ps`, fixed sleeps, ad hoc model-listing HTTP probes, or separate manual wait steps; the helper owns the start-and-wait policy.

Step 1 — start VLM (if needed)

Run this exact command. Do not add `sleep`, `status`, `curl`, or shell control operators around it:

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
"${REPO_ROOT}/workflows/agentic/annotator/vllm.sh" ensure

Run (Offline HDF5)

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

Step 1 — setup and resolve HDF5

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"

# LLM endpoint + model (OpenAI-compatible vLLM). Defaults match annotator/vllm.sh; override to use an
# external vision server — e.g. VLM_BASE_URL=http://localhost:8000/v1 VLM_MODEL=qwen3-vl-32b
VLM_BASE_URL="${VLM_BASE_URL:-http://localhost:8000/v1}"
VLM_MODEL="${VLM_MODEL:-Qwen/Qwen3-VL-8B-Instruct}"

# Point HDF5_PATH at a real recording (absolute path). Recordings come from teleop, mimic, or
# validate (which writes data/verify.hdf5 under each runs/eval_* dir). If HDF5_PATH is not set,
# choose one recording: an HDF5 inside runs/.latest if present, otherwise the newest non-annotation
# HDF5 under runs/. "All recorded episodes" means all episodes inside this one HDF5.
HDF5_PATH="${HDF5_PATH:-}"
if [ ! -f "${HDF5_PATH}" ]; then
  LATEST_RUN="$(readlink -f "${RUNS_ROOT}/.latest" 2>/dev/null || true)"
  if [ -n "${LATEST_RUN}" ] && [ -d "${LATEST_RUN}" ]; then
    HDF5_PATH="$(
      find "${LATEST_RUN}" -name '*.hdf5' -type f -printf '%T@ %p\n' 2>/dev/null \
        | sort -nr | awk 'NR==1 { $1=""; sub(/^ /, ""); print; exit }'
    )"
  fi
fi
if [ ! -f "${HDF5_PATH}" ]; then
  HDF5_PATH="$(
    find "${RUNS_ROOT}" \( -path '*/annotate_*' -o -path '*/.latest' \) -prune -o \
      -name '*.hdf5' -type f -printf '%T@ %p\n' 2>/dev/null \
      | sort -nr | awk 'NR==1 { $1=""; sub(/^ /, ""); print; exit }'
  )"
fi
if [ ! -f "${HDF5_PATH}" ]; then
  echo "annotate: set HDF5_PATH to an existing .hdf5 (got '${HDF5_PATH:-<unset>}'). Candidates:" >&2
  find "${RUNS_ROOT}" \( -path '*/annotate_*' -o -path '*/.latest' \) -prune -o \
    -name '*.hdf5' -type f -printf '%TY-%Tm-%Td %TH:%TM  %p\n' 2>/dev/null | sort -r | head
  exit 1
fi

RUN_DIR="${RUNS_ROOT}/annotate_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${RUN_DIR}/data" "${RUN_DIR}