/i4h-workflow-dataset-teleop

name: i4h-workflow-dataset-teleop
version: "0.6.0"
description: Record episodes for an agentic env via teleoperation (keyboard, SO-ARM leader, or VR) into HDF5. Use when the user wants to teleop or record human demos.
license: Apache-2.0
metadata:
  author: "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>"
  tags:
    - isaac-for-healthcare
    - i4h
    - dataset
    - teleoperation
    - recording

i4h Workflow — Teleop Record

Purpose

Record episodes for an agentic env via teleoperation (keyboard, SO-ARM leader, or VR) into HDF5. Use when the user wants to teleop or record human demos.

Base Code

These steps drive the i4h-workflows base code (the `workflows/agentic/` tree). To reuse an existing checkout, set `I4H_WORKFLOWS` to its path (no clone happens). Otherwise this resolves the current repo, or clones to `~/i4h-workflows` — pick that default without prompting. Run every command below from the resolved root:

# Resolve the i4h-workflows base code (provides workflows/agentic/).
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
if [ ! -d "$ROOT/workflows/agentic" ]; then
  ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
  [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
fi
export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"

Basics

• **Env config (source of truth):** `workflows/agentic/config/environments/<env>.yaml` — `robot.type`, `zenoh.camera_names`, and the task for `<env>`.
• Teleop runs through `arena/run.sh --teleop`.
• Device support is env-specific. Check `arena/run.sh --env <env> --help` for valid `--teleop-device` values.
• Do not run teleop headless unless the user explicitly asks. Teleop is a GUI/hardware handoff: launching the recorder is not the same as recording successful demos.

Controls

Reserved keys (consistent across devices):

| Key | Action | |---|---| | `B` | Start episode | | `N` | Mark success, save, advance | | `R` | Discard, reset | | `F` | Reserved by Isaac Sim — do not bind |

Device-specific keybindings (move, rotate, gripper, mode switches) are printed by the teleop process at startup and vary by `--teleop-device`. Report them to the user from the log; they cannot drive the sim without them. See "Surface Device Keybindings". For `keyboard_23d`, the banner can print late, after scene creation and recorder setup; wait for mode/control markers such as `BOTH_HANDS`, `HAND MODE`, `BASE NAVIGATION MODE`, `SPECIAL KEYS`, or `Current Mode` before saying the controls are missing.

Stop from terminal:

workflows/agentic/stop.sh arena --env <env>

Known Devices

| Env | Devices | |---|---| | `scissor_pick_and_place` | `keyboard`, `so101_leader` | | `locomanip_tray_pick_and_place` | `keyboard_23d` | | `locomanip_push_cart` | `keyboard_23d` |

For other envs, consult `arena/run.sh --env <env> --help`.

Run

Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.

Step 1 — setup

REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
ENV_ID=scissor_pick_and_place
RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"
RUN_DIR="${RUNS_ROOT}/teleop_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p "${RUN_DIR}/data" "${RUN_DIR}/logs"
ln -sfn "${RUN_DIR}" "${RUNS_ROOT}/.latest"

Step 2 — teleop record

Use the foreground form when the human operator is ready to drive immediately and the agent can stay attached until completion:

"${REPO_ROOT}/workflows/agentic/arena/run.sh" \
  --env "${ENV_ID}" \
  --teleop \
  --teleop-device <device> \
  --episodes 3 \
  --record-to "${RUN_DIR}/data/demo.hdf5" \
  2>&1 | tee "${RUN_DIR}/logs/teleop.log"

If no human operator is ready, do not fake demos and do not leave a teleop process behind. You may launch long enough to confirm the visible GUI, recorder, and controls, then stop cleanly and report that recording is waiting on a human operator.

Surface Device Keybindings

The teleop process prints its keybinding table to stdout shortly after launch (look for sections such as `Keybindings`, `Controls`, `Key Map`, `BOTH_HANDS`, `HAND MODE`, `BASE NAVIGATION MODE`, `SPECIAL KEYS`, `WORKSPACE LIMITS`, `Current Mode`, mode-switch lines, or any block enumerating keys/actions). Wait for the table to appear, extract it from the log, and report it to the user before they need to drive the sim.

# After launching teleop and tailing the log:
for i in $(seq 1 180); do
  if grep -qiE 'Keybind|Controls|Key Map|BOTH_HANDS|HAND MODE|BASE NAVIGATION MODE|SPECIAL KEYS|WORKSPACE LIMITS|Current Mode|Mode\] Switched|run complete|Traceback|Error' "${RUN_DIR}/logs/teleop.log" 2>/dev/null; then
    break
  fi
  sleep 1
done
grep -n -A80 -E 'Keybind|Controls|Key Map|BOTH_HANDS|HAND MODE|BASE NAVIGATION MODE|SPECIAL KEYS|WORKSPACE LIMITS|Current Mode|Mode\] Switched' "${RUN_DIR}/logs/teleop.log" | head -120

If the block is multi-section (e.g. `BOTH_HANDS`, `BASE_NAV`, `LEFT_HAND`, `RIGHT_HAND` modes), include every mode in the report. Append the reserved keys above so the user has one consolidated reference.

Notes

• `--record-to` must be absolute. The recorder resolves relative paths against `workflows/agentic/arena` (its CWD) and writes to a nested orphan dir. `${RUN_DIR}/data/demo.hdf5` built from `${REPO_ROOT}` is absolute.
• Use `--save-all-episodes` only when failed attempts must be kept.
• If the prompt asks "Run teleop for N episodes" and no human/operator input is available, launch the visible recorder only for readiness/control verification, stop it cleanly, and report that no demos were recorded; do not claim N episodes were recorded.

Verify

• `${RUN_DIR}/data/demo.hdf5` exists.
• Log contains `run complete: N/M episodes succeeded`, and N must equal the requested episode count before reporting success. A tiny HD