/hyperpod-ssm

name: hyperpod-ssm
description: Remote command execution and file transfer on SageMaker HyperPod cluster nodes via AWS Systems Manager (SSM). This is the primary interface for accessing HyperPod nodes — direct SSH is not available. Use when any skill, workflow, or user request needs to execute commands on cluster nodes, upload files to nodes, read/download files from nodes, run diagnostics, install packages, or perform any operation requiring shell access to HyperPod instances. Other HyperPod skills depend on this skill for all node-level operations.
metadata:
  version: "1.0.0"

HyperPod SSM Access

Prerequisites

• **`aws` CLI v2**, authenticated for the target account/Region.
• **`session-manager-plugin`** — installed alongside the AWS CLI.
• **`jq`** — the scripts build JSON payloads with it.
• **`unbuffer`** (from the `expect` package) — wraps `aws ssm start-session` with a PTY so the session-manager-plugin flushes stdout instead of racing to close. Without it, calls intermittently return empty output with `Cannot perform start session: EOF` even when the command ran. Install with `sudo yum install expect`, `sudo apt install expect`, or `brew install expect`. `ssm-exec.sh` detects and uses it automatically; falls back with a warning if missing.

SSM Target Format

Target: `sagemaker-cluster:<CLUSTER_ID>_<GROUP_NAME>-<INSTANCE_ID>`

• `CLUSTER_ID`: Last segment of cluster ARN (NOT the cluster name). Extract via `get-cluster-info.sh`.
• `GROUP_NAME`: Instance group name — retrieve via `list-nodes.sh`.
• `INSTANCE_ID`: EC2 instance ID (e.g., `i-0123456789abcdef0`)

Scripts

Three scripts under `scripts/`. Resolve cluster info and nodes **once**, then execute per node.

get-cluster-info.sh — Resolve cluster name → ID (call once)

scripts/get-cluster-info.sh CLUSTER_NAME [--region REGION]
# Output: {"cluster_id":"...","cluster_arn":"...","cluster_name":"...","region":"..."}

list-nodes.sh — List all nodes with pagination (call once)

scripts/list-nodes.sh CLUSTER_NAME [--region REGION] [--instance-group GROUP] [--instance-id ID]
# Output: JSON array of ClusterNodeSummaries (InstanceId, InstanceGroupName, InstanceStatus, etc.)

`list-cluster-nodes` paginates at 100 nodes. This script handles pagination automatically.

ssm-exec.sh — Execute command on a node (call per node)

# Execute — with pre-built target
scripts/ssm-exec.sh --target "sagemaker-cluster:CLUSTERID_GROUP-INSTANCEID" 'command' [--region REGION]

# Execute — with parts
scripts/ssm-exec.sh --cluster-id ID --group GROUP --instance-id INSTANCE_ID 'command' [--region REGION]

# Upload
scripts/ssm-exec.sh --target TARGET --upload LOCAL_PATH REMOTE_PATH [--region REGION]

# Read remote file
scripts/ssm-exec.sh --target TARGET --read REMOTE_PATH [--region REGION]

Running Commands Across Many Nodes

SSM `start-session` rate limit: **3 TPS** per account. Plan batch size and delay accordingly.

`aws ssm send-command` does NOT support `sagemaker-cluster:` targets — only `start-session` works.

Manual SSM Commands

When the scripts aren't suitable, use `aws ssm start-session` directly with `AWS-StartNonInteractiveCommand`. Wrap every invocation in `unbuffer` — without it, stdout is intermittently empty (see Prerequisites).

cat > /tmp/cmd.json << 'EOF'
{"command": ["bash -c 'echo hello && whoami'"]}
EOF

unbuffer aws ssm start-session \
  --target sagemaker-cluster:{CLUSTER_ID}_{GROUP_NAME}-{INSTANCE_ID} \
  --region REGION \
  --document-name AWS-StartNonInteractiveCommand \
  --parameters file:///tmp/cmd.json

• Always use a JSON file for `--parameters` — inline parameters break with special characters.
• The document's `command` parameter is argv, not shell input. Wrap multi-statement scripts in `bash -c '...'` so pipes, semicolons, and redirects evaluate.

Common Diagnostic Commands

| Task | Command | | ---------------- | -------------------------------------------------------------- | | Lifecycle logs | `cat /var/log/provision/provisioning.log` | | Memory | `free -h` | | Disk/mounts | `df -h && lsblk` | | GPU status | `nvidia-smi` | | GPU memory | `nvidia-smi --query-gpu=memory.used,memory.total --format=csv` | | EFA/network | `fi_info -p efa` | | CloudWatch agent | `sudo systemctl status amazon-cloudwatch-agent` | | Top processes | `ps aux --sort=-%mem \| head -20` |

Key Details

• Default SSM non-interactive user is `root`.
• SSM rate limit: **3 TPS** per account.
• For interactive sessions (rare), omit `--document-name` to get a shell.
• Interactive commands (vim, top) are not supported via `AWS-StartNonInteractiveCommand`.
• Large outputs may be truncated by SSM.
• For troubleshooting common errors, see [references/troubleshooting.md](references/troubleshooting.md).