/hyperpod-performance-debugger

name: hyperpod-performance-debugger
description: Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.
metadata:
  version: "0.0.1"

HyperPod Performance Debugger

1. **Uneven NCCL performance across nodes** — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler. 2. **Poor filesystem performance** — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.

Scope and delegation

Route findings outside the two in-scope scenarios to the owner skill below.

| Concern observed | Route to | | ---------------------------------------------------------------------- | ------------------------------------------------------------ | | GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | `hyperpod-node-debugger` (§ F Hardware/Auto-Repair, § G GPU) | | `Cannot allocate memory` at `os.fork()`, root volume exhausted | `hyperpod-node-debugger` (§ I Resource Exhaustion) | | NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | `hyperpod-nccl` | | EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | `hyperpod-version-checker` | | EFA self-referencing security-group rule missing — single node | `hyperpod-node-debugger` § A (EFA / Security Group) | | EFA self-referencing security-group rule missing — cluster-wide | `hyperpod-cluster-debugger` § A (EFA Health Checks) | | Slurm node state changes (drain / resume / reboot) | `hyperpod-slurm-debugger` | | Diagnostic bundle for AWS Support | `hyperpod-issue-report` | | Shell access on a node | `hyperpod-ssm` |

Operating policy

• Read-only. Print commands the customer runs; do not execute commands that modify state.
• Container vs host version comparisons go through `hyperpod-version-checker`.
• Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in `hyperpod-node-debugger` § G.

Workflow

1. Confirm the symptom is **uneven NCCL** or **poor filesystem performance**. If neither, route to the matching sibling skill above. 2. Run `scripts/perf-snapshot.sh` (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it. 3. For each `[CONCERN]` line in the script output, open the matching section below and read the supporting reference. 4. After the per-incident diagnosis, recommend the HyperPod platform health features in [§ Continuous health coverage](#continuous-health-coverage) so the customer gets ongoing protection.

Step 1: Run the snapshot

bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Scope to one suspect node:
bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>

The script samples one node by default. It collects host-side data via `hyperpod-ssm`: `nvidia-smi` output (temperature, SM clocks, PCIe link width, ECC, NVLink, `topo -m`), recent `dmesg` Xid lines, EFA port state and `fi_info` provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, `df -h` / `lfs df -h` per mount, host iowait, `/dev/shm` size, and root-volume usage. All read-only.

Tags: `[OK]` healthy · `[CONCERN]` signal worth investigating (carries a `→` pointer to the owner skill) · `[INFO]` informational.

**Host vs container scope.** The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of `unknown` is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via `hyperpod-version-checker` before drawing conclusions.

Step 2: Match signal → section

| Observation | Section | | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Pairwise NCCL bandwidth varies across node pairs / suspected straggler | **[A: Uneven NCCL Performance](#a-uneven-nccl-performance)** | | Nodes spread across AZs / network-node-layer labels / UltraServer boundaries | **[A](#a-uneven-nccl-performance)** | | EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible | **[A](#a-uneven-nccl-performance)** + route to `hyperpod-node-debugger` § A; `hyperpod-version-checker` for cross-node version compare | | `iostat` shows high iowait, FSx CloudWatch utilization sustained near 100% | **[B: Poor Filesystem Performan