/hyperpod-nccl

name: hyperpod-nccl
description: Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce / collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, /dev/shm or memlock issues, NCCL version mismatch across pods, container OOM / exit-137 / OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff / Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA / SSM failures (→ hyperpod-cluster-debugger § A / § F).
metadata:
  version: "0.0.1"

HyperPod NCCL Debugger

**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.

Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts/nccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references/<file>.md § <section>`. Read-only.

**Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.

---

Workflow

1. Collect cluster name, region, namespace/job (EKS), exact NCCL error string. 2. Run the diagnostic (always — the output drives everything else). 3. For every `[FAIL]` line, `Read` the referenced section. 4. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval. 5. Re-run the diagnostic to confirm.

If a finding has no matching section, report it as a bug — do not invent a fix.

Step 1: Authenticate kubectl (EKS)

EKS_ARN=$(aws sagemaker describe-cluster --cluster-name <HYPERPOD-NAME> --region <REGION> \
  --query 'Orchestrator.Eks.ClusterArn' --output text)
EKS_NAME=$(echo "$EKS_ARN" | awk -F'/' '{print $NF}')
aws eks update-kubeconfig --name "$EKS_NAME" --region <REGION>
kubectl get nodes

Step 2: Run the diagnostic

# Basic:
bash scripts/nccl-diagnose.sh --cluster <HYPERPOD-NAME> --region <REGION>

# Scope to an EKS job/namespace:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --namespace <NS> --job <JOB>

# Force orchestrator:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --orchestrator slurm

# Larger hardware sample (default 3):
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --sample-nodes 10

# Specific node only:
bash scripts/nccl-diagnose.sh --cluster <NAME> --region <REGION> --node i-0abc123def456

Tags: `[PASS]` · `[FAIL]` (counted in `Issues Found`, has reference pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks training · **P1** degraded · **P2** informational.

---

Remediation index

Each `[FAIL]` line in the script already points directly at the right section. This table is a lookup for manual triage.

| Finding | Section | | ------------------------------------------ | --------------------------------------------------------------------------------------------------- | | SG missing inbound/outbound self-reference | [operations.md § 8](references/operations.md) | | Blocking NetworkPolicy / allow-all missing | [operations.md § 8](references/operations.md) | | Slurm node DOWN / DRAINING / RemoveIPC | [operations.md § 7](references/operations.md) | | GPU XID / SYSTEM_ERROR / hardware fault | [hyperpod-node-debugger § F / § G](../hyperpod-node-debugger/references/node-diagnostics-detail.md) | | GPU row-remap / DCGM Fail / silent NaNs | [hyperpod-node-debugger § G.1.a/b](../hyperpod-node-debugger/references/node-diagnostics-detail.md) | | NCCL timeout / rendezvous / straggler | [debugging-guide.md § 1](references/debugging-guide.md) | | EFA configuration / not used | [debugging-guide.md § 6](references/debugging-guide.md) | | EFA TCP fallback (`NET/OFI Using TCP`) | [debugging-guide.md § 13](references/debugging-guide.md) | | NCCL version mismatch across pods | [debugging-guide.md § 10](references/debugging-guide.md) | | Container OOM (pod killed, exit 137) | [debugging-guide.md § 4](references/debugging-guide.md) | | GPU OOM (`CUDA out of memory`) | [debugging-guide.md § 11](references/debugging-guide.md) | | RDMA memlock / `/dev/shm` too small | [debugging-guide.md § 17](references/debugging-guide.md) | | MASTER_ADDR DNS / headless Service | [debugging-guide.md § 12](references/debugging-guide.md) | | NVLS / PXN / topology tuning | [debugging-guide.md § 19](references/debugging-guide.md) | | Any NCCL / EFA / rendezvous log pattern | [error-patterns-quick-ref.md](references/error-patterns-quick-r