/hyperpod-slurm-debugger

name: hyperpod-slurm-debugger
description: Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs."
metadata:
  version: "0.0.1"

HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation.

When to invoke

Invoke when the user reports any of the symptoms in the [decision table](#decision-table).

When NOT to invoke

• Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.
• Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.
• NCCL training-hang investigation — invoke `hyperpod-nccl`.
• Node unreachable via SSM — invoke `hyperpod-ssm`.

Constraints

• Read-only. Do not run, recommend, or print state-mutating commands.
• For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
• IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation

drifts the live state from the IaC plan.

Canonical recovery URLs: [references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).

Prerequisites

• AWS CLI v2, authenticated for the target account and region with permissions:
• `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`
• `ssm:StartSession` on the HyperPod-created SSM document
• [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)

installed locally.

• `jq` ≥ 1.6.
• `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`

returns empty stdout intermittently with `Cannot perform start session: EOF` and every check silently misreports. Install: `expect` package on Amazon Linux / RHEL / Debian / Ubuntu / macOS. Script exits at prerequisite check if missing.

Procedure

Step 1 — Collect inputs

Ask the user for:

1. HyperPod cluster name (not Slurm partition name). 2. AWS region. 3. Optional: a specific Slurm node name.

Step 2 — Confirm orchestrator

aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
  --query 'Orchestrator' --output json

If `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).

Step 3 — Run the diagnostic script

bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
# Scope to a node:
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>

Relay the script output to the user verbatim.

Step 4 — Map findings → docs

For each finding, look up the section in the [decision table](#decision-table) and link the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.

Decision table

| Symptom (`sinfo -o "%N %T %30E"` or script finding) | Section | | ----------------------------------------------------------- | ------------------------------------------------------ | | Node state = `down` or `down*`, reason other than below | [A: Node Down](#a-node-down) | | Node state = `down*`, Reason = `Node unexpectedly rebooted` | [B: Unexpected Reboot](#b-unexpected-reboot) | | Jobs `PENDING` with `REASON=Resources` while nodes are idle | [C: Controller State](#c-controller-state) | | Jobs stuck `COMPLETING` after node replacement | [C: Controller State](#c-controller-state) | | `scontrol ping` returns `DOWN` for the controller | [C: Controller State](#c-controller-state) | | GRES (GPU) counts incorrect or not released | [C: Controller State](#c-controller-state) | | `state=fail` issued but no recovery occurred | [D: Action Reason Mismatch](#d-action-reason-mismatch) | | Accounting errors or RPC errors mentioning `dbd` | [C: Controller State](#c-controller-state) (slurmdbd) | | `slurm.conf` edited; new partitions or nodes not visible | [C: Controller State](#c-controller-state) (config) | | Job exited on a hardware failure but did not restart | [E: Auto-resume](#e-auto-resume) |

Defaults

| Behavior | Default | Override | | -------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- | | Mode | read-only — always; no remediation flag exists | n/a | | Region | `$AWS_DEFAULT_REGION`, falling back to `us-east-1` | `--region <R>` | | Scope | all nodes in `down` / `drain` / `fail` / "unexpectedly rebooted" | `--node <SLURM_NODE_NAME>` | | Output | colorized terminal | `--no-color` | | SSM target format | `sagemaker-cluster:<clusterId>_<instanceGroupName>