/hyperpod-cluster-debugger

name: hyperpod-cluster-debugger
description: Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation / deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.
metadata:
  version: "0.0.1"

HyperPod Cluster Debugger

**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

**Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

`scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/<file>.md § <section>`.

| Reference | Open when | | ------------------------------------------------------------------------- | ------------------------------------------------------------------- | | [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L) | | [cluster-operations.md](references/cluster-operations.md) | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) | | [cloudformation-errors.md](references/cloudformation-errors.md) | § H needs the full per-resource CFN error catalog | | [capacity-planning.md](references/capacity-planning.md) | § B or `--validate` flags capacity / subnet sizing | | [lifecycle-scripts.md](references/lifecycle-scripts.md) | § C points at a specific lifecycle failure | | [iam-permissions.md](references/iam-permissions.md) | Full IAM policy for the diagnostic |

---

Workflow

1. Collect HyperPod cluster name (not EKS name), region, exact error string. 2. Run `scripts/diagnose-cluster.sh` (or `--validate` for pre-create). 3. For every `[FAIL]` line, `Read` the referenced section. 4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval. 5. Re-run the diagnostic to confirm.

---

Step 1: Run diagnostics

# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]

Pass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: `[PASS]` · `[FAIL]` (counted, has `→ references/...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational.

---

Step 2: Match signal → section

**Error messages / events:**

| Signal | Section | | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | | `"EFA health checks did not run successfully"` (public-doc verbatim signal) | **[A: EFA Health Checks](#a-efa-health-checks)** | | Insufficient-capacity or AZ-mismatch failure at creation | **[B: Capacity & AZ](#b-capacity--az)** | | Lifecycle-script failure or timeout during provisioning | **[C: Lifecycle Scripts](#c-lifecycle-scripts)** | | kubectl auth error (server asks for credentials / no API group list) | **[D: EKS Access](#d-eks-access--kubectl)** | | `InService` but not all instances visible | **[E: Cluster Provisioning](#e-cluster-provisioning)** | | `"Target is not connected"` / SSM errors | **[F: SSM Connectivity](#f-ssm-connectivity)** | | Node replacement not happening / `batch-replace` not working | **[G: Node Replacement](#g-node-replacement)** | | `"Embedded stack failed"` / any CloudFormation error | **[H: CloudFormation Errors](#h-cloudformation-errors)** | | `UpdateClusterSoftware` failed or cluster in post-maintenance rollback state | **[J: AMI & Cluster Updates](#j-ami--cluster-updates)** | | Dangling / orphaned nodes in EKS vs `list-cluster-nodes` | **[K: Dangling Nodes & Cleanup](#k-dangling-nodes--cleanup)** | | Cluster Autoscaler breaks after HyperPod attached | **[L: Autoscaler Compatibility](#l-autoscaler-compatibility)** | | Slow I/O, FSx throughput saturated | [cluster-operations.md § 9](references/cluster-operations.md) | | Slurm node name → instance ID lookup