/gke-ai-troubleshooting-tpu-slice-mig-stuck-updating

name: gke-ai-troubleshooting-tpu-slice-mig-stuck-updating
description: >-
  Diagnose and resolve GKE TPU Dynamic Slice creation and JobSet scheduling
  failures blocked by underlying Compute Engine managed instance groups (MIGs)
  stuck in an updating or unstable state (isStable=False). Use when TPU slice
  creation fails with a SliceCreationFailed status reason, when JobSets remain
  stuck in pending, or when MIG status reflects isStable=False or
  versionTarget.isReached=False during TPU slice formation. Don't use for
  SliceCreationFailed causes unrelated to MIG stability (use
  gke-ai-troubleshooting-tpu-dynamic-slices-monitoring) or general pod
  scheduling failures unrelated to TPU slices.
metadata:
  version: "1.0.0"
  category: Containers

Troubleshoot GKE TPU slice creation blocked by an unstable MIG

This skill helps diagnose GKE TPU Dynamic Slice creation failures and JobSet scheduling delays when the Compute Engine managed instance groups (MIGs) that back the TPU node pool aren't stable (`status.isStable: false`).

A `SliceCreationFailed` status reason means that slice creation failed because prerequisites weren't met (for example, required Compute Engine resources are missing) or preflight checks failed (see [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)). A MIG that is still creating, updating, or repairing instances is one cause to check. This skill checks the slice status first, then the stability of the backing MIGs.

Prerequisites

• **Google Cloud SDK & Tools**:
• For installing and initializing `gcloud` (project selection and

authentication) and enabling the GKE API, follow [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).

• For installing `kubectl` and configuring cluster credentials, follow the

"Install kubectl", "Install required plugins", and "Store cluster information for kubectl" sections of [Install kubectl and configure cluster access](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).

• **API Enablement & Billing**:
• Verify that the project has an active Cloud Billing account.
• Enable APIs: `container.googleapis.com`, `compute.googleapis.com`, and

`logging.googleapis.com`.

• **Required IAM Roles**:
• Logs Viewer (`roles/logging.viewer`): Inspect Cloud Audit Logs for MIG

errors.

• Compute Viewer (`roles/compute.viewer`): Check MIG stability (`isStable`)

and instance progress.

• Kubernetes Engine Viewer (`roles/container.viewer`): Inspect Slice custom

resources (`slices.accelerator.gke.io`), nodes, and cluster operations.

• **Documentation**:
• [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)

(Sections: "Before you begin", "Create a Slice custom resource", "Monitor the status of the Slice custom resource", "Verify the status of the nodes and the partitions", "Upgrade your node pools")

• [Install kubectl and configure cluster access](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)

(Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")

• [Add and manage node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)

(Sections: "View node pools in a Standard cluster")

• [View information about MIGs and managed instances](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)

(Sections: "Check whether a MIG is stable", "Check the status of managed instances", "List instance errors")

• [Automatically apply VM configuration updates in a MIG](https://docs.cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups.md.txt)

(Sections: "Verify whether an update rollout is complete")

• [Manually upgrade a cluster's control plane or node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/upgrading-a-cluster.md.txt)

(Sections: "Check node pool upgrade status", "Cancel a node pool upgrade", "Resume a node pool upgrade", "Roll back a node pool upgrade")

• [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)

(Sections: "Get information about recent automated repair events")

• [Troubleshoot GKE Standard node pools](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-pools.md.txt)

(Sections: "Issue: Node pool creation fails due to insufficient resources")

• [Troubleshoot TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/tpus.md.txt)

(Sections: "Insufficient quota to satisfy the TPU request", "Insufficient TPU resources to satisfy the TPU request")

• [REST Resource: projects.locations.clusters.nodePools](https://docs.cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters.nodePools.md.txt)

(Sections: "Resource: NodePool")

• [Schedule dynamic slices with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)

(Sections: "Monitor the status of the slice")

• [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)
• [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)

(Sections: "Before you contact support")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain, > delete, or re-create nodes, or run any other command that changes the cluster. > Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

Diagnostic workflow

Step 0: Collect context and set the investigation window `[Low Risk]`

Collect the nec