/gke-ai-troubleshooting-tpu-vm-repair-stuck

name: gke-ai-troubleshooting-tpu-vm-repair-stuck
description: >-
  Diagnose GKE TPU VM instances stuck in Compute Engine repair (REPAIRING state)
  after host errors, maintenance events, or faulty host reports in single-host,
  multi-host, dynamic slicing, and TPU All Capacity mode node pools. Use when
  TPU instances in a slice node pool remain in REPAIRING, host error events
  appear in audit logs, or TPU slices are unschedulable due to a VM in repair.
  Don't use for nodes reporting Ready: Unknown when no VM is in REPAIRING (use
  gke-ai-troubleshooting-node-unresponsive-timeout), other NotReady nodes when
  no VM is in REPAIRING (use gke-node-notready), Pending Pods when all nodes are
  Ready (use gke-workload-troubleshooting), JobSet preemption events (use
  gke-ai-troubleshooting-jobset-interruption), or Slice custom resource
  lifecycle failures (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring).
metadata:
  version: "1.0.0"
  category: Containers

Troubleshoot GKE TPU VMs stuck in Compute Engine repair (`REPAIRING`)

This skill checks the VM status, the host events, and the GKE repair status, and routes recovery through GKE instead of direct Compute Engine changes.

Prerequisites

• **Tools**:
• For installing and initializing `gcloud` (which covers authentication and

project selection), follow [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt).

• For installing `kubectl` and required plugins, follow

[Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt) and [Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).

• **Cloud Billing & Project Configuration**: Ensure the project has an active

Cloud Billing account. Follow [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt) to initialize `gcloud` and enable the Google Kubernetes Engine API (`container.googleapis.com`), ensure the Compute Engine API (`compute.googleapis.com`) and Cloud Logging API (`logging.googleapis.com`) are enabled, and follow [Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt) to configure `kubectl` cluster credentials.

• **Required IAM Roles**:
• Kubernetes Engine Viewer (`roles/container.viewer`)
• Compute Viewer (`roles/compute.viewer`)
• Logs Viewer (`roles/logging.viewer`)
• For the user who enables node auto-repair in Step 5, item 1: Kubernetes

Engine Cluster Admin (`roles/container.clusterAdmin`)

• **Documentation**:
• [View information about MIGs and managed instances](https://docs.cloud.google.com/compute/docs/instance-groups/getting-info-about-migs.md.txt)

(Sections: "List MIGs", "Check the status of managed instances")

• [Compute Engine instance lifecycle](https://docs.cloud.google.com/compute/docs/instances/instance-lifecycle.md.txt)

(Sections: "`REPAIRING` state")

• [Troubleshooting VM shutdowns and reboots](https://docs.cloud.google.com/compute/docs/troubleshooting/troubleshooting-reboots.md.txt)

(Sections: "Querying Cloud Audit Logs", "Reviewing Cloud Audit Logs")

• [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)

(Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")

• [Add and manage node pools](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-pools.md.txt)

(Sections: "View node pools in a Standard cluster", "Deploy a Pod to a specific node pool")

• [Troubleshoot NodeNotReady state](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-notready.md.txt)

(Sections: "Check the node's status and conditions", "Confirm node preemption")

• [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)

(Sections: "Multi-host TPU slice node pools")

• [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)

(Sections: "Requirements", "Verify the status of the nodes and the partitions", "Monitor the status of the Slice custom resource", "Hardware maintenance and failures")

• [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)

(Sections: "Key characteristics of dynamic slicing configurations")

• [Deploy TPUs in All Capacity mode in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpu-all-capacity-mode.md.txt)

(Sections: "Manage node failures", "Node failure and recovery")

• [Report and repair faulty TPU hosts in All Capacity mode](https://docs.cloud.google.com/tpu/docs/all-capacity-report-faulty-hosts.md.txt)

(Sections: "Understand the faulty host report process", "Review report faulty host operations")

• [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)

(Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events", "Node auto repair in TPU slice nodes")

• [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)

(Sections: "Before you begin", "Configure auto repair for TPU slice nodes")

• [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)

(Sections: "Before you contact support", "Recommendations for efficient support")

• [REST Resource: projects.locations.clusters.nodePools](https://docs.cloud.google.com/kubernetes-engine/docs/ref