/gke-ai-troubleshooting-tpu-reservation-stockout

name: gke-ai-troubleshooting-tpu-reservation-stockout
description: >-
  Diagnose and resolve cluster-wide GKE TPU node pool creation or scale-up
  failures caused by Compute Engine reservation stockouts (GCE_STOCKOUT,
  ZONE_RESOURCE_POOL_EXHAUSTED) when existing unhealthy or NotReady TPU node
  pools in the cluster have node auto-repair disabled and hold reserved TPU
  capacity. Use when TPU node pool creation or scale-up fails with GCE_STOCKOUT,
  ZONE_RESOURCE_POOL_EXHAUSTED, or ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
  despite having a specific Compute Engine TPU reservation. Don't use for
  general pod scheduling without a TPU reservation or workload-level container
  crashes.
metadata:
  version: "1.0.0"
  category: Containers

Troubleshoot GKE TPU reservation stockouts caused by leaked node pool capacity

Use this skill to diagnose and resolve GKE TPU node pool creation or scale-up failures that fail with `GCE_STOCKOUT`, `ZONE_RESOURCE_POOL_EXHAUSTED`, or `ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS` (or queue due to lack of capacity) even though they target a specific Compute Engine TPU reservation.

If node auto-repair is disabled on an existing TPU node pool, GKE doesn't repair its nodes when they become `NotReady`. While their VMs are still running, they keep consuming slots in the reservation (`inUseCount` is the number of running instances that consume the reservation). Per [Ensure reservation availability](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt), the reservation must have sufficient available TPU chips at the time of node pool creation, and per [Multi-host TPU slice node pools](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt), if GKE is unable to deploy one node in a slice, no nodes in the TPU slice node pool are deployed. When the free slots in the reservation (`specificReservation.count - specificReservation.inUseCount`; see [Consumable instances in a reservation](https://docs.cloud.google.com/compute/docs/instances/reservations-view.md.txt)) are fewer than the number of nodes that the new TPU node pool needs, creation fails with `GCE_STOCKOUT` or `ZONE_RESOURCE_POOL_EXHAUSTED` (or queues due to lack of capacity).

Prerequisites

• **Required IAM Roles**:
• Logs Viewer (`roles/logging.viewer`): Query Cloud Audit Logs in Cloud

Logging.

• Monitoring Viewer (`roles/monitoring.viewer`): Query GKE node readiness

metrics in Cloud Monitoring.

• Kubernetes Engine Viewer (`roles/container.viewer`): Inspect GKE node pools

(`management.autoRepair`), cluster operations, and node conditions.

• Compute Viewer (`roles/compute.viewer`): Inspect Compute Engine reservation

utilization (`specificReservation.inUseCount` vs `specificReservation.count`), consuming instances, and VM labels (including in the reservation's owner project if using a shared reservation).

• For remediation (`[High Risk]` steps): Kubernetes Engine Cluster Admin

(`roles/container.clusterAdmin`)

• **API Enablement & Billing**:
• Ensure the target project has an active Google Cloud billing account

attached: `gcloud billing projects describe {project_id}`.

• Enable required APIs: `container.googleapis.com`, `compute.googleapis.com`,

`logging.googleapis.com`, and `monitoring.googleapis.com`.

• **CLI & Authentication Setup**:
• Install Google Cloud SDK (`gcloud`) and `kubectl`:

https://cloud.google.com/sdk/docs/install.

• **Cluster mode**: This skill applies to Standard clusters. Autopilot clusters

always auto-repair nodes, and you can't turn this off (see [Settings for Autopilot and Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)).

• **Reference Documentation**:
• [Auto-repair nodes](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair.md.txt)

(Sections: "Settings for Autopilot and Standard", "Repair criteria", "Node repair process", "Verify node auto-repair is enabled for a Standard node pool", "Node auto repair in TPU slice nodes", "Enable auto-repair for an existing Standard node pool", "Get information about recent automated repair events")

• [Troubleshoot GKE Standard node pools](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/node-pools.md.txt)

(Sections: "Issue: Node pool creation fails due to insufficient resources")

• [Troubleshoot TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/troubleshooting/tpus.md.txt)

(Sections: "How GKE handles capacity issues", "Insufficient TPU resources to satisfy the TPU request")

• [About TPUs in GKE](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/tpus.md.txt)

(Sections: "Multi-host TPU slice node pools")

• [Consuming reserved zonal resources](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/consuming-reservations.md.txt)

(Sections: "Consuming a specific shared reservation")

• [About reservations](https://docs.cloud.google.com/compute/docs/instances/reservations-overview.md.txt)

(Sections: "How reservations work")

• [View reservations or future reservation requests](https://docs.cloud.google.com/compute/docs/instances/reservations-view.md.txt)

(Sections: "Limitations", "View reservations", "Consumable instances in a reservation", "View a list of instances that are consuming a reservation")

• [Organize resources using labels](https://docs.cloud.google.com/compute/docs/labeling-resources.md.txt)

(Sections: "View labels")

• [Create and manage cluster and node pool labels](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/creating-managing-labels.md.txt)

(Sections: "Automatically applied labels")

• [Deploy TPU workloads in GKE Standard](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/tpus.md.txt)

(Sections: "Ensure reservation availability", "Monitor health metrics for TPU nodes and node pools", "Configure auto repair