/gke-ai-troubleshooting-tpu-partition-overlap

name: gke-ai-troubleshooting-tpu-partition-overlap
description: >-
  Diagnose and resolve GKE TPU slice creation failures (SliceCreationFailed)
  caused by partition overlap conflicts where requested TPU partitions or their
  nodes are already used by another slice or JobSet. Use when diagnosing GKE TPU
  slice creation failures with SliceCreationFailed partition or node conflicts.
  Don't use for general slice lifecycle states or slice provisioning failures
  without node overlap (use
  gke-ai-troubleshooting-tpu-dynamic-slices-monitoring), quota exhaustion, or
  unresponsive TPU nodes.
metadata:
  version: "1.0.0"
  category: Containers

Troubleshoot TPU slice creation failures caused by partition overlap

A dynamic slice fails to form with the status reason `SliceCreationFailed` when, among other documented causes, any of its nodes are already used by another slice. This skill confirms the failure, rules out the other documented causes, finds the slice that uses the same nodes, and routes the fix through GKE.

Prerequisites

• **Tools**:
• For installing and initializing `gcloud` (which covers authentication and

project selection), follow [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).

• For installing `kubectl` and required plugins, follow

[Install kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt) and [Install required plugins](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt).

• **Cloud Billing & Project Configuration**: Ensure the project has an active

Cloud Billing account. Follow [Before you begin](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt) to initialize `gcloud` and enable the Google Kubernetes Engine API (`container.googleapis.com`), ensure the Cloud Monitoring API (`monitoring.googleapis.com`) is enabled, and follow [Store cluster information for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt) to configure `kubectl` cluster credentials.

• **Required IAM Roles**:
• Kubernetes Engine Viewer (`roles/container.viewer`)
• Monitoring Viewer (`roles/monitoring.viewer`)
• **Documentation**:
• [Use dynamic slicing with a custom scheduler](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt)

(Sections: "Before you begin", "Target your node pool to belong to a block or a sub-block", "Verify the status of the nodes and the partitions", "Create a Slice custom resource", "Monitor the status of the Slice custom resource", "Delete the slice")

• [Configure cluster access for kubectl](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl.md.txt)

(Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")

• [Slice](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)

(Sections: "Slice status", "Slice specification")

• [Schedule dynamic slices with Kueue and TAS](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/use-gke-dynamic-slicing.md.txt)

(Sections: "Target your node pool to belong to a block or a sub-block", "Create a dynamic slice with Kueue and TAS", "Define the partition health selection", "Optimize workload scheduling with slice reuse", "Enable Slice reuse", "Enforce slice deletion per workload", "Monitor the status of the slice", "Monitor the metrics of the slice", "Clean up")

• [GKE system metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes.md.txt)

(Sections: "Kubernetes metrics")

• [Monitored resource types](https://docs.cloud.google.com/monitoring/api/resources.md.txt)

(Sections: "k8s_node_pool")

• [About GKE dynamic slicing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dynamic-slicing.md.txt)

(Sections: "Dynamic slicing configurations")

• [Get support](https://docs.cloud.google.com/kubernetes-engine/docs/getting-support.md.txt)

(Sections: "Before you contact support", "Recommendations for efficient support")

> **Read-only rule**: Run read-only diagnostic commands only. Never drain, > delete, or re-create nodes, or run any other command that changes the cluster. > Give the user any fix to apply themselves.

When you recommend a fix, link the doc section that describes it.

---

Diagnostic workflow

Step 0: Collect context `[Low Risk]`

Collect the target parameters:

• `{project_id}`: Google Cloud project ID
• `{cluster_name}`: GKE cluster name
• `{location}`: Cluster region or zone
• `{slice_name}`: Name of the Slice custom resource that failed to form
• `{node_pool_name}`: Name of the TPU node pool (or node pools) backing the

partitions in `{slice_name}`, if known. If the user doesn't know `{node_pool_name}`, after Step 1 run the Cloud Monitoring query in Step 3 for the partition IDs in `{partition_ids}` and read the `node_pool_name` label on the [k8s_node_pool](https://docs.cloud.google.com/monitoring/api/resources.md.txt) monitored resource.

• `{issue_time}`: Incident timestamp in RFC3339 UTC, if known. If the user

doesn't know `{issue_time}`, use the `Last Transition Time` (`lastTransitionTime`, "A timestamp of the most recent status change" in [Slice status](https://docs.cloud.google.com/kubernetes-engine/docs/reference/crds/slice.md.txt)) of the `Ready` condition from Step 1. By default, query Cloud Monitoring over a 60-minute window around `{issue_time}`.

---

Step 1: Confirm the `SliceCreationFailed` status `[Low Risk]`

To describe `{slice_name}`, follow the instructions in the section [Monitor the status of the Slice custom resource](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/create-dynamic-slices.md.txt).

• In `Status` > `Conditions`, check