/gke-tpu-dynamic-slices-monitoring

name: gke-tpu-dynamic-slices-monitoring
description: >-
  Monitors, troubleshoots, and manages GKE TPU Dynamic Slices custom resources. Use when checking TPU slice lifecycle states, troubleshooting slice provisioning failures, validating single-slice or multi-slice (JobSet) workload manifests, or safely patching stuck finalizers and disabling the slice controller. Don't use for generic GKE cluster node pool creation or standard non-TPU workload management (use gke-basics or gke-cluster-creation instead).
metadata:
  category: Containers

GKE TPU Dynamic Slices Monitoring & Management

Monitors the status of TPU Slice custom resources, troubleshoots provisioning failures, validates workload manifests on dynamic slices, and performs cleanups.

Prerequisites

• Cloud Logging enabled for the project.
• `kubectl` and `gcloud` CLIs configured to access the GKE cluster.

Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Gather project, cluster, and slice context using cluster tools or the following parameters:

• **Project ID**: `{project_id}` (e.g., `my-gcp-project`)
• **Cluster Name**: `{cluster_name}` (e.g., `tpu-cluster`)
• **Region/Zone**: `{location}` (e.g., `us-central1-a`)
• **Slice Name**: `{slice_name}` (e.g., `test-slice`)
• **Issue Time**: `{timestamp}` (Optional; default to the last 30 minutes

window `[T - 30m]` to `[T + 30m]`)

--------------------------------------------------------------------------------

Step 1: Describe the Slice Custom Resource [Low Risk]

When asked to inspect, troubleshoot, or check a slice status, immediately execute `kubectl describe slice {slice_name}` using available cluster tools to perform the inspection. Parse the resulting `Status.Conditions` output against the condition table below to diagnose the exact state and provide concrete recommendations.

• **Command**:

    kubectl describe slice {slice_name}

State & Reason Analysis

Analyze the `Status.Conditions` (especially `Type: Ready` and its `Reason` and `Status`):

| Lifecycle State / Reason | Meaning | Recommended Action | | :------------------------ | :--------------------- | :---------------------- | | **`SliceNotCreated`** | GKE Slice Controller | Wait a few minutes and | : : is initializing the : re-check slice status. : : : slice and performing : : : : resource checks. : : | **`SliceCreationFailed`** | Prerequisites | Verify selected nodes | : : validation failed : exist, are unallocated, : : : (e.g., selected nodes : and topology matches : : : don't exist, nodes are : partition count. : : : already used by : : : : another slice, or the : : : : topology doesn't match : : : : the number of : : : : partitions). : : | **`ACTIVATING`** | GKE is actively | Monitor node | : : forming and : provisioning. : : : provisioning the TPU : : : : slice. : : | **`ACTIVE`** | The TPU slice is | Proceed to deploy or | : : successfully formed : check workloads. : : : and ready to host : : : : workloads. : : | **`ACTIVE_DEGRADED`** | The slice is usable, | Monitor workload logs | : : but one or more : for interconnect or : : : sub-blocks are : device errors. Check : : : degraded. : faulty node VMs. : | **`FAILED`** | GKE failed to form the | Ensure all selected | : : TPU slice (e.g., : nodes belong to the : : : selected nodes are not : same reservation block. : : : part of the same : : : : reservation block). : : | **`DEACTIVATING`** | The slice is | Wait for dismantling to | : : dismantling (triggered : finish, or patch : : : by user deletion or a : finalizers if stuck. : : : critical systemic : : : : failure). : : | **`INCOMPLETE`** | The terminal phase | No action required; the | : : before the Slice CR is : resource will be : : : deleted from the : removed shortly. : : : cluster. : :

Provisioning Failure Troubleshooting Checklist

When investigating slice creation or provisioning failures (`SliceCreationFailed` or `FAILED`), perform the following verification steps:

1. **Node Existence & Allocation Check**: Verify that the selected TPU nodes exist in the cluster and are not already allocated to another slice (`kubectl get nodes -l cloud.google.com/gke-tpu-slice`, `kubectl get slice -A`). 2. **Topology Alignment**: Confirm that the partition count matches the requested topology dimensions (e.g. topology `2x2` requires 4 nodes). 3. **Reservation Block Alignment