/gke-storage-troubleshooting

name: gke-storage-troubleshooting
metadata:
  category: Storage
description: >-
  Diagnoses GKE persistent-storage failures — volume attach/mount errors (Regional PD on optimized VMs, fsGroup mount timeouts), disk-performance and node storage-pressure issues, slow-disk Pod-creation failures, volume-expansion problems, Local SSD / Hyperdisk Storage Pool creation errors, and Cloud Storage FUSE OOM. Use when Pods are stuck in ContainerCreating, volumes fail to attach or mount, or nodes report storage pressure. Don't use for routine storage provisioning or StorageClass/PVC authoring (see the gke-storage skill).

GKE Storage Troubleshooting Skill

Use this skill to systematically diagnose and resolve **persistent-storage failures** for workloads running on GKE — volume attach/mount errors, disk performance and node storage pressure, volume expansion, storage-related cluster/node-pool creation errors, and Cloud Storage FUSE memory issues. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or configuration corrections.

> For routine storage **provisioning** and StorageClass/PVC authoring, use the > `gke-storage` skill instead. This skill focuses on **failure diagnosis**.

🔍 Diagnosis & Resolution Workflow

Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1. **Parameter Extraction**: Extract required context (`project_id`, `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`, `pod_name`, and the relevant `pvc_name` / `pv_name` / `node_name`) non-interactively from the user prompt, active `SETTINGS.md`, or environment defaults:

• Default `workload_namespace` to `default` if omitted.
• Infer missing cluster parameters from the active environment (`kubectl

config current-context` or `gcloud config get-value project`).

2. **Cluster Credentials & Fallback Mode**:

• Attempt credential fetch: `gcloud container clusters get-credentials

{cluster_name} --location {cluster_location} --project {project_id}`.

• **Fallback / Dry-Run Mode**: If the cluster is unreachable,

non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):

• Limit retry attempts to avoid resource exhaustion and context

overflow.

• Immediately present the exact `kubectl` / `gcloud` diagnostic

commands for the human operator to run.

• Synthesize the root-cause analysis and output the proposed GitOps

correction based on the reported symptoms.

--------------------------------------------------------------------------------

Step 1: Classify the Storage Symptom

Gather the primary signals, then jump to the **matching branch under Step 2 (Resolution)** — you normally perform **only the one branch** that matches your diagnosis, not all of them.

**Diagnostic Commands:**

kubectl describe pod {pod_name} -n {workload_namespace}
kubectl get pvc,pv -n {workload_namespace}
kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
kubectl describe node {node_name}

• **Pod stuck in `ContainerCreating` with an attach/mount event** → **Volume

Attach & Mount Failures**.

• **Node-level slowness, `PLEG is not healthy`, or `StoragePressureDetected`

events** → **Disk Performance & Node Storage Pressure**.

• **Cluster / node-pool creation or provisioning error** → **Storage

Provisioning & Creation Failures**.

• **A resized volume is not reflected inside the container** → **Volume

Expansion Not Reflecting in the Container**.

• **Cloud Storage FUSE Pod / sidecar OOM** → **Cloud Storage FUSE

Out-Of-Memory (OOM) Events**.

--------------------------------------------------------------------------------

Step 2: Resolution

Perform **only the branch that matches your Step 1 diagnosis**. These branches are mutually exclusive alternatives, not sequential steps.

Volume Attach & Mount Failures

• **`Error 400: Cannot attach RePD to an optimized VM`**: Regional persistent

disks are restricted from being used with **memory-optimized** or **compute-optimized** machine types.

• If a regional PD is not a hard requirement, switch the workload to a

**non-regional persistent disk** StorageClass.

• If a regional PD **is** required, use **taints and tolerations** so that

Pods needing regional PDs are scheduled onto a node pool that does **not** use optimized machine types.

• **Pods stay `Pending` / `FailedScheduling` after a node pool is moved to a

4th-generation (N4, N4A, N4D) machine series while the workload uses a Persistent Disk StorageClass**: N4/N4A/N4D machines **do not support Persistent Disk** (they support Hyperdisk only), so a PVC bound to a `pd-*` StorageClass cannot bind or schedule on those nodes. Events typically show `FailedScheduling` with a volume node-affinity / topology conflict.

• Switch the workload to a **Hyperdisk** StorageClass (for example `type:

hyperdisk-balanced`) for the Gen4 node pool.

• For existing Persistent Disk volumes, migrate the data to a Hyperdisk

volume; the original PD cannot be attached to a Gen4 node.

• If the workload must keep Persistent Disk, keep it on a PD-capable

machine series (for example N2) via node selection. This is a machine-type/disk-type incompatibility, **not** a capacity problem, so increasing disk size or quota does not help.

• **Hyperdisk Pods become unschedulable when a compute class falls back across

VM generations (for example N4 priority, N2 fallback), or one StorageClass must serve mixed generations**: a single static disk type in the StorageClass is not compatible with every machine series in the fallback list, so Pods cannot bind their volume on the fallback nodes.

-