/gke-workload-troubleshooting

name: gke-workload-troubleshooting
metadata:
  category: Containers
description: >-
  Diagnoses GKE workload failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, etc.) via logs and events. Use when pods fail to start or crash repeatedly. Don't use for GKE cluster infrastructure provisioning, node pool creation, or non-Kubernetes Google Cloud services.

GKE Workload Troubleshooting Skill

Use this skill to systematically diagnose and resolve failures in application workloads deployed in GKE clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or config corrections.

🔍 Diagnostic Workflow

Step 0: Non-Interactive Context Discovery & Time Window Definition

1. **Parameter Extraction**: Extract required context (`project_id`, `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`) non-interactively from the user prompt, active `SETTINGS.md`, or active environment defaults:

• Default `workload_namespace` to `default` if omitted.
• Infer missing cluster parameters from active environment (`kubectl

config current-context` or `gcloud config get-value project`).

• Prioritize non-interactive context discovery from prompts and

environment defaults to ensure autonomous execution flow.

2. **Cluster Credentials & Fallback Mode**:

• Attempt credential fetch: `gcloud container clusters get-credentials

{cluster_name} --region/--zone {cluster_location}`

• **Fallback / Dry-Run Mode**: If the cluster is unreachable,

non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):

• Limit retry attempts to avoid resource exhaustion and context

overflow in unreachable cluster scenarios.

• Immediately present the exact sequence of `kubectl` diagnostic

commands for the human operator to run.

• Synthesize the root cause analysis and output the proposed GitOps

manifest fix based on the reported symptoms.

3. **Time Handling & Fallbacks**:

• **Determine Issue Timestamp ({issue_time})**:
• **Specific Time Provided**: If the user provides a specific

timestamp, use it as `{issue_time}`.

• **Relative Time Provided (e.g., "5 minutes ago")**: Dynamically

calculate the corresponding UTC timestamp based on current system time, and use it as `{issue_time}`.

• **No Time Provided (Default)**: Use current system time as

`{issue_time}`.

• **Window Calculation**: Center a 1-hour query window around

`{issue_time}` (`start_time` = `{issue_time} - 30m`, `end_time` = `{issue_time} + 30m`).

--------------------------------------------------------------------------------

Step 1: Analyze Pod Status and Conditions

Inspect the workload's active pod states and controller status.

**Diagnostic Commands:**

# 1. Inspect the deployment's actual selector labels:
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
# 2. Query the pods using the returned labels, for example:
kubectl get pods -l {selector_labels} -n {workload_namespace}
kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml

Diagnostic Decision Tree:

• **Phase: Pending**:
• The Pod cannot schedule on any node. Proceed directly to **Step 2 (Query

Namespace Events)**.

• **State: CrashLoopBackOff / Error**:

• Container is booting but exiting repeatedly. Check the terminated status

using:

    kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

• **ExitCode: 137 (OOMKilled)**: Memory limit reached. Proceed to **Step 3

(Inspect Logs)** and inspect container startup command to differentiate between an application-level memory leak/loop vs an infrastructure capacity limit mismatch, then proceed to **Step 5** to propose fixes.

• **ExitCode: 1 or other non-zero codes**: The application code crashed.

Proceed directly to **Step 3 (Inspect Logs)**.

• **State: ContainerCreating**:

• The container is blocked during volume mount, networking setup, or image

pulling. Proceed directly to **Step 2 (Query Namespace Events)**.

--------------------------------------------------------------------------------

Step 2: Query Namespace Events

Look for infrastructure, volume, image, or scheduling alerts in GKE.

**Diagnostic Command:**

kubectl get events -n {workload_namespace} --sort-by='.metadata.creationTimestamp'
# Or query Cloud Logging for historical GKE events within the time window:
gcloud logging read "resource.type=\"k8s_cluster\" AND logName=\"projects/{project_id}/logs/events\" AND jsonPayload.involvedObject.namespace=\"{workload_namespace}\"" --start-time="{start_time}" --end-time="{end_time}" --project="{project_id}"

*Note: Retrieve the sorted events list and manually inspect the event timestamps (CreationTimestamp/LastSeen) to identify failures occurring within the `{start_time}` and `{end_time}` window.*

Signature Identifiers:

• **`FailedScheduling`**: Node resource exhaustion. Look for messages like

`0/3 nodes are available: 3 Insufficient memory.` or missing node affinity tolerations (e.g. Spot VM taints).

• **`FailedMount`**:
• Missing PersistentVolumeClaim (`PVC`).
• Missing Secret (`Secret "{secret_name}" not found`).
• Missing ConfigMap (`ConfigMap "{configmap_name}" not found`).
• **`Failed` / `BackOff` (Image Pull)**:
• Wrong image tag, missing image registry authentication (e.g.,

ImagePullBackOff).

• **Resolution Steps for Wrong Image Tag**:
• Identify the failing container image name and the invalid tag.