/gke-workload-troubleshooting

name: gke-workload-troubleshooting
metadata:
  category: Containers
description: >-
  Diagnoses GKE workload failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, Pending, etc.) via logs and events. Use when pods fail to start or crash repeatedly. Don't use for GKE cluster infrastructure provisioning, node pool creation, or non-Kubernetes Google Cloud services.

GKE Workload Troubleshooting Skill

Use this skill to systematically diagnose and resolve failures in application workloads deployed in GKE clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary: it only **proposes** fixes — whether Kubernetes manifest/config patches or Google Cloud changes (for example `gcloud` IAM bindings or node-pool recreation) — and never executes live mutations itself.

🔍 Diagnostic Workflow

Step 0: Non-Interactive Context Discovery & Time Window Definition

1. **Parameter Extraction**: Extract required context (`project_id`, `cluster_name`, `cluster_location`, `workload_name`, `workload_namespace`) non-interactively from the user prompt, active `SETTINGS.md`, or active environment defaults:

• Default `workload_namespace` to `default` if omitted.
• Infer missing cluster parameters from active environment (`kubectl

config current-context` or `gcloud config get-value project`).

• Prioritize non-interactive context discovery from prompts and

environment defaults to ensure autonomous execution flow.

2. **Cluster Credentials & Fallback Mode**:

• Attempt credential fetch: `gcloud container clusters get-credentials

{cluster_name} --region/--zone {cluster_location}`

• **Fallback / Dry-Run Mode**: If the cluster is unreachable,

non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):

• Limit retry attempts to avoid resource exhaustion and context

overflow in unreachable cluster scenarios.

• Immediately present the exact sequence of `kubectl` diagnostic

commands for the human operator to run.

• Synthesize the root cause analysis and output the proposed GitOps

manifest fix based on the reported symptoms.

3. **Time Handling & Fallbacks**:

• **Determine Issue Timestamp ({issue_time})**:
• **Specific Time Provided**: If the user provides a specific

timestamp, use it as `{issue_time}`.

• **Relative Time Provided (e.g., "5 minutes ago")**: Dynamically

calculate the corresponding UTC timestamp based on current system time, and use it as `{issue_time}`.

• **No Time Provided (Default)**: Use current system time as

`{issue_time}`.

• **Window Calculation**: Center a 1-hour query window around

`{issue_time}` (`start_time` = `{issue_time} - 30m`, `end_time` = `{issue_time} + 30m`).

--------------------------------------------------------------------------------

Step 1: Analyze Pod Status and Conditions

Inspect the workload's active pod states and controller status.

**Diagnostic Commands:**

# 1. Inspect the deployment's actual selector labels:
kubectl get deployment {workload_name} -n {workload_namespace} -o jsonpath='{.spec.selector.matchLabels}'
# 2. Query the pods using the returned labels, for example:
kubectl get pods -l {selector_labels} -n {workload_namespace}
kubectl get deploy/{workload_name} -n {workload_namespace} -o yaml

Diagnostic Decision Tree:

• **Phase: Pending**:
• The Pod cannot schedule on any node. Proceed directly to **Step 2 (Query

Namespace Events)**.

• **State: CrashLoopBackOff / Error**:

• The container boots but exits repeatedly; the `kubelet` restarts it with

an increasing back-off delay of up to five minutes. First read the terminated **reason** and **exit code**:

    kubectl describe pod {pod_name} -n {workload_namespace}
    kubectl get pod {pod_name} -n {workload_namespace} -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

• **Reason: OOMKilled (Exit Code 137)**: The container's memory limit was

reached. Proceed to **Step 3 (Inspect Logs) → OOM Analysis** to classify container-level vs node-level, then **Step 5** to propose fixes.

• **Exit Code 0 (successful exit)**: Unexpected for a long-running

Deployment/StatefulSet — `restartPolicy: Always` restarts the finished process, creating the loop. Common causes: the `command`/`entrypoint` does not start a persistent process, a worker exits on an empty queue, or a missing/invalid config (e.g., an unattached or mis-keyed `ConfigMap` volume) makes the app exit cleanly. Proceed to **Step 3 (Inspect Logs)**.

• **Exit Code 128**: Invalid `command`/`entrypoint` — the executable path

is wrong or absent in the image. Verify the container command in the manifest.

• **Exit Code 1 or other non-zero**: The application crashed —

configuration errors, missing/invalid env vars or config files, unreachable dependencies, or auth failures (`401`/`403`) on Google Cloud calls (check the Pod's IAM / Workload Identity Federation). Proceed directly to **Step 3 (Inspect Logs)**.

• If the exit code looks healthy but the container keeps restarting,

suspect a **liveness probe failure** (see Step 3).

• **State: ImagePullBackOff / ErrImagePull**:

• The kubelet cannot pull the container image. `ImagePullBackOff` means it

keeps retrying with back-off; `ErrImagePull` is a general, non-recoverable pull error. Related statuses: `InvalidImageName`, `RegistryUnavailable`, `SignatureValidationFailed`, `ImageInspectError`. Proceed to **Step 2 (Query Namespace Events)** to read the exact pull error message.

• **S