/gke-workload-scaling-troubleshooting

name: gke-workload-scaling-troubleshooting
metadata:
  version: "1.0.0"
  category: Containers
description: >-
  Diagnoses GKE HorizontalPodAutoscaler (HPA) failures — metrics showing as
  <unknown>, FailedGetResourceMetric / FailedGetScale / FailedComputeMetricsReplicas
  events, missing Pod resource requests, custom/external metrics-pipeline breakage
  (FailedGetExternalMetric / FailedGetCustomMetric, unavailable metrics adapter,
  control-plane firewall blocking the adapter), HPA that won't scale up or down
  (tolerance / stabilization window / unavailable rate metrics), scale-to/from-zero
  problems, and slow HPA reaction on large clusters. Use when an HPA isn't scaling a
  workload as expected or reports metric errors. Don't use for configuring or
  authoring new HPA/VPA objects or scaling best practices (see the
  gke-workload-scaling skill), or for Cluster Autoscaler / node-pool sizing.

GKE Workload Scaling Troubleshooting Skill

Use this skill to systematically diagnose and resolve **HorizontalPodAutoscaler (HPA) failures** on GKE — metrics reported as `<unknown>`, `FailedGet*` events, missing resource requests, custom/external metrics-pipeline breakage, HPA that refuses to scale up or down, scale-to/from-zero issues, and slow HPA reaction on large clusters. This skill operates non-interactively and enforces a read-only diagnostics boundary before proposing manifest or configuration corrections.

> For configuring HPA/VPA objects and scaling **best practices**, use the > `gke-workload-scaling` skill instead. This skill focuses on **failure > diagnosis**.

🔍 Diagnosis & Resolution Workflow

Step 0: Non-Interactive Context Discovery & Dry-Run Fallback

1. **Parameter Extraction**: Extract required context (`project_id`, `cluster_name`, `cluster_location`, `hpa_name`, `workload_name`, `workload_namespace`) non-interactively from the user prompt, active `SETTINGS.md`, or environment defaults:

• Default `workload_namespace` to `default` if omitted.
• Infer missing cluster parameters from the active environment (`kubectl

config current-context` or `gcloud config get-value project`).

2. **Cluster Credentials & Fallback Mode**:

• Attempt credential fetch: `gcloud container clusters get-credentials

{cluster_name} --location {cluster_location} --project {project_id}`.

• **Fallback / Dry-Run Mode**: If the cluster is unreachable,

non-existent, or live command execution fails (such as in sandboxed evaluations, dry-run mode, or offline analysis):

• Limit retry attempts to avoid resource exhaustion and context

overflow.

• Immediately present the exact `kubectl` / `gcloud` diagnostic

commands for the human operator to run.

• Synthesize the root-cause analysis and output the proposed GitOps

correction based on the reported symptoms.

--------------------------------------------------------------------------------

Step 1: Inspect the HPA and Classify the Symptom

Start every investigation with `kubectl describe hpa`, then route to the matching branch. The three key sections are **`Metrics`** (an `<unknown>` value means the HPA hasn't fetched the metric or the pipeline is broken), **`Conditions`** (`AbleToScale`, `ScalingActive`, `ScalingLimited` — a `False` status marks a failure), and **`Events`** (specific reasons such as `FailedGetScale` or `FailedGetResourceMetric`).

**Diagnostic Commands:**

kubectl describe hpa {hpa_name} -n {workload_namespace}
kubectl get hpa {hpa_name} -n {workload_namespace} -o yaml

For historical events, query Cloud Logging (the HPA events survive after the live `Events` list rolls over):

resource.type="k8s_cluster"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
logName="projects/{project_id}/logs/events"
jsonPayload.involvedObject.kind="HorizontalPodAutoscaler"

Route by signal:

• **`FailedGetScale`, `FailedComputeMetricsReplicas`, `Error 400 ... label

is not allowed`, or fluctuating replicas from competing HPAs** → **Branch A** (Configuration Errors).

• **`FailedGetResourceMetric`, `unable to fetch pod metrics`, or `multiple

services selecting the same target`** → **Branch B** (Workload & Service Errors).

• **`<unknown>` custom/external metric, `FailedGetExternalMetric` /

`FailedGetCustomMetric`, or `no known available metric versions found`** → **Branch C** (Metrics API & Data Availability).

• **Conditions all `True` / no errors but the workload won't scale up or down**

→ **Branch D** (Healthy but Unexpected Scaling).

• **Workload configured with `minReplicas: 0` won't scale to or from zero** →

**Branch E** (Scale To / From Zero).

• **Correct HPA but slow reaction on a cluster with many HPA objects** →

**Branch F** (Slow Recalculation on Large Clusters).

--------------------------------------------------------------------------------

Step 2: Resolution — Route to the Matching Branch

Based on the signal you classified in Step 1, jump to **one** of the mutually-exclusive branches below (A–F). These are alternatives — you do **not** run them in sequence. After applying the branch's fix, go to Step 3 to present it as a reviewable GitOps change.

Branch A: HorizontalPodAutoscaler Configuration Errors

• **`FailedGetScale` — `unable to get the target's current scale: ... "TARGET"

not found`**: the `scaleTargetRef` doesn't resolve to an existing scalable workload.

• Verify the `scaleTargetRef` `name`, `kind`, and `apiVersion` exactly

match the target workload's metadata.

• Confirm the target workload exists **in the same namespace** as the HPA

(a missing `-n` puts objects in `default`, causing a mismatch).

• The target must be a scalable kind (Deployment, StatefulSet,

ReplicaSet) — you **cannot** autoscale a DaemonSet.

• **`FailedC