/gke-alert-configuration

name: gke-alert-configuration
metadata:
  category: CloudInfrastructure
  canonical_source: https://github.com/google/skills/tree/main/skills/cloud/gke-alert-configuration
description: >-
  Configures alerting policies in Terraform for Google Kubernetes Engine (GKE)
  clusters, workloads, and services using PromQL and Google Cloud Managed Service
  for Prometheus. Use when writing, analyzing, validating, or deploying Terraform
  alerting policies to monitor GKE service latency, traffic, error rates using
  Multi-Window Multi-Burn-Rate SLO alerts, memory saturation, and cluster health
  such as CrashLoopBackOff and Node NotReady conditions.
  Don't use for non-GKE compute runtimes such as standalone Compute Engine VMs or
  standalone Cloud Run services without GKE.

GKE Alert Configuration

This skill provides guidelines and best practices for creating robust, high-signal alerting policies for Google Kubernetes Engine workloads using Google Cloud Managed Service for Prometheus and Terraform. It ensures comprehensive coverage of the **4 Golden Signals** and key cluster health metrics while minimizing alert noise.

--------------------------------------------------------------------------------

Critical Rules

• **Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes**:
• This skill is strictly scoped to Google Kubernetes Engine (GKE)

workloads, clusters, and services using PromQL and Google Cloud Managed Service for Prometheus.

• **Do not use for non-GKE compute runtimes**, such as standalone Compute

Engine virtual machines or standalone Cloud Run services without GKE.

• **STOP AND RESPOND DIRECTLY (Do Not Edit Files)**: When the user

requests alert configuration for non-GKE compute infrastructure: 1. **Do not write, create, edit, or validate any Terraform files on disk**. 2. **Immediately stop and respond directly to the user in chat**:

• **Explicitly Clarify Out-of-Scope**: State clearly that

standalone Compute Engine virtual machine monitoring or standalone Cloud Run monitoring is out of scope for this GKE-specific PromQL alerting skill, which is designed specifically for GKE workloads using Google Cloud Managed Service for Prometheus and PromQL.

• **Do Not Generate GKE PromQL Alerts**: Do not create or generate

Kubernetes PromQL alert policies or fabricate Kubernetes container, pod, or node resources for non-GKE infrastructure.

• **Redirect the User**: Guide and redirect the user to standard

Google Cloud Monitoring metrics, such as `compute.googleapis.com/instance/cpu/utilization` or `run.googleapis.com/request_latencies`, using standard `google_monitoring_alert_policy` with `condition_threshold` or MQL, or recommend the relevant specialized Cloud observability skill.

• **Mandatory `kube-state-metrics` (KSM) Cost Guardrail**:
• Deploying open-source `kube-state-metrics` in Google Cloud Managed

Service for Prometheus incurs billable metric ingestion costs.

• **STOP AND ASK PERMISSION FIRST (Do Not Edit Files)**: When a requested

alert rule relies on **Tier 2 KSM metrics** (such as `kube_cronjob_*`, `kube_pod_status_phase`, `kube_persistentvolume_*`, `kube_deployment_*`, `kube_statefulset_*`, `kube_job_*`, or `kube_daemonset_*`), **do not write, create, edit, or validate any Terraform files or generate alert policies before obtaining user approval**.

• Instead, you **must immediately stop and respond directly to the user**

to: 1. **Alert the user** that the requested alert requires `kube-state-metrics`. 2. **Explain the cost impact**: Detail that `kube-state-metrics` incurs billable sample ingestion costs in Google Cloud Managed Service for Prometheus. 3. **Ask for explicit permission**: Ask the user for explicit permission before assuming, enabling, or generating KSM-dependent alert configurations. 4. **Recommend filtering or allowlisting**: Suggest and recommend filtering or allowlisting only the specific required metrics, such as using a `PodMonitoring` resource with `metricRelabeling` (`action: keep`) or KSM `--metric-allowlist` to minimize ingestion costs. Provide a concrete allowlist example.

• **Always prefer Non-KSM Native Alternatives** (Tier 1 cAdvisor or native

GKE metrics documented in [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)) whenever possible, such as using `container_memory_working_set_bytes` and `container_spec_memory_limit_bytes` instead of `kube_pod_container_resource_limits`.

• **Explicit Tier and Cost Surcharge Identification in Response**: In

every response where you generate or recommend an alerting policy, you **must explicitly state its classification tier and cost impact**:

• **Tier 1 native or standard metric** (GKE built-in metrics, cAdvisor

`container_*`, kubelet volume stats, kubelet node conditions, and control-plane metrics; see [metrics_and_alerts_catalog.md](references/metrics_and_alerts_catalog.md)): State that it is a **Tier 1 native or standard metric with zero KSM cost surcharge**.

• **Tier 2 KSM metric**: State that it is a **Tier 2 KSM-dependent

metric** and follow the permission and allowlisting guardrail above. *(Tip: Generally, metrics with the `kube_` prefix that represent resource state or metadata belong to Tier 2).*

• **Plan-Validate-Execute Loop for Appr