/gke-observability

name: gke-observability
description: >-
  Configures GKE observability, including Cloud Logging, Cloud Monitoring, and
  managed Prometheus. Use when configuring GKE monitoring, setting up GKE
  logging, or configuring Prometheus metrics collection, to diagnose GKE control
  plane, API server, admission webhook, etcd storage, node registration or
  label, and pod eviction, preemption, or scheduling issues using GKE metrics,
  Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for
  Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets,
  PodMonitoring misconfiguration, rule/alert evaluation failures, and monitoring
  permission errors. Don't use to configure local application logging frameworks
  or external APMs outside GKE.
metadata:
  version: "1.2.1"
  category: CloudObservabilityAndMonitoring

GKE Observability

This reference covers monitoring, logging, and metrics configuration and troubleshooting for Google Kubernetes Engine (GKE). The golden path enables comprehensive observability including control-plane metrics and Managed Service for Prometheus (GMP).

> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`, > `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud > container clusters update --monitoring=...`, `gcloud logging read`

Usage Attribution

Do not run live `kubectl`, `gcloud`, or `curl` commands unless the user names a target cluster/project or asks to inspect a live cluster (output commands and queries with placeholders, and skip [Usage Attribution](references/attribution.md)). Before running any live command, read [Usage Attribution](references/attribution.md), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)"`, and append `agent-skills/1.0 (skill:gke-observability)` to `User-Agent` headers (never in generated scripts, Terraform, or user-facing code).

Sub-Skill Reference Routing (Progressive Disclosure)

> **Complete Reference Index** (no `scripts/` directory or unlisted reference > files exist — do not run `ls` on the skill folder; read only the matching > reference below when answering deep diagnostic, metric, log, or GMP questions):

1. **[Control-Plane Metrics, Storage, Webhooks & Capacity](references/control-plane-metrics-and-storage.md)**: Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not publicly exposed) and customer-visible alternatives; cluster etcd / etcd-compatible storage left, quota & per-resource scalability limits; admission webhook latency, system namespace/Lease/upgrade exclusions, `APIService` discovery bloat, client discovery caching bursts (`/apis/custom.metrics.k8s.io`, `/apis/metrics.k8s.io`) & proactive GKE Recommender insights; watch storms; controller-manager workqueue & scheduler lag; or control-plane headroom, VM autoscaling & node-count pre-scaling, network egress, connections (`65k`, no per-replica metric) & Protobuf. 2. **[Cloud Logging, Audit Attribution, APF & API Server Traces](references/logging-audit-and-apiserver-traces.md)**: Read for `kube-apiserver` slow-request `Trace[` logs (`trace.go` & KEP-647); `httplog.go` client load (`user_agent`, `verb`, `apf_iseats`); Cloud Audit Log noisy-neighbor, heavy `LIST` & unindexed CRD `fieldSelector` scan tail-latency attribution; API Priority and Fairness (APF) `429`/`504`/`404` throttling; `409 Conflict` hot objects (`ConfigMap` -> `Lease` heartbeats) & `RenewDeadline / 2` leader-election timeouts; `pods/exec` saturation, Konnectivity (`No agent available`), disabled Cloud KMS keys, or in-cluster GCP IAM OAuth token webhook authorizer saturation. 3. **[Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP)](references/cluster-node-pod-health-and-gmp.md)**: Read for control-plane availability (`kubernetes.io/cluster/kcp_healthchecks` rate ratio & ruling out KCP outages); maintenance windows (`in_maintenance_window`), zonal resize downtime & cluster access errors (`EOF` / `masterAuthorizedNetworksConfig`); node readiness, registration (`AUTO_REPAIR_NODES`, Private Google Access) & persistent node-pool labels; pod startup timelines, crashing pod logs, `tolerationSeconds: 0` mass eviction & Autopilot preemption; or multi-burn-rate SLO alerting (`GlobalRules`) & GMP / Grafana / `PodMonitoring` manifests & troubleshooting.

Golden Path Observability Defaults

Setting | Golden Path Value | Notes --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----- `loggingConfig` components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging `monitoringConfig` components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane `managedPrometheusConfig.enabled` | `true` | Google-managed Prometheus `advancedDatapathObservabilityConfig.enableMetrics` | `true` | Dataplane V2 flow metrics `loggingService` | `logging.googleapis.com/kubernetes`