/gke-observability

name: gke-observability
description: >-
  Configures GKE observability, including Cloud Logging, Cloud Monitoring, and
  managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging,
  or configuring Prometheus metrics collection. Don't use to configure local
  application logging frameworks or external APMs outside GKE.
metadata:
  category: CloudObservabilityAndMonitoring

GKE Observability

This reference covers monitoring, logging, and metrics configuration for GKE. The golden path enables comprehensive observability including control-plane metrics.

> **MCP Tools:** `get_cluster`, `list_k8s_events`, `get_k8s_logs`, > `get_k8s_cluster_info`, `describe_k8s_resource`. **CLI-only:** `gcloud > container clusters update --monitoring=...`, `gcloud logging read`

Golden Path Observability Defaults

Setting | Golden Path Value | Notes --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----- `loggingConfig` components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging `monitoringConfig` components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane `managedPrometheusConfig.enabled` | `true` | Google-managed Prometheus `advancedDatapathObservabilityConfig.enableMetrics` | `true` | Dataplane V2 flow metrics `loggingService` | `logging.googleapis.com/kubernetes` | Cloud Logging `monitoringService` | `monitoring.googleapis.com/kubernetes` | Cloud Monitoring

Control-Plane Metrics (Golden Path Addition)

The golden path adds three control-plane monitoring components not present in default clusters:

| Component | What It Monitors | | -------------------- | ---------------------------------------------------------------------- | | `APISERVER` | API server request latency, error rates, admission webhook performance | | `SCHEDULER` | Scheduling latency, pending pods, scheduling failures | | `CONTROLLER_MANAGER` | Controller work queue depth, reconciliation latency |

These are critical for diagnosing cluster-level issues (slow API responses, scheduling delays, stuck controllers).

Enabling Full Monitoring

**Say this whenever you hand over a `--monitoring` command:**

1. **Control-plane metrics are NOT enabled by default.** State this outright in your answer — do not leave it implied by the fact that you are supplying an enable command. `API_SERVER`, `SCHEDULER`, and `CONTROLLER_MANAGER` are off on every new cluster and collect nothing until explicitly turned on, and the same is true of `DCGM`, `CADVISOR`, `KUBELET`, and kube-state (`POD`, `DEPLOYMENT`, `STATEFULSET`, `DAEMONSET`, `HPA`, `STORAGE`, `JOBSET`). `SYSTEM` is the only package on by default. A user asking "why are there no API server metrics" has almost always simply never enabled them. 2. **The flag replaces, it does not append.** The set supplied to `--monitoring` overrides the previous setting entirely, so omitting a component silently turns it off. Always pass the full desired list, and always include `SYSTEM` — it cannot be disabled while monitoring is on, and never on Autopilot. 3. **These metrics bill per sample ingested** via Managed Service for Prometheus. Enabling the full suite on a large cluster is a real cost increase; mention it rather than presenting the list as free.

> **The gcloud flag and the API field use different spellings for the same > components.** Do not copy names between them: > > Component | `gcloud --monitoring=` | `monitoringConfig` API enum > ---------------- | ---------------------- | --------------------------- > System | `SYSTEM` | `SYSTEM_COMPONENTS` > API server | `API_SERVER` | `APISERVER` > Controller mgr | `CONTROLLER_MANAGER` | `CONTROLLER_MANAGER` > > The remaining components share a spelling. Using an API enum in the CLI flag > (or the reverse) fails the command — this is a common and confusing error.

# Enable golden path monitoring suite
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
  --quiet

# Enable Managed Prometheus
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --enable-managed-prometheus \
  --quiet

# Enable Dataplane V2 observability metrics
gcloud container clusters update <CLUSTER_NAME> --region <REGION> \
  --enable-dataplane-v2-flow-observability \
  --quiet

Managed Prometheus

Golden path enables Google Managed Prometheus for metrics collection and querying.

**Querying metrics:**

• Use Cloud Monitoring Metrics Explorer in the c