/gke-reliability

name: gke-reliability
description: >-
  Improves GKE workload reliability, using PDBs, health probes, and topology
  spread constraints. Use when configuring GKE workload reliability, setting up
  PDBs, or configuring GKE health probes (liveness, readiness, startup). Don't
  use for generating K8s YAML manifests (use gke-manifest-generation) or
  disaster recovery and cluster backups (use gke-backup-dr).
metadata:
  version: "1.0.2"
  category: Containers

GKE Reliability

> **Routing Note:** To generate Kubernetes YAML manifests (`Deployment`, `StatefulSet`, `Service`, `ConfigMap`, `HTTPRoute`, `PodDisruptionBudget`), open `gke-manifest-generation/SKILL.md`.

This reference covers high availability and reliability configuration for GKE clusters and workloads.

> **MCP Tools:** `get_cluster`, `get_k8s_resource`, `describe_k8s_resource`, > `apply_k8s_manifest`, `list_k8s_events`

Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-reliability)"` and append `agent-skills/1.0 (skill:gke-reliability)` to `User-Agent` headers.

Golden Path Reliability Defaults

| Setting | Golden Path Value | Notes | | ---------------- | --------------------- | -------------------------------- | | Cluster type | Regional (4 zones: | Control plane replicated across | : : us-central1-a/b/c/f) : zones : | Upgrade strategy | SURGE (`maxSurge: 1`) | Rolling upgrades with extra | : : : capacity : | Auto-repair | `true` | Unhealthy nodes replaced | : : : automatically : | Auto-upgrade | `true` | Nodes follow control plane | : : : version : | Release channel | REGULAR | Balanced freshness and stability | | Stateful HA | Enabled | Leader election for stateful | : : : workloads :

Workflows

1. Verify Cluster High Availability

# MCP (preferred)
get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",
  readMask="location,locations,nodePools.locations")

# gcloud fallback
gcloud container clusters describe <CLUSTER> --region <REGION> \
  --format="json(location, locations)" \
  --quiet

• If `location` is a region (e.g., `us-central1`), the control plane is

regional

• If `locations` has multiple entries, nodes span multiple zones

2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).

**Check existing PDBs:**

# MCP (preferred)
get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

# kubectl fallback
kubectl get pdb --all-namespaces

**Create PDB:**

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

> Every production Deployment with 2+ replicas should have a PDB.

3. Health Probes

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.

**Check existing probes:**

# MCP (preferred)
describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

# kubectl fallback
kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

**Recommended probe configuration:**

spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time

• **Readiness**: Determines when a pod can accept traffic
• **Liveness**: Determines when to restart a container
• **Startup**: Disables liveness/readiness until the app is ready (prevents

premature restarts)

4. Graceful Shutdown

Ensure applications handle `SIGTERM` and drain in-flight requests:

spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

• **Zone spread** (`DoNotSchedule`): Hard requirement -- pods must be balanced

across zones

• **Node spread** (`ScheduleAnyway`): Best-effort -- prefer distribution but

don't block scheduling

6. Replicas

| Workload Type | Minimum Replicas | Reason | | -------------------- | ----------