/gke-reliability

name: gke-reliability
description: >-
  Improves GKE workload reliability, using PDBs, health probes, and topology
  spread constraints. Use when configuring GKE workload reliability, setting up
  PDBs, or configuring GKE health probes (liveness, readiness, startup). Don't
  use for disaster recovery setup or full cluster backups (use gke-backup-dr
  instead).
metadata:
  category: Containers

GKE Reliability

This reference covers high availability and reliability configuration for GKE clusters and workloads.

> **MCP Tools:** `get_cluster`, `get_k8s_resource`, `describe_k8s_resource`, > `apply_k8s_manifest`, `list_k8s_events`

Golden Path Reliability Defaults

| Setting | Golden Path Value | Notes | | ---------------- | --------------------- | -------------------------------- | | Cluster type | Regional (4 zones: | Control plane replicated across | : : us-central1-a/b/c/f) : zones : | Upgrade strategy | SURGE (`maxSurge: 1`) | Rolling upgrades with extra | : : : capacity : | Auto-repair | `true` | Unhealthy nodes replaced | : : : automatically : | Auto-upgrade | `true` | Nodes follow control plane | : : : version : | Release channel | REGULAR | Balanced freshness and stability | | Stateful HA | Enabled | Leader election for stateful | : : : workloads :

Workflows

1. Verify Cluster High Availability

# MCP (preferred)
get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",
  readMask="location,locations,nodePools.locations")

# gcloud fallback
gcloud container clusters describe <CLUSTER> --region <REGION> \
  --format="json(location, locations)" \
  --quiet

• If `location` is a region (e.g., `us-central1`), the control plane is

regional

• If `locations` has multiple entries, nodes span multiple zones

2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).

**Check existing PDBs:**

# MCP (preferred)
get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

# kubectl fallback
kubectl get pdb --all-namespaces

**Create PDB:**

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

> Every production Deployment with 2+ replicas should have a PDB.

3. Health Probes

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.

**Check existing probes:**

# MCP (preferred)
describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

# kubectl fallback
kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

**Recommended probe configuration:**

spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time

• **Readiness**: Determines when a pod can accept traffic
• **Liveness**: Determines when to restart a container
• **Startup**: Disables liveness/readiness until the app is ready (prevents

premature restarts)

4. Graceful Shutdown

Ensure applications handle `SIGTERM` and drain in-flight requests:

spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

• **Zone spread** (`DoNotSchedule`): Hard requirement -- pods must be balanced

across zones

• **Node spread** (`ScheduleAnyway`): Best-effort -- prefer distribution but

don't block scheduling

6. Replicas

| Workload Type | Minimum Replicas | Reason | | -------------------- | -------------------- | ------------------------------ | | Stateless web/API | 2 | Survive single pod/node | : : : failure : | Critical services | 3 | Survive zone failure with zone | : : : spread : | Stateful (databases) | 3 (with replication) | Application-level quorum | | Batch/jobs | 1 | Ephemeral by nature |

Best Practices & Production Guidelines

1. **Regional clusters for production**: Always use regional clusters to survive zone failures. 2. **PDBs fo