/devops-k8s-cluster

name: devops-k8s-cluster
description: Use when building, upgrading, repairing or choosing the Kubernetes cluster itself - kubeadm, k3s or Talos (RKE2 and Cluster API as pointers), HA control plane and API endpoint, etcd health/backup/restore, version skew and upgrade order, node join/drain/replace/remove, certificate expiry, Cilium as CNI (kube-proxy replacement, Gateway API support), bare-metal LoadBalancer IPs (Cilium LB IPAM and L2 announcements, MetalLB, kube-vip) and cluster-level troubleshooting (NotReady nodes, etcd quorum loss, API server down, CNI broken). Not for workloads, Services, Gateway resources, RBAC or day-to-day kubectl (devops-kubernetes), Helm/Kustomize/Argo CD/Flux (devops-gitops-packaging), CSI backend internals (devops-storage-backup) or VM provisioning (devops-proxmox, devops-virtualization).
versions:
  kubernetes: "1.37.2 (supported minors 1.37, 1.36, 1.35)"
  k3s: "v1.37.2+k3s1"
  talos: "v1.14.5"
  cilium: "1.20.4"
  metallb: "v0.16.2"
  etcd: "v3.7.3 (GitHub tag, single source)"
  kubeadm-config-api: "kubeadm.k8s.io/v1beta4"
user-invocable: true
references:
  - references/distribution-choice.md
  - references/kubeadm-ha-bootstrap.md
  - references/k3s-cluster.md
  - references/talos-cluster.md
  - references/etcd-backup-restore.md
  - references/version-skew-upgrades.md
  - references/node-lifecycle.md
  - references/cni-cilium.md
  - references/baremetal-loadbalancing.md
  - references/cluster-troubleshooting.md
  - references/templates/kubeadm-ha-config.yaml
  - references/templates/k3s-server-config.yaml
  - references/templates/talos-patches.yaml
  - references/templates/cilium-values.yaml
  - references/templates/cilium-lb-ipam-l2.yaml
  - references/templates/metallb-l2.yaml
  - references/templates/etcd-snapshot-restore.md
  - references/templates/cluster-upgrade-checklist.md
related-skills:
  - devops-kubernetes
  - devops-gitops-packaging
  - devops-storage-backup
  - devops-networking
  - devops-linux
  - devops-proxmox
  - devops-virtualization
  - devops-openstack
  - devops-iac
  - devops-ansible
  - devops-observability
  - devops-secrets
  - devops-sre-ops

<objective> Covers the cluster itself: choosing a distribution, bootstrapping an HA control plane, etcd backup/restore/health, version skew and upgrades, node lifecycle, certificate expiry, Cilium as the default CNI, bare-metal LoadBalancer IPs and cluster-level troubleshooting. Does not cover workloads, Services, Gateway resources, RBAC or kubectl usage (devops-kubernetes), packaging and GitOps (devops-gitops-packaging), CSI backends (devops-storage-backup) or VM provisioning (devops-proxmox, devops-virtualization). </objective>

DevOps Kubernetes Cluster

Agent Workflow (MANDATORY)

Before ANY cluster change, spawn in parallel, one `Agent` call each with a `name`:

1. **fuse-ai-pilot:explore-codebase** - find existing cluster config (kubeadm YAML, k3s `config.yaml`, Talos patches, inventories, Terraform/Ansible that built the nodes) 2. **fuse-ai-pilot:research-expert** - verify flags and versions at kubernetes.io, docs.k3s.io, docs.siderolabs.com, docs.cilium.io, etcd.io 3. **mcp__context7__query-docs** - `/websites/etcd_io`, `/websites/cilium_io_en_stable`, `/websites/kubernetes_io`

Then inspect read-only first (`kubectl get nodes -o wide`, `kubectl version`, etcd health, certificate expiry). After changes to files, run **fuse-ai-pilot:sniper**.

Versions (checked 2026-11-10)

Kubernetes v1.37.2 / v1.36.6 / v1.35.10 are the latest patches; supported minors are 1.37, 1.36, 1.35 (https://kubernetes.io/releases/version-skew-policy/). k3s v1.37.2+k3s1, Talos v1.14.5 (docs.siderolabs.com/talos/v1.14), Cilium 1.20.4, MetalLB v0.16.2. etcd v3.7.3 comes from a GitHub tag only; the Kubernetes etcd page names 3.4.29+ and 3.5.11+ as the minimum for production. Cilium documents Gateway API CRDs v1.6.3 although Gateway API v1.7.0 exists: install the version Cilium documents. Kubernetes 1.37 kubelet defaults to cgroup v2 only (`FailCgroupV1: true`).

Overview

| Area | Decision in one line | |------|----------------------| | **Distribution** | kubeadm = upstream, you own OS and etcd; k3s = single binary, embedded etcd HA; Talos = immutable, API-only, no SSH; RKE2 and Cluster API are pointers ([distribution-choice.md](references/distribution-choice.md)) | | **Control plane** | 3 stacked nodes minimum behind a TCP endpoint (VIP or load balancer); kubeadm `controlPlaneEndpoint` must equal that address | | **etcd** | Odd member count; 3 tolerate 1 loss, 5 tolerate 2; snapshot on a schedule, off the node, restore-tested | | **Upgrades** | One minor at a time, API server first, kubelet may lag up to three minors; drain before a minor kubelet upgrade | | **CNI** | Cilium by default; install it before joining more nodes; kube-proxy replacement needs `k8sServiceHost` and `k8sServicePort` | | **Bare-metal LB** | Cilium LB IPAM + L2 announcements if Cilium is the CNI; MetalLB otherwise; kube-vip for the control-plane VIP | | **Troubleshooting** | kubelet and container runtime first, then etcd, then API server, then CNI ([cluster-troubleshooting.md](references/cluster-troubleshooting.md)) |

Critical Rules

1. **Version first** - read server, kubelet and etcd versions before any upgrade; never skip a minor ([version-skew-upgrades.md](references/version-skew-upgrades.md)) 2. **Backup before touching etcd** - a fresh snapshot, copied off the node, with `etcdutl snapshot status` output attached ([etcd-backup-restore.md](references/etcd-backup-restore.md)) 3. **Destructive verbs are proposed, not run** - `kubeadm reset`, `etcdutl snapshot restore`, `etcdctl member remove`, `talosctl reset`, `talosctl bootstrap --recover-from`, `k3s server --cluster-reset`, `kubectl delete node`, `k3s-uninstall.sh`: show ACTION / EFFECT / QUORUM / PROOF and wait for the user's go 4. **Quorum arithmetic before any member change** - state members, quorum and how many can still fail after the action 5. **One contro