/operating-kubernetes

name: operating-kubernetes
description: Operating production Kubernetes clusters effectively with resource management, advanced scheduling, networking, storage, security hardening, and autoscaling. Use when deploying workloads to Kubernetes, configuring cluster resources, implementing security policies, or troubleshooting operational issues.

Kubernetes Operations

Purpose

Operating Kubernetes clusters in production requires mastery of resource management, scheduling patterns, networking architecture, storage strategies, security hardening, and autoscaling. This skill provides operations-first frameworks for right-sizing workloads, implementing high-availability patterns, securing clusters with RBAC and Pod Security Standards, and systematically troubleshooting common failures.

Use this skill when deploying applications to Kubernetes, configuring cluster resources, implementing NetworkPolicies for zero-trust security, setting up autoscaling (HPA, VPA, KEDA), managing persistent storage, or diagnosing operational issues like CrashLoopBackOff or resource exhaustion.

When to Use This Skill

**Common Triggers:**

• "Deploy my application to Kubernetes"
• "Configure resource requests and limits"
• "Set up autoscaling for my pods"
• "Implement NetworkPolicies for security"
• "My pod is stuck in Pending/CrashLoopBackOff"
• "Configure RBAC with least privilege"
• "Set up persistent storage for my database"
• "Spread pods across availability zones"

**Operations Covered:**

• Resource management (CPU/memory, QoS classes, quotas)
• Advanced scheduling (affinity, taints, topology spread)
• Networking (NetworkPolicies, Ingress, Gateway API)
• Storage operations (StorageClasses, PVCs, CSI)
• Security hardening (RBAC, Pod Security Standards, policies)
• Autoscaling (HPA, VPA, KEDA, cluster autoscaler)
• Troubleshooting (systematic debugging playbooks)

Resource Management

Quality of Service (QoS) Classes

Kubernetes assigns QoS classes based on resource requests and limits:

**Guaranteed (Highest Priority):**

• Requests equal limits for CPU and memory
• Never evicted unless exceeding limits
• Use for critical production services

resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"  # Same as request
    cpu: "500m"

**Burstable (Medium Priority):**

• Requests less than limits (or only requests set)
• Can burst above requests
• Evicted under node pressure
• Use for web servers, most applications

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"  # 2x request
    cpu: "500m"

**BestEffort (Lowest Priority):**

• No requests or limits set
• First to be evicted under pressure
• Use only for development/testing

Decision Framework: Which QoS Class?

| Workload Type | QoS Class | Configuration | |---------------|-----------|---------------| | Critical API/Database | Guaranteed | requests == limits | | Web servers, services | Burstable | limits 1.5-2x requests | | Batch jobs | Burstable | Low requests, high limits | | Dev/test environments | BestEffort | No limits |

Resource Quotas and LimitRanges

Enforce multi-tenancy with ResourceQuotas (namespace limits) and LimitRanges (per-container defaults):

# ResourceQuota: Namespace-level limits
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

For detailed resource management patterns including Vertical Pod Autoscaler (VPA), see `references/resource-management.md`.

Advanced Scheduling

Node Affinity

Control which nodes pods schedule on with required (hard) or preferred (soft) constraints:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - g4dn.xlarge  # GPU instance

Taints and Tolerations

Reserve nodes for specific workloads (inverse of affinity):

# Taint GPU nodes to prevent non-GPU workloads
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

# Pod tolerates GPU taint
tolerations:
- key: "workload"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

Topology Spread Constraints

Distribute pods evenly across failure domains (zones, nodes):

topologySpreadConstraints:
- maxSkew: 1  # Max difference in pod count
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app

For advanced scheduling patterns including pod priority and preemption, see `references/scheduling-patterns.md`.

Networking

NetworkPolicies (Zero-Trust Security)

Implement default-deny security with NetworkPolicies:

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# Allow specific ingress (frontend → backend)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Ingress vs. Gateway API

**Ingress (Legacy):**

• Widely supported, mature ecosystem
• Limited expressiveness
• Use for existing applications

**Gateway API (Modern):**

• Role-oriented design (cluster ops vs. app devs)
• More expressive (HTTPRoute, TCPRoute, TLSRoute)
• Recommended for new applications (GA in Kubernetes 1.29+)

# Gateway API example
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
spec:
  parentRefs:
  - name: production-gateway