/k8s-debug

name: k8s-debug
description: Diagnose and fix Kubernetes pods, CrashLoopBackOff, Pending, DNS, networking, storage, and rollout failures with kubectl.

Kubernetes Debugging Skill

Overview

Systematic toolkit for debugging Kubernetes clusters, workloads, networking, and storage with a deterministic, safety-first workflow.

Trigger Phrases

Use this skill when requests resemble:

• "My pod is in `CrashLoopBackOff`; help me find the root cause."
• "Service DNS works in one pod but not another."
• "Deployment rollout is stuck."
• "Pods are `Pending` and not scheduling."
• "Cluster health looks degraded after a change."
• "PVC is pending and pods cannot mount storage."

Prerequisites

Run from the skill directory (`devops-skills-plugin/skills/k8s-debug`) so relative script paths work as written.

Required

• `kubectl` installed and configured.
• An active cluster context.
• Read access to namespaces, pods, events, services, and nodes.

Quick preflight:

kubectl config current-context
kubectl auth can-i get pods -A
kubectl auth can-i get events -A
kubectl get ns

Optional but Recommended

• `jq` for more precise filtering in `./scripts/cluster_health.sh`.
• Metrics API (`metrics-server`) for `kubectl top`.
• In-container debug tools (`nslookup`, `getent`, `curl`, `wget`, `ip`) for deep network tests.

Fallback behavior:

• If optional tools are missing, scripts continue and print warnings with reduced output.
• If `kubectl top` is unavailable, continue with `kubectl describe` and events.

When to Use This Skill

Use this skill for:

• Pod failures (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled)
• Service connectivity or DNS resolution issues
• Network policy or ingress problems
• Volume and storage mount failures
• Deployment rollout issues
• Cluster health or performance degradation
• Resource exhaustion (CPU/memory)
• Configuration problems (ConfigMaps, Secrets, RBAC)

Safety Rules for Disruptive Commands

Default mode is read-only diagnosis first. Only execute disruptive commands after confirming blast radius and rollback.

Commands requiring explicit confirmation:

• `kubectl delete pod ... --force --grace-period=0`
• `kubectl drain ...`
• `kubectl rollout restart ...`
• `kubectl rollout undo ...`
• `kubectl debug ... --copy-to=...`

Before disruptive actions:

# Snapshot current state for rollback and incident notes
kubectl get deploy,rs,pod,svc -n <namespace> -o wide
kubectl get pod <pod-name> -n <namespace> -o yaml > before-<pod-name>.yaml
kubectl get events -n <namespace> --sort-by='.lastTimestamp' > before-events.txt

Reference Navigation Map

Load only the section needed for the observed symptom.

| Symptom / Need | Open | Start section | | --- | --- | --- | | You need an end-to-end diagnosis path | `./references/troubleshooting_workflow.md` | `General Debugging Workflow` | | Pod state is `Pending`, `CrashLoopBackOff`, or `ImagePullBackOff` | `./references/troubleshooting_workflow.md` | `Pod Lifecycle Troubleshooting` | | Service reachability or DNS failure | `./references/troubleshooting_workflow.md` | `Network Troubleshooting Workflow` | | Node pressure or performance regression | `./references/troubleshooting_workflow.md` | `Resource and Performance Workflow` | | PVC / PV / storage class issues | `./references/troubleshooting_workflow.md` | `Storage Troubleshooting Workflow` | | Quick symptom-to-fix lookup | `./references/common_issues.md` | matching issue heading | | Post-mortem fix options for known issues | `./references/common_issues.md` | `Solutions` sections |

Scripts Overview

| Script | Purpose | Required args | Optional args | Output | Fallback behavior | | --- | --- | --- | --- | --- | --- | | `./scripts/cluster_health.sh` | Cluster-wide health snapshot (nodes, workloads, events, common failure states) | None | `--strict`, `K8S_REQUEST_TIMEOUT` env var | Sectioned report to stdout | Continues on check failures, tracks them in summary and exit code | | `./scripts/network_debug.sh` | Pod-centric network and DNS diagnostics | `<pod-name>` (`<namespace>` defaults to `default`) | `--strict`, `--insecure`, `K8S_REQUEST_TIMEOUT` env var | Sectioned report to stdout | Uses secure API probe by default; insecure TLS requires explicit `--insecure` | | `./scripts/pod_diagnostics.py` | Deep pod diagnostics (status, describe, YAML, events, per-container logs, node context) | `<pod-name>` | `-n/--namespace`, `-o/--output` | Sectioned report to stdout or file | Fails fast on missing access; skips optional metrics/log blocks with clear messages |

Script Exit Codes

`./scripts/cluster_health.sh` and `./scripts/network_debug.sh` share the same contract:

• `0`: checks completed with no check failures (warnings allowed unless `--strict` is set).
• `1`: one or more checks failed, or warnings occurred in `--strict` mode.
• `2`: blocked preconditions (for example: missing `kubectl`, no active context, inaccessible namespace/pod).

Deterministic Debugging Workflow

Follow this systematic approach for any Kubernetes issue:

1. Preflight and Scope

kubectl config current-context
kubectl get ns
kubectl auth can-i get pods -n <namespace>

If preflight fails, stop and fix access/context first.

2. Identify the Problem Layer

Categorize the issue:

• **Application Layer**: Application crashes, errors, bugs
• **Pod Layer**: Pod not starting, restarting, or pending
• **Service Layer**: Network connectivity, DNS issues
• **Node Layer**: Node not ready, resource exhaustion
• **Cluster Layer**: Control plane issues, API problems
• **Storage Layer**: Volume mount failures, PVC issues
• **Configuration Layer**: ConfigMap, Secret, RBAC issues

3. Gather Diagnostics with the Right Script

Use the appropriate diagnostic script based on scope:

Pod-Level Diagnostics

Use `./scripts/pod_diagnostics.py` for comprehensive pod analysis:

python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace>

Th