/k8s-incident-triage

name: k8s-incident-triage
category: sre
tags:
  - kubernetes
  - incident-response
  - triage
  - reliability
  - observability
  - kubectl
  - oncall
  - runbooks
description: Read-only Kubernetes incident triage using kubectl. Accepts natural language or structured input. Produces root-cause hypotheses, evidence, and next-step commands.
metadata:
  forge:
    requires:
      bins:
        - kubectl
      env:
        required: []
        one_of: []
        optional:
          - KUBECONFIG
          - K8S_API_DOMAIN
          - DEFAULT_NAMESPACE
          - TRIAGE_MAX_PODS
          - TRIAGE_LOG_LINES
    egress_domains:
      - "$K8S_API_DOMAIN"
    denied_tools:
      - http_request
      - web_search

Kubernetes Incident Triage

Performs read-only triage of Kubernetes workloads and namespaces using `kubectl`.

Supports:

• Natural language input (human mode)
• Structured JSON input (automation mode)

This skill NEVER mutates cluster state.

---

Tool Usage

This skill uses `cli_execute` with `kubectl` commands exclusively. NEVER use http_request or web_search to interact with Kubernetes. All cluster operations MUST go through kubectl via the cli_execute tool.

---

Tool: k8s_triage

Diagnose unhealthy Kubernetes workloads, pods, or namespaces.

---

Input Modes

1) Human Mode (Natural Language)

Input is a plain string.

Examples:

• `triage payments-prod`
• `triage deployment payments-api in payments-prod`
• `why are pods pending in checkout-prod?`
• `investigate crashloop in payments-prod`
• `triage pod api-7c9f6d7f86-abcde in payments-prod`
• `check rollout of deployment payments-api in prod`

Behavior:

• Parse namespace, workload, pod, or selector intent.
• If namespace omitted, use `$DEFAULT_NAMESPACE` if set.
• If ambiguity exists, default to namespace-level triage.
• Never require the user to remember JSON fields.

---

2) Automation Mode (Structured JSON)

Input JSON schema:

{ "namespace": "payments-prod", "workload_kind": "deployment", "workload_name": "payments-api", "pod_name": null, "label_selector": null, "include_logs": true, "logs_tail_lines": 200, "include_previous_logs": true, "events_limit": 50, "include_node_diagnostics": true, "include_metrics": false, "output_format": "markdown" }

Rules:

• `namespace` is required.
• If `pod_name` provided → pod-level triage.
• If workload fields provided → workload-level triage.
• Else → namespace scan.

---

Triage Process

Step 0 — Preconditions

Verify cluster access:

kubectl version --client kubectl cluster-info

If RBAC denies access:

• Continue with allowed operations.
• Explicitly report denied commands in output.

---

Step 1 — Fast Health Snapshot

Namespace scope:

kubectl get pods -n <ns> -o wide kubectl get deploy,sts,ds,job,cronjob -n <ns>

Workload scope:

kubectl get <kind> <name> -n <ns> kubectl rollout status <kind>/<name> -n <ns> --timeout=10s

Select pods in states:

• CrashLoopBackOff
• ImagePullBackOff
• ErrImagePull
• Pending / Unschedulable
• Error
• NotReady
• OOMKilled
• High restart count

Limit deep triage to `$TRIAGE_MAX_PODS` (default 5).

---

Step 2 — Events Timeline

kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n <events_limit>

Look for:

• FailedScheduling
• FailedMount
• Unhealthy (probe failures)
• Back-off pulling image
• Evicted
• Node pressure signals

---

Step 3 — Describe Pods & Workloads

For each selected pod:

kubectl describe pod <pod> -n <ns>

Capture:

• Container state and reason
• Restart count
• Last termination reason
• Probe failures
• Volume mount issues
• Node assignment
• Taints / tolerations
• Affinity constraints

If workload-level triage:

kubectl describe <kind> <name> -n <ns>

---

Step 4 — Node Diagnostics (Optional)

kubectl get nodes -o wide kubectl describe node <node>

Check for:

• NotReady
• MemoryPressure
• DiskPressure
• PIDPressure
• Evictions

---

Step 5 — Logs (Optional)

kubectl logs <pod> -n <ns> --all-containers --tail=<N>

If restart loops and include_previous_logs=true:

kubectl logs <pod> -n <ns> --previous --all-containers --tail=<N>

Rules:

• Prefer previous logs for CrashLoopBackOff.
• Redact obvious sensitive patterns.
• Never print Secret values.

---

Step 6 — Optional Metrics

If enabled:

kubectl top pods -n <ns> kubectl top node

Gracefully skip if metrics-server is unavailable.

---

Detection Heuristics

Classify detected issues into:

• CrashLoop / Application Crash
• OOMKilled / Resource Limits
• Image Pull Failure
• Scheduling Constraint
• Probe Failure
• PVC / Volume Failure
• Node Pressure / Eviction
• Rollout Stuck
• Unknown / Escalation Needed

For each issue provide:

• Hypothesis
• Supporting evidence
• Confidence score (0.0–1.0)
• Recommended next commands

---

Output Structure

1) Incident Summary

• Namespace
• Scope (pod/workload/namespace)
• Affected resources count
• Timestamp

2) Top Findings

Concise bullet summary.

3) Likely Root Causes (Top 3)

For each:

• Description
• Evidence excerpt
• Confidence
• Recommended actions

4) Next Commands

Copy-paste kubectl commands.

5) Evidence Appendix

• Events (recent)
• Describe excerpts
• Log excerpts

---

Safety Constraints

This skill MUST:

• Perform read-only kubectl operations only.
• Never execute:
• apply
• patch
• delete
• exec
• port-forward
• scale
• rollout restart
• Never print Secret values.
• Avoid dumping full environment variables.

---

Autonomous Compatibility

This skill is designed to be invoked by:

• k8s_alert_handler (alert-triggered triage)
• k8s_patrol (scheduled patrol scans)
• Humans via natural language CLI

It must:

• Be idempotent
• Produce deterministic fingerprints
• Avoid excessive cluster load
• Limit deep triage scope