/k8s-triage

name: k8s-triage
description: Kubernetes alert triage — dedup via YT search, deep control plane investigation, auto-escalation for recurring/flapping/control-plane alerts.
allowed-tools: Bash
user-invocable: true
metadata:
  openclaw:
    always: true

Kubernetes Alert Triage

When you see a message containing `k8s-triage.sh` in `#infra-nl-prod` or `#infra-gr-prod`, you MUST run it using the `exec` tool immediately. Do NOT ask questions.

Usage

./skills/k8s-triage/k8s-triage.sh "<alertname>" "<severity>" "<namespace>" "<summary>" "<node>" "<pod>" [--site nl|gr]

Site is auto-detected from node name prefix (`grskg*` → GR, otherwise NL). Use `--site` to override.

Environment Variables

| Variable | Purpose | |----------|---------| | `FORCE_ESCALATE=true` | Escalate regardless of severity (set by n8n for flapping alerts) | | `EXISTING_ISSUE=ID` | Reuse this YT issue instead of creating new | | `SKIP_ESCALATION=true` | Skip escalation step (for testing) |

CRITICAL RULES

1. **ALWAYS use the `exec` tool** to run the script. Do NOT describe what to do. 2. **Do NOT make changes.** All investigation is READ-ONLY (kubectl get/describe/logs only). 3. **The script handles everything** — YT dedup, issue creation, investigation, findings, escalation.

What's New (v2)

Issue Deduplication (Step 0)

Before creating a new YT issue, the script searches YouTrack for existing open issues with the same Alert Rule created within 24h. If found, it reuses that issue instead of creating a duplicate. Also searches for **related** issues (same node or namespace within 12h) and lists them in the findings.

Control Plane Deep Investigation

When the alert involves a control plane component (apiserver, etcd, controller-manager, scheduler), the script performs cross-component investigation:

• etcd pod status + logs (all members)
• apiserver pods + restart counts + error logs (all instances)
• controller-manager + scheduler status
• Control plane resource usage (kubectl top)

Smart Escalation (Step 5)

Escalation is triggered by ANY of these conditions:

• **Critical severity** (unchanged)
• **Recurring alert** (reusing an existing issue)
• **Forced escalation** (`FORCE_ESCALATE=true`, set by n8n for flapping alerts)
• **Control plane warning** (all control plane alerts escalate regardless of severity)

Integration Details

Register Callback

After creating/reusing the YouTrack issue, the script POSTs a register callback to n8n so the Prometheus Alert Receiver can track the issue:

POST /webhook/prometheus-alert (or /webhook/prometheus-alert-gr for GR site)
{
  "action": "register",
  "alertKey": "<alertname>:<namespace>",
  "issueId": "<IFRNLLEI01PRD-NNN or IFRGRSKG01PRD-NNN>"
}

YouTrack Custom Fields Set

The script sets 6 custom fields on newly created issues:

• **Hostname** — K8s node name (or "k8s-cluster" for cluster-wide alerts)
• **Alert Rule** — Prometheus alert name (e.g., `KubePodCrashLooping`)
• **Severity** — `critical` or `warning`
• **Namespace** — K8s namespace where the alert fired
• **Pod** — affected pod name (if applicable)
• **Alert Source** — `prometheus` (distinguishes from LibreNMS alerts)