/dt-obs-kubernetes

name: dt-obs-kubernetes
description: >-
  Kubernetes cluster, pod, node, and workload monitoring. Use when analyzing K8s health, resource
  optimization, pod failures, OOMKills, scheduling, or security posture. Also use for Kubernetes
  operational events like pod restarts, OOM events, evictions, and cluster event history.
  Trigger: "Kubernetes pods", "K8s cluster health", "OOMKill", "pod restarts", "container CPU",
  "namespace resource usage", "over-provisioned pods", "privileged containers",
  "pod placement", "K8s node capacity", "running containers by cluster",
  "workload scheduling", "pod evictions", "K8s labels and annotations",
  "kubernetes events", "pod restart events", "OOM events", "K8s event history".
  Do NOT use for explaining existing queries, product documentation questions,
  AWS-specific resource queries, service-level RED metrics, distributed tracing, or
  log analysis — use the relevant skill instead.
license: Apache-2.0

Infrastructure Kubernetes

Monitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query cluster resources, monitor workload health, analyze pod placement, optimize costs, and assess security posture.

When to Use This Skill

• Monitoring Kubernetes cluster health and capacity
• Analyzing pod and container resource utilization
• Investigating pod failures, OOMKills, evictions, or crash loops
• Debugging degraded deployments, stuck rollouts, or node pressure
• Optimizing Kubernetes resource costs
• Assessing security posture and compliance
• Troubleshooting workload scheduling and placement
• Auditing ingress routing and network policies

Knowledge Base Structure

Core Monitoring (Start Here)

1. **Cluster Inventory** → `references/cluster-inventory.md` - Clusters, namespaces, resource distribution 2. **Node Monitoring** - Node capacity, CPU/memory usage, pod density 3. **Pod Monitoring** - Pod CPU, memory, lifecycle events 4. **Workload Monitoring** - Deployment, StatefulSet, DaemonSet resources

Advanced Topics

1. **Configuration Analysis** → `references/labels-annotations.md` - Parse k8s.object, labels, annotations 2. **Scheduling & Placement** → `references/pod-node-placement.md` - Node selectors, affinity, taints, HA 3. **Cost Optimization** - Right-sizing, waste detection, efficiency scoring 4. **Security & Compliance** - Privileged containers, security contexts

Key Concepts

Entity Types

**Workloads:** `K8S_DEPLOYMENT`, `K8S_STATEFULSET`, `K8S_DAEMONSET`, `K8S_JOB`, `K8S_CRONJOB`, `K8S_HORIZONTALPODAUTOSCALER` **Infrastructure:** `K8S_CLUSTER`, `K8S_NAMESPACE`, `K8S_NODE`, `K8S_POD` **Configuration:** `K8S_SERVICE`, `K8S_CONFIGMAP`, `K8S_SECRET`, `K8S_PERSISTENTVOLUMECLAIM`, `K8S_PERSISTENTVOLUME`, `K8S_INGRESS`, `K8S_NETWORKPOLICY`

> **Note:** HPA, Job, CronJob, and configuration entity types are listed for > inventory queries only. For HPA scaling analysis see `references/workload-health.md`; > for PVC/PV see `references/pv-pvc.md`; for Ingress/NetworkPolicy see the > corresponding reference files.

Query Types

**smartscapeNodes** - Query K8s entities (current state, no time range needed):

smartscapeNodes K8S_POD
| filter k8s.namespace.name == "production"
| fields k8s.cluster.name, k8s.pod.name

**timeseries** - Monitor metrics over time (always specify a `from:` range):

timeseries cpu = sum(dt.kubernetes.container.cpu_usage),
  by: {k8s.pod.name, k8s.namespace.name},
  from: now()-1h
| fieldsAdd avg_cpu = arrayAvg(cpu)

> `timeseries` returns each metric as a time-bucket array. To collapse it to a > scalar in a downstream `fieldsAdd`, use `arrayAvg(series)` or `arraySum(series)` > — calling `avg()` or `sum()` on a series field outside the `timeseries {}` block > is not supported.

**fetch logs** - Analyze log events:

fetch logs
| filter k8s.namespace.name == "production" and loglevel == "ERROR"

Core Fields

• `k8s.cluster.name`, `k8s.namespace.name`, `k8s.pod.name`, `k8s.node.name`
• `k8s.workload.name`, `k8s.workload.kind`, `k8s.container.name`
• `k8s.object` - Full JSON configuration for deep inspection
• `tags[label]` - Access labels and annotations

**`k8s.workload.kind` values** (always lowercase in DT — not Pascal-case as in the K8s API): `"deployment"`, `"statefulset"`, `"daemonset"`, `"replicaset"`, `"job"`, `"cronjob"`

**`k8s.object` availability:**

| Entity type | `k8s.object` available? | |---|---| | `K8S_POD` | Yes | | `K8S_NAMESPACE` | Yes | | `K8S_NODE` | Yes | | Workload types (`K8S_DEPLOYMENT`, etc.) | Yes | | `K8S_CLUSTER` | **No** |

Available Metrics

**CPU:** `dt.kubernetes.container.cpu_usage`, `cpu_throttled`, `limits_cpu`, `requests_cpu` **Memory:** `dt.kubernetes.container.memory_working_set`, `limits_memory`, `requests_memory` **Operations:** `dt.kubernetes.container.restarts`, `oom_kills` **Node:** `dt.kubernetes.node.pods_allocatable`, `cpu_allocatable`, `memory_allocatable`, `dt.kubernetes.pods`

> **Aggregation rule for requests/limits:** Always use `sum()` — never `avg()` — > when aggregating `requests_cpu`, `limits_cpu`, `requests_memory`, or > `limits_memory`. `avg()` returns a per-container average and silently > undercounts multi-container pods and multi-replica workloads. For DaemonSets > (one pod per node), this error is proportional to cluster size.

Entity Disambiguation

`K8S_POD` vs `CONTAINER`: these are different entity types in Dynatrace.

• **`K8S_POD`** — K8s-native entities with `k8s.object` JSON, scheduling state, conditions, and K8s metrics. Use this skill.
• **`CONTAINER`** — Host-level container inventory (image, lifetime, host assignment). Use `dt-obs-hosts` skill instead.

The smartscape edge is `CONTAINER --(is_part_of)--> K8S_POD`. To reach containers from a pod, traverse backward:

smartscapeNodes K8S_POD
| filter k8s.namespace.name == "<namespace>"
| traverse edgeTypes: {is_part_of}, targetTypes: {CONTAINER}, direction: backward, fieldsKeep: {id}
|