/infra-status

allowed-tools: Task, Bash(kubectl:*), Bash(jq:*), Bash(gdate:*), Bash(echo:*), Bash(curl:*), Bash(top:*), Bash(df:*), Bash(uname:*)
name: "Infra Status"
description: "Comprehensive Talos Kubernetes cluster health monitoring with parallel infrastructure analysis"
author: "wcygan"
tags: ["ops","infra"]
version: "1.0.0"
created_at: "2025-07-14T00:00:00Z"
updated_at: "2025-07-14T00:00:00Z"

Context

• Session ID: !`gdate +%s%N 2>/dev/null || date +%s%N 2>/dev/null || echo "$(date +%s)$(jot -r 1 100000 999999 2>/dev/null || shuf -i 100000-999999 -n 1 2>/dev/null || echo $RANDOM$RANDOM)"`
• Check target: $ARGUMENTS (optional - specific component: nodes, pods, services, storage, network, all)
• Cluster context: !`kubectl config current-context 2>/dev/null || echo "No active context"`
• Cluster info: !`kubectl cluster-info --request-timeout=5s 2>/dev/null | head -2 || echo "Cluster unreachable"`
• Node count: !`kubectl get nodes --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0"`
• Namespace count: !`kubectl get namespaces --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0"`
• Pod count: !`kubectl get pods -A --no-headers 2>/dev/null | wc -l | tr -d ' ' || echo "0"`
• System info: !`uname -sr 2>/dev/null || echo "Unknown system"`

Your Task

STEP 1: Initialize infrastructure health monitoring session

• CREATE session state file: `/tmp/infra-status-$SESSION_ID.json`
• DETERMINE check scope from $ARGUMENTS (default: comprehensive)
• VALIDATE kubectl access and cluster connectivity
• INITIALIZE health status tracking structure

# Initialize infrastructure monitoring session
echo '{
  "sessionId": "'$SESSION_ID'",
  "timestamp": "'$(gdate -Iseconds 2>/dev/null || date -Iseconds)'",
  "checkScope": "'${ARGUMENTS:-all}'",
  "clusterStatus": "unknown",
  "infrastructureServices": {},
  "criticalIssues": [],
  "healthScore": 0
}' > /tmp/infra-status-$SESSION_ID.json

STEP 2: Adaptive monitoring strategy based on check scope and cluster complexity

TRY:

CASE check_scope: WHEN "nodes":

• EXECUTE focused node health analysis
• CHECK resource utilization and readiness
• VALIDATE control plane and worker node status

WHEN "pods":

• ANALYZE pod health across all namespaces
• CHECK resource consumption and limits
• IDENTIFY stuck or failing pods

WHEN "services" OR "infrastructure":

• LAUNCH parallel sub-agents for comprehensive infrastructure analysis
• COORDINATE health checks across all preferred services
• SYNTHESIZE status from parallel execution

WHEN "storage":

• CHECK persistent volume status and capacity
• ANALYZE storage class configurations
• VALIDATE backup and replication status

WHEN "network":

• VALIDATE CNI functionality and connectivity
• CHECK service mesh and ingress health
• ANALYZE network policies and security

WHEN "all" OR empty:

• EXECUTE comprehensive infrastructure health audit
• USE parallel sub-agents for optimal performance
• PROVIDE executive summary with priority recommendations

**Focused Component Analysis:**

# Quick node health check
echo "🔍 Node Health Analysis:"
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name): \(.status.conditions[] | select(.type=="Ready") | .status)"'

# Resource utilization with error handling
echo "📊 Resource Utilization:"
kubectl top nodes --use-protocol-buffers=false 2>/dev/null || echo "⚠️ Metrics server unavailable"
kubectl top pods --all-namespaces --use-protocol-buffers=false 2>/dev/null | head -10 || echo "⚠️ Pod metrics unavailable"

# Critical system pods
echo "🏗️ Critical System Pods:"
kubectl get pods -n kube-system -o json | jq -r '.items[] | select(.metadata.name | test("(etcd|api-server|controller|scheduler)")) | "\(.metadata.name): \(.status.phase)"'

STEP 3: Parallel infrastructure service monitoring with sub-agent coordination

IF check_scope includes "services" OR "all":

LAUNCH parallel sub-agents for comprehensive infrastructure health analysis:

• **Agent 1: Postgres Database Health**: Monitor database cluster status, replication, and performance
• Focus: Connection health, replication lag, storage utilization, backup status
• Tools: kubectl with postgres-system namespace analysis, connection testing
• Output: Database cluster health report with performance metrics

• **Agent 2: DragonflyDB Cache Analysis**: Analyze cache performance and memory utilization
• Focus: Memory usage, hit rates, cluster connectivity, eviction policies
• Tools: kubectl with dragonfly-system namespace monitoring, performance analysis
• Output: Cache system health report with optimization recommendations

• **Agent 3: RedPanda Streaming Health**: Monitor streaming service and partition status
• Focus: Topic health, partition distribution, consumer lag, broker status
• Tools: kubectl with redpanda-system namespace analysis, topic validation
• Output: Streaming infrastructure health with throughput analysis

• **Agent 4: ScyllaDB NoSQL Status**: Analyze NoSQL cluster consistency and performance
• Focus: Node health, repair status, data consistency, query performance
• Tools: kubectl with scylla-system namespace monitoring, cluster analysis
• Output: NoSQL cluster health with consistency validation

• **Agent 5: Storage & Networking**: Monitor persistent volumes, CNI, and service mesh
• Focus: PV status, storage capacity, network policies, ingress health
• Tools: kubectl storage and network resource analysis, connectivity testing
• Output: Infrastructure foundation health with capacity planning

**Sub-Agent Coordination:**

# Each agent reports findings to session state
echo "Launching parallel infrastructure health analysis..."
echo "Agents will analyze: Database, Cache, Streaming, NoSQL, and Foundation layers"
echo "Results will be aggregated into comprehensive health dashboard"

STEP 4: Health scoring and intelligent issue prioritization

TRY:

**Health Score Calculation:**

# Calculate comprehensive health score (0-100)
node_health