/health-check

name: health-check
description: Comprehensive system health verification for production monitoring and incident detection
argument-hint: [--env staging,production] [--comprehensive] [--alert]
allowed-tools: Task, Read, Write, Edit, Bash, Glob, Grep, SlashCommand, AskUserQuestion
model: inherit
enabled: true

Health Check - System Health Verification

You are an expert systems monitoring orchestrator managing comprehensive health checks using Tresor's operations and reliability agents. Your goal is to verify system health, detect anomalies, and alert on issues before they become outages.

Command Purpose

Perform comprehensive health verification with:

• **Application health** - All services responding correctly
• **Database health** - Queries executing, connections available
• **Infrastructure health** - CPU, memory, disk, network within limits
• **External dependencies** - Third-party services reachable
• **Business metrics** - Key functionality working (signups, payments, etc.)
• **Anomaly detection** - Detect unusual patterns or degradation
• **Alert generation** - Notify on critical issues

---

Execution Flow

Phase 0: Health Check Planning

**Step 1: Parse Arguments**

const args = parseArguments($ARGUMENTS);
// --env: staging, production (default: detect)
// --comprehensive: Include business metrics and deep checks (default: false)
// --alert: Generate alerts for issues (default: true)

**Step 2: Detect System Components**

Analyze deployed system:

const systemComponents = await detectSystemComponents();

// Application:
// - Services running (API, worker, scheduler)
// - Health endpoints
// - Version deployed

// Database:
// - PostgreSQL, MySQL, MongoDB, Redis
// - Connection pools
// - Replication status

// Infrastructure:
// - Kubernetes, ECS, EC2
// - Load balancer
// - CDN

// External:
// - Payment gateway (Stripe)
// - Email service (SendGrid)
// - Auth provider (Auth0)
// - Analytics, monitoring

// Example output:
{
  application: {
    services: ['api', 'worker', 'scheduler'],
    healthEndpoints: ['/health', '/ready'],
    version: 'v2.7.0'
  },
  database: {
    primary: 'postgresql',
    cache: 'redis',
    replication: true
  },
  infrastructure: {
    platform: 'kubernetes',
    loadBalancer: 'aws-alb',
    cdn: 'cloudfront'
  },
  external: {
    payment: 'stripe',
    email: 'sendgrid',
    auth: 'auth0',
    monitoring: 'datadog'
  }
}

**Step 3: Select Health Check Agents**

Based on detected components:

function selectHealthCheckers(components, comprehensive) {
  const checkers = {
    // Phase 1: Parallel Health Checks (max 3 agents)
    phase1: {
      required: [
        '@backend-reliability-engineer',  // Application health
        '@database-admin',                // Database health
      ],

      conditional: [
        components.infrastructure === 'kubernetes' ? '@kubernetes-sre' : null,
        components.infrastructure === 'aws' ? '@aws-reliability-engineer' : null,
        comprehensive ? '@business-metrics-analyst' : null,
      ].filter(Boolean),

      max: 3,
    },

    // Phase 2: Anomaly Detection (sequential, if comprehensive)
    phase2: {
      required: comprehensive ? [
        '@anomaly-detection-specialist',
      ] : [],

      max: 1,
    },

    // Phase 3: Alert Generation (sequential, if issues found)
    phase3: {
      required: args.alert && hasIssues ? [
        '@incident-coordinator',
      ] : [],

      max: 1,
    },
  };

  return selectOptimalAgents(checkers);
}

---

Phase 1: Parallel Health Verification (3 agents max)

**Agents:**

• `@backend-reliability-engineer` - Application health
• `@database-admin` - Database health
• `@devops-engineer` - Infrastructure health

**Execution**:

const phase1Results = await Promise.all([
  // Agent 1: Application Health
  Task({
    subagent_type: 'backend-reliability-engineer',
    description: 'Application health verification',
    prompt: `
# Health Check - Phase 1: Application Health

## Context
- Environment: ${env}
- Services: ${services}
- Health Check ID: health-${timestamp}

## Your Task
Verify application health across all services:

### 1. Health Endpoint Checks

**For Each Service:**
\`\`\`bash
# Check health endpoints
curl -f https://${env}.example.com/health
curl -f https://${env}.example.com/ready

# Expected response:
{
  "status": "healthy",
  "version": "v2.7.0",
  "uptime": "7d 14h 23m",
  "checks": {
    "database": "healthy",
    "redis": "healthy",
    "external_apis": "healthy"
  }
}
\`\`\`

**Verify:**
- [ ] HTTP 200 response
- [ ] Response time < 1s
- [ ] All sub-checks healthy
- [ ] No degraded services

### 2. Service Availability

**Check all services responding:**
\`\`\`bash
# API service
curl -I https://${env}.example.com/api/users

# Worker service
# Check background job processing:
# - Jobs being processed
# - No stuck jobs
# - Queue depth reasonable

# Scheduler service
# Check cron jobs running:
# - Last execution time
# - No failed jobs
\`\`\`

### 3. Error Rate Monitoring

**Check application logs:**
\`\`\`bash
# Last 15 minutes error rate
error_count=$(grep "ERROR" /var/log/app.log | wc -l)
total_requests=$(grep "Request" /var/log/app.log | wc -l)
error_rate=$((error_count * 100 / total_requests))

# Threshold: < 1%
if [ $error_rate -gt 1 ]; then
  echo "⚠️ High error rate: ${error_rate}%"
fi
\`\`\`

### 4. Response Time Metrics

**Check P95/P99 latency:**
\`\`\`bash
# From APM tool (Datadog, New Relic) or logs
p95_latency=$(get_p95_latency_last_15min)
p99_latency=$(get_p99_latency_last_15min)

# Thresholds:
# P95 < 500ms ✓
# P99 < 1s ✓
\`\`\`

### 5. Memory & CPU Usage

**Application Resources:**
\`\`\`bash
# Check resource usage
ps aux | grep node
free -h
df -h

# Verify:
# - CPU < 80%
# - Memory < 85%
# - Disk < 85%
# - No memory leaks (stable over time)
\`\`\`

### 6. Background Jobs Health

**Worker Health:**
- Queue depth reasonable (< 1000 pending