/service-health-check

name: service-health-check
description: "Service health monitoring, endpoint validation, and CVE source auditing."
user-invocable: false
allowed-tools:
  - Bash
  - Read
  - Write
  - Glob
  - Grep
  - Edit
routing:
  triggers:
    - "service status"
    - "process health"
    - "uptime check"
    - "is service running"
    - "check health"
    - "validate endpoints"
    - "smoke test API"
    - "health check endpoints"
    - "test endpoint"
    - "check API"
    - "smoke test"
    - "check cve sources"
    - "cve source coverage"
    - "audit cve feeds"
    - "vulnerability source audit"
    - "verify cve sources"
    - "security feed audit"
  category: infrastructure
  pairs_with:
    - kubernetes
    - condition-based-waiting
    - e2e-testing

Service Health Check Skill

Overview

This skill provides deterministic service health monitoring using the **Discover-Check-Report** pattern. It finds services, gathers health signals from multiple sources (process table, health files, port binding), and produces actionable reports identifying degraded or failed services.

**Core principle**: Health assessment is evidence-based. Never report a service healthy without verifying process status independently of health file content. Never assume a running process is functional — always cross-check against health files and port binding.

---

Reference Loading Table

| Signal | Load These Files | Why | |---|---|---| | Endpoint validation request | `references/endpoint-validator.md` | Full endpoint validation methodology | | Security header WARNs, HSTS/CSP/X-Frame issues | `references/security-headers.md` | Deep security header reference | | Config errors, hardcoded IPs, timeout problems | `references/endpoint-config-preferred-patterns.md` | Endpoint config patterns | | 401/403 failures, Bearer/API-key/cookie auth | `references/auth-endpoint-patterns.md` | Auth endpoint patterns | | CVE source audit request | `references/cve-source-check.md` | Full CVE source check methodology | | CVE registry schema questions | `references/registry-schema.md` | Registry shape and entry format | | CVE source URL verification | `references/source-verification.md` | HEAD-check semantics | | CVE report format questions | `references/output-formats.md` | JSON schema and Markdown sections |

Instructions

Phase 1: DISCOVER

**Goal**: Identify all services to check before running any health probes.

**Step 1: Locate service definitions**

Search for service configuration in this order: 1. `services.json` in project root 2. Docker/docker-compose files for service definitions 3. systemd unit files or process manager configs 4. User-provided service specification

**Step 2: Build service manifest**

For each service, establish:

## Service Manifest
| Service | Process Pattern | Health File | Port | Stale Threshold |
|---------|----------------|-------------|------|-----------------|
| api-server | gunicorn.*app:app | /tmp/api_health.json | 8000 | 300s |
| worker | celery.*worker | /tmp/worker_health.json | - | 300s |
| cache | redis-server | - | 6379 | - |

**Validation constraints**:

• Each process pattern must be specific enough to avoid false matches (e.g., "python" matches all Python processes—use full paths or arguments instead)
• Health file paths must be absolute
• Port numbers must be valid (1-65535)
• Pattern specificity matters: narrow patterns with full command paths, distinguishing arguments, or specific binary names

**Step 3: Validate manifest**

Confirm each entry passes the constraints above. If a pattern is too broad, use `ps aux | grep` to identify distinguishing arguments, then update the pattern.

**Gate**: Service manifest complete with at least one service. Proceed only when gate passes.

Phase 2: CHECK

**Goal**: Gather health signals for every service in the manifest. Always check process status independently of health file content—a running process and a healthy health file are separate signals.

**Step 1: Check process status**

For each service, run process check:

pgrep -f "<process_pattern>"

Record: running (true/false), PIDs, process count.

**Rationale**: Process existence is the primary signal. A missing process always means the service is DOWN. A running process alone is insufficient—the service may have crashed or failed to bind to its port.

**Step 2: Parse health files (if configured)**

Read and parse JSON health files. Evaluate:

• Does the file exist?
• Does it parse as valid JSON?
• How old is the timestamp (staleness)? Default stale threshold is 300 seconds.
• What status does the service self-report?
• What is the connection state?

**Critical constraint**: Never trust health file content alone. The file could be stale from before a process crash. Always verify: 1. Process is still running 2. Health file timestamp is fresh (within configured threshold) 3. Status field matches evidence (e.g., "error" requires restart)

**Step 3: Probe ports (if configured)**

Check if expected ports are listening:

ss -tlnp "sport = :<port>"

**Rationale**: Verify ports are actually bound. A process can start but fail to bind to its configured port—that is effectively a DOWN state, not HEALTHY.

**Step 4: Evaluate health per service**

Apply this decision tree (constraints embedded in logic):

1. **Process not running** → **DOWN** (definitive) 2. **Process running + health file missing** → **WARNING** (limited visibility, but process is alive) 3. **Process running + health file stale** (> threshold) → **WARNING** (file hasn't updated in configured time, suggests no activity or crash recovery in progress) 4. **Process running + status=error** → **ERROR** (restart recommended immediately) 5. **Process running + disconnected > 30 minutes** → **WARNING** (long disconnect suggests stuck state, restart recommended) 6. **Process running + disconnected < 30 minutes** → **DEGRADED** (allow reconnection window, monitor) 7. **Process running + port