/cg-indie-observability

name: cg-indie-observability
description: Set up production observability for indie-scale apps — structured logging, uptime monitoring, error tracking, and basic metrics. Use when the user mentions logging, monitoring, alerts, error tracking, uptime checks, "how do I know if my app is down", Sentry, journalctl, structured logs, or wants to know what's happening in production. NOT for enterprise observability stacks (Grafana+Prometheus+Loki+Jaeger), OpenTelemetry distributed tracing, or APM platforms — this is for solo devs and small teams who want to know when something breaks and why.

cg-indie-observability

Give an indie-scale production app enough visibility to answer three questions reliably: *Is it up? Did something break? Why?* Without running a six-container observability stack.

Scope

**This skill covers:**

• Structured logging (JSON to stdout, picked up by journald)
• Log rotation and retention
• External uptime monitoring with alerts
• Error tracking with Sentry-compatible tools
• Basic system and application metrics
• Alerting that actually reaches you (email, Telegram, Discord, Slack)
• Log querying with `journalctl` and simple grep workflows

**This skill does NOT cover:**

• Grafana + Prometheus + Loki + Tempo + Jaeger stacks (too much to run and maintain)
• Full OpenTelemetry distributed tracing (overkill for single-service apps)
• APM platforms with flame graphs, session replay, or LLM trace capture
• Kubernetes cluster observability
• Multi-cluster federation or high-availability monitoring
• Custom Grafana dashboards and PromQL

If the user has multiple services, high traffic, or needs distributed tracing across a microservice mesh — recommend SigNoz, Grafana LGTM, or a managed platform instead.

Required context — gather before generating anything

MUST have answers to these before producing code or config. Ask the user directly if unclear.

1. **What language/runtime is the app?** — determines logging library:

• Go → `log/slog` (standard library, 1.21+)
• Node.js / Bun → `pino`
• Python → `structlog` or `python-json-logger`
• Rust → `tracing` with `tracing-subscriber`
• Java / Spring Boot → Logback with `logstash-logback-encoder`
• .NET → Serilog with `Serilog.Formatting.Compact` (CLEF format)
• Kotlin → same as Java (Logback) or `kotlin-logging` wrapper
• Swift (server-side) → `swift-log` with a JSON backend like `swift-log-format-and-pipe`

2. **Is the app already running in production?** — determines workflow:

• New app → instrument from the start
• Existing app with `printf` or `console.log` → migration path, not a rewrite

3. **How is the app deployed?** — affects log destination:

• systemd service → logs go to stdout/stderr, journald captures them
• Docker container → logs go to stdout, Docker captures them, forwarded via `docker logs` or a log driver
• Bare process → needs explicit file redirection with log rotation

4. **What's the team size and error volume expectation?**

• Solo dev, low volume → free tier of Sentry cloud, or skip error tracker entirely (structured logs are enough)
• Small team, moderate volume → self-hosted Bugsink (single container) or GlitchTip
• Privacy-sensitive (EU data, healthcare, etc.) → self-hosted only

5. **What alerting channels does the user actually check?**

• Email is reliable but slow
• Telegram/Discord are fast and mobile-friendly
• Slack is fine if the team lives there
• SMS/phone calls only for true emergencies (PagerDuty-style)

The best alerting system is the one the user will not mute.

The four pillars

Indie observability is four loosely-coupled concerns. Set them up in order of value.

Pillar 1 — Structured logging (highest value, lowest cost)

**Why first**: Every other pillar is easier when logs are structured. A single `jq` query replaces an entire dashboard. Debugging goes from hours to minutes.

**Rules**:

• **MUST** use structured logging (JSON key-value pairs), not `printf` or string concatenation
• **MUST** log to stdout/stderr, not files directly — let the process supervisor handle log collection
• **MUST** include these fields on every log entry: `timestamp` (ISO 8601 UTC), `level`, `service`, `message`
• **SHOULD** add request-scoped context: `request_id`, `user_id`, `route`, `duration_ms`
• **SHOULD** generate `request_id` via middleware at the entry point and propagate through the entire request chain — this is single-service tracing and gives you 90% of what distributed tracing provides without the complexity
• **SHOULD** use log levels correctly: `DEBUG` (dev only), `INFO` (normal operations), `WARN` (unexpected but handled), `ERROR` (something failed), `FATAL` (process must exit)
• **MUST NOT** log secrets, passwords, API keys, full credit card numbers, or raw PII
• **MUST NOT** use `printf` / `console.log` / `print` in production code paths

**Go example with slog**:

import (
    "log/slog"
    "os"
)

logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo,
}))
slog.SetDefault(logger)

// Use it
slog.Info("request completed",
    "request_id", reqID,
    "method", "POST",
    "path", "/api/users",
    "status", 201,
    "duration_ms", 42,
)

**Node.js with pino**:

import pino from 'pino';

const logger = pino({
    level: process.env.LOG_LEVEL || 'info',
    formatters: {
        level: (label) => ({ level: label }),
    },
    timestamp: pino.stdTimeFunctions.isoTime,
});

logger.info({
    request_id: reqId,
    method: 'POST',
    path: '/api/users',
    status: 201,
    duration_ms: 42,
}, 'request completed');

**Python with structlog**:

import structlog
import logging

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.JSONRenderer(),
    ],
    wrapper_class=s