/observability

name: observability
description: "Structured logging, debugging (pdb/ipdb), profiling (cProfile/line_profiler), and performance monitoring. Use when adding logging, debugging issues, or optimizing performance. TRIGGER when: logging, debug, profiling, performance monitoring, metrics, stack trace. DO NOT TRIGGER when: feature implementation, testing, documentation, config changes."
allowed-tools: [Read, Grep, Glob, Bash]

Observability Skill

Comprehensive guide to logging, debugging, profiling, and performance monitoring in Python applications.

When This Skill Activates

• Adding logging to code
• Debugging production issues
• Profiling performance bottlenecks
• Monitoring application metrics
• Analyzing stack traces
• Performance optimization
• Keywords: "logging", "debug", "profiling", "performance", "monitoring"

---

Core Concepts

1. Structured Logging

Structured logging with JSON format for machine-readable logs and rich context.

**Why Structured Logging?**

• Machine-parseable (easy to search, filter, aggregate)
• Context-rich (attach metadata to log entries)
• Consistent format across services

**Key Features**:

• JSON-formatted logs
• Log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)
• Context logging with extra metadata
• Best practices for meaningful logs

**Example**:

import logging
import json

logger = logging.getLogger(__name__)
logger.info("User action", extra={
    "user_id": 123,
    "action": "login",
    "ip": "192.168.1.1"
})

**See**: `docs/structured-logging.md` for Python logging setup and patterns

---

2. Debugging Techniques

Interactive debugging with pdb/ipdb and effective debugging strategies.

**Tools**:

• **Print debugging** - Quick and simple
• **pdb** - Python's built-in debugger
• **ipdb** - IPython-enhanced debugger
• **Post-mortem debugging** - Debug after crash

**pdb Commands**:

• `n` (next) - Execute current line
• `s` (step) - Step into function
• `c` (continue) - Continue execution
• `p variable` - Print variable value
• `l` - List source code
• `q` - Quit debugger

**Example**:

import pdb; pdb.set_trace()  # Debugger starts here

**See**: `docs/debugging.md` for interactive debugging patterns

---

3. Profiling

CPU and memory profiling to identify performance bottlenecks.

**Tools**:

• **cProfile** - CPU profiling (built-in)
• **line_profiler** - Line-by-line CPU profiling
• **memory_profiler** - Memory usage analysis
• **py-spy** - Sampling profiler (no code changes)

**cProfile Example**:

python -m cProfile -s cumulative script.py

**Profile Decorator**:

import cProfile
import pstats

def profile(func):
    def wrapper(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()
        result = func(*args, **kwargs)
        profiler.disable()
        stats = pstats.Stats(profiler)
        stats.sort_stats('cumulative')
        stats.print_stats(10)  # Top 10 functions
        return result
    return wrapper

@profile
def slow_function():
    # Your code here
    pass

**See**: `docs/profiling.md` for comprehensive profiling techniques

---

4. Monitoring & Metrics

Performance monitoring, timing decorators, and simple metrics.

**Timing Patterns**:

• **Timing decorator** - Measure function execution time
• **Context manager timer** - Measure code block duration
• **Performance assertions** - Fail if too slow

**Simple Metrics**:

• **Counters** - Track event occurrences
• **Histograms** - Track value distributions

**Example**:

import time
from functools import wraps

def timer(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start = time.time()
        result = func(*args, **kwargs)
        duration = time.time() - start
        print(f"{func.__name__} took {duration:.2f}s")
        return result
    return wrapper

@timer
def process_data():
    # Your code here
    pass

**See**: `docs/monitoring-metrics.md` for stack traces, timers, and metrics

---

5. Best Practices & Anti-Patterns

Debugging strategies and logging anti-patterns to avoid.

**Debugging Best Practices**: 1. **Binary Search Debugging** - Narrow down the problem area 2. **Rubber Duck Debugging** - Explain the problem to someone (or something) 3. **Add Assertions** - Catch bugs early 4. **Simplify and Isolate** - Reproduce with minimal code

**Logging Anti-Patterns to Avoid**:

• Logging sensitive data (passwords, tokens)
• Logging in loops (use counters instead)
• No context in error logs
• Inconsistent log formats
• Too verbose logging (noise)

**See**: `docs/best-practices-antipatterns.md` for detailed strategies

---

Quick Reference

| Tool | Use Case | Details | |------|----------|---------| | Structured Logging | Production logs | `docs/structured-logging.md` | | pdb/ipdb | Interactive debugging | `docs/debugging.md` | | cProfile | CPU profiling | `docs/profiling.md` | | line_profiler | Line-by-line profiling | `docs/profiling.md` | | memory_profiler | Memory analysis | `docs/profiling.md` | | Timer decorator | Function timing | `docs/monitoring-metrics.md` | | Context timer | Code block timing | `docs/monitoring-metrics.md` |

---

Logging Cheat Sheet

import logging

# Setup
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

# Usage
logger.debug("Debug message")       # Detailed diagnostic
logger.info("Info message")         # General information
logger.warning("Warning message")   # Warning (recoverable)
logger.error("Error message")       # Error (handled)
logger.critical("Critical message") # Critical (unrecoverable)

# With context
logger.info("User action", extra={"user_id": 123, "action": "login"})

---

Debugging Cheat Sheet

# pdb
import pdb; pdb.set_trace()

# ipdb (enhanced)
import ipdb; ipdb.set_trace()

# Post-mortem (debug after crash)
import pdb, sys
try:
    # Your code
    pass
except Exc