/authoring-log-alerts

name: authoring-log-alerts
description: >
  Author useful, low-noise log alerts on services in a PostHog project. Use when the user asks to set up
  alerts for their logs, suggest alerts they should add, or evaluate whether a service is worth monitoring.
  Covers service triage, baseline characterisation, threshold drafting, back-testing via simulate, and
  shipping with a notification destination.

Authoring log alerts

Authoring an alert is a _measurement_ problem, not a guessing problem. You are not trying to be exhaustive — you are trying to land thresholds that fire 0–3 times per week on real production patterns, on services that matter.

When to use this skill

• The user asks to "set up alerts" / "suggest alerts" for their project.
• The user wants to evaluate whether a service is producing alertable signal.
• The user has just enabled log alerting and wants a starter set.

When _not_ to use this skill

• Tuning an alert that already exists — that's a different job (use `posthog:logs-alerts-events-list` to inspect

fire/resolve cadence and `posthog:logs-alerts-partial-update` to adjust).

• Investigating an active incident — pull rows with `posthog:query-logs`, don't author an alert mid-incident.

Tools

| Tool | Job | Where it fits | | --------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------------ | | `posthog:logs-services` | Top-25 services in window with log_count, error_count, error_rate, sparkline. | Step 1 — triage. | | `posthog:logs-attributes-list` / `posthog:logs-attribute-values-list` | Discover keys/values for narrower filters. | Step 2, optional. | | `posthog:logs-count-ranges` | Adaptive time-bucketed counts for a filter. | Step 3 — baseline. | | `posthog:logs-alerts-simulate-create` | Replay a draft config against `-7d` history with full state machine. | Step 4 — validate. | | `posthog:logs-alerts-create` | Persist the alert. | Step 5 — ship. | | `posthog:logs-alerts-destinations-create` | Wire the alert to Slack, webhook, or Microsoft Teams. | Step 5: ship. |

Do **not** call `posthog:query-logs` during authoring. You need distributions, not rows. Reserve `posthog:query-logs` for the very end if the user asks "show me a sample of what would have fired" — `limit: 10` is plenty.

Workflow

1. Triage — pick candidate services

Call `posthog:logs-services` for the last 24h with no filters. The response is capped at 25 services and includes a sparkline, so it is small and bounded.

A service is a candidate when **both** are true:

• `log_count` is non-trivial (≥ ~1k in 24h — quieter services produce too little signal to alert on).
• `error_rate` is non-zero, **or** the user has named the service explicitly.

Skip services with high volume but `error_rate == 0` unless the user wants a volume-shape alert (e.g. "warn me if api-gateway suddenly stops producing logs"). Volume-floor alerts use `threshold_operator: below` and need different reasoning — see [references/volume-floor-alerts.md](./references/volume-floor-alerts.md).

If the user names a service, treat it as a candidate even without error signal.

2. (Optional) Narrow the filter

If a service has many error sub-types, an alert on "all errors" is usually too broad. Use `posthog:logs-attributes-list` (try `attribute_type: log`) and `posthog:logs-attribute-values-list` to find a discriminator — common ones are `http.status_code`, `error.type`, `k8s.container.name`. Add the narrowing filter to your draft.

Keep it simple: one severity filter + one or two attribute filters is plenty. Multi-clause filters are harder to reason about and rarely improve precision.

3. Baseline — characterise the candidate over 7 days

Call `posthog:logs-count-ranges` with the candidate's filters, `dateRange: { date_from: "-7d" }`, and `targetBuckets: 24` (one bucket ≈ 7h). The response gives you bucket counts.

**Do not eyeball the percentiles or scale the threshold to the alert window manually.** Pipe the count-ranges response into the helper script:

echo '<count-ranges JSON>' | python3 scripts/baseline_stats.py --window-minutes 5

The script returns:

{
  "n_buckets": 12,
  "bucket_minutes": 420.0,
  "alert_window_minutes": 5,
  "stats": { "p50": 12.0, "p95": 71.25, "p99": 126.25, "max": 140 },
  "suggested_threshold_count": 5,
  "rationale": "max(p99=126.25, median*3=36.0, floor=5) scaled from 420m bucket to 5m window",
  "health": []
}

Use `suggested_threshold_count` as your starting threshold. Read `health`:

| `health` flag | What it means | What to do | | ----------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `sparse:N_of_M_buckets` | Too few non-empty buckets for a 7d baseline. | Widen filter, extend to `-30d`, or skip. | | `empty` | All buckets are zero. | Skip — no signal. | | `spiky` | `max` is 10×+ `p95`. | Count-threshold alerts work well. Proceed.