/dt-alerting

name: dt-alerting
description: "End-to-end Dynatrace alerting lifecycle — anomaly detector setup and model selection (static threshold, adaptive baseline, seasonal baseline), alert event storage in Grail, problem grouping and denoising by root cause analysis, and workflow-based notification routing (email, Slack, ServiceNow, webhook). Use when configuring alerting, choosing between detector types, querying alert event history, understanding why alerts merged into a problem, or setting up problem-triggered notifications."
license: Apache-2.0

dt-alerting

Configure and understand the full alerting lifecycle in Dynatrace — from anomaly detector setup through Grail event storage, problem grouping, and workflow notification delivery.

The Alerting Lifecycle

┌─────────────────────────────────────────────────────────────────────┐
│  Alert Sources — five categories, each fires a DAVIS_EVENT          │
│  ─────────────────────────────────────────────────────────────────  │
│  1. DQL-based  · Grail-scheduled server-side detector               │
│  2. Edge       · OneAgent on monitored host or process              │
│  3. Pipeline   · OpenPipeline ingest-stream filter matcher          │
│  4. Synthetic  · Worldwide synthetic checker node                   │
│  5. External   · Events API, Workflow, or OneAgent local ingest     │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ DAVIS_EVENT created per trigger per entity
                               ▼
             ┌─────────────────────────────────────┐
             │  Event stored in Grail              │  Persisted and queryable via DQL.
             └──────────────────┬──────────────────┘  One event per trigger per entity.
                                │ correlated by root-cause and impact graph
                                ▼
             ┌─────────────────────────────────────┐
             │  Problem (Denoising)                │  Events sharing the same root-cause
             └──────────────────┬──────────────────┘  and impact graph → one Problem.
                                │ problem event triggers workflow
                                ▼
             ┌─────────────────────────────────────┐
             │  Workflow Notification              │  Filters problems and routes to
             └─────────────────────────────────────┘  email, Slack, ServiceNow, webhook.

When to Use This Skill

• **Detector setup** — "How do I create an anomaly detector?", "What kind of

detector should I use?", "What is the difference between adaptive and seasonal?"

• **Alert event history** — "Query all alert events for this service", "Show me

which metrics triggered alerts last week"

• **Problem denoising** — "Why did these two alerts merge into one problem?",

"How does Davis group alerts?"

• **Notification setup** — "How do I send a Slack message when a problem opens?",

"Set up a ServiceNow ticket on critical problems"

• **Best practices** — "How do I avoid alert storms?", "Which sensitivity setting

should I use?"

• **Over-alerting analysis** — "Why am I getting too many alerts?", "How do I

reduce alert fatigue?", "Which detector is firing the most?", "How do I tune sensitivity or thresholds to avoid noise?"

• **Notification routing** — "How do I route alerts to the right team?", "Set up

scalable problem filters in workflows", "Send Slack notifications only to the team responsible for the affected service"

Agent Instructions

**First step for any alerting setup request** — Before recommending a specific detector or model, load `references/anomaly-detectors.md` and use its category and model decision guide to identify which detector category (DQL-based, Edge, Pipeline, Synthetic, External) and which model (Static, Adaptive, Seasonal) best fits the user's use-case. Only proceed with configuration guidance once the right detector type has been established.

**Consolidate, don't multiply** — When a user asks to alert on multiple entities of the same kind (e.g. "alert on services A, B, and C"), always recommend a **single combined detector** rather than one detector per entity. Use `by: { <dimension> }` in the DQL `timeseries` call to split results per entity, and a single `filter:` clause to scope to the relevant entities. Pair the combined detector with a **single `dt.alert_group` tag** shared across all alert conditions and the corresponding workflow notification filter. This keeps the number of detector configs small, ensures consistent routing, and makes the workflow notification channel reusable for future entities added to the same group.

Example for three services — one detector, one workflow:

timeseries avg(dt.service.request.response_time),
  by: { dt.smartscape.service },
  filter: { in(dt.smartscape.service, {toSmartscapeId("SERVICE-0000000000000001"), toSmartscapeId("SERVICE-0000000000000002"), toSmartscapeId("SERVICE-0000000000000003")}) }

Set `dt.alert_group: "checkout-team"` in the detector's event properties, then filter the notification workflow on `matchesPhrase(dt.alert_group, "checkout-team")`. If a new service must be covered, add it to the single `filter:` list — no new detector or workflow rule needed.

Intent Mapping

| User Request | Action | Reference | |---|---|---| | "how to alert on ...", "create an alert on ...", "create anomaly detector", "set up alerting", "configure alert rule" | Explain detector categories and variants, guide through model selection | anomaly-detectors.md | | "what kinds of anomaly detectors", "edge alert", "pipeline alert", "synthetic alert", "OneAgent alert" | Explain the five alert source categories and their trade-offs | anomaly-detectors.md | | "static vs adaptive", "which detector model", "seasonal detector" | Compare models, apply decision guide | anomaly-detectors.md | | "query alert history", "which alerts fired", "Davis events in Grail" | Query `dt.davis.events` in Grail via `fetc