/elasticsearch-anomaly-detection-explainer

name: elasticsearch-anomaly-detection-explainer
description: >
  Explain Elasticsearch ML anomaly detection scores, model behavior, and result interpretation.
  Use when the user asks why a score is high or low, how the model learns, what the
  numbers mean, or how to troubleshoot unexpected anomaly scores.
compatibility: >
  Requires Elasticsearch 8.x+ or Elastic Cloud Serverless with ML anomaly detection.
  Uses the ML REST API and the standard _search API against .ml-anomalies-* — no ES|QL.
  User needs monitor_ml privilege for ML APIs.
metadata:
  author: elastic
  version: 0.3.0
  universal: true

Anomaly Detection Score Explainer

Explain anomaly scores, model behavior, and why results look the way they do. Use the **ML REST API** for job config and the **standard `_search` API** against `.ml-anomalies-*` for results — no ES|QL, fully compatible with Elastic Serverless. For job lifecycle (create, start, stop), use the `elasticsearch-anomaly-detection` skill.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

> **Prerequisite:** ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless > projects include ML. The caller needs `monitor_ml` to read job config and anomaly results. > > **Serverless note:** The `_ml/.../results/*` REST endpoints return HTTP 410 in Elastic Serverless. Always use > `POST /.ml-anomalies-*/_search` for result queries instead — fully supported everywhere this skill runs.

Process

1. **Decide whether to fetch data or interpret what the user supplied.** If the user embeds an anomaly record (or job config) in the prompt, interpret it directly using the domain knowledge below — do not call APIs to re-fetch fields already present. If the job ID, time range, or record is missing, retrieve it from the cluster.

The decision: proceed with judgment-only explanation when the record contains `record_score`, `initial_record_score`, `actual`, `typical`, and `function`; otherwise fetch the missing pieces before explaining.

2. **Verify connectivity when calling the cluster.** Call `GET /`. If the call fails, stop and surface the connection error — do not guess endpoints or credentials.

3. **Resolve the job ID and load config.** When the job ID is unknown, call `GET /_ml/anomaly_detectors` to list candidates. Call `GET /_ml/anomaly_detectors/{job_id}` for full `analysis_config` (bucket_span, detectors, custom_rules, use_null, model_plot_config) and `GET /_ml/anomaly_detectors/{job_id}/_stats` for `state`, `model_size_stats.memory_status`, and data counts.

The decision: confirm detector function and direction match the user's question before interpreting scores. A `low_count` job legitimately fires on drops; a `high_count` job does not.

4. **Retrieve anomaly records for the time range.** Call `POST /.ml-anomalies-*/_search` with `result_type: record`, the job ID, a timestamp range, and optional `record_score` filter. Read `initial_record_score`, `record_score`, `actual`, `typical`, `function`, `multi_bucket_impact`, and `anomaly_score_explanation`.

Always show **both** `initial_record_score` and `record_score`. The gap is the renormalization story.

5. **Classify the score pattern before speculating on causes.**

• **`initial_record_score` >> `record_score`** — **Renormalization.** A later, more extreme anomaly rescale this

record downward. This is expected, healthy model behavior — not a broken model or reason to distrust the detection. Use `initial_record_score` for alerting severity; show both scores and explain the gap explicitly.

• **`initial_record_score` == `record_score`** — No renormalization has occurred since detection.
• **`actual` << `typical` with `low_count`, `count`, or `low_mean`** — **Absence / drop anomaly.** A high score is

legitimate — the job detected an outage, pipeline stall, or service failure. This is **not** a false positive. Recommend incident investigation, not score tuning.

• **`actual` >> `typical` with `high_count` or `high_mean`** — Spike anomaly; confirm with `single_bucket_impact`.

**Only cite `anomaly_score_explanation` factors present in the record.** If `high_variance_penalty` is `false`, do not blame variance. If a factor is absent, note that it was not returned — do not invent it.

6. **Quantify renormalization across the job (optional).** Re-query `POST /.ml-anomalies-*/_search` for records in the time range sorted by `timestamp` ascending. Compute `score_drift = initial_record_score − record_score` per record and filter to `|score_drift| ≥ 20`. Large negative drift (initial >> record) confirms renormalization after a more extreme anomaly appeared later.

7. **Add context when the user asks "what caused this?" or "why so low/high?"**

• **Model bounds** — If `model_plot_config.enabled` is true, call `POST /.ml-anomalies-*/_search` with

`result_type: model_plot` for the same job and time range. Compare `actual` to `model_lower` / `model_upper`.

• **Influencers** — Call `POST /.ml-anomalies-*/_search` with `result_type: influencer` for the bucket time range;

sort by `influencer_score` descending.

• **Categorization jobs** — Call `POST /.ml-anomalies-*/_search` with `result_type: category_definition` to li