/dd-triage-flaky-test

name: dd-triage-flaky-test
description: Load when investigating a specific flaky test. Gets history, failure pattern, and category, then recommends fix, quarantine, or escalate.
metadata:
  version: "1.1.0"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,ci,cicd,flaky,flaky-tests,test-optimization
  alwaysApply: "false"

Triage Flaky Test

One-line summary: Investigate a specific flaky test — get history, failure pattern, and category, then recommend fix, quarantine, or escalate.

Requires: `dd-pup` skill (pup CLI installed and authenticated).

---

Input

| Parameter | Description | |---|---| | Test name | Fully qualified test name (e.g. `TestMyFunc` or `com.example.MyTest`) | | Repository | Lowercase, no-schema URL (e.g. `github.com/org/repo`). Derive from `git remote get-url origin` if not provided. |

---

Workflow

STEP 0 — Parse Input

Derive repository ID from git if not provided:

git remote get-url origin
# Strip protocol and trailing .git, normalize SSH git@host:org/repo to host/org/repo, then lowercase
# e.g. https://github.com/DataDog/my-repo.git → github.com/datadog/my-repo

**Validation fallback:** If STEP 1 returns no results, confirm the correct repository by searching without a repo filter:

pup cicd tests search \
  --query "test_level:test @test.name:\"<test-name>\"" \
  --from 30d \
  --limit 5

Extract `@git.repository.id_v2` from results and retry STEP 1 with the confirmed value.

STEP 1 — Get Flaky Test Details

**Preferred — use `fingerprint_fqn` if known** (`fingerprint_fqn` is a valid CI Visibility search facet, distinct from `flaky_state`):

pup cicd flaky-tests search \
  --query "fingerprint_fqn:<fqn>" \
  --sort="-last_flaked" \
  --limit 5

**Fallback — use name + suite + repo:**

pup cicd flaky-tests search \
  --query "@test.name:\"<test-name>\" @test.suite:\"<suite>\" @git.repository.id_v2:\"<repo>\"" \
  --sort="-last_flaked" \
  --limit 10

Omit `@test.suite` if unknown. If multiple suites or modules match, resolve the test identity from the failing event before continuing; add `@test.module` when needed to distinguish them.

Do not filter by `flaky_test_state` — return the test regardless of state.

Note: the query filter facet is `flaky_test_state`; the returned response attribute is `flaky_state` — these are different names for the same concept; do not use `flaky_state:active` as a query filter.

Extract from results:

• `fingerprint_fqn` — unique test identifier; used as the `id` in STEP 5 write call. **If absent, do not proceed to quarantine — see STEP 5.**
• `flaky_state` — current state (active / quarantined / disabled / fixed)
• `test_stats.failure_rate_pct` — percentage of runs that fail
• `flaky_category` — root cause category
• `codeowners` — owning team
• `pipeline_stats.total_lost_time_ms` — total CI time lost in milliseconds
• `attempt_to_fix_id` — non-empty when a Bits Code fix has been generated
• `web_url` — datacenter-correct FTM link, when supplied by the backend

Pup returns raw JSON:API records: read fields from `data[].attributes` and use the matching record’s `data[].id` as `fingerprint_fqn` if it is not repeated in attributes. The MCP enriches responses with `fingerprint_fqn` and `web_url`; pup may not return `web_url`. Keep the ID tied to the confirmed test’s module, suite, name, and repository. If no link is supplied, use the FTM UI on the configured Datadog site; do not assume a US1 hostname.

STEP 2 — Get Recent Failure History

pup cicd tests search \
  --query "test_level:test @test.name:\"<test-name>\" @test.suite:\"<suite>\" @test.status:fail @git.repository.id_v2:\"<repo>\"" \
  --from 7d \
  --limit 20

Extract:

• Error messages and stack traces (`@error.message`, `@error.stack`)
• Failing branches (`@git.branch`) — branch-specific vs. widespread
• Frequency pattern — random timing or specific conditions
• Unique `@ci.pipeline.id` values for blast radius (STEP 3)

**Reading stack traces:** Read error chains from bottom to top. Chained errors hide the real cause — the innermost error at the bottom is the root cause, not the first line. If the identified root cause would cause near-100% failure but the actual flake rate is much lower, the root cause identification is wrong — look deeper.

STEP 2.5 — Determine Attribution

Decide whether this flake was **introduced by the current working change** (self-introduced) or is a **pre-existing, inherited** flake. This drives the fix-vs-quarantine steer in STEP 4/5 — you should not suppress a flake you just introduced.

**Preserve the confirmed test identity.** If STEP 1 identified a module, add `@test.module:"<module>"` to the STEP 2 query and rerun it before continuing. Use only history matching the confirmed module, suite, name, and repository for attribution, error analysis, and blast radius. If history cannot be scoped to the confirmed test, treat it as inconclusive.

**Failure history takes precedence.** If STEP 2 shows the test failing on the default branch, or failing across multiple unrelated branches (not just the current one), the flake pre-dates the current change — classify it `pre-existing` even if the current diff touches the test or nearby code, and even if the user claims ownership. Diff overlap or an author editing nearby code does not make an already-widespread flake self-introduced. Apply the signals below only when this pre-existing failure history is absent.

A flake is **self-introduced** when *any* signal holds (and the failure history above has not already classified it `pre-existing`):

1. **Diff overlap** (primary) — the current branch's changes touch the failing test's file, or a changed non-test file that plausibly feeds the failing code path. **First confirm the local checkout is the repository under investigation:** normalize `git remote get-url origin` (lowercase, strip scheme and trailing `.git`, and convert SSH re