/dd-triage-flaky-test

name: dd-triage-flaky-test
description: Load when investigating a specific flaky test. Gets history, failure pattern, and category, then recommends fix, quarantine, or escalate.
metadata:
  version: "1.0.0"
  author: datadog-labs
  repository: https://github.com/datadog-labs/agent-skills
  tags: datadog,ci,cicd,flaky,flaky-tests,test-optimization
  alwaysApply: "false"

Triage Flaky Test

One-line summary: Investigate a specific flaky test — get history, failure pattern, and category, then recommend fix, quarantine, or escalate.

Requires: `dd-pup` skill (pup CLI installed and authenticated).

---

Input

| Parameter | Description | |---|---| | Test name | Fully qualified test name (e.g. `TestMyFunc` or `com.example.MyTest`) | | Repository | Lowercase, no-schema URL (e.g. `github.com/org/repo`). Derive from `git remote get-url origin` if not provided. |

---

Workflow

STEP 0 — Parse Input

Derive repository ID from git if not provided:

git remote get-url origin
# Strip protocol and trailing .git, then lowercase the result
# e.g. https://github.com/DataDog/my-repo.git → github.com/datadog/my-repo

**Validation fallback:** If STEP 1 returns no results, confirm the correct repository by searching without a repo filter:

pup cicd tests search \
  --query "@test.name:\"<test-name>\"" \
  --from 30d \
  --limit 5

Extract `@git.repository.id_v2` from results and retry STEP 1 with the confirmed value.

STEP 1 — Get Flaky Test Details

**Preferred — use `fingerprint_fqn` if known** (`fingerprint_fqn` is a valid CI Visibility search facet, distinct from `flaky_state`):

pup cicd flaky-tests search \
  --query "fingerprint_fqn:<fqn>" \
  --sort="-last_flaked" \
  --limit 5

**Fallback — use name + suite + repo:**

pup cicd flaky-tests search \
  --query "@test.name:\"<test-name>\" @test.suite:\"<suite>\" @git.repository.id_v2:\"<repo>\"" \
  --sort="-last_flaked" \
  --limit 10

Omit `@test.suite` if unknown; if the same test name appears in multiple suites, pick the entry whose suite matches the failing test.

Do not filter by `flaky_test_state` — return the test regardless of state.

Note: the query filter facet is `flaky_test_state`; the returned response attribute is `flaky_state` — these are different names for the same concept; do not use `flaky_state:active` as a query filter.

Extract from results:

• `fingerprint_fqn` — unique test identifier; used as the `id` in STEP 5 write call. **If absent, do not proceed to quarantine — see STEP 5.**
• `flaky_state` — current state (active / quarantined / disabled / fixed)
• `test_stats.failure_rate_pct` — percentage of runs that fail
• `flaky_category` — root cause category
• `codeowners` — owning team
• `pipeline_stats.total_lost_time_ms` — total CI time lost

STEP 2 — Get Recent Failure History

pup cicd tests search \
  --query "@test.name:\"<test-name>\" @test.suite:\"<suite>\" @test.status:fail @git.repository.id_v2:\"<repo>\"" \
  --from 7d \
  --limit 20

Extract:

• Error messages and stack traces (`@error.message`, `@error.stack`)
• Failing branches (`@git.branch`) — branch-specific vs. widespread
• Frequency pattern — random timing or specific conditions
• Unique `@ci.pipeline.id` values for blast radius (STEP 3)

STEP 3 — Check Blast Radius

Count distinct pipelines impacted using pipeline IDs from STEP 2:

pup cicd events aggregate \
  --query "@ci.status:error @ci.pipeline.id:(<id1> OR <id2> OR ...) @git.repository.id_v2:\"<repo>\"" \
  --compute count \
  --group-by "@ci.pipeline.name" \
  --from 7d

Use the first 10 pipeline IDs from STEP 2 (cap at 10; if more are available, run a second batch and merge results by summing counts per `@ci.pipeline.name` across batches). Report blast radius as: total number of unique pipelines impacted and whether failures are branch-specific or widespread.

Note: a pipeline failure is not necessarily caused solely by this flaky test — treat blast radius as a signal, not a definitive count.

STEP 4 — Recommend Fix or Quarantine

Use `flaky_category` from STEP 1 and error messages from STEP 2.

**Root cause first:**

• Read the full error trace from bottom to top — chained errors hide the real cause; the innermost error is the root cause, not the first line.
• Identify the exact source of nondeterminism (race, ordering, stale state, timing).
• If the root cause is a CI infrastructure problem (runner unavailable, Docker daemon failure, network outage) → do NOT propose a code fix; classify as `infra` and recommend retry instead.
• If root cause is uncertain and cannot be confirmed from the stack trace → skip fix, go to quarantine.

**Fix at the correct layer:**

• Test issue → fix in test or test helper only.
• Production bug exposed by the test → fix in production code.
• Shared helper used by multiple tests → fix the helper AND update all call sites; never patch a single test when the root cause is in shared code.

**Forbidden — do not propose these:**

• Timing hacks: increasing timeouts, adding sleeps, widening time windows, adding retries. A fix that only reduces flake probability without eliminating the root cause is invalid.
• Masking: relaxing assertions (e.g., exact match → at least 1), dropping validations.
• Partial fixes: touching one call site when multiple share the root cause.

**Fix patterns by category:**

| Category | Approach | |---|---| | `timeout` | Identify the slow operation and make it synchronous or deterministic — do NOT simply raise the timeout constant | | `concurrency` | Add deterministic synchronization (barriers, channels, locks); remove shared mutable state between tests | | `network` | Mock or stub network calls at the boundary; if the test requires a real connection, isolate it with a test server | | `time` | Inject a controllable clock; replace wall-clock assertions with relative or event-driven checks | | `order_dependency` | Isolate test state with setup/teardown; eliminat