/performance-rca

name: performance-rca
license: Apache-2.0
compatibility: >
  Requires gcx v1.3.0 or later (https://github.com/grafana/gcx) with a context
  that can query a Grafana Pyroscope datasource. Optional go tool pprof for
  profile diffs.
description: >
  Investigates CPU, memory, and contention problems with Grafana Pyroscope
  profiles via the gcx CLI. Compares affected and baseline workloads, locates
  changed application stacks, and tests root-cause hypotheses against symptom
  and change evidence. Use when a service shows high CPU, growing memory, GC
  pressure, lock contention, or hot functions, when code regressed after a
  deploy, or when given a suspect service, flame graph, trace, or profiling
  anomaly report. For profiling setup use pyroscope; for profilecli-based
  analysis use profilecli-insights.

Performance RCA

Explain the observed performance problem: which workload changed, which code path accounts for it, and what evidence supports the mechanism. A hotspot is a location; an anomaly is a lead; neither alone proves a root cause. Stop at the strongest supported conclusion, including an inconclusive result. Code changes and proactive optimization are separate tasks.

Investigation contract

1. Follow Sections 1–5 in order. Reuse verified evidence from an earlier step or referring investigation; do not repeat discovery merely to satisfy the sequence. Establish each step's prerequisites before interpreting its result. 2. Fix the context, datasource, UTC windows, selectors, and profile type before comparison. Discover missing identifiers; never invent them. Record any justified change to the scope or comparison. 3. Qualify the control in Section 2 before claiming a regression. Without one, continue only to current cost attribution and label the comparison missing. 4. Distinguish empty results, unsupported capabilities, and query failures. Apply Section 1's exits to the affected profiling path; use its documented fallbacks only when their prerequisites hold. 5. Select output formats explicitly: JSON for discovery/metrics, table to rank functions, DOT to trace callers and callees, and pprof for export. Retain diagnostics and limits. 6. Always produce Section 5's report, including after an exit. Fill unsupported fields with `unknown`, `not measured`, or `not applicable`, plus a reason. A missing measurement must never become evidence that the service is healthy.

1. Preserve the question and check usable profiles

Reuse the context, datasource, service, symptom, incident window, deployment facts, and trace/profile links supplied by the user or a referring skill. Pin absolute UTC incident and baseline times. Derive a missing incident interval from timestamped symptom evidence, or ask for it before querying. Carry the same `--context <context>` on remote commands (omitted below), and pass the datasource UID explicitly. Keep the investigation read-only; do not switch contexts or change collection.

gcx help-tree profiles -o text

The examples below need gcx v1.3.0 or later; check with `gcx --version`. If gcx is missing or older, ask the user to install or upgrade it from https://github.com/grafana/gcx before continuing. Do not guess command renames or silently drop time filters. Commands, flags, and output formats can still be unsupported by the backend; Section 4 and the advanced reference describe those fallbacks.

# Discover only if the datasource UID is unknown; choose the target environment.
gcx datasources list -t grafana-pyroscope-datasource -o json
gcx profiles data-range -d <pyro-uid> -o json
gcx profiles list-profile-types -d <pyro-uid> \
  --from <incident-from> --to <incident-to> -o json

CLI support does not imply backend support. `data-range` returns tenant-wide `dataIngested`, `oldestProfileTime`, and `newestProfileTime` (Unix milliseconds; zero means unknown). These bounds do not prove that the selected service or lens has continuous coverage. Do not assume a fixed retention period.

Apply these outcomes before proceeding. Every exit stops the affected profile queries and produces the Section 5 report; other usable signals may continue through gcx's bundled `debug-with-grafana` skill (`gcx agent skills get debug-with-grafana`) with the missing-evidence question made explicit.

• **EXIT A — profiling unavailable:** no target datasource, missing access, or

a required CLI capability is unavailable. Record the exact limitation; use `gcx login` and `gcx config check` for missing setup or access, or a compatible gcx build for a CLI mismatch.

• **EXIT B — no ingestion reported:** a successful stats response says

`dataIngested: false`. Report the instrumentation/data gap, not service health.

• **EXIT C — incident not covered:** the incident has no overlap with known

queryable bounds. Report both ranges; do not replace the incident with today. For partial overlap, analyze only that overlap, record the coverage gap, and limit conclusions to the observed interval.

• **EXIT D — no usable incident profiles:** after checking the selector, window,

and lens against discovered facts, incident samples remain absent or zero. A missing baseline alone is handled in Section 2, not by this exit. Report the unresolved coverage/instrumentation gap, not proof of zero cost.

**Continue with scoped checks** when stats are absent, unsupported, or have unknown bounds. Missing stats alone must not block usable type/profile queries. These queries still need matching samples before they can support attribution.

For query errors, retain the operation and diagnostic. Correct a demonstrated selector/flag mistake; use a documented fallback for unsupported capabilities. Do not retry authentication failures or vary selectors blindly. A failed query is not an empty successful result.

2. Choose the lens and a qualified comparison

Copy exact profile type IDs from discovery, including their units. Discover labels r