/diagnose-entity-graph

name: diagnose-entity-graph
description: >-
  Diagnoses Entity Graph (knowledge graph) problems: missing entities or
  services, missing edges, disconnected clusters, environment filters that do
  not work. Use when the Entity Graph is empty or looks wrong.

Diagnose Entity Graph

Systematic diagnosis of Entity Graph problems using gcx commands. Follow the steps in order — each step narrows the cause. Be direct and report findings concisely.

Start here: read the Orientation block

`gcx kg diagnose` prints an Orientation block above its check table. Read it first. It tells you which (if any) of the five common Entity Graph scenarios the run matches:

• "I see no entities at all"
• "Some expected entities are missing"
• "I see entities with no edges"
• "I see disconnected clusters of entities"
• "I can't filter to the entities I want"

When the Orientation block names a matched scenario, follow its `Next:` commands. The per-step playbook below is the fallback for runs where no scenario matched or the matched scenario's hints didn't resolve the issue.

Reading Diagnose Output

Treat `gcx kg diagnose`'s verdicts as authoritative for the queries it ran. Two non-obvious classifications:

• **`WARN — metric exists … but no series match the requested

env/namespace scope`** means the metric is flowing on the stack but doesn't carry the scoped label value. This is a label-mapping issue (asserts_env vs. deployment_environment, etc.), not a missing-data issue. Investigate the label pipeline (Step 6) before suggesting the user enable new telemetry.

• **`FAIL — no data` (without the WARN above)** means the metric was

not found, even unscoped — the integration / recording rule is genuinely absent.

For ad-hoc PromQL outside `kg diagnose`, apply the same discipline: re-run the query without the env / namespace filter before concluding the data is missing.

If a user reports an entity by name, see Step 7's entity-existence workflow before assuming it exists on this stack.

Prerequisites

gcx must be installed and configured with a valid context. `gcx kg diagnose` bundles most of the per-step checks below; the individual commands produce equivalent results when you need to drill into one check.

gcx config view
gcx kg status

If `kg status` fails with a config or auth error (no context, connection refused, 401), use the `setup-gcx` skill first. Do not route every error there: a 404 means the Asserts plugin isn't installed on this stack, and 403s from entity endpoints while `kg status` itself succeeds usually mean the Knowledge Graph isn't onboarded (or the token lacks plugin access) — those are Asserts onboarding/permission issues `setup-gcx` cannot fix; handle them via Step 1's onboarding stop instead.

Step 1: Stack Health

gcx kg status

**Check:** `status` must be `"complete"` and `enabled` must be `true`. If not, the Knowledge Graph hasn't been onboarded — stop here and direct the user to the Asserts app onboarding flow.

**Shortcut:** `gcx kg diagnose` runs this plus all subsequent checks in parallel.

Step 2: Entity Counts and Scopes

gcx kg stats --since 1h
gcx kg meta scopes

**Check:** entity counts should be > 0. The `meta scopes` output shows available `env`, `site`, and `namespace` values.

If scoping to a specific environment, note the exact `env` value — you'll use it in all subsequent queries.

Step 3: Source Metrics in Mimir

Check whether the raw telemetry that feeds Entity Graph exists. Raw Tempo metrics use `deployment_environment`, not `asserts_env`.

Note the label shape difference between the two metrics: `traces_target_info` describes a single service so it has one `deployment_environment` label; `traces_service_graph_request_total` describes an edge between two services and exposes the env on both sides as `client_deployment_environment` and `server_deployment_environment` — there is no unified `deployment_environment` label.

# Service identity (OTel traces)
gcx metrics query 'count(traces_target_info)' --since 1h
gcx metrics query 'count(traces_target_info{deployment_environment="ENV"})' --since 1h

# Call data (inter-service HTTP/gRPC)
gcx metrics query 'count(traces_service_graph_request_total)' --since 1h
# Filter on server side (use client_deployment_environment for outbound view):
gcx metrics query 'count(traces_service_graph_request_total{server_deployment_environment="ENV"})' --since 1h

**Interpret:**

• Both have data → traces are flowing. Continue to Step 4.
• Both empty → no OTel traces for this environment. Entities may still exist

via Prometheus scraping. Continue to Step 4.

For more specific verdicts on this metric pair (Tempo metrics generation disabled, broken trace context propagation), run `gcx kg diagnose --env ENV` and read the check results — the command encodes the detection logic for those two cases and emits a targeted recommendation. Service-name collisions are **not** detected by the command; spot them manually via self-loop edges (Step 7).

Step 4: Recording Rules

Recording rules convert raw metrics into the `asserts:*` metrics that Entity Graph consumes. These use `asserts_env`, not `deployment_environment`.

# Entity discovery (central to how services appear)
gcx metrics query 'count(asserts:mixin_workload_job{asserts_env="ENV"})' --since 1h

# CALLS edges
gcx metrics query 'count(asserts:relation:calls{asserts_env="ENV"})' --since 1h

# Request rate KPI
gcx metrics query 'count(asserts:request:rate5m{asserts_env="ENV"})' --since 1h

**Interpret:**

• `asserts:mixin_workload_job` has data but `asserts:relation:calls` doesn't →

entities are discovered but no edges exist. Continue to Step 5.

• All empty → recording rules aren't producing output. Check Step 6 (labels).
• All have data → pipeline is healthy. For a specific missing service, go to Step 7.

Step 5: Edge Source Analysis

CALLS edges can come from many sources, not just OTel traces