/gke-workload-identity

name: gke-workload-identity
metadata:
  version: "1.0.0"
  category: Security
description: >-
  Diagnoses Workload Identity Federation for GKE authentication failures for Pods
  (403 "iam.serviceAccounts.getAccessToken" / permission denied, "could not find
  default credentials", or GKE metadata server unreachable) by verifying
  cluster and node-pool Workload Identity configuration, the Kubernetes
  ServiceAccount (KSA) to IAM binding (direct principal binding and legacy Google
  ServiceAccount impersonation), target-resource IAM roles, and gke-metadata-server
  health. Use when a Pod cannot authenticate to Google Cloud APIs even though
  Workload Identity is expected to be in effect. Don't use for in-cluster
  Kubernetes RBAC errors (API-server authorization), general workload crashes
  (use gke-workload-troubleshooting), or Workload Identity setup and hardening
  (use gke-workload-security).

GKE Workload Identity Federation Troubleshooting Skill

Use this skill to systematically diagnose why a Pod using **Workload Identity Federation for GKE** cannot authenticate to Google Cloud APIs. Typical symptoms:

• `HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on

resource`

• `google.auth.exceptions ... could not find default credentials` /

`ComputeEngineCredentials cannot find the metadata server`

• API calls that unexpectedly use the node's default Compute Engine service

account instead of the workload's identity.

Read-only boundary

This skill is **diagnostic and non-interactive**. It only **reads** cluster, IAM, and logging state and **proposes** fixes as commands or GitOps manifest changes for a human to apply. It must **never** create or modify IAM bindings, KSA annotations, node pools, or clusters automatically. When evidence is missing or the fix requires a privileged change, summarize findings and hand off to a human (see Step 7).

Output discipline (apply to every conclusion)

When you report a diagnosis, always:

1. **Name the single most-likely root cause** (not an open-ended list of possibilities). 2. **Explicitly rule out** the other plausible causes, citing the evidence that excludes them. In particular, when the cause is node-pool configuration, state plainly that it is **not** a KSA annotation or IAM binding problem; when the cause is a missing role on a target resource, state that Workload Identity itself is **not** misconfigured. 3. **Give the exact remediation as a proposed change for a human** — a concrete `gcloud` command (or GitOps manifest edit) with the real identifiers filled in — and never apply it automatically.

--------------------------------------------------------------------------------

Diagnostic Workflow

Step 0: Context discovery & time window

Collect (from the user or the failing resource): `PROJECT_ID`, `PROJECT_NUMBER`, `CLUSTER`, cluster `LOCATION`, `NAMESPACE`, the **KSA** the Pod runs as, the **node and node pool** the Pod is scheduled on (used in Step 2), the **target resource / API** being called, and the **exact error string**. Define a time window around the first observed failure for log queries.

# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"

# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
  -o jsonpath='{.spec.serviceAccountName}'

# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'

--------------------------------------------------------------------------------

Step 1: Capture the exact error signature

Read the workload's own logs and the `gke-metadata-server` logs to classify the failure.

kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"

Equivalent via Cloud Logging (preferred for historical events). Open it as a **Logs Explorer deep link** — URL-encode the query and append the project and time window: `https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}` (encode `/` as `%2F`, or use `;duration=PT1H` for a rolling hour):

resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING

**Classify the signature:**

• `iam.serviceAccounts.getAccessToken` denied / `HTTP/403` → the workload is

using the **GSA impersonation** path. This is expected for the legacy setup, but if you intend to use **direct binding**, it means the KSA still carries a leftover `iam.gke.io/gcp-service-account` annotation (or the client SDK is configured to impersonate) and is unintentionally impersonating a GSA; go to Step 3.

• `403 PERMISSION_DENIED` on the **target API/resource** (no `getAccessToken`

in the error) → the resolved identity lacks the required IAM role on that resource; go to Step 4.

• `could not find default credentials` / `cannot find the metadata server` →

metadata-server connectivity or a startup race; go to Step 5.

• Calls succeed but as the **node default service account** → Workload

Identity is not in effect for this node pool; go to Step 2.

--------------------------------------------------------------------------------

Step 2: Verify Workload Identity is enabled (cluster + node pool)

Both the cluster and the **node pool the Pod runs on** must have Workload Identity enabled. A node pool with `GCE_METADATA` (instead of `GKE_METADATA`) causes Pods to fall back to the node's default Compute Engine service account.

# Cluster must have a workload identity po