/debug-cluster

name: Debug Cluster
description: "Provides systematic debugging approaches for HyperShift hosted-cluster issues. Auto-applies when debugging cluster problems, investigating stuck deletions, or troubleshooting control plane issues."

HyperShift Cluster Debugging Guide

This skill provides structured debugging workflows for common HyperShift hosted-cluster issues.

When to Use This Skill

This skill automatically applies when:

• Investigating hosted-cluster deletion issues
• Debugging stuck resources or finalizers
• Troubleshooting control plane problems
• Analyzing NodePool lifecycle issues
• Reviewing operator logs for cluster problems

Provider-Specific Troubleshooting

For provider-specific issues and detailed troubleshooting steps, refer to these subskills:

• **AWS**: [aws-troubleshooting.md](./aws-troubleshooting.md) - AWS-specific CAPI issues, infrastructure cleanup, and reinstallation

The main skill below provides provider-agnostic debugging workflows. When you encounter provider-specific issues, consult the relevant subskill for detailed resolution steps.

Key Components to Understand

Resource Hierarchy

• **HostedCluster (HC)**: Main cluster resource in the management cluster
• **HostedControlPlane (HCP)**: Control plane representation of the HC in HCP namespace
• **NodePool (NP)**: Worker node pool resources
• **CAPI Resources**: Cluster API resources (Cluster, Machine, etc.) in HCP namespace

Operators

• **hypershift-operator (HO)**: Manages HC and NP resources
• **control-plane-operator (CPO)**: Manages HCP and control plane components
• **hosted-cluster-config-operator (HCCO)**: Manages configuration and in-cluster resources for hosted clusters

Namespaces

• **HC namespace**: Where HostedCluster and NodePool resources live (e.g., `default`, `clusters`)
• **HCP namespace**: Where control plane pods and CAPI resources run (e.g., `clusters-<cluster-name>`)

Common Debugging Scenarios

Scenario: Hosted Cluster Stuck Deleting

When a hosted-cluster is stuck in deleting state, follow this systematic debugging process:

1. Node Pools Deletion

Check and verify NodePool deletion is progressing:

# Check NodePool resources in HC namespace
kubectl get nodepool -n <hc-namespace>

# Check CAPI cluster resource status in HCP namespace
kubectl get cluster -n <hcp-namespace> -o yaml

# Check CAPI provider pod logs
kubectl logs -n <hcp-namespace> deployment/capi-provider

# Check CAPI machines status in HCP namespace
kubectl get machines -n <hcp-namespace>
kubectl describe machines -n <hcp-namespace>

# Review HyperShift operator logs for NodePool issues
kubectl logs -n hypershift deployment/operator --tail=100 | grep -i nodepool
kubectl logs -n hypershift deployment/operator --tail=100 | grep -i <cluster-name>

**What to look for:**

• Finalizers blocking NodePool deletion
• CAPI machines that aren't terminating
• Provider errors preventing machine deletion
• HO logs showing reconciliation errors

2. HostedControlPlane Resource Deletion

Verify HCP resource and pods are being cleaned up:

# Check HCP resource status
kubectl get hostedcontrolplane -n <hcp-namespace> -o yaml

# Check pods in HCP namespace
kubectl get pods -n <hcp-namespace>

# Check for stuck pods
kubectl get pods -n <hcp-namespace> --field-selector=status.phase!=Running

# Review control-plane-operator logs
kubectl logs -n <hcp-namespace> deployment/control-plane-operator --tail=100

**What to look for:**

• HCP finalizers blocking deletion
• Pods with finalizers or in Terminating state
• CPO logs showing errors in resource cleanup
• PVC or other resources preventing namespace deletion

3. HCP Namespace Deletion

Investigate why the HCP namespace isn't being removed:

# Check namespace status
kubectl get namespace <hcp-namespace> -o yaml

# List all remaining resources in namespace
kubectl api-resources --verbs=list --namespaced -o name | \
  xargs -n 1 kubectl get --show-kind --ignore-not-found -n <hcp-namespace>

# Check for resources with finalizers
kubectl get all -n <hcp-namespace> -o json | \
  jq '.items[] | select(.metadata.finalizers != null) | {kind: .kind, name: .metadata.name, finalizers: .metadata.finalizers}'

# Review HO logs for namespace cleanup
kubectl logs -n hypershift deployment/operator --tail=100 | grep -i namespace

**What to look for:**

• Resources with finalizers preventing deletion
• API resources that HO should have cleaned up
• Webhook or admission controller errors
• Namespace stuck in Terminating state

4. HostedCluster Resource Deletion

Final check on the HostedCluster resource itself:

# Check HostedCluster status
kubectl get hostedcluster -n <hc-namespace> <cluster-name> -o yaml

# Check HostedCluster finalizers
kubectl get hostedcluster -n <hc-namespace> <cluster-name> -o jsonpath='{.metadata.finalizers}'

# Review HO logs for HostedCluster deletion
kubectl logs -n hypershift deployment/operator --tail=200 | grep -i "hostedcluster.*<cluster-name>"

**What to look for:**

• Finalizers blocking HostedCluster deletion
• HO errors in reconciliation loop
• Dependencies that haven't been cleaned up
• Cloud resources that failed to delete

Quick Debugging Checklist

When investigating cluster deletion issues:

• [ ] Check resource status and conditions
• [ ] Review relevant operator logs (HO, CPO)
• [ ] Inspect finalizers on stuck resources
• [ ] Verify CAPI resources are reconciling
• [ ] Check for events indicating failures
• [ ] Look for provider errors
• [ ] Verify namespace cleanup progress
• [ ] Check for webhook or admission errors

Common Issues and Resolutions

Issue: NodePool won't delete

• **Cause**: CAPI machines stuck due to provider errors
• **Resolution**: Check provider credentials, investigate machine deletion errors in HO logs

Issue: Machines stuck in "Deleting" phase with "WaitingForInfrastructureDeletion"

• **Cause**: Cluster resource deleted before machin