/gke-ai-troubleshooting-jobset-interruption

name: gke-ai-troubleshooting-jobset-interruption
metadata:
  category: Containers
description: >-
  Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI/ML training workloads autonomously.
  Use when troubleshooting JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes.
  Don't use for general GKE cluster creation, basic workload deployment, or non-JobSet application issues.

GKE JobSet Interruption Troubleshooting

Use this skill to systematically diagnose and resolve JobSet interruptions, restarts, and preemptions on GKE clusters hosting large-scale AI/ML workloads.

⚠️ Prerequisites & Sandbox Rules

• JobSet metrics package must be enabled in `kube-state-metrics` for your

cluster.

• Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.
• **Autonomous & Sandbox Execution Rule**: If API queries, tools, or commands

encounter `403 Permission Denied`, authentication errors, or network isolation, **do NOT enter authentication or credential troubleshooting loops**. Populate the query templates with the acquired variables (`{project_id}`, `{cluster_name}`, `{workload_name}`, `{start_time}`, `{end_time}`), inspect any locally staged telemetry or mock data files if available, and complete the diagnostic workflow and resolution recommendations autonomously.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather context using tools, workspace files, environment details, or user prompt context:

• **Project ID** (`{project_id}`)
• **Cluster Name** (`{cluster_name}`)
• **Workload Name (JobSet Name)** (`{workload_name}`)
• **Workload Namespace** (`{namespace}`)
• **Issue Time** (`{issue_time}`)

If specific variables are not explicitly provided by the user, inspect cluster resources or logs to determine them, or use the `{variable}` placeholders provided.

Time Handling Rules

1. **Autonomous Time Window**: If a relative time (e.g., "X minutes ago") or no exact timestamp is provided, calculate the query window based on current time or available log timestamps. 2. **Window Calculation**: If a timestamp `{issue_time}` is available (or calculated as `T`), set `{start_time}` = `T - 30m` and `{end_time}` = `T + 30m`.

--------------------------------------------------------------------------------

Step 1: Identify JobSet Restarts and Attempts [Low Risk]

Verify if the JobSet is experiencing restart loops and determine the frequency of restarts.

Visual Chart / MQL Query - restarts

• **MQL Query Specification**:

    fetch prometheus_target
    | metric 'prometheus.googleapis.com/kube_jobset_restarts/gauge'
    | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'
    | align next_older(1m)
    | every 1m
    | group_by [metric.jobset_name], [val: max(value)]

PromQL Metric Query - restarts

• **PromQL Query Specification**:

    kube_jobset_restarts{jobset_name="{workload_name}", cluster="{cluster_name}"}

• **Diagnostic Logic**: A non-zero or increasing value for restarts indicates

that the JobSet is being actively restarted by the controller due to worker failure or interruption.

• **Automation**: Proceed to Step 2 automatically after reporting findings.

--------------------------------------------------------------------------------

Step 2: Inspect Nodepool Interruptions [Low Risk]

Determine if the JobSet restarts were triggered by physical nodepool-level events (such as spot preemptions, maintenance, or host terminations).

A. Metrics Query (Nodepool Interruption Counts)

Visual Chart / MQL Query - interruptions

• **MQL Query Specification**:

    fetch k8s_node_pool
    | metric 'kubernetes.io/node_pool/interruption_count'
    | filter cluster_name == '{cluster_name}'
    | align next_older(10m)
    | every 10m
    | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]

PromQL Query - interruptions

• **PromQL Query Specification**:

    sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (
      avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name="{cluster_name}"}[10m])
    )

B. Log Query (Nodepool Life Events)

• **LQL Log Filter Specification**:

    resource.type="gke_nodepool"
    AND resource.labels.cluster_name="{cluster_name}"
    AND timestamp >= "{start_time}"
    AND timestamp <= "{end_time}"

• **Diagnostic Logic**:

• **PreemptionEvent**: Spot VMs were preempted, or node was scale-down.
• **MaintenanceEvent**: Node pool updated or Google scheduled maintenance.
• **TerminationEvent**: Serious host failures. Check `interruption_reason`

or logs for host issues.

• See [Failure Signatures](references/failure_signatures.md) for examples

of node termination logs and preemption events.

• **Automation**: Proceed to Step 3 automatically.

--------------------------------------------------------------------------------

Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]

Correlate node readiness failures with physical host VMs to see if a single faulty host repeatedly fails coordinator pods.

A. Metrics Query (Node Ready Status Check)

Visual Chart / MQL Query - node status

• **MQL Query Specification**:

    fetch k8s_node
    | metric 'kubernetes.io/node/status_condition'
    | filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'
    | align next_older(1m)
    | every 1m
    | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]

PromQL Query - node status

• **PromQL Query Specification**: