/gke-inference

name: gke-inference
description: >-
  Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and
  model servers. Use when deploying GKE inference servers, configuring GKE GPU
  resources for inference, or deploying LLMs on GKE. Don't use for migrating
  existing AI workloads to GKE (use google-cloud-solution-guided-gke-ai-migration),
  GKE RAG with Cloud SQL/AlloyDB (use google-cloud-solution-rag-enterprise-search-gke-sqldb),
  or batch/HPC (use gke-batch-hpc).
metadata:
  version: "1.0.2"
  category: Containers

GKE AI/ML Inference

> **Routing Note:** For migrating existing AI workloads to GKE, open `google-cloud-solution-guided-gke-ai-migration/SKILL.md`. For GKE RAG with Cloud SQL or AlloyDB (`pgvector`), open `google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md`.

This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.

> **MCP Tools:** `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`, > `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`. > **CLI-only:** `gcloud container ai profiles *`

Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-inference)"` and append `agent-skills/1.0 (skill:gke-inference)` to `User-Agent` headers.

When to Use

• Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
• Generate optimized Kubernetes manifests for inference
• Select GPU/TPU accelerators for model serving
• Configure autoscaling for LLM inference

Prerequisites

• A golden path GKE Autopilot cluster (GPU workloads are supported via

ComputeClasses and NAP)

• `gcloud` CLI authenticated
• Sufficient GPU/TPU quota in the target region

Workflow

1. Discovery: Find Models and Hardware

# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet

2. Generate Manifest

gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml

**Parameters:**

• `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`)
• `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`)
• `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`,

`nvidia-h100-80gb`)

• `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token

(optional, for latency optimization)

**Example:**

gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml

3. Review and Deploy

# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>

> Some models require Hugging Face tokens. Create a Kubernetes Secret and > reference it in the manifest.

GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16

Accelerator Selection Guide

| Accelerator | Best For | Memory | Relative Cost | | ------------------- | ------------------------ | ----------- | ------------- | | NVIDIA T4 | Budget inference, | 16 GB | Lowest | : : lightweight legacy : : : : : models : : : | NVIDIA L4 (G2) | Small-medium model | 24 GB | Low | : : inference, video, : : : : : graphics : : : | NVIDIA RTX PRO 6000 | Multimodal AI, | 96 GB | Medium | : (G4) : high-fidelity 3D, : : : : : fine-tuning : : : | Cloud TPU v5e | Cost-effective | Varies | Medium | : : transformer inference : : : | Cloud TPU v5p | High-performance | Varies | High | : : training : : : | Cloud TPU v6e | High-efficiency next-gen | 32 GB/chip | Medium-High | : (Trillium) : training & serving : : : | Cloud TPU v7x | Ultra-scale inference & | 192 GB/chip | High | : (Ironwood) : agentic workflows : : : | NVIDIA A100 | Large model inference, | 40/80 GB | High | : : enterprise ML : : : | NVIDIA H100 / H200 | Frontier model training, | 80/141 GB | Highest | : : high throughput : : : | NVIDIA B200 (A4) | Blackwell-scale | 192 GB | Highest | : : training, FP4 precision : : : | NVIDIA GB200 (A4X) | Rack-scale AI (Grace | Massive | Highest | :