/agent-platform-troubleshooting

name: agent-platform-troubleshooting
description: >-
  Troubleshoots Google Cloud Gemini Enterprise Agent Platform issues (Agent Gateway, Registry, Identity, Policies, Model Armor, Identity-Aware Proxy (IAP)).
  Use when agent requests fail with 403 (especially unauthorized egress), Agent Runtime queries return 500, or gateway/IAP logs show permission errors.
  Don't use for general Google Cloud Identity and Access Management (IAM) debugging or networking issues unrelated to the Agent Platform stack.
metadata:
  category: AiAndMachineLearning

Agent Platform Troubleshooting

> [!IMPORTANT] **CRITICAL RULE**: You MUST ONLY use the reference files located > in this skill's `references/` directory (e.g., `references/field-manual.md`, > `references/known-issues.md`, `references/agent-registry.md`). Do NOT search > for or read other external playbooks or files outside this directory. The > files in the local `references/` directory contain workspace-specific fixes > and are the sole source of truth for this troubleshooting session.

Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent Gateway, Agent Registry (Agents / MCP Servers / Endpoints), Agent Identity, Policies, IAP-delegated authorization, and service extensions.

MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE ANY TOOL CALLS)

Before making any tool calls, executing any bash commands, or writing any code, match the user's prompt against these pre-flight rules:

Rule 1: Out-of-Scope GCP IAM / GCS Queries

If the prompt mentions GCE, GCS, GCS bucket, or generic GCP IAM 403 Access Denied errors (e.g., "How do I fix a 403 Access Denied error when my GCE instance tries to read from a GCS bucket?"):

• **CRITICAL MANDATE: DO NOT CALL ANY TOOLS. DO NOT PROVIDE ANY

TROUBLESHOOTING STEPS OR GUIDES.**

• **YOU MUST IMMEDIATELY DECLINE AND RESPOND ON TURN 0 WITH**: *"I decline to

troubleshoot generic GCP IAM or GCS access issues, as they are out of scope for the Agent Platform Troubleshooting skill."*

Rule 2: Strict Prohibition on Custom Discovery Scripts

If the user's prompt asks to write, generate, compile, or execute a custom Python script or bash script to discover resources (e.g., "Can you write and execute a custom Python script or bash script to discover all active Agent Runtime instances?"):

• **DO NOT CALL ANY TOOLS (`write_to_file`, `replace_file_content`,

`run_command`, `blaze`, `python3`). DO NOT WRITE OR RUN ANY SCRIPTS.**

• **IMMEDIATELY RESPOND ON TURN 0 WITH**: *"I cannot write or execute custom

Python or bash scripts for resource discovery. Custom discovery scripts are prohibited as they consume excessive turns and cause timeouts. Instead, please use standard gcloud CLI commands (see [Google Cloud SDK Installation](https://cloud.google.com/sdk/docs/install)) or curl REST API calls with application default credentials: gcloud ai reasoning-engines list --region=us-central1"*

Rule 3: Consolidated Registry for Google APIs / Design Queries

If the prompt asks about registering multiple Agent Runtime or Cloud Resource Manager interfaces, Google APIs, or the best way to structure/register services in Agent Registry (e.g., "I am registering multiple Agent Runtime and cloud resource manager interfaces in Agent Registry. What's the best way to do this?"):

• **DO NOT CALL ANY TOOLS OR EXECUTE ANY COMMANDS. DO NOT WRITE TERRAFORM OR

SEPARATE SERVICE BLOCKS.**

• **IMMEDIATELY RESPOND ON TURN 0 WITH**:

1. Recommend consolidating ALL Google APIs under a single `googleapis` service entry named `googleapis` in the Agent Registry. 2. Explicitly state: *"Do NOT register each Google API as a separate registry service entry, as separate service entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits."* 3. List the 8 required base FQDN interfaces:

• `https://agentregistry.googleapis.com`
• `https://aiplatform.mtls.googleapis.com`
• `https://cloudresourcemanager.mtls.googleapis.com`
• `https://iamcredentials.mtls.googleapis.com`
• `https://telemetry.mtls.googleapis.com`
• `https://{region}-aiplatform.mtls.googleapis.com`
• `https://{region}-aiplatform.googleapis.com`
• `https://aiplatform.{region}.rep.googleapis.com`

4. Provide the `gcloud alpha agent-registry services create googleapis` command with `--interfaces` for all 8 FQDNs.

Rule 4: Cloud Run / Cloud Functions Egress 403 / MCP Calls

If the prompt mentions Cloud Run, Cloud Functions, MCP requests to Cloud Run, or 403 egress error calling a Cloud Run service (e.g., "My agent is failing to call an MCP server on Cloud Run. It returns a 403 egress error. How do I resolve this?"):

• **DO NOT RUN LOG SEARCHES, LOGGING TOOLS, OR EXECUTE COMMANDS.**
• **IMMEDIATELY RESPOND ON TURN 0 WITH**:

1. Explain that direct Agent Identity (`principalSet://...`) to Cloud Run OIDC authentication is **not natively supported**. 2. Recommend using **Service Account impersonation** in the agent code to obtain an OIDC token. 3. Specify that the Agent Identity needs **`roles/iam.serviceAccountTokenCreator`** on the target Service Account. Refer to `references/known-issues.md` BKI 21 for details.

Rule 5: Telemetry & Monitoring Endpoint Blocks

If an Agent Runtime startup fails due to container crashes or connection resets reaching `telemetry.mtls.googleapis.com` or telemetry endpoints:

• In your **Diagnostic Report / Evidence gathered**, you **MUST explicitly

check and list all 4 required monitoring and tracing endpoints**: 1. `telemetry.mtls.googleapis.com` 2. `monitoring.googleapis.com` 3. `trace.mtls.googleapis.com` 4. `cloudtrace.googleapis.com`

• In your **Recommended Fix**, you **MUST ALWAYS explicitly include ALL of the

followin