/workload-troubleshooting

name: workload-troubleshooting
description: "Diagnoses unhealthy Control Plane workloads. Use when asked why a workload is crashing, not starting, OOMKilled, ImagePullBackOff, returning 502s, failing health checks, unreachable, or stuck deploying."

Workload Troubleshooting

The symptom-first companion to the `workload` skill (which owns workload types, the spec shape, and the create/update tools). Given an unhealthy workload, map what you observe to its platform-specific root cause and a fix the schema will actually accept. Diagnosis is **read-only and MCP-first**; most failures trace to a Control Plane rule a generic engineer would not guess — deny-by-default firewalls, the secret identity+policy chain, blocked ports, the sleep-binary shutdown rule. The single most common is OOMKilled. Deep remediation for each area lives in the domain skill named in that section; this skill is the diagnostic map.

Step 1 — Gather state (read-only)

| Tool | What it tells you | |---|---| | `mcp__cpln__list_deployments` | **Start here.** Per-location readiness with reason/message. Pass `location` to drill into one failing location. | | `mcp__cpln__get_workload_events` | Image pulls, crashes, scheduling, probe failures, `OOMKilled`. | | `mcp__cpln__get_workload_logs` | App logs (LogQL); the `_accesslog` container holds HTTP status codes and latency. | | `mcp__cpln__get_resource` (kind=`workload`) | The spec and current status. | | `mcp__cpln__list_metrics` then `mcp__cpln__query_metrics` | Resource pressure — memory before OOM, CPU, latency. | | `mcp__cpln__list_workload_replicas` | Confirm which replicas are currently running. Use the `cpln` CLI after reading the `cpln` skill if in-container inspection is essential. | | `mcp__cpln__query_traces` then `mcp__cpln__get_trace` | For a slow or intermittently failing request: which span in the path spent the time or errored. Needs tracing enabled on the GVC (opt-in). Deep dive in `metrics-observability`. |

CLI fallback (MCP unavailable, interactive shell, or CI/CD):

cpln workload get WORKLOAD --gvc GVC -o json
cpln workload eventlog WORKLOAD --gvc GVC -o json
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --limit 50          # |= "error" filters; container="_accesslog" for HTTP codes
cpln workload connect WORKLOAD --gvc GVC --location LOCATION      # interactive shell

Failure catalog

Out of memory (OOMKilled) — the most common issue

**Symptoms:** container restarts repeatedly, events show `OOMKilled`, crashes under load.

`memory` is a hard cap — exceed it (app + runtime + GC + buffers) and the kernel kills the container. Usual culprits: Java without `-Xmx`, Node without `--max-old-space-size`, Python loading large datasets. Each container, sidecars included, has its own limit. With **Capacity AI** on, spiky workloads can be downsized too aggressively — set `minMemory` as a floor (Capacity AI never downscales CPU below 25 millicores).

**Fix:** check real usage with `query_metrics`, then raise `memory` — but memory (MiB) must stay within **8× CPU (millicores)**, so raise `cpu` alongside it or the update is rejected (the default `cpu: 50m` caps memory at 400Mi). See *Apply fixes within the schema's limits*.

Image pull failures

**Symptoms:** events show `ImagePullBackOff` / `ErrImagePull`, deployment stuck.

• **Reference format** — `//image/NAME:TAG` (org registry), bare `NAME:TAG` (Docker Hub, no `docker.io/`), full URL (other registries).
• **Platform** — images must be `linux/amd64` for managed locations (BYOK allows more).
• **Pull secret** — a private external registry needs a pull secret in the GVC's `pullSecretLinks`; only `docker`, `ecr`, `gcp` secret types work, and org `//image/` images need none.

The registry secret must already exist (created by the user — offer a manifest scaffold for them to fill and apply, `setup-secret` skill); attach with `update_gvc`. Deep setup: `image` skill.

Secret access failures

**Symptoms:** env vars empty, logs show missing config, secret-access errors in events.

A workload reaches a secret only with all three in place: an **identity linked** to it (`spec.identityLink`), a **policy granting that identity `reveal`** on the secret, and a **correct reference**. Fastest fix: `grant_workload_secret_access` builds the whole chain in one call. Reference format by type:

| Type | Reference | |---|---| | Opaque (decoded / raw) | `cpln://secret/NAME.payload` / `cpln://secret/NAME` | | Dictionary | `cpln://secret/NAME.KEY` | | Username & password | `cpln://secret/NAME.username` / `.password` | | Keypair | `cpln://secret/NAME.secretKey` / `.publicKey` / `.passphrase` | | TLS | `cpln://secret/NAME.cert` / `.key` / `.chain` | | AWS | `cpln://secret/NAME.accessKey` / `.secretKey` / `.roleArn` / `.externalId` |

The manual chain: `access-control` and `setup-secret` skills.

Port mismatch — healthy but 502/503

• The spec port must match what the process listens on — compare the workload spec with application startup logs. If that is inconclusive, use the `cpln` CLI after reading the `cpln` skill for an in-container socket check.
• On serverless, the runtime injects `PORT` and rejects a `PORT` env var that doesn't equal the exposed port.
• **Type rules:** serverless exposes exactly one port (zero is rejected; TCP needs a dedicated LB — see *Dedicated load balancer & domain*); standard and stateful expose zero or more; cron serves no endpoint.
• **Blocked ports** (cannot bind, invalid for TCP probes): `8012, 8022, 9090, 9091, 15000, 15001, 15006, 15020, 15021, 15090, 41000`.

Firewall blocking traffic

**Symptoms:** unreachable externally, can't reach external APIs, or can't talk to other workloads.

Deny-by-default: external inbound disabled, external outbound disabled, internal `none`. Fix via `update_workload`:

• **Inbound** — `external.inboundAllowCIDR` (e.g. `0.0.0.0/0`, or specific CIDRs).
• **Outbound** — `external.outboundAllowCIDR`, or `outboundAllowHostname` (hostname r