/logql-observability

name: logql-observability
description: "Queries workload logs with LogQL on Control Plane. Use to troubleshoot a workload from its logs, or for log search, access or egress logs, cron run logs, missing or truncated logs, retention, or logs in Grafana."

LogQL & Log Observability

Control Plane stores workload stdout/stderr in Loki and queries it with LogQL. The org is the Loki tenant — it comes from the endpoint path, so `org` is never a query label and queries cannot cross orgs. Reading logs requires the org-level `readLogs` permission, and the in-pod `CPLN_TOKEN` cannot authenticate to the logs endpoint — use a user or service-account token (see the `workload` skill). The recurring agent failure is passing a raw `query` to the MCP tool alongside structured params: a raw query replaces them entirely (the tool rejects the combination), so a raw query must embed every label itself.

Two ways to query

• **MCP (primary for agents):** `mcp__cpln__get_workload_logs`: structured params `gvc`, `workload`, `container`, `location`, `filter` (literal substring, LogQL `|=`, not regex); window `since` (default `1h`) or `from`/`to` (ISO 8601, `from` inclusive, `to` exclusive); `limit` (default 30, max 999, single request: `truncated: true` means narrow the window or filter harder); `order` (`oldest_first` default, or `newest_first`). For regex, parsers, or the `replica`/`stream`/`version` labels, pass a raw `query` (max 500 chars) instead of the structured selectors.
• **mk8s cluster logs** (the cluster's `logs` add-on): the same tool with `cluster` in place of `gvc`, narrowed by `namespace`, `pod`, `container`, `node`, and `job` (`kubernetes-pods`, `kubelet`, `kernel`, `docker-containers`, `audit`). These streams carry `cluster_name` and `cluster_alias`, never `gvc`; Control Plane workloads running on the cluster log under their GVC.
• **CLI:** `cpln logs '<LOGQL>'` — interactive debugging (live `--tail`), scripts, CI.

# Defaults: --since 1h, --limit 30, --direction forward
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --org ORG
cpln logs '{gvc="GVC", workload="WORKLOAD"} |= "error"' --limit 100
cpln logs '{gvc="GVC", workload="WORKLOAD", container="main"}' --since 7d
cpln logs '{gvc="GVC", workload="WORKLOAD"}' --tail              # live follow; the server ends a tail session after 30m
cpln logs '{gvc="GVC", workload="WORKLOAD"}' \
  --from 2026-06-01T00:00:00Z --to 2026-06-02T00:00:00Z          # ISO 8601 or relative (7d, now-1M); from inclusive, to exclusive
cpln logs '{gvc="GVC", workload="WORKLOAD"} |= "error"' --since 24h --limit 0   # 0 = unlimited, auto-paginates
cpln logs '{gvc="GVC", workload="WORKLOAD"}' -o jsonl            # one JSON object per line; -o raw = bare lines

• `--since` takes relative durations (`ms s m h d w mo y`, compound like `1h30m`). `--from`/`--to` (CLI) take an ISO 8601 timestamp **or** a relative duration meaning that long ago — `--from 7d`, `--from now-1M`, `--to now-30m` (the CLI accepts `M` for months; the `mcp__cpln__get_workload_logs` tool takes ISO 8601 only).
• `cpln workload eventlog WORKLOAD` (alias `cpln workload log`) is resource event history, not container output — for application logs always use `cpln logs`.

Labels

| Label | Value | |:---|:---| | `gvc` | GVC name | | `workload` | Workload name | | `container` | Container name, or a built-in stream below | | `location` | Deployment location, e.g. `aws-us-east-1` | | `provider` | Cloud provider | | `replica` | Replica (pod) name — unique per cron execution | | `stream` | `stdout` or `stderr` | | `version` | Workload deployment version that wrote the line |

At least one non-empty matcher is required; regex matchers work — `{gvc=~".+"}` spans every GVC in the org.

Filters and LogQL features

| Operator | Meaning | Example | |:---|:---|:---| | `\|= "text"` | contains | `\|= "error"` | | `!= "text"` | does not contain | `!= "health"` | | `\|~ "regex"` | matches regex | `\|~ "timeout\|crash"` | | `!~ "regex"` | does not match | `!~ "debug\|trace"` |

Loki is current (3.x), so full LogQL works: parsers (`| json`, `| logfmt`, `| pattern`, `| regexp`), post-parse label filters (`| latency > 100`), `line_format`, and metric queries (`count_over_time`, `rate`, `sum ... by`). The CLI and MCP tool print log lines only — run metric queries in Grafana: the `Explore on Grafana` link on the console Logs page opens it with the query prefilled (the org `grafanaAdmin` permission grants the Grafana Admin role; everyone else is Viewer).

{gvc="GVC", workload="WORKLOAD"} |= "error" != "health"               # errors minus noise
{gvc="GVC", workload="WORKLOAD"} |~ "panic|fatal|exception"           # crashes and stack traces
{gvc="GVC", workload="WORKLOAD", container="_accesslog"} |= "\" 50"   # HTTP 5xx in access logs
sum(count_over_time({gvc="GVC", container="_accesslog"} |= "\" 50"[1m])) by (workload)   # 5xx rate (Grafana)

Built-in log streams

| Selector | Contents | |:---|:---| | `container="_accesslog"` | Inbound requests (Envoy access-log format) on the workload's ports | | `container="_requestlog"` | The workload's outbound (egress) requests through the sidecar, same format | | `workload="_loadbalancer"` | Access logs of the GVC's dedicated load balancer | | `container="_alerts"` | Threat-detection (Falco) alerts, with extra labels `rule`, `priority`, `source` |

Platform health probes and unroutable-request noise are filtered out of `_accesslog` by design, so probe traffic never shows up there. Sidecar and system containers (`istio-init`, `istio-validation`, `cpln-*`, `debugger-*`) are never collected.

Why logs go missing (pipeline limits)

• **Lines over 16 KiB are cut** at 16 KiB with a `... [truncated N bytes]` suffix; empty lines are dropped.
• **Per-replica rate limit:** each replica+container pair is capped (roughly 10,000 lines/s on managed clusters; effectively unlimited on BYOK). Excess lines are dropped, and when collection resumes one marker line appe