/integration-test

name: integration-test
description: Run a live-instance verification of traceway-cli that goes beyond the Go smoke suite — exercises real-data detail endpoints, TTY-default rendering, adaptive metric-name discovery, and emits a human-readable coverage report. Invoke ONLY when the user explicitly asks (e.g. "run integration tests", "verify the CLI against stormwind"). Never invoke automatically after edits or commits. Assumes the user is already authenticated and a default project is configured.

integration-test — traceway-cli

A repeatable protocol for verifying the CLI end-to-end against a live Traceway server, focused on the things the Go smoke suite can't easily cover.

Trigger

Invoke ONLY when the user explicitly asks. Never run as a side effect of edits, commits, or builds.

Relationship to the Go smoke suite

The `just smoke-test` target (`test/smoke/*_test.go`, build tag `smoke`) is the primary regression check. Against a live instance it already covers:

• JSON shape of every list endpoint and `profiles list`.
• Output-format coverage (json/table/yaml) for projects, profiles, exceptions, endpoints, logs.
• Client-side enum validation (`--search-type`, `--order-by`, `--sort-direction`, `--aggregation`, `--tag`).
• Time-range parsing edges (`--since 7D`, missing `--to`, `--since` + absolute mix, far-future windows).
• `metrics query` missing `--name`, bogus name → empty series, malformed tag.
• `exceptions show <zeros>` → exit 5 `not_found`.
• `--profile no-such-profile` → exit 4; `--project 0…0` → non-zero, no panic, no `connection_failed`.

**Do not re-implement these here.** If they regress, that's a Go-test bug, not a skill failure.

What this skill adds beyond `just smoke-test`:

1. **Real-data detail endpoints** — `exceptions show <captured-hash>`, populated `metrics query --name <real>` with every aggregation + group-by. 2. **Adaptive metric-name discovery** — walk a candidate list until one populates. 3. **TTY-vs-pipe default** — table rendering to a real terminal. 4. **Coverage matrix report** — a human-readable artifact, on demand. 5. **Safety doctrine** — the forbidden-verb blocklist and `confirmMutation` env hygiene, applied to every probe.

Hard constraints

**Read-only. No exceptions.** Even if a subcommand looks safe by name, check `--help` for mutating flags before running.

Forbidden verbs and flags

Skip any subcommand whose name or `--help` mentions:

• `archive`, `unarchive`, `resolve`, `unresolve`, `mute`, `ack`, `acknowledge`
• `create`, `delete`, `update`, `set`, `put`, `post`, `add`, `remove`, `rm`
• `assign`, `claim`, `close`, `reopen`
• `login`, `logout`, `token`, `rotate`, `regenerate`
• `--archive`, `--resolve`, `--delete`, `--write`, `--mutate`, `--apply`, `--commit`

If a new subcommand is ambiguous (`sync`, `refresh`, `replay`, `export`), do not run it — list it under "skipped — manual review". If `--dry-run` exists, **still skip** write-shaped subcommands.

Mutation safeguards

The CLI gates mutations via `confirmMutation` (`cmd/traceway/querycommon.go`). The harness MUST:

• Never pass `--yes`.
• `unset TRACEWAY_ASSUME_YES` at the top of the script.
• Run with stdin from `/dev/null`.

So that if a forbidden verb slips through, the gate refuses with exit 2 `usage_error` instead of hanging on a prompt.

Pre-flight

Run in order. Stop if any fails.

1. Build: `nix develop --command go build -o ./bin/traceway ./cmd/traceway`. 2. Config exists (don't print — JWT inside): `test -f "${XDG_CONFIG_HOME:-$HOME/.config}/traceway/config.json"`. 3. Reachability + capture `TW_PROJECT_ID`:

   ./bin/traceway projects list --output json | jq -e 'type=="array" and length>=1' >/dev/null
   TW_PROJECT_ID=$(./bin/traceway projects list --output json | jq -r '.[0].id')

`projects list --output json` returns a **bare array**, not a `{data, pagination}` envelope.

Detail-endpoint probes

`exceptions show <captured-hash>`

1. Capture a real hash:

   HASH=$(./bin/traceway exceptions list --since 720h --page-size 1 --output json | jq -r '.data[0].exceptionHash // empty')

If empty, retry against other projects via `--project <id>`. If still empty, skip with reason `no exception found across all projects`. 2. With a real hash: three output formats + `--help`. JSON shape: `{group: {...}, occurrences: [...], pagination: {...}}` — assert `.group and .occurrences`. 3. Capture `.occurrences[0].traceId` if present for the logs probe below.

`metrics query --name <real-metric>` (adaptive)

Probe these names in order until one returns a populated `series`:

system.cpu.utilization
system.network.io
system.network.errors
system.network.dropped
http.server.duration
traceway.requests

If none populates, skip the live block with reason `no live metric name found`.

For the first metric that populates:

• Three output formats + `--help`.
• All aggregations: `avg`, `sum`, `count`, `min`, `max`, `p50`, `p95`, `p99`.
• `--interval-minutes 15`.
• `--group-by direction` for network metrics (splits `__all__` into `receive`/`transmit`).

JSON shape: `{results: [{name, unit, series: {<tag-key>: [{timestamp, value}, ...]}}]}` — `series` is a map keyed by group tag, default key `__all__`.

`logs query --trace-id <captured>`

If a real trace id was captured above, run `logs query --trace-id $TRACE --since 720h`. Assert exit 0 and `{data, pagination}` shape.

By-id detail commands (captured id + recordedAt)

These all **require** a timestamp flag; capture the id and its `recordedAt` together from `exceptions show`, then exercise them. All read-only.

1. Capture one occurrence's id + recordedAt (+ optional trace/session ids):

   OCC=$(./bin/traceway exceptions show "$HASH" --output json | jq -c '.occurrences[0]')
   OID=$(jq -r '.id'                 <<<"$OCC")
   OTS=$(jq -r '.recordedAt'         <<<"$OCC")
   DT=$(jq -r '.traceId // empty'    <<<"$OCC")
   SID=$(jq -r '.sessionId // empty' <<<"$O