/deploy-status

name: deploy-status
description: Check civitai PROD deployment status across the live Tekton -> Flux -> Flagger chain on the DataPacket cluster (kubectl, read-only). Tekton/Flagger cluster state is the primary truth; the GitHub Deployments API is kept as a public cross-check. Use to see where a deploy is in the chain, watch it to completion, or debug a build/canary failure.
argument-hint: status [<tag|sha>] | watch [<tag|sha>] | gh-recent | gh-sha:<commit> | logs:<run-name>
allowed-tools:
  - Bash(kubectl:*)
  - Bash(bash:*)
  - Bash(gh:*)
  - Bash(curl:*)
  - Bash(open:*)
  - Bash(xdg-open:*)

Deploy Status

Monitor the **real** civitai PROD deploy chain — Tekton build -> Flux image automation -> Flagger canary -> prod primaries — by reading live cluster state with `kubectl`. The GitHub Deployments API is retained as a public cross-check (its source is the `github-create-deploy` Tekton task).

**Primary truth = cluster state** (this skill's `status` / `watch`). **Cross-check = GitHub Deployments** (`gh-*` commands below).

Cluster requirement (READ-ONLY)

All `status` / `watch` queries run against the DataPacket cluster via kubectl context **`civit-datapacket`**. One kubeconfig covers the whole chain (build ns `tekton-builds`, Flux ns `flux-system`, app+canary ns `civitai-dp-prod`).

This skill performs **status reads only** — `kubectl get` / `kubectl logs`. It NEVER mutates the cluster (no apply/delete/patch/rollout/scale). To *trigger* or *roll back* a deploy, use the talos-infra `dp-build-deploy` skill instead.

CRITICAL — prod is NOT keyed off the semver tag

A release maps to prod via the **release-branch commit**, not the tag object. Getting this wrong reports a stage/next build as prod.

| Run name pattern | Branch label | Image | Target | Prod? | |---|---|---|---|---| | `civitai-web-build-*` | `release` | `ghcr.io/civitai/civitai-prod` | civitai.com | **YES** | | `civitai-web-tag-build-*` | `<semver>` (e.g. `v5.0.1817`) | `ghcr.io/civitai/civitai-web` | next/stage | no — EXCLUDE | | `civitai-web-main-build-*` | `main` | civitai-prod (throttled) | — | no — EXCLUDE | | `pr-preview-*` / `pr-check-*` | _(none)_ | — | PR preview | no — EXCLUDE |

The semver tag (`vX.Y.Z`) fires a **different** trigger (`civitai-app-tag-trigger`) that builds `civitai-web` (stage) on the tag commit. The PROD run is the `civitai-web-build-*` on the **release-branch** commit. Example: tag `v5.0.1817` -> tag run on commit `5f19831` (civitai-web, NOT prod); the prod run is release commit `3dcd1e8` = `civitai-web-build-cp9kz`.

The prod selector that enforces this:

kubectl --context civit-datapacket -n tekton-builds get pipelinerun \
  -l pipeline=build-and-push,pipeline.jquad.rocks/git.repository.branch.name=release \
  --sort-by=.metadata.creationTimestamp

Usage (cluster — primary)

Driver script: `deploy-chain.sh` (in this skill dir). Dependency-light bash + kubectl.

• `bash deploy-chain.sh status` — full chain snapshot for the **latest** release-branch prod run.
• `bash deploy-chain.sh status <sha>` — snapshot for the prod run matching that commit (full or short sha, matched on the `git.repository.branch.commit` label). A raw sha for a tag/stage build correctly resolves to **no prod run** (it is never reported as prod).
• `bash deploy-chain.sh status <tag>` — a semver tag (e.g. `v5.0.1817` / `5.0.1817`, must contain a dot) is resolved to the **release-branch HEAD** via `gh`, then matched to its prod run.
• `bash deploy-chain.sh watch [<tag|sha>]` — poll the chain to completion, printing each transition; exits 0 only when **every serving pool is fully ROLLED** onto the new image (`updatedReplicas == readyReplicas == desired` for all app deployments), exits 1 if the build fails or a canary rolls back. Not merely when the deploy spec image flips.

What `status` prints (the full phase chain)

1. **BUILD** — PipelineRun `.status.conditions` + per-task TaskRuns (ns `tekton-builds`). Task order: `notify-preparing -> github-create-deploy -> fetch-repository -> build-image -> migrations` (plus trailing `github-*`/`notify-*` tasks). `github-create-deploy` is what writes the GitHub Deployments entries. 2. **IMAGE PICKED UP** — Flux ImagePolicy `flux-system/civitai-prod-release` `.status.latestImage` (`ghcr.io/civitai/civitai-prod:<14-digit-ts>-<sha7>`). ~1m GHCR scan. 3. **TAG PINNED TO GIT** — `ImageUpdateAutomation` (`flux-system/civitai-dp-prod`) commits the new tag to trunk (<=5m); reflected once primaries pick it up. 4. **CANARY** — two Flagger Canary CRs in `civitai-dp-prod`: `civitai-dp-prod` (**SSR**) and `civitai-dp-prod-api` (**API / tRPC path**). Reads `.status.phase`, `.status.canaryWeight`, `.status.iterations`, `.status.failedChecks` + recent Warning events (rollbacks). 5. **PRIMARIES (100% prod)** — `civitai-dp-prod-primary` (SSR), `civitai-dp-prod-api-primary` (API), plus the other serving pools (`-api-heavy`, `-jobs`, …). **Deploy is fully live ONLY when every app deployment is fully ROLLED onto the new image — `updatedReplicas == readyReplicas == desired` — NOT when the deploy spec image flips.** Flagger promotes the spec image minutes before the pods finish their rolling update, so a procedure added in the new image is NOT_FOUND on the not-yet-rolled pods. **Never tell anyone a change is live off the spec image / `status`'s image line alone — wait for the "FULLY ON PROD — every app deployment rolled" summary (or `watch` exit 0).** The API primary fully rolled is what makes tRPC procedures live; the jobs pool fully rolled is what makes cron changes live.

Then an overall **SUMMARY** line: where in the chain + ETA (building / awaiting Flux pickup / canary progressing / rolled back / fully on prod).

Canary progression: `Initialized -> Progressing` (10->50% in 2-min steps) `-> Promoting -> Finalising -> Succeeded`. At rest between deploys a canary sits at `Succeeded weight=0`. End-to-end ~35-40 min (build 15-25).

Manual one-liners (same queries the script