/triage-leaked-infra

name: triage-leaked-infra
description: "Assess whether an AWS VPC or infra set from HyperShift CI is safe to delete. Use when the user pastes cleanleaked output and asks 'can I delete this?', 'is this safe to remove?', 'triage this infra', asks about a LEAKED or UNCERTAIN verdict, provides a VPC ID or infraID and wants to know if it's orphaned, or says 'check this VPC'. Also use when the user asks 'should I delete this?' about any AWS resource in the HyperShift CI account."

Triage Leaked Infrastructure

Assess whether an AWS infrastructure set (VPC + associated resources) from HyperShift CI is safe to delete. Every claim must be backed by an empirical AWS query — never assume, always verify.

Input

The user provides one of:

• cleanleaked tool output (an infra set block)
• A VPC ID (e.g., `vpc-0abc123...`)
• An infraID (e.g., `00ab3695c5f73d4354b9` or `node-pool-78vcg`)

Extract the **infraID** and **VPC ID** from the input. If only one is given, derive the other:

• VPC ID → `aws ec2 describe-vpcs --vpc-ids <VPC> --query 'Vpcs[0].Tags'` → extract infraID from `kubernetes.io/cluster/<infraID>` tag or `Name` tag (strip `-vpc` suffix)
• infraID → `aws ec2 describe-vpcs --filters "Name=tag:Name,Values=<infraID>-vpc"` → get VPC ID

Checks

Run these in order. For each, report **PASS** (safe signal), **FAIL** (do NOT delete), or **UNKNOWN** (could not determine). Use `--region us-east-1` for all commands.

1. Protection tags

aws ec2 describe-vpcs --vpc-ids <VPC> --query 'Vpcs[0].Tags' --output json

• Check for `hypershift.openshift.io/do-not-delete=true` → if present: **FAIL**
• Check for `hypershift.openshift.io/ci-cluster` → if present: **FAIL** (this is a management cluster)

2. Protected VPC name

From the `Name` tag: if it is `hypershift-ci-2-vpc`, `hypershift-ci-3-vpc`, or `hypershift-ci-metrics-vpc` → **FAIL**

3. Protected user

Check if the infraID contains any of these usernames: `aabdelre`, `agarcial`, `ahmed`, `alamela`, `alesross`, `bclement`, `brcox`, `celebdor`, `cewong`, `dario`, `dari2o`, `dmace`, `glipceanu`, `jiezhao`, `jparrill`, `mbhalodi`, `mbrown`, `meha`, `mgencur`, `mulham`, `mraee`, `rkshirsa`, `sdminonne`, `sjenning`, `tsegura`, `vismishr`

If match → **FAIL** (developer cluster, contact the owner)

4. Expiration date / Age

From the VPC tags, check `expirationDate`:

• If present and **not yet expired** → **FAIL**
• If present and **expired** → **PASS**
• If absent → check resource age instead (the `expirationDate` tag only exists on resources created in the last ~2 weeks)

When `expirationDate` is absent, determine age from the earliest timestamped sub-resource:

aws ec2 describe-vpc-endpoints --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'VpcEndpoints[*].CreationTimestamp' --output text
aws ec2 describe-network-interfaces --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'NetworkInterfaces[*].Attachment.AttachTime' --output text

Take the earliest timestamp found. If the resource is **older than 24 hours** AND the CI pattern check (step 5) is **PASS** → **PASS**. If younger than 24 hours → **FAIL**. If no timestamp found at all → **UNKNOWN**.

5. CI pattern match

Does the infraID match a known CI test pattern?

• **Hex ID** (20+ lowercase hex chars, e.g., `00ab3695c5f73d4354b9`) → CI generic e2e → **PASS**
• **Named test**: `create-cluster-*`, `node-pool-*`, `control-plane-upgrade-*`, `autoscaling-*`, `karpenter-*`, `karpenter-upgrade-control-plane-*`, `scale-from-zero-*`, `kms-verify-*`, `request-serving-*`, `private-*`, `proxy-*`, `spot-demo-*`, `ha-break-glass-creds-*`, `custom-config-*`, `ho-upgrade-*`, `multi-hop-upgrade-*` → **PASS**
• **None of the above** (e.g., `hc1-*`, `clust-*`, `dev-*`, `test-dev-*`) → **UNKNOWN** (might be a developer cluster)

6. OIDC S3 liveness

aws s3api head-object --bucket hypershift-ci-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1
aws s3api head-object --bucket hypershift-ci-2-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1
aws s3api head-object --bucket hypershift-ci-3-oidc --key "<infraID>/.well-known/openid-configuration" --region us-east-1

If any returns 200 → **FAIL** (cluster still active). If all return 404/error → **PASS**.

7. EC2 instances

aws ec2 describe-instances --region us-east-1 \
  --filters "Name=tag:kubernetes.io/cluster/<infraID>,Values=owned" \
           "Name=instance-state-name,Values=pending,running,stopping,stopped" \
  --query 'Reservations[*].Instances[*].[InstanceId,State.Name,Tags[?Key==`Name`].Value|[0]]' --output text

If any instances returned → **FAIL**. If none → **PASS**.

8. red-hat-managed check

aws ec2 describe-instances --region us-east-1 \
  --filters "Name=vpc-id,Values=<VPC>" "Name=tag:red-hat-managed,Values=true" \
  --query 'Reservations[*].Instances[*].InstanceId' --output text

If any instances returned → **FAIL** (ROSA managed infrastructure). If none → **PASS**.

9. Sub-resources inventory (VPC-scoped)

Query each and report **ID + Name** for every resource, not just counts:

aws elbv2 describe-load-balancers --region us-east-1 --query "LoadBalancers[?VpcId=='<VPC>'].[LoadBalancerName,Type,Scheme,DNSName]" --output text
aws ec2 describe-vpc-endpoints --region us-east-1 --filters "Name=vpc-id,Values=<VPC>" --query 'VpcEndpoints[*].[VpcEndpointId,VpcEndpointType,ServiceName,State]' --output text
aws ec2 describe-vpc-endpoint-service-configurations --region us-east-1 --output json | # filter by kubernetes.io/cluster/<infraID> tag
aws ec2 describe-nat-gateways --region us-east-1 --filter "Name=vpc-id,Values=<VPC>" --query 'NatGateways[*].[NatGatewayId,State,Tags[?Key==`Name`].Value|[0],NatGatewayAddresses[0].PublicIp]' --output text
aws ec2 describe-internet-gateways --region us-east-1 --filters "Name=attachment.vpc-id,Values=<VPC>" --query 'InternetGateways[*].[InternetGatewayId,Tags[?Key=