prombench malformed PR runs after Exaustion errors (or other start errors)

[bwplotka]

We can't run more than 3 benchmarks. When this happens we have exhaustion error for some node pools.

This gets prometheus_start to never finish - it waits for node pool

prometheus_stop then never finish too

waiting for nodepools to be deleted
infra gke nodes check-deleted -a *** \
	-v ZONE:europe-west3-a -v GKE_PROJECT_ID:macro-mile-203600 \
	-v EKS_WORKER_ROLE_ARN: -v EKS_CLUSTER_ROLE_ARN: \
	-v EKS_SUBNET_IDS: -v SEPARATOR: \
	-v CLUSTER_NAME:test-infra -v PR_NUMBER:18000 \
	-f ./manifests/prombench/nodes_gke.yaml
11:35:06 gke.go:517: nodepool running name: prometheus-18000
make: *** [Makefile:120: all_nodes_deleted] Error 1

No timeouts on either of those jobs (I see it's running for 2h just fine) https://github.com/prometheus/prometheus/actions/runs/21665627212/job/62460234151

We need to make it robust so:

• We have some eventual timeouts
• Running cancel WHEN start is still running cancels starts
• restart == start ideally
• We don't need to manually remove prometheus-xyz node pool

[bwplotka]

NOTE: This always happens when you are running 4th instance of prombench (easy to check by looking on dashboard prNumber variable count http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1 as we have room for 3x prombench runs max at the moment.

To unblock yourself (maintainers):

1. Log in @prometheus.io account
2. Go to test-infra cluster in GCP GKE console.cloud.google.com "Kubernetes Engine > Clusters"
3. You should notice unhealthy nodepools

For example here you see nodes-18111 and prometheus-18111. Pair means it worked. If you see only one, it might mean deploy is stuck or pending. In this example prometheus-18272 is stuck.

4. Remove stuck nodepools via UI. In the above example I need to kill prometheus-18272

you can rerun only when you see deletion completed in the UI (otherwise this can stuck again).

Help wanted to fix our deploy script to clean for us and handle this gracefully. (something for us @SuperQ to hack on)