Retry the macOS integration tests only when the VM's resolver times out

[JamBalaya56562]

Why

Docker runs inside a QEMU-backed Lima VM on the macOS runners, and name resolution in that VM times out often enough to take down reactor runs that have nothing wrong with them. Most recently on this job:

#2 [internal] load metadata for registry.access.redhat.com/ubi8/ubi-minimal:8.6
#2 ERROR: failed to do request:
   Head "https://registry.access.redhat.com/v2/ubi8/ubi-minimal/manifests/8.6":
   dial tcp: lookup registry.access.redhat.com: i/o timeout

It is not about one registry. The same signature has hit Docker Hub, and there the error names the resolver it gave up on:

lookup registry-1.docker.io on 127.0.0.53:53:
read udp 127.0.0.1:33196->127.0.0.53:53: i/o timeout

127.0.0.53 is the VM's own systemd-resolved, so what fails is the client side, inside the VM — not the registries, which resolve and serve manifests fine from a healthy network.

Why not just retry the step

A full reactor run on these runners takes 29 to 34 minutes (measured on two green macOS jobs). Retrying the step outright means every flake costs another half hour, and a genuine failure pays that too before failing a second time.

So the retry is narrowed twice over:

1. Only for that signature. Anything else fails immediately, with an annotation saying why. The two real failures found on macOS this month — a Postgres healthcheck timeout and a 409 during container removal — would not be retried.

2. Only from where it broke. Maven prints the module to resume from, and the step reuses it. The run above failed on module 14 of 36 after 17 minutes; resuming re-runs 23 modules instead of 36.

It is a single attempt, and it always leaves a ::warning:: behind, so a resolver that has genuinely stopped working still turns the job red, and how often this fires stays visible in the logs.

Verification

The step's script was extracted from the YAML and run against a stubbed mvn, so what was tested is what the workflow contains.

scenario	exit	mvn invocations	outcome
build succeeds	0	1	no retry
resolver timeout, second attempt succeeds	0	2	resumed with -rf :dmp-it-buildx-push
resolver timeout on both attempts	1	2	retried once, still red
real failure (healthcheck timeout)	1	1	not retried, ::error::Integration tests failed for a reason other than name resolution
resolver timeout with no -rf hint	1	2	falls back to a full re-run

The log is written to ${RUNNER_TEMP} rather than into it/, so nothing new appears in the workspace during the build — confirmed in the same runs.

Not covered by this

This does not make the resolver reliable, it only stops one timeout from discarding 17 minutes of work. Two plugin-side levers were measured first and neither addresses it:

• <buildx><configFile> with a [dns] section configures the containers BuildKit runs for RUN steps, not buildkitd's own registry lookups. Pointed at an unroutable resolver (203.0.113.1), load metadata still completed in 2.7s while the RUN step reported nameserver 203.0.113.1.
• <buildx><driverOpts><network>host</network> puts buildkitd on the VM's resolver at 127.0.0.53 — the one that timed out above.

Independent of #1968, which covers the Windows daemon readiness and pre-pulls base images for the classic build path. If that lands first this needs a trivial rebase, since it moves DOCKER_HOST to the job's env.

[rohanKanojia]

This may not be accurate because the initial invocation includes clean, which removes the reactor's build outputs. Could we clarify what Maven actually avoids rebuilding when -rf is used after a clean install failure

[JamBalaya56562]

You are right, and the comment was wrong. clean does run for every module in
the resumed reactor, so those are rebuilt from scratch.

What resuming actually skips is the modules before the resume point: they are
left out of the reactor altogether, so no phase runs for them and their images
and containers are not built a second time. I checked it on a throwaway
three-module reactor — after mvn clean install -rf :b, a marker file left in
a/target survived while the ones in b/target and c/target were removed.

That is still where the saving is: the run that prompted this failed on module
14 of 36 after 17 minutes, so resuming re-runs 23 modules instead of 36, against
29 to 34 minutes for a full reactor. Their artifacts are already in ~/.m2 from
the first attempt, so the resumed build resolves against them.

Reworded in the follow-up commit.