Building pexes in docker on macs sometimes breaks

[gcbirzan-plutoflume]

Describe the bug

Trying to build pexes for Linux on a mac via a docker container will sometimes fail. This only affects mac->linux builds, because of the weirdness of docker on mac.

In 2.33.0, pants started using python 3.14. A change introduced there causes pex's multiprocessing use to try to create a UNIX socket in the bind mount (since the temp dir is inside the sandbox, which is shared with the host OS). This fails with

[Errno 22] Invalid argument: '/pants-sandbox/pants-sandbox-uOpBP2/.tmp/pymp-j_z_u9v6/sock-d3zx1djz'

This is because UNIX sockets on bind mounts are not supported. @jsirois released a new pex version that lets you skip using mulitprocessing when using --jobs=1, but that's not really a correct solution, just a workaround.

I think correct solution is setting the TMPDIR to be in the filesystem of the docker container, even though... it's a sandbox. But, as long as it works...

The reason it sometimes works is that pex sometimes picks not to use multiprocessing.

Pants version
2.33.0

OS
macOS only as it's because of the interaction between the docker VM and the host OS.

Additional info
Add any other information about the problem here, such as attachments or links to gists, if relevant.

[jsirois]

This is because UNIX sockets on bind mounts are not supported.

IIUC, this is not true. It's just that certain operations on such special files are not allowed - namely chmod in this case. If the chmod were done on the host and there was proper uid/gid alignment with the container, then I believe all is well. That is not an option in this case since Python in the container is the one creating + chmod'ing the unix socket in the bind mount (i.e.: the socket is anonymous to all but the Python process in the container).

Back to the point though; the current setup (6 years old):

pants/src/python/pants/backend/python/util_rules/pex_cli.py

Line 155 in 8eb7d2f

tmpdir = ".tmp"

Relevant background on current choice:

pants/src/python/pants/backend/python/util_rules/pex_cli.py

Lines 170 to 182 in 8eb7d2f

	global_args = [
	# Ensure Pex and its subprocesses create temporary files in the process execution
	# sandbox. It may make sense to do this generally for Processes, but in the short term we
	# have known use cases where /tmp is too small to hold large wheel downloads Pex is asked to
	# perform. Making the TMPDIR local to the sandbox allows control via
	# --local-execution-root-dir for the local case and should work well with remote cases where
	# a remoting implementation has to allow for processes producing large binaries in a
	# sandbox to support reasonable workloads. Communicating TMPDIR via --tmpdir instead of via
	# environment variable allows Pex to absolutize the path ensuring subprocesses that change
	# CWD can find the TMPDIR.
	"--tmpdir",
	tmpdir,
	]

Whatever the choice for TMPDIR, it must work both for local execution and under remoting. The current sandbox relative path satisfies that constraint as well as the size constraint by implication as explained in the large comment.

So it appears there is a push-pull here - "remoting" on host mac to local Linux via docker wants TMPDIR not in the sandbox whereas all other use cases want this.

As further color, IIUC, the only reason the sandboxes are volume mounted in to the container is as a hack to allow the Pants running on the host to populate the sandbox dir from the host side. IOW, Pants or some narrower LMDB-aware process need not be run in the container to prepare a sandbox.