Skip to content

CI: restrict UCX transports to fix MPI init aborts on MANA runners - #80

Merged
mrogowski merged 1 commit into
masterfrom
fix-ci-ucx-mana
Aug 27, 2026
Merged

CI: restrict UCX transports to fix MPI init aborts on MANA runners#80
mrogowski merged 1 commit into
masterfrom
fix-ci-ucx-mana

Conversation

@mrogowski

Copy link
Copy Markdown
Contributor

The weekly native workflow has been failing intermittently since late July (e.g. runs 30190900147, 30735658852, 32617182070), with a random subset of matrix jobs dying at "Test - 1 process" while the same commit passes on other
weeks. The failed jobs abort inside MPI_Init_thread, before running any test:

    UCX  ERROR mana_0: iface ... failed to create UD QP ... failed: Operation not supported
    UCX  ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
    Abort(943843343) on node 0: Fatal error in internal_Init_thread: ...
    MPIDI_UCX_init_world(260).....:
    init_worker(38)...............: ucx function returned with failed status

Some Azure runner VMs expose the MANA NIC as an ibverbs device; UCX probes it and tries to open a ud_verbs transport, but MANA does not support UD QPs, so UCX init fails and the apt MPICH (ch4:ucx) aborts. Whether a job fails depends only on which VM it lands on, which explains the random failing Python versions.

Fix: set UCX_TLS=tcp,sm,self at the job level, matching what mpi4py already uses in .ci/env.mpich. These tests run on a single host, so TCP, shared memory, and self transports are all that is needed.

@mrogowski mrogowski self-assigned this Aug 27, 2026
@mrogowski
mrogowski merged commit d306db9 into master Aug 27, 2026
19 checks passed
@mrogowski
mrogowski deleted the fix-ci-ucx-mana branch August 27, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant