Segmentation fault when running with multi-threads

[WernherwW]

Hello, all. When running my code (based on STRUMPACK solver) on CPUs using more than one threads, the code crashes once the matrix to solve is slightly heavy (~GB memory consumption). The error log only gives
Segmentation fault
or
Segmentation error - invalid memory reference
The output log of my code ends with

# multifrontal factorization:
#   - estimated memory usage (exact solver) = 6483.55 MB
#   - minimum pivot, sqrt(eps)*|A|_1 = 9.91058e-07
#   - replacing of small pivots is not enabled

I tried to recompile STRUMPACK with cmake or easyBuild but it didn't work. If I use OMP_NUM_THREADS=1, the code works well until the work size exceeds the capability of one node (then the code ends with out of memory as expected).

This error is machine dependent since it only occurs after I moved to a new HPC: LUMI-C (128 cores per node, 2GB memory per core). The memory I use in multi-thread runs should be much more than enough compared to what I used on previous machines. I would accuse it on the multi-thread adaption problem of STRUMPACK on LUMI-C, but I failed to solve this issue with the LUNI-C's helpdesk. I would appreciate it a lot if I get any clues here.

For more information, similar Segmentation fault also happens when I run the test_BLR_mpi in STRUMPACK examples. Once the
matrix N reaches 100000, the test crashes with segmentation fault. More specifically, I tested OMP_NUM_THREADS=128 (2 nodes * 1 task per node * 128 cores per task) and OMP_NUM_THREADS=1 (2 nodes * 128 tasks per node * 1 core per task). Both work under N=60000, while under N=100000, OMP_NUM_THREADS=128 case crashes with Segmentation fault while OMP_NUM_THREADS=1 case crashes with out of memory.

Many thanks in advance.

Best regards,

Chizhou

[pghysels]

I'm running some tests, but I suspect it is also just running out of memory.
Using more OpenMP threads will use more memory. For N=100000, it would need at least (for double), 80GB. But in practice, there is overhead, which increases with the number of OMP threads. In test_BLR_mpi, there is also a redistribution from scalapack layout to some other distribution, which creates a copy.
You can try to monitor the memory usage, with for instance watch -n .1 free -h, or some other tool like top, htop, bashtop.

[pghysels]

It could also be that there is a 32bit integer overflow somewhere for these larger sizes.
In test_BLR_mpi, does it crash before printing # from_ScaLAPACK done! ?

[pghysels]

I pushed some fixes for 32bit integer overflows: c088a06
This fixes segfaults for me in test_BLR_mpi.

You might still see an error like # ERROR: 32bit integer overflow in all_to_all_v!!.
This is harder to fix. It's because MPI uses 32bit ints for everything, including the message size.
But you should be able to get around this by running with more MPI ranks and fewer OpenMP threads, so the messages become smaller.

[WernherwW]

Hello, Dr. Ghysels,

Thank you for your reply! I looked at the memory usage: it's true that N=100000 test_BLR_mpi takes over 150 GB of memory. It works with more nodes.
Some crashed cases stop before giving any output, while others stop after printing # from_ScaLAPACK done!. The latter can be fixed by the patch you gave. I will see if this new STRUMPACK version can accelerate my code.
The error # ERROR: 32bit integer overflow in all_to_all_v!! doesn't show up in my logs.

Thanks again for your help!

Best regards,
Chizhou