Runtime NaN and segfault with Kokkos::Profiling::ScopedRegion region()

[mihelog]

Hello everyone,

I have a cuda and Kokkos code I'm trying to profile using region names. I have compiled kokkos-tools and generated "libkp_nvtx_connector.so". Then in the source code, I add where applicable:

#include <Kokkos_Profiling_ScopedRegion.hpp>

and in various code scopes I give them names by at the beginning of the scope adding this (with various names, here is just one example):

Kokkos::Profiling::ScopedRegion region("HorizInterpRemapperBase::HorizInterpRemapperBase");

Then in the sbatch (slurm) script, I invoke it by:

export KOKKOS_TOOLS_LIBS=/global/homes/m/mihelog/ThrustE/mihelog/kokkos-tools/profiling/nvtx-connector/kp_nvtx_connector.so
export KOKKOS_PROFILE_LIBRARY=/global/homes/m/mihelog/ThrustE/mihelog/kokkos-tools/profiling/nvtx-connector/kp_nvtx_connector.so
srun  --label  -n 4 -N 1 -c 32  --cpu_bind=cores   -m plane=4 <executable> <application command-line arguments>

The problem is that I get runtime errors shown at the end of this post. I've isolated the issue to these calls because when I comment them out, the application completes successfully.

Any thoughts would be great. Thanks in advance!

 number of MPI processes per node: min,max=           4           4
0:
2:  var,nvars:           1           1
2:            3  **ABORTING WITH ERROR: NaNs detected in repro sum input**
0: Note: nsplit=-1, while nsplit must be >=1. We know SCREAM does not know nsplit until runtime, so this is fine.
0:       Make sure nsplit is set to a valid value before calling prim_advance_subcycle!
0: gfr> nelemd 1350 qsize 10
0: compose> nelemd 1350 qsize 10 hv_q 1 hv_subcycle_q 6 lim 9 independent_time_steps 1
0:  var,nvars:           1           1
1:  var,nvars:           1           1
1:            2  **ABORTING WITH ERROR: NaNs detected in repro sum input**
2: MPICH ERROR [Rank 2] [job id 37885683.0] [Sat Apr 19 09:02:20 2025] [nid001413] - Abort(128) (rank 2 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 128) - process 2
2:
2: aborting job:
2: application called MPI_Abort(MPI_COMM_WORLD, 128) - process 2
3:  var,nvars:           1           1
3:            4  **ABORTING WITH ERROR: NaNs detected in repro sum input**
0:            1  **ABORTING WITH ERROR: NaNs detected in repro sum input**
1: MPICH ERROR [Rank 1] [job id 37885683.0] [Sat Apr 19 09:02:20 2025] [nid001413] - Abort(128) (rank 1 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 128) - process 1
1:
0: MPICH ERROR [Rank 0] [job id 37885683.0] [Sat Apr 19 09:02:20 2025] [nid001413] - Abort(128) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 128) - process 0
1: aborting job:
1: application called MPI_Abort(MPI_COMM_WORLD, 128) - process 1
0:
0: aborting job:
0: application called MPI_Abort(MPI_COMM_WORLD, 128) - process 0
3: MPICH ERROR [Rank 3] [job id 37885683.0] [Sat Apr 19 09:02:20 2025] [nid001413] - Abort(128) (rank 3 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 128) - process 3
3:
3: aborting job:
3: application called MPI_Abort(MPI_COMM_WORLD, 128) - process 3
1: Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
1:
1: Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
1:
1: Backtrace for this error:
2: Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize()
2:
2: Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

[dalg24]

Typically, issues with regions are unbalanced push/pop but you are using the RAII wrapper so I suppose I'd check that all your scoped regions are out of scope by the time Kokkos is finalized.

[mihelog]

That's a good point. The code is huge and complicated so I'm not sure how to check that, but it appears that this error happens early in the execution based on stdout prints. So it's unlikely that kokkos is called to finalize this early... At least I think. Is there a way to check when kokkos is called to finalize?

[dalg24]

Please confirm that the code execute properly with the scoped regions but without loading the nvtx connector tool.

If it does then build a modified version of the tool by applying

diff --git a/profiling/nvtx-connector/kp_nvtx_connector.cpp b/profiling/nvtx-connector/kp_nvtx_connector.cpp
index 6c89635..9960671 100644
--- a/profiling/nvtx-connector/kp_nvtx_connector.cpp
+++ b/profiling/nvtx-connector/kp_nvtx_connector.cpp
@@ -87,11 +87,9 @@ void kokkosp_begin_parallel_reduce(const char* name, const uint32_t /*devID*/,

 void kokkosp_end_parallel_reduce(const uint64_t /*kID*/) { nvtxRangePop(); }

-void kokkosp_push_profile_region(const char* regionName) {
-  nvtxRangePush(regionName);
-}
+void kokkosp_push_profile_region(const char* regionName) {}

-void kokkosp_pop_profile_region() { nvtxRangePop(); }
+void kokkosp_pop_profile_region() {}

 // TODO: move this to kp_core?
 namespace {

and run with that version of the tool loaded to confirm the issue is with the regions.

[mihelog]

It does not. By that I mean that I do not set the environment variables I set above and do not get a printout from the connector tool. The error and seg fault are the same. I'm not sure if this helps or I should still try what you suggest?

[dalg24]

Making sure I understand. You are saying that when you add Kokkos regions to your code, you start seeing errors at runtime without even loading a tool. Is that correct?

That sounds very weird. Kokkos::ScopedRegion is just a RAII wrapper that pushes and pops the region in the constructor and destructor respectively. If no tool is loaded, then the push/pop are essentially a no-op.

[mihelog]

I see. Ok let me re-do this and get back to you. It's possible I misconfigured something. Thank you.

[mihelog]

Here's what I did:

I re-introduced the region calls and their corresponding includes. It compiles well.

I do NOT load the environment variables nor do I get a printout at stdout from the ntvx connector.

I still get the error and seg fault I mentioned above.

So I wonder if there is something else going on here. Such as, inserting the calls messes with something else in the application.

[dalg24]

You could try UBSan.