"ACC: ERROR non contiguous transfer" from internal/ltinv_mod.F90:366

[okkevaneck]

What happened?

Running ectrans on GPU with default arguments works fine.
But as soon as we scale up, we run into problems regarding memory transfers with the following error:

ACC: ERROR non contiguous transfer from ../../../pfs/lustrep2/scratch/project_462000713/ectrans-main-acc/src/ectrans-HEAD^1.6.0.src/build/src/trans/gpu/generated/ectrans_gpu_dp/internal/ltinv_mod.F90:366

This was the error message for double precision, but we observe the same behavior for single precision:

ACC: ERROR non contiguous transfer from ../../../pfs/lustrep2/scratch/project_462000713/ectrans-main-acc/src/ectrans-HEAD^1.6.0.src/build/src/trans/gpu/generated/ectrans_gpu_sp/internal/ltinv_mod.F90:366

This line corresponds to the following call to PRFI1B:

364         IF(PRESENT(PSPSC3A) .AND. NF_SC3A > 0) THEN
365           DO J3=1,UBOUND(PSPSC3A,3)
366             CALL PRFI1B(PSCALARS(IFIRST:IFIRST+2*NF_SC3A-1,:,:),PSPSC3A(:,:,J3),NF_SC3A,UBOUND(PSPSC3A,2))
367             IFIRST  = IFIRST+2*NF_SC3A
368           ENDDO
369         ENDIF

What are the steps to reproduce the bug?

Environment

We are running on LUMI dev-g using 1 node, and load the following modules before the installation and execution:

module load LUMI/24.03 partition/G PrgEnv-cray
module load cpe/24.03 craype-x86-trento craype-accel-amd-gfx90a
module load cray-mpich cray-libsci cray-fftw cray-python
module load buildtools
module load rocm/6.0.3
module load cce/17.0.1

These are different from the ones we found in the GitHub actions, so we also tried those, but unfortunately, both led to the same error.
The ones from GitHub actions for reference:

module load CrayEnv
module load PrgEnv-cray
module load cce/17.0.1
module load craype-accel-amd-gfx90a
module load rocm/6.0.3
module load cray-fftw
module load buildtools

We also set the following environment variables:

export FC=ftn
export FC90=ftn
export CXX=CC
export CC=cc

Installation

The subsections below contain the installation instructions for each module.
Note that we set environment variables for linking OpenMP, which are isolated per module.

Ecbuild

Version: 3.8.5
Installation through git pull, followed by:

mkdir -p build
cd build
cmake .. 
make clean
make -j16
make install

Fiat

Version: 1.4.1
Installation through git pull followed by:

export LDFLAGS="-fopenmp"
mkdir -p build
cd build
$DNB_INSTALL_DIR/ecbuild.bin/bin/ecbuild \
    -DCMAKE_BUILD_TYPE=Release \
    -DENABLE_MPI=ON \
    -DENABLE_OMP=ON \
    -DENABLE_TESTS=OFF \
    .. 
make clean
make -j16
make install

Ectrans

Version: 1.6.1
Installation through git pull followed by:

export LDFLAGS="-fopenmp -lcraymp"
mkdir -p build
cd build
$DNB_INSTALL_DIR/ecbuild.bin/bin/ecbuild \
    -DCMAKE_BUILD_TYPE=Release \
    -Dfiat_ROOT=$DNB_INSTALL_DIR/fiat.bin \
    -DENABLE_TESTS=OFF \
    -DENABLE_SINGLE_PRECISION=ON \
    -DENABLE_SINGLE_PRECISION=ON \
    -DENABLE_MKL=OFF \
    -DENABLE_FFTW=OFF \
    -DENABLE_OMP=OFF \
    -DENABLE_ACC=ON \
    -DENABLE_ACC=ON \
    -DENABLE_GPU_AWARE_MPI=ON \
    .. 
make clean
make -j16
make install

Execution

These are examples of our executions using double precision, but it works the same for single precision.
All executions used 1 node, and the environment described above is loaded beforehand.

When we run with default arguments, the following line is executed:

numactl -l --all --physcpubind=49-55 -- ./bin/ectrans-benchmark-gpu-dp --norms --nlev 137 --vordiv --scders -t 600 --niter 20

The numactl is required on LUMI for optimal bindings.
For our "scaled-up" executions, we run the following:

numactl -l --all --physcpubind=49-55 -- ./bin/ectrans-benchmark-gpu-dp --vordiv --scders --uvders --nfld 1 --norms --niter 10 --nlev 79 --truncation 1279

Version

v1.6.0

Platform (OS and architecture)

LUMI, Cray-OS, running kernel 5.14.21-150500.55.49_13.0.56-cray_shasta_c

Relevant log output

ACC: ERROR non contiguous transfer from ../../../pfs/lustrep2/scratch/project_462000713/ectrans-main-acc/src/ectrans-HEAD^1.6.0.src/build/src/trans/gpu/generated/ectrans_gpu_dp/internal/ltinv_mod.F90:366

Accompanying data

No response

Organisation

Barcelona Supercomputing Center

[okkevaneck]

To extend upon this bug, we just tried another configuration, which gave the same error:
./bin/ectrans-benchmark-gpu-dp --vordiv --scders --uvders --nfld 1 --norms --niter 10 --nlev 137 --truncation 399

[samhatfield]

Will take a look at this @okkevaneck - thanks for flagging it 👍🏻

By the way, you could try with OpenMP offload instead of OpenACC - -DENABLE_OMP=ON -DENABLE_ACC=OFF. It would be interesting to see how that behaves.

[okkevaneck]

Thanks @samhatfield!

I've run the OpenMP double-precision version using the default args and the ones from the issue.
The default arguments seem to work fine:

# Default args ACC
TOTAL WALLCLOCK TIME     183.748 CPU TIME     97.215 VECTOR TIME      97.215

# Default args OMP
TOTAL WALLCLOCK TIME     197.861 CPU TIME   1290.033 VECTOR TIME    1290.033

When scaling up, the program becomes quite slow, and exits with an OOM error on 1 node:

slurmstepd: error: Detected 1 oom_kill event in StepId=10693934.0. Some of the step tasks have been OOM Killed.
slurmstepd: error: *** STEP 10693934.0 ON nid005003 CANCELLED AT 2025-05-06T13:54:13 ***

This is as expected, since before using the redgreengpu branch, it didn't fit on 1 node either.
The job runtime was 00:26:13.

Afterwards, I ran it on 4 nodes, and got a successful run with the following timing:

# LUMI dev-g - scaled-up OpenMP - 4 nodes
TOTAL WALLCLOCK TIME    2083.740 CPU TIME  13445.508 VECTOR TIME   13445.508

So the job ran for 00:34:56.

I've found old results of redgreengpu on Leonardo for the same workload, but on Booster nodes.
Just for a comparison, these are the timings we acquired back then:

# Leonardo Booster - scaled-up - 4 nodes
TOTAL WALLCLOCK TIME      58.229 CPU TIME     58.229 VECTOR TIME      58.229
# Leonardo Booster - scaled-up - 8 nodes
TOTAL WALLCLOCK TIME      32.517 CPU TIME     32.517 VECTOR TIME      32.517
# Leonardo Booster - scaled-up - 16 nodes
TOTAL WALLCLOCK TIME      22.914 CPU TIME     22.914 VECTOR TIME      22.914
# Leonardo Booster - scaled-up - 32 nodes
TOTAL WALLCLOCK TIME      15.966 CPU TIME     15.965 VECTOR TIME      15.965

Note that this is with OpenACC, and not OpenMP.
Even though the porting method and HPC systems are different, I am surprised by the difference in runtime comparing the 4-node setups.

Note for the issue

Note that the scaled-up workload didn't fit on 1 node for redgreengpu, and neither does it for the new OpenMP version. So before submitting this issue, I also ran 1.6.0 with OpenACC on 4 nodes on LUMI, but unfortunately, I got the same errors as described. Hence, I've submitted a 1-node example as it's probably easier to debug.

[samhatfield]

Okay, I followed your instructions, but at runtime it failed with:

ectrans/build/bin/ectrans-benchmark-gpu-sp: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory

The LUMI software environment is extremely brittle and life is too short to debug these kinds of errors. What I can do is prepare instructions for building and running on LUMI-G "state of the art" (which will be different to your instructions above) and demonstrate (hopefully) that it works. Then you can either just adopt those instructions or use them as a reference to figure out what's going wrong with yours, how does that sound?

As for what you mention above: so you have something workable if you use OpenMP offload instead of OpenACC? That's great. Regarding wall times, I would not recommend relying on TOTAL WALLCLOCK TIME. It is easily skewed by expensive initialisation steps. It's better to look at Inverse-direct transforms - med (s) which is the median time for the inverse and direct transforms combined. Also, if your runs have a very expensive first step, that likely means that your FFT plans are being recompiled from scratch every time you run ecTrans. The preferred method is to cache the plans to disk first time and then read from that cache subsequent times. My instructions will explain how :)

[okkevaneck]

Ah, but is this with the OpenACC version for GPU offloading?
A working example would be great! We could use that to debug our installation as well 👍🏼.

Okay, good to know about the timings!
Thanks for the info on caching the FFT plans, I wasn't aware of this setup.
Where can I find the instructions?

[samhatfield]

That's right, it looks like you may have discovered a bug specifically in the OpenACC code path so that's the one I'll test. I'm a little overwhelmed with some other tasks today but I'll see if I can prepare some instructions at the end of the day or tomorrow.

[samhatfield]

Okay here are the instructions:

1. Prepare source code

git clone https://github.com/ecmwf/ecbuild -b 3.10.0
git clone https://github.com/ecmwf-ifs/fiat -b 1.5.0
git clone https://github.com/ecmwf-ifs/ectrans -b 1.6.0

2. Prepare environment files

$ cat env.sh
module load CrayEnv
module load PrgEnv-cray
module load cce/17.0.1
module load craype-accel-amd-gfx90a
module load rocm/6.0.3
module load cray-fftw
module load buildtools
$ cat toolchain.cmake
set( OpenMP_C_LIB_NAMES       "craymp" )
set( OpenMP_CXX_LIB_NAMES     "craymp" )
set( OpenMP_Fortran_LIB_NAMES "craymp" )
set( OpenMP_craymp_LIBRARY    "craymp" )

3. Build FIAT

cmake -S fiat -B fiat/build -DCMAKE_TOOLCHAIN_FILE=$PWD/toolchain.cmake
cmake --build fiat/build -j10

4. Build ecTrans

cmake -S ectrans -B ectrans/build -Dfiat_ROOT=fiat/build -DENABLE_OMP=OFF -DENABLE_GPU=ON -DENABLE_ACC=ON -DENABLE_GPU_AWARE_MPI=ON -DCMAKE_TOOLCHAIN_FILE=$PWD/toolchain.cmake
cmake --build ectrans/build -j10

5. Prepare run script

$ cat select_gpu
#!/bin/bash
export ROCR_VISIBLE_DEVICES=$SLURM_LOCALID
exec $*
$ chmod +x select_gpu
$ cat ectrans_execute.sbatch
#!/bin/bash
#SBATCH --partition=standard-g
#SBATCH --time=01:00:00
#SBATCH --output=ectrans.out
#SBATCH --error=ectrans.err
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=7
#SBATCH --gpus-per-node=8
#SBATCH --hint=nomultithread
#SBATCH --mem=200G
#SBATCH --account=project_{your project here}

set -x
set -uo pipefail

# Set truncation
truncation=1279

# Set the build directory
BUILD_DIR=$PWD/ectrans/build

# Load the environment
source $PWD/env.sh

# Required for direct GPU-GPU communication
export MPICH_GPU_SUPPORT_ENABLED=1

# Cache rocFFT plans
rocfft_cache_dir=$PWD/rocfft_cache/rocFFT_${CRAY_ROCM_VERSION}_${truncation}
mkdir -p $rocfft_cache_dir
export ROCFFT_RTC_CACHE_PATH=$rocfft_cache_dir/rocfft_kernel_cache.db

# Explicitly set CPU pinning
CPU_BIND1="map_cpu:49,57,17,25,1,9,33,41"

# Run the executable
srun --cpu-bind=$CPU_BIND1 \
    ./select_gpu ${BUILD_DIR}/bin/ectrans-benchmark-gpu-dp \
    --norms --truncation $truncation --niter 100 --nlev 137 --nfld 1 --vordiv --uvders --scders --meminfo

5. Submit

sbatch ectrans_execute.sbatch

The first time will take ages, like 40 minutes. The second time should only take about 6 minutes, since the FFT plans will have been cached from the first run.

You might want to set a lower truncation to start just to test it's working, like 159.

Have a go and let me know if you have any issues :)

[okkevaneck]

Thank you for the instructions @samhatfield!
I've integrated them into our workflow and tried out the caching.

For runs with default arguments, it seems to work flawlessly, but when I scale up to --truncation 1279, I have many errors regarding the DB, for example:

Error: failed to store code object for bluestein_single_fwd_len1178_dim1_dp_op_R_CI: database is locked

The out file still contains the timings, so I guess that everything is right.
Moreover, the initial run took around 46 minutes, and a 2nd one around 2 minutes, so it does speed up things significantly.
For this second run I didn't get the error files, so I'm curious if they are really problematic.
It seems like they are related to the initial creation of the db files.

I've added the out and err files of rank 0 of the initial run for you to take a look at in case you're interested.
err.10748196.0.txt
out.10748196.0.txt

[samhatfield]

Yes I'm really not sure what's happening there. My guess is each rank is trying to write their plans to the same file, but each write takes long enough that for any one rank trying to write, another already has the file lock. As you've seen, it doesn't seem to have any consequences for the subsequent runs.

In the long term I'm not sure what we should do with this caching thing when running on AMD platforms with rocFFT. cuFFT has no such issue as far as I'm aware - it's fast first time round. Probably we will need to keep a universal persistent cache somewhere safe.

Anyway, looks like everything's working for you, but keep me posted if you spot anything weird.

[okkevaneck]

Curious how cuFFT doesn't have similar problems, maybe it's something we can address with AMD.
For now, this workaround is good enough for our project, thanks!

Let me know if you find a solution to the OpenACC version!

[reuterbal]

Note that you can force the use of an in-memory cache for rocFFT: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#things-to-remember-when-programming-for-amd-gpus

export ROCFFT_RTC_CACHE_PATH=/dev/null

Later versions also allow providing a read-only system-installed cache, but I think that is not yet available in the ROCm versions on LUMI:
https://rocm.docs.amd.com/projects/rocFFT/en/latest/design/runtime_compilation.html#cache-location

[samhatfield]

Curious how cuFFT doesn't have similar problems, maybe it's something we can address with AMD. For now, this workaround is good enough for our project, thanks!

Let me know if you find a solution to the OpenACC version!

The instructions above build ecTrans with OpenACC so as long as you followed that you'll already be using OpenACC!

[okkevaneck]

Thanks @reuterbal for the extra info!

@samhatfield : Ah I applied it to OpenMP, my bad! But for OpenACC we would still have the core problem of this issue when scaling up --truncation right? I don't see how caching the plans would change that.

[samhatfield]

@samhatfield : Ah I applied it to OpenMP, my bad! But for OpenACC we would still have the core problem of this issue when scaling up --truncation right? I don't see how caching the plans would change that.

I would say, try it out anyway. I'm not sure what caused the error you originally reported. There are a few other things different in my instructions other than the plan caching. E.g. exporting MPICH_GPU_SUPPORT_ENABLED=1 is essential when GPU_AWARE_MPI=ON, so if you weren't doing that before then that may be partially responsible. Have a play around with the OpenACC version following my instructions and see if you can find the error?