GPU-AMG Libraries Evaluation

This project evaluates several libraries implementing an AMG-preconditioned conjugate gradient solver on the GPU for solving linear systems derived from the TRUST platform CFD code (TRUST platform).

Libraries Evaluated

• Ginkgo
• AMGX
• amgcl
• Hypre

1. Clone the Project and Submodules

git clone https://github.com/rbourgeois33/GPU-AMG-libraries-evaluation
cd GPU-AMG-libraries-evaluation
git submodule update --init --recursive

2. Load Required Modules

NVIDIA A5000 (Petra)

module load cuda/12.1.0 \
            cmake/3.18.4 \
            openmpi/gcc_11.2.0/4.1.4

NVIDIA Ada 6000 (Ada)

module load cuda/12.4.0 openmpi/gcc_13.3.0/

NVIDIA A100 (Jean Zay)

module load cuda/12.4.1 \
            openmpi/4.1.5 \
            cmake/3.18.0

AMD Radeon PRO W7900

module load rocm/6.2.0 \
            openmpi/gcc_13.3.0/ \
            cmake/3.21.1

3. Build All Libraries

Ginkgo

cd lib/ginkgo
mkdir build && cd build
mkdir install

CMake Configuration

• On Jean Zay, Petra & Ada

cmake -DGINKGO_BUILD_EXAMPLES=ON \
      -DGINKGO_BUILD_OMP=ON \
      -DGINKGO_BUILD_CUDA=ON \
      -DGINKGO_CUDA_ARCHITECTURES=Ampere \
      -DCMAKE_INSTALL_PREFIX=install/ ..

• On AMD GPU Machine

cmake -DGINKGO_BUILD_EXAMPLES=ON \
      -DGINKGO_BUILD_OMP=ON \
      -DGINKGO_BUILD_HIP=ON \
      -DCMAKE_INSTALL_PREFIX=install/ ..

Build and Install

• On Petra & AMD GPU Machine

make -j 48 install

• On Jean Zay

srun -p compil -A pri@v100 -t 00:30:00 -c 24 --hint=nomultithread make -j24 install

AMGX

cd lib/AMGX
mkdir build && cd build
cmake ..
make -j 16

Hypre

cd lib/hypre
mkdir build && cd build
mkdir install
cmake -DHYPRE_ENABLE_OPENMP=ON -DHYPRE_ENABLE_CUDA=ON -DCMAKE_INSTALL_PREFIX=install/ ../src/
make -j12 install

4. Build the Main Program

mkdir build
cd build
cmake ..
make -j 12

5. Run the Programs

Ginkgo Evaluation

./ginkgo_eval _backend_

where _backend_ can be:

• cuda
• omp
• reference
• hip

AMGX Evaluation

Generate the AMGX Matrix Market formatted input file (AMGX requires A and b in the same file):

cd data/
./AMGX_formatter.sh A.mtx rhs.mtx

Run AMGX solver:

./AMGX_eval ../config_AMGX/file.json

Optional: Add write_w to the command to write the output of the resolution in .mtx format.

AMGCL CUDA Evaluation

./amgcl_cuda_eval ../data/aij_2592000.mtx ../data/rhs_2592000.mtx

Hypre Evaluation

Run the Hypre solver with appropriate input files and configuration.