OpenMP target offload

Build the Steinhardt kernel for an NVIDIA GPU. This is not the gpulite TUM path and it is not the host-only pixi environment.

The Meson option is -Dwith_openmp_offload=enabled. Configure probes a real target teams distribute parallel for and records OpenMP target offload: YES only when that link succeeds. nvc++ is tried first (-mp=gpu, then -gpu=cc80, then -gpu=cuda12.2 / -gpu=cuda11.6 so an HPC SDK that does not ship a toolkit matching the driver can still use a bundled release). clang with libomptarget is the fallback (-fopenmp -fopenmp-targets=nvptx64-nvidia-cuda).

Elja A100

GPU partitions: gpu-1xA100, gpu-2xA100, gpu-long. Account: chem-ui. Do not compile on the login node.

GPU batch shells do not have Lmod on PATH. Use EasyBuild NVHPC 23.7-CUDA-12.2.0 (the A100 driver is 12.2; OpenHPC nvc++ 22.3 ships only CUDA 11.6 and refuses -mp=gpu against that driver unless -gpu=cuda11.6 is set) and the GCCcore-13.3.0 EasyBuild prefixes for Meson, Ninja, Eigen, FlexiBLAS, and Python 3.12. Cluster Catch2 is 2.x; the tree carries a Catch2 3 wrap. CUDA 12.4 supplies nsys 2023.4.

EB=/hpcapps/lib-edda/easybuild/software
NVHPC_ROOT=$EB/NVHPC/23.7-CUDA-12.2.0/Linux_x86_64/23.7
export NVHPC_CUDA_HOME=$NVHPC_ROOT/cuda/12.2
export PATH=$EB/CUDA/12.4.0/bin:$NVHPC_ROOT/compilers/bin:$EB/Meson/1.4.0-GCCcore-13.3.0/bin:$EB/Ninja/1.12.1-GCCcore-13.3.0/bin:$EB/Python/3.12.3-GCCcore-13.3.0/bin:$PATH
export PKG_CONFIG_PATH=$EB/Eigen/3.4.0-GCCcore-13.3.0/share/pkgconfig:$EB/FlexiBLAS/3.4.4-GCC-13.3.0/lib/pkgconfig
export LD_LIBRARY_PATH=$EB/Python/3.12.3-GCCcore-13.3.0/lib:$NVHPC_ROOT/compilers/lib:$NVHPC_ROOT/cuda/12.2/lib64:$NVHPC_ROOT/math_libs/12.2/lib64:$EB/GCCcore/13.3.0/lib64:$EB/FlexiBLAS/3.4.4-GCC-13.3.0/lib:$EB/Z3/4.13.0-GCCcore-13.2.0/lib:$EB/hwloc/2.9.2-GCCcore-13.2.0/lib
export PYTHONPATH=$EB/Meson/1.4.0-GCCcore-13.3.0/lib/python3.12/site-packages
# GPU nodes lack glibc-devel. Before sbatch, from a login node:
#   cp /usr/lib64/crt1.o crti.o crtn.o Scrt1.o elja-crt/
#   rsync -a /usr/include/ elja-sysroot/usr/include/
export LIBRARY_PATH=$PWD/elja-crt:/lib64
export CFLAGS="-B$PWD/elja-crt"
export CXXFLAGS="-B$PWD/elja-crt"
export LDFLAGS="-B$PWD/elja-crt -L$PWD/elja-crt -L/lib64 -L$EB/hwloc/2.9.2-GCCcore-13.2.0/lib -lhwloc"

# nvc++ 23.7 accepts -mp=gpu against driver 12.2. Meson does not
# list C++20 for this release, so setup uses -Dcpp_std=none and
# -std=c++20 (the tree uses concepts).
export CC=$EB/GCCcore/13.3.0/bin/gcc
export CXX=nvc++
export CFLAGS="-idirafter $PWD/elja-sysroot/usr/include"
export CXXFLAGS="-std=c++20 -idirafter $PWD/elja-sysroot/usr/include"
meson setup build-offload --buildtype=debugoptimized \
  -Dcpp_std=none \
  -Dwith_openmp_offload=enabled \
  -Dwith_python=false \
  -Dwith_lua=disabled \
  -Dwith_mpi=disabled \
  -Dwith_gpulite=disabled \
  -Dcatch2:tests=false
meson compile -C build-offload
meson test -C build-offload --print-errorlogs

meson setup / compile / test run under srun or sbatch on a GPU node, for example:

srun --partition=gpu-2xA100 --gres=gpu:2 --account=chem-ui \
  --time=02:00:00 --pty bash
# gpu-1xA100 nodes do not advertise GRES, so --gres=gpu:1 is
# rejected. gpu-2xA100 advertises gpu:2.

Eigen, Catch2, FlexiBLAS, Meson, and Ninja come from the GCC/13.3.0 EasyBuild toolchain on this cluster (module spider the name if a version is missing). They do not come from pixi.

If nvc++ refuses the sources, swap the compiler to a clang that ships libomptarget and keep -Dwith_openmp_offload=enabled. The probe then takes -fopenmp-targets=nvptx64-nvidia-cuda.

Runtime

SEAMS_OFFLOAD=0 forces the host path in an offload build. Unset, or any value other than 0, uses the device when omp_get_num_devices() > 0.

seams --features prints whether the binary was compiled with the offload define.

The Catch2 case steinhardtQl offload matches serial and threaded bit for bit compiles only when the probe succeeded. It compares the device path to one-thread and four-thread host paths on the FCC lattice and on input/traj/mW_cubic.lammpstrj, and it checks the FCC q4 / q6 reference values.

Profile

scripts/elja_gpu_job.sh is the sbatch body: it configures the offload build, runs the test suite, then nsys profile and nsys stats --force-export=true. The stats file must be non-empty.