Building hpic2 on Ubuntu from source

Developers should build hPIC2 from source so that they can make changes, compile, and test locally. We have found that it is easiest to install dependencies via Spack, then manually run cmake to set up the hPIC2 build.

It is highly recommended installing all the dependencies of hPIC2 via Spack, even if you already have some of them installed on your system. This will ensure that the correct versions are used. Before installing hPIC2 from source, set up Spack and the dependencies as follows.

Ensure you have some basic compilers installed:

sudo apt install -y build-essential gfortran

Setting up your Spack environment for hPIC2 dependencies

Download the Spack repository (suggested location, your $HOME directory):

git clone -c feature.manyFiles=true https://github.com/spack/spack.git

Source the spack environment:

source $HOME/spack/share/spack/setup-env.sh

Note

Since the source step is needed every time, we suggest to add the source command to your .bashrc file to avoid forgetting to re-type it every time you open a new terminal.

echo "source $HOME/spack/share/spack/setup-env.sh" >> ~/.bashrc

Let Spack find your compilers,

spack compiler find

make a new spack repository for hpic2,

spack repo create hpic2_dev

register a Spack repository for hpic2,

spack repo add hpic2_dev

change to the spack repository packages directory,

cd hpic2_dev/packages

Download the hpic2 source with its submodules,

git clone --recurse-submodules https://github.com/lcpp-org/hpic2.git

Optionally, download the rustbca source at the same location, and rename it to rustbca

git clone https://github.com/lcpp-org/RustBCA.git
mv RustBCA rustbca

Building hPIC2 with OpenMP

Create and enter a spack env

spack env create hpic2_omp_opt
spack env activate hpic2_omp_opt

Add dependencies to spack env

spack add googletest
spack add hypre+openmp
spack add kokkos~cuda+openmp
spack add mfem~cuda+openmp~zlib
spack add mpi
spack add spdlog
spack add hdf5+cxx+mpi

Note

If you would like to specify a version for each dependency, you can specify it with the @: operator, e.g. spack add kokkos@:3.8 ~cuda+openmp; otherwise Spack will automatically choose the last version available from the main Spack package list.

Optionally add also RustBCA

spack add rustbca

And install all dependencies

spack install

Note

Spack installation without RustBCA takes about 30 minutes on a HP-Z230. Including RustBCA increases the Spack install time to about 10 hours.

Create a build directory

mkdir ~/hpic2_dev/hpic2_omp_opt

Create a CMake script to configure the build

cat > ~/hpic2_dev/hpic2_omp_opt.sh << 'EOF'
spacktivate hpic2_omp_opt
cmake -DCMAKE_BUILD_TYPE=Release -DWITH_MFEM=ON -DWITH_TESTS=ON -DWITH_RUSTBCA=ON ~/hpic2_dev/packages/hpic2
EOF

Source the configure script from the build directory

cd ~/hpic2_dev/hpic2_omp_opt
. ../hpic2_omp_opt.sh

Compile

make

When you make changes to the source code in ~/hpic2_dev/hpic2, you need only run make from the build directory again to recompile. Note that when you open a fresh terminal, you must spacktivate the Spack env or source the configure script again before you can make.

Building hPIC2 with CUDA

Look up your GPU on the CUDA GPUs website. Remove the decimal from its Compute Capability, so that 3.7 becomes 37, for example. Store this temporarily as an environment variable

export MY_CC = <compute capability without decimal>

Create and enter a spack env

spack env create hpic2_cuda_opt
spack env activate hpic2_cuda_opt

Add dependencies to spack env and install them

spack add googletest
spack add hypre+openmp+cuda cuda_arch=$MY_CC
spack add kokkos+cuda+cuda_lambda+openmp+wrapper cuda_arch=$MY_CC
spack add mfem+cuda+openmp~zlib cuda_arch=$MY_CC
spack add openmpi +cuda cuda_arch=$MY_CC
spack add rustbca
spack add spdlog
spack add hdf5+cxx+mpi
spack install

Create a build directory

mkdir ~/hpic2_dev/hpic2_cuda_opt

Create a CMake script to configure the build

cat > ~/hpic2_dev/hpic2_cuda_opt.sh << 'EOF'
spacktivate hpic2_cuda_opt
cmake -DCMAKE_BUILD_TYPE=Release -DWITH_MFEM=ON -DWITH_TESTS=ON -DWITH_RUSTBCA=ON ~/hpic2_dev/hpic2
EOF

Source the configure script from the build directory

cd ~/hpic2_dev/hpic2_cuda_opt
. ../hpic2_cuda_opt.sh

Compile

make

When you make changes to the source code in ~/hpic2_dev/hpic2, you need only run make from the build directory again to recompile.

Note

Note that when you open a fresh terminal, you must spacktivate the Spack env or source the configure script again before you can make.