bigdft/runtime repository overview

Runtime containers : For BigDFT users who don't need to modify the code, this is a smaller version already compiled that should run on most platforms. It also includes jupyter notebook support with BigDFT bindings, to ease interactions with BigDFT.

⁠Usage on a local system with docker :

You can avoid using "sudo" by adding your user to the "docker" group. Warning : read https://docs.docker.com/install/linux/linux-postinstall/⁠

To match your user id and group, add the --user $(id -u):$(id -g) option after docker run. By default the image uses user id 1000, which could not have permission to write files on your local directory.

• From a folder with input files, to run with one process (using all cores with OpenMP) :

sudo docker run -ti -v <abs_path_to_inputfiles>:/results -w /results bigdft/runtime:openmpi bigdft

• To change the number of OpenMP processes to 4 :

sudo docker run -ti -e OMP_NUM_THREADS=4 -v <abs_path_to_inputfiles>:/results -w /results bigdft/runtime:openmpi bigdft

• Other option : export OMP_NUM_THREADS=4 sudo docker run -ti -e OMP_NUM_THREADS=$OMP_NUM_THREADS -v <abs_path_to_inputfiles>:/results -w /results bigdft/runtime:openmpi bigdft

• To run with MPI+OpenMP sudo docker run -ti -e OMP_NUM_THREADS=$OMP_NUM_THREADS -v <abs_path_to_inputfiles>:/results -w /results bigdft/runtime:openmpi mpirun -np 2 bigdft

• Interactive session which can provide more flexibility: sudo docker run -ti -e OMP_NUM_THREADS=$OMP_NUM_THREADS -v <abs_path_to_inputfiles>:/results -w /results bigdft/runtime:openmpi bash

⁠Usage with jupyter notebooks

From a folder with a notebook (or you can create it inside) : sudo docker run -ti -p 8888:8888 -e OMP_NUM_THREADS=$OMP_NUM_THREADS -v <abs_path_to_notebookfile>:/results -w /results bigdft/runtime:openmpi

• then connect to localhost:8888, with password 'bigdft'. You can also run this command on a distant server and connect to serveraddress:8888 to remotely use BigDFT if port is correctly forwarded.

⁠Using with shifter (example for Piz Daint, more info on https://user.cscs.ch/scientific_computing/supported_applications/shifter/⁠)

module load daint-gpu module load shifter

We need to get the mvapich-based image, as OpenMPI is not yet compatible with Cray MPI

This will download the docker image, and turn it into a shifter image, available locally for launch with slurm.

shifterimg pull bigdft/runtime:mvapich_cuda8

and from the folder where the input files are

MPICH_RDMA_ENABLED_CUDA=1 srun -N2 -n2 -C gpu shifter --mpi --image=bigdft/runtime:mvapich_cuda8 bash -c "LD_PRELOAD=/opt/shifter/site-resources/gpu/lib64/libcuda.so bigdft"

Or with a script :

#!/bin/bash -l
#SBATCH --job-name="shifter"
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=12
#SBATCH --constraint=gpu
#SBATCH --time=01:00:00
 export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
#comment out this line for OpenCl jobs !
 export MPICH_RDMA_ENABLED_CUDA=1
#comment out this line for OpenCl jobs !
 export CRAY_CUDA_MPS=1

 module load daint-gpu
 module load shifter
 srun -n $SLURM_NTASKS --ntasks-per-node=$SLURM_NTASKS_PER_NODE -c $SLURM_CPUS_PER_TASK shifter --mpi --image=bigdft/runtime:mvapich_cuda8 bash -c "LD_PRELOAD=/opt/shifter/site-resources/gpu/lib64/libcuda.so bigdft"

⁠Usage with singularity + slurm

example with one task on one node :

module load Singularity/2.3.2 singularity pull docker://bigdft/runtime:openmpi Note: on singularity > 2.3, there might be a problem with the pull, as a new dependency is not installed on all systems. Ignore this step in this case

srun -N1 --ntasks-per-node=1 --pty mpirun -np 1 singularity exec --nv docker://bigdft/runtime:openmpi bigdft

Or with a script :

#!/bin/sh
#SBATCH --job-name="singularity"
#SBATCH --exclusive=user
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=4 
#SBATCH --time=06:00:00

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

module purge
module load Singularity/2.3.2

srun --pty mpirun -np $SLURM_NTASKS singularity exec --nv docker://bigdft/runtime:openmpi bigdft