mlawsonca/benchmarking_suite_range_searching_libraries repository overview

⁠Benchmarking Suite for Range Tree Libraries

This project contains all of the code needed to replicate the experiments for the arXiv paper: "Exploring Spatial Indexing for Accelerated Feature Retrieval in HPC." The code compares 20 different C/C++ spatial indexing libraries, and includes 1). correctness tests for the libraries and 2). the set of benchmarks used in the paper. These benchmarks include performance, strong and weak scaling, and memory usage tests.

⁠Getting Started

Since this work uses High Performance computing, our instructions will focus on Singularity.

To start, pull this container image from Docker:

singularity pull --docker-login docker://mlawsonca/benchmarking_suite_range_searching_libraries

⁠Usage

To generate the job scripts, execute the following:

singularity exec benchmarking_suite_range_searching_libraries_latest.sif cd $HOME/paper_benchmarks/src/benchmark/make_job_scripts && g++ make_job_scripts.cpp -o make_job_scripts && ./make_job_scripts [slurm | lsf | none] max_number_of_concurrent jobs /path/to/singularity/image/benchmarking_suite_range_searching_libraries_latest.sif [all | small | large | scaling] num_procs_per_node [job_scheduler_account_number] && g++ make_massif_commands.cpp -o make_massif_commands && ./make_massif_commands 0 && ./make_massif_commands 1

• The first parameter is the job scheduler (slurm, lsf, or none to indicate the commands should be added to a single bash file)
• The second parameter is the maximum number of concurrent jobs to run
• The third parameter is the full path to the container
• The fourth parameter is which benchmarks to run: all of them ('all'), the small jobs ('small'), which have a small amount of data per process, the large jobs ('large'), which a large amount of data per process, or the scaling jobs ('scaling'), which test strong and weak scaling.
• The fifth parameter is the number of processes to use per node. For 'all', 'small' or 'scaling' jobs, this must be divisible by 10 (to enable us to use 10x as many processes for the small scale jobs vs the large scale jobs). For the 'all' or 'scaling' jobs, this number must additionally be divisible by 8 (to enable us to test with up to 8x more or fewer processors (for the large and small cases, respectively).
• The sixth parameter is optional, and is your account name to use in the job scheduler scripts (so you don't have to add this manually).

⁠The performance and scalability tests

First, copy the job scripts to a local folder:

singularity exec benchmarking_suite_range_searching_libraries_latest.sif cp -r $HOME/paper_benchmarks/src/benchmark/job_scripts /my/desired/path

The performance and scaling job scripts can be added to the queue with:

chmod u+x sbatch_script && ./sbatch_script

The results can then be retrieved:

singularity exec benchmarking_suite_range_searching_libraries_latest.sif cp -r $HOME/paper_benchmarks/src/benchmark/output my/desired/path

⁠The memory tests

The commands for the memory tests are found in $HOME/paper_benchmarks/src/benchmark/memory_results_large/massif_commands.bash and $HOME/paper_benchmarks/src/benchmark/memory_results_small/massif_commands.bash. These could be executed interactively. e.g.,

singularity shell benchmarking_suite_range_searching_libraries_latest.sif
Singularity/Debian.sif> chmod u+x $HOME/paper_benchmarks/src/benchmark/memory_results_[large|small]/massif_commands.bash && $HOME/paper_benchmarks/src/benchmark/memory_results_[large|small]/massif_commands.bash

Or, using a simple job scheduler script, e.g.,

#!/bin/bash

#BSUB -nnodes 1
#BSUB -core_isolation 2
#BSUB -Jd Memory_Tests
#BSUB -W 04:00

singularity exec benchmarking_suite_range_searching_libraries_latest.sif  $HOME/paper_benchmarks/src/benchmark/memory_results_[large|small]/massif_commands.bash

The results can then be retrieved:

singularity exec cp $HOME/paper_benchmarks/src/benchmark/memory_results_[large|small]/output.log my/desired/path

⁠Prerequisities

In order to run this container you'll need Singularity installed. Docker could be used as well with minor adjustments to the above commands. When generating the job scripts, instead of supplying lsf or slurm as the job scheduler, 'none' should be supplied. This will cause all of the necessary commands to be added to a single bash file in the job_scripts folder (e.g., 'scaling_run_script.sh'). The file is still designed to be used with singularity, but the singularity command (singularity exec path_to_container) can be stripped from each line, leaving a script that can be run entirely within the container.

⁠Find Us

• GitHub⁠

⁠Authors

• Margaret Lawson

⁠License

This project is licensed under the MIT License - see the COPYING file (paper_benchmarks/COPYING) file for details.