gaunab/bencher repository overview

⁠Docker Container

The Docker container can be pulled from the Docker Hub⁠ or built locally. It contains all benchmarks and dependencies and exposes the benchmark server via port 50051.

We give an exemplary usage of the Docker container in the bencherclient⁠ repository.

pwd # /path/to/bencher
docker build -t bencher .
# always keep the container running, can be stopped with docker stop <container-id>
docker run -p 50051:50051 --restart always -d gaunab/bencher:latest

or

docker pull gaunab/bencher:latest
# always keep the container running, can be stopped with docker stop <container-id>
docker run -p 50051:50051 --restart always -d gaunab/bencher:latest

⁠Apptainer / Singularity Container

You can build an Apptainer container from the Docker image:

Bootstrap: docker
From: gaunab/bencher:latest
Namespace:
Stage: build

%environment
    export LANG=C.UTF-8
    export PATH="/root/.local/bin:$PATH"

%post
    cd /opt
    git clone your-repo
    cd your-repo
    pip install bencherscaffold # you'll need bencherscaffold to call bencher
    pip install your-dependencies

%startscript
    bash -c "/docker-entrypoint.sh"

%runscript
    bash -c "your-command-to-run-your-app"

This will create an Apptainer container with the Docker image gaunab/bencher:latest and the repository your-repo with the dependencies your-dependencies installed.

⁠Usage

⁠Starting the instance

apptainer build container.sif your-apptainer-file

⁠Start the Apptainer instance

This starts all the benchmarks in the container (as defined in the startscript of the Apptainer file).

apptainer instance start container.sif your-instance-name

⁠Run your command that depends on the benchmarks

This runs your command in the instance your-instance-name as defined in the runscript of the Apptainer file.

apptainer run instance://your-instance-name

⁠Evaluating a benchmark

We show how to run all benchmarks in the bencherclient⁠ repository. You don't need to use this repository, it is mainly used to test the benchmarks. The general setup to evaluate a benchmark is as follows. First, install the bencherscaffold⁠ package:

pip install git+https://github.com/LeoIV/BencherScaffold

Then, you can use the following code to evaluate a benchmark:

from bencherscaffold.client import BencherClient
from bencherscaffold.protoclasses.bencher_pb2 import Value, ValueType

# Create a client to communicate with the Bencher server
# By default, it connects to 127.0.0.1:50051
client = BencherClient()

# Create a list of values to evaluate
values = [Value(type=ValueType.CONTINUOUS, value=0.5) for _ in range(180)]
# The benchmark name is the name of the benchmark you want to evaluate
benchmark_name = 'lasso-dna'

# Evaluate the benchmark with the given values
# This will send the values to the server and return the result
# If the server is not running, it will raise an error
result = client.evaluate_point(
    benchmark_name=benchmark_name,
    point=values,
)
print(f"Result: {result}")

⁠Available Benchmarks

The following benchmarks are available:

⁠Footnotes

1. LassoBench⁠ ( Šehić Kenan, Gramfort Alexandre, Salmon Joseph and Nardi Luigi, "LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso", AutoML conference, 2022.) ↩⁠ ↩²⁠ ↩³⁠ ↩⁴⁠ ↩⁵⁠ ↩⁶⁠ ↩⁷⁠ ↩⁸⁠ ↩⁹⁠

2. BAxUS⁠ Leonard Papenmeier, Luigi Nardi, and Matthias Poloczek, "Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces", NeurIPS 2022 ↩⁠ ↩²⁠ ↩³⁠ ↩⁴⁠ ↩⁵⁠ ↩⁶⁠ ↩⁷⁠ ↩⁸⁠ ↩⁹⁠ ↩¹⁰⁠ ↩¹¹⁠

3. The LassoBench paper states 19,959 features, but the number of features in the RCV1 dataset is 47,236. ↩⁠

4. SAASBO⁠ David Eriksson and Martin Jankowiak, "High-dimensional Bayesian optimization with sparse axis-aligned subspaces", UAI 2021 ↩⁠ ↩²⁠

5. BODi⁠ Aryan Deshwal, Sebastian Ament, Maximilian Balandat, Eytan Bakshy, Janardhan Rao Doppa, and David Eriksson, "Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings", AISTATS 2023 ↩⁠ ↩²⁠

6. Bounce⁠ Leonard Papenmeier, Luigi Nardi and Matthias Poloczek, "Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces", NeurIPS 2023 ↩⁠ ↩²⁠ ↩³⁠

7. TurBO⁠ ( David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner and Matthias Poloczek, "Scalable Global Optimization via Local Bayesian Optimization." NeurIPS 2019) ↩⁠ ↩²⁠ ↩³⁠

8. LA-MCTS⁠ Linnan Wang, Rodrigo Fonseca, and Yuandong Tian, "Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search", NeurIPS 2020 ↩⁠ ↩²⁠ ↩³⁠ ↩⁴⁠ ↩⁵⁠ ↩⁶⁠

9. The SVM benchmark is not included in the repository and was obtained by corresponding with the authors of the paper. ↩⁠

10. Oh, Changyong, et al. "Combinatorial bayesian optimization using the graph cartesian product." Advances in Neural Information Processing Systems 32 (2019). ↩⁠

11. Each category has 5 possible values. The benchmark expects an integer between 0 and 4 for each category. ↩⁠

12. de Nobel, Jacob, et al. "Iohexperimenter: Benchmarking platform for iterative optimization heuristics." Evolutionary Computation 32.3 (2024): 205-210. ↩⁠ ↩²⁠ ↩³⁠ ↩⁴⁠ ↩⁵⁠ ↩⁶⁠ ↩⁷⁠ ↩⁸⁠ ↩⁹⁠ ↩¹⁰⁠ ↩¹¹⁠ ↩¹²⁠ ↩¹³⁠ ↩¹⁴⁠ ↩¹⁵⁠ ↩¹⁶⁠ ↩¹⁷⁠ ↩¹⁸⁠ ↩¹⁹⁠ ↩²⁰⁠ ↩²¹⁠ ↩²²⁠ ↩²³⁠ ↩²⁴⁠ ↩²⁵⁠ ↩²⁶⁠ ↩²⁷⁠ ↩²⁸⁠ ↩²⁹⁠ ↩³⁰⁠ ↩³¹⁠ ↩³²⁠ ↩³³⁠ ↩³⁴⁠ ↩³⁵⁠ ↩³⁶⁠ ↩³⁷⁠ ↩³⁸⁠ ↩³⁹⁠ ↩⁴⁰⁠ ↩⁴¹⁠ ↩⁴²⁠ ↩⁴³⁠ ↩⁴⁴⁠ ↩⁴⁵⁠ ↩⁴⁶⁠ ↩⁴⁷⁠ ↩⁴⁸⁠ ↩⁴⁹⁠ ↩⁵⁰⁠ ↩⁵¹⁠ ↩⁵²⁠ ↩⁵³⁠ ↩⁵⁴⁠ ↩⁵⁵⁠ ↩⁵⁶⁠

13. Hansen, Nikolaus, et al. "COCO: A platform for comparing continuous optimizers in a black-box setting." Optimization Methods and Software 36.1 (2021): 114-144. ↩⁠ ↩²⁠ ↩³⁠ ↩⁴⁠ ↩⁵⁠ ↩⁶⁠ ↩⁷⁠ ↩⁸⁠ ↩⁹⁠ ↩¹⁰⁠ ↩¹¹⁠ ↩¹²⁠ ↩¹³⁠ ↩¹⁴⁠ ↩¹⁵⁠ ↩¹⁶⁠ ↩¹⁷⁠ ↩¹⁸⁠ ↩¹⁹⁠ ↩²⁰⁠ ↩²¹⁠ ↩²²⁠ ↩²³⁠ ↩²⁴⁠