Public | Automated Build

Last pushed: 13 days ago
Short Description
A repository for benchmark testing of OME Files
Full Description

OME Files Benchmark

The current repository contains the set of benchmark scripts used to
test the performance of the OME Files library.

Available benchmarks

A few different benchmarks are available via this repository. Each benchmark
contains its own set of tests and metrics. The following section describes how
to build and execute the benchmark. In addition, each benchmark might require
a set of predefined datasets.

Refer to the following pages for more details about the scope, datasets, tests,
and results of:

Building and executing the benchmark scripts

See the
OME Files C++ and
Bio-Formats building instructions.


The Windows build requirements are Cmake,
Maven, Visual
and a local version of the
standalone OME Files bundle matching the Visual Studio version.

For running our builds, we used the Continuous Integration software
Jenkins to trigger the Windows
benchmark builds. A single script executing the building and execution
steps is available under

To build the OME Files performance scripts manually, within a build
directory, execute the following cmake command:

$ cmake -G "Ninja" -DCMAKE_VERBOSE_MAKEFILE:BOOL=%verbose%
$ cmake --build .
$ cmake --build . --target install

The Bio-Formats performance script can be built within the source
directory using Maven:

$ cd source
$ call mvn clean install


The Linux benchmark was performed on Ubuntu 16.04. To ease the
distribution and reproducibility of the suite, the benchmark
environment is built using Docker via a
Dockerfile including the software prerequisites and the
benchmark code.

To retrieve the
Docker image,

$ docker pull openmicroscopy/ome-files-performance:latest

For testing, you can pull the
Docker image
built daily including all the Pull Requests:

$ docker pull snoopycrimecop/ome-files-performance:master_merge_trigger

To build the Docker image locally, run:

$ docker build -t ome-files-performance .

Create a unique benchmark directory which will be mounted as the /data
volume when running the Docker container, e.g.:

$ mkdir /temporary/benchmark-DD-MM-YYYY

where DD-MM-YYYY would be the date of execution of the benchmark.

This structure of this benchmark directory will be the following:

  • if the benchmark expects specific input datasets as specified in the
    ome-files-performance, copy this data under the root folder of
    the benchmark directory e.g.
  • the results generated by the benchmark will be stored under a subfolder
    called results,
  • the temporary files generated by the benchmark will be stored under a
    subfolder called :file:out.

The default docker run command will execute the benchmarks defined by the scripts/run_benchmarking script without argument:

$ docker run --rm -it -v /temporary/benchmark-DD-MM-YYYY:/data openmicroscopy/ome-files-performance:latest

To execute a particular benchmark, start a shell in the Docker container using
docker run --entrypoint, e.g.:

$ docker run --rm -it --entrypoint=bash -v /temporary/benchmark-DD-MM-YYYY:/data snoopycrimecop/ome-files-performance:master_merge_trigger

Then from the shell, execute the desired benchmark::

root@084bb88d5a62:/git/ome-files-performance$ ./scripts/run_benchmarking metadata


Docker Pull Command