faroit/python-audio-benchmark repository overview

⁠Python Audio-Loading Benchmark

The aim of his repository is to evaluate the loading performance of various audio I/O packages interfaced from python.

This is relevant for machine learning models that today often process raw (time domain) audio and assembling a batch on the fly. It is therefore important to load the audio as fast as possible. At the same time a library should ideally support a variety of uncompressed and compressed audio formats and also is capable of loading only chunks of audio (seeking). The latter is especially important for models that cannot easily work with samples of variable length (convnets).

⁠Tested Libraries

⁠Not tested

• audioread (coreaudio)⁠: only available on macOS.
• madmom⁠: same ffmpeg interface as ar_ffmpeg.
• tensorflow 2 decode_wav⁠: Not released yet.
• python builtin wave⁠: TODO

⁠Results

The benchmark loads a number of (single channel) audio files of different length (between 1 and 151 seconds) and measures the time until the audio is converted to a tensor. Depending on the target tensor type (either numpy, pytorch or tensorflow) a different number of libraries were compared. E.g. when the output type is numpy and the target tensor type is tensorflow, the loading time included the cast operation to the target tensor. Furthermore, multiprocessing was disabled for data loaders that support them.

All results shown below, depict loading time in seconds for wav and mp3 files.

⁠Load to Numpy Tensor

⁠Load to PyTorch Tensor

⁠Load to Tensorflow Tensor

⁠Getting metadata information

In addition to loading the file, one might also be interested in extracting metadata. To benchmark this we asked for every file to provide metadata for sampling rate, channels, samples, and duration. All in consecutive calls, which means the file is not allowed to be opened once and extract all metadata together. Note, that we have excluded pydub from the benchmark results on metadata as it was significantly slower than the other tools.

⁠Running the Benchmark

⁠Installation using Docker

Build the docker container using docker build -t audio_benchmark ., it installs all the package requirements for all audio libraries.

⁠Generate sample data

To test the loading speed, we generate different durations of random (noise) audio data and encode it either to PCM 16bit WAV, MP3 CBR, or MP4. The data is generated by using a shell script. To generate the data in the folder AUDIO, run generate_audio.sh.

⁠Start Benchmark

Mount the data directory into the docker container and run run.sh inside the container. Afterwards, run plot.py to visualze the results.

⁠Authors

@faroit, @hagenw

⁠Contribution

We encourage interested users to contribute to this repository in the issue section and via pull requests. Particularly interesting are notifications of new tools and new versions of existing packages. Since benchmarks are subjective, I (@faroit) will reran the benchmark on our server again.