flaviostutz/benchmark repository overview

⁠Table of Content

• Overview⁠
  • Measurements⁠
• Test Cases⁠
  • Brainfuck⁠
    • bench.b⁠
    • mandel.b⁠
  • Base64⁠
  • Json⁠
  • Matmul⁠
• Tests Execution⁠
  • Environment⁠
  • Using Docker⁠
  • Manual Execution⁠
    • Prerequisites⁠
• Contribution⁠
  • Makefile guide⁠
    • Binary executables⁠
    • Compiled artifacts⁠
    • Scripting language⁠

⁠Overview

The benchmarks follow the criteria:

• They are written as the average software developer would write them, i.e.

  • The algorithms are implemented as cited in public sources;
  • The libraries are used as described in the tutorials, documentation and examples;
  • The used data structures are idiomatic.

• The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

• All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks⁠, crystal-benchmarks-game⁠

⁠Measurements

The measured values are:

• time spent for the benchmark execution (loading required data and code self-testing are not measured);
• memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
• energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2020-12-01

⁠Test Cases

⁠Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

• Verbose (default). Prints the output immediately.
• Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck⁠

⁠bench.b

⁠mandel.b

Mandel in Brainfuck⁠

⁠Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64⁠

⁠Json

Testing parsing and simple calculating of values from a big JSON file.

Json⁠