kmgowda/sbk-file repository overview
Storage Benchmark Kit 
Any Storage System... Any Payload... Any Time Stamp...
The SBK (Storage Benchmark Kit) is an open source software frame-work for the performance benchmarking of any storage system. If you are curious to measure the maximum throughput performance of your storage device/system, then SBK is the right software for you. The SBK itself is a very high-performance benchmark tool/framework. It massively writes the data to the storage system and reads the data from the storage system. The SBK supports multi writers and readers and also the End to End latency benchmarking. The percentiles are calculated for complete data written/read without any sampling; hence the percentiles are 100% accurate.
Supported By
The design principle of SBK is the Performance Benchmarking of 'Any Storage System' with 'Any Type of data payload' and 'Any Time Stamp', because, the SBK is not specific to particular type of storage system, it can be used for performance benchmarking of any storage system, let it be file system, databases , any distributed storage systems or message queues by adding SBK driver which specifies the IO operations of storage system. you can find the list of supported drivers below. The SBK supports a variety of payloads too, such as byte array, byte buffer, string, and you can add your own payload type. The Latency values can be measured either in milliseconds, microseconds or nanoseconds using SBK.
SBK supports performance benchmarking of following storage systems
| # | Driver | # | Driver |
|---|---|---|---|
| 1. | Artemis | 21. | Minio |
| 2. | Asyncfile | 22. | Mongodb |
| 3. | Bookkeeper | 23. | Mssql |
| 4. | Cassandra | 24. | Mysql |
| 5. | Cephs3 | 25. | Nats |
| 6. | Concurrentq | 26. | NatsStream |
| 7. | Couchdb | 27. | Nsq |
| 8. | Csv | 28. | Null |
| 9. | Db2 | 29. | Openio |
| 10. | Derby | 30. | Postgresql |
| 11. | Fdbrecord | 31. | Pravega |
| 12. | File | 32. | Pulsar |
| 13. | Filestream | 33. | Rabbitmq |
| 14. | Foundationdb | 34. | Redis |
| 15. | Hdfs | 35. | Redpanda |
| 16. | Hive | 36. | Rocketmq |
| 17. | Ignite | 37. | Rocksdb |
| 18. | Jdbc | 38. | Seaweeds3 |
| 19. | Kafka | 39. | Sqlite |
| 20. | Mariadb |
In the future, many more storage systems drivers will be plugged in
We welcome open source developers to contribute to this project by adding a driver for your storage device and any features to SBK. Refer to :
- Contributing to SBK for the Contributing guidelines.
- Add your storage driver to SBK to know how to add your driver (storage device driver or client) for performance benchmarking.
Build SBK
Prerequisites
- Java 11+
- Gradle 6+
Building
Checkout the source code:
git clone https://github.com/kmgowda/SBK.git
cd SBK
Build the SBK:
./gradlew build
untar the SBK to local folder
tar -xvf ./build/distributions/sbk.tar -C ./build/distributions/.
Running SBK locally:
<SBK directory>/./build/distributions/sbk/bin/sbk -help
...
usage: sbk
Storage Benchmark Kit
-class <arg> Storage Driver Class,
Available Drivers [Artemis, AsyncFile, BookKeeper,
CSV, Cassandra, CephS3, ConcurrentQ, CouchDB, Db2,
Derby, FdbRecord, File, FileStream, FoundationDB,
HDFS, Hive, Ignite, Jdbc, Kafka, MariaDB, MinIO,
MongoDB, MsSql, MySQL, Nats, NatsStream, Nsq, Null,
OpenIO, PostgreSQL, Pravega, Pulsar, RabbitMQ,
RedPanda, Redis, RocketMQ, RocksDB, SQLite,
SeaweedS3]
-context <arg> Prometheus Metric context; default: 9718/metrics;
'no' disables the metrics
-help Help message
-ram <arg> SBK RAM host; default: no; disable if this parameter
is set to 'no'
-ramport <arg> SBK RAM Port; default: 9716
-readers <arg> Number of readers
-records <arg> Number of records(events) if 'seconds' not specified;
otherwise, Maximum records per second by writer(s)
and/or Number of records per reader
-rsec <arg> Number of seconds/step for readers, default: 0
-rstep <arg> Number of readers/step, default: 1
-seconds <arg> Number of seconds to run; if not specified, runs
forever
-size <arg> Size of each message (event or record)
-sync <arg> Each Writer calls flush/sync after writing <arg>
number of of events(records) ; <arg> number of
events(records) per Write or Read Transaction
-throughput <arg> if > 0 , throughput in MB/s
if 0 , writes/reads 'records'
if -1, get the maximum throughput (default: -1)
-time <arg> Latency Time Unit [ms:MILLISECONDS, mcs:MICROSECONDS,
ns:NANOSECONDS]; default: ms
-writers <arg> Number of writers
-wsec <arg> Number of seconds/step for writers, default: 0
-wstep <arg> Number of writers/step, default: 1
Please report issues at https://github.com/kmgowda/SBK
Just to check the SBK build issue the command
./gradlew check
Build only the SBK install binary
./gradlew installDist
executable binary will be available at : [SBK directory]/./build/install/sbk/bin/sbk
Running Performance benchmarking
The SBK can be executed to
- write/read a specific amount of events/records to/from the storage driver (device/cluster)
- write/read the events/records for the specified amount of time
SBK outputs the data written/read , average throughput and latency , maximum latency and the latency percentiles 10th, 25th, 50th, 75th, 95th, 99th , 99.9th and 99.99th for every 5 seconds time interval as show below.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 832450 records, 166423.4 records/sec, 15.87 MB/sec, 5.8 ms avg latency, 67 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 6 ms 75th, 8 ms 90th, 9 ms 95th, 12 ms 99th, 16 ms 99.9th, 26 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 978068 records, 195418.2 records/sec, 18.64 MB/sec, 5.1 ms avg latency, 70 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 8 ms 99th, 66 ms 99.9th, 67 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 994458 records, 198732.6 records/sec, 18.95 MB/sec, 5.0 ms avg latency, 18 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 7 ms 99th, 13 ms 99.9th, 13 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 991436 records, 198089.1 records/sec, 18.89 MB/sec, 5.0 ms avg latency, 32 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 8 ms 99th, 28 ms 99.9th, 28 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 967234 records, 193408.1 records/sec, 18.44 MB/sec, 5.1 ms avg latency, 104 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 7 ms 99th, 100 ms 99.9th, 100 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 971933 records, 194308.9 records/sec, 18.53 MB/sec, 5.1 ms avg latency, 18 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 7 ms 95th, 9 ms 99th, 12 ms 99.9th, 14 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 995450 records, 199050.2 records/sec, 18.98 MB/sec, 5.0 ms avg latency, 26 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 7 ms 99th, 18 ms 99.9th, 23 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 1005407 records, 201041.2 records/sec, 19.17 MB/sec, 4.9 ms avg latency, 13 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 7 ms 99th, 9 ms 99.9th, 10 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 971825 records, 194209.6 records/sec, 18.52 MB/sec, 5.1 ms avg latency, 15 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 8 ms 99th, 10 ms 99.9th, 11 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 996529 records, 199226.1 records/sec, 19.00 MB/sec, 5.0 ms avg latency, 26 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 8 ms 99th, 17 ms 99.9th, 18 ms 99.99th.
Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 999380 records, 199836.0 records/sec, 19.06 MB/sec, 5.0 ms avg latency, 13 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 7 ms 99th, 9 ms 99.9th, 10 ms 99.99th.
At the end of the benchmarking session, SBK outputs the total data written/read , average throughput and latency , maximum latency and the latency percentiles 10th, 25th, 50th, 75th, 95th, 99th , 99.9th and 99.99th for the complete data records written/read. An example final output is show as below:
Total : Pulsar Writing 1 Writers, 0 Readers, 1 Max Writers, 0 Max Readers, 11692470 records, 194874.5 records/sec, 18.58 MB/sec, 5.1 ms avg latency, 104 ms max latency; 0 invalid latencies; Discarded Latencies: 0 lower, 0 higher; 4 ms 10th, 5 ms 25th, 5 ms 50th, 5 ms 75th, 6 ms 90th, 6 ms 95th, 8 ms 99th, 14 ms 99.9th, 67 ms 99.99th.
Grafana Dashboards of SBK
When you run the SBK, by default it starts the http server and all the output benchmark data is directed to the default port number: 9718 and metrics context. If you want to change the port number and context, you can use the command line argument -context to change the same. you have to run the prometheus monitoring system (server [default port number is 9090] cum client) which pulls/fetches the benchmark data from the local/remote http server. If you want to include additional SBK nodes/instances to fetch the performance data or from port number other than 9718, you need to extend or update targets.json In case, if you are fetching metrics/benchmark data from remote http server , or from the context other than metrics then you need to change the default prometheus server configuration too. Run the grafana server (cum a client) to fetch the benchmark data from prometheus. For example, if you are running a local grafana server then by default it fetches the data from the prometheus server at the local port 9090. You can access the local grafana server at localhost:3000 in your browser using admin/admin as default username / password. You can import the grafana dashboards to fetch the SBK benchmark data of the existing supported storage drivers from grafana dashboards.
The sample output of Standalone Pulsar benchmark data with grafana is below
Port conflicts between storage servers and grafana/prometheus
- If you are running Pravega server in standalone/local mode or if you are running SBK in the same system in which Pravega controller is also running, then Prometheus port 9090 conflicts with the Pravega controller. So, either you change the Pravega controller port number or change the Prometheus port number in the Prometheus targets file before deploying the prometheus.
- If you find that using the local port 9718 conflicts with a storage server or any other application. Then, you can change the SBK's http port using -metrics option, and you need change the [Prometheus targets.json] (https://github.com/kmgowda/SBK/blob/master/grafana/prometheus/targets.json) too
Distributed SBK
SBK can be deployed in a distributed clusters using SBK-RAM
SBK Docker Containers
The SBK Docker images are available at SBK Docker
The SBK docker image pull command is
docker pull kmgowda/sbk
you can straightaway run the docker image too, For example
docker run -p 127.0.0.1:9718:9718/tcp kmgowda/sbk:latest -class rabbitmq -broker 192.168.0.192 -topic kmg-topic-11 -writers 5 -readers 1 -size 100 -seconds 60
- Note that the option -p 127.0.0.1:9718:9718/tcp redirects the 9718 port to local port to fetch the performance metric data for Prometheus.
- Avoid using the --network host option , because this option overrides the port redirection.
SBK Docker Compose
The SBK docker compose consists of SBK docker image, Grafana and prometheus docker images. The grafana image contains the dashboards which can be directly deployed for the performance analytics.
As an example, just follow the below steps to see the performance graphs
-
In the SBK directory build the 'SBK' service of the docker compose file as follows.
<SBK dir>% docker-compose build -
Run the 'SBK' service as follows.
<SBK dir>% docker-compose run sbk -class concurrentq -writers 1 -readers 5 -size 1000 -seconds 120 -
login to grafana local host port 3000 with username admin and password sbk
-
go to dashboard menu and pick the dashboard of the storage device on which you are running the performance benchmarking. in the above example, you can choose the Concurrent Queue dashboard.
-
The SBK docker compose runs the SBK image as docker container. In case, if you are running SBK as an application, and you want to see the SBK performance graphs using Grafana, then use Grafana Docker compose
SBK Kubernetes
check these SBK Kubernetes Deployments samples for details on SBK as kubernetes pod. If you want to run the Grafana and prometheus as Kubernetes pods, then use [Grafana Kubernetes deployment] (https://github.com/kmgowda/SBK/tree/master/grafana#grafana-with-kubernetes)
SBK Execution Modes
The SBK can be executed in the following modes:
1. Burst Mode (Max rate mode)
2. Throughput Mode
3. Rate limiter Mode
4. End to End Latency Mode
1 - Burst Mode / Max Rate Mode
In this mode, the SBK pushes/pulls the messages to/from the storage client(device/driver) as much as possible. This mode is used to find the maximum and throughput that can be obtained from the storage device or storage cluster (server). This mode can be used for both writers and readers. By default, the SBK runs in Burst mode.
For example: The Burst mode for pulsar single writer as follows
<SBK directory>./build/distributions/sbk/bin/sbk -class Pulsar -admin http://localhost:8080 -broker tcp://localhost:6650 -topic topic-k-223 -partitions 1 -writers 1 -size 1000 -seconds 60 -throughput -1
The -throughput -1 indicates the burst mode. Note that, you don't supply the parameter -throughput then also its burst mode.
This test will be executed for 60 seconds because option -seconds 60 is used.
This test tries to write and read events of size 1000 bytes to/from the topic 'topic-k-223'.
The option '-broker tcp://localhost:6650' specifies the Pulsar broker IP address and port number for write operations.
The option '-admin http://localhost:8080' specifies the Pulsar admin IP and port number for topic creation and deletion.
Note that -producers 1 indicates 1 producer/writers.
in the case you want to write/read the certain number of records.events use the -records option without -seconds option as follows
<SBK directory>/build/distributions/sbk/bin/sbk -class Pulsar -admin http://localhost:8080 -broker tcp://localhost:6650 -topic topic-k-223 -partitions 1 -writers 1 -size 1000 -records 100000 -throughput -1
-records <number> indicates that total <number> of records to write/read
2 - Throughput Mode
In this mode, the SBK pushes/pulls/from the messages to the storage client(device/driver) with specified approximate maximum throughput in terms of Mega Bytes/second (MB/s). This mode is used to find the least latency that can be obtained from the storage device or storage cluster (server) for given throughput.
For example: The throughput mode for pulsar 5 writers as follows
<SBK directory> ./build/distributions/sbk/bin/sbk -class Pulsar -admin http://localhost:8080 -broker tcp://localhost:6650 -topic topic-k-223 -partitions 1 -writers 5 -size 1000 -seconds 120 -throughput 10
The -throughput <positive number> indicates the Throughput mode.
This test will be executed with approximate max throughput of 10MB/sec.
This test will be executed for 120 seconds (2 minutes) because option -seconds 120 is used.
This test tries to write and read events of size 1000 bytes to/from the topic 'topic-k-223' of 1 partition.
If the topic 'topic-k-223' is not existing , then it will be created with 1 segment.
if the steam is already existing then it will be deleted and recreated with 1 segment.
Note that -writers 5 indicates 5 producers/writers .
in the case you want to write/read the certain number of events use the -records option without -seconds option as follows
<SBK directory>./build/distributions/sbk/bin/sbk -class Pulsar -admin http://localhost:8080 -broker tcp://localhost:6650 -topic topic-k-223 -partitions 1 -writers 5 -size 1000 -records 1000000 -throughput 10
-records 1000000 indicates that total 1000000 (1 million) of events will be written at the throughput speed of 10MB/sec
3 - Rate limiter Mode
This mode is another form of controlling writers/readers throughput by limiting the number of records per second. In this mode, the SBK pushes/pulls the messages to/from the storage client (device/driver) with specified approximate maximum records per sec. This mode is used to find the least
Tag summary
Content type
Image
Digest
Size
161.5 MB
Last updated
about 5 years ago
docker pull kmgowda/sbk-file