SPDK (Storage Performance Development Kit) Docker Image
Simple image built following the official SPDK Documentation. This image intents to work with the NVMe device emulation image ljishen/qemu-nvme while can be also used in a system equipped with NVMe device.
The container startup process encompasses compiling the SPDK source so that it is aware of the underlying system supports, e.g. SSSE3, RDRAND. Relying on the docker build may cause errors like ERROR: This system does not support "XXXX". The container also sets up the BlobFS so that you can freely try
db_bench for performance evaluation. Note that 5GB (2560 huge pages) is allocated by default.
When work with the NVMe device emulation image ljishen/qemu-nvme, start the container without argument
docker run -ti \ --privileged \ --ipc host \ -v /dev:/dev \ -v /var/lib/docker/aufs/diff:/var/lib/docker/aufs/diff \ ljishen/spdk
For other systems with external module(s) installed (check
/lib/modules/$(uname -r)) and NVMe devices enabled
docker run -ti \ --privileged \ --ipc host \ -v /dev:/dev \ -v /var/lib/docker/aufs/diff:/var/lib/docker/aufs/diff \ ljishen/spdk \ /bin/bash
1) This docker command bind-mounts to the
/dev which is necessary since it needs to access
Here is the error message if we don't bind-mounts to /dev
EAL: Cannot open /dev/uio0: No such file or directory EAL: Requested device 0000:00:04.0 cannot be used mkfs.c: 77:spdk_mkfs_run: *ERROR*: bdev Nvme0n1 not found
2) The bind-mount to path
/var/lib/docker/aufs/diff is used when generating perf report during running
Here is the error message if we don't bind-mounts to /var/lib/docker/aufs/diff
Generating perf report for insert test phase...Failed to open /var/lib/docker/aufs/diff/d82991ccac9a6a30672470d40fdd864a8147ee4dc2e9bbef27057c05315c5775/lib/x86_64-linux-gnu/ld-2.23.so, continuing without symbols Failed to open /var/lib/docker/aufs/diff/4f877e8daca1d319bec46bfc85dea20d9fa25182ffc8345479a70b84aea0d2eb/root/rocksdb/db_bench, continuing without symbols Failed to open /var/lib/docker/aufs/diff/d82991ccac9a6a30672470d40fdd864a8147ee4dc2e9bbef27057c05315c5775/lib/x86_64-linux-gnu/libpthread-2.23.so, continuing without symbols Failed to open /var/lib/docker/aufs/diff/d82991ccac9a6a30672470d40fdd864a8147ee4dc2e9bbef27057c05315c5775/lib/x86_64-linux-gnu/libc-2.23.so, continuing without symbols Failed to open /var/lib/docker/aufs/diff/72db9e97d30e822bb272b33655f06878fa53f7acd8852cd187c9f39306130f2f/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21, continuing without symbols Failed to open /var/lib/docker/aufs/diff/d82991ccac9a6a30672470d40fdd864a8147ee4dc2e9bbef27057c05315c5775/lib/x86_64-linux-gnu/libm-2.23.so, continuing without symbols done.
Testing RocksDB with SPDK
The BlobFS Getting Started Guide has a detail instruction of how to run
db_bench against a variety of workloads with
test/blobfs/rocksdb/run_tests.sh. Since there are so many options for the
db_bench, it is also worthwhile to read its home page right here.
When running the
run_tests.sh, check the results in folder
/root/spdk/test/blobfs/rocksdb/results/last. If you see messages in file
insert_db_bench.txtsimilar to the following, you system may not have enough memory for the 5GB huge pages.
EAL: Not enough memory available! Requested: 5120MB, available: XXXXMB EAL: FATAL: Cannot init memory EAL: Cannot init memory Failed to initialize DPDK