avs75alatau/minidfscluster repository overview

⁠MiniDFSCluster Docker Image

⁠Overview

This Docker image provides a local MiniDFSCluster environment to simplify testing HDFS-based applications such as Apache Spark.

⁠Key Features

• NameNode with fixed ports for both RPC and Web UI.
• Configurable number of DataNodes (default: 1, max: 9).
• Automatic creation of /tmp and /user directories in HDFS.
• Verified functionality on Ubuntu, CentOS, Windows, and macOS.

⁠Pull Image

docker pull avs75alatau/minidfscluster:latest

⁠Running the Container

⁠Linux

docker run -d --name hdfs-minicluster --network=host -e NUM_DATANODES=3 avs75alatau/minidfscluster:latest

⁠Windows/macOS

docker run -d --name hdfs-minicluster -p 35200:35200 -p 35100:35100 -p 30000-30030:30000-30030 -e NUM_DATANODES=3 avs75alatau/minidfscluster:latest

⁠Notes

• The NUM_DATANODES environment variable is optional. If omitted, the cluster will start with one DataNode by default.
• Maximum NUM_DATANODES is 9.

⁠Ports

• NameNode RPC: 35200
• NameNode Web UI: 35100
• DataNode RPC, xferaddr, and http: Static range 30000-30030

⁠Usage

⁠Log Output

Wait for about 30 seconds after starting the container, then check the logs:

docker logs hdfs-minicluster

This will display connection information, including the configured ports.

⁠Connecting to HDFS

Use the full HDFS path to interact with the file system. For example:

hdfs dfs -ls hdfs://localhost:35200/

⁠Pre-Created Directories

The container automatically creates the following directories in HDFS:

• /tmp
• /user

⁠Example: Using with Apache Spark

To use this image with spark-shell for testing:

/opt/spark-3.5.1/bin/spark-shell --master local[*] \
  --conf spark.hadoop.fs.defaultFS=hdfs://localhost:35200

⁠Feedback and Contributions

For issues, improvements, or contributions, please contact GitHub⁠.

Happy Testing!