unicorn666/bigdata_ml_env repository overview

⁠unicorn666/bigdata_ml_env

⁠Overview

unicorn666/bigdata_ml_env v1.0.0 is an all-in-one Docker image for big data processing, analytics, and machine learning.
It provides a pseudo-distributed Hadoop cluster (HDFS + YARN) along with Hive, Pig, Mahout, Spark, and a Python ML environment ready for development, experimentation, and learning.

⁠Included Tools

⁠Big Data Stack

• Hadoop 3.3.6 (HDFS, NameNode, DataNode, YARN)
• Hive 2.3.9 (Data warehouse)
• Pig 0.17.0 (Dataflow scripting)
• Mahout 14.1 (Scalable ML on Hadoop)
• Scala (for Spark and Hadoop jobs)

⁠Python & ML Environment

• Python 3.10.17 via pyenv
• PySpark 3.5.0
• Data Analysis & ML Libraries:
  • pandas 2.1.4
  • numpy 1.24.3
  • scikit-learn 1.3.2
• Utilities:
  • py4j 0.10.9.7
  • pyyaml 6.0.1
  • python-dateutil 2.8.2
  • psutil 5.9.6
• Web UI:
  • Flask 2.3.3
  • Gunicorn 21.2.0

⁠System Utilities

• openjdk-11-jdk
• SSH, curl, wget, git, nano, net-tools, iputils-ping
• build-essential and libraries required for Python compilation

⁠Pyenv & Python Version Management

This image uses pyenv to manage Python versions.

• Default Python: 3.10.17
• You can install other Python versions inside the container easily:

# List available Python versions
pyenv install --list

# Install a new version
pyenv install 3.11.5

# Set global default
pyenv global 3.11.5

# Use a specific version locally in a project
pyenv local 3.9.18

• The Python binaries installed via pyenv are available in your shell automatically.

⁠Setup & Installation

⁠Prerequisites

• Docker installed

  • Linux: sudo apt install docker.io or Docker Engine
  • macOS / Windows: Docker Desktop⁠

• Docker Hub account (to pull the image)

⁠Pull the Image

docker pull unicorn666/bigdata_ml_env:v1.0.0

⁠Running the Container

⁠Linux / macOS

docker run -it \
  -v $(pwd)/uploads:/app/uploads \
  -v $(pwd)/output:/app/output \
  -v $(pwd)/logs:/app/logs \
  -v $(pwd)/src:/app/src \
  unicorn666/bigdata_ml_env:v1.0.0 shell

⁠Windows (PowerShell)

docker run -it `
  -v ${PWD}/uploads:/app/uploads `
  -v ${PWD}/output:/app/output `
  -v ${PWD}/logs:/app/logs `
  -v ${PWD}/src:/app/src `
  unicorn666/bigdata_ml_env:v1.0.0 shell

⁠🔹 Windows (Command Prompt)

docker run -it --rm ^
  -v %cd%/uploads:/app/uploads ^
  -v %cd%/output:/app/output ^
  -v %cd%/logs:/app/logs ^
  -v %cd%/src:/app/src ^
  unicorn666/bigdata_ml_env:v1.0.0 shell

Notes:

• Starts Hadoop/YARN automatically.
• Waits for HDFS and ResourceManager to be ready.
• Drops into an interactive Bash shell.

⁠Accessing Hadoop & Spark

• HDFS Web UI: http://localhost:9870
• YARN ResourceManager: http://localhost:8088
• Run Spark Jobs inside /app directory:

spark-submit /app/src/recommendation_system.py

⁠Persistent Storage

• Mount local directories to persist data:

docker run -v /local/uploads:/app/uploads \
           -v /local/output:/app/output \
           -v /local/logs:/app/logs \
           unicorn666/bigdata_ml_env:v1.0.0 shell

• Hadoop HDFS can also be persisted:

docker run -v /local/hdfs:/hdfs unicorn666/bigdata_ml_env:v1.0.0 shell

⁠Logging

• HDFS/YARN daemon logs are available in /hdfs/logs inside the container:

tail -f /hdfs/logs/namenode.log
tail -f /hdfs/logs/datanode.log
tail -f /hdfs/logs/resourcemanager.log

⁠Usage Examples

⁠PySpark Job

spark-submit /app/src/recommendation_system.py

⁠Hive CLI

hive

⁠Pig Script

pig /app/src/sample.pig

⁠Start Python Web UI

cd /app/src/web_ui
gunicorn -w 2 -k gthread --threads 4 -b 0.0.0.0:5000 app:app

⁠Notes for New Users

• HDFS NameNode is formatted automatically on first run if needed.
• Python environment is pre-configured via pyenv.
• All Hadoop, Hive, Pig, Mahout, Spark, and Python dependencies are ready-to-use.
• /app is the working directory; mount your projects here for persistence.

Enjoy a fully configured Big Data & ML development environment!