gusseppe/smartdeploy repository overview

⁠Docker Image for MLOps

⁠Image Description:

This Docker image is designed for data science and machine learning workflows, providing a robust and versatile environment for both Python and R language projects. It extends from the gusseppe/smartdeploy:base image.

⁠Key Features:

1. Apache Toree Installation: Integration with Apache Toree for Jupyter notebooks, enabling Scala, PySpark, SparkSQL, SparkR, and SQL interpreters.
2. Python Environment: A comprehensive set of Python packages tailored for machine learning and data analysis, including MLflow, FastAPI, Ray, and Boto3.
3. R Environment: A robust R language setup with packages like dplyr, tidyverse, caret, and xgboost, catering to a wide range of data science needs.
4. User and Permissions Management: Configured to operate with non-root user permissions, ensuring security and good practices in container deployment.

⁠Dockerfile Configuration:

• Base Image: gusseppe/smartdeploy:base
• User: Root initially, then switches to a non-root user.
• Apache Toree Setup: Installed and configured for multiple interpreters with Spark integration.
• Python Package Management: Custom requirements from requirements.txt are installed after removing ortools. Additionally, unwanted caches and files are cleaned up post-installation.
• R Package Management: R language dependencies are managed using mamba for efficient installation and cleanup, as listed in r_requirements.txt.

⁠Python Packages (requirements.txt):

• Core packages for SmartDeploy, BigData cluster, and user-specific needs.
• Notable inclusions: mlflow==1.30.0, deepchecks==0.9.2, fastapi==0.88.0, boto3==1.26.95, imbalanced-learn>=0.9.1, psycopg2-binary==2.9.9.
• Release dates for each package are mentioned for clarity.

⁠R Libraries (r_requirements.txt):

• A selection of libraries focusing on data manipulation, machine learning, and visualization.
• Includes r-data.table, r-mlmetrics, r-dplyr, r-tidyverse, r-caret, r-xgboost, and more.

⁠Usage and Applications:

This Docker image is ideal for data scientists and machine learning engineers who require a comprehensive, ready-to-use environment for both Python and R projects. It's particularly useful for projects involving complex data processing, machine learning model development, and interactive data exploration.