johntorrestensor/pyspark-ml repository overview

Create Volume at "/opt/ml/code/", create main.py as entry file and then run any pyspark data processing job or train machine learning models.

This image is based on python version 3.10.6 and spar version 3.3.

Contains following libraries:

• numpy
• pandas
• scikit-learn
• hyperopt
• mlflow
• xgboost
• pyarrow
• pyspark

Arguments for docker run : train --> In order to run the main.py file.

To run main.py volume use following command. Go to your working directory where your main.py file is stored. Then use:

docker run -it --rm -p 8889:8889 -v "${PWD}":/opt/ml/code/ johntorrestensor/pyspark-ml train