datagovsg/python-spark repository overview

⁠python-spark

This image is based off the python:2.7⁠ image and contains Hadoop, Sqoop and Spark binaries.

Useful packages included for Spark and Sqoop:

• Spark-csv
• Spark-avro
• MySQL JDBC driver
• PostgreSQL JDBC driver
• MS SQL JDBC driver

⁠Logging in to Docker Hub Container Registry

Credentials are managed by docker login. Login with docker login

⁠Building and pushing new images

Assuming that the Python version we want to build is 2.7, the Hadoop version is 2.7.3 and the Spark version is 2.0.2, then we can build the images as such:

docker build -t datagovsg/python-spark:2.7-2.0 --build-arg HADOOP_VERSION=2.7.3 --build-arg SPARK_VERSION=2.0.2 .
docker push datagovsg/python-spark:2.7-2.0

⁠Supported tags and respective Dockerfile links

• latest: Python 2.7 with Spark 1.6.1 (Dockerfile⁠)
• 2.7-1.6.1: Python 2.7 with Spark 1.6.1 (Dockerfile⁠)
• 2.7-2.0: Python 2.7 with Spark 2.0 (Dockerfile⁠)