OpenJDK, Python, PySpark, Hadoop and Cron to run datascience scripts on a dockerized Spark Cluster
175
This docker image is intended to periodically (using cron) run pyspark scripts on a remote or dockerized spark cluster to create a private ETL pipeline. It is based on the official python docker image and uses fixed versions (see below) to ensure compatibility with the spark cluster. It is optimized for the bitnami spark cluster docker image (docker.io/bitnami/spark:3.2.1), which can be used to easily create a private spark cluster with a variable number of workers using cocker.
| Location | Description |
|---|---|
| /crontab | Standard linux crontab file. Use to define periodical runs of your python scripts |
| /requirements.txt | Standard requirements file to load additional python libraries. IS automatically processed during container startup. |
| /pyspark/pipelines | Folder for your pipeline scripts |
| /pyspark/logs | Logs from cronjobs or pipeline runs |
| /pyspark/datalake | Storage for delta lake parquets |
version: '3.9'
services:
spark-master:
image: "docker.io/bitnami/spark:3.2.1"
hostname: "master"
container_name: "spark-master"
networks:
- "sparkcluster"
environment:
- "SPARK_MODE=master"
- "SPARK_RPC_AUTHENTICATION_ENABLED=no"
- "SPARK_RPC_ENCRYPTION_ENABLED=no"
- "SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no"
- "SPARK_SSL_ENABLED=no"
ports:
- '8080:8080'
restart: always
spark-worker:
image: "docker.io/bitnami/spark:3.2.1"
container_name: "spark-worker-1"
networks:
- "sparkcluster"
depends_on:
- "spark-master"
environment:
- "SPARK_MODE=worker"
- "SPARK_MASTER_URL=spark://master:7077"
- "SPARK_WORKER_MEMORY=1G"
- "SPARK_WORKER_CORES=1"
- "SPARK_RPC_AUTHENTICATION_ENABLED=no"
- "SPARK_RPC_ENCRYPTION_ENABLED=no"
- "SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no"
- "SPARK_SSL_ENABLED=no"
ports:
- '8081:8081'
restart: always
pyspark:
image: "trasrik/pyspark"
container_name: "pyspark"
networks:
- "sparkcluster"
volumes:
- "./crontab:/crontab"
- "./requirements.txt:/requirements.txt"
- "./scripts:/scripts"
restart: always
networks:
sparkcluster:
external: true
Content type
Image
Digest
Size
1 GB
Last updated
about 4 years ago
docker pull trasrik/pyspark