datamechanics/spark repository overview

⁠Optimized Docker images for Apache Spark⁠

Brought to you by Data Mechanics⁠, the cloud-native Spark platform for data engineers. Read our blog post⁠ for more background on our motivation and this tutorial⁠ for help on getting started.

⁠What's a Docker image for Spark ?

When you run Spark on Kubernetes⁠, the Spark driver and executors are Docker containers. These containers use an image specifically built for Spark. This image contains the Spark distribution itself, you can also add your application code and dependencies in it.

It can also be used to run Spark locally. For example you can run Spark in a driver-only mode (in a single container), or run Spark on Kubernetes on a local minikube cluster.

⁠What's in these images?

They contain the Spark distribution itself - from open-source code, without any modifications.

They come built-in with connectors to common data sources:

• AWS S3 ( s3a:// )
• Google Cloud Storage ( gs:// )
• Azure Blob Storage ( wasbs:// )
• Azure Datalake generation 1 ( adls:// )
• Azure Datalake generation 2 ( abfss:// )
• Snowflake⁠
• Delta Lake⁠
• Hadoop cloud magic committer for AWS

They also come built-in with Python & PySpark support, as well as pip and conda so that it's easy to to install additional Python packages. (If you don't need PySpark, you can use the lighter image with the tag prefix jvm-only)

Finally, they contain a combination of versions from the components listed below. Note that not all the combination exist.

⁠Tag Formats Explained

For each image we provide multiple tags. Let's take the example of an image, with Spark 3.1.2, Java 8, Scala 2.12, Hadoop 3.2.0 and python 3.8. It has 3 different tags:

• A long-form tag: 3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest where all versions are exposed.
• A short form tag, with the patch version of Spark specified: datamechanics/spark:3.1.2-latest: only the Spark distribution version is set and all other dependencies are set to the latest compatible version. We allow ourselves to upgrade the version of a dependency if a new, compatible version is released. For example, we may upgrade 3.1.2-latest to Hadoop 3.3.0 once it is compatible with Spark 3.1.2.
• A short form tag, with the minor version of Spark specified: datamechanics/spark:3.1-latest: that is because Spark 3.1.2 is the latest patch version of Spark 3.1. If Spark 3.1.3 is released, tag 3.1-latest will point to 3.1.3-latest.

In most cases, we encourage using our short-form tags like datamechanics/spark:3.1-latest or datamechanics/spark:3.1.2-latest.

We recommend to use a long-form only if you need a specific combination. For instance, you may require a specific combination of versions when migrating an existing Scala or Java project to Spark on Kubernetes. On the other hand, new JVM projects and PySpark projects should work just fine with short-form tags!

⁠Tag Revisions

By default, we recommend to use image tags ending with -latest. This will always contain the most up to date version of the image for a specific flavor.

When pushing bug fixes, (e.g. a broken spark.jar.packages), or upgrading connectors versions, we'll publish a new revision of this image, with a suffix -dmXX where XX is the image version number.

If you want to use a specific version, use the appropriate image tag, so to use the revision number 16 of the pyspark image with Spark 3.1.2, use datamechanics/spark:3.1.2-dm17

The release notes⁠ of the versions we publish are available on our website.

⁠Where are the Dockerfiles?

The images in this repository are built with Earthly⁠, not with Docker. There is in fact a single Earthfile⁠ for the whole fleet of images.

This Earthfile along with the build context are automatically generated. We are working to make this code open-source, stay tuned!

⁠List of supported tags

All tags on a single line are aliases of the same image.

⁠spark

This image family includes PySpark and a python version.

Spark 3

• 3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-dm17, 3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-latest, 3.2-dm17, 3.2-latest, 3.2.1-dm17, 3.2.1-latest,
• 3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-dm17, 3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-latest, 3.2.0-dm17, 3.2.0-latest,
• 3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.1-dm17, 3.1-latest, 3.1.3-dm17, 3.1.3-latest,
• 3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.1.2-dm17, 3.1.2-latest,
• 3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.1.1-dm17, 3.1.1-latest,
• 3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.0-dm17, 3.0-latest, 3.0.3-dm17, 3.0.3-latest,
• 3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.0.2-dm17, 3.0.2-latest,
• 3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.0.1-dm17, 3.0.1-latest,
• 3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, 3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, 3.0.0-dm17, 3.0.0-latest,
• 3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-dm17, 3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-latest,
• 3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-dm17, 3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-latest,
• 3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• 3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• 3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• 3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• 3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• 3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, 3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,

Spark 2

• 2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, 2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, 2.4.7-dm17, 2.4.7-latest, 2.4-dm17, 2.4-latest,
• 2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, 2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, 2.4.6-dm17, 2.4.6-latest,
• 2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, 2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, 2.4.5-dm17, 2.4.5-latest,
• 2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, 2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,
• 2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, 2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,
• 2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, 2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,

⁠spark:jvm-only

This lighter variant does not embed PySpark.

Spark 3

• jvm-only-3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-latest, jvm-only-3.2-dm17, jvm-only-3.2-latest, jvm-only-3.2.1-dm17, jvm-only-3.2.1-latest,
• jvm-only-3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-latest, jvm-only-3.2.0-dm17, jvm-only-3.2.0-latest,
• jvm-only-3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.1-dm17, jvm-only-3.1-latest, jvm-only-3.1.3-dm17, jvm-only-3.1.3-latest,
• jvm-only-3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.1.2-dm17, jvm-only-3.1.2-latest,
• jvm-only-3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.1.1-dm17, jvm-only-3.1.1-latest,
• jvm-only-3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.0-dm17, jvm-only-3.0-latest, jvm-only-3.0.3-dm17, jvm-only-3.0.3-latest,
• jvm-only-3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.0.2-dm17, jvm-only-3.0.2-latest,
• jvm-only-3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.0.1-dm17, jvm-only-3.0.1-latest,
• jvm-only-3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17, jvm-only-3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-latest, jvm-only-3.0.0-dm17, jvm-only-3.0.0-latest,
• jvm-only-3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,
• jvm-only-3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-dm17, jvm-only-3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-latest,

Spark 2

• jvm-only-2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, jvm-only-2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, jvm-only-2.4.7-dm17, jvm-only-2.4.7-latest, jvm-only-2.4-dm17, jvm-only-2.4-latest,
• jvm-only-2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, jvm-only-2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, jvm-only-2.4.6-dm17, jvm-only-2.4.6-latest,
• jvm-only-2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-dm17, jvm-only-2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-latest, jvm-only-2.4.5-dm17, jvm-only-2.4.5-latest,
• jvm-only-2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, jvm-only-2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,
• jvm-only-2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, jvm-only-2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,
• jvm-only-2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-dm17, jvm-only-2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-latest,

⁠Tutorial

See this Tutorial⁠ for help on getting started with these images, as well as this github repository⁠ for code examples.