wiseupdata/pyspark repository overview

⁠

⁠

⁠

⁠

⁠

Pyspark docker images! 🚀️

⁠Simple to use ❤️

docker run -it wiseupdata/pyspark bash

⁠Summary 📃

• Versions info 🐍

  • 3.4.0⁠

• Utils 🛠️

  • Utils⁠

  • Create your own image to dockerhub⁠

  • References⁠

  • Dockerhub versions⁠

⁠Version 3.4.0 ✨️

• Pyspark 3.4.0

• Ubuntu 22.04.2 LTS end of life April 2032 - Official

• Python 3.10 - Official

• OpenJDK8 - Official⁠

• Kubernetes Ready and Tested!

• Userspark Restrict user NO sudo

• UserUbuntu sudo

• Simple shell, type:

  • pyspark and have fun with Python and Spark! ❤️
  • spark and have fun with Scala and Spark! 🫀
  • python import the pyspark and have fun!

• Made with A.I. contribution 🤖

• Dockerfile⁠

  This setup it's the same for the tags: 3.4.0 and 3.4.0-ubuntu-22.04

🚀 This docker was built on top of Apache, to be resilient and ready for debug, so it's light enough! If you need something different try the pure Apache docker!

Example with specific version 💻:

docker run -it wiseupdata/pyspark:3.4.0 bash
docker run -it wiseupdata/pyspark:3.4.0-ubuntu-22.04 bash

⁠Inside made easy! 😍

⁠Type pyspark and enjoy Python with Spark!

pyspark

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.4.0
      /_/

Using Python version 3.10.11 (main, May  2 2023 14:04:10)

⁠Type spark and enjoy it Scala with Spark!

pyspark

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.0
      /_/
         
Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 1.8.0_372)

⁠Info 🔎

• Pyspark 3.4.0 it's official from spark.apache.org⁠
  • To use pyspark 3.4.0 just type: pyspark

⁠Tests 🧪

⁠Pyspark

Type python or pyspark and test!

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TestPySpark").getOrCreate()
data = [("Alice", 1), ("Bob", 2), ("Charlie", 3)]
df = spark.createDataFrame(data, ["Name", "Age"])
df.show()

spark.stop()
exit()

⁠Scala

Type spark and test!

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder.appName("Spark Scala Test").getOrCreate()
val df = Seq((1, "John"), (2, "Jane"), (3, "Jim")).toDF("id", "name")
df.show()

spark.stop()
:quit

⁠Create your own image to dockerhub 🥳

docker build -t pyspark ./versions/3.4.0 --no-cache

docker run -it --rm pyspark bash

docker run --name pyspark -d --rm pyspark bash run
docker exec -it pyspark bash

# Exit and kill
exit
docker rm pyspark -f

docker login -u wiseupdata

docker tag pyspark wiseupdata/pyspark
docker tag pyspark wiseupdata/pyspark:3.4.0
docker tag pyspark wiseupdata/pyspark:3.4.0-ubuntu-22.04
docker tag pyspark wiseupdata/pyspark:3.4.0-ubuntu-22.04-jdk8

docker push wiseupdata/pyspark
docker push wiseupdata/pyspark:3.4.0
docker push wiseupdata/pyspark:3.4.0-ubuntu-22.04
docker push wiseupdata/pyspark:3.4.0-ubuntu-22.04-jdk8

docker run -it wiseupdata/pyspark:3.4.0 bash

⁠Utils 🎁

docker ps -a

docker rm $(docker ps -a -q) -f

docker image ls -a

docker image rm pyspark -f

docker image rm  $(docker image ls -a ) -f

docker run -it --entrypoint /bin/bash pyspark

⁠References 🌍 🗄️

1. GitHub Repository⁠
2. WiseUpData⁠
3. Emojis⁠
4. OpenJDK⁠
5. Apache Dockerfile⁠
6. Apache Dockerfile 2⁠
7. Kubectl⁠

⁠Maintainer 🤗 👨‍💻

Silvio Liborio

📧 [email protected]⁠

silvio-de-melo-liborio ⁠

back to top ⏫ ⁠