holsi/spark_master

By holsi

•Updated over 4 years ago

Image

10K+

Overview Tags

holsi/spark_master repository overview

⁠Docker Spark Cluster Images

Spark version: 2.4.2 Scala version: 2.12.8 Hadoop version: 3.3.0

⁠Updates

20220107 - v0.2.1: Upgrade Hadoop version from 2.4.7 to 3.3.0 for integration with working S3, Delta lake version:0.6.1 and resolution of related exceptions

⁠Prerequisites

I assume knowledge of basic Docker commands such as run, build, etc. You will need to set up multiple machines with a cloud provider such as AWS or Azure or private cluster such as Swarm docker.

⁠Getting Started

To get started, pull the following three docker images

docker pull holsi/spark_master:0.2.1
docker pull holsi/spark_worker:0.2.1
docker pull holsi/spark_submit:0.2.1

Create a docker swarm using

docker swarm init

then attach the other machines you wish to be in the cluster to the docker swarm by copying and pasting the output from the above command.

Create an overlay network by running the following on one of the machines

docker network create -d overlay --attachable spark-net

On the machine you wish to be the master node of the Spark cluster run

docker run -it --name spark-master --network spark-net -p 4080:8080 holsi/spark_master
:0.0.5

On the machines you wish to be workers run

docker run -it --name spark-worker1 --network spark-net -p 4081:8081 -e MEMORY=8G -e
     CORES=4 holsi/spark_worker:0.0.5

Start a driver node by running

docker run -it --name spark-submit --network spark-net -p 4040:4040 holsi/spark_submit
:latest bash

You can now either submit files to spark using

$SPARK_HOME/bin/spark-submit [flags] file

Docker compose Swarm example:

version: '3.3'
services:
  spark-master:
    image: holsi/spark_master:0.2.1
    ports:
     - 4080:8080
    networks:
     - spark-net
    logging:
      driver: json-file
  spark-worker:
    image: holsi/spark_worker:0.2.1
    environment:
      CORES: '4'
      MEMORY: 8G
    ports:
     - 4081:8081
    networks:
     - spark-net
    logging:
      driver: json-file
    deploy:
      replicas: 2
  spark-submit:
    image: holsi/spark_submit:0.2.1
    environment:
      MASTER_CONTAINER_NAME: 'spark-master'
      SPARK_EXECUTOR_MEMORY: 4G
      SPARK_EXECUTOR_CORES: 2
    ports:
      - 4040:4040
    networks:
      - spark-net
    logging:
      driver: json-file
networks:
  spark-net:
    external: true

⁠Exceptions resolved

build 0.2.1:

Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/services/s3/model/MultiObjectDeleteException
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2602)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2567)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2663)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3378)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3411)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:160)
    at org.apache.spark.sql.delta.DeltaTableUtils$.isDeltaTable(DeltaTable.scala:93)
    at io.delta.tables.DeltaTable$.isDeltaTable(DeltaTable.scala:657)
    at com.enel.remote.control.deltalake.poc.utils.DLakeUtils$.initDeltaSchema(DLakeUtils.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3$.main(DeltaLakeOracleToS3.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3.main(DeltaLakeOracleToS3.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.services.s3.model.MultiObjectDeleteException


tYZXlbS5679DhrqsmOfIUMIS0zPtBaG67CQmlhY9OHY2a2P9WPkk257lOXdLadcwu/cAR8Uq5cc0ztkJkPEYyCZ6zMpwz8z3WTN1vnR4wI+YZ4gfOUniE4Hq+WQki3D1eWF3P1UFiqVmHFEkpzaUstzAnyif4IaOBjItLYaloYLc+/MyW0JY3OOYgjHBLyICRodjDs24Y2JPjRRv3Con/oE6aORsWxpG
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/SdkBaseException
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2602)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2567)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2663)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3378)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3411)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:160)
    at org.apache.spark.sql.delta.DeltaTableUtils$.isDeltaTable(DeltaTable.scala:93)
    at io.delta.tables.DeltaTable$.isDeltaTable(DeltaTable.scala:657)
    at com.enel.remote.control.deltalake.poc.utils.DLakeUtils$.initDeltaSchema(DLakeUtils.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3$.main(DeltaLakeOracleToS3.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3.main(DeltaLakeOracleToS3.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.SdkBaseException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)


    21/12/21 10:35:25 WARN DeltaLakeOracleToS3: getSessionToken: FwoGZXIvYXdzECwaDMfZ9Q05mFUpajft9yK0AWczYxmBMzejwbTX5Weqc89gTRcdtr74dgAAXq5nLlhl5YDSWkzMhH4fDNjraIivVYqVXFt6VLvNy7FifJx3PaZ2rBGC5FYlSUtw7mxhTDZqbTaxPu92D0leGNq5VXumFTZHJcqWABh6BmOrM0HrT29ogCMcdEfPybE4Fe6gTsKlQg7eI6dBtuI8eCzhMfzk1b+LPJhMxRU4wUWwlU/azb8pmxA49T7rVP6qoZT+WaSDxmuQiyjt24aOBjItosPybgVRp4vw1kHiTwh0GatHanjdb62mqrjxSsUIYUdQ9qH/insmkHD6Tmm9
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2602)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2567)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2663)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3378)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3411)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:160)
    at org.apache.spark.sql.delta.DeltaTableUtils$.isDeltaTable(DeltaTable.scala:93)
    at io.delta.tables.DeltaTable$.isDeltaTable(DeltaTable.scala:657)
    at com.enel.remote.control.deltalake.poc.utils.DLakeUtils$.initDeltaSchema(DLakeUtils.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3$.main(DeltaLakeOracleToS3.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3.main(DeltaLakeOracleToS3.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)





A1XKCNBUNNO3F/SNl5TdKGdQCZrVijIcBdnWGF3moEwL2pGwyHTo/Ocuvu4bBkrT7rTC6tfJZbUJhWjcVuiMdeBptY8aq3vLOZD8xdEt44JssbQMqC/OGWAsw8i8OUyNIFwRRQx4se9ieO8Z8g8tgffeuijN24aOBjItWg5oZeFLbKY8hiP+pbybRTBgTX6hqFAIlB5KxwchVmnAX5+QxUyZCj1z2DK2
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/event/ProgressListener
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2602)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2567)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2663)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3378)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3411)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.delta.DeltaTableUtils$.findDeltaTableRoot(DeltaTable.scala:160)
    at org.apache.spark.sql.delta.DeltaTableUtils$.isDeltaTable(DeltaTable.scala:93)
    at io.delta.tables.DeltaTable$.isDeltaTable(DeltaTable.scala:657)
    at com.enel.remote.control.deltalake.poc.utils.DLakeUtils$.initDeltaSchema(DLakeUtils.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3$.main(DeltaLakeOracleToS3.scala:16)
    at com.enel.remote.control.deltalake.poc.DeltaLakeOracleToS3.main(DeltaLakeOracleToS3.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.event.ProgressListener



Caused by: java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.5-8beca2a2-4ce6-461e-bca6-c0241d32c10c-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /tmp/snappy-1.0.5-8beca2a2-4ce6-461e-bca6-c0241d32c10c-libsnappyjava.so)
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:179)
    at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:154)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
    at org.apache.parquet.hadoop.codec.SnappyDecompressor.decompress(SnappyDecompressor.java:62)
    at org.apache.parquet.hadoop.codec.NonBlockedDecompressorStream.read(NonBlockedDecompressorStream.java:51)
    at java.io.DataInputStream.readFully(DataInputStream.java:195)

Tag summary

Recent tags

Content type

Image

Digest

Size

814.6 MB

Last updated

over 5 years ago

docker pull holsi/spark_master