Sign inSign up

jacintoarias/sparkdev

By jacintoarias

•Updated over 7 years ago

Spark Development with scala and sbt

Image
0

252

jacintoarias/sparkdev repository overview

Docker Automated buil Docker Build Statu

⁠docker-sparkdev

This is a base base image for developing Apache Spark applications.

⁠Docker Hub

One can easily obtain the latest image using:

docker pull jacintoarias/docker-sparkdev:latest

⁠Usage

⁠SBT

The following command starts the sbt prompt, mounts your working directory as project root and builds the dependencies cache on the local .ivy folder.

docker run \
    --rm \
    -v $(pwd):/home/work/project \
    -v $(pwd)/.ivy:/sbtlib \
    jacintoarias/docker-sparkdev \
    sbt -Dsbt.global.staging=./.staging package \
    -ivy ./.ivy \
⁠Spark

Runs an application withing a jar file in your local sbt build

docker run \
    --rm \
    -v $(pwd):/home/work/project \
    jacintoarias/docker-sparkdev \
    spark-submit \
    --class <YOUR MAIN CLASS> \
    target/scala-2.11/<YOUR JAR FILE> \
    <ARGUMENTS>

Runs spark shell or any other program

docker run \
    --rm \
    -v $(pwd):/home/work/project \
    jacintoarias/docker-sparkdev \
    spark-shell

Tag summary

Content type

Image

Digest

Size

617.7 MB

Last updated

over 7 years ago

docker pull jacintoarias/sparkdev