Public Repository

Last pushed: a year ago
Short Description
Official container for REST Spark Job Server. Designed for Mesos or Hadoop.
Full Description

Spark-jobserver is available as a Docker container! This might be the easiest way to get started and deploy.

NOTE: In all the docker commands below, you need to append the tag, such as velvia/spark-jobserver:0.6.2.mesos-0.28.1.spark-1.6.1.

To get started:

docker run -d -p 8090:8090 velvia/spark-jobserver

This will start job server on port 8090 in a container, with H2 database and Mesos support, and expose that port to the host on which you run the container.

If you would like to debug job server using JMX / VisualVM etc., then also expose port 9999.

Configuration

By default, the container has an embedded Spark distro and runs using Spark local mode (local[4]).

To change the spark master the container runs against, set SPARK_MASTER when you start the container:

docker run -d -p 8090:8090 -e SPARK_MASTER=mesos://zk://mesos.master:5050 velvia/spark-jobserver

You can easily change the amount of memory job server uses with JOBSERVER_MEMORY, or replace the entire config job server uses at startup with JOBSERVER_CONFIG.

The standard way to replace the config is to derive a custom Docker image from the job server one by overwriting the default config at app/docker.conf. The Dockerfile would look like this:

from velvia/spark-jobserver
add /path/to/my/jobserver.conf /app/docker.conf

Similarly, to change the logging configuration, inherit from this container and overwrite /app/log4j-server.properties.

Jars / Passing Arguments to the Start Script

Any spark-submit arguments can be passed to the tail of the docker run command. A very common use of this is to add custom jars to your Spark job environment. For example, to add the Datastax Spark-Cassandra Connector to your job:

docker run -d -p 8090:8090 velvia/spark-jobserver --packages com.datastax.spark:spark-cassandra-connector_2.10:1.3.0-M1

Database, Persistence, Logs

Docker containers are usually stateless, but it wouldn't be very useful to have the jars and job config reset every time you had to kill and restart a container.

The job server docker image is configured to use H2 database by default and to write the database to a Docker volume at /database, which will be persisted between container restarts, and can even be shared amongst multiple job server containers on the same host. Note that in order to persist them to new containers, you need to create a local directory, something like this:

docker run -d -p 8090:8090 -v /opt/job-server-db:/database velvia/spark-jobserver

See the Docker Volumes Guide for more info.

Another option is to configure job server to persist metadata in PostGres, MySQL, or similar database. To do that, create a new config, pass it into the docker container as above using JOBSERVER_CONFIG and the /config volume, and point to your shared database, perhaps using --link to a PostGres or MySQL container.

Logging goes to stdout, as per standard Docker conventions. Therefore:

  • Use docker logs -f <containerHash> to follow logs
  • Use docker logs --tail=100 <containerHash> to list the last 100 lines
  • Use Docker logging drivers to redirect logs to syslog, SumoLogic, etc.
Docker Pull Command
Owner
velvia

Comments (9)
vvsh
3 months ago

Could you please push an image with spark-2.0-preview and latest mesos (1.2.0)?

xho22
7 months ago

@ kiambogo
This dockerfile link is invalid, who can support a new one?Thanks

javierporras
a year ago

Thanks for uploading 0.6.2, we needed it just yesterday. It saved our day, because we need to be able to change the Java version of the jars.

jxchanghe
2 years ago

When I add a argument --jars at the end of docker run, trigger a warning :
Warning: Local jar /spark/lib/JobServerDepentTest.jar does not exist, skipping.
and It result the job:
"errorClass": "java.lang.NoClassDefFoundError",
Could you help me resolve the problem?
Thanks.

yohanliyanage
2 years ago

+1 for Dockerfile / GitHub link

emmekappa
2 years ago

+1 for the Dockerfile source.

jcderr
2 years ago

Could you please provide the Dockerfile for this project, or a link to it on Github?

anldisr
2 years ago

This container is missing the Mesos library files and i cannot find the docker file to read how i can pass this to the start script. Can you provide either one?