Welcome to the Spark 4 Devs course by Data Fellas.
You can here pull the image that will be used the whole three days!
Pull the docker image
docker pull datafellas/spark-training
Run the image
docker run -it --rm --name spark-training -p 9000:9000 -p 4040-4060:4040-4060 --volume $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent datafellas/spark-training /bin/bash
Note: we mapped all ports from
4060, 20 opened notebooks should be enough...
Warn: the port
4040-4060 mapping might not work depending on your docker version, hence must be split manually like for
Starting the training
bash, you need to launch
Run Image: SSH-AGENT
This is required to privately access the last git changes using the
git:// protocol that will use SSH.
- your ssh key must be known by the training git repo
- your host must add this key in a ssh-agent.
Some problem may happen with ssh-agen, for instance of ec2 we can have:
Could not open a connection to your authentication agent.
Here is the SO to solve it: http://serverfault.com/questions/325402/issues-with-ssh-add.
You'll need to execute
eval ssh-agent then
exec ssh-agent bash and finally