ubuntu 16.04 hadoop 3.1.1 spark 2.4.0
$ sudo docker run -dit --name [spark-container-name] --network [network-hadoop-container-is-connected] [image-name] /bin/bash
or
$ ./run_spark.sh [network-name] [spark-container-name]
if ./run_spark.sh fails with Permission denied error,
# chmod +x run_spark.sh
$ docker exec -it [spark-container-name] bash
# spark-shell --master yarn --deploy-mode client
or
# cd /usr/local/spark/
# ./run-sparkshell.sh
:quit or ctrl+D
Check if master and slaves are connected. Lists all running nodes.
# yarn node -list
if spark-shell gets stuck, check running applications and kill unnecessary ones.
# yarn application -list
# yarn application -kill [application-id]
install sbt which is a build tool.
sbt install
# echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
# sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
# sudo apt-get update
# sudo apt-get install sbt
if sudo apt-get update fails with E: The method driver /usr/lib/apt/methods/https could not be found.
# sudo apt-get install apt-transport-https
Content type
Image
Digest
Size
1.6 GB
Last updated
about 6 years ago
docker pull kmubigdata/ubuntu-spark