bigdatateam/spark-course2 repository overview

NB: In course02_week05-Demo.ipynb' in line 'sparkSession = SparkSession.builder.enableHiveSupport().master("local [2]").getOrCreate() the "local [2]" string should be without space thus "local[2]"is right.

To run container you need to install docker and then execute:

docker run -it --rm -v YYYYYY:/home/jovyan/dir_on_host -p XXXX:8888 bigdatateam/spark-course2

-it allows to see the information from the container

--rm means that the container will be deleted after exiting/terminating

-v means that the files from the Jupyter of the container will be transferred to the local file system directory YYYYYY

YYYYYY - a local path, for example, for Linux $(pwd)/bigdata, $(pwd) - the path to the current directory

(--rm and -v YYYYYY:/home/jovyan/dir_on_host can be safely removed if it is not needed.)

-p XXXX:8888 allows to connect to the Jupyter of the container on 127.0.0.1:XXXX or localhost:XXXX, XXXX - a port number of the local system

-p AAAA:4040 allows to connect to the Spark web UI of the container on 127.0.0.1:AAAA or localhost:AAAA, AAAA - a port number of the local system (if you do not need this UI, delete this option)

For more options see the "docker run" documentation page⁠.