liuedy/spark repository overview

Play Spark with K8S

⁠Simple Steps:

⁠Create K8S cluster with https://kubespray.io/⁠

// Get your cluster info.

sudo /usr/local/bin/kubectl cluster-info

Kubernetes master is running at https://192.168.1.10:6443⁠

// Create SA for Spark

sudo /usr/local/bin/kubectl create serviceaccount spark

// Create SA roles.

sudo /usr/local/bin/kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

⁠Build you own docker image with spark or use this one.

curl -O https://archive.apache.org/dist/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz⁠

tar xf spark-2.4.1-bin-hadoop2.7.tgz

cd spark-2.4.1-bin-hadoop2.7/

sudo docker build -t spark:2.4.1 -f kubernetes/dockerfiles/spark/Dockerfile .

⁠Run you test program.

// Submit Spark PI example, replace your own image name or use my build.

sudo bin/spark-submit --master k8s://https://192.168.1.10:6443⁠ --deploy-mode cluster --name spark-pi --class org.apache.spark.examples.SparkPi --conf spark.executor.instances=2 --conf spark.kubernetes.container.image=liuedy/spark:2.4.1 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:////opt/spark/examples/jars/spark-examples_2.11-2.4.1.jar

// View the logs for the run 19/04/07 01:04:27 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 2.826370 s

Pi is roughly 3.1434157170785855

19/04/07 01:04:27 INFO SparkUI: Stopped Spark web UI at http://spark-pi-1554030869270-driver-svc.default.svc:4040⁠

19/04/07 01:04:27 INFO KubernetesClusterSchedulerBackend: Shutting down all executors

⁠Refer: http://spark.apache.org/docs/latest/running-on-kubernetes.html⁠