vastdataorg/spark-vast repository overview

⁠VAST DATA – Spark Docker Image with VastDB Plugin

⁠Introduction

• VAST DATA provides a Docker image that includes the Spark runtime, preconfigured with the VASTDB connector, enabling seamless querying and ingestion into VastDB and Vast-S3 data-sources.

• The Docker image is hosted on Docker-Hub and is ready-to-use for launching Spark Master, Worker, and Client container nodes.

  • This allows you to run end-to-end Spark jobs directly on VastDB / VAST S3

• Without specifying the container image tag, it defaults to latest, but a number of any released Spark image by VAST can be used, by specifying a TAG.

vastdataorg/spark-vast:TAG

⁠Running Spark with VAST DB Docker

⁠Prerequisites

• You must have the Docker engine installed on your host machine.

  • You can download it from the official Docker website or use your OS’s native package manager.

• You also need to have:

  • A valid spark-defaults.conf file with all VAST plugin configurations (see below).
  • spark-defaults.conf should be mounted under /opt/spark/conf/ path inside the container.

⁠Configuration File: spark-defaults.conf

• To make Spark connect and query VastDB through the plugin, the user needs to configure a few parameters:
  • spark.ndb.endpoint : VAST single VIP or VIP-pool DNS name → Network Access through Virtual IPs⁠
  • spark.ndb.access_key_id : S3 Access → Managing S3 User Access⁠
  • spark.ndb.secret_access_key : S3 Secret
  • spark.ndb.data_endpoints : List of all desired VIP's within the same VIP-pool, in order to distribute the query acceleration to multiple CNODES. (Specify multiple IP's with commas; e.g: http://vast-vip1,http://vast-vip2 ..)

To connect Spark with your VastDB cluster, the following configuration must be included in your spark-defaults.conf:

spark.ndb.endpoint=http://VAST-VIP
spark.ndb.data_endpoints=http://VAST-VIP1,http://VAST-VIP2
spark.ndb.access_key_id=USER-ACCESS-KEY
spark.ndb.secret_access_key=USER-SECRET-KEY

# tuning parameters
spark.ndb.num_of_splits=256
spark.ndb.num_of_sub_splits=8
spark.ndb.rowgroups_per_subsplit=1
spark.ndb.query_data_rows_per_split=4000000
spark.ndb.retry_max_count=3
spark.ndb.retry_sleep_duration=1
spark.ndb.parallel_import=true
spark.ndb.dynamic_filter_compaction_threshold=100
spark.ndb.dynamic_filtering_wait_timeout=2
spark.sql.catalog.ndb=spark.sql.catalog.ndb.VastCatalog
spark.sql.extensions=ndb.NDBSparkSessionExtension

⁠Running Spark Cluster

⁠Step 1: Create a Docker Network

To allow Spark containers to discover and communicate with each other, first create a user-defined Docker network:

docker network create spark-network

⁠Step 2: Start Spark Master

Launch the Spark Master container and mount your custom configuration:

docker run -d --name spark-master \
  --network spark-network \
  -p 8080:8080 -p 7077:7077 \
  -v ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf \
  vastdataorg/spark-vast \
  /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master

• Master WebUI available on http://localhost:8080⁠
• Port 7077 is used by workers to register

⁠Step 3: Start Spark Worker

Start the Worker container, providing access to spark-defaults.conf:

docker run -d --name spark-worker \
  --network spark-network \
  -p 8081:8081 \
  -v ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf \
  vastdataorg/spark-vast \
  /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077

• Worker WebUI available on http://localhost:8081⁠

⁠Step 4: Start Spark Client

The client container is used to submit Spark jobs or run Spark SQL.
Keep it running in background.

docker run -d --name spark-client \
  --network spark-network \
  -p 4040:4040 \
  -v ./spark-defaults.conf:/opt/spark/conf/spark-defaults.conf \
  vastdataorg/spark-vast \
  /bin/sleep infinity

⁠Running Spark-SQL

Running Spark SQL – Query VastDB Tables

⁠Interactively query existing tables on VastDB:

docker exec -it spark-client /opt/spark/bin/spark-sql --master spark://spark-master:7077 \
  --driver-class-path $(echo '/opt/spark/vast/*.jar' | tr ' ' ':') \
  --conf spark.executor.extraClassPath=$(echo '/opt/spark/vast/*.jar' | tr ' ' ':') \
  --jars $(echo '/opt/spark/vast/*.jar' | tr ' ' ',') \
  --conf spark.executor.userClassPathFirst=true \
  --conf spark.driver.userClassPathFirst=true \
  --conf spark.driver.maxResultSize=4g \
  --conf spark.driver.memory=16g \
  --conf spark.executor.cores=8 \
  --conf spark.executor.memory=8g \
  --conf spark.sql.catalogImplementation=in-memory

• Driver WebUI available on http://localhost:4040⁠

Once in the Spark SQL prompt:

SELECT * FROM `ndb`.`vastdb`.`schema1`.`table1` LIMIT 10;

⁠Cleaning Up

To stop and remove all Spark containers:

docker stop spark-client spark-worker spark-master
docker rm spark-client spark-worker spark-master
docker network rm spark-network

⁠Additional Notes

The spark-vast Docker image comes preloaded with all required VAST JAR's files.

For advanced tuning, refer to the Apache Spark documentation: https://spark.apache.org/docs/latest/tuning.html⁠