Sign inSign up

archongum/spark

By archongum

•Updated about 3 years ago

https://github.com/archongum

Image
0

179

archongum/spark repository overview

⁠Overview

  • Can use as STS (Spark Thrift Server): Set entrypoint to /entrypoint-sts.sh
  • Can use as spark-executor on kubernetes: Default entrypoint
  • Also publish on Aliyun Registry, add prefix registry.cn-guangzhou.aliyuncs.com/, e.g. registry.cn-guangzhou.aliyuncs.com/archongum/spark:3.3.2-hadoop3

⁠Limitations

  • hadoop2 are not tested
  • spark 3.4.1 raise Error retrieving next row NullPointerException but still functional

⁠Addtional Jars

  • Support S3 location table
  • Support Iceberg - iceberg-spark-runtime-${SPARK_VERSION:0:3}_2.12-1.3.0.jar
  • Support Nessie - nessie-spark-extensions-${SPARK_VERSION:0:3}_2.12-0.64.0.jar
  • Support MySQL - mysql-connector-java-5.1.48.jar
  • Support Elasticsearch - elasticsearch-spark-20_2.12-7.17.9.jar
  • Support Mongo - mongo-spark-connector_2.12-3.0.2-assembly.jar
  • Support Auth (Simple or LDAP) - hiveserver-custom-auth-1.2-jar-with-dependencies.jar

⁠Quickstart

⁠Start STS
docker run -it --rm \
-p 10000:10000 \
-p 4040:4040 \
--name sts \
--entrypoint /entrypoint-sts.sh \
archongum/spark:3.3.2-hadoop3 \
--master local[2] \
--conf spark.hadoop.fs.s3a.access.key=minioadmin \
--conf spark.hadoop.fs.s3a.secret.key=minioadmin \
--conf spark.hadoop.fs.s3a.endpoint=http://127.0.0.1:9000 \
--conf spark.hadoop.fs.s3a.path.style.access=true \
--conf spark.hadoop.fs.s3a.connection.ssl.enabled=false \
--conf spark.sql.warehouse.dir=s3a://warehouse/hive \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.s3=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.s3.type=hadoop \
--conf spark.sql.catalog.s3.warehouse=s3a://warehouse/s3 \
--conf spark.sql.defaultCatalog=s3

If you need minio, run minio and create a bucket on minio website(port 9001) after minio started

docker run \
--rm \
-d \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin \
-p 9000:9000 \
-p 9001:9001 \
--name minio \
-v $(pwd)/data:/data \
minio/minio server /data --console-address ":9001"
⁠Run example

hive/spark native table

create table spark_catalog.default.tmp (id int, name string) using parquet;
insert into spark_catalog.default.tmp values (1, 'A');
select * from spark_catalog.default.tmp;
describe extended spark_catalog.default.tmp;

describe table result

Database	default
Table	tmp
Owner	root
Created Time	Tue Jul 04 10:37:58 CST 2023
Last Access	UNKNOWN
Created By	Spark 3.3.2
Type	MANAGED
Provider	parquet
Location	s3a://warehouse/hive/tmp
Serde Library	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat

iceberg table

create table s3.default.tmp (id int, name string) using parquet;
insert into s3.default.tmp values (1, 'A');
select * from s3.default.tmp;
describe extended s3.default.tmp;

describe table result

Name	s3.default.tmp
Location	s3a://warehouse/s3/default/tmp
Provider	iceberg
Owner	spark
Table Properties	[current-snapshot-id=707103439661033832,format=iceberg/parquet,format-version=1,write.format.default=parquet]

⁠Auth

Mount hive-site.xml to /etc/spark/hive-site.xml.

More info: https://github.com/archongum/hiveserver-custom-auth⁠

Tag summary

Content type

Image

Digest

sha256:9bf49bef6…

Size

533.9 MB

Last updated

about 3 years ago

docker pull archongum/spark:3.2.4-hadoop3