Tag: v3.1.1
Clean spark image with jre 8 and root user
Step 1:
➜ tar -zxf ~/Downloads/spark-3.1.1-bin-hadoop3.2.tgz
➜ cd spark-3.1.1-bin-hadoop3.2
➜ vi kubernetes/dockerfiles/spark/Dockerfile
####
#ARG java_image_tag=11-jre-slim
ARG java_image_tag=8-jre-slim
......
# Specify the User that the actual main process will run as
#USER ${spark_uid}
USER root
####
Step 2:
➜ docker build -t echang0929/dswspark-runner:v3.1.1 -f kubernetes/dockerfiles/spark/Dockerfile .
➜ docker push echang0929/dswspark-runner:v3.1.1
Tag: v3.0.1-2
merged all stocator, schema-registry and avro-serializer jars and their dependency jars into spark jars/ folder, and removed conflict low version jars.
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-schema-registry</artifactId>
<version>5.4.1</version>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>5.4.1</version>
</dependency>
<dependency>
<groupId>com.ibm.stocator</groupId>
<artifactId>stocator</artifactId>
<version>1.1.3</version>
</dependency>
Use the command:
mvn dependency:copy-dependencies
to download all dependency jars.
ls spark-3.0.1-bin-hadoop3.2/jars | sort > jarlist.txt
for ii in $(cat toremove.txt); do mv spark-3.0.1-bin-hadoop3.2/jars/$ii bkpjars/; done
Tag: v3.0.1-3
To match compatibility between kubernetes-client with our K8s version 1.19.5, upgraded all kubernetes related jar into 4.13.1:
<dependencies>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
<version>4.13.1</version>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model</artifactId>
<version>4.13.1</version>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-model-common</artifactId>
<version>4.13.1</version>
</dependency>
</dependencies>
And upgraded all jackson related jars into 2.11.2
Content type
Image
Digest
Size
450.3 MB
Last updated
almost 5 years ago
docker pull echang0929/dswspark-runner:v3.1.2