truebyteinnovationllp/spark repository overview

⁠spark

Minimal, security-hardened Apache Spark 4.1 (PySpark) on Debian 13 (trixie). Two-stage build (download + GPG/SHA512 verify → clean runtime), runs as a non-root user, multi-arch (amd64/arm64), built to follow the Docker Hardened Images⁠ Spark conventions (Debian 13 base, non-root uid 65532, /opt/spark layout, JRE under /opt/java/openjdk).

⁠Highlights

• 🔒 Hardened — non-root (uid 65532, the DHI convention), compatible with a read-only root filesystem and restricted PodSecurity. Roots-drop uses setpriv (util-linux, C) — no gosu, so no Go runtime / Go-stdlib CVE surface.
• 📌 Reproducible — Debian base pinned by digest; Spark pinned by version and SHA-512 and verified against the release manager's detached GPG signature.
• 🩹 Patched JARs — patch-jars.sh upgrades the cleanly-swappable bundled libraries to fixed releases (Netty 4.2.15, Jackson 2.21.4, Derby 10.17, ZooKeeper 3.9.5), each verified against Maven Central's .sha1. The optional Spark Connect REPL bundle (which shaded a vulnerable Netty) is removed.
• ☕ Java 21 — Debian's openjdk-21-jre-headless (apt-patched, no AWT/GUI surface).
• 🐍 PySpark — Python 3 (python3 + pip); PYSPARK_PYTHON=python3 preset.
• 🧮 Scala 2.13 — Spark 4.1.x / Hadoop 3 distribution.
• 🪶 Small — tarball fetched + verified in a builder stage; the final image carries runtime deps only (gnupg/wget never reach it).
• 🧱 Multi-arch — linux/amd64, linux/arm64.
• 🚀 K8s-ready — ships the upstream Spark-on-Kubernetes entrypoint (driver / executor / pass-through modes) and the decommission helper at /opt/decom.sh.

⁠Tags

• latest — newest build.
• 4.1.2-python3-java21 — Spark 4.1.2, Scala 2.13, Java 21, Python 3, Debian 13.

⁠Image facts

⁠Quick start

⁠PySpark shell

docker run --rm -it truebyteinnovationllp/spark:4.1.2-python3-java21 \
  /opt/spark/bin/pyspark

⁠Run the SparkPi example (local mode)

docker run --rm truebyteinnovationllp/spark:4.1.2-python3-java21 \
  /opt/spark/bin/spark-submit \
  --master 'local[*]' \
  --class org.apache.spark.examples.SparkPi \
  /opt/spark/examples/jars/spark-examples.jar 100

⁠Submit a PySpark script

docker run --rm -v "$PWD":/work \
  truebyteinnovationllp/spark:4.1.2-python3-java21 \
  /opt/spark/bin/spark-submit --master 'local[*]' /work/job.py

⁠Spark on Kubernetes

The entrypoint understands the driver and executor commands used by spark-submit --master k8s://…. Point spark.kubernetes.container.image at this image; it runs unprivileged by default and switches to the spark user via gosu if a pod starts as root.

⁠Hardening notes

• Immutable runtime — add --read-only (mount a writable volume at /opt/spark/work-dir if your job writes there).
• Drop capabilities — --cap-drop=ALL; the image needs none.
• No build tooling — wget/gnupg2 exist only in the builder stage.
• Supply chain — every release is pushed multi-arch with SBOM + provenance attestations and signed with Cosign (see the repo CI). Verify with:

  cosign verify --key cosign.pub truebyteinnovationllp/spark:4.1.2-python3-java21
  

  Copy

⁠Security posture (CVEs)

Unlike the from-source images in this repo, Spark ships a large, fixed set of prebuilt upstream JARs, so literal "zero CVE" is not achievable by patching a tarball — the upstream Docker Hardened Image reaches zero only by rebuilding Spark from source against patched dependencies. This image instead does the honest, maintainable thing:

1. Upgrade every cleanly-swappable standalone JAR to a fixed release (patch-jars.sh: Netty 4.2.15, Jackson 2.21.4, Derby 10.17, ZooKeeper 3.9.5).
2. Remove dead/optional vulnerable surface (Spark Connect REPL bundle with its shaded Netty; macOS/Windows Netty native jars; gosu/Go).
3. Triage the irreducible remainder in .trivyignore⁠ and spark.openvex.json⁠, each with a written justification.

The CI gate is Trivy honouring .trivyignore, so the build still fails on any new, undocumented HIGH/CRITICAL — only the reviewed residuals pass.

The residual ~34 are: Jackson shaded inside Hadoop/Parquet uber-jars (Spark's own classpath Jackson is patched); JLine telnet / Jetty UI / Thrift paths not reachable in normal operation; lz4-java (no published fix); and Debian userland packages marked fix_deferred (perl/expat/ncurses/python) that apt upgrade will patch automatically once Debian ships them. Re-review on every Spark/Debian bump and delete entries as they clear.

⁠Build locally

docker build -t spark:4.1.2-python3-java21 spark-docker/

# Override the version (also update SHA512 + GPG key together):
docker build \
  --build-arg SPARK_VERSION=4.1.2 \
  --build-arg SPARK_SHA512=<sha512> \
  --build-arg GPG_KEY=<release-manager-fingerprint> \
  -t spark:4.1.2-python3-java21 spark-docker/

Upstream sources: https://spark.apache.org⁠ · signing keys: https://downloads.apache.org/spark/KEYS⁠