Sign inSign up

glazole/spark-stack

By glazole

ā€¢Updated 11 months ago

Apache Spark 3.5 + Iceberg 1.6 + GraphFrames + JupyterLab ā€” complete data-engineering stack

Image
0

132

glazole/spark-stack repository overview

ā šŸ§Š Glazkov-OI Spark Stack

Apache Spark 3.5 + Iceberg 1.6 + GraphFrames 0.9 + JupyterLab

Pre-built data-engineering environment optimized for local & cloud analytics.
Includes Spark 3.5 with AWS SDK v2 / Hadoop 3.3 S3A, Iceberg REST Catalog + PostgreSQL metastore,
and designed to work with an external MinIO container as an S3-compatible object storage backend.

ā šŸ§© Available Tags

TagDescription
3.5.0-iceberg-sparkSpark master + worker image with Iceberg and GraphFrames
3.5.0-iceberg-jupyterJupyterLab client for PySpark + Iceberg + GraphFrames

ā āš” Quick Start

docker run -d --name spark-master -p 7077:7077 -p 8080:8080 glazole/spark-stack:3.5.0-iceberg-spark
docker run -d --name spark-worker --link spark-master:spark-master glazole/spark-stack:3.5.0-iceberg-spark
docker run -d --name jupyter -p 8888:8888 --link spark-master:spark-master glazole/spark-stack:3.5.0-iceberg-jupyter

Then open šŸ‘‰ http://localhost:8888ā  Default token: lab

ā šŸ§  Features

  • šŸ§Š Iceberg REST + PostgreSQL metadata store
  • ā˜ļø External MinIO S3A backend
  • āš” Spark 3.5 + AWS SDK v2 support
  • šŸ§  GraphFrames 0.9.0
  • šŸ“˜ JupyterLab with PyIceberg 0.5.1

ā šŸ§± Used in Docker Compose

services:
  spark-master:
    image: glazole/spark-stack:3.5.0-iceberg-spark
    container_name: spark-master
    ports:
      - "7077:7077"   # Spark master
      - "8080:8080"   # Spark UI

  spark-worker:
    image: glazole/spark-stack:3.5.0-iceberg-spark
    container_name: spark-worker
    depends_on:
      - spark-master
    ports:
      - "8081:8081"   # Worker UI (optional)

  jupyter:
    image: glazole/spark-stack:3.5.0-iceberg-jupyter
    container_name: jupyter
    depends_on:
      - spark-master
    ports:
      - "8888:8888"   # JupyterLab

  iceberg-rest:
    image: apache/iceberg-rest-fixture
    container_name: iceberg-rest
    depends_on:
      - postgresql
    ports:
      - "8181:8181"   # Iceberg REST API
    environment:
      CATALOG_WAREHOUSE: s3://warehouse/iceberg/
      CATALOG_IO__IMPL: org.apache.iceberg.aws.s3.S3FileIO
      # === JDBC Catalog ===
      CATALOG_IMPL: org.apache.iceberg.jdbc.JdbcCatalog
      CATALOG_URI: jdbc:postgresql://postgresql:5432/iceberg_catalog
      CATALOG_JDBC_USER: iceberg
      CATALOG_JDBC_PASSWORD: iceberg
      # === MinIO / S3 ===
      CATALOG_S3_ENDPOINT: http://minio:9000
      # === AWS ===
      AWS_REGION: us-east-1
      AWS_ACCESS_KEY_ID: minio
      AWS_SECRET_ACCESS_KEY: minio_minio
      LOG_LEVEL: INFO

  postgresql:
    image: postgres:15-alpine
    container_name: postgresql
    environment:
      POSTGRES_DB: iceberg_catalog
      POSTGRES_USER: iceberg
      POSTGRES_PASSWORD: iceberg
    ports:
      - "5432:5432"
ā āš™ļø Included JARs (for Spark runtime)
  • iceberg-spark-runtime-3.5_2.12-1.6.1.jar
  • aws-java-sdk-bundle-1.12.517.jar
  • hadoop-aws-3.3.4.jar
  • bundle-2.25.48.jar, s3-2.25.48.jar, url-connection-client-2.25.48.jar
  • graphframes-spark3_2.12-0.9.0-spark3.5.jar

ā šŸ“‚ Source code and Docker setup: āž”ļø https://github.com/glazole/msij-projectā 

Ā© Glazkov-OI / Data Office Stack

Tag summary

Content type

Image

Digest

sha256:5bf0c6ffbā€¦

Size

1.9 GB

Last updated

11 months ago

docker pull glazole/spark-stack:3.5.0-iceberg-jupyter