infrahelpers/javabigdata repository overview

⁠Docker images to support big data development on Scala/Java stacks

⁠Introduction

That project⁠ produces Docker images, hosted on dedicated public Docker Cloud site⁠. Those Docker images are intended to bring Linux-based ready-to-use environment for Scala/Java big data developers. Both programming languages are relying on the Java Virtual Machine (JVM) ecosystem. Apache Spark⁠ is featured in particular. SBT-built artefacts can for instance be launched on a stand-alone Spark version.

The supported Linux distributions are:

• CentOS 7⁠
• CentOS 8⁠
• Ubuntu 18.04 LTS (Bionic Beaver)⁠
• Ubuntu 20.04 LTS (Focal Fossal)⁠
• Debian 10 (Buster)⁠
• Debian 9 (Stretch)⁠

Every time some changes are committed on the project's GitHub repository⁠, the Docker images are automatically rebuilt⁠ and pushed onto Docker Cloud.

When some more components are needed, which may be of interest to other Scala/Java big data developers, the Docker image may be amended so as to add those extra components. The preferred way to propose amendment of the Docker image is through pull requests on the GitHub project⁠. Once the pull request has been merged, i.e., once the Dockerfile amendment has been committed in GitHub⁠, Docker Cloud then rebuilds the corresponding Docker images, which become available for every one to use.

⁠Images on Docker Cloud

• Docker Cloud dashboard⁠

⁠Using the pre-built development images

• Start the Docker container featuring the target Linux distribution (<linux-distrib> may be one of centos8, centos7, debian11, debian10, ubuntu2204, ubuntu2004 or ubuntu1804):

$ docker pull infrahelpers/javabigdata:<linux-distrib>
$ docker run --rm -v ~/.ssh/id_rsa:/home/build/.ssh/id_rsa -v ~/.ssh/id_rsa.pub:/home/build/.ssh/id_rsa.pub -it infrahelpers/javabigdata:<linux-distrib>
[[email protected] dev]$ 

• Setup the user name and email address for Git:

[[email protected] dev]$ git config --global user.name "Firstname Lastname"
[[email protected] dev]$ git config --global user.email "[email protected]"

• Clone some Scala/Java-based project (e.g., BOM4V meta-models⁠ is used as an example here):

[[email protected] dev]$ git clone https://github.com/bom4v/metamodels.git
Cloning into 'metamodels...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (35/35), done.
Receiving objects: 100% (5813/5813), 61.53 MiB | 211.00 KiB/s, done.
remote: Total 5813 (delta 12), reused 19 (delta 8), pack-reused 5769
Resolving deltas: 100% (3665/3665), done.
[[email protected] dev]$ cd metamodels
[[email protected] metamodels (master)]$ 

• Do some development:

[[email protected] metamodels (trunk)]$ 

⁠Customize a Docker Image

The images may be customized, and pushed to Docker Cloud; <linux-distrib> may be one of centos8, centos7, debian11, debian10, ubuntu2204, ubuntu2004 or ubuntu1804:

$ mkdir -p ~/dev
$ cd ~/dev
$ git clone https://github.com/bom4v/docker-images.git jvm-docker-images
$ cd jvm-docker-images
$ vi <linux-distrib>/Dockerfile
$ docker build -t infrahelpers/javabigdata:<linux-distrib> <linux-distrib>/
$ docker run --rm -v ~/.ssh/id_rsa:/home/build/.ssh/id_rsa -v ~/.ssh/id_rsa.pub:/home/build/.ssh/id_rsa.pub -it infrahelpers/javabigdata:<linux-distrib>
[[email protected] jvm-docker-images]$ exit
$ docker push infrahelpers/javabigdata:<linux-distrib>

⁠TODO

For any of the following features, an issue may be open on GitHub⁠:

1. Support other Linux distributions, for instance Ubuntu 16.04 LTS or Fedora (e.g., fedora)
2. Automate regular rebuilds (e.g., once a month for CentOS or Ubuntu)