This project is a general purpose container to run shell scripts and python3 scripts.
500K+
This project is used to build General Purpose Container (GPC) images for use in DataKitchen Platform Docker container nodes. These images in, conjunction with the DataKitchen Platform, run shell scripts, python3 code, and Jupyter notebooks. This facilitates the automation of virtually any current or future tool from within the DataKitchen Platform.
Currently, only Ubuntu based images are provided. However, other base images could be used as well. For instance, an Amazon Linux 2 base image was built and tested and could be made available if desired.
As of this writing, there are 2 Dockerfiles available for building the GPC. The first is DockerfileUbuntu20Base which builds a base image that contains a minimum set of APT and python packages. This image can run simple shell and python scripts. It can also be used as a base image for building custom GPC images with packages and configuration specific to an organization or project. The second is DockerfileUbuntu20 which inherits from the base image and adds APT and python packages deemed useful for automating tools from within the DataKitchen Platform. Customers are encouraged to build their own custom image (or base image, if need be) containing the suite of tools required by their organiziation or project teams.
The DOCKERFILE variable assignment below tells Make which Dockerfile to build. The VERSION variable assignment tells Make how to tag the created image. It is highly recommended to use semantic versions when building your images.
make build DOCKERFILE=<dockerfile> VERSION=<semantic_version>
example:
make build DOCKERFILE=DockerfileUbuntu20 VERSION=0.0.1
* By default, the images are built with the --no-cache option (i.e. NO_CACHE=--no-cache). To disable this option,
add NO_CACHE= to the make command above.
By default, the Makefile deploys the image to DataKitchen's private repository on Dockerhub in the datakitchenprod
namespace and dk_general_purpose_container repository. To customize the destination of the images, users may provide
REGISTRY_LOCATION, NAMESPACE, and REPOSITORY variable assignments when issuing a Makefile command. For
instance, to push to a registry other than Dockerhub, set REGISTRY_LOCATION with the address and port of the registry,
making sure to include the trailing slash (e.g. my.registry.address:port/). Currently, the versioned tag is either
ubuntu20-base-<VERSION> for DockerfileUbuntu20Base and ubuntu20-<VERSION> for DockerfileUbuntu20. The tags
for the two images are datakitchenprod/dk_general_purpose_container:ubuntu20-base- and
datakitchenprod/dk_general_purpose_container:ubuntu20- (i.e. REGISTRY_LOCATION is empty by default). If
REGISTRY_LOCATION was set to my.registry.address:port/, the tag would be:
my.registry.address:port/datakitchenprod/dk_general_purpose_container:ubuntu20-base-)
make push DOCKERFILE=<dockerfile> VERSION=<semantic_version>
Typically, users will want to test and validate this versioned image before tagging it with latest and pushing it. The
tag_latest and push_latest targets are for this very purpose. The commands below will tag the image defined by
DOCKERFILE and VERSION with latest (i.e. ubuntu20-base-latest or ubuntu20-latest) and then push that image. The
tag_latest target performs a pull before the tag to ensure the image exists locally.
make tag_latest DOCKERFILE=<dockerfile> VERSION=<semantic_version>
make push DOCKERFILE=<dockerfile> VERSION=<semantic_version>
Three make targets exist to run tests: unittest, imagetest, and ordertest. The unittest target runs locally (i.e. not in a Docker container) to test the Analytic Container source code. By the way, this code lives in the src directory and is executed via the ENTRYPOINT to container. Therefore, if this code is changed, ensure the unittests pass. The imagetest target runs a test inside a Docker container, similar to how container nodes are executed in the DataKitchen Platform. This test will run the local image specified by the DOCKERFILE and VERSION variables. Therefore, you can build the image locally and test it before pushing to Dockerhub. Finally, the ordertest target runs an Order in the DataKitchen platform using the new image specified by the DOCKERFILE and VERSION variables. However, the image must be pushed and available on Dockerhub so the platform can pull it and run it.
The Makefile targets are intended to facilitate a CI/CD pipeline for deploying new GPC images. Internally, DataKitchen's CI/CD pipeline does the following:
--no-binary option (see Change in binary packages between Psycopg 2.7 and 2.8)Content type
Image
Digest
sha256:5c5d1cb2a…
Size
888.4 MB
Last updated
almost 3 years ago
docker pull datakitchenprod/dk_general_purpose_container