gucharbon/microb repository overview

⁠microb - build python based container images the easy way

🐳 microb is an alternative to the Dockerfile format for creating best practice Python based container images. It relies on the same pyproject.toml file which is already used to configure Python projects.

As a buildkit frontend, microb does not need to be installed. It is seamlessly integrated and run by docker buildkit⁠ (respectively docker⁠).
Create best practice docker images for packaging your python app with ease, without beeing a docker pro!

⁠Pyproject.toml

Modern Python packages can contain a pyproject.toml file, first introduced in PEP 518⁠ and later expanded in PEP 517⁠, PEP 621⁠ and PEP 660⁠. This file contains build system requirements and information, which are used by pip or other package managers to build the package.

microb uses the pyproject.toml file to gather the required dependencies and build the container image. No need for an additional Dockerfile or a separate build configuration !

#syntax=gucharbon/microb                           # [1]  Enable automatic microb syntax support
[project]
name = "my_example"
authors = [
    {name = "Guillaume Charbonnier", email = "[email protected]"},
]
requires-python = ">=3.8,<3.12"
dependencies = [
    "nats-py",
    "nkeys"
]
dynamic = ["version"]

[tool.setuptools]
py-modules = ["example"]

[tool.setuptools.dynamic]
version = {attr = "example.__version__"}

[tool.microb.target.default]
api_version = "v1"                                 # [2] Configure the microb API version used
python_version = "3.11"                            # [3] Configure the python interpreter version to use
build_deps = ["build-essential", "libffi-dev"]     # [4] Additional apt packages to install during build (not installed in final image)
system_deps = ["gettext"]                          # [5] Additional apt packages to install in final image (not installed in build image)
env = { "FOO" = "bar" }                            # [6] Additional environment variables to set in final image
indices = [{ "url" = "https://pypi.org/simple" }]  # [7] Configure pip index to use
labels = { "com.example.foo" = "bar" }             # [8] Additional labels to add to the final image
entrypoint = ["micro", "run"]                      # [9] Configure the entrypoint used in the final image
command = ["example:setup"]                        # [10] Configure the command used in the final image

The above pyproject.toml file can be used to produce a docker image with the following command:

docker build -t example:latest -f pyproject.toml .

The most important part of the file is the first line #syntax=gucharbon/microb. It tells docker buildkit to use the microb frontend. This can also be achieved by setting the frontend to solve the dockerfile by the running engine itself. For instance for the docker build command one can append the following build-arg to tell docker to use microb without the in-file syntax directive: --build-arg BUILDKIT_SYNTAX=gucharbon/microb:v1. However, the recommended way is to set it in the pyproject.toml, as this is independent of the used builder cli.

The pyproject.toml file is a standard file for python projects. It is used to configure the project and its dependencies.

Several build configurations can be defined in a single pyproject.toml. Each build configuration is defined in a separate section under [tool.microb.target]. The default build configuration is defined under [tool.microb.target.default]. To build a specific target, use the microb_target build argument:

docker build -t example:latest --build-arg microb_target=default -f pyproject.toml .

The frontend is compatible with linux, windows and mac. It also supports various cpu architectures. Currently i386, amd64, arm/v6, arm/v7, arm64/v8 are supported. Buildkit automatically picks the right version for you from docker hub.

Available configuration options are listed in the table below.

⁠Copy

⁠Add

⁠Index

The example folder⁠ contains a few examples how you can use microb.

⁠Recommendations for using microb

• use https in favor of http if possible (for registries, for direct whl files and for git)
• try to avoid setting trust in an index definition, rather use a trusted https url
• prefer git+ssh://[email protected]/moskomule/anatome.git over http / https links like git+https://user:[email protected]/moskomule/anatome.git
• in general prefer setting up an index under the indices key for authentication of existing pip registries, rather than using in-url credentials

⁠Build microb

microb can be build with every docker buildkit compatible cli. The following are a few examples:

⁠docker:

DOCKER_BUILDKIT=1 docker build --ssh default --build-arg BUILDKIT_SYNTAX=gucharbon/microb:v1 -t example:latest -f pyproject.toml .

In that particular case even the syntax directive from [1] is not required anymore, as it is set on the docker build command directly.
If the syntax directive is set in the pyproject.toml, --build-arg BUILDKIT_SYNTAX=gucharbon/microb:v1 can be omitted in the command.

⁠nerdctl:

nerdctl build --ssh default -t example:latest -f pyproject.toml .

⁠buildctl:

buildctl build \
--frontend=gateway.v0 \
--opt source=gucharbon/microb:v1 \
--ssh default \
--local context=. \
--local dockerfile=pyproject.tml \
--output type=docker,name=example:latest \
| docker load

In that particular case even the syntax directive from [1] is not required anymore, as it is set on the buildctl command directly.
If the syntax directive is set in the pyproject.toml, --opt source=gucharbon/microb:v1 can be omitted in the command.

The resulting image is build as a best practice docker image and employs a multistage build- It uses official python slim images⁠ image as final base image. It runs as non-root user and only includes the minimal required runtime dependencies.

⁠SSH dependencies

If at least one ssh dependency is present in the deps list, pay attention to add the --ssh default flag to the build command. Also make sure, that your ssh-key is loaded inside the ssh agent.
If you receive an error invalid empty ssh agent socket, make sure SSH_AUTH_SOCK is set your SSH agent is not running or improperly set up. You can start or configure it and adding your ssh key by executing:

eval `ssh-agent`
ssh-add /path/to/ssh-key

The ssh flag is only required if you're including a ssh dependency. If no ssh dependency is present, the ssh flag can be omitted.

⁠Run a container from the built image

The built image can be run like any other container:

$ docker run --rm example:latest

⁠microb development

⁠Installation as cmd

$ go install github.com/charbonats/microb

⁠Arguments

The following arguments are supported running the frontend:

For instance to show the created equivalent Dockerfile, use the command go run ./cmd/microb/main.go -dockerfile -filename example/minimal/pyproject.toml.

⁠Example generated Dockerfile

The example present in example/minimal⁠ contains a pyproject.toml⁠ file. The dockerfile produced by microb is the following:

FROM docker.io/python:3.11 AS builder

RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt apt-get update && apt-get install -y --no-install-recommends build-essential libffi-dev

ENV PIP_DISABLE_PIP_VERSION_CHECK=1
ENV PIP_NO_WARN_SCRIPT_LOCATION=0
ENV PIP_USER=1
ENV PYTHONPYCACHEPREFIX=/.pycache
RUN --mount=type=cache,target=/root/.cache --mount=type=ssh,required=true GIT_SSH_COMMAND='ssh -o StrictHostKeyChecking=no' python -m pip install --user --retries 2 --extra-index-url https://pypi.org/simple nats-py nats-micro@git+ssh://[email protected]/charbonats/nats-micro.git nkeys
COPY . /projectdir
RUN --mount=type=cache,target=/root/.cache python -m pip install --no-deps /projectdir
RUN find /root/.local/lib/python*/ -name 'tests' -exec rm -r '{}' + && find /root/.local/lib/python*/site-packages/ -name '*.so' -exec sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; && find /root/.local/lib/python*/ -type f -name '*.pyc' -delete && find /root/.local/lib/python*/ -type d -name '__pycache__' -delete

FROM python:3.11-slim

RUN apt-get update && apt-get install -y --no-install-recommends  gettext  && rm -rf /var/lib/apt/lists/*
RUN useradd --uid=65532 --user-group --home-dir=/home/nonroot --create-home nonroot
USER 65532:65532

COPY --from=builder /root/.local /home/nonroot/.local
ENV PATH=$PATH:/home/nonroot/.local/bin

ENTRYPOINT ["micro","run"]
CMD ["example:setup"]

LABEL org.opencontainers.image.description="autogenerated by microb"
LABEL moby.buildkit.frontend="microb"
LABEL microb.version="v1"
LABEL com.example.foo="World"
LABEL org.opencontainers.image.authors="Guillaume Charbonnier <[email protected]>, Someone else <[email protected]>"

⁠Credits

• https://github.com/cmdjulian/mopy⁠
• https://earthly.dev/blog/compiling-containers-dockerfiles-llvm-and-buildkit/⁠
• https://github.com/moby/buildkit/blob/master/docs/merge%2Bdiff.md⁠
• https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md⁠
• https://gitlab.wikimedia.org/repos/releng/blubber⁠