lfdominguez/kaniko_with_git repository overview

⁠kaniko - Build Images In Kubernetes

NOTE: kaniko is not an officially supported Google product

kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a standard Kubernetes cluster.

kaniko is meant to be run as an image: gcr.io/kaniko-project/executor. We do not recommend running the kaniko executor binary in another image, as it might not work.

We'd love to hear from you! Join us on #kaniko Kubernetes Slack⁠

:mega: Please fill out our quick 5-question survey⁠ so that we can learn how satisfied you are with kaniko, and what improvements we should make. Thank you! :dancers:

If you are interested in contributing to kaniko, see DEVELOPMENT.md⁠ and CONTRIBUTING.md⁠.

Table of Contents generated with DocToc⁠

• Community⁠
• How does kaniko work?⁠
• Known Issues⁠
• Demo⁠
• Tutorial⁠
• Using kaniko⁠
  • kaniko Build Contexts⁠
  • Using Azure Blob Storage⁠
  • Using Private Git Repository⁠
  • Using Standard Input⁠
  • Running kaniko⁠
    • Running kaniko in a Kubernetes cluster⁠
      • Kubernetes secret⁠
    • Running kaniko in gVisor⁠
    • Running kaniko in Google Cloud Build⁠
    • Running kaniko in Docker⁠
  • Caching⁠
    • Caching Layers⁠
    • Caching Base Images⁠
  • Pushing to Different Registries⁠
    • Pushing to Docker Hub⁠
    • Pushing to Google GCR⁠
    • Pushing to Google GCR - Workload Identity⁠
    • Pushing to Amazon ECR⁠
  • Additional Flags⁠
    • --build-arg⁠
    • --cache⁠
    • --cache-dir⁠
    • --cache-repo⁠
    • --cache-ttl duration⁠
    • --cleanup⁠
    • --context-sub-path⁠
    • --customPlatform⁠
    • --digest-file⁠
    • --dockerfile⁠
    • --force⁠
    • --git⁠
    • --image-name-with-digest-file⁠
    • --image-name-tag-with-digest-file⁠
    • --insecure⁠
    • --insecure-pull⁠
    • --insecure-registry⁠
    • --label⁠
    • --log-format⁠
    • --log-timestamp⁠
    • --no-push⁠
    • --oci-layout-path⁠
    • --push-retry⁠
    • --registry-certificate⁠
    • --registry-mirror⁠
    • --reproducible⁠
    • --single-snapshot⁠
    • --skip-tls-verify⁠
    • --skip-tls-verify-pull⁠
    • --skip-tls-verify-registry⁠
    • --skip-unused-stages⁠
    • --snapshotMode⁠
    • --tarPath⁠
    • --target⁠
    • --use-new-run⁠
    • --verbosity⁠
    • --whitelist-var-run⁠
    • --ignore-path⁠
  • Debug Image⁠
• Security⁠
  • Verifying Signed Kaniko Images⁠
• Comparison with Other Tools⁠
• Community⁠
• Limitations⁠
  • mtime and snapshotting⁠

⁠Community

We'd love to hear from you! Join #kaniko on Kubernetes Slack⁠

⁠How does kaniko work?

The kaniko executor image is responsible for building an image from a Dockerfile and pushing it to a registry. Within the executor image, we extract the filesystem of the base image (the FROM image in the Dockerfile). We then execute the commands in the Dockerfile, snapshotting the filesystem in userspace after each one. After each command, we append a layer of changed files to the base image (if there are any) and update image metadata.

⁠Known Issues

• kaniko does not support building Windows containers.
• Running kaniko in any Docker image other than the official kaniko image is not supported (ie YMMV).
  • This includes copying the kaniko executables from the official image into another image.
• kaniko does not support the v1 Registry API (Registry v1 API Deprecation⁠)

⁠Demo

⁠Tutorial

For a detailed example of kaniko with local storage, please refer to a getting started tutorial⁠.

Please see References⁠ for more docs & video tutorials

⁠Using kaniko

To use kaniko to build and push an image for you, you will need:

1. A build context⁠, aka something to build
2. A running instance of kaniko⁠

⁠kaniko Build Contexts

kaniko's build context is very similar to the build context you would send your Docker daemon for an image build; it represents a directory containing a Dockerfile which kaniko will use to build your image. For example, a COPY command in your Dockerfile should refer to a file in the build context.

You will need to store your build context in a place that kaniko can access. Right now, kaniko supports these storage solutions:

• GCS Bucket
• S3 Bucket
• Azure Blob Storage
• Local Directory
• Local Tar
• Standard Input
• Git Repository

Note about Local Directory: this option refers to a directory within the kaniko container. If you wish to use this option, you will need to mount in your build context into the container as a directory.

Note about Local Tar: this option refers to a tar gz file within the kaniko container. If you wish to use this option, you will need to mount in your build context into the container as a file.

Note about Standard Input: the only Standard Input allowed by kaniko is in .tar.gz format.

If using a GCS or S3 bucket, you will first need to create a compressed tar of your build context and upload it to your bucket. Once running, kaniko will then download and unpack the compressed tar of the build context before starting the image build.

To create a compressed tar, you can run:

tar -C <path to build context> -zcvf context.tar.gz .

Then, copy over the compressed tar into your bucket. For example, we can copy over the compressed tar to a GCS bucket with gsutil:

gsutil cp context.tar.gz gs://<bucket name>

When running kaniko, use the --context flag with the appropriate prefix to specify the location of your build context:

If you don't specify a prefix, kaniko will assume a local directory. For example, to use a GCS bucket called kaniko-bucket, you would pass in --context=gs://kaniko-bucket/path/to/context.tar.gz.

⁠Using Azure Blob Storage

If you are using Azure Blob Storage for context file, you will need to pass Azure Storage Account Access Key⁠ as an environment variable named AZURE_STORAGE_ACCESS_KEY through Kubernetes Secrets

⁠Using Private Git Repository

You can use Personal Access Tokens for Build Contexts from Private Repositories from GitHub⁠.

You can either pass this in as part of the git URL (e.g., git://[email protected]/acme/myproject.git#refs/heads/mybranch) or using the environment variable GIT_TOKEN.

You can also pass GIT_USERNAME and GIT_PASSWORD (password being the token) if you want to be explicit about the username.

⁠Using Standard Input

If running kaniko and using Standard Input build context, you will need to add the docker or kubernetes -i, --interactive flag. Once running, kaniko will then get the data from STDIN and create the build context as a compressed tar. It will then unpack the compressed tar of the build context before starting the image build. If no data is piped during the interactive run, you will need to send the EOF signal by yourself by pressing Ctrl+D.

Complete example of how to interactively run kaniko with .tar.gz Standard Input data, using docker:

echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
  --interactive -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest \
  --context tar://stdin \
  --destination=<gcr.io/$project/$image:$tag>

Complete example of how to interactively run kaniko with .tar.gz Standard Input data, using Kubernetes command line with a temporary container and completely dockerless:

echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | kubectl run kaniko \
--rm --stdin=true \
--image=gcr.io/kaniko-project/executor:latest --restart=Never \
--overrides='{
  "apiVersion": "v1",
  "spec": {
    "containers": [
      {
        "name": "kaniko",
        "image": "gcr.io/kaniko-project/executor:latest",
        "stdin": true,
        "stdinOnce": true,
        "args": [
          "--dockerfile=Dockerfile",
          "--context=tar://stdin",
          "--destination=gcr.io/my-repo/my-image"
        ],
        "volumeMounts": [
          {
            "name": "cabundle",
            "mountPath": "/kaniko/ssl/certs/"
          },
          {
            "name": "docker-config",
            "mountPath": "/kaniko/.docker/"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "cabundle",
        "configMap": {
          "name": "cabundle"
        }
      },
      {
        "name": "docker-config",
        "configMap": {
          "name": "docker-config"
        }
      }
    ]
  }
}'

⁠Running kaniko

There are several different ways to deploy and run kaniko:

• In a Kubernetes cluster⁠
• In gVisor⁠
• In Google Cloud Build⁠
• In Docker⁠

⁠Running kaniko in a Kubernetes cluster

Requirements:

• Standard Kubernetes cluster (e.g. using GKE⁠)
• Kubernetes Secret⁠
• A build context⁠

⁠Kubernetes secret

To run kaniko in a Kubernetes cluster, you will need a standard running Kubernetes cluster and a Kubernetes secret, which contains the auth required to push the final image.

To create a secret to authenticate to Google Cloud Registry, follow these steps:

1. Create a service account in the Google Cloud Console project you want to push the final image to with Storage Admin permissions.
2. Download a JSON key for this service account
3. Rename the key to kaniko-secret.json
4. To create the secret, run:

kubectl create secret generic kaniko-secret --from-file=<path to kaniko-secret.json>

Note: If using a GCS bucket in the same GCP project as a build context, this service account should now also have permissions to read from that bucket.

The Kubernetes Pod spec should look similar to this, with the args parameters filled in:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=<path to Dockerfile within the build context>"
    - "--context=gs://<GCS bucket>/<path to .tar.gz>"
    - "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"
    volumeMounts:
    - name: kaniko-secret
      mountPath: /secret
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /secret/kaniko-secret.json
  restartPolicy: Never
  volumes:
  - name: kaniko-secret
    secret:
      secretName: kaniko-secret

This example pulls the build context from a GCS bucket. To use a local directory build context, you could consider using configMaps to mount in small build contexts.

⁠Running kaniko in gVisor

Running kaniko in gVisor⁠ provides an additional security boundary. You will need to add the --force flag to run kaniko in gVisor, since currently there isn't a way to determine whether or not a container is running in gVisor.

docker run --runtime=runsc -v $(pwd):/workspace -v ~/.config:/root/.config \
gcr.io/kaniko-project/executor:latest \
--dockerfile=<path to Dockerfile> --context=/workspace \
--destination=gcr.io/my-repo/my-image --force

We pass in --runtime=runsc to use gVisor. This example mounts the current directory to /workspace for the build context and the ~/.config directory for GCR credentials.

⁠Running kaniko in Google Cloud Build

Requirements:

• A build context⁠

To run kaniko in GCB, add it to your build config as a build step:

steps:
- name: gcr.io/kaniko-project/executor:latest
  args: ["--dockerfile=<path to Dockerfile within the build context>",
         "--context=dir://<path to build context>",
         "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"]

kaniko will build and push the final image in this build step.

⁠Running kaniko in Docker

Requirements:

• Docker⁠

We can run the kaniko executor image locally in a Docker daemon to build and push an image from a Dockerfile.

For example, when using gcloud and GCR you could run kaniko as follows:

docker run \
    -v "$HOME"/.config/gcloud:/root/.config/gcloud \
    -v /path/to/context:/workspace \
    gcr.io/kaniko-project/executor:latest \
    --dockerfile /workspace/Dockerfile \
    --destination "gcr.io/$PROJECT_ID/$IMAGE_NAME:$TAG" \
    --context dir:///workspace/

There is also a utility script run_in_docker.sh⁠ that can be used as follows:

./run_in_docker.sh <path to Dockerfile> <path to build context> <destination of final image>

NOTE: run_in_docker.sh expects a path to a Dockerfile relative to the absolute path of the build context.

An example run, specifying the Dockerfile in the container directory /workspace, the build context in the local directory /home/user/kaniko-project, and a Google Container Registry as a remote image destination:

./run_in_docker.sh /workspace/Dockerfile /home/user/kaniko-project gcr.io/$PROJECT_ID/$TAG

⁠Caching

⁠Caching Layers

kaniko can cache layers created by RUN and Copy (configured by flag --cache-copy-layers) commands in a remote repository. Before executing a command, kaniko checks the cache for the layer. If it exists, kaniko will pull and extract the cached layer instead of executing the command. If not, kaniko will execute the command and then push the newly created layer to the cache.

Note that kaniko cannot read layers from the cache after a cache miss: once a layer has not been found in the cache, all subsequent layers are built locally without consulting the cache.

Users can opt into caching by setting the --cache=true flag. A remote repository for storing cached layers can be provided via the --cache-repo flag. If this flag isn't provided, a cached repo will be inferred from the --destination provided.

⁠Caching Base Images

kaniko can cache images in a local directory that can be volume mounted into the kaniko pod. To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming image at gcr.io/kaniko-project/warmer:

docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:latest --cache-dir=/workspace/cache --image=<image to cache> --image=<another image to cache>

--image can be specified for any number of desired images. This command will cache those images by digest in a local directory named cache. Once the cache is populated, caching is opted into with the same --cache=true flag as above. The location of the local cache is provided via the --cache-dir flag, defaulting to /cache as with the cache warmer. See the examples directory for how to use with kubernetes clusters and persistent cache volumes.

⁠Pushing to Different Registries

kaniko uses Docker credential helpers to push images to a registry.

kaniko comes with support for GCR, Docker config.json and Amazon ECR, but configuring another credential helper should allow pushing to a different registry.

⁠Pushing to Docker Hub

Get your docker registry user and password encoded in base64

echo -n USER:PASSWORD | base64

Create a config.json file with your Docker registry url and the previous generated base64 string

Note: Please use v1 endpoint. See #1209 for more details

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "xxxxxxxxxxxxxxx"
		}
	}
}

Run kaniko with the config.json inside /kaniko/.docker/config.json

docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination=yourimagename

⁠Pushing to Google GCR

To create a credentials to authenticate to Google Cloud Registry, follow these steps:

1. Create a service account⁠ or in the Google Cloud Console project you want to push the final image to with Storage Admin permissions.
2. Download a JSON key for this service account
3. (optional) Rename the key to kaniko-secret.json, if you don't rename, you have to change the name used the command(in the volume part)
4. Run the container adding the path in GOOGLE_APPLICATION_CREDENTIALS env var

docker run -ti --rm -e GOOGLE_APPLICATION_CREDENTIALS=/kaniko/config.json \
-v `pwd`:/workspace -v `pwd`/kaniko-secret.json:/kaniko/config.json:ro gcr.io/kaniko-project/executor:latest \
--dockerfile=Dockerfile --destination=yourimagename

⁠Pushing to GCR using Workload Identity

If you have enabled Workload Indentity on your GKE cluster then you can use the workload identity to push built images to GCR without adding a GOOGLE_APPLICATION_CREDENTIALS in your kaniko pod specification.

Learn more on how to enable⁠ and migrate existing apps⁠ to workload identity.

To authenticate using workload identity you need to run the kaniko pod using the Kubernetes Service Account (KSA) bound to Google Service Account (GSA) which as Storage.Admin permissions to push images to Google Container registry.

Please follow the detailed steps here⁠ to create a Kubernetes Service Account, Google Service Account and create an IAM policy binding between the two to allow the Kubernetes Service account to act as the Google service account.

To grant the Google Service account the right permission to push to GCR, run the following GCR command

gcloud projects add-iam-policy-binding $PROJECT \
  --member=serviceAccount:[gsa-name]@${PROJECT}.iam.gserviceaccount.com \
  --role=roles/storage.objectAdmin

Please ensure, kaniko pod is running in the namespace and with Kubernetest Service Account.

⁠Pushing to Amazon ECR

The Amazon ECR credential helper⁠ is built into the kaniko executor image. To configure credentials, you will need to do the following:

1. Update the credsStore section of config.json⁠:

{ "credsStore": "ecr-login" }

You can mount in the new config as a configMap:

kubectl create configmap docker-config --from-file=<path to config.json>

1. Configure credentials

   1. You can use instance roles when pushing to ECR from a EC2 instance or from EKS, by configuring the instance role permissions⁠.

   2. Or you can create a Kubernetes secret for your ~/.aws/credentials file so that credentials can be accessed within the cluster. To create the secret, run:

      kubectl create secret generic aws-secret --from-file=<path to .aws/credentials>
      

      Copy

The Kubernetes Pod spec should look similar to this, with the args parameters filled in. Note that aws-secret volume mount and volume are only needed when using AWS credentials from a secret, not when using instance roles.

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=<path to Dockerfile within the build context>"
    - "--context=s3://<bucket name>/<path to .tar.gz>"
    - "--destination=<aws_account_id.dkr.ecr.region.amazonaws.com/my-repository:my-tag>"
    volumeMounts:
    - name: docker-config
      mountPath: /kaniko/.docker/
    # when not using instance role
    - name: aws-secret
      mountPath: /root/.aws/
  restartPolicy: Never
  volumes:
  - name: docker-config
    configMap:
      name: docker-config
  # when not using instance role
  - name: aws-secret
    secret:
      secretName: aws-secret

⁠Additional Flags

⁠--build-arg

This flag allows you to pass in ARG values at build time, similarly to Docker. You can set it multiple times for multiple arguments.

⁠--cache

Set this flag as --cache=true to opt into caching with kaniko.

⁠--cache-dir

Set this flag to specify a local directory cache for base images. Defaults to /cache.

This flag must be used in conjunction with the --cache=true flag.

⁠--cache-repo

Set this flag to specify a remote repository that will be used to store cached layers.

If this flag is not provided, a cache