icebal/ingress-enforcer repository overview

⁠ingress-enforcer

Ingress Enforcer, Kubernetes Admission Plugin

⁠About This Tool

ingress-enforcer is a Kubernetes Admission Plugin⁠ that does the following:

1. Adds a default ingress class⁠ to an ingress object if left undefined. Some ingress controllers⁠ assume they should be the default if a user doesn't define this, so ingress-enforcer allows you to be explicit even when the user isn't.
2. It enforces allowed ingress classes, with exemptions per namespace.

⁠Example Use Case

ingress-enforcer was born out of the need to control ingress usage in a multi tenant Kubernetes cluster. Here's some problems that this admission plugin may solve for you:

• Some ingress controllers attempt to latch onto ingress objects if the user doesn't define a class, and may not allow you to turn this behavior off. With multiple controllers, this can lead to race conditions and constant updating of the ingress object's status as the controllers "fight". ingress-enforcer lets you define a default ingress class to add to ingress objects if the user forgets to define one. This prevents race conditions and lets you decide what the default ingress controller is on your cluster.
• You may want to limit access to certain ingress controllers. For instance, GKE's default controller "gce" allows users to create publicly accessible ingresses. In a private cluster situation, you may want to restrict usage of this controller to those teams that understand how to secure it with edge auth (or just ensure the team has been educated on risks before using). ingress-enforcer tackles this problem by letting you define what classes can be used by default, and allows you to define exemptions to other classes per namespace. For instance, you may set a policy allowing all namespaces to use class "nginx", but only certain namespaces can use "gce".

⁠Current Features

The following features are currently working:

• Setting the default ingress class to apply to ingress objects without a class defined,
• Setting cluster wide allowed ingress classes,
• Setting namespace scoped exemptions to certain classes.
• Supports ingress objects from API groups extensions/v1beta1 and networking.k8s.io/v1beta1.

⁠Upcoming Features

The following features are coming soon:

• All proposed ideas have been implemented! Can you think of a feature you'd like to see added?

⁠Other To-do

This project was started by a System Admin who is new to Go, but wanting to do things right eventually. The project needs the following things done eventually and contributions are welcome!

• While the project is using Kubernetes' official e2e webhook code as an example to live by, it would be nice to have a Go veteran review the code for improvements / issues,
• Wrap up CI/CD testing and builds with automatic container generation to Docker Hub.
• Finish up Sonarqube code smells (only a couple).

⁠Documentation

⁠Docker Hub Container

ingress-enforcer containerized builds are uploaded to Docker Hub⁠. The container is built from scratch⁠ image and only includes the ingress-enforcer binary making it very small in both size and security footprint. The container runs as non-root and listens on port 8443.

⁠Installation Into a Kubernetes Cluster

ingress-enforcer can run within the Kubernetes cluster it is serving and is the recommended configuration option.

⁠Preparing Certificates

Kubernetes requires that communication to the webhook happens over a verified TLS connection. If you stick with the default configurations in /deploy, you will need to provide a cert keypair for common name ingress-enforcer-service.kube-system.svc (change this if you change your service name, they should match). If this certificate pair is created by a certificate authority not trusted by the system's certificate store (i.e. because you don't want to go through the trouble of adding the CA or you don't have access to it such as in GKE or EKS) you'll need to also include the caBundle option in your webhook configuration so Kubernetes will trust your cert keypair.

A quick tutorial for creating your own CA and certs is available here⁠.

⁠Preparing the Webhook Configuration

As stated above, if you are creating a cert keypair using a certificate authority that isn't in the Kubernetes system certificate store, you have to state a caBundle in your webhook configuration in base64 format. Assuming you've followed the tutorial above and have a CA certificate ready:

base64 -w 0 cacert.pem

The above command should provide you with a base64 encoded string for use in the webhookconfigurations.yaml in the /deploy directory of this repository within the caBundle fields.

If you are using a cert pair that is trusted by your cluster by default, you can remove the caBundle fields.

⁠Creating the ingress-enforcer Cert Secret

You will need to create a Kubernetes Secret to hold your cert keypair. In the /deploy directory of this repository we have a template you can use called secret-template.yaml. Either use this file once and trash it or pipeline its secret data so the cert keypair doesn't get committed to version control.

The secret data will need to be base64 encoded. You can do that with the following commands:

base64 -w 0 server.crt
base64 -w 0 server.key

⁠Deploying Into Kubernetes

Now that you have your cert prep out of the way, deploy ingress-enforcer as follows:

1. Apply configmap.yaml and secret.yaml,
2. Apply deployment.yaml and service.yaml,
3. Apply webhookconfigurations.yaml.

If all goes as planned, you should have a functioning installation of ingress-enforcer running in the kube-system namespace.

⁠Platform Specific Information

⁠GKE

GKE clusters enable the --enable-aggregator-routing option because the API server is unable to route service IP addresses. This causes it to attempt a connection directly to the ingress-enforcer pod on port 8443 instead. If creating ingresses hangs with an eventual timeout, you may need to adjust your firewall to allow the GKE master to communicate in to your nodes on port 8443.

This seems troublesome, but running the pod on port 443 (usually automatically opened by GKE) would have required running the container as root which is bad practice and may not be allowed in some environments.