ahodges22/eks-rolling-update repository overview

⁠EKS Rolling Update

EKS Rolling Update is a utility for updating the launch configuration or template of worker nodes in an EKS cluster.

• Intro⁠
• Requirements⁠
• Installation⁠
• Usage⁠
• Configuration⁠
• Contributing⁠
• License⁠

⁠Intro

EKS Rolling Update is a utility for updating the launch configuration or template of worker nodes in an EKS cluster. It updates worker nodes in a rolling fashion and performs health checks of your EKS cluster to ensure no disruption to service. To achieve this, it performs the following actions:

• Pauses Kubernetes Autoscaler (Optional)
• Finds a list of worker nodes that do not have a launch config or template that matches their ASG
• Scales up the desired capacity
• Ensures the ASGs are healthy and that the new nodes have joined the EKS cluster
• Cordons the outdated worker nodes
• Suspends AWS Autoscaling actions while update is in progress
• Drains outdated EKS outdated worker nodes one by one
• Terminates EC2 instances of the worker nodes one by one
• Detaches EC2 instances from the ASG one by one
• Scales down the ASG to original count (in case of failure)
• Resumes AWS Autoscaling actions
• Resumes Kubernetes Autoscaler (Optional)

⁠Requirements

• kubectl⁠ installed
• KUBECONFIG environment variable set, or config available in ${HOME}/.kube/config per default
• AWS credentials configured⁠

⁠Installation

⁠From PyPi

pip3 install eks-rolling-update

⁠From source

virtualenv -p python3 venv
source venv/bin/activate
pip3 install -r requirements.txt

⁠Usage

usage: eks_rolling_update.py [-h] --cluster_name CLUSTER_NAME [--plan]

Rolling update on cluster

optional arguments:
  -h, --help            show this help message and exit
  --cluster_name CLUSTER_NAME, -c CLUSTER_NAME
                        the cluster name to perform rolling update on
  --plan, -p            perform a dry run to see which instances are out of
                        date

Example:

eks_rolling_update.py -c my-eks-cluster

⁠Configuration

⁠Run Modes

There are a number of different values which can be set for the RUN_MODE environment variable.

1 is the default.

Each of them have different advantages and disadvantages.

• Scaling up all ASGs at once may cause AWS EC2 instance limits to be exceeded
• Only cordoning the nodes on a per-ASG basis will mean that pods are likely to be moved more than once
• Cordoning the nodes for all ASGs at once could cause issues if new pods needs to start during the process

⁠Examples

• Plan

$ python eks_rolling_update.py --cluster_name YOUR_EKS_CLUSTER_NAME --plan

• Apply Changes

$ python eks_rolling_update.py --cluster_name YOUR_EKS_CLUSTER_NAME

• Cluster Autoscaler

If using cluster-autoscaler, you must let eks-rolling-update know that cluster-autoscaler is running in your cluster by exporting the following environment variables:

$ export  K8S_AUTOSCALER_ENABLED=1 \
          K8S_AUTOSCALER_NAMESPACE="CA_NAMESPACE" \
          K8S_AUTOSCALER_DEPLOYMENT="CA_DEPLOYMENT_NAME"

• Disable operations on cluster-autoscaler

$ unset K8S_AUTOSCALER_ENABLED

• Configure tool via .env file

Rather than using environment variables, you can use a .env file within your working directory to load updater settings. e.g:

$ cat .env
DRY_RUN=1

⁠Docker

Although no public Docker image is currently published for this project, feel free to use the included Dockerfile⁠ to build your own image.

make docker-dist version=1.0.DEV

After building the image, run using the command

docker run -ti --rm \
  -e AWS_DEFAULT_REGION \
  -v "${HOME}/.aws:/root/.aws" \
  -v "${HOME}/.kube/config:/root/.kube/config" \
  eks-rolling-update:latest \
  -c my-cluster

Pass in any additional environment variables and options as described elsewhere in this file.

⁠Contributing

Please read CONTRIBUTING.md⁠ for details on our code of conduct, and the process for submitting pull requests to us.

⁠License

This project is licensed under the Apache 2.0 License - see the LICENSE⁠ file for details