amdatu/amdatu-kubernetes-scalerctl repository overview

⁠Introduction

Scalerd is a component that takes care of (auto) scaling Kubernetes clusters. It exposes a REST API which is consumed by the scalerctl command line tool. Scalerd orchestrates scaling between the Kubernetes API and the cloud provider API. Currently there is only support for Amazon AWS, but other cloud APIs, and even support for "private cloud" infrastructure like OpenStack could be added as well. Examples of the mechanism described in this document are based on the AWS support to make it more concrete.

⁠Scaling up explained

In AWS a AutoScaling Group is defined for Kubernetes nodes. The AutoScaling configuration is set up so that a CloudInit script runs that registers a new node to the Kubernetes cluster completely automatic. Scalerd only has to modify the desired nodes property to scale up the number of machines. After AWS starts the new machines, they will automatically become available to Kubernetes.

⁠Scaling down explained

Scaling down is a bit more tricky than scaling up. In a fully clustered environment it's common to say that there should be at least 3 replicas of everything. In such an environment we could safely turn of a server, and let Kubernetes take care of rescheduling the deleted Pod. The application will continue to work, although with a slightly lower capacity for a short while. Unfortunately, not all our customers are running all pods in a replicated fashion. In such a scenario it might be that the node runs Pods that are not replicated on other nodes. Turning of the node will trigger a reschedule in Kubernetes, but until a new Pod is fully active there is downtime. Because Java applications often take a while to start, this is not acceptable. To mitigate this issue we implemented node draining functionality in scalerd, which includes the following steps on a scale down action: The node is set to unscheduable. This prevents Kubernetes to schedule new pods on this node. For each pod, all labels are removed from the pod. This will make the ReplicationController forget about the pod, which will result in a reschedule. While the new pod is being started, the original pod is still active and still accepts load. scalerd performs health checks on the new pods the same way as during a initial deployment When enough pods are healthy, the original pod is being deleted. When all pods are removed, the node is detached from the AWS AutoScaling group (detaching is an atomic operation that shuts down a specific machine, and reduces the desired nodes property)

⁠scalerd install guide

Scalerd uses the AWS API internally. This API requires AWS credentials to be set in ~/.aws/credentials.

[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
Alternatively the credentials can be provided as environment variables.
AWS_ACCESS_KEY_ID=AKID1234567890
AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY

To run scalerd you can either take one of the OS specific binaries, or the Docker container.

docker run -e AWS_ACCESS_KEY_ID=AKID1234567890 -e AWS_SECRET_ACCESS_KEY=MY-SECRET-KEY amdatu/rti-scalerd -kubernetes http://kubernetes-master:8080 -httpport 8100

8100 is the default port if none specified

⁠scalerctl usage guide

Scalerctl is available as a binary. Before executing any other commands, scalerctl must be configured with the endpoint of the scalerd server:

./scalerctl-macos configure
scalerd url: http://localhost:8100

This configuration is persistent, so that it can be reused by following commands, which is more convenient than passing in an argument each time. Executing scalerctl without arguments will print an overview of available commands. Below is an overview of the most important commands.

⁠scalerctl get nodes

This command gives an overview of the cluster similar to kubectl get nodes, but includes an overview of running pods on each node.

./scalerctl-macos get nodes
2015/11/09 22:59:14 Configured to Scalerd API on http://localhost:8100
Number of nodes: 4
Total CPU: 8
Total Memory: 16201088Ki
Total Pod Capacity: 160
Node: ip-10-150-18-191.eu-west-1.compute.internal - CPU: 8 - Memory: 16201088Ki - Max Pods: 160
Ready: True
Allows scheduling: true
Pods running on node: None
Node: ip-10-150-47-212.eu-west-1.compute.internal - CPU: 2 - Memory: 4050272Ki - Max Pods: 40
Ready: True
Allows scheduling: true
Pods running on node:
 #0 todo-release-189-zv696
Node: ip-10-150-51-222.eu-west-1.compute.internal - CPU: 2 - Memory: 4050272Ki - Max Pods: 40
Ready: True
Allows scheduling: true
Pods running on node: None
Node: ip-10-150-55-85.eu-west-1.compute.internal - CPU: 2 - Memory: 4050272Ki - Max Pods: 40
Ready: True
Allows scheduling: true
Pods running on node:
 #0 todo-release-189-zv696
 #1 todo-release-189-zv696

⁠scalerctl addinstances

Add instances to the cluster

#Add two instances to the cluster. Will be available after a few minutes.
./scalerctl-macos addinstances 2

⁠scalerctl shutdown

Gracefully shutdown a node, and set the number of desired nodes in the autoscaling group n-1. The argument to this command is the node name as known by both the cloud provider and Kubernetes.

./scalerctl-macos shutdown ip-10-150-18-191.eu-west-1.compute.internal

⁠scalerctl enable

Enable a node for scheduling. This removes the "unschedulable" status from a node if set. If no node name is passed as an argument to the command, all nodes in the cluster will be re-enabled.

./scalerctl-macos enable ip-10-150-18-191.eu-west-1.compute.internal

⁠Scheduled scaling

Scalerd / scalerctl can be used to schedule scaling activities. For example a schedule to scale down the number of machines during nighttime, while scaling up again in the morning. Scalerctl can be used to define such a schedule.

scalerctl create daytime.json

#daytime.json
{
  "name": "daytime",
  "cron": "0 0 8 * * *",
  "description": "Switch to full capacity during day time",
  "desiredCapacity": 5,
  "appScaleTemplates": [
    {
      "app": "todo",
      "replicas": 5
    }
  ]
}

Scalerd / scalerctl does not do any verification if the number of pods "fit" on the remaining nodes. This is up to capacity planning of the ops team of the cluster. Note that the number of replicas and the number of nodes doesn't have to match, but usually the number of nodes should be larger than the number of replicas of any application. Scalerd persists schedules in etcd, and re-enables them after restarts. The schedules are stored in the following key structure: /scalerd/schedules/[name-of-schedule], which will contain the JSON above.

⁠Supported cloud providers

Currently scalerd only supports Amazon AWS, because integration with a cloud provider is required to actually add/remove machines beyond the Kubernetes level. Other cloud providers (or private clouds) can be supported as well, but require scalerd to be extended to work with these APIs as well.