alphanoob1337/failover-controller repository overview

⁠Introduction

Let's say you have a kubernetes cluster running deployment A and deployment B and a service which is pointing to deployment A but you want it to failover to deployment B in case deployment A goes down. Whenever deployment A is back, you want to have the service switch back to that.

These priority-based failover scenarios can be implemented using this controller: it monitors its namespace for services with the label failoverLabel and a selector including this failover label. It will also check the namespace for any pods matching the selector defined by the service excluding the failover label. It will then check the pods for a failoverPriority label (defaults to 0). Finally, the pod(s) with the highest priority and status Ready will be patched to have the failover label with the value specified in the service selector and remove the failover label value from all other pods matching the service selector.

Additionally, you can specify the label failoverMinReplicas for your pods (if not specified it defaults to 1). This prevents the service from being re-routed when less than the specified number of replicas are ready. The number of replicas are counted across all pods with the same value for the pod-template-hash. In case you prefer to count the number of live replicas across different pod templates, you can add a label failoverGroup; any pods with the same value are grouped together to determine the number of replicas.

⁠Environment variables

There are two environment variables which can be specified:

• FAILOVER_CONTROLLER_LOG_LEVEL can have the values "DEBUG", "INFO" (default), "WARNING" or "ERROR"
• FAILOVER_CONTROLLER_UPDATE_INTERVAL expects a positive numeric value (float or integer) and will limit the number of requests to the kubernetes API server by checking the services and pods in the namespace less often (defaults is 0.1 s)

⁠Example

Bringing everything together in a kubernetes yaml file:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: failover-controller
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: failover-controller
rules:
- apiGroups: [ '' ] # Core API only
  resources: [ 'pods', 'services' ]
  verbs: [ 'get', 'list', 'watch', 'patch' ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: failover-controller-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: failover-controller
subjects:
- kind: ServiceAccount
  name: failover-controller
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: failover-controller
  namespace: default
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: failover-controller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: failover-controller
    spec:
      serviceAccountName: failover-controller
      containers:
        - name: app
          image: alphanoob1337/failover-controller:latest
          env:
            - name: FAILOVER_CONTROLLER_LOG_LEVEL
              value: "DEBUG"
            - name: FAILOVER_CONTROLLER_UPDATE_INTERVAL
              value: "0.05"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-a
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      failoverGroup: 'whoami-a'
  template:
    metadata:
      labels:
        failoverGroup: 'whoami-a'
        failoverPriority: '100'
        failoverMinReplicas: '2'
        app.kubernetes.io/name: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami:v1.4.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami-b
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      failoverGroup: 'whoami-b'
  template:
    metadata:
      labels:
        failoverGroup: 'whoami-b'
        app.kubernetes.io/name: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami:v1.4.0
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: default
  labels:
    failoverLabel: failoverStatus
spec:
  selector:
    app.kubernetes.io/name: whoami
    failoverStatus: active
  ports:
    - name: whoami
      protocol: TCP
      port: 80

This will create a service account (required for the failover controller to access the kubernetes API server and modify pod labels), a deployment for the failover controller and two example deployments whoami-a and whoami-b as well as a service whoami. The failover controller will add a label failoverStatus with value active to all pods from the deployment whoami-a since the failoverPriority is a higher numerical value than for deployment whoami-b (which has the default value 0) but it will only do so, if more than 2 replicas are ready (due to the label failoverMinReplicas). If whoami-a has less pods in the ready state but whoami-b has at least one pod ready, the failoverStatus will be set to active for whoami-b.

Please note that technically you don't need to specify labels for failoverGroup in the example above since the replicas of whoami-a will all have the same pod-template-hash. This is just done for illustration.

⁠Repository

You can find the source code and build instrunctions in the Github repository⁠.