kuberhost/kube-backup repository overview

⁠Kube-backup

Kubernetes resource state backup to git

⁠Git structure

_global_ - global resources such as Node, ClusterRole, StorageClass
_grafana_ - grafana configs (when grafana enabled)
<namespace> - such as kube-system, default, etc...
  <ResourceType> - folder for each resource type
    <resource-name.yaml> - file for each resource

⁠Screenshots

⁠Deployment

Yaml manifests are in deploy folder⁠.

⁠Create Deployment Key

Github and gitlab support adding key only for one repository

• Create repo
• Generate ssh key ssh-keygen -f ./new_key
• Add new ssh key to repo with write access
• Save key to 2_config_map.yaml⁠ (see comments in file)

⁠Testing Deployment

I recommend to run it periodically with kubernetes' CronJob resource, if you want to test how it works without waiting then can change running schedule or create pod with same parameters

⁠Commands

• kube_backup backup - pull remote git repository, save kubernetes state, make git commit in local repository
• kube_backup push - push changes to remote repository
• kube_backup help - shows help

Docker image by default runs kube_backup backup && kube_backup push

⁠Config

• GIT_REPO_URL - remote git URL like [email protected]:kuberhost/kube-backup.git (required)
• BACKUP_VERBOSE use 1 to enable verbose logging
• TARGET_PATH - local git repository folder, default ./kube_state
• SKIP_NAMESPACES - namespaces to exclude, separated by coma (,)
• ONLY_NAMESPACES - whitelist namespaces
• GLOBAL_RESOURCES - override global resources list, default is node, apiservice, clusterrole, clusterrolebinding, podsecuritypolicy, storageclass, persistentvolume, customresourcedefinition, mutatingwebhookconfiguration, validatingwebhookconfiguration, priorityclass
• EXTRA_GLOBAL_RESOURCES - use it to add resources to GLOBAL_RESOURCES list
• SKIP_GLOBAL_RESOURCES - blacklist global resources
• RESOURCES - default list of namespaces resources, see KubeBackup::TYPES
• EXTRA_RESOURCES - use it to add resources to RESOURCES list
• SKIP_RESOURCES - exclude resources
• SKIP_OBJECTS - use it to skip individual objects, such as kube-backup/ConfigMap/kube-backup-ssh-config (separated by coma, spaces around coma ignored)
• GIT_USER - default is kube-backup
• GIT_EMAIL - default is kube-backup@$(HOSTNAME)
• GIT_BRANCH - Git branch, default is master
• GIT_PREFIX - Path to the subdirectory in your repository
• GRAFANA_URL - grafana api URL, e.g. https://grafana.my-cluster.com
• GRAFANA_TOKEN - grafana API token, create at https://your-grafana/org/apikeys⁠
• TZ - timezone of commit times. e.g. :Europe/Berlin

⁠Security

To avoid man in a middle attack it's recommended to provide known_hosts file. Default known_hosts contain keys for github.com, gitlab.com and bitbucket.org

⁠Custom Resources

Let's say we have a cluster with prometheus and certmanager, they register custom resources and we want to add them in backup.

Get list of custom resource definitions:

$ kubectl get crd

NAME                                    CREATED AT
alertmanagers.monitoring.coreos.com     2018-06-27T10:33:00Z
certificates.certmanager.k8s.io         2018-06-27T09:39:43Z
clusterissuers.certmanager.k8s.io       2018-06-27T09:39:43Z
issuers.certmanager.k8s.io              2018-06-27T09:39:44Z
prometheuses.monitoring.coreos.com      2018-06-27T10:33:00Z
prometheusrules.monitoring.coreos.com   2018-06-27T10:33:00Z
servicemonitors.monitoring.coreos.com   2018-06-27T10:33:00Z

Or get more useful output:

$ kubectl get crd -o json | jq -r '.items | (.[] | [.spec.names.singular, .spec.group, .spec.scope]) | @tsv'
alertmanager    monitoring.coreos.com  Namespaced
certificate     certmanager.k8s.io     Namespaced
clusterissuer   certmanager.k8s.io     Cluster
issuer          certmanager.k8s.io     Namespaced
prometheus      monitoring.coreos.com  Namespaced
prometheusrule  monitoring.coreos.com  Namespaced
servicemonitor  monitoring.coreos.com  Namespaced

Set env variables in container spec:

env:
  - name: EXTRA_GLOBAL_RESOURCES
    value: clusterissuer
  - name: EXTRA_RESOURCES
    value: alertmanager, prometheus, prometheusrule, servicemonitor, certificate, issuer

Special thanks to Pieter Lange for original idea⁠