nuancemobility/data-volume-cleaner repository overview

This process (docker image) runs in a kubernetes cluster by default as a daemonset, which means on each nodes by default. It's default configuration can be overwritten in it's kubernetes manifest file by either using environment variables or command line arguments.

Clean up Logic Overview

This process is meant to run on each node and perform cleanup of expired user data that gets mounted under a specific folder.

The main cleanup logics are around the log folders for the different podIds as well as namespace folders. The default location and structure is the following

/var/lib/docker/volumes/nuance//logs// /var/lib/docker/volumes/nuance///logs//

Namespace folder cleanup

The cleaner will loop over all folders and check in k8s if the same namespace exists. If it does, it will leave it alone and move to the next namespace folder on disk. If it doesn't, it will check it's last modified date, walking through sub-folders as well, and compare it to its namespaceExpiryPeriod config. If the folder is marked as expired it will be deleted.

Namesapce sub-folders cleanup

Once all namespace folders have been analyzed, the namespace direct sub-folders are entered by the cleaner tool and a new analysis is performed on them. The cleaner tool uses patterns (eg "*./logs") to select which namespace sub-folders to process. The patterns are relative to the namespace folder or to the pod folder directly under the namespace folder. The default is to lookup for pods. Whether pods are looked up and processed is configured per pattern. If the cleaner is configured to process pods, the same logic as for namespace is used: PodIds are queried in kubernetes and are removed if absent from kubernetes and expired.

Configuration per namespace through K8S Config Map

The tool uses the dvc Config Map containing the data-cleanup-policies key, as in the sample Sample DVC config map. The config map makes it possible to specify

set of patterns to match directories and expiry duration per set of patterns whether to lookup for pods per set of patterns

The config map can be create with kubectl, eg

kubectl -s 12.34.56.78:8080 --namespace create configmap dvc --from-file=data-cleanup-policies=DVCConfigMap.sample.yaml Note how data-cleanup-policies is specified.

If the dvc Config Map exists and contains the data-cleanup-policies key, podsParentPathPatterns and podExpiryPeriod are ignored.

Folder patterns

The folders to be check for expiry can be specified using patterns in the Config Map described in the previous section, or in the podsParentPathPatterns parameter. The patterns are relative to the namespace directory being processed. The patterns are separated using the OS separator (":" on Linux)

Expiry check frequeny

The cleaner divides the shortest expiry duration configured for namespace and pods by checksPerExpiryDuration to determine how often to check for expiry. The greater checksPerExpiryDuration, the more often the checks are performed.

Eg with

podExpiryPeriod=10h checksPerExpiryDuration=5 expiry checks are performed every two hours

Service Information

Property list (with default values)

Usage of /cleaner: -apiserverAddress="": http address for the k8s apiserver (eg http://10.56.10.104:8080⁠) -namespaceExpiryPeriod=24h0m0s: duration after which to consider a namespace directory expired for deletion -podsParentPathPatterns="logs:*/logs": list of patterns to select which folders are inspected for expired data -podExpiryPeriod=96h0m0s: duration after which to consider a pod directory expired for deletion -checksPerExpiryDuration=10: specifies how many times expiry is checked per expiry duration -rootPath="/var/opt/nuance/kubernetes/volumes": root path for the container data volumes