Automatically snapshot and upload tars to s3
3.5K
Config, settings and documentation for the creation and management of the Flowmoco Kubernetes cluster.
Please ensure the following is installed in order to successfully use kops
brew update && brew install kops. See for details https://github.com/kubernetes/kopsaws configure --profile kops, inputting the keys and selecting json as the output format to make your life easier later on.This repo is used to create and manage K8s via Kops. To use Kops commands first we must set some environment variables. This is done by running the following
source environment.sh
You can create an empty repo simply by loading the config from the Kops file like so.
# Set your environtment variables
source environment.sh
# Create the empty cluster
kops create -f cluster.yaml
# Add the public key for the admin user as a secret
kops create secret sshpublickey admin -i ~/.ssh/id_rsa.pub
# Actually create the resources on AWS
kops update cluster --yes # Cluster name is saved as environment variable so no need to pass.
To start dashboard and monitoring daemons
# from root of repo
kubectl create -f kube-system/
# to connect to dashboard
kubectl proxy
# and navigate to http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
To start the detailed monitoring processes run
kubectl apply -f kube-prometheus/
# It can take a few seconds for the above 'create manifests' command to fully create the following resources, so verify the resources are ready before proceeding.
until kubectl get customresourcedefinitions servicemonitors.monitoring.coreos.com ; do date; sleep 1; echo ""; done
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f kube-prometheus/ # This command sometimes may need to be done twice (to workaround a race condition).
To forward ports you can run one or all of the following to connect.
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
kubectl --namespace monitoring port-forward svc/grafana 3000
kubectl --namespace monitoring port-forward svc/alertmanager-main 9093
The default grafana password is admin:admin. This is changed on first launch.
The ingress controller creates an application load balancer for AWS. Create the service.
kubectl apply -f alb-ingress-controller/
Now let's start an example service to test the new controller.
kubectl apply -f alb-ingress-controller/examples/
External DNS project. To install run the following.
kubectl apply -f external-dns/
To test you can run the following, which will create a load-balancer and Nginx.
kubectl create -f external-dns/examples/service-example-1.yaml
In order to start ceph you may need to run this command multiple times to avoid race conditions.
kubectl apply -f rook-ceph/
# Look out for errors and retry
kubectl apply -f rook-ceph/
Maybe you want to test out the block storage volumes by running the following.
kubectl apply -f rook-ceph/exampels/mysql.yaml
It uses a persistent volume claim to ask for storage and mounts and uses that storage in a way that can be failed over in triple redundancy.
Don't ask me how I managed to get it working in the end. I think it's included within kops now and would be installed and set up automatically. I did discover that using IAM roles is rubbish and not quite what we want. Instead I reverted to using AIM user levels and groups.
Groups and permissions can be (and typically are) set up per namespace basis by creating kubernetes Roles and RoleBindings.
In order to configure which users have which groups you need to edit kubesystem/aws-aim-authenticator.yaml and add the users at the end. The userARN is the aws IAM userARN.
In order to refresh after changes perform the following...
# apply new config
kubectl -n kube-system apply -f kube-system/aws-iam-authenticator.yaml
# Visually validate AIM daemon is running pods properly on masters
kubectl -n kube-system get pods -l k8s-app=aws-iam-authenticator
# Restart all pods
kubectl -n kube-system delete pods -l k8s-app=aws-iam-authenticator
You must have the following installed in order to connect to the cluster and to develop locally
You can install by going to https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html and following the instructions
You also need to log in, and you can do that by creating a new profile, which is the default.
# configure profile
aws configure --profile default
# switch profile
export AWS_DEFAULT_PROFILE=default
Check if it works try listing buckets.
aws s3 ls
Download and install Docker Community Edition for mac or linux from https://www.docker.com
On Docker for mac go to docker in toolbar > preferences > kubernetes and enable.
Install https://github.com/kubernetes-sigs/aws-iam-authenticator in your path so you can run aws-iam-authenticator token -i cluster.flowmo.cloud to get a token (this will error for now until cluster.flowmo.cloud is configured in kubectl.)
The installation steps mentioned on the readme 'go get...' failed as the asset was a 404 page on GitHub. Installation had to be done via a manual download of the binary asset from the releases page: https://github.com/kubernetes-sigs/aws-iam-authenticator/releases
More info needed here...
You can save the current state of the kops cluster by running the following command.
kops get -o yaml > cluster.yaml
kops create cluster --zones eu-west-2a,eu-west-2b,eu-west-2c --master-zones eu-west-2a,eu-west-2b,eu-west-2c --admin-access 82.69.103.158/32 --ssh-access 82.69.103.158/32 --cloud-labels Owner=flowmoco,ClusterName=cluster.flowmo.cloud --encrypt-etcd-storage --master-size t3.small --node-size t3.micro --node-count 1 --master-count 3
Content type
Image
Digest
Size
230.8 MB
Last updated
about 6 years ago
docker pull flowmoco/ceph-automated-backup