CoreOS and FreeIPA provisioning on DigitalOcean
683
This project uses Ansible to automate bootstrapping a Kubernetes cluster around FreeIPA infrastructure, with cluster nodes running CoreOS Container Linux on DigitalOcean droplets. It also installs a number of basic containerized services, including email and web.
FreeIPA provides core services essential to clusters, such as DNS, user/host authentication/authorization, and SSL certificate authorities. A Kubernetes cluster built on this infrastructure orchestrates various containerized services, including FreeIPA itself, simplifying service management across the cluster. Kubernetes runs in CoreOS Container Linux, a minimal OS that might be seen as a cluster node appliance requiring minimal maintenance. And the OS runs on DigitalOcean droplets for inexpensive, automated provisioning on the cloud. Provisioning and configuration management is automated at all levels with Ansible, and a full cluster may be bootstrapped and running within an hour by issuing a few simple commands.
This project means to automate setup of a basic Internet site with self-hosted email, web and phone system. Service is redundant where possible to increase reliability. The components are chosen to scale up, but the down-scaled, three-node bootstrap configuration is complete, and meant to be ready for production.
Although the goal is production readiness, this is unfinished and experimental work. It is not represented to be fit for any purpose. Readers are strongly advised to contemplate the risks before using this work in any critical scenario.
At present, the Ansible playbooks can complete the following tasks for a 3-node cluster with no manual intervention. The end result is a running Kubernetes cluster, still with a few minor known issues.
etcdctl client on nodeskubectl client with TLS client certsInitial site setup (only run once):
# Copy `hosts.yaml` template and edit
cp hosts.yaml.example hosts.yaml
# You will certainly want to edit:
# - domain_name and kerberos_realm
# - network_prefix: each host, first 2 octets MUST BE unique
# You might want to edit:
# - host names: host1 etc.
# - additional hosts: minimum 3 hosts
# - defaults: size_id (default 1gb), region_id (nyc1)
$EDITOR hosts.yaml
# Optionally build container if not pulling from docker hub
./container -b
# Start shell in container; the following `ansible-*` commands
# should all be run in the container
./container
# Set up password vault once; will prompt for FreeIPA admin
# and directory passwords
ansible-playbook playbooks/init-site.yaml
Install whole cluster in one command:
ansible-playbook playbooks/site.yaml
Delete a node or the whole cluster
# Destroy host1
ansible-playbook playbooks/destroy.yaml -e confirm=host -l host1
# Destroy whole cluster
ansible-playbook playbooks/destroy.yaml -e confirm=all
Misc commands:
# Re-collect facts about host
ansible host1 -m setup \
-e ansible_python_interpreter=/home/core/bin/python \
-e ansible_ssh_user=core
# List all variables for a host
ansible host1 -m debug -a "var=hostvars[inventory_hostname]"
# Also vars, environment, group_names, groups
Ansible ipa_* tests:
env \
PYTHONPATH=$(pwd)/lib \
IPA_HOST=host1.example.com \
IPA_USER=admin \
IPA_PASS=mysecretpw \
IPA_DOMAIN=example.com \
IPA_NSRECORD=host1.example.com. \
nosetests -v test/units/modules/identity/ipa/
Run nosetests in ansible repo
PYTHONPATH=$(pwd)/lib nosetests -v test/units/modules/identity/ipa/Run etcdctl with SSL:
cd /media/state/etcd
etcdctl --endpoint=https://127.0.0.1:2380 \
--ca-file=ca.pem --cert-file=client.pem --key-file=client-key.pem \
cluster-health
Run shell in running container:
ssh -t core@$HOST0 docker exec -it ipa bash
FreeIPA information commands
# List all servers and replicas
ipa-replica-manage list
# List agreements for a server
ipa-replica-manage list host1.example.com
Querying LDAP needs SASL auth mech explicitly defined
docker exec -it ipa ldapsearch -H ldaps://host1.example.com -Y GSSAPI
Run shell in ipaclient container, ready to run emacs
docker exec -it --detach-keys ctrl-^ ipaclient env TERM=screen bash
Misc. kubernetes commands
kubectl config *
kubectl cluster-info
kubectl get cs
kubectl --namespace=kube-system get pods
kubectl --namespace=kube-system describe pods kube-dns-v20-d7crm
kubectl --namespace=kube-system logs kube-dns-v20-d7crm kubedns
kubectl --namespace=kube-system replace --force -f var/k8s/dns-addon.yaml
kubectl --namespace=kube-system delete pods kube-dns-v20-d7crm
kubectl --namespace=kube-system port-forward kubernetes-dashboard-v1.6.0-xcgh7 9090
kubectl --namespace=kube-system exec kube-dns-v20-jh9sb -c kubedns -- nslookup host1
kubectl --namespace=kube-system describe pods kube-dns-v20-jh9sb
Ansible automates the bootstrapping before anything else.
At this time, Ansible has an enormous number of modules that handle 90% of our needs. The missing 10% primarily handle FreeIPA object classes that the roles in this repo use extensively, such as the SSL-related objects CA, CA ACL, certificate and service, and also DNS zones and records. There is also a parted module copied from upstream with bugfixes.
It also turned out to simplify playbooks to write a number of filter plugins, some for specific purposes and some general.
ansible.cfg: fact_path = /home/centos/ansible_facts.dFirst step is to provision DigitalOcean droplets with CoreOS image.
Next step is to install and configure FreeIPA in a Docker container.
Man-pages:
Docs:
certutilCoreOS Container Linux must be configured for clustering with etcd3 and Kubernetes. This depends on FreeIPA being bootstrapped.
The initial node will be bootstrapped with a static cluster configuration and no TLS. TLS will be introduced after FreeIPA is running and DNS and the CA are available.
The etcd cluster will be bootstrapped from DNS with the
--discovery-srv flag.
All communication will be over TLS with certs generated from FreeIPA (in a dedicated sub-CA) and monitored with certmonger.
Basic configuration:
SSL on CoreOS:
Launching containers in CoreOS:
IPA integration
Kubernetes depends on Flannel networking.
Install Kubernetes after the Container Linux cluster is configured.
Ansible examples incl. etcd2, docker in kubernetes
Kubernetes is replacing fleet
Other projects to set up Kubernetes on CoreOS with Ansible
Kubernetes module in Ansible
HAProxy for load balancing, but really for reverse-proxying multiple web services on a single IP.
These should be added to automation
Connect replica servers to eliminate host1 as SPOF
ipa-replica-manage connect host2.example.com host3.example.com
Create ipa sidekick /etc/resolv.conf service to install
FreeIPA/Google DNS servers at start/stop
DO isn't the cheapest anymore, and Dogtag struggles in a $10/month 1gb droplet.
Possibilities:
Content type
Image
Digest
Size
51.9 MB
Last updated
about 9 years ago
docker pull zultron/freeipa-cloud-prov:dnsmasq