mesoform/concierge-consul repository overview

⁠Role to create a Concierge managed application

⁠Using this role

Generally this role should be wrapped by a top level playbook for generating application config⁠ to be built into the image. However, if ran manually, there is a concierge-tests.yml file which can be passed to ansible-playbook but many of the required variables and files will need to be passed at runtime.

Primarily the role simply pulls all of your application variables and scripts together into a set Ansible tasks used for performing system and integration tests on a running container which includes your application.

CURRENTLY, there are no required variables to run this as a standalone playbook but as we develop more tests there may be some which are. These will be able to be set using ansible-playbook --extra-vars some_key=some_value.

⁠Set environment variables

export DOCKER_HOST=tcp://lab3:2376 export DOCKER_TLS_HOSTNAME=tcp://lab2:2376 export DOCKER_TLS=yes export DOCKER_CERT_PATH=~/.docker/contexts/tls/06f49a4ba48aeee06f338b503add8763c7d8552ba19575d3c50a5e9a41e91a4f/docker/ export IMAGE_VER="0.7"

⁠Secure access with ACL system

https://learn.hashicorp.com/tutorials/consul/access-control-setup-production⁠

⁠Just build

ansible-playbook -v main.yml -t build_image

⁠Check Dockerfile template

ansible-playbook -v main.yml -t create_build_script

⁠Just deploy

ansible-playbook -v main.yml -t deploy_resources

⁠Check logs

$ docker service logs -f consul-server            
consul-server.1.eq6acv2doxex@lab2    | ==> Starting Consul agent...
consul-server.1.eq6acv2doxex@lab2    |            Version: '1.9.1'
consul-server.1.eq6acv2doxex@lab2    |            Node ID: 'fd7f15ae-a4f1-19aa-d165-b26a163b59cf'
consul-server.1.eq6acv2doxex@lab2    |          Node name: 'ba7e9a231812'
consul-server.1.eq6acv2doxex@lab2    |         Datacenter: 'euw0' (Segment: '<all>')
consul-server.1.eq6acv2doxex@lab2    |             Server: true (Bootstrap: true)
consul-server.1.eq6acv2doxex@lab2    |        Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 53)
consul-server.1.eq6acv2doxex@lab2    |       Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302)
consul-server.1.eq6acv2doxex@lab2    |            Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
consul-server.1.eq6acv2doxex@lab2    | 
consul-server.1.eq6acv2doxex@lab2    | ==> Log data will now stream in as it occurs:
consul-server.1.eq6acv2doxex@lab2    | 
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:31.230Z [WARN]  agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:31.297Z [WARN]  agent: bootstrap = true: do not enable unless necessary
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:32.034Z [WARN]  agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:32.039Z [WARN]  agent.auto_config: bootstrap = true: do not enable unless necessary
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.592Z [INFO]  agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:fd7f15ae-a4f1-19aa-d165-b26a163b59cf Address:172.17.0.2:8300}]"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.652Z [INFO]  agent.server.raft: entering follower state: follower="Node at 172.17.0.2:8300 [Follower]" leader=
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.833Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: ba7e9a231812.euw0 172.17.0.2
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.847Z [INFO]  agent.server.serf.wan: serf: Attempting re-join to previously known node: 5bcbfe4f9e45.euw0: 172.17.0.4:8302
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.857Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: ba7e9a231812 172.17.0.2
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.859Z [INFO]  agent.server.serf.lan: serf: Attempting re-join to previously known node: 5b503f14bdcc: 172.17.0.2:8301
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.860Z [INFO]  agent.router: Initializing LAN area manager
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.894Z [INFO]  agent.server: Handled event for server in area: event=member-join server=ba7e9a231812.euw0 area=wan
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.912Z [INFO]  agent.server: Adding LAN server: server="ba7e9a231812 (Addr: tcp/172.17.0.2:8300) (DC: euw0)"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.929Z [INFO]  agent: Started DNS server: address=0.0.0.0:53 network=tcp
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:33.942Z [INFO]  agent: Started DNS server: address=0.0.0.0:53 network=udp
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:34.109Z [INFO]  agent.server.serf.lan: serf: Re-joined to previously known node: 5b503f14bdcc: 172.17.0.2:8301
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:34.286Z [WARN]  agent.http.ui_server: ui_config.dir does not contain an index.html. Index templating and redirects to index.html are disabled.
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:34.397Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:34.472Z [WARN]  agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:34.699Z [INFO]  agent: started state syncer
consul-server.1.eq6acv2doxex@lab2    | ==> Consul agent running!
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:37.132Z [INFO]  agent.server.serf.wan: serf: Attempting re-join to previously known node: 5b503f14bdcc.euw0: 172.17.0.2:8302
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:37.376Z [INFO]  agent.server.serf.wan: serf: Re-joined to previously known node: 5b503f14bdcc.euw0: 172.17.0.2:8302
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:42.567Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:42.938Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader=
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:42.945Z [INFO]  agent.server.raft: entering candidate state: node="Node at 172.17.0.2:8300 [Candidate]" term=2
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.201Z [INFO]  agent.server.raft: election won: tally=1
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.201Z [INFO]  agent.server.raft: entering leader state: leader="Node at 172.17.0.2:8300 [Leader]"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.261Z [INFO]  agent.server: cluster leadership acquired
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.370Z [INFO]  agent.server: New leader elected: payload=ba7e9a231812
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.913Z [INFO]  agent.leader: started routine: routine="federation state anti-entropy"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.913Z [INFO]  agent.leader: started routine: routine="federation state pruning"
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:43.923Z [INFO]  agent.server: member joined, marking health alive: member=ba7e9a231812
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:44.221Z [INFO]  agent.server: federation state anti-entropy synced
consul-server.1.eq6acv2doxex@lab2    |     2020-12-31T10:45:45.853Z [INFO]  agent: Synced node info

⁠Check member status

$ docker exec -it $(docker container ls -qf volume=/data) consul members -detailed
Node          Address          Status  Tags
ba7e9a231812  172.17.0.2:8301  alive   acls=0,bootstrap=1,build=1.9.1:ca5c3894,dc=euw0,ft_fs=1,ft_si=1,id=fd7f15ae-a4f1-19aa-d165-b26a163b59cf,port=8300,raft_vsn=3,role=consul,segment=<all>,vsn=2,vsn_max=3,vsn_min=2,wan_join_port=8302

⁠Docker Compose files

This playbook makes use of the multiple compose files principle⁠ as well as the extends keyword for extending services. It will autogenerate a docker-compose-override.yml file which will automatically be read by docker-compose up and can be used for integration tests. It will also create a docker-compose-units.yml file which can be passed in with -f docker-compose-units.yml and used for running unit tests of your application and ran without the containerpilot entrypoint. The general idea behind these tests is you will have another file where you define your specific test requirements defined in a service named the same as your application service and makes use of the extends keyword for defining specific commands (e.g scripts) for your tests

⁠Testing the playbook

Add a test tag to the tasks you want to test and run ansible-playbook -vv concierge-tests.yml -t test

However, because it's generally unlikely this role will be used on its own much, and because it is so easy, we recommend testing whilst integrated as a submodule⁠

⁠Troubleshooting

"msg": "Error: Docker SDK for Python version is 1.10.6 (gMacBookPro's Python /usr/local/Cellar/ansible/2.9.11/libexec/bin/python3.8). Minimum version required is 2.6.0.Try `pip uninstall docker-py` followed by `pip install docker`."

Ansible is using its own virtual environment, so this needs to be specified. I.e.

/usr/local/Cellar/ansible/2.9.11/libexec/bin/python3.8 -m pip uninstall docker-py 
/usr/local/Cellar/ansible/2.9.11/libexec/bin/python3.8 -m pip install --upgrade docker 

----------------------------------------------------------------------------------------------------------------- Template by Mesoform Limited⁠

⁠Some older docs

⁠Role to create an image for a Concierge managed application

⁠Introduction

The Concierge Paradigm⁠ is an extension of The Autopilot Pattern⁠, and is a rich and powerful method of automating the management of running containers by simply using a service discovery system like Consul, and an event management system, like Zabbix. By using these, already well developed systems, you gain incredible control and information about the state of the system as a whole and fine-grained detail of all applications.

A concierge managed application is one that fits naturally into this concierge environment and automatically registers itself for discovery, monitoring and scheduling. This playbook asks only a few simple questions about your application and the environment in which you expect to run it in, then spits out a Docker image at the end and performs the required system and integration tests to be Concierge managed and any custom tests you require for your application.

⁠About this role

Primarily the role generates a Dockerfile, a set of Docker Compose files and a Containerpilot⁠ file. Then builds an image and runs a set of tests against the build. It wraps up some other common roles for creating our Docker images ready to be used in a Concierge Paradigm environment. The role has been split into 4 parts:

1. concierge-app-playbook: This repository. The purpose of which is to get you your own custom repository setup to start building your application container
2. create-concierge-app: This submodule role takes the variables, scripts and any files needed for your application and constructs the necessary application configuration files (if using templates) and orchestration files for managing the lifecycle of your application.
3. create-concierge-image: Constructs your Dockerfile and builds your image.
4. create-concierge-tests: Performs basic system tests, integration to service discovery and event management. Plus any user-defined application tests

The Dockerfile has some default attributes automatically set and allows for others to be included by creating the required lists or variables.

Currently these are as follows:

• os_distro (string) = The flavour of operating system. Current options are alpine and debian - versions 3.4 and jessie, respectively
• install_scripts (list) = the location of the script or scripts to install the application you want to package into the image. A list is used so as to logically separate different install steps into separate RUN commands and better make use of UnionFS image layers
• build_args (list) = a list of additional Docker ARG options for required variables when building
• container_vars (list) = a list of environment variables which will be set as defaults inside the running container. E.g. container_vars: FOO=bar
• env_vars (list) = a list of additional environment variables which will be passed to the container at runtime. E.g. env_vars: FOO=bar
• labels (list) = a list of additional labels to add to the container
• ports (list) = a list of ports to expose
• service_port (integer) = the port of your application which will be registered with with the service discovery system so that downstream services know which port to use when communicating with your application. service_port is a required field if you want your application to register as a service in Consul and be discoverable
• volumes (list) = a list of volume the container should create
• entrypoint (string) = process or script to run as ENTRYPOINT. For the concierge containers, it is assumed that unless you're creating a base image, this will always be containerpilot and already set in the base image.
• command[REQUIRED] (string) = the command used to start your application. This will be executed as part of a Containerpilot job.
• app_config_dir (string) = the directory where you application's configuration files will reside. Currently this must be in /etc. Default = /etc/{{ project_name }}
• custom_orchestration_dir = the location where you want your custom application orchestration config template to output to. This is not container orchestration (e.g. docker-compose.yml) but how you want to orchestrate your application (e.g. containerpilot.json). The default is /etc inside your container.
• upstreams (list) = a list of service names, registered in Consul and which your application depends upon
• downstreams (list) = a list of clients which are registered in Consul as a service; and your application may want to configure access for
• test_script (string) = if you want to perform any kind of unit tests of your application, set this value to the location of that script. The playbook will then use this to create a docker-compose-units.yml file and set the entry point to this script. **This will completely override the startup, so your script must include starting your application in the way expected.

See vars/main.yml and defaults/main.yml for others variables and their descriptions and any non-declared container orchestration options like mem_limit (defaults to 128MB)are best updated in the compose files at the end.

⁠Submodules

Within this playbook there are some additional roles included as git submodules. These modules are synchronised with their upstream repositories every time you run the playbook and any changes you made locally will be stashed. Every effort is made to make these submodule roles backward compatible but sometimes things accidents happen and sometimes its just not feasible. We've added some output messages to indicate that changes have happened but also advise that you watch the included each roles'repositories as well.

If you want to run the playbook without doing this, either remove the relevant entry from .gitmodules or run with --skip-tags=update_submodules. Likewise, if you are pulling your own repository, which is generated from this one, and you just want to update just the submodules, run ansible-playbook app.yml -t update_submodules.

⁠Service discovery

Service discovery as a subject is beyond the scope of this documentation but if you're choosing to use this repository it's because you value it. What you also need to know is that we choose to use active discovery⁠ and chose to use Consul⁠ to perform that function. We aren't opinionated about using Consul but currently this playbook is. Partly this is due to time writing the code but also, we do genuinely believe Consul is the best product on the market for this, right now.

The use of upstreams variable will set some conditions in your orchestration. Firstly, when starting up, your application will wait on these services being marked as healthy if you have a pre_start job; and will reload when any changes happen to these services (reload variable is required to have a value of the command to reload the application)

Some other things you need to know:

1. If you already have a service discovery or event management system running and you want to test against these, instead of spinning up new container instances, you can set svc_discovery_server and/or event_management_server variables to the server addresses and you'll configure your container to use these and won't get additional container starting up.
2. We've written this code in such a way as to use DNS search domains so that containers and services can more easily be recognised within your domains. For example, if you set dns_domain=mesoform.com, oaas_domain=svc.mesoform.com and svc_discovery=consul then the Consul agent running in your container will look for your service discovery system with the following addresses consul, consul.mesoform.com and consul.svc.mesoform.com.
3. Therefore, if you have a system already running and in place that is reachable on the address service-discovery.ops.myservices.internal you can set svc_discovery_server to service-dicovery or service-discovery.ops.myservices.internal. Then, as long as dns_domain or oaas_domain match ops.myservices.internal, normal DNS searching will get your requests to the right place
4. You don't have to have a full blown discovery system running to build your application container image. If you don't set a value for svc_discovery, then the consul agent running inside the container will simply start in development mode so that even if your job is a service, it will still run as expected.
5. If you set svc_discovery=consul, you will also get service defined in the file docker-compose-integrations.yml. This will allow you to perform a proper integration test and see if your application registers as a service correctly. See Standard Integration tests below⁠

⁠Event management

Likewise, event management and monitoring (EMM) is a large subject an beyond the scope of this documentation. The Concierge Paradigm makes use of already well-defined systems in this area to improve the scheduling of our systems. For example, if our EMM system has metrics from our platform, application, database and infrastructure all together we have a much more powerful tool for auto-scaling our apps. This means we can use metrics from applications and databases, we can use predictive scaling and we have a powerful tool for debugging.

Again, we're not opinionated about which system you use, but currently code uses Zabbix⁠. We like Zabbix because it is an enterprise-grade, flexible and open-source monitoring solution with a really powerful event management system which we can use for scheduling our containers.

All the points above in the Service discovery⁠ section are relevant for hooking into EMM automatically. For example, if event_management is defined, the monitoring agent will be started and registration and container heartbeats will be sent to the address specified in the value. The same goes with the DNS points, so if event_management is set to event_management=zabbix and oaas_domain=svc.mesoform.com dns_domain=mesoform.com, your container will look for the EMM at zabbix, zabbix.svc.mesoform.com and zabbix.mesoform.com.

(WIP) You will also get a service definition for our pre-configured EMM in docker-compose-integrations.yml

⁠Setting up

⁠Environment variables

Ansible default output isn't great when trying to debug large data output so firstly set ANSIBLE_STDOUT_CALLBACK

export ANSIBLE_STDOUT_CALLBACK=yaml

⁠Clone the repository

cd {{ roles_dir }}
mkdir my-app-name # Only use hyphens, not underscores because this is used as the service name in our service discovery system
cd my-app-name
git clone https://github.com/mesoform/configure-concierge-app.git .

⁠create your project repository

In Github, Bitbucket or whatever system you like and copy the URL to your clipboard because you'll need it when you...

⁠Run the setup script to set up the playbook for your application

./setup.sh  --initialise-git

This will initialise and pull down the submodules, set some defaults for your project and perform an initial commit. NOTE: --initialise-git is only needed once, after you first clone the repository.

⁠Add custom files to the right directories

⁠Custom application scripts

Any scripts to be used as part of your application deployment can be added to {{ playbook_dir }}/files/bin and will be automatically copied to /usr/local/bin on the container. You can find an example scripts already in this directory.

⁠Custom application configuration

Any Jinja2 templates added to {{ playbook_dir }}/templates/app with the .j2 extension will automatically be processed and copied to files/etc/{{ project_name }} where they will be uploaded to the application configuration directory (default = /etc/{{ project_name }}). You can find an example of one already in the directory. Even if your files need no processing, simply drop the basic files in this directory with a .j2 extension and they will be copied to you application configuration directory.

You can add any static scripts and configuration to any other directory in the container by simply adding the full path, relative to the root directory where you want the file to end up at, beneath the files/ directory. For example, If you want something copied to /usr/libexec/bin inside the container, you will need to create files/usr/libexec/bin/some_file.

⁠Custom application configuration templates

If you need to process any Jinja templates specific to your application which don't belong in the standard application configuration directory (default: /etc/{{ project_name }}), copy your template (with a .j2 extension) into the relative path in the templates directory and it will be processed into the matching path in your container. E.g. if you want to process a template into /etc/mail/spamassassin/local.cf, create templates/etc/mail/spamassassin/local.cf.j2

⁠Custom application tests

Not implemented but this will be where to manually add or templates will be copied to for tests. These will be copied to /tmp in the container

⁠Custom application orchestration

Any Jinja2 templates added to {{ playbook_dir }}/templates/orchestration with the .j2 extension will automatically be processed and copied to files/etc/ where they will be uploaded to the application orchestration directory (default = /etc). You can find an example of one already in the directory. If you want the file to be copied to a different location, set custom_orchestration_dir to a path value relative to files/etc/ (e.g. custom_orchestration_dir=files/etc/my_orchestration).

⁠Custom application test templates

Not implemented Any Jinja2 templates added to {{ playbook__dir }}/templates/tests with the .j2 extension will automatically be processed and uploaded to the application orchestration directory (default = /etc). You can find an example of one already in the directory

⁠Configure any variables you need

{{ playbook_dir }}/vars

⁠Run the playbook

• Run the playbook: ansible-playbook -v app.yml
• List all of the playbook tags: ansible-playbook --list-tags app.yml
• Skip updating the upstream submodules: ansible-playbook -v --skip-tags=update_submodules app.yml
• View al