mesoform/concierge-debian-base-image repository overview

⁠About

• Uses http://autopilotpattern.io⁠ and our Concierge Paradigm⁠ to keep orchestration, load balancing, scheduling and monitoring application-centric.
• using Zabbix event management platform allows us to auto-scale applications based on reported metrics from the agent running inside the container
• Just give the Zabbix Server access to running docker-compose against your Docker API
• Aggregate metrics⁠ across instances and create highly intelligent trigger expressions⁠ that upon firing run docker-compose scale service=N+1
• Even predictive trigger functions⁠ that could pre-scale your service in preparation of forthcoming load
• Have a metric-centric view of system as a whole
• Perform vertical scaling on some metrics and horizontal scaling on others
• Keeps humans in the loop by keeping automation tasks and even callouts (e.g. to Slack) centralised with alerts. Scaling can more easily be associated with issues.
• Create cross component triggers for scaling. E.g. queue size has increase isn't always a good reason to scale - it could be because the upstream DB has an issue
• Discover docker volumes and interface stats automatically with Zabbix Low-Level Discovery⁠
• Separation of concerns around container monitoring. Developers can focus on application metrics and SysAdmins can still easily monitor OS.
• Metrics can be added on-the-fly without having to redeploy
• Enterprise grade monitoring platform already have well defined means to monitor off-the-shelf applications and databases
• Automatically discover metrics from known application frameworks/libraries like Dropwizard⁠, Spring⁠ and Prometheus⁠
• Capture metrics as a snapshot so as not to suffer with windowing issues when comparing
• Collect high-volumes of metrics with almost no impact
• Gain advantages of both "push" and "pull" based metrics collection
• provide a service discovery function out-of-the-box by running a Consul agent
• provide a sidecar proxy service to the installed application by running a Consul Connect (WIP)

⁠How do I get set up?

Rebuild FROM {{ this.image }} with your own containerpilot.json, zabbix_agentd.conf (or other event management process) by placing them in /etc from your build directory so ONBUILD⁠ hook will pick them up (ONBUILD COPY etc /etc). Then COPY application binary (ONBUILD COPY bin /bin) and pass start script or binary path using CMD

⁠Dockerfile example:

FROM mesoform/concierge-debian-base-image
CMD ["/usr/local/bin/my_app_binary", "-a", "parameter"]

⁠The build directory

Should look as follows:

├── Dockerfile
├── bin/
|   └── my_app_binary 
└── etc/
      ├── app/
      |   └── my_app.conf
      ├── containerpilot.json
      └── coprocesses/
          └── zabbix/
             └── zabbix_agentd.conf

• bin/ for your binaries.
• etc/ is all copied up automatically using ONBUILD hooks

⁠Containerpilot.json example:

{
  "consul": "{{ if .CONSUL_AGENT }}localhost{{ else }}{{ if .CONSUL }}{{ .CONSUL }}{{ else }}consul{{ end }}{{ end }}:8500",
  "preStart": ["/usr/local/bin/setdomain.sh"],
  "preStop": ["zabbix_sender",
    "-c", "/etc/coprocesses/zabbix/zabbix_agentd.conf",
    "--key", "container.state",
    "--value", "removing"],
  "postStop": ["zabbix_sender",
    "-c", "/etc/coprocesses/zabbix/zabbix_agentd.conf",
    "--key", "container.state",
    "--value", "exited"],
  "logging": {
    "level": "{{ .LOG_LEVEL }}",
    "format": "{{ .LOG_FORMAT }}"
  },
  "services": [
    {
      "name": "hello-world",
      "port": 80,
      "health": "ping -c 1 -q www.bbc.co.uk",
      "poll": 10,
      "ttl": 30,
      "tags": ["hello", "world"],
      "consul": {
        "enableTagOverride": true,
        "deregisterCriticalServiceAfter": "90m"
      }
    }
  ],
  "tasks": [
    {
      "name": "scheduling_status",
      "command": ["zabbix_sender",
        "-c", "/etc/coprocesses/zabbix/zabbix_agentd.conf",
        "--key", "container.state",
        "--value", "running"],
      "frequency": "10000ms",
      "timeout": "3000ms"
    }
  ],
  "coprocesses": [
    {
      "name": "zabbix_agent",
      "command": ["/usr/sbin/zabbix_agentd", "-fc", "/etc/coprocesses/zabbix/zabbix_agentd.conf"],
      "restarts": 3
    },
    {{ if .CONSUL_AGENT }}
    {
      "command": ["/usr/local/bin/consul-agent.sh"],
      "restarts": "unlimited"
    }{{ end }}
  ]
}

Alternatively you can look at using the concierge-app-playbook⁠ where the configuration code is already written and helps you get setup in a declarative manner.

⁠Zabbix Setup

⁠Create service host

• in zabbix with
  • ITEM = Service::MyApp::node::count grpsum["Service Discovery/MyApp/Mesoform DC1","agent.ping",last]
  • ITEM = populate deployment status inventory field
  • TRIGGER = Information MyApp nodes scaled to {ITEM.VALUE} {service.MyApp:grpsum["Service Discovery/MyApp/Mesoform DC1","agent.ping",last].diff()}=1
  • TRIGGER = add event tag - MyAppCount: {{ITEM.VALUE}.regsub("^([0-9]+)", \1)}
  • TRIGGER = Warning MyApp::load::high::{ITEM.VALUE} {service.MyApp:grpavg["Service Discovery/MyApp/Mesoform DC1","system.cpu.load[percpu,avg1]",last].last()}>0.8
  • TRIGGER ACTION = external script which runs "docker-compose scale MyApp={INVENTORY.DEPLOYMENT.STATUS}+1"

⁠Setup auto-registration for application

• add an auto-registration rule that adds you container to the correct host groups and templates
• this is done using dns_domain values passed to the container and picked up by zabbix_agent system.hostname

Example repository will follow and more details can be found on our Concierge Paradigm⁠ blog

⁠Support

• Please comment if you have any questions