Sign inSign up

gitopsorg/docker-devops-http-monitor

By gitopsorg

•Updated about 5 years ago

Image
0

67

gitopsorg/docker-devops-http-monitor repository overview

Gatus

build Go Report Card codecov Go version Docker pulls Follow TwinProduction

Gatus is a health dashboard that gives you the ability to monitor your services using HTTP, ICMP, TCP, and even DNS queries as well as evaluate the result of said queries by using a list of conditions on values like the status code, the response time, the certificate expiration, the body and many others. The icing on top is that each of these health checks can be paired with alerting via Slack, PagerDuty, Discord and even Twilio.

I personally deploy it in my Kubernetes cluster and let it monitor the status of my core applications: https://status.twinnation.org/⁠

Quick start
docker run -p 8080:8080 --name gatus twinproduction/gatus

For more details, see Usage⁠

⁠Table of Contents

⁠Why Gatus?

Before getting into the specifics, I want to address the most common question:

Why would I use Gatus when I can just use Prometheus’ Alertmanager, Cloudwatch or even Splunk?

Neither of these can tell you that there’s a problem if there are no clients actively calling the endpoint. In other words, it's because monitoring metrics mostly rely on existing traffic, which effectively means that unless your clients are already experiencing a problem, you won't be notified.

Gatus, on the other hand, allows you to configure health checks for each of your features, which in turn allows it to monitor these features and potentially alert you before any clients are impacted.

A sign you may want to look into Gatus is by simply asking yourself whether you'd receive an alert if your load balancer was to go down right now. Will any of your existing alerts by triggered? Your metrics won’t report an increase in errors if there’s no traffic that makes it to your applications. This puts you in a situation where your clients are the ones that will notify you about the degradation of your services rather than you reassuring them that you're working on fixing the issue before they even know about it.

⁠Features

Gatus dark mode

The main features of Gatus are:

  • Highly flexible health check conditions: While checking the response status may be enough for some use cases, Gatus goes much further and allows you to add conditions on the response time, the response body and even the IP address.
  • Ability to use Gatus for user acceptance tests: Thanks to the point above, you can leverage this application to create automated user acceptance tests.
  • Very easy to configure: Not only is the configuration designed to be as readable as possible, it's also extremely easy to add a new service or a new endpoint to monitor.
  • Alerting: While having a pretty visual dashboard is useful to keep track of the state of your application(s), you probably don't want to stare at it all day. Thus, notifications via Slack, Mattermost, Messagebird, PagerDuty and Twilio are supported out of the box with the ability to configure a custom alerting provider for any needs you might have, whether it be a different provider or a custom application that manages automated rollbacks.
  • Metrics
  • Low resource consumption: As with most Go applications, the resource footprint that this application requires is negligibly small.
  • GitHub uptime badges: Uptime 1h Uptime 24h Uptime 7d

⁠Usage

By default, the configuration file is expected to be at config/config.yaml.

You can specify a custom path by setting the GATUS_CONFIG_FILE environment variable.

Here's a simple example:

metrics: true         # Whether to expose metrics at /metrics
services:
  - name: twinnation  # Name of your service, can be anything
    url: "https://twinnation.org/health"
    interval: 30s     # Duration to wait between every status check (default: 60s)
    conditions:
      - "[STATUS] == 200"         # Status must be 200
      - "[BODY].status == UP"     # The json path "$.status" must be equal to UP
      - "[RESPONSE_TIME] < 300"   # Response time must be under 300ms
  - name: example
    url: "https://example.org/"
    interval: 5m
    conditions:
      - "[STATUS] == 200"

This example would look like this:

Simple example

Note that you can also use environment variables in the configuration file (e.g. $DOMAIN, ${DOMAIN})

If you want to test it locally, see Docker⁠.

⁠Configuration

ParameterDescriptionDefault
debugWhether to enable debug logsfalse
metricsWhether to expose metrics at /metricsfalse
storageStorage configuration{}
storage.fileFile to persist the data in. If not set, storage is in-memory only.""
servicesList of services to monitorRequired []
services[].nameName of the service. Can be anything.Required ""
services[].groupGroup name. Used to group multiple services together on the dashboard. See Service groups⁠.""
services[].urlURL to send the request toRequired ""
services[].methodRequest methodGET
services[].insecureWhether to skip verifying the server's certificate chain and host namefalse
services[].conditionsConditions used to determine the health of the service. See Conditions⁠.[]
services[].intervalDuration to wait between every status check60s
services[].graphqlWhether to wrap the body in a query param ({"query":"$body"})false
services[].bodyRequest body""
services[].headersRequest headers{}
services[].dnsConfiguration for a service of type DNS. See Monitoring a service using DNS queries⁠.""
services[].dns.query-typeQuery type for DNS service""
services[].dns.query-nameQuery name for DNS service""
services[].alerts[].typeType of alert. Valid types: slack, discord, pagerduty, twilio, mattermost, messagebird, customRequired ""
services[].alerts[].enabledWhether to enable the alertfalse
services[].alerts[].failure-thresholdNumber of failures in a row needed before triggering the alert3
services[].alerts[].success-thresholdNumber of successes in a row before an ongoing incident is marked as resolved2
services[].alerts[].send-on-resolvedWhether to send a notification once a triggered alert is marked as resolvedfalse
services[].alerts[].descriptionDescription of the alert. Will be included in the alert sent""
alertingConfiguration for alerting. See Alerting⁠.{}
securitySecurity configuration{}
security.basicBasic authentication security configuration{}
security.basic.usernameUsername for Basic authenticationRequired ""
security.basic.password-sha512Password's SHA512 hash for Basic authenticationRequired ""
disable-monitoring-lockWhether to disable the monitoring lock⁠false
skip-invalid-config-updateWhether to ignore invalid configuration update. See Reloading configuration on the fly⁠.
webWeb configuration{}
web.addressAddress to listen on0.0.0.0
web.portPort to listen on8080
⁠Conditions

Here are some examples of conditions you can use:

ConditionDescriptionPassing valuesFailing values
[STATUS] == 200Status must be equal to 200200201, 404, ...
[STATUS] < 300Status must lower than 300200, 201, 299301, 302, ...
[STATUS] <= 299Status must be less than or equal to 299200, 201, 299301, 302, ...
[STATUS] > 400Status must be greater than 400401, 402, 403, 404400, 200, ...
[STATUS] == any(200, 429)Status must be either 200 or 429200, 429201, 400, ...
[CONNECTED] == trueConnection to host must've been successfultrue, false
[RESPONSE_TIME] < 500Response time must be below 500ms100ms, 200ms, 300ms500ms, 501ms
[IP] == 127.0.0.1Target IP must be 127.0.0.1127.0.0.10.0.0.0
[BODY] == 1The body must be equal to 11{}, 2, ...
[BODY].user.name == johnJSONPath value of $.user.name is equal to john{"user":{"name":"john"}}
[BODY].data[0].id == 1JSONPath value of $.data[0].id is equal to 1{"data":[{"id":1}]}
[BODY].age == [BODY].idJSONPath value of $.age is equal JSONPath $.id{"age":1,"id":1}
len([BODY].data) < 5Array at JSONPath $.data has less than 5 elements{"data":[{"id":1}]}
len([BODY].name) == 8String at JSONPath $.name has a length of 8{"name":"john.doe"}{"name":"bob"}
has([BODY].errors) == falseJSONPath $.errors does not exist{"name":"john.doe"}{"errors":[]}
has([BODY].users) == trueJSONPath $.users exists{"users":[]}{}
[BODY].name == pat(john*)String at JSONPath $.name matches pattern john*{"name":"john.doe"}{"name":"bob"}
[BODY].id == any(1, 2)Value at JSONPath $.id is equal to 1 or 21, 23, 4, 5
[CERTIFICATE_EXPIRATION] > 48hCertificate expiration is more than 48h away49h, 50h, 123h1h, 24h, ...
⁠Placeholders
PlaceholderDescriptionExample of resolved value
[STATUS]Resolves into the HTTP status of the request404
[RESPONSE_TIME]Resolves into the response time the request took, in ms10
[IP]Resolves into the IP of the target host192.168.0.232
[BODY]Resolves into the response body. Supports JSONPath.{"name":"john.doe"}
[CONNECTED]Resolves into whether a connection could be establishedtrue
[CERTIFICATE_EXPIRATION]Resolves into the duration before certificate expiration24h, 48h, 0 (if not using HTTPS)
[DNS_RCODE]Resolves into the DNS status of the responseNOERROR
⁠Functions
FunctionDescriptionExample
lenReturns the length of the object/slice. Works only with the [BODY] placeholder.len([BODY].username) > 8
hasReturns true or false based on whether a given path is valid. Works only with the [BODY] placeholder.has([BODY].errors) == false
patSpecifies that the string passed as parameter should be evaluated as a pattern. Works only with == and !=.[IP] == pat(192.168.*)
anySpecifies that any one of the values passed as parameters is a valid value. Works only with == and !=.[BODY].ip == any(127.0.0.1, ::1)

NOTE: Use pat only when you need to. [STATUS] == pat(2*) is a lot more expensive than [STATUS] < 300.

⁠Alerting

Gatus supports multiple alerting providers, such as Slack and PagerDuty, and supports different alerts for each individual services with configurable descriptions and thresholds.

Note that if an alerting provider is not configured properly, all alerts configured with the provider's type will be ignored.

ParameterDescriptionDefault
alerting.slackConfiguration for alerts of type slack{}
alerting.slack.webhook-urlSlack Webhook URLRequired ""
alerting.discordConfiguration for alerts of type discord{}
alerting.discord.webhook-urlDiscord Webhook URLRequired ""
alerting.pagerdutyConfiguration for alerts of type pagerduty{}
alerting.pagerduty.integration-keyPagerDuty Events API v2 integration key.Required ""
alerting.twilioSettings for alerts of type twilio{}
alerting.twilio.sidTwilio account SIDRequired ""
alerting.twilio.tokenTwilio auth tokenRequired ""
alerting.twilio.fromNumber to send Twilio alerts fromRequired ""
alerting.twilio.toNumber to send twilio alerts toRequired ""
alerting.mattermostConfiguration for alerts of type mattermost{}
alerting.mattermost.webhook-urlMattermost Webhook URLRequired ""
alerting.mattermost.insecureWhether to skip verifying the server's certificate chain and host namefalse
alerting.messagebirdSettings for alerts of type messagebird{}
alerting.messagebird.access-keyMessagebird access keyRequired ""
alerting.messagebird.originatorThe sender of the messageRequired ""
alerting.messagebird.recipientsThe recipients of the messageRequired ""
alerting.telegramConfiguration for alerts of type telegram{}
alerting.telegram.tokenTelegram Bot TokenRequired ""
alerting.telegram.idTelegram User IDRequired ""
alerting.customConfiguration for custom actions on failure or alerts{}
alerting.custom.urlCustom alerting request urlRequired ""
alerting.custom.methodRequest methodGET
alerting.custom.insecureWhether to skip verifying the server's certificate chain and host namefalse
alerting.custom.bodyCustom alerting request body.""
alerting.custom.headersCustom alerting request headers{}
alerting.*.default-alert.enabledWhether to enable the alertN/A
alerting.*.default-alert.failure-thresholdNumber of failures in a row needed before triggering the alertN/A
alerting.*.default-alert.success-thresholdNumber of successes in a row before an ongoing incident is marked as resolvedN/A
alerting.*.default-alert.send-on-resolvedWhether to send a notification once a triggered alert is marked as resolvedN/A
alerting.*.default-alert.descriptionDescription of the alert. Will be included in the alert sentN/A
⁠Configuring Slack alerts
alerting:
  slack: 
    webhook-url: "https://hooks.slack.com/services/**********/**********/**********"

services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: slack
        enabled: true
        description: "healthcheck failed 3 times in a row"
        send-on-resolved: true
      - type: slack
        enabled: true
        failure-threshold: 5
        description: "healthcheck failed 5 times in a row"
        send-on-resolved: true
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"

Here's an example of what the notifications look like:

Slack notifications

⁠Configuring Discord alerts
alerting:
  discord: 
    webhook-url: "https://discord.com/api/webhooks/**********/**********"

services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: discord
        enabled: true
        description: "healthcheck failed"
        send-on-resolved: true
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[RESPONSE_TIME] < 300"
⁠Configuring PagerDuty alerts

It is highly recommended to set services[].alerts[].send-on-resolved to true for alerts of type pagerduty, because unlike other alerts, the operation resulting from setting said parameter to true will not create another incident, but mark the incident as resolved on PagerDuty instead.

alerting:
  pagerduty: 
    integration-key: "********************************"

services:
  - name: twinnation
    url: "https://twinnation.org/health"
    interval: 30s
    alerts:
      - type: pagerduty
        enabled: true
        failure-threshold: 3
      

Tag summary

Content type

Image

Digest

Size

16.5 MB

Last updated

about 5 years ago

docker pull gitopsorg/docker-devops-http-monitor:1.0