adammillerio/dogwhistle repository overview

⁠dogwhistle

dogwhistle is a tool for managing DataDog resources using declarative configuration files. This allows for the ability to store configuration in source control for better accountability and management.

This project is largely based on DogPush⁠, however it has some fundamental design differences, and is also a total rewrite in order to clean up code structure and update to Python 3.

dogwhistle can be installed on all platforms via Docker or pip.

⁠Installation

dogwhistle can be installed via either Docker or pip.

⁠Pip

On Windows, it is recommended that you use the ActivePython⁠ distribution of Python 3, as it comes with Pip.

On MacOS, Homebrew⁠ is recommended.

⁠PyPi

Once Python 3 and Pip are installed, run the following command to install dogwhistle from PyPi:

pip install dogwhistle

⁠From Source

To install dogwhistle from source, run the following command from the cloned repository:

pip install .

⁠Docker

⁠Docker Hub

DogWhistle is available as a Docker image on Docker Hub⁠.

Example run:

docker run --rm -it adammillerio/dogwhistle:latest init

⁠From Source

To build the Docker container from source, run the following command from the cloned repository:

docker build -t dogwhistle:latest .

⁠Setup

⁠Overview

At the very least, DogWhistle requires a minimum configuration of API and Application keys from DataDog.

DogWhistle configuration can be provided in multiple ways.

The first option is a config file. DogWhistle's configuration file is located by default in ~/.dogwhistlerc, however, you can manually specify a config location with the --config flag

Alternatively, configuration can be provided via command line flag or environment variable.

⁠Options

⁠Usage

⁠Writing Spec Files

A spec file in DogWhistle is a set of resources as they are to be represented in the DataDog API. Here is an example of a spec file with a single monitor retrieved via the init subcommand:

monitors:
- name: '[team] my-stream IteratorAge is over 4 hours'
  type: metric alert
  query: avg(last_4h):avg:aws.kinesis.get_records_iterator_age_milliseconds{streamname:my-stream} > 3600000
  message: '@[email protected]'
  multi: false
  options:
    escalation_message: ''
    evaluation_delay: 900
    include_tags: true
    new_host_delay: 300
    no_data_timeframe: null
    notify_audit: true
    notify_no_data: false
    renotify_interval: 30
    require_full_window: true
    thresholds:
      critical: 3600000.0
    timeout_h: 0

The main thing to know is that the monitor represented in the spec file above is a direct representation of a Monitor object in the DataDog Python API. This makes it easy to reference the DataDog API documentation to learn what the various values mean.

You can view information about building Monitors here⁠.

⁠Templating Spec Files

DogWhistle is also able to generate yaml spec files via jinja2 templating. For example, we could make the spec file above take two parameters, priority and stream and use these in a template file called kinesis.yaml.j2.

Note: DogWhistle detects template files via the .j2 file extension.

monitors:
- name: '[team] {{ stream }} IteratorAge is over 4 hours'
  type: metric alert
  query: avg(last_4h):avg:aws.kinesis.get_records_iterator_age_milliseconds{streamname:{{ stream }}} > 3600000
  {% if priority == 'high' %}
  message: '@pagerduty'
  {% else %}
  message: '@[email protected]'
  {% endif %}
  multi: false
  options:
    escalation_message: ''
    evaluation_delay: 900
    include_tags: true
    new_host_delay: 300
    no_data_timeframe: null
    notify_audit: true
    notify_no_data: false
    renotify_interval: 30
    require_full_window: true
    thresholds:
      critical: 3600000.0
    timeout_h: 0

Now, we can invoke dogwhistle with -p priority=high -p stream=my-stream and it will substitute in the stream name, and also dynamically set the monitor to notify the PagerDuty integration since it is high priority.

Parameters can also be stored in key value format in a json file and loaded via --params-file params.json:

{
  "priority": "high",
  "stream": "my-stream"
}

There is a wide range of options that are available when using the jinja2 templating library that allow for dynamic configuration of DataDog resources.

⁠Resources in DataDog

All resources that are created by DogWhistle have a special tag applied to them, dogwhistle.managed:true. This tag is transparently used to allow the tool to filter out resources that it does not manage.

⁠Init

DogWhistle uses spec files to define DataDog resources. These spec files are essentially direct representations of the DataDog API, with a few small changes and the ability to use jinja2 templating. If you already have an initial set of monitors deployed in DataDog through other means, you can retrieve them with the init command:

dogwhistle init --prefix '[team]'

This will retrieve all resources in a given account that start with [team] and output a spec file representation of them that can be used with other DogWhistle commands. This output can then be redirected to a file to be edited:

dogwhistle init --prefix '[team]' > team-spec.yaml

⁠Diff

Diff will take a set of provided spec files and check them against what is deployed in DataDog. It will then output a diff with two sections:

• local - Monitors that have been defined locally but have not been deployed to DataDog yet, these will be CREATED upon push.
• changed - Monitors that have differing state between the local spec file and the DataDog API, these will be UPDATED upon push. A summary of the differences is provided.

dogwhistle diff kinesis-spec.yaml.j2

⁠Push

Push will take a set of provided spec files and check them against what is deployed in DataDog. If the force option is not provided, it will then print a diff for the user and ask them to manually approve the changes. Then, it will create/update resources in DataDog as necessary.

Note: DogWhistle does not delete resources.

dogwhistle push kinesis-spec.yaml.j2

⁠Mute

Mute will take a set of provided spec files and mute the associated monitors in DataDog, disabling all notification targets. This is useful for running deployments.

Note: Currently rules for muting are not implemented, use the --forever flag to mute indiscriminately.

dogwhistle mute kinesis-spec.yaml.j2

⁠Unmute

Unmute will take a set of provided spec files and unmute the associated monitors in DataDog.

dogwhistle unmute kinesis-spec.yaml.j2

⁠Development

⁠Building

To develop on DogWhistle, it is recommended to install it in development mode with pip:

pip install -e .

This will install dogwhistle so that it points to the current working directory, allowing for iteration without having to reinstall.

⁠mypy

This repository has MyPy type hints for type enforcement.

To use these, first install mypy:

pip install mypy

Then, run the following command from the root of the repository:

mypy dogwhistle