blippar/aragorn repository overview

⁠Regression testing made easy

⁠Description

Aragorn is a regression testing tool that can run a set of tests periodically. It acts as a server that can execute or schedule test suites. (a JSON file with the .suite.json extension, see example below for more information)

When a test suite is run, each of its tests is executed. For each test, there are 3 possible outcomes:

• the test's request is successful and its response is expected: nothing happens
• the test's request is successful but its response is unexpected: failures are reported
• the test's request can not be performed : the error is reported

Where the failures/errors are reported and how they are reported depends on the notifier configured. For now the only one implemented is the SlackNotifier. It is set at the initialization of a test suite. During a test suite execution, it stacks every potential failure and/or error. At the end of the suite execution, it will send a notification describing what happened if there was any failure or error.

⁠Quick Start

You can add Aragorn to your project by doing aragorn init

The directories to watch can be specified via command line positional arguments : aragorn watch dir1 dir2.

⁠Config

The config is only used by the run command. It contains a list of suites to execute or schedule and the notifiers.

Example:

{
  "notifiers": {
    "slack": {
      "webhook": "https://hooks.slack.com/services/T/B/1",
      "verbose": true
    }
  },
  "suites": [
    {
      "path": "./test/service.suite.json",
      "runEvery": "1h",
      "failFast": true,
      "suite": {
        "base": {
          "url": "https://example.com",
          "insecure": true
        }
      }
    }
  ]
}

⁠Notifiers

⁠Slack

⁠Test Suites

⁠SuiteConfig

A test suite describes a combination of tests to be run. It is composed of some configuration fields for the scheduling and notification handling. The tests are described in the suite field depending on the type field (HTTP or GRPC).

Example:

{
  "name": "Service 1 HTTP test suite",
  "type": "HTTP",
  "runEvery": "12h",
  "retryCount": 3,
  "retryWait": "100ms",
  "timeout": "10s",
  "suite": {"..."}
}

⁠HTTPSuite

An HTTP test suite contains a base configuration and list of tests.

⁠HTTPBase

⁠OAUTH2Config

See golang.org/x/oauth2/clientcredentials Config documentation⁠ for more info.

⁠HTTPTest

⁠HTTPRequest

⁠HTTPExpect

1. See json-schema.org⁠ and Understanding JSON Schema⁠ for more info.

⁠HTTP URL Templating

The URL path and query can be constructed from previous tests through templating.

{
  "tests": [
    {
      "id": "add_todo",
      "name": "Add Todo",
      "request": {
        "method": "POST",
        "path": "/todo"
      },
      "saveDocument": true
    },
    {
      "name": "Get Todo",
      "request": {
        "method": "GET",
        "path": "/todo/{{add_todo.id}}"
      }
    }
  ]
}

⁠HTTPDocument

Document is any type with some special behaviors like reference.

Load a JSON document from a file:

{
  "$ref": "user.json"
}

Load a RAW document from a file:

{
  "$ref": "image.jpg",
  "$raw": true
}

Inline RAW document:

{
  "$raw": "OK"
}

⁠HTTPObject

Object must be a JSON object (map[string]interface{}). It can load a JSON object from a file (see Document doc).

⁠HTTP Example

This example shows how to create an HTTP test suite file that has 2 tests:

• The first one is a GET http://localhost:8080/ request and expects a 200 OK JSON response {"key": "value"}.
• The second test is a POST http://localhost:8080/echo request with a 201 Created response and JSON body containing {"key":"value", "a": [1, 2, 3], "b": {"c": "d"}}. It expects a JSON response matching the JSON schema schema.json and the given JSON values.

{
  "name": "Service 1",
  "type": "HTTP",
  "suite": {
    "base": {
      "url": "http://localhost:8080",
      "header": {
        "Accept-Encoding": "application/json"
      },
      "oauth2": {
        "clientID": "id",
        "clientSecret": "secret",
        "tokenURL": "https://localhost:8080/token"
      }
    },
    "tests": [
      {
        "name": "root",
        "request": {
          "path": "/",
          "method": "GET"
        },
        "expect": {
          "header": {
            "Content-Type": "application/json",
            "Content-Length": "15"
          },
          "jsonDocument": { "key": "value" }
        }
      },
      {
        "name": "echo",
        "request": {
          "path": "/echo",
          "method": "POST",
          "header": {
            "Content-Type": "application/json"
          },
          "body": { "key": "value" }
        },
        "expect": {
          "statusCode": 201,
          "header": {
            "Content-Type": "application/json"
          },
          "jsonSchema": { "$ref": "schema.json" },
          "jsonValues": {
            "key": "value",
            "a.0": 1,
            "a.length": 3,
            "b.c": "d"
          }
        }
      }
    ]
  }
}

⁠GRPC Suite

An GRPC test suite contains a base configuration and list of tests.

⁠GRPCTest

⁠GRPCRequest

⁠GRPCExpect

⁠GRPCDocument

Document is any type with some special behaviors like reference.

Load a JSON document from a file:

{
  "$ref": "user.json"
}

⁠GRPC Example

This example shows how to create an GRPC test suite file that has 2 tests.

{
  "name": "Service 2",
  "type": "GRPC",
  "suite": {
    "address": "localhost:50051",
    "header": { "hello": "world" },
    "tests": [
      {
        "name": "Empty Call",
        "request": { "method": "grpcexpect.testing.TestService/EmptyCall" },
        "expect": { "code": "OK" }
      },
      {
        "name": "Simple Call",
        "request": {
          "method": "grpcexpect.testing.TestService/SimpleCall",
          "document": { "username": "world" }
        },
        "expect": {
          "code": "OK",
          "header": { "hello": "world" },
          "document": { "message": "Hello world!" }
        }
      }
    ]
  }
}

⁠Tracing

This project use OpenTracing⁠, a vendor-neutral open standard for distributed tracing. When aragorn run a test suite it will create a span that will be propagated in the context, a sub span is created for each test. More details are filled by the test suite package that implement the execution of the test. For example, the http call in the httpexpect package is traced in a sub span.

By default, the aragorn command will run with no tracer. You can set a tracer with the tracer flag. More info with aragorn help exec.

⁠Credits

Thanks to Malenea⁠ for his awesome work on the logo.

Aragorn's logo is licensed under the Creative Commons Attribution 4.0 International License⁠ and was widely inspired on the original Go gopher designed by Renee French⁠.