area51/dataretriever repository overview

⁠dataretriever

This is a simple go application that can bridge a remote service and submit messages to a local RabbitMQ instance.

Note: This is still alpha quality so bugs will probably exist until I have fully tested it.

⁠Introduction

This application supports multiple data sources using multiple protocols ranging from simple http polling to connecting to remote Message Brokers using Stomp.

⁠Configuration

The application requires a single configuration file which is in yaml format. This file consists of several sections listed below. A template is available in the github repository.

⁠amqp

This section is mandatory and defines the connection to RabbitMQ.

# Required: Details on logging in to RabbitMQ
amqp:
  # Required: RabbitMQ connection string: amqp://user:pass@hostname
  url: amqp://user:password@rabbitHostname
  # Required: The routing key to use for all messages
  routingKey: test
  # Optional: The exchange to use, defaults to amq.topic
  #exchange: amq.topic

⁠debug

An optional setting that turns on additional debugging. For production set this to false or leave out.

# Turn on debugging
debug: true

⁠STOMP bridge

In this mode this will bridge a remote STOMP server to RabbitMQ. The instance will connect to a single remote server but you can subscribe to multiple topics, each one will forward each message to an individual routingKey.

Example of connecting to a STOMP server & subscribing to 2 topics

stomp:
  server: "example.com:61618"
  username: "[email protected]"
  password: "s3cr3t"
  clientId: "[email protected]"
  host: "/"
  heartBeat:
    sendTimeout: 15s
    receiveTimeout: 15s
  subscription:
  -
    topic: "/topic/TOPIC1"
    routingKey: feed.topic1
  -
    # durable subscription - valid for activemq only
    topic: "/topic/TOPIC2"
    routingKey: feed.topic2
    headers:
      "activemq.subscriptionName": "topic2"

⁠HTTP polling

This mode is intended for remotely accessing a remote webservice. Any response received is then subbitted to the local RabbitMQ instance.

http:
  # Required: The url to retrieve
  url: https://api.example.com/
  # Optional: Basic authentication
  #basicAuth:
  #  user: MyUser
  #  password: MyPassword
  # Optional: Headers to send in the request. This is simply a map of key/value pairs sent for each request
  #headers:
  #  Authorization: SomeAuthorizationKey
  #  key: value
  # The schedule to retrieve
  schedule: "0/10 * * * * *"

⁠Schedule expression format

A cron expression represents a set of times, using 6 space-separated fields.

Note: Month and Day-of-week field values are case insensitive. "SUN", "Sun", and "sun" are equally accepted.

⁠Special Characters

Asterisk ( * )

The asterisk indicates that the cron expression will match for all values of the field; e.g., using an asterisk in the 5th field (month) would indicate every month.

Slash ( / )

Slashes are used to describe increments of ranges. For example 3-59/15 in the 1st field (minutes) would indicate the 3rd minute of the hour and every 15 minutes thereafter. The form "*/..." is equivalent to the form "first-last/...", that is, an increment over the largest possible range of the field. The form "N/..." is accepted as meaning "N-MAX/...", that is, starting at N, use the increment until the end of that specific range. It does not wrap around.

Comma ( , )

Commas are used to separate items of a list. For example, using "MON,WED,FRI" in the 5th field (day of week) would mean Mondays, Wednesdays and Fridays.

Hyphen ( - )

Hyphens are used to define ranges. For example, 9-17 would indicate every hour between 9am and 5pm inclusive.

Question mark ( ? )

Question mark may be used instead of '*' for leaving either day-of-month or day-of-week blank.

⁠Predefined schedules

You may use one of several pre-defined schedules in place of a cron expression.

⁠Intervals

You may also schedule a job to execute at fixed intervals. This is supported by formatting the cron spec like this:

@every where "duration" is a string accepted by time.ParseDuration (http://golang.org/pkg/time/#ParseDuration⁠).

For example, "@every 1h30m10s" would indicate a schedule that activates every 1 hour, 30 minutes, 10 seconds.

Note: The interval does not take the job runtime into account. For example, if a job takes 3 minutes to run, and it is scheduled to run every 5 minutes, it will have only 2 minutes of idle time between each run.

⁠Time zones

By default, all interpretation and scheduling is done in the machine's local time zone (as provided by the Go time package http://www.golang.org/pkg/time⁠). The time zone may be overridden by providing an additional space-separated field at the beginning of the cron spec, of the form "TZ=Asia/Tokyo"

Be aware that jobs scheduled during daylight-savings leap-ahead transitions will not be run!

⁠Building

To build run the following:

docker build -t myimage .

This will build a new image called myimage with everything compiled.

⁠Running

Ensure you have config.yaml setup then run the following replacing path/to/config.yaml to the absolute path to your config.yaml file:

docker run -d --name myimage -v path/to/config.yaml:/config.yaml:ro myimage