Public | Automated Build

Last pushed: 18 hours ago
Short Description
RESTFull service indexing time series events in Elasticsearch
Full Description

What is espipe?

espipe is a RESTFull service indexing documents in bulk requests to Elasticsearch.

Run

docker run -p 5000:5000 -v /etc/espipe:/etc/espipe khezen/espipe:5

Supported tags and respective Dockerfile links

Services

Request Response Description
POST /espipe/{template.name}/{documentType} JSON body 200 OK indexes JSON body as {documentType} in Elasticsearch {template.name}-yyyy.MM.dd
GET /espipe/health 200 OK healthcheck

Configure

{
    "elasticsearch": "http://localhost:9200",
    "templates": [{
            "name": "logs",
            "bufferSizeKB": 5000,
            "timerMS": 5000,
            "body": {...}
        }
    ]
}

Configure templates

  • templates is an array of template configurations:
    • espipe creates templates in elasticsearch if they do not exist yet
    • an index template will automatically be applied when new indices are created
    • for each template, espipe creates indices on daily basis
      • example:
        • template=logs-*,
        • indices=logs-2017.01.05, logs-2017.01.06, etc..
  • espipe indexes documents in bulk requests. For each index, espipe triggers bulk requests by:
    • bulk size >= template.bufferSizeKB
    • ticker(period=template.timerMS)

template.body

template.body takes the template settings and mappings with types definition.
See the Create Template API documentation.

The mapping itself is flexible and is schema-free. New fields are automatically added to the type mapping definition when espipe indexes a new document. Check out the mapping section for more information on mapping definitions.

Security

AWS Sign

espipe supports AWS authentication for Amazon Elasticsearch Service

{
  "elasticsearch": "https://host.eu-west-1.es.amazonaws.com",
  "AWSAuth": {
    "accessKeyId": "changeme",
    "secretAccessKey": "changeme",
    "region": "eu-west-1"
  },
  "templates": [...]
}

Basic Auth

espipe supports basic authentication for Elasticsearch (shield, search-guard)

{
    "elasticsearch": "http://localhost:9200",
    "basicAuth":{
      "username": "elastic",
      "password": "changeme"
    },
    "templates": [...]
}

Default config.json

Request Response Description
POST /espipe/logs/log JSON body 200 OK indexes JSON body as log in Elasticsearch logs-yyyy.MM.dd
POST /espipe/web/trace JSON body 200 OK indexes JSON body as trace in Elasticsearch web-yyyy.MM.dd
GET /espipe/health 200 OK healthcheck

User Feedback

Issues

If you have any problems or questions, please ask for help through a GitHub issue.

Docker Pull Command
Owner
khezen
Source Repository

Comments (0)