ikats/ingestion repository overview

⁠ IKATS Ingestion

This module provides features in order to ingest data in IKATS.
The application is deployed on a Tommee server and runs independantly of IKATS datamodel. It can be reached by two means:

1. an IKATS operator: import TS⁠
2. or via HTTP API (see below).

⁠HTTP API

• The ingestion tool's HTTP API manages some ingestion sessions as jobs.
• Currently, the tool can only run one ingestion session at the same time.
• Practical Use cases⁠ are presented as user guidelines.

The API is available from the URL base: http://host:port/ikats-ingestion/api where host is the IP of the application server host.

Services available from HTTP API are:

• Create and start ingestion session⁠
• Restart a session⁠
• Get the list of the sessions handled⁠
• Get the session by ID⁠
• Get statistics about one session with ID⁠

⁠Create and start an ingestion session

Request:

• API resource: /sessions
• HTTP Url: http://host:port/ikats-ingestion/api/sessions
• HTTP Verb: POST
• HTTP Body: provides the DATA: with the JSON describing the session⁠.

Nominal response:

• header code HTTP 201
• header location property, useful to grep the session <ID> from the value.
  Location example: http://host:port/ikats-ingestion/api/sessions/<ID>

Error response:

• header code HTTP: error code 500|400|...

• body: HTML content for additional error information.

  See also how to submit HTTP requests⁠

Ingestion session description

The JSON data to be sent shall contain the following properties:

• dataset The name of the future dataset into IKATS database

• description A description of that dataset for the end user

• rootPath The root path of the dataset on the import server where files are located

  • Could be absolute, in that case, represent the path on the server
  • If relative, a configuration property will be used as prefix to the path (default: /IKATSDATA)

• pathPattern Regex pattern rules for defining tags and metric of dataset:
  

  • The path is described with a regex
  • The root of the absolute path is the rootPath, and is not included in the pattern
  • The metric and tags should be matched into regex named groups. Metric and tags will be saved as Meta-data useful for querying timeseries in IKATS.
  • The pattern must define one metric with: (?<metric>.*), it defines mandatory information (for OpenTSDB )
  • Each tag is defined with a regex group defined with: (?<tagname>.*)
    Examples: patterns encoded in JSON string (\ needs to be doubled \\):

    \\/(?<equipement>\\w*)\\/(?<metric>.*?)_(?<validity>bad|good)\\.csv

• funcIdPattern Pattern configuring how is generated the Functional Identifier. This substitution pattern defines sections ${metric} or ${<tagname>} refering to pathPattern groups, and replaced by respective group values matched by pathPattern. It follows Apache Commons Lang StrSubstitutor variable format, with any <tagname> or metric as variables names.
  Examples:

  ${equipement}_${metric}_${validity}

• importer Fully Qualified Name of the java importer used to transfer the Time-Serie data to the IKATS dedicated database.

  • This is a plugin definiton adapted to a database. The default plugin is in example below: applicable to OpenTSDB database.
  • serializer Set the Fully Qualified Name of the input serializer.
    • This is a plugin definition adapted to a specific file format. The default plugin is in example below

Example of the JSON document:

  {
    "dataset": "Dataset_name",
    "description": "Dataset exemple",
    "rootPath": "data_dataset_example",
    "pathPattern": "\\/(?<equipement>\\w*)\\/(?<metric>.*?)_(?<validity>bad|good)\\.csv",
    "funcIdPattern": "${equipement}_${metric}_${validity}",
    "serializer": "fr.cs.ikats.datamanager.client.opentsdb.importer.CommonDataJsonIzer",
    "importer": "fr.cs.ikats.ingestion.process.opentsdb.OpenTsdbImportTaskFactory"
  }

⁠Restart a session

• API resource: /sessions/{id}/restart
• HTTP URL: http://host:port/ikats-ingestion/api/sessions/{id}/restart
  • Where {id} is the id of the session.
• HTTP Verb: PUT

See also how to submit HTTP requests⁠

⁠Get sessions list

Warning: Do not use it for a large dataset, the current output is the full data of the sessions !

• API resource: /sessions
• HTTP URL: http://host:port/ikats-ingestion/api/sessions
• HTTP Verb: GET

See also how to submit HTTP requests⁠

⁠Get ingestion session

Request:

• API resource: /sessions/<ID>
• HTTP Url: http://host:port/ikats-ingestion/api/sessions/x
• HTTP Verb: GET

Response: the session JSON content

See also how to submit HTTP requests⁠

⁠Get statistics about an ingestion session

Request:

• API resource: /sessions/<ID>/stats
• HTTP Url: http://host:port/ikats-ingestion/api/sessions/x/stats
• HTTP Verb: GET

Response: the session statistics JSON content

See also how to submit HTTP requests⁠

⁠Use case: launch new ingestion session

1. Make sure that the application server is started⁠ or start it⁠
2. Make sure that the ingestion tool services are started⁠ or start them from the Tomee admin page.
3. Prepare the imported folder:

• IKATS convention: prepared under /var/lib/ikats/IKATSDATA/ filer, must be visible from server host⁠

1. Prepare your new session JSON content in your favourite text editor (JSON describing the session⁠)
2. Submit the service Create and start ingestion session⁠

⁠Use Case: start another ingestion while one is running

You can only run one ingestion at once! There are 2 workarounds:

• Solution 1: starting another application server on another node
  • required: install the application server on a new node of the same cluster: not straightforward !!
  • And follow the use case launch new ingest session⁠ from the new node.
• Solution 2: stopping the running session, launching the second one, and finally restarting the stopped session.
  1. Stop the services from the Tomee admin page⁠ or more drastically stop⁠ and start⁠ the Tomee server.
  2. follow the use case launch new ingest session⁠
  • Submit the stats service⁠ to follow the new session by its ID⁠.
  1. Once the session is finished, restart the stopped session, with stopped ID.

⁠Ingestion tool services

⁠Manage the ingestion tool services

Firstly make sure that Tomee server⁠ is installed and started.

The services are provided by the application ikats-ingestion running on the Tomee server.

You can manage the services from the Tomee admin page, at line ikats-ingestion as described in Check the server status⁠

• Démarrage: is inactive when services are started
• Arrêt: is inactive when services are not started

⁠How to submit HTTP requests

⁠From linux bash

Use the command http installed by httpie⁠

1. For a body with JSON content: edit the JSON content in a file req.json

2. Launch the http command and read the response header and body

Example:
http POST http://host:port/ikats-ingestion/api/sessions < req.json

HTTP/1.1 201
Content-Length: 0
Date: Fri, 12 May 2018 07:02:50 GMT
Location: http://host:port/ikats-ingestion/api/sessions/6
Server: Apache TomEE

⁠From firefox

1. Browse your <HTTP URL>: for a specific body or a verb different from GET, this does not yet work, and you have to go to step 2
2. Open Development tools and select the Net frame
3. From the browser: refresh the page
4. Select the Net tab of the development tool: view this request and edit it: * Select the Header frame * and click on Modify and resend: from that new panel, you can customize your HTTP request
   • edit the HTTP Verb: PUT|POST|DELETE|GET
   • add the json content in request:
     1. complete the Request Request Header with property content-type Content-Type: application/json
     2. edit the Request HTTP body: <JSON content>
   • change the HTTP URL if needed.
   • Submit * Check the Response Header from the submitted requests: visible from the Header.
     For example when required to retrieve the session ID: read the header response location property.

⁠Check the server status

Check that the server is started: browse the http://host:port: TomEE welcome page should be available.

For further informations, select Server status from the welcome page, and enter the well-known admin login/password to reach admin page:

• Follow the link Lister les applications

• The ingesting tool services should be visible at line having the path /ikats-ingestion

⁠Start and stop the server

⁠Start server

To start the application server you have to run from the server host⁠

cd /home/ikats/ingestion # will be the directory where the ingestion database file will be stored
cd apache-tomee-plume-7.0.3
bin/./catalina.sh jpda start # to tell the server to start with JPDA remote capabilities activated

⁠Stop server

To stop the server from the server host⁠ use:

/home/ikats/ingestion/apache-tomee-plume-7.0.3/bin/./shutdown.sh