synchronicityiot/ai-prediction-engine repository overview

NOTE: This image (>= v0.9) does not work alone and needs its combination with other modules to behave correctly. As a matter of fact, this module relies on Elastic Search to store train samples, models and historics. For the sake of a holistic documentation, the reader may refer here⁠, where we describe a fully-fledged approach that embraces all the micro-services needed to run an estimator over FIWARE-compliant datasets.

⁠IOT AI Engine

Regarding the way to accurately estimate next possible values based on the context information gathered from the various cities, we will make use of different Machine Learning techniques that will produce the prediction. Technically speaking, we make us of Keras⁠, a Python-based framework that operates on top of some Deep Learning frameworks as TensorFlow⁠.

⁠Training the model

With the data produced by the sensors every day, a database is collected for every entity/id so that a predictive model can be produced. This model will recognize and detect the patterns so that it is able to forecast (predict), considering some input data (current and periodic past measures in the database reformed as a time series), the values for the next 15, 30, 45 and 60 minutes. In order to train the model, json data format is used along with an Elasticsearch⁠ database. For every new data it is needed an @id and a @timestamp field, more values will be interpreted as features. The algorithm reframe the data as a time series and will transform it into a supervised learning problem: for every measure, previous periodic measures are used as features of a time series, which are then loaded into a neural network that will try to fit to the pattern of each time series.

⁠Configuration

The engine uses a configuration file to store its relevant communication and iniial parameters. In addition, there are several parameters that can be modified to change how the system learns. You should create this configuration file ( named config.ini) with values for every of these parameters, inside a 'config' folder that you will later load into the container (to substitute the one already there). Every parameter is explained here:

DEFAULT section:

• num_forecasts: Number of values that the system will output when a prediction is requested.
• num_previous_measures: (IMPORTANT) This parameter affects the learning algorithm itself, it stands for the length of the time series, which is the number of previous values considered for the learning.

ML section:

• algorithm: Name of the algorithm to be used as Neural Network, currently 'lstm' (default), 'phasedlstm' and 'dense' models are available.
• epoch_internal: Number of times the full training set is passed though the neural network for a specific batch size. Default is 50
• epoch_external: Number of times the batch_size is increased and the neural network is retrained (epoch_internal times). Default is 1
• batch_size: Number of examples or size of the batches in which each epoch_internal is divided. Default is 200 (increases with each epoch_external)
• initial_validation_split: Percentage of validation examples used to test the model when training (for the optimizing function). Default is 0.05
• validation_split_multiplayer: Multiplier of the validation split used in each epoch_external. Default is 1.75
• batch_size_multiplier: Multiplier of the size of the batches for every epoch external. Default is 1.5
• minimum_samples: Minimum number of samples to train the system
• training_size: Number of samples which will be used to train the model
• time_window_to: Period of time for looking back for data in elasticsearch. One week is 1w. One month would be 1M
• time_window_from: Time to start looking back to for data in elasticsearch. Default is now

Format for time_window parameters follow Date Math⁠ from elasticsearch API.

ELASTIC section:

• index_entities: Name of the index that stores the entities
• index_data: Name of the index that stores the data
• index_scalers: Name of the index that stores the scalers
• index_predictions: Name of the index that stores the predictions
• index_models: Name of the index that stores the models
• mapping_data: Name of the mapping that defines the format in the index of data
• mapping_entities: Name of the mapping that defines the format in the index of entities
• mapping_models: Name of the mapping that defines the format in the index of models
• mapping_predictions: Name of the mapping that defines the format in the index of predictions

DATA section:

• time_index: Column label used for indexing data (timestamp column)
• prediction_feature: Name of the feature that is going to be forecasted
• other_features: Name of the other features used for training

ELK section

• elastic_host: Hostname of elasticsearch. Example: localhost
• elastic_port: Port of communication with elasticsearch. Example: 9200 (default of elasticsearch)

Here we provide an example of the configuration file (config.ini):

[DEFAULT]
num_forecasts = 4
num_previous_measures = 10

[ML]
algorithm = lstm
epoch_internal = 50
; If the algorithm allows warm-start (sklearn-MLP or sklearn-RandomForest) or it is a keras model this can be greater than 1
epoch_external = 1
batch_size = 200
initial_validation_split = 0.1
training_ratio = 0.9
validation_split_multiplayer = 1.75
batch_size_multiplier = 1.5
; Minimum number of samples to train the system
minimum_samples = 1000
; Number of samples which will be used to train the model
training_size = 10000
; Period of time for looking back for data in elasticsearch. One week is 1w. One month would be 1M
time_window_to = 1w
; Time to start looking back to for data in elasticsearch
time_window_from = now

[ELASTIC]
index_entities = entities
index_data = data-parking
; index_scalers = scalers-demo
index_predictions = predictions-parking
index_models = models-parking
mapping_data = context_data
mapping_entities = context_entities
mapping_models = model_keras
mapping_predictions = predictions_keras

[DATA]
time_index = timestamp
prediction_feature = ratio
; other_features = humidity,pressure


[ELK]
;elastic_host = elasticsearch
;elastic_host = localhost
;Synchronicity VM (ATOS)
;elastic_host = 94.75.227.183
elastic_host = 127.0.0.1
elastic_port = 9200

⁠Instructions

You may run the engine with command:

docker run synchronicityiot/ai-prediction-engine:@tag -e ELASTIC_HOST='@elastic_host' -e ELASTIC_PORT='@elastic_port' -v ./config/:/usr/src/app/config -p 5000:5000

Note that you must specify at least @tag (for example :v1.0) and the folder that contains the configuration file as a volume. You can specify environment variables for the elasticsearch host (@elastic_host) and the elasticsearch port (@elastic_port), if you do not provide them, the ones in the config.ini file will be used. You should also open the port used in the engine (5000).

In case you might want to include the engine in your docker-compose file, we provide here the code needed to launch it (please configure properly the docker image tag and the elasticsearch environment variables):

networks:
  ai-engine-network:
    driver: bridge

volumes:
  aimodelsdata:
    driver: local

services:

  iot-ai-engine:
    image: synchronicityiot/ai-prediction-engine:v1.0
    container_name: iot-ai-engine
    hostname: iot-ai-engine_host
    volumes:
      - ./config/:/usr/src/app/config
      - aimodelsdata:/usr/src/app/storage
    environment:
      NODE_ENV: development
      ELASTIC_HOST: 127.0.0.1
      ELASTIC_PORT: 9200
    ports:
      - 5000:5000
    networks:
      ai-engine-network:
        aliases:
          - iot-ai-engine_host

The engine should start running (it waits some time to connect to elasticsearch). The docker image of the engine is available in docker hub synchronicityiot⁠, please make sure you are using the newest tag (not always :latest).

The engine is defined as an API REST, a server in localhost and port 5000 (http://localhost:5000⁠) will start running ready to work when the docker image starts. Plase take a look at the logs, the engine needs the conection with elasticsearch in order to work properly.

Firstly, in order to make predictions, the module must be fed with data. The data must be already stored in the elasticsearch database and should at least contain the following parameters:

{
    "id": "urn:ngsi-ld:OnStreetParking:santander:parking:onStreet:StaLuciaEast",
    "timestamp": "2019-06-11T10:05:09.00Z",
    "type": "OnStreetParking",
    "ratio": 10.344827586206897
}

• id: Id or name of the entity with the data.
• timestamp: Marks the time index.
• type: Type of the data, here is parking.
• ratio: Feature that will be used to predict. The name can vary.

Additional parameters may be used depending on your configuration or your needs (for example more features). However, everything should match the config.ini file.

The engine will automatically train a model for every entity (with enough new data) every day at 4:30am. Models are stored in the /storage folder. With the model trained, it is now possible to make predictions, which are done also automatically by the engine every hour. You can access those predictions in elasticsearch at the index and mapping that you have indicated in the config.ini file. An example of the format of the predictions stored in elasticsearch is:

{
    "model_file": "Q7NwDWsB2UYPRYj1IUbr",
    "timestamp": "2019-05-31T10:30:30.737034Z",
    "algorithm": "lstm",
    "id": "urn:ngsi-ld:OnStreetParking:santander:parking:onStreet:StaLuciaEast",
    "values": [
        17.101938247680664,
        16.920570373535156,
        16.93316650390625,
        16.768169403076172
    ],
    "prediction_times": [
        "2019-05-31T10:45:30.737034Z",
        "2019-05-31T11:00:30.737034Z",
        "2019-05-31T11:15:30.737034Z",
        "2019-05-31T11:30:30.737034Z"
    ],
    "latency": "0.45788"
}

It is also possible to ask for a prediction via a GET method to the engine like:

curl -X GET \
  http://127.0.0.1:5000/predict/urn:ngsi-ld:OnStreetParking:santander:parking:onStreet:StaLuciaEast

Notice that it is necessary to indicate the id of the entity where the estimations will be done. This will return a JSON object with the format:

{
    "timestamp": "2019-05-31T10:30:30.737034Z",
    "id": "urn:ngsi-ld:OnStreetParking:santander:parking:onStreet:StaLuciaEast",
    "prediction": [
        17.101938247680664,
        16.920570373535156,
        16.93316650390625,
        16.768169403076172
    ],
    "prediction_times": [
        "2019-05-31T10:45:30.737034Z",
        "2019-05-31T11:00:30.737034Z",
        "2019-05-31T11:15:30.737034Z",
        "2019-05-31T11:30:30.737034Z"
    ],
    "latency": "0.1",
    "status": "success"
}

In addition to this, it is also possible to force the system to train now, with a request like:

curl -X GET http://localhost:5000/train

⁠License

• AGPLv3