monicaproject/hldfad_worker repository overview

⁠High Level Data Fusion and Anomaly Detection Module (HLDFAD) Quick Start Guide

⁠Open Source Code

Open Source code and complete docker-compose solution is available at GitHub Repository⁠, with complete README documentation. It has to be remarked that contents reported in such documentation partially reflects and shall be referred to most complete source code repository. A working example of whole MONICA toolchain, including HLDFAD module, is freely avaiable at this GitHub Repository⁠.

⁠Overview

HLDFAD is a back end deputed to provide high level outputs based on acquisition and elaboration of selected on field observations received from MQTT Platform Broker. MQTT input topics are filtered according to contents reported in Platform GOST Observation Catalog. Output Messages can be provided in MQTT format through internal MQTT Output Broker and\or OGC Service Catalog Output.

The complete bottom-up data chain is the following:

- On Field Data Sensors (wristband and Security Fusion Nodes for cameras input)
- On Field Gateway
- SCRAL
- LinkSmart
- HLDFAD
- OGC and DSS

Figure below provides a quick overview to give a general understanding about HLDFAD position module.

HLDFAD Module actually interacts directly with exchange end point, that are middleware between different MONICA modules. Such elements are reported in the following:

• WP6 Service Catalog: Beginning End Point that communicates IOT ID and MQTT output broker for MQTT output messages;
• OGC Service Catalog: Beginning End Point that reports all Things and Datastreams available. HLDFAD extracts MQTT Observations topics from there
• MQTT Broker Observations and Output: Runtime MQTT endpoint for input observations acquisition and output provisioning

⁠Main functionalities

Tag 02.06.01.05 of HLFDAD Docker hub images offer the following functionalities:

• Crowd Heatmap, based on Wristbands Localization input
• Queue detection Alert, based on Security Fusion Node density maps.

⁠Crowd Heatmap

Based on localization received by Wristbands Gateway, HLDFAD calculates people density map, i.e. the occurrency of the positions within a geographic monitored area expressed in terms of geo spatial rectangular matrix.

⁠Queue Detection Alert

Based on people geospatial density maps received by Security Fusion Nodes, HLDFAD generates geographic polygons with adjacent cells with specific mean of people.

⁠Detailed Information

HLDFAD is a Docker-composed solution relying on Django Python framework. Docker containers that compose the HLDFAD solution are the following:

- worker: Django-Celery main docker, that includes task generation and management (this docker hub image)
- rabbit: Exchange and Queue for Django task system management
- redis: Cache for temporarily storage of incoming observation input before elaboration (second most important container after celery) 

GitHub⁠ reports open source code python code and docker-compose solution. Most of the following information are referred to such repository.

⁠Test

It has been created a dedicated repository that allows to easily performs complete test with HLDFAD module on GitHub⁠. Follows the instructions reported in README.

⁠Definition and Terms

⁠Environment Variables

In the following, the environment variables available for module configuration are reported. In the following, the environment variable are categorized in different sub lists. For each variable, it is indicated the name, the matching with default .env value (based on .env reported in Git Hub repository), the type (intended in terms of internal parsing) , an explaination and default value (if available).

⁠Environment Variables: External Interfaces

Such variables allows to set up this module towards external end point.

• WP6_CATALOG_CONNECTIONURL: ${V_WP6_CATALOG_CONNECTIONURL} --> [str] WP6 Service Catalog Connection Hostname
• WP6_CATALOG_CONNECTIONPORT: ${V_WP6_CATALOG_CONNECTIONPORT} --> [int] WP6 Service Catalog Connection Port
• ENV_MQTT_OBSERVATION_URL: ${V_ENV_MQTT_OBSERVATION_URL} --> [str] MQTT Broker Observations and output IP Address
• ENV_MQTT_OBSERVATION_PORT: ${V_ENV_MQTT_OBSERVATION_PORT} --> [int] MQTT Broker Observations and output Port
• OUTPUT_MQTTBROKER_USERNAME: ${V_OUTPUT_MQTTBROKER_USERNAME} --> [str] MQTT Broker Observations and output Username
• OUTPUT_MQTTBROKER_PASSWORD: ${V_OUTPUT_MQTTBROKER_PASSWORD} --> [str] MQTT Broker Observations and output Password
• ENV_CATALOG_PORT: ${V_ENV_CATALOG_PORT} --> [int] OGC Catalog Port
• ENV_WEB_BASE_URL: ${V_ENV_WEB_BASE_URL} --> [str] OGC IP Address-Domain
• ENV_CATALOG_USERNAME: ${V_ENV_CATALOG_USERNAME} --> [str] OGC Catalog Username
• ENV_CATALOG_PASSWORD: ${V_ENV_CATALOG_PASSWORD} --> [str] OGC Catalog Password
• DB_PORT_5432_TCP_ADDR: ${PGSQL_WORKER_HOST} --> [str] PosgreSQL Connection Database IP Address (note: PosgreSQL Service Name, when it runs locally)
• DB_PORT_5432_TCP_PORT: ${PGSQL_WORKER_PORT} --> [int] PosgreSQL Connection Database TCP Port (default=5432)
• DB_USER: ${PGSQL_WORKER_USER} --> [str] PosgreSQL Connection Username
• DB_PASSWORD: ${PGSQL_WORKER_PASSWORD} --> [str] PosgreSQL Connection Password
• DB_NAME: ${PGSQL_WORKER_DATABASE} --> [str] PosgreSQL Connection Database Name

⁠Environment Variables: Internal Subnetwork interfaces

Such variables allows to set up this module towards internal sub net docker components interfaces (inside docker-compose file).

• RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER} --> [str] RabbitMQ Username
• RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASS} --> [str] RabbitMQ Password
• RABBITMQ_HOSTNAME: rabbit --> [str] RabbitMQ Hostname
• RABBITMQ_PORT: 5672 --> [str] RabbitMQ Port
• CACHEREDIS_DEFAULT_HOSTNAME: redis --> [str] Cache Redis Hostname
• CACHEREDIS_DEFAULT_PORT: 6379 --> [int] Cache Redis Port

⁠Environment Variables: Main Application Configurations

Such variables allows to set up main internal configuration data, in particular the geographic monitored area (useful for Crowd Heatmap computation based on Wristband Localization).

• APPSETTING_MONITORINGAREA_LATITUDE: ${V_APPSETTING_MONITORINGAREA_LATITUDE} --> [float] Crowd Heatmap Output Ground Plane Position Latitude
• APPSETTING_MONITORINGAREA_LONGITUDE: ${V_APPSETTING_MONITORINGAREA_LONGITUDE} --> [float] Crowd Heatmap Output Ground Plane Position Longitude
• APPSETTING_MONITORINGAREA_HORIZONTALSIZE_M: ${V_APPSETTING_MONITORINGAREA_HORIZONTALSIZE_M} --> [int] Crowd Heatmap Output Ground Plane Position Horizontal Size, in meters
• APPSETTING_MONITORINGAREA_VERTICALSIZE_M: ${V_APPSETTING_MONITORINGAREA_VERTICALSIZE_M} --> [int] Crowd Heatmap Output Ground Plane Position Vertical Size, in meters
• APPSETTING_MONITORINGAREA_CELLSIZE_M: ${V_APPSETTING_MONITORINGAREA_CELLSIZE_M} --> [int] Crowd Heatmap Output Ground Plane Position Cell Size, in meter (The single size of square cell)

⁠Environment Variables: Additional Application Configurations

Such variables allows to set up main additional configuration data to regulate internal software behaviour.

• APPSETTING_ENABLE_EMPTY_CROWD_HEATMAP: ${V_APPSETTING_ENABLE_EMPTY_CROWD_HEATMAP} --> [bool] Enable Creation of empty Crowd Heatmap when no observation are received (default=False)
• APPSETTING_ENABLE_RANDOM_OUTPUT: ${V_APPSETTING_ENABLE_RANDOM_OUTPUT} --> [bool] Enable Creation of random Crowd Heatmap when no observation are received (default=False)
• APPSETTING_ENABLE_RANDOM_QUEUEDETECTIONALERT: ${V_APPSETTING_ENABLE_RANDOM_QUEUEDETECTIONALERT} --> [bool] Enable Creation of random Queue Detection Alert when no observation are received (default=False)
• APPSETTING_TASK_ELABORATION_FREQ_SECS: ${V_APPSETTING_TASK_ELABORATION_FREQ_SECS} --> [int] Interval of forcing elaboration expressed in seconds (independently from observations received)
• APPSETTING_TASK_ALIVEAPP_FREQ_SECS: ${V_APPSETTING_TASK_ALIVEAPP_FREQ_SECS} --> [int] Interval of Task Alive in seconds (it just provides evidence via log that HLDFAD is up and running and the thread are up))
• APPSETTING_ENABLE_OBS_IOTIDRETRIEVE: ${V_APPSETTING_ENABLE_OBS_IOTIDRETRIEVE} --> [bool] Enable Retrieving of observation IoT Identifier from OGC Catalog (default=True)
• APPSETTING_GOST_NAME: ${V_APPSETTING_GOST_NAME} --> [str] Beginning Label in composition of observation topics (default="GOST")
• APPSETTINGS_ENABLE_IMMEDIATEELABORATION_FEEDBYNUMBEROBS: ${V_APPSETTINGS_ENABLE_IMMEDIATEELABORATION_FEEDBYNUMBEROBS} --> [bool] Enable immediate trigger elaboration of Crowd Heatmap (and-or Queue Detection) when the number of observations unprocessed reaches up the number of associated datastreams (default=True)
• CONFENVIRONMENT_GLOBALINFO: ${V_CONFENVIRONMENT_GLOBALINFO} --> [str] Label To Identify Environment

⁠Affiliation

This work is supported by the European Commission through the MONICA H2020 PROJECT⁠ under grant agreement No 732350.