arkonatechnologies/vtelemetry2 repository overview

⁠Telemetry Package

⁠What is this?

This image allows you to monitor all operational parameters exposed by the arkona BLADE//runner⁠ and the Lawo V__matrix⁠, and stream the results to an influxdb database for further analysis or interactive visual inspection.

⁠Where do I get it from?

To download the docker image, please install docker-ce and docker-compose first and pull the image with the docker command: docker pull arkonatechnologies/vtelemetry2. If you want to use podman, pull the image from the docker.io hub : podman pull docker.io/arkonatechnologies/vtelemetry2.

⁠How do I set it up?

The container is configured by environment variables defined on container start as shown in the example below:

docker run -d \
    -e MACHINES=172.16.4.10,172.16.4.20,172.16.4.30 \
    -e BLACKLIST='\.audio\.signals,\.output\.signals,\.user_labels,\.monitoring_objects\.audio,software_update,audio_source_slices,peak_meter' \
    -e WHITELIST='p_t_p,genlock,system' \
    -e INFLUX_API=1 \
    -e DB_HOST=192.168.1.23 \
    -e DB_USER=someguy \
    -e DB_PASSWORD=secret \
    -e DB_NAME=main_db \
    -e DB_PORT=8086 \
    -e VERBOSITY=1 \
    arkonatechnologies/vtelemetry2

⁠Machines

Decide on the list of machines you wish to monitor and enter their respective IP addresses. For example, if you wish to monitor three AT300/C100 blades running at 172.16.4.10, 172.16.4.20, 172.16.4.30, the environment variable should look like: -e MACHINES=172.16.4.10,172.16.4.20,172.16.4.30

Every machine may be monitored through any of its IP addresses (i.e., whether you choose to use a management port or one of the front ports is irrelevant). However, vtelemetry will not allow you to open concurrent connections to the same blade as this would do nothing but put unnecessary load both on your monitoring PC and the blade itself. Should you have requested multiple connections to the same blade by mistake, vtelemetry will tell you about the offending IP address(-es) and exit.

⁠Whitelist / Blacklist

To monitor everything the BLADE//runner or V__matrix has to offer, you may simply omit the BLACKLIST and WHITELIST variables. However, this will likely log large amounts of irrelevant information such as peak meter levels, and thus place an unnecessarily high burden on your database server.

To only monitor those parts of the state you are actually interested in, set BLACKLIST to a list of comma-separated regular expressions (using JavaScript syntax). Those will be matched against every keyword's entire path <keyword_list>.<keyword_name>, and anything that matches will be ignored. For most use cases, the following is a good default to begin with:

-e BLACKLIST='\.audio\.signals,\.output\.signals,\.user_labels,\.monitoring_objects\.audio,software_update,audio_source_slices,peak_meter'.

You may also use WHITELIST to re-include state parts that had previously been excluded by BLACKLIST. For example, the easiest way to log everything under PTP and Genlock but nothing else would be to set BLACKLIST=.* (i.e., skip everything) and WHITELIST=p_t_p,genlock.

⁠I wish to use this with a preexisting influxdb/Grafana setup

Fill in all fields starting with DB_ according to your particular influxdb setup. For example, if you wish to log to a database named main_db on an influxdb server at IP address 192.168.1.23, using the default port of 8086, username someguy and password secret, the config file should now look as follows:

docker run -d \
    -e MACHINES=172.16.4.10,172.16.4.20,172.16.4.30 \
    -e BLACKLIST='\.audio\.signals,\.output\.signals,\.user_labels,\.monitoring_objects\.audio,software_update,audio_source_slices,peak_meter' \
    -e WHITELIST='p_t_p,genlock,system' \
    -e INFLUX_API=1 \
    -e DB_HOST=192.168.1.23 \
    -e DB_USER=someguy \
    -e DB_PASSWORD=secret \
    -e DB_NAME=main_db \
    -e DB_PORT=8086 \
    -e VERBOSITY=1 \
    arkonatechnologies/vtelemetry2

⁠InfluxDB v2 support

The vtelemetry package can also be used with influxDB v2. Just change following variables:

• Set INFLUX_API to 2
• Change DB_NAME to DB_BUCKET=<bucket name>
• Change DB_PASSWORD to DB_TOKEN=<influx token>

Note that while testing this, we found v1 performance to be significantly better; this may have changed by now, but should you experience performance issues with a v2 installation, you might want to consider downgrading to v1.x for the time being.

⁠I do not have a preexisting influxdb/Grafana setup, but would like to try this anyway

To get your feet wet with an example setup of influxdb, grafana and vtelemetry, first install docker-compose⁠.

Go to a directory where you want to store your configuration and database data and create a file named docker-compose.yml with the following contents:

version: '2'
services:
  influxdb:
    image: influxdb:1.8
    ports:
      - '8086:8086'
    volumes:
      - ./influxdb/:/var/lib/influxdb
    environment:
      - INFLUXDB_DB=test
      - INFLUXDB_ADMIN_USER=test
      - INFLUXDB_ADMIN_PASSWORD=test
  grafana:
    image: grafana/grafana:latest
    ports:
      - '3000:3000'
    depends_on:
      - influxdb
    environment:
      - GF_SECURITY_ADMIN_USER=test
      - GF_SECURITY_ADMIN_PASSWORD=test
  vtelemetry:
    image: arkonatechnologies/vtelemetry2
    depends_on:
      - influxdb
    environment:
      - MACHINES=172.16.4.10,172.16.4.20,172.16.4.30
      - BLACKLIST=\.audio\.signals,\.output\.signals,\.user_labels,\.monitoring_objects\.audio,software_update,audio_source_slices,peak_meter
      - WHITELIST=p_t_p,genlock,system
      - INFLUX_API=1
      - DB_HOST=<IP address of your host>
      - DB_USER=test
      - DB_PASSWORD=test
      - DB_NAME=test
      - DB_PORT=8086
      - VERBOSITY=1

Create an empty directory named influxdb and start docker compose (or docker-compose in older docker versions) using the following command:

docker compose -f docker-compose.yml up

After docker compose has made it through the initialization process (this may take half a minute or so), navigate to http://127.0.0.1:3000 using a browser of your choice.

• Enter username test, password test and click on Log In.
• Go to Connections -> Data Sources and then click "Add data source";
• Select InfluxDB.
• Fill in the following four fields:

URL=http://influxdb:8086
Database=test
User=test
Password=test

• Click Save & Test and make sure Grafana tells you that the data source is working.
• Go to the Explore section (compass icon in the left toolbar)
• By clicking on the select_measurement button, you should see a popup menu with a wealth of items to choose from. Enter system.temperature and set field to cpu. Group the resulting data by IP address and set the displayed measurement interval to at least 15 minutes (in the top right corner of the screen).
• Click Run query to show your graph.

⁠Where do I go from here?

Play around with the Explore section and maybe set up a few dashboards. If you like what you see, you may now wish to consult the official influxdb documentation on how to set up a properly configured server. At the very least, please make sure to set up a retention policy⁠ as influxdb will otherwise keep inflating its database until your storage medium signals surrender.

⁠Compose file with influxDB configuration and loki

version: '3'

networks:
  telemetry:

services:
# influxDB
  influxdb:
    image: influxdb:2.7.1
    networks:
      - telemetry
    ports:
      - '8086:8086'
    volumes:
      - ./influxdb2:/var/lib/influxdb2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=influxdbuser
      - DOCKER_INFLUXDB_INIT_PASSWORD=influxdbpassword
      - DOCKER_INFLUXDB_INIT_ORG=influxdborg
      - DOCKER_INFLUXDB_INIT_BUCKET=bladerunner
      - V1_DB_NAME=v1-bladerunner
      - V1_RP_NAME=v1-rp
      - V1_AUTH_USERNAME=v1-influxdbuser
      - V1_AUTH_PASSWORD=v1-influxdbpassword
# grafana
  grafana:
    image: grafana/grafana-oss
    user: root
    networks:
      - telemetry
    ports:
      - '3000:3000'
    depends_on:
      - influxdb
    volumes:
      - ./grafana/var/:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=test
      - GF_SECURITY_ADMIN_PASSWORD=test
      - GF_SECURITY_ALLOW_EMBEDDING=true
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
# loki for logs
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - telemetry
# promtail for loki
  promtail:
    image: grafana/promtail:2.9.0
    user: root
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/config.yml
    networks:
      - telemetry
# vtelemetry
  vtelemetry:
    image: arkonatechnologies/vtelemetry2
    networks:
      - telemetry
    depends_on:
      - influxdb
    environment:
      - MACHINES=172.16.4.10,172.16.4.20,172.16.4.30
      - BLACKLIST=\.audio\.signals,\.output\.signals,\.user_labels,\.monitoring_objects\.audio,software_update,audio_source_slices,peak_meter
      - WHITELIST=p_t_p,genlock,system
      - INFLUX_API=2
      - DB_HOST=influxdb
      - DB_USER=influxdbuser
      - DB_BUCKET=bladerunner
      - DB_NAME=bladerunner
      - DB_ORG=influxdborg
      - DB_TOKEN=<whatever token from influxDB>
      - DB_PASSWORD=influxdbpassword
      - VERBOSITY=1
      - DB_PORT=8086

⁠Hardware requirements

The hardware requirements for vtelemetry depend on the setup of vtelemetry, especially the number of blades and the list of keywords being monitored. Including keywords that are by default excluded by the blacklist, like peak meters, will result in huge data loads with questionable purpose.

It furthermore depends on how long the data has to be stored, 7d of data (default configuration) of ~45 blades results in ~1TB of data. There are possibilities in the database to reduce that load by downsample older data, like adding a recursing task with something like the following:

// Task Options
option task = {name: "downsample", every: 8h}

// Defines a data source
data =
    from(bucket: "bladerunner")
        |> range(start: -duration(v: int(v: task.every) * 2))

//    |> filter(fn: (r) => r._measurement == "mem")
data
    // Windows and aggregates the data in to 1h averages
    |> aggregateWindow(fn: last, every: 10m)
    // Stores the aggregated data in a new bucket
    |> to(bucket: "downsampled", org: "influxdborg")

Also, querying the data in a non-wasteful way is often more important than further optimizing the collection process. Unspecific queries for a long time period can result in request timeouts or even an unresponsive PC. Best way to do queries is to be as specific as possible. This is even more important when there's not plenty of RAM in which the database usually caches data.

TL;DR

As a very rough recommendation there are two scenarios:

1. Small installations like up to 8 blades with the default configuration can be monitored by a mid range PC, roughly: {cpu: i5/Ryzen 5, ram: 16GB RAM, disk: SSD}.

2. Bigger installations like up to ~40 blades or more* with the default configuration can be monitored by a higher range PC (i7/i9/Ryzen 7/Ryzen 9, 64GB RAM, SSD raid). It scales better with more cores/threads than pure clock speed. 64GB of RAM are not needed, but 32GB could be right at the edga without any in-mem handling for the database, and therefore not recommended for that scenario.

*Not having the possibility to test telemetry with more blades than being installed in our lab, the numbers are to be handled with caution. The lab has around 45 blades running, all being monitored on one server (32 core EPYC, 192GB RAM, SSD raid), data stored for 7d plus downsampled data for a year,plus some other data and services with a load of 30GB RAM used, 140GB RAM cached for influx, 30% CPU in average.

⁠Caveats

Even with a lot of optimizations, the huge amount of data being written to the database in very short time periods could lead to problems if multiple data batches want to to write at the exact same time to the database. To resolve issues like that, consider looking for a database clustering or other optimizations.