atlasplatformdocker/rta-datasvc repository overview

⁠RTA Data Service

This service is part of the RTA Toolkit Services suite.

It provides compact, high-performance telemetry storage, backed by either a file system or Amazon S3⁠ and DynamoDB⁠. It handles backfill (late samples and channels) and copes with sample rates from 0.001 Hz to 1 MHz with stable memory use and back-pressure control.

It exposes a GRPC interface to write or read data from your environment, and an outward-facing REST interface to be exposed to users via the Gateway Service.

This service is intended to be used alongside the Session Service, Config Service, Stream Service (if live updates are required) and Gateway Service as a reverse proxy to bring the capabilties together.

It is available as a binary for Windows and Linux, and as a Docker image.

⁠Performance

Expected throughput on an AWS EC2 m5.xlarge Linux instance, per thread:

For comparison, a well-tuned InfluxDB⁠ facade can expect to read at 2-3 million samples/sec on equivalent hardware.

⁠When should you use this service?

This service does not offer any search or analytical capabilities — only high-performance sample storage and retrieval, optimized for RTA.

Other options to consider first:

Can you access data in its existing format? : If you have a significant amount of data in an existing format, it might be better to implement the data service interface to access this data in place. If the format is slow — such as CSV — you could cache data aggressively to compensate.

Can you use a time-series database, such as InfluxDB⁠ or TimescaleDB⁠? : Much slower to ingest and query, and much more expensive to run: check your data rates to see what your running costs would be. However, in return you get powerful query and aggregation capabilities which — while not used by ATLAS — can be really useful for dashboarding sofware like Grafana⁠.

Can you use a columnar file format, like Parquet⁠? : Slow to ingest and query, and will likely require some specialized ingest tooling to buffer and organize the data before it is written. But if you don't have strict ingest latency or query latency requirements, you can have the flexibility to use the same data storage with ATLAS and Big Data tools like Apache Spark⁠ or Amazon Athena⁠.

If McLaren don't provide a service for your existing format, perhaps we can help you build one? Refer to our developer documentation for more information.

You might prefer this Data Service if you need:

• High speed and efficiency
• Bulk access to subsets of data — without resampling or aggregation
• Reasonably compact, inexpensive storage (expect around 1.2 bytes/sample)
• To store data as it arrives without a slow finalization stage
• To work with minimal supporting infrastructure
• Support for more data types than double-precision
• McLaren ECU-specific channel types, such as slow-row data

⁠Usage

⁠Docker

Storing data on the file system:

docker run --rm \
  -v /path/to/store:/data \
  -e RTA_Store=File \
  -e RTA_FilePath=/data \
  -p 2670:2670 -p 2671:2671 -p 2672:2672 \
  mclarenapplied/rta-datasvc

Storing data in AWS:

docker run --rm \
  -e RTA_Store=Aws \
  -e RTA_AwsS3Bucket=mydatabucket \
  -e RTA_AwsDynamoTable=mydatatable \
  -p 2670:2670 -p 2671:2671 -p 2672:2672 \
  mclarenapplied/rta-datasvc

⁠Command Line

Storing data on the file system:

rta-datasvc --Store File --FilePath /data

Storing data in AWS:

rta-datasvc --Store Aws --AwsS3Bucket mydatabucket --AwsDynamoTable mydatatable

⁠Ports

⁠Monitoring

HTTP GET / : Returns 200 OK and reports the software version. In Kubernetes, use this for liveness probes.

HTTP GET /health : Returns 200 OK if the service is up and the file system path or AWS S3 bucket and DynamoDB table are accessible.
In Kubernetes, use this for readiness probes.

HTTP GET /metrics : Prometheus⁠ metrics, including the health check status.

⁠Configuration

The service can be configured to store data either on the file system, or on AWS using S3⁠ for the data and DynamoDB⁠ for indexes.

Options can be set using both environment variables and command line arguments, or by the provided appsettings.config JSON configuration file.

Environment variables are all prefixed with RTA_, and can be written in ALLCAPS or MixedCase.
For example, FilePath is RTA_FILEPATH or RTA_FilePath.

For Linux and on Kubernetes, also replace : with __.
For example, Category:Item is RTA_CATEGORY__ITEM.

We recommend that separate instances are run in either the read or write roles. This helps balance resource use and allows the write service to run with different permissions and network security rules.

⁠Init Store

InitStoreAndExit=true will cause the service to initialize the store and exit immediately. This enables the service to be used to setup or upgrade the environment.

⁠Memory Management

These options manage the memory pool when writing data — and so are relevant only when EnableWrite=true.

• Write buffers are used to accumulate incoming data;
• Flush buffers are used to collate and hold data while is it being flushed to storage;
• The high threshold determines when data is transferred from write buffers to flush buffers;
• The low threshold determines how much data should remain in the write buffers to be merged with back-fill.

The default options are quite aggressive, consuming 256 MiB (write) + 128 MiB (flush), beginning a flush at 50% write buffer usage and flushing down to 10%. You probably don't need to change these settings unless you want to raise or lower the memory requirement.

These are soft limits: additional memory is required to merge late data (back-fill), and run the .NET Core Garbage Collector.

Pre-allocation helps ensure that the process is stable when under load in a resource-constrained environment — such as Kubernetes.

When tuning for maximum throughput, aim for a write:flush ratio of about 2:1 to balance the buffers after allowing for data compression. Set the flush buffer size so that a full flush will span 4-16 buffers for increased I/O parallelism.

⁠Authentication and Authorization

These options enable OAuth 2.0 support.

This service is intended to run within a private network, behind the Gateway Service, which will forward the Bearer token if present.

⁠Footnotes

1. Slow-path access happens when the data needs to be decoded before it can be served to the user — for example, to apply a time-shift. ↩⁠

2. Fast-path access happens when no decoding is required and data can be routed directly from storage to ATLAS with minimal intermediate processing. ↩⁠