mtsrus/data-rentgen repository overview

⁠What is Data.Rentgen?

Data.Rentgen is a Data Motion Lineage service, compatible with OpenLineage⁠ specification.

Currently we support consuming lineage from:

• Apache Spark
• Apache Airflow
• Apache Hive
• Apache Flink
• dbt
• SyncMaster (proprietary integration, part of MWS Data Bridge)
• StarRocks (proprietary integration, part of MWS Data Engine)

Note: service is under active development, so API can be unstable.

⁠Goals

• Collect lineage events produced by OpenLineage clients & integrations.
• Store operation-grained events for better detalization.
• Provide API for fetching both job/run ↔ dataset lineage and dataset ↔ dataset lineage.

⁠Features

• Support consuming large amounts of lineage events, use Apache Kafka as event buffer.
• Store data in tables partitioned by event timestamp, to speed up lineage graph resolution.
• Lineage graph is build with user-specified time boundaries.
• Lineage graph can be build with different granularity. e.g. merge all individual Spark commands into Spark applicationId or Spark applicationName.
• Column-level lineage support.
• Authentication support.

⁠Non-goals

• This is not a Data Catalog. DataRentgen doesn't track dataset schema change, owner and so on. Use Datahub⁠ or OpenMetadata⁠ instead.
• Static Data Lineage like view → table is not supported.

⁠Limitations

• OpenLineage have integrations with Trino, Debezium and some other lineage sources. DataRentgen support may be added later.
• DataRentgen parses only limited set of OpenLineage facets, and doesn't store custom facets. This can be changed in future.

⁠Documentation

See https://data-rentgen.readthedocs.io/⁠

⁠Screenshots

⁠Lineage graph

Dataset-level lineage graph

Dataset column-level lineage graph

Job-level lineage graph

Run-level lineage graph

⁠Hierarchy graph

⁠Datasets

⁠Runs

⁠Spark application

⁠Spark run

⁠Spark command

⁠Hive query

⁠Airflow DagRun

⁠Airflow TaskInstance