pilillo/mastro repository overview

  __  __           _             
 |  \/  |         | |            
 | \  / | __ _ ___| |_ _ __ ___  
 | |\/| |/ _` / __| __| '__/ _ \ 
 | |  | | (_| \__ \ |_| | | (_) |
 |_|  |_|\__,_|___/\__|_|  \___/ 

👷 Data and Feature Catalogue in Go

⁠Data What?

⁠Goals

Managing Metadata to achieve:

• Lineage - understanding data dependencies
• Quality - enrich data with dependability information
• Democratization - foster self-service culture

Modeling of the data transformation layer to foster:

• Reuse - of domain specific knowledge and pre-computed features in different use cases
• Enforcing - of established industry or domain-specific transformations and practices
• Interaction - between different roles through teams and projects

⁠Disclaimer

Mastro is still on development and largely untested. Please fork the repo and extend it at wish.

⁠TL-DR

Terminology:

• Connector⁠ - component handling the connection to volumes and data bases
• FeatureStore⁠ - service to manage features (i.e., featureSets and featureStates);
• Catalogue⁠ - service to manage data assets (i.e., static data definitions and their relationships);
• Crawler⁠ - any agent able to list and walk a file system, filter and parse asset definitions (i.e. manifest files) and push them to the catalogue;

Help:

• PlantUML Diagram of the repo⁠
• Configuration⁠

Status: