transferwiseworkspace/pipelinewise repository overview

⁠PipelineWise

PipelineWise is a Data Pipeline Framework using the Singer.io⁠ specification to ingest and replicate data from various sources to various destinations. Documentation is available at https://transferwise.github.io/pipelinewise/⁠

⁠Supported tags and images

For more information on PipelineWise and a range of available versions of docker images, please go to Github at https://github.com/transferwise/pipelinewise⁠

⁠Features

• Built with ELT in mind: PipelineWise fits into the ELT landscape and is not a traditional ETL tool. PipelineWise aims to reproduce the data from the source to an Analytics-Data-Store in as close to the original format as possible. Some minor load time transformations are supported but complex mapping and joins have to be done in the Analytics-Data-Store to extract meaning.

• Managed Schema Changes: When source data changes, PipelineWise detects the change and alters the schema in your Analytics-Data-Store automatically

• Load time transformations: Ideal place to obfuscate, mask or filter sensitive data that should never be replicated in the Data Warehouse

• YAML based configuration: Data pipelines are defined as YAML files, ensuring that the entire configuration is kept under version control

• Lightweight: No daemons or database setup are required

• Extensible: PipelineWise is using Singer.io⁠ compatible taps and target connectors. New connectors can be added to PipelineWise with relatively small effort

⁠License

PipelineWise as a standalone software is licensed under Apache License Version 2.0 but bundled components can use different licenses and may overwrite the terms and conditions detailed in Apache License Version 2.0.

The final license of the build depends on the included connectors, so please check which connectors included in the image. For further details please check the Licenses⁠ section in the documentation. If you're not satisfied with the license of the docker image then you can build your own image and customise which connectors you want to include into your build.