koroslak/nifi repository overview

⁠What is it ?

It's a framework for small scale out-of-the-box BI infrastructure.

Each compoment can be replaced by a technology of you choice. Don't like postgres ? replace it with Oracle. The principles will still be the same.

⁠How to run it

1. Install docker https://docs.docker.com/install/⁠
2. From root folder run. docker-compose up
3. Wait until the containers are running.
4. Done :)

⁠Core Components

⁠NiFi

• Fetch data
• Store them in Postgres
• Execute Postgres stored procedures
• Single point of scheduling/dataflow

⁠Postgres

• Data store - staging
• Clean integrated data - core
• Reports - mart
• +Transformations between areas

⁠Metabase

• View Reports/tables stored in mart
• Make aggregates/filters
• Share them via URLs

⁠NiFi

• I'm using NiFi as a batch ingest tool and orchestrator for Data Warehouse.

⁠Principles

From 2 years of working and trying to make workflow as generic, modularized and simple to understadnd here are my principles I used to archive it.

1. Every logical step should output only 1 flow file. e.g. One Processor group that ingests data from a source system and outputs one flow file into success or error connection.
2. Speficy credentials only once. e.g. If you use InvokeHTTP processor, create a workflow so that there won't be 2 InvokeHTTP processors with the same credentials set.
3. If you can, create a Processor group that will act as a blackbox. Show the user what's going on without going to unnecessary detail.

⁠Naming Convention:

⁠Postgres

⁠3 schemas

⁠4 table types (defined by postfixes)

⁠Metabase