Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with tailored integrations with Hadoop, AWS, GCP and more, a lot of it community contributed.

https://github.com/spotify/luigi⁠