dl1998/workflows-manager repository overview

⁠Workflows-Manager

Effortlessly manage and execute Python-based workflows locally.

GitHub Repository⁠ | Documentation⁠

⁠Overview

Workflows-Manager is a lightweight and flexible tool for running locally defined workflows. Designed for developers and data engineers, it allows seamless execution of Python-based reusable steps defined in JSON/YAML configuration files.

⁠Features:

• Local Execution: Execute workflows entirely on your local machine. No external dependencies required.
• Customizable Steps: Define workflows with reusable Python steps and JSON-based or YAML-based configurations.
• Lightweight Design: Simplified setup and usage tailored for personal and small team projects.
• Extensible: Easily extendable for various use cases like data processing, automation, and testing.

⁠Quick Start

⁠Pull the Image

docker pull dl1998/workflows-manager:latest

⁠Run the Container

Mount your workflow directory to the container for local execution:

docker run --rm -v $(pwd)/your-workflows:/app dl1998/workflows-manager:latest

⁠Example Directory Structure

your-workflows/
│
├── steps/
│   ├── step1.py
│   └── step2.py
│
├── workflows.yaml

⁠Execute a Workflow

docker run --rm -v $(pwd)/your-workflows:/app dl1998/workflows-manager:latest \
    run <workflow-name>

⁠Configuration

1. Steps Directory: Contains Python files with reusable workflow steps.
2. Workflow JSON/YAML: Defines the workflow structure, including step order, parameters, and dependencies.

⁠Example Workflow Configuration Syntax

workflows:
  workflow-name:
    steps:
      - name: Execute step
        step: registered-step1
        parameters:
          param1: value1
      - name: Execute another step
        step: registered-step2
        parameters:
          param2: value2

⁠Building Locally

To build and run the Docker image locally:

docker build -t workflows-manager .
docker run --rm -v $(pwd)/your-workflows:/app workflows-manager:latest --help

⁠License

This project is licensed under the MIT License⁠.

⁠Feedback and Contributions

Found a bug or have a feature request? Feel free to submit an issue or contribute via pull requests on GitHub⁠.