prefecthq/prefect repository overview

⁠Prefect

Prefect is a workflow orchestration framework for building data pipelines in Python. It's the simplest way to elevate a script into an interactive workflow application. With Prefect, you can build resilient, dynamic workflows that react to the world around them and recover from unexpected changes.

With just a few decorators, Prefect supercharges your code with features like automatic retries, distributed execution, scheduling, caching, and much more.

Workflow activity is tracked and can be monitored with a self-hosted Prefect server⁠ instance or managed Prefect Cloud⁠ dashboard.

⁠Getting started

Prefect requires Python 3.9 or later. To install the latest or upgrade to the latest version of Prefect⁠, run the following command:

pip install -U prefect

Then create and run a Python file that uses Prefect flow and task decorators to orchestrate and observe your workflow - in this case, a simple script that fetches the number of GitHub stars from a repository:

from prefect import flow, task
from typing import list
import httpx


@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")


@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)


# run the flow!
if __name__=="__main__":
    github_stars(["PrefectHQ/Prefect"])

Fire up the Prefect UI to see what happened:

prefect server start

To run your workflow on a schedule, turn it into a deployment and schedule it to run every minute by changing the last line of your script to the following:

if __name__ == "__main__":
    github_stars.serve(name="first-deployment", cron="* * * * *")

You now have a server running locally that is looking for scheduled deployments! Additionally you can run your workflow manually from the UI or CLI - and if you're using Prefect Cloud, you can even run deployments in response to events⁠.

⁠Join the community

Join us on GitHub⁠ or Slack⁠.