whipharsh/airflow repository overview

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

DAGs - In Airflow, a DAG (directed Acyclic Graph) is a collection of all the tasks you want to run. Operator - Class describing an operation that you want to do in a DAG. Example of an operator is Extract data from Mongo to S3. Transfer data from S3 to Snowflake. Airflow comes with a set of prebuilt operators - Tasks - instantiation of an Operator. The instantiation defines specific values when calling the abstract operator, and the parameterized task becomes a node in a DAG. Task instance - one execution or run of a task. Characterized as the combination of a dag, a task and a point in time. Connections - Credentials for connecting to external entities - DB, AWS S3, Snowflake. You define connections to external entities through the Airflow's Admin UI and use the connection name through Hooks in code. Hooks - Classes that are used in code and that encapsulate functions that can take place on a connection.