Skip to content

Draft: Instrument lineage of airflow pipelines

Milimetric requested to merge add-datahub-lineage into main

The docs below outline the setup necessary to enable an Airflow instance to talk to our Datahub instance. Once these are done, we can test this MR to make sure it works, and then review it for code style/approach.

The goal is to have lineage defined for all existing Airflow DAGs so we can set a good example going forward.

Refernce: https://datahubproject.io/docs/lineage/airflow/ https://airflow.apache.org/docs/apache-airflow/stable/lineage.html

Edited by Milimetric

Merge request reports