The mw CLI tool (with development environment & gitlab commands)

Collection of data engineering DAGs to be executed by the WMF Airflow instances.

Collection of data engineering DAGs to be executed by the WMF Airflow instances.

Proof of concept for Terraform and Flink on WMCS

For experimentation

Utilities and libraries for working with data pipelines at WMF. E.g. distributing conda envs, building and syncing artifacts, etc.

Repository for configuration of general purpose GitLab Cloud Runner.

WMF fork of moby/buildkit

A development environment for Scap3 and the Scap self-installer

A Scap3 deployment repo for RelEng's Jenkins instances

In the heart of the Section topics data pipeline lies a set of Spark jobs.

Data jobs owned by the Global Data and Insights team.

These should use conda-dist from workflow_utils to generate conda env artifacts, which are then deployed to the Analytics Data Lake for scheduling and running by Airflow.

Toolforge Local Kubernetes setup for development