Data Orchestration with Apache Airflow
Building a unified data orchestration layer with over 600 DAGs on Apache Airflow.
Client: Bergfreunde GmbH · Period: 2020–2022 · Role: Data Engineer
For Bergfreunde, we rebuilt the data orchestration layer from scratch. The result was a unified orchestration system on Apache Airflow with a complete ETL pipeline spanning over 600 DAGs, including data modelling jobs. Bergfreunde gained a single engine that executes all data jobs on schedule with 100% consistency. Comprehensive documentation was produced for the handover, and a team of 4 data engineers was trained on the new platform.
Technologies
Google Cloud, Kubernetes, Apache Airflow, BigQuery, Python, Terraform