#AIRFLOW DAG UNIT TESTING CODE#

get ( 'S3_BUCKET', default_var = None ) logger = logging. Here’s my basic DAG template for a classic ETL job: import datetime import logging from import EmptyOperator from corators import dag, task, task_group from .hooks.s3 import S3Hook S3_BUCKET = Variable. If not, the beauty of Airflow is mutability, but starting from basic, familiar components will reduce the complexity of your DAGs and make them easier to interpret. In my own projects, I start from a template- pretty much every job follows an extract-transform-load pattern that can be mapped to a few tasks. Simply using TaskFlow is a great start! With the new structure, readability is greatly improved. The result? Without the discipline and rigor to standardize a deployment, DAGs can quickly turn into untested scripts. Unfortunately, it was introduced without any framework for modularity or testing. py file where they could do whatever they want. Airflow revolutionized orchestration by providing engineers with a blank canvas- a. Perhaps the biggest weakness of Airflow is also its biggest strength: possibility. I think one of the most important aspects of DAG creation is structure- DAGs should be standardized as much as possible, prioritizing readability and consistency. Here are a few common Airflow problems & my proposed solutions. Coupled with some external tools, we can make the most of Airflow by adhering to best practices.

#AIRFLOW DAG UNIT TESTING CODE#

Much of this new functionality can be used to provide a smoother experience during development and improve code testability. Quite a bit has changed in the last few versions- though the TaskFlow API was released in December 2020, the Airflow team has continued to provide improved functionality that competes with newer orchestrators (Dagster, Prefect, Mage) on a number of fronts, fixing bugs and expanding the scope of Dynamic Task Mapping. If you’ve read this far, you’re either using Airflow or going forward with an implementation… 😬 My goal this week is to give Airflow a fair shake 🧂 and talk about how you can make the most of your Airflow deployment to build robust, production-ready DAGs.

I agree with many of Jake’s points and hope to extend his argument on why Airflow “sometimes wins.” While I was writing this post, Jake Watson reached out about a similar peice. For the same reasons, understanding how Airflow works can be a game-changer for data engineers- up-skilling popular tools improves marketability and potential impact.Ī few weeks ago, I sat down with Daniel Imberman of Astronomer, who walked me through some features of their AstroSDK, which I found quite helpful. ”ĭue to the popularity, adoption, and momentum of Airflow, it will be around for years, if not decades, regardless of competition. In the words of Annath at Data Engineering Weekly: “ a known devil is better than an unknown. Perhaps you’re on an established team with hundreds of DAGs or your org is one of many using hosted Airflow, which is available on every major cloud provider ( Amazon / Google / Azure). There are so many tools that can move faster, have better support, or come with advanced functionality.įor many, however, this is not reality. We have minimal tech debt and our current tools can easily be migrated- what a luxury! It’s obvious Airflow isn’t a top choice. My perspective is one of a solo data guy on a small team. One specific point of contention was Apache Airflow. While “hot takes” are fun and generate a buzz, they might not tell the whole story. Of course, the truth is rarely black and white- often just a shade of grey. In my last post, 🌶️ Hot Takes on the Modern Data Stack, I presented my opinions in a raw, unfiltered format, opting for starkness & contrast over nuance.

Airflow dag unit testing