DAG

Now I want to share one of the topics in Data Engineering: DAG (Directed Acyclic Graph). At first glance, it may seem hard or strange, but it is much simpler than you think.

I think it is better to start with the last word, which is Graph. A graph has nodes and edges. Each node represents a step or a process, and each edge represents a pipeline that moves data through the process.

Graphs have two types: Acyclic and Cyclic. Here, we have the first one, which means there are no loops. This helps us understand that we do not repeat the same processes or nodes again and again, and each process follows a clear direction.

It helps you manage each step, and each node can contain running Python code, execute SQL commands, run another node, or even interact with another system like NiFi. Each node is an operator (Airflow’s building block, such as PythonOperator, BashOperator, SQLOperator, etc.) that performs one discrete step.

Edges between nodes only define the order and dependencies, meaning: ‘this task runs only after that one has finished.