n8n templatesn8n templates

airflow dag run

This workflow automatically triggers and monitors the execution of specified DAGs by calling the REST API of Apache Airflow, allowing real-time retrieval of task execution results. It has built-in status checks and timeout mechanisms to intelligently handle different states, ensuring the stability and controllability of the workflow. It is suitable for scenarios that require remote triggering and monitoring of data pipeline tasks, improving work efficiency, reducing human intervention, and ensuring the smooth progress of task processes.

Workflow Diagram
airflow dag_run Workflow diagram

Workflow Name

airflow dag_run

Key Features and Highlights

This workflow leverages Apache Airflow’s REST API to automatically trigger the execution of a specified DAG (Directed Acyclic Graph), monitor its runtime status in real-time, and ultimately retrieve the task execution results. It incorporates built-in status checks and timeout mechanisms to intelligently handle various states such as queued, running, success, and failure, ensuring the stability and controllability of the workflow execution.

Core Problems Addressed

Automates the triggering and monitoring of Airflow DAG executions, resolving issues related to manual operations being cumbersome, difficulty in tracking statuses, and untimely exception handling. It is especially suitable for scenarios requiring remote or integrated invocation of Airflow tasks with timely feedback on execution results.

Use Cases

  • Data engineers needing to remotely trigger and monitor data pipeline tasks
  • Automated operations teams triggering and checking Airflow task statuses
  • Business systems integrating Airflow job execution results for subsequent processing
  • Seamless integration between orchestration platforms and Airflow

Main Workflow Steps

  1. Input Parameter Reception: Accepts inputs including DAG ID, task ID, configuration parameters (conf), polling interval (wait), and maximum wait time (wait_time).
  2. Configure Airflow API Endpoint: Sets the Airflow server API base URL.
  3. Trigger DAG Execution: Initiates the specified DAG via an HTTP POST request to the Airflow API.
  4. Evaluate DAG Status: Processes the returned DAG run status with branching logic:
    • If status is queued, wait for the specified interval before polling again;
    • If status is running, continue polling similarly;
    • If status is success, retrieve the task execution result;
    • If status is failed, terminate immediately and raise an error.
  5. Timeout Mechanism: Maintains a polling counter and stops execution with an error if the maximum wait time is exceeded.
  6. Return Results: On success, fetches and outputs the specified task’s XCom return value.

Involved Systems or Services

  • Apache Airflow: Utilized for DAG execution and status queries via its REST API.
  • n8n: Serves as the automation workflow engine orchestrating and controlling the entire process.

Target Users and Value Proposition

Ideal for data engineers, DevOps engineers, and automation platform developers, this workflow facilitates automated triggering and status monitoring of Airflow tasks, enhancing operational efficiency, reducing manual intervention, and ensuring stable execution of data pipelines and task workflows. It is particularly valuable for technical teams aiming to integrate Airflow jobs into broader automation ecosystems.