Airflow

Apache Airflow, maintained by the Apache Software Foundation, is a workflow orchestration platform for programmatically authoring, scheduling, and monitoring data pipelines, with active development shown by recent releases (version 3.3.0, July 2026). Workflows are written in standard Python rather than XML or a proprietary DSL, a web UI provides monitoring and task management through a modern dashboard, a modular queue-based architecture scales across multiple workers, Jinja templating enables dynamic, code-driven pipeline generation, and users can build custom operators or use built-in ones for AWS, Google Cloud, and Azure. An active community spans Slack, Stack Overflow, and GitHub, with a regular Airflow Summit conference. Completely free and open source. Good fit for a data engineering team that needs to author, schedule, and monitor complex, code-defined data pipelines with real dependency management, rather than a simpler cron-based scheduler.