Scheduler Power
Airflow is one of those tools that’s easy to get running but rarely examined beneath the surface. Most teams interact with it through the UI or by writing DAGs — and that’s enough to be productive. But understanding the components underneath helps when things break, when you need to scale, or when you’re trying to explain why a job that “should have run” didn’t.
This isn’t a deep dive into every configuration knob. It’s a walkthrough of the major components, how they interact, and why the architecture is designed the way it is.
DAG Processor
The DAG processor is responsible for reading your Python DAG files, parsing them, serialising the resulting DAG structures, and storing them in the metadata database. This is a separate process from the scheduler — an important distinction, because it means DAG parsing failures don’t crash the scheduler itself.
In earlier versions of Airflow, the scheduler handled DAG parsing directly. Separating the two was a deliberate architectural improvement: it isolates user code (which can import anything, fail in unpredictable ways, or consume excessive resources) from the core scheduling loop.
Metadata Database
Every Airflow deployment needs a metadata database. In our setup, this is a managed Postgres instance. It stores everything: serialised DAGs, DAG run history, task instance states, connections, variables, and XCom data.
Postgres is the source of truth for the entire system. The scheduler reads from it to determine what needs to run. The web server reads from it to render the UI. Workers write back to it to update task status. If the metadata database goes down, Airflow is effectively down — nothing can be scheduled, monitored, or recorded.
This is worth emphasising because it’s a common bottleneck. Under heavy load — hundreds of DAGs, thousands of task instances — the metadata database can become the limiting factor before CPU or memory on the workers does. Connection pooling, query performance, and database sizing all matter here.
Scheduler
The scheduler is the core of Airflow. It continuously polls the metadata database, evaluates which DAG runs need to be created based on schedules and triggers, determines which task instances are ready to execute (dependencies met, upstream tasks succeeded), and queues them for execution.
Think of it as a manager: it doesn’t do the work, but it decides what work needs to happen and when. It reads the state of the world from the database, applies the dependency logic encoded in your DAGs, and pushes eligible tasks into the message broker.
The scheduler is designed to be stateless — all state lives in the database. This means you can run multiple scheduler instances for high availability. If one goes down, another picks up where it left off, because the database is the single source of truth.
Message Broker (Redis)
Between the scheduler and the workers sits a message broker — in our case, Redis, used via the Celery executor. Redis acts as a lightweight task queue: the scheduler enqueues tasks, and workers dequeue and execute them.
You can think of it like a simpler Kafka without the durability guarantees. Messages are transient — once a worker picks up a task, it’s popped from the queue. If Redis goes down and tasks are lost, the scheduler will detect that those task instances never reported back and can re-queue them.
Redis is fast and operationally simple, which is why it’s the default choice for most Celery-based Airflow deployments. For teams that need stronger delivery guarantees, RabbitMQ is the other common option.
Workers
Workers are where the actual computation happens. Each worker pulls tasks from the message broker, executes them, and writes the result (success, failure, retry) back to the metadata database.
This is where resource constraints surface. If a large number of tasks are scheduled simultaneously — a thundering herd scenario — workers can hit CPU and memory limits. Airflow provides several levers for this: worker concurrency settings, pool slots to limit parallelism across specific groups of tasks, and priority weights to control execution order when the queue is backed up.
In a Celery-based setup, workers are independent processes (often running on separate nodes), which means you can scale them horizontally. Need more capacity? Add more worker nodes. Need to isolate workloads? Route tasks to specific queues consumed by dedicated workers.
Triggerer
The triggerer is an optional but increasingly important component, introduced in Airflow 2.2. It handles deferrable tasks — tasks that are waiting for an external condition (an API response, a file landing, a database state change) without occupying a worker slot.
Before the triggerer existed, sensors would sit on a worker slot for the entire time they were waiting. A sensor polling every 30 seconds for a file that arrives in 2 hours would hold a worker slot for 2 hours doing almost nothing. At scale, this wastes significant capacity.
Deferrable tasks solve this by yielding their worker slot back to the pool and registering a trigger with the triggerer process. The triggerer uses asyncio to monitor many triggers concurrently with minimal resources. When the condition is met, the task is re-queued to a worker for completion.
Web Server / API Server
The web server provides the UI and REST API. It reads from the metadata database to render DAG status, task logs, run history, and configuration. Through the UI, you can pause and unpause DAGs, trigger manual runs, mark tasks as succeeded or failed, and inspect logs.
It’s a Flask application backed by Gunicorn, and it’s entirely stateless — it’s just a view layer on top of the metadata database. You can run multiple web server instances behind a load balancer without any session affinity concerns.
How It All Fits Together
┌─────────────────┐
│ DAG Files │
│ (Python code) │
└────────┬────────┘
│ parse & serialise
▼
┌─────────────────┐
│ DAG Processor │
└────────┬────────┘
│ write serialised DAGs
▼
┌──────────────┐ ┌─────────────────┐ ┌──────────────┐
│ Web Server │────────▶│ Postgres │◀────────│ Triggerer │
│ (UI/API) │ read │ (metadata db) │ read/ │ (deferred │
└──────────────┘ └────────┬────────┘ write │ tasks) │
│ └──────▲───────┘
read │ │ re-queue
▼ │ when ready
┌─────────────────┐ │
│ Scheduler │───────────────┘
│ (poll & queue) │
└────────┬────────┘
│ enqueue tasks
▼
┌─────────────────┐
│ Redis │
│ (message broker)│
└────────┬────────┘
│ dequeue & execute
┌────────┴────────┐
▼ ▼
┌────────────┐ ┌────────────┐
│ Worker 1 │ │ Worker N │
│ │ │ │
└─────┬──────┘ └─────┬──────┘
│ │
└────────┬────────┘
│ write results
▼
┌──────────┐
│ Postgres │
└──────────┘The lifecycle of a scheduled task flows through every component:
- DAG processor parses your DAG file and writes the serialised DAG to Postgres.
- Scheduler reads the DAG, determines a run is due, creates a DAG run and task instances in the database, and enqueues ready tasks into Redis.
- Redis holds the task in the queue until a worker is available.
- Worker picks up the task, executes it, and writes the outcome back to Postgres.
- Triggerer (if applicable) monitors deferred tasks and re-queues them when their conditions are met.
- Web server displays all of this in real time by reading from Postgres.
Each component is independently scalable and, with the exception of the metadata database, stateless. The database is the single point of coordination — which is both the strength of the design (simplicity, recoverability) and its primary scaling concern.
What Went Wrong: Lessons from Production
Silent OOM on Worker Nodes
We had a task that ran reliably for months — a data processing job that loaded a dataset into memory, transformed it, and wrote the output. Then one day it started getting killed mid-execution with no useful error in the Airflow logs. Just a task marked as failed with no traceback.
The root cause was upstream: the data source had grown significantly over time, but the task’s resource footprint hadn’t been re-evaluated. What used to fit comfortably in memory no longer did. The worker node’s OOM killer terminated the process silently — Airflow never got a chance to capture the exception because the process was killed at the OS level, not by Python.
This is an insidious failure mode because nothing about the DAG changed. The code was the same, the schedule was the same. The data just grew. The fix was straightforward — increase the worker node’s memory and add monitoring on task memory consumption — but the lesson was that any task processing variable-size input data needs resource limits and alerting that account for growth, not just the current state.
Worker Slot Exhaustion from Uncontrolled Concurrency
We gave users the ability to freely add DAGs to the platform — which was the right call for adoption, but created a capacity problem we didn’t anticipate soon enough.
Our worker slot count was sized roughly to match the total CPU across worker nodes. That arithmetic works when each task uses approximately one CPU core. It breaks when users write DAGs with high levels of task parallelism — a DAG with 50 concurrent tasks consumes 50 worker slots regardless of whether each task is CPU-light or CPU-heavy.
What happened in practice: a few teams deployed DAGs that fanned out aggressively. Their tasks claimed the majority of available worker slots, and every other DAG on the platform started queuing. The scheduler was doing its job — it was creating task instances and enqueuing them — but there were no slots available to execute them. From the user’s perspective, their DAGs were “stuck” for no apparent reason.
The fix involved several changes. We introduced pool slots to enforce per-team and per-DAG concurrency limits, preventing any single DAG from monopolising the cluster. We also revised our capacity model: total worker slots need to account for peak concurrent task demand across all DAGs, not just the number of DAGs or the CPU count. And we added guardrails for new DAGs — a review step that evaluates expected concurrency before a DAG goes live.
The broader takeaway: setting the right capacity at the start — and enforcing concurrency boundaries — is critical for a healthy scheduler. Airflow will faithfully queue everything you ask it to, even if there’s no realistic chance it can all run in time.
Seven components, one scheduling loop, and a few hard lessons.