Monitoring DAGs in Apache Airflow
SLAs, Alerts, Email, Webserver & Metrics 📈🚨
The Story: When Pipelines Go Silent
Imagine you’ve built a perfect Airflow DAG.
It runs daily, processes business-critical data, and everyone trusts it.
One morning, dashboards are empty.
No errors. No emails. No alerts.
The DAG did run — but it finished 4 hours late.
That’s when you realize:
Scheduling is not enough. Monitoring is mandatory.
In production Airflow, monitoring DAGs is as important as writing them. This article explains how Airflow monitors DAGs, how to configure SLAs, alerts, email notifications, UI monitoring, and metrics, and how to do it professionally at scale.
What Does “Monitoring a DAG” Really Mean?
Monitoring in Airflow answers five critical questions:
- Did the DAG run?
- Did it finish on time?
- Did any task fail or retry?
- Did performance degrade?
- Did anyone get notified?
Airflow provides multiple monitoring layers, not just one.
Monitoring Layers in Airflow
| Layer | Purpose |
|---|---|
| SLA | Detects slow tasks |
| Alerts | Reacts to failures |
| Notifies humans | |
| Webserver UI | Visual monitoring |
| Metrics | Long-term observability |
Each layer covers a different failure mode.
SLA Monitoring – Detecting “Slow Success”
What Is an SLA in Airflow?
An SLA (Service Level Agreement) defines how long a task is allowed to run.
📌 Important:
An SLA does not fail the task.
It triggers an SLA miss event.
SLA Example in a DAG
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta
with DAG(
dag_id="sales_reporting",
start_date=datetime(2024, 1, 1),
schedule_interval="@daily",
catchup=False,
) as dag:
generate_report = BashOperator(
task_id="generate_report",
bash_command="sleep 120",
sla=timedelta(minutes=1),
)
Input
| Parameter | Value |
|---|---|
| Task runtime | 120 seconds |
| SLA | 60 seconds |
Output (SLA Miss)
- Task succeeds ✅
- SLA is missed ❌
- SLA callback is triggered
- Visible in Airflow UI → Browse → SLA Misses
SLA Best Practices
✅ Use SLAs only for business-critical tasks
❌ Do not add SLAs to every task
✅ Combine SLAs with alert callbacks
Alerts & Callbacks – Reacting to Failures
Airflow allows callbacks to react to events.
Common Callbacks
| Callback | Trigger |
|---|---|
on_failure_callback | Task fails |
on_success_callback | Task succeeds |
sla_miss_callback | SLA missed |
Failure Alert Example
def notify_failure(context):
dag_id = context["dag"].dag_id
task_id = context["task_instance"].task_id
print(f"ALERT: {dag_id}.{task_id} failed")
task = BashOperator(
task_id="load_data",
bash_command="exit 1",
on_failure_callback=notify_failure,
)
Input
| Event | Value |
|---|---|
| Task exit code | 1 |
Output
ALERT: sales_reporting.load_data failed
📌 In production, this callback usually sends:
- Slack message
- PagerDuty alert
Email Alerts – The Simplest Notification Layer
Email is built-in and widely used.
Email Configuration Example
default_args = {
"email": ["data-team@example.com"],
"email_on_failure": True,
"email_on_retry": False,
}
with DAG(
dag_id="email_alert_dag",
default_args=default_args,
schedule_interval="@daily",
):
...
Input
| Condition | Value |
|---|---|
| Task state | FAILED |
Output
📧 Email sent with:
- DAG ID
- Task ID
- Execution date
- Log link
Email Best Practices
❌ Avoid email floods
✅ Email only on final failure
✅ Combine email with chat alerts
Webserver UI – Real-Time Visual Monitoring
The Airflow Web UI is the first monitoring tool most teams use.
Key UI Views for Monitoring
| View | Use Case |
|---|---|
| DAGs View | Overall health |
| Grid View | Task state history |
| Graph View | Dependencies |
| Gantt View | Performance bottlenecks |
| SLA Misses | Slow tasks |
What the UI Tells You Instantly
- Which DAG is failing
- Which task is retrying
- Where time is being spent
- Historical stability
📌 The UI reads directly from the metadata database.
Metrics – Production-Grade Monitoring
For serious environments, metrics are non-negotiable.
Airflow emits metrics via StatsD.
Common Airflow Metrics
| Metric | Meaning |
|---|---|
dagrun.duration | DAG runtime |
task.duration | Task runtime |
task.failures | Failure count |
scheduler.heartbeat | Scheduler health |
Example Metric Flow
Airflow → StatsD → Prometheus → Grafana
Input
| Task runtime | 95 seconds |
Output (Metric)
airflow.task.duration{dag="sales_reporting",task="generate_report"} 95
📊 This enables:
- SLA trend analysis
- Capacity planning
- Early performance warnings
Putting It All Together – A Monitoring Strategy
| Scenario | Monitoring Layer |
|---|---|
| Task failed | Callback + Email |
| DAG slow | SLA + Metrics |
| Scheduler down | Metrics |
| Manual investigation | Web UI |
No single tool is enough. Monitoring is a system.
Common Monitoring Mistakes
❌ Relying only on email
❌ Ignoring SLA misses
❌ No metrics in production
❌ Alert fatigue
❌ No ownership of alerts
Input & Output – Realistic Monitoring Scenario
Input
- DAG runs daily
- 20 tasks
- SLA = 30 minutes
- Email on failure
- Metrics enabled
Output
| Event | Result |
|---|---|
| Task fails | Email + alert |
| Task slow | SLA miss logged |
| DAG slower over time | Grafana trend |
| Scheduler unhealthy | Ops alert |
Summary 🧠
- Monitoring is not optional in Airflow
- SLAs detect slow success
- Alerts react to failures
- Email notifies humans
- UI helps investigate
- Metrics ensure long-term stability
Key Takeaways
- ✅ Use SLAs sparingly but meaningfully
- ✅ Always configure failure callbacks
- ✅ Email is a baseline, not a strategy
- ✅ Metrics are required for production
- ✅ Monitoring prevents silent failures
What’s Next? 🚀
➡️ Task Retries & Error Handling in Airflow
(How retries work, backoff strategies, and failure design patterns)