Skip to main content
DatabricksPySparkSnowflakeApache Airflow

Templating & Jinja Expressions in Apache Airflow

You’ve learned how to create DAGs, set dependencies, schedule them, and manage variables and connections. ✅

But what if your DAG needs dynamic behavior, like:

  • Running with today’s date
  • Referencing yesterday’s processed file
  • Passing parameters to tasks dynamically

This is where templating and Jinja expressions shine.

What is Templating in Airflow?​

Templating allows dynamic substitution of values in:

  • Bash commands
  • Python functions
  • SQL queries
  • File paths

Airflow uses Jinja as the templating engine.

📌 Rule:

Any parameter that supports templating can use Airflow Jinja variables like ds, ts, etc. at runtime.

Real-World Story: Dynamic Pipelines​

Imagine a pipeline that moves daily sales reports:

  • Input file: /data/sales_2024-12-16.csv
  • Output table: sales_2024_12_16

Hardcoding filenames every day is tedious.
With templating, Airflow automatically substitutes dates.

Jinja Expression Basics​

Common Placeholders​

{{ ds }}              Execution date (YYYY-MM-DD)
{{ ds_nodash }}       Execution date without dashes
{{ ts }}              Timestamp of DAG run
{{ ts_nodash }}       Timestamp without special characters
{{ macros.ds_add(ds, 1) }}  Add days to a date

Example 1: BashOperator with Templating​

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id="templating_bash_dag",
    start_date=datetime(2024, 1, 1),
    schedule_interval="@daily",
    catchup=False,
    tags=["templating", "bash"],
) as dag:

    task = BashOperator(
        task_id="print_date_file",
        bash_command="echo /data/sales_{{ ds_nodash }}.csv"
    )

Output​

/data/sales_20241216.csv

Example 2: PythonOperator with Templating​

from airflow.operators.python import PythonOperator

def print_file(ds, **kwargs):
    filename = f"/data/sales_{ds.replace('-', '')}.csv"
    print(f"Processing file: {filename}")

Example 3: PostgresOperator with Templating​

from airflow.providers.postgres.operators.postgres import PostgresOperator

query_task = PostgresOperator(
    task_id="insert_sales",
    postgres_conn_id="my_postgres",
    sql="""
        INSERT INTO sales_{{ ds_nodash }}
        SELECT * FROM raw_sales
        WHERE sale_date = '{{ ds }}';
    """
)

Runtime SQL​

INSERT INTO sales_20241216
SELECT * FROM raw_sales
WHERE sale_date = '2024-12-16';

Using Macros for Advanced Logic​

from airflow import macros

yesterday = "{{ macros.ds_add(ds, -1) }}"

Best Practices​

  • Use templating for dynamic filenames and queries
  • Use macros for date calculations
  • Keep templates simple and readable
  • Test using airflow tasks render

Common Mistakes​

  • Using unsupported templated fields
  • Hardcoding dates instead of ds
  • Overcomplicating Jinja logic in DAGs
  • Not testing templates before production

Key Takeaways​

  • Templating enables dynamic DAGs
  • Jinja variables like ds and ts are runtime values
  • Macros help with date manipulation
  • Works in Bash, Python, and SQL operators

Summary​

You learned:

  • What Airflow templating is
  • How Jinja expressions work
  • Built-in runtime variables
  • Macros for date operations
  • Real-world DAG examples

🎯 Your DAGs are now dynamic and production-ready.

What’s Next?​

PythonOperator Deep Dive