Incremental Filter

How can we query only records updated after the highest timestamp in our target table?

Advanced⏱ 3 Minutes⭐ 1910 XP

🤔 THE QUESTION

How can we query only records updated after the highest timestamp in our target table?

💡 WHAT IS IT?

Using a subquery for MAX(updated_at) implements a dynamic high-watermark incremental ETL filter.

🎯 WHAT IS IT USED FOR?

High-watermark incremental loads, minimizing compute costs, continuous data warehouse synchronization.

💻 EXAMPLE

SELECT *
FROM source_orders
WHERE updated_at > (
    SELECT MAX(updated_at)
    FROM target_orders
);

Practice typing production-grade SQL code for Incremental Filter.