The Customer Master pipeline currently performs a full data load every night. Due to rapid business growth, execution time has increased significantly, impacting SLA compliance and increasing infrastructure costs. The Data Engineering team has been asked to enhance the existing pipeline by implementing an incremental loading strategy that processes only newly created and modified customer records.
The existing Customer Master pipeline successfully delivers accurate data, however its full-load implementation has become increasingly expensive and time-consuming as customer volume has grown. Leadership has approved this enhancement to improve operational efficiency while maintaining the same business functionality.
Before implementing incremental loading, the engineering team reviewed the existing production pipeline to understand its current architecture, execution flow, and processing pattern. The pipeline is stable and functionally correct, but it reloads the complete customer dataset every night.