Production datasets are reaching downstream systems with invalid, duplicate, and incomplete records. Design a centralized Data Quality Framework to validate incoming data, quarantine failed records, maintain audit logs, and improve trust across Databricks, Snowflake, and Executive Dashboards.
Production pipelines currently process every incoming record without validating business rules. Invalid, duplicate, and incomplete data is reaching Silver, Gold, Snowflake, and Executive Dashboards, creating reporting inconsistencies and reducing business confidence.
Engineering leadership has approved a centralized Data Quality Framework that validates incoming datasets, quarantines failed records, captures audit information, and continuously monitors data quality before records are consumed by downstream systems.
Implement a reusable Data Quality Framework that validates incoming production datasets before they are processed by downstream systems.
Validate null values, duplicates, schema mismatches and invalid business data.
Move failed records into quarantine for investigation while allowing valid records to continue.
Capture validation status, failure reasons and execution history for every pipeline run.
Track data quality metrics and provide operational visibility for production datasets.
During the past month, multiple production incidents have been traced back to poor data quality. Records with missing mandatory fields, duplicate customer IDs, invalid business values and schema mismatches are reaching downstream reporting systems.
The engineering team has been requested to design a reusable Data Quality Framework that automatically validates incoming datasets before they reach Silver, Gold, Snowflake and Executive Dashboards.
Poor data quality is affecting multiple business functions. Without a centralized validation framework, invalid records continue to propagate across the analytics platform, resulting in operational inefficiencies and reduced confidence in enterprise reporting.
Revenue dashboards contain duplicate and invalid business metrics.
Engineering teams spend significant time manually correcting production data issues.
Missing audit history and inconsistent validation reduce trust in enterprise data.
Poor-quality data increases the likelihood of incorrect business decisions.