Development Arena
Build production data engineering pipelines from the ground up across modern lakehouse and warehouse architectures. Architect Bronze, Silver, and Gold layers, configure Delta tables, integrate external APIs, design dimensional data marts, and implement CDC pipelines.
Build Customer Lakehouse Pipeline
Build a production-ready customer data pipeline from source ingestion through Bronze, Silver, and Gold layers for analytics consumption.
Incremental Sales Pipeline
Build a reliable incremental sales pipeline that processes cloud storage transaction files, handles duplicates and late data, and delivers trusted analytics datasets.
REST API Integration Pipeline
Build an ingestion pipeline consuming third-party REST API customer activity with pagination, transient error handling, rate limiting, and Silver/Gold curation.
Enterprise Sales Reporting Data Mart
Design and implement a dimensional reporting data mart with explicit grain definition, fact/dimension relationships, SCD handling, and Snowflake reconciliation.
CDC-Based Customer Data Pipeline
Design a Change Data Capture (CDC) processing pipeline handling INSERT, UPDATE, and DELETE event streams with out-of-order sequencing, replay, and current-state materialization.
Real-Time Sales Streaming Pipeline
Build a low-latency streaming pipeline continuously ingesting real-time sales events, handling watermarking, late data, stateful micro-batch aggregations, and Snowflake synchronization.
Customer ML Feature Engineering Pipeline
Design a production-grade ML feature engineering pipeline transforming customer and transaction history into reusable feature stores with strict point-in-time leakage prevention.
Production CI/CD Data Platform
Implement automated GitHub Actions CI/CD workflows for multi-environment data pipelines with linting, unit/integration testing, secret management, and rollback mechanisms.
Multi-Source Enterprise Data Platform
Build a unified enterprise platform integrating Customer DB, Sales Files, Product REST API, and Store Reference Data with entity resolution, SLA freshness, and cross-source reconciliation.
Enterprise Data Engineering Capstone
Architect and build an end-to-end enterprise lakehouse platform combining batch ingestion, real-time streaming, CDC, REST APIs, ML feature stores, CI/CD, and Snowflake data marts.