PySpark Certification Practice Test 10

Debug The ETL Pipeline

Question 1 of 20

📋 Multiple Choice

Question 181

Problem: ```python df.filter(col("salary") > 10000) ``` Expected Output: 100 rows Actual Output: 1000 rows Most likely issue?

A. Partition issue

✓

B. Result not assigned back to DataFrame

C. Cache issue

D. Broadcast issue

Answer

✅ B

Explanation

### Fix ```python df = df.filter(col("salary") > 10000) ```

Answered: 0 / 20

PySpark Certification Practice Test 10

Debug The ETL Pipeline

Insightful Saga — Modern Data Engineering Certification Preparation

Question 181

Problem: ```python df.filter(col("salary") > 10000) ``` Expected Output: 100 rows Actual Output: 1000 rows Most likely issue?

A. Partition issue

B. Result not assigned back to DataFrame

C. Cache issue

D. Broadcast issue

Answer: ✅ B

Explanation:

### Fix ```python df = df.filter(col("salary") > 10000) ```

Question 182

Problem: ```python df.join(df2) ``` Output row count is extremely large. Most likely fix?

A. Add cache

B. Add collect

C. Add count

D. Add join condition

Answer: ✅ D

Question 183

Problem: ```python df.collect() ``` Job fails with Driver OOM. Most likely fix?

A. Add distinct

B. Add orderBy

C. Avoid collect on huge datasets

D. Increase columns

Answer: ✅ C

Question 184

Problem: ```python df.write.mode("append") ``` Duplicates appear every day. Best fix?

A. Use coalesce(1)

B. Cache data

C. Use collect

D. Implement incremental/deduplication logic

Answer: ✅ D

Question 185

Problem: A join takes 2 hours. Tables: ```text Transactions = 5 TB Reference = 20 MB ``` Fix?

A. collect()

B. coalesce(1)

C. Broadcast Reference Table

D. Full Outer Join

Answer: ✅ C

Question 186

Problem: ```python df.repartition(1) ``` Output size: ```text 600 GB ``` Best fix?

A. Remove single partition design

B. Use count()

C. Use more filters

D. Add collect()

Answer: ✅ A

Question 187

Problem: Pipeline runtime increased suddenly. Logs show: ```python Python UDF ``` processing billions of rows. Best recommendation?

A. Add union()

B. Add more print()

C. Use Spark Native Functions

D. Add count()

Answer: ✅ C

Question 188

Problem: Pipeline loads customer data. Business finds outdated records. Most likely fix?

A. Drop NULLs

B. Add repartition

C. Use collect

D. Latest Record Logic using row_number

Answer: ✅ D

Question 189

Problem: ```python dropDuplicates() ``` still leaves duplicates. Most likely issue?

A. Driver issue

B. Cache issue

C. Business keys not defined properly

D. Executor issue

Answer: ✅ C

Question 190

Problem: ```python orderBy() ``` takes very long. Dataset: ```text 3 Billion Rows ``` Most likely explanation?

A. Large Global Shuffle

B. Broadcast Failure

C. Missing Union

D. Missing Cache

Answer: ✅ A

Question 191

Problem: Job succeeds. Gold table row count: ```text 0 ``` Most likely next step?

A. Increase executors

B. Use cache

C. Increase cluster size

D. Validate upstream filters

Answer: ✅ D

Question 192

Problem: SCD Type 2 table has multiple active rows. Expected: ```text One active record ``` Most likely fix?

A. Add collect

B. Add cache

C. Add repartition

D. Correct end-date/current-flag logic

Answer: ✅ D

Question 193

Problem: Incremental load misses records. Watermark: ```python last_load_date ``` Most likely area to review?

A. Watermark Logic

B. Window Function

C. Broadcast Join

D. Cache Storage

Answer: ✅ A

Question 194

Problem: ```python df.count() ``` used repeatedly for debugging. Pipeline becomes slow. Best recommendation?

A. Remove unnecessary actions

B. Add more partitions

C. Add unions

D. Add more count()

Answer: ✅ A

Question 195

Problem: Customer table contains: ```text 1 customer 5 duplicate orders ``` Join multiplies rows. Best fix?

A. Add repartition

B. Add cache

C. Add count

D. Verify data cardinality before join

Answer: ✅ D

Question 196

Problem: Source schema changed. Pipeline starts failing. Best engineering solution?

A. Cache Everything

B. Schema Validation Framework

C. count()

D. collect()

Answer: ✅ B

Question 197

Problem: Driver CPU constantly at 100%. Logs show: ```python collect() foreach() local loops ``` Most likely root cause?

A. Large Processing on Driver

B. Missing Broadcast

C. Missing Cache

D. Missing Filter

Answer: ✅ A

Question 198

Problem: A skewed join causes one task to run much longer than others. Most likely technique?

A. collect()

B. Salting / AQE

C. union()

D. count()

Answer: ✅ B

Question 199

Problem: Business reports inconsistent revenue numbers. Two ETL pipelines calculate revenue differently. Best correction?

A. Repartition Data

B. Add Executors

C. Standardize Business Logic

D. Cache More

Answer: ✅ C

Question 200

Problem: A PySpark ETL solution works perfectly in QA but fails at production scale. What is the MOST likely missing consideration?

A. Print Statements

B. Column Naming

C. Scalability Testing

D. Notebook Comments

Answer: ✅ C

Explanation:

### Explanation Many ETL solutions work on millions of rows but fail on billions because scalability, skew, shuffles, memory usage, and execution plans were not evaluated properly. --- --- title: PySpark Certification Practice Test 11 description: Spark Optimization Scenarios --- # PySpark Certification Practice Test 11