PySpark Certification Practice Test 07
Find The Problem In The Code
PySpark Certification Practice Test 07
Find The Problem In The Code
Insightful Saga — Modern Data Engineering Certification Preparation
Question 121
A developer writes: ```python df = spark.read.parquet("/sales") result = df.collect() print(len(result)) ``` The dataset contains 2 billion records. What is the biggest problem?
### Explanation collect() moves all data to the Driver.
Question 122
A developer writes: ```python large_df.join(small_df, "customer_id") ``` Fact Table = 5 TB Lookup Table = 10 MB What optimization opportunity is missing?
Question 123
Review the code: ```python df.repartition(1) .write .parquet("/output") ``` Output size = 800 GB What is the main concern?
Question 124
A developer writes: ```python for i in range(100): print(df.count()) ``` What is the problem?
Question 125
Review: ```python df.show() df.show() df.show() ``` What is the concern?
Question 126
A Data Engineer writes: ```python df1.crossJoin(df2) ``` Both tables contain 50 million rows. What is the likely issue?
Question 127
Review the code: ```python df.withColumn("new_col1", ...) .withColumn("new_col2", ...) .withColumn("new_col3", ...) ... ``` More than 60 withColumn statements exist. What is the primary concern?
Question 128
Developer writes: ```python df.cache() ``` The DataFrame is used only once. What issue exists?
Question 129
Review: ```python df.write.mode("overwrite") ``` Target table = Production Customer Table What question should be asked?
Question 130
A developer creates: ```python df.dropDuplicates() ``` Business key is: (customer_id, order_id) What concern exists?
Question 131
Review: ```python customers.join( transactions, customers.id == transactions.id ) ``` Output records are unexpectedly higher. Most likely cause?
Question 132
Developer writes: ```python df.filter(col("amount") > 100) ``` but never stores the result. What is the issue?
Question 133
Review: ```python df.collect() for row in rows: process(row) ``` Used for 500 million rows. What is the main problem?
Question 134
Developer writes: ```python from pyspark.sql.functions import udf ``` A UDF is applied to 2 billion records even though Spark built-in functions exist. Concern?
Question 135
Review: ```python df.coalesce(1) ``` File size = 1.5 TB. Issue?
Question 136
Developer performs: ```python large_df.join(large_df2) ``` No filters. No partitions. No broadcast. Performance is terrible. What should be investigated first?
Question 137
Review: ```python df.write.mode("append") ``` Business users report duplicate records daily. Most likely concern?
Question 138
Developer writes: ```python df.count() ``` solely for troubleshooting. Table contains 5 billion records. Concern?
Question 139
Review: ```python df.orderBy("customer_name") ``` Table has 3 billion rows. What operation should engineers expect?
Question 140
A Principal Data Engineer reviews a notebook containing: - collect() - repartition(1) - repeated count() - unnecessary cache() - Python UDFs What is the overall feedback?
### Explanation The code may work on small datasets but will struggle significantly in production-scale environments. --- --- title: PySpark Certification Practice Test 08 description: Predict The Output Questions --- # PySpark Certification Practice Test 08