Question 1

A Spark application needs to read multiple Parquet files from a directory where the files have differing but compatible schemas. The data engineer wants to create a DataFrame that includes all columns from all files. Which code should the data engineer use to read the Parquet files and include all columns using Apache Spark?

Options :

A : spark.read.parquet("/data/parquet/")

B : spark.read.option("mergeSchema", True).parquet("/data/parquet/")

C : spark.read.format("parquet").option("inferSchema", "true").load("/data/parquet/")

D : spark.read.parquet("/data/parquet/").option("mergeAllCols", True)

Answer: B

Question 2

A data engineer needs to write a Streaming DataFrame as Parquet files. Given the code:

Given the code:

Options :

A :

format("parquet") .option("location", "path/to/destination/dir")

B :

CopyEdit .option("format", "parquet") .option("destination", "path/to/destination/dir")

C :

.option("format", "parquet") .option("location", "path/to/destination/dir")

D :

format("parquet") .option("path", "path/to/destination/dir")

Answer: D

Question 3

A Spark developer is developing a Spark application to monitor task performance across a cluster. One requirement is to track the maximum processing time for tasks on each worker node and consolidate this information on the driver for further analysis. Which technique should the developer use?

Options :

A : Broadcast a variable to share the maximum time among workers.

B : Configure the Spark UI to automatically collect maximum times.

C : Use an RDD action like reduce() to compute the maximum time.

D : Use an accumulator to record the maximum time on the driver.

Answer: C

Question 4

A Spark application suffers from too many small tasks due to excessive partitioning. How can this be fixed without a full shuffle?

Options :

A : Use the distinct() transformation to combine similar partitions

B : Use the coalesce() transformation with a lower number of partitions

C : Use the sortBy() transformation to reorganize the data

D : Use the repartition() transformation with a lower number of partitions

Answer: B

Question 5

A data scientist of an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user. Before further processing the data, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns in this DataFrame. The PII columns in df_user are first_name, last_name, email, and birthdate. Which code snippet can be used to meet this requirement?

Options :

A : df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

B : df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")

C : df_user_non_pii = df_user.dropfields("first_name", "last_name", "email", "birthdate")

D : df_user_non_pii = df_user.dropfields("first_name, last_name, email, birthdate")

Answer: A

Confidently Practice Online with Free Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Exam Cram

Buying Options: