▸case-01 I am running a Spark 3.2 SQL query that joins several tables with unpredictable data sizes, leading to suboptimal execution plans. Provide the specific Spark configuration property I should enable to allow Spark to dynamically optimize the physical plan during runtime. Return just the configuration key and the boolean value. | pass→pass | 2,646 | 2,635 | -0% | 1 | 1 | 0% | 312 | 285 | -9% | 0 | 0 | — |
▸case-02 Write a PySpark snippet to read a CSV file hosted on AWS S3. Many legacy examples use the s3:// or s3n:// prefixes, but I need the current recommended protocol prefix for Hadoop 3.x that supports multipart uploads and better performance. Provide the spark.read.csv line. | pass→pass | 4,173 | 3,942 | -6% | 1 | 1 | 0% | 757 | 802 | +6% | 0 | 0 | — |
▸case-03 I am writing large Parquet datasets to AWS S3 using Spark. The default FileOutputCommitter is extremely slow due to S3's eventual consistency and rename penalties. Provide the specific Spark Hadoop configuration key and value to enable the high-performance committer designed specifically for S3A. | fail→fail | 7,800 | 6,890 | -12% | 1 | 1 | 0% | 1,527 | 1,340 | -12% | 0 | 0 | — |
▸case-04 I need to join a 500GB fact table with a 5MB dimension table in PySpark. A standard join causes a massive shuffle. Write the PySpark code to perform this join efficiently, ensuring the small table is sent to all worker nodes. | pass→pass | 6,200 | 5,259 | -15% | 1 | 1 | 0% | 1,111 | 954 | -14% | 0 | 0 | — |
▸case-05 I have an RDD of (String, Int) representing word counts. I need to aggregate the total count per word. Provide the most efficient PySpark RDD method to achieve this without causing out-of-memory errors during the shuffle phase. | pass→pass | 7,886 | 6,487 | -18% | 1 | 1 | 0% | 1,704 | 1,091 | -36% | 0 | 0 | — |
▸case-06 I am joining two massive Spark DataFrames on a 'customer_id' column. The join is hanging because 20% of the rows have a null or 'unknown' customer_id, causing severe data skew on a single partition. Describe the standard programmatic technique to distribute these skewed keys across multiple partitions before the join. | pass→pass | 17,111 | 16,603 | -3% | 1 | 1 | 0% | 3,087 | 3,120 | +1% | 0 | 0 | — |
▸case-07 I have a PySpark DataFrame with a string column containing JSON data. I need to extract a specific field from this JSON. While I could write a Python UDF using the json library, provide the more performant native PySpark SQL function to parse this column. | pass→pass | 9,409 | 8,089 | -14% | 1 | 1 | 0% | 1,848 | 1,662 | -10% | 0 | 0 | — |
▸case-08 My PySpark DataFrame currently has 2000 partitions after a massive filter operation. I want to reduce the number of partitions to 20 before writing to disk. Provide the PySpark method that minimizes network shuffle to achieve this reduction. | pass→pass | 3,408 | 3,623 | +6% | 1 | 1 | 0% | 654 | 634 | -3% | 0 | 0 | — |
▸case-09 I am deploying a Spark application to a multi-tenant YARN cluster. Instead of hardcoding the number of executors, I want Spark to scale executors up and down based on the workload. Provide the Spark configuration key and value to enable this feature. | pass→pass | 3,704 | 4,594 | +24% | 1 | 1 | 0% | 651 | 917 | +41% | 0 | 0 | — |
▸case-10 When enabling dynamic allocation in a Spark YARN cluster, executors may be removed when idle. To prevent losing shuffle data written by these removed executors, what additional Spark configuration property must strictly be enabled? | pass→pass | 5,915 | 5,691 | -4% | 1 | 1 | 0% | 1,041 | 931 | -11% | 0 | 0 | — |
▸case-11 My Spark application processes complex custom Java objects, and the default Java serialization is causing a major performance bottleneck. Provide the Spark configuration key and the fully qualified class name to switch to the faster, more compact serialization framework recommended for Spark. | pass→pass | 2,998 | 3,170 | +6% | 1 | 1 | 0% | 575 | 591 | +3% | 0 | 0 | — |
▸case-12 My Spark job runs on a large cluster where occasionally a single task takes 10x longer than the rest due to a degraded hardware node. Instead of writing custom timeout logic, provide the Spark configuration key to automatically re-launch these straggler tasks on other nodes. | pass→pass | 2,838 | 3,786 | +33% | 1 | 1 | 0% | 545 | 703 | +29% | 0 | 0 | — |
▸case-13 I am writing a PySpark DataFrame to disk for long-term analytical storage. I want to use the Parquet format, but I need to override the default Snappy compression with a modern codec that offers a better compression ratio while maintaining fast decompression speeds. Provide the spark.conf.set command for this. | pass→pass | 3,545 | 3,663 | +3% | 1 | 1 | 0% | 599 | 686 | +15% | 0 | 0 | — |
▸case-14 I need to check if a PySpark DataFrame contains any data before proceeding with a heavy transformation. Calling df.count() > 0 triggers a full dataset scan. Provide the most efficient PySpark DataFrame method to check for the existence of rows. | pass→pass | 9,989 | 8,184 | -18% | 1 | 1 | 0% | 1,915 | 1,302 | -32% | 0 | 0 | — |
▸case-15 I have a PySpark DataFrame that is 500GB, which exceeds my cluster's total memory. I need to reuse this DataFrame multiple times. Calling cache() will cause memory eviction issues. Provide the specific PySpark method and storage level argument to safely store this data across memory and disk. | pass→fail | 7,738 | 6,007 | -22% | 1 | 1 | 0% | 1,534 | 1,187 | -23% | 0 | 0 | — |
▸case-16 I am running a Spark SQL aggregation on a 5TB dataset. The default configuration causes out-of-memory errors because each shuffle partition is too large. Provide the Spark configuration key I must increase from its default value of 200. | pass→pass | 3,419 | 3,822 | +12% | 1 | 1 | 0% | 609 | 729 | +20% | 0 | 0 | — |
▸case-17 I have a PySpark DataFrame with 50 columns. I want to remove duplicate rows, but only considering the columns 'user_id' and 'timestamp'. Provide the PySpark DataFrame method and arguments to achieve this without using window functions. | pass→pass | 2,818 | 3,617 | +28% | 1 | 1 | 0% | 483 | 598 | +24% | 0 | 0 | — |
▸case-18 I have a PySpark DataFrame with 'date' and 'daily_sales'. I need to calculate a running total (cumulative sum) of sales ordered by date. Provide the PySpark code using the pyspark.sql.window module to define the ordering specification. | pass→pass | 7,189 | 5,263 | -27% | 1 | 1 | 0% | 1,541 | 1,106 | -28% | 0 | 0 | — |
▸case-19 My PySpark DataFrame has a column 'tags' which contains an array of strings. I need to transform the DataFrame so that each string in the array gets its own row, duplicating the other columns. Provide the specific PySpark SQL function to achieve this. | pass→pass | 3,703 | 3,115 | -16% | 1 | 1 | 0% | 591 | 525 | -11% | 0 | 0 | — |
▸case-20 I am running a PySpark application that relies heavily on complex Python libraries (like Pandas and Scikit-learn) inside UDFs. The executors keep getting killed by YARN for exceeding memory limits, even though the JVM heap is fine. Provide the specific Spark configuration key to increase the off-heap memory allocated for the Python worker processes. | pass→pass | 8,383 | 6,983 | -17% | 1 | 1 | 0% | 1,548 | 1,212 | -22% | 0 | 0 | — |
▸case-21 I need to schedule a daily data pipeline that runs a series of Python scripts in a specific dependency order. Provide a basic Apache Airflow script to define this schedule and execution order. | pass→pass | 9,408 | 8,858 | -6% | 1 | 1 | 0% | 2,075 | 1,784 | -14% | 0 | 0 | — |
▸case-22 I have a 5MB CSV file on my local laptop. I need to load it into memory and calculate a 7-day rolling average on the 'price' column. Provide the Python code using the standard single-node data manipulation library. | pass→pass | 6,210 | 5,785 | -7% | 1 | 1 | 0% | 1,197 | 1,103 | -8% | 0 | 0 | — |
▸case-23 I am configuring a Java Apache Kafka producer. I want to increase throughput by batching messages together before sending them over the network, even if it adds a few milliseconds of latency. Provide the specific Kafka producer configuration property to set this delay. | pass→pass | 3,930 | 3,905 | -1% | 1 | 1 | 0% | 756 | 713 | -6% | 0 | 0 | — |