▸case-01 Our nightly PySpark job is taking over four hours to run because of a huge join operation between a large event dataset and a smaller lookup table, leading to frequent out-of-memory errors on worker nodes. Can you review this issue and create a practical troubleshooting guide with code updates and cluster configuration recommendations to streamline the join and eliminate memory bottlenecks? | fail→fail | 26,749 | 24,851 | -7% | 1 | 1 | 0% | 3,830 | 4,086 | +7% | 0 | 0 | — |
▸case-02 We are scaling up a data pipeline that processes terabytes of data daily, but our cluster execution suffers from severe data skew and poor core utilization. Please give me a detailed action plan including partition strategy adjustments, executor memory tuning settings, and verification steps to balance task distribution and lower execution time. | fail→fail | 22,833 | 25,687 | +12% | 1 | 1 | 0% | 4,119 | 4,183 | +2% | 0 | 0 | — |
▸case-03 Our Spark pipeline runs an iterative algorithm that recalculates the same intermediate DataFrame repeatedly, causing long stage durations and high shuffle spill. I need a clear step-by-step performance optimization checklist detailing appropriate caching techniques, persistence storage levels, and config tweaks to eliminate redundant computations. | fail→fail | 19,708 | 21,594 | +10% | 1 | 1 | 0% | 3,488 | 3,424 | -2% | 0 | 0 | — |
▸case-04 A PySpark pipeline processes streaming event logs and writes out millions of tiny 1KB Parquet files to S3, causing extremely slow read performance for downstream Athena queries. A junior developer proposed setting spark.sql.shuffle.partitions to 1 right before writing to solve this. Provide an architectural review and technical recommendation to fix the small file problem without causing driver out-of-memory errors. | fail→fail | 23,237 | 24,697 | +6% | 1 | 1 | 0% | 3,818 | 4,778 | +25% | 0 | 0 | — |
▸case-05 Our Spark batch job aggregates a 500GB dataset by user_id to compute daily metrics. The job spends 80% of its time in stage shuffles with high network spill. The lead dev suggested converting the PySpark DataFrame to a Pandas DataFrame and using Pandas groupby. Recommend a Spark-native solution for optimizing the aggregation and reducing shuffle write overhead. | fail→fail | 23,319 | 21,940 | -6% | 1 | 1 | 0% | 3,241 | 3,447 | +6% | 0 | 0 | — |
▸case-06 We attempted to optimize a join by applying explicit PySpark broadcast hints on a dimension table. However, the driver node crashes with an OutOfMemoryError during physical plan execution. A developer suggested setting spark.sql.autoBroadcastJoinThreshold to -1 and forcing broadcast on all tables. Provide a root-cause analysis and configuration remedies. | fail→fail | 21,500 | 23,736 | +10% | 1 | 1 | 0% | 2,858 | 3,594 | +26% | 0 | 0 | — |
▸case-07 During long-running Spark jobs processing Parquet files on Kubernetes, executor pods frequently exit with code 137 (OOM killed) or freeze during GC pauses lasting over 30 seconds. Someone suggested increasing spark.executor.memory to 64GB per executor with 16 cores per executor. Evaluate this suggestion and provide an optimal executor configuration and memory tuning strategy. | fail→fail | 23,902 | 26,439 | +11% | 1 | 1 | 0% | 3,269 | 4,253 | +30% | 0 | 0 | — |
▸case-08 We are upgrading an existing PySpark batch workflow to Apache Spark 3.x and want to leverage Adaptive Query Execution to handle fluctuating daily data volumes. The current codebase has hardcoded spark.sql.shuffle.partitions = 2000. Recommend the proper AQE configuration parameters to dynamically handle shuffle partitions and skew. | fail→fail | 20,367 | 16,861 | -17% | 1 | 1 | 0% | 2,950 | 2,800 | -5% | 0 | 0 | — |
▸case-09 A machine learning pipeline repeatedly fits models on a 150GB preprocessed Spark DataFrame across 50 iterations. The team currently uses df.cache() default storage level, but executor nodes keep dropping cached blocks to disk, causing severe execution latency. Recommend the optimal storage level strategy and memory tuning configuration for this iterative job. | fail→fail | 17,132 | 19,269 | +12% | 1 | 1 | 0% | 3,162 | 3,732 | +18% | 0 | 0 | — |
▸case-10 A PySpark data transformation applies a custom Python function via spark.sql.functions.udf to extract regex patterns from a 100 million row text column. The pipeline is running 10x slower than expected. A teammate suggested wrapping the UDF in a Python process pool executor inside the function. Provide a proper Spark optimization approach to replace or accelerate this UDF. | fail→fail | 19,341 | 15,629 | -19% | 1 | 1 | 0% | 2,742 | 3,443 | +26% | 0 | 0 | — |
▸case-11 Our Spark data lake ingests IoT telemetry data into Parquet format every 5 minutes. Querying daily partitions takes over 15 minutes due to partition scan overhead across tens of thousands of files. Explain how to design a file compaction job in Spark, contrasting repartition vs coalesce, and specifying file size target parameters. | fail→fail | 25,705 | 24,227 | -6% | 1 | 1 | 0% | 3,744 | 4,025 | +8% | 0 | 0 | — |
▸case-12 When running PySpark jobs with heavy MLlib matrix multiplications and C-extensions, workers crash with 'Container killed by YARN for exceeding memory limits'. A developer recommended increasing spark.executor.memory by 20GB without touching overhead configurations. Diagnose the issue and provide the exact config parameters to adjust. | fail→fail | 22,127 | 15,705 | -29% | 1 | 1 | 0% | 3,062 | 3,215 | +5% | 0 | 0 | — |
▸case-13 An analytics pipeline performs a full outer join between two 1TB datasets on country_code and status columns. A single partition takes 3 hours to complete while all other 199 partitions finish in 2 minutes. Explain how to handle this skew using two-pass joining or key salting in PySpark. | fail→fail | 31,193 | 44,196 | +42% | 1 | 1 | 0% | 5,020 | 8,306 | +65% | 0 | 0 | — |
▸case-14 Our shared Spark cluster experiences high resource contention during peak hours because batch jobs hold onto fixed executor allocations even when idle during driver-side data validation stages. Outline how to configure dynamic executor allocation to scale down idle executors. | fail→fail | 18,669 | 18,280 | -2% | 1 | 1 | 0% | 2,419 | 2,708 | +12% | 0 | 0 | — |
▸case-15 Two large tables (Orders and OrderItems, each 2TB) are joined 50 times a day on order_id. Every query execution incurs massive shuffle overhead. A team member suggested caching both 2TB tables in memory. Provide a persistent storage optimization strategy using Spark SQL bucketing. | fail→fail | 21,016 | 20,618 | -2% | 1 | 1 | 0% | 2,885 | 3,271 | +13% | 0 | 0 | — |
▸case-16 A PySpark job reads historical transaction logs partitioned by year/month/day from Amazon S3. The job spends 10 minutes listing S3 objects before starting processing. Someone suggested disabling partition discovery altogether. Recommend proper partition pruning and Spark configuration to accelerate directory listing and file scanning. | fail→fail | 23,128 | 22,558 | -2% | 1 | 1 | 0% | 3,178 | 3,495 | +10% | 0 | 0 | — |
▸case-17 A window function operation Window.partitionBy("customer_id").orderBy("transaction_timestamp") fails with OutOfMemoryError because a single corporate customer account has over 500 million transactions. Provide a strategy to rewrite or restructure this window calculation in PySpark. | fail→fail | 24,082 | 31,373 | +30% | 1 | 1 | 0% | 3,575 | 5,422 | +52% | 0 | 0 | — |
▸case-18 A legacy Spark job running on PySpark/Scala spends excessive CPU time serializing Java objects during inter-stage shuffles. The current configuration uses default Java serialization. Recommend the optimal serialization framework and configurations for Spark shuffle and caching. | fail→fail | 18,853 | 19,520 | +4% | 1 | 1 | 0% | 2,642 | 3,322 | +26% | 0 | 0 | — |
▸case-19 A PySpark script executes df.collect() on a dataset that filtered down to 50GB, triggering driver Java heap space OutOfMemory errors. A developer proposed increasing spark.driver.maxResultSize to 100g to allow the collect operation to complete. Evaluate this proposal and provide a production-ready alternative. | fail→fail | 19,795 | 14,571 | -26% | 1 | 1 | 0% | 2,707 | 3,084 | +14% | 0 | 0 | — |
▸case-20 We are configuring an Apache Flink streaming pipeline that processes Clickstream events from Kafka. The Flink job suffers from backpressure on the window operator during traffic spikes. How should we tune Flink checkpointing intervals, managed memory fractions, and task slot allocations to mitigate backpressure? | pass→pass | 26,101 | 18,267 | -30% | 1 | 1 | 0% | 3,614 | 3,571 | -1% | 0 | 0 | — |
▸case-21 Our Apache Kafka cluster is experiencing severe partition skew on a key topic with 32 partitions, causing specific consumer instances in a consumer group to lag significantly. How can we rebalance Kafka topic partitioning and adjust consumer configuration settings like max.poll.records to equalize partition distribution? | pass→pass | 18,435 | 24,604 | +33% | 1 | 1 | 0% | 3,145 | 3,921 | +25% | 0 | 0 | — |
▸case-22 Our PostgreSQL analytical data warehouse is experiencing slow query response times on complex multi-table joins involving millions of rows. How should we tune PostgreSQL work_mem, shared_buffers, and CREATE INDEX parameters (such as BRIN or B-tree indices) to improve join query execution speed? | pass→pass | 33,465 | 21,309 | -36% | 1 | 1 | 0% | 4,916 | 2,876 | -41% | 0 | 0 | — |
▸case-23 We run ad-hoc queries on Trino (formerly Presto) against Amazon S3 Parquet tables. Queries with large GROUP BY operations fail with query exceeded memory limit per node (user memory limit). How should we adjust Trino query memory configs like query.max-memory-per-node and query spill settings to resolve this? | pass→pass | 19,273 | 20,864 | +8% | 1 | 1 | 0% | 3,231 | 3,272 | +1% | 0 | 0 | — |