▸case-01 I am configuring a spark-submit script for a heavy aggregation job on AWS EMR. I have assigned 8GB per executor, but memory-intensive off-heap operations and driver/executor overhead are causing OutOfMemory errors. A colleague suggested setting executor memory to 32GB without adjusting off-heap or overhead fractions. How should the spark-submit memory overhead flag be specified to allocate explicit overhead memory beyond executor memory? | pass→pass | 17,669 | 21,744 | +23% | 1 | 1 | 0% | 2,335 | 3,610 | +55% | 0 | 0 | — |
▸case-02 We need to parse a standard ISO timestamp string in a PySpark DataFrame column into a Unix epoch integer across billions of rows. Developers want to write a custom Python function wrapped in `udf(lambda x: ...)` for quick implementation. What is the recommended native PySpark expression or function to avoid Python process serialization overhead? | pass→pass | 15,393 | 16,153 | +5% | 1 | 1 | 0% | 2,013 | 2,463 | +22% | 0 | 0 | — |
▸case-03 We are joining a 50MB dim_store lookup dataset with a 2TB fact_sales dataset in Spark SQL. Spark is defaulting to a expensive SortMergeJoin and shuffling the 2TB table across all nodes. What configuration parameter controls the maximum table size in bytes for auto-broadcasting, and how should it be adjusted for this join? | pass→pass | 13,749 | 14,440 | +5% | 1 | 1 | 0% | 1,689 | 2,095 | +24% | 0 | 0 | — |
▸case-04 Our Scala Spark job processes custom domain objects during shuffle operations, resulting in slow Java default serialization and huge spill files. What Spark configuration setting switches the default Java serializer to the high-performance Kryo serializer? | pass→pass | 11,285 | 11,434 | +1% | 1 | 1 | 0% | 1,239 | 1,624 | +31% | 0 | 0 | — |
▸case-05 When writing a PySpark Structured Streaming query from Kafka to Delta Lake, we want to ensure exact-once processing and state recovery across driver restarts. Where should state metadata be stored, and what option parameter must be passed to the DataStreamWriter? | pass→pass | 12,036 | 11,886 | -1% | 1 | 1 | 0% | 1,359 | 1,625 | +20% | 0 | 0 | — |
▸case-06 After running a filter on a large dataset in PySpark, our DataFrame is distributed across 1000 partitions, but total remaining records are small. We want to write out 10 clean Parquet files to S3 without triggering a full stage shuffle across the cluster. Which DataFrame method should be called? | pass→pass | 10,687 | 10,783 | +1% | 1 | 1 | 0% | 1,044 | 1,398 | +34% | 0 | 0 | — |
▸case-07 We run interactive and batch Spark jobs on a shared YARN/Kubernetes cluster where resource utilization needs to scale up during bursts and scale down when idle. Which main Spark SQL configuration property enables dynamic executor allocation? | pass→pass | 8,945 | 10,126 | +13% | 1 | 1 | 0% | 674 | 1,321 | +96% | 0 | 0 | — |
▸case-08 We have a mathematical model that requires scipy/numpy vectorized operations on PySpark DataFrame groups. Standard Python UDFs process one row at a time and run extremely slowly. Which PySpark UDF feature uses Apache Arrow for batch processing across PyArrow / Pandas series? | fail→pass | 13,721 | 14,935 | +9% | 1 | 1 | 0% | 1,525 | 2,181 | +43% | 0 | 0 | — |
▸case-09 In our production batch ingestion job reading millions of multi-nested JSON files from S3, the team relies on `spark.read.json(path)` without specifying schema, causing job launch latency while Spark scans the entire dataset to infer types. How should production Spark readers declare data structure? | pass→pass | 19,274 | 20,553 | +7% | 1 | 1 | 0% | 2,623 | 3,330 | +27% | 0 | 0 | — |
▸case-10 During Spark writes to AWS S3 using PySpark, final file rename operations take up to 40% of job execution time due to S3's lack of atomic directory renames. Which specialized S3A committer protocol should be configured to eliminate rename overhead? | pass→pass | 16,235 | 13,863 | -15% | 1 | 1 | 0% | 2,134 | 2,672 | +25% | 0 | 0 | — |
▸case-11 In Spark 3.x, we want Spark to dynamically combine small post-shuffle partitions, optimize join strategies at runtime, and handle skew automatically. Which top-level adaptive feature configuration flag manages this capability? | pass→pass | 3,980 | 4,283 | +8% | 1 | 1 | 0% | 773 | 1,159 | +50% | 0 | 0 | — |
▸case-12 We are joining two tables on `user_id`, but 80% of incoming events have `user_id = NULL` or `user_id = 'GUEST'`, creating extreme skew where one executor task runs for hours while others finish in seconds. What data engineering pattern breaks up skewed keys using synthetic random suffixes before join? | pass→pass | 10,467 | 19,110 | +83% | 1 | 1 | 0% | 2,038 | 2,474 | +21% | 0 | 0 | — |
▸case-13 We are orchestrating a PySpark script execution from Apache Airflow on a remote YARN cluster. The team considers using BashOperator with raw curl commands. Which native Airflow provider operator is designed specifically to execute spark-submit jobs with typed application arguments? | pass→pass | 13,220 | 17,330 | +31% | 1 | 1 | 0% | 1,455 | 2,731 | +88% | 0 | 0 | — |
▸case-14 In a cloud cluster, a few random node instances experience transient CPU throttling, causing 99% of Spark tasks to finish in 2 minutes while 2 tasks hang for 30 minutes. Which Spark setting detects stragglers and launches duplicate copies of slow tasks? | pass→pass | 10,037 | 11,543 | +15% | 1 | 1 | 0% | 898 | 1,529 | +70% | 0 | 0 | — |
▸case-15 A PySpark job reads raw events, performs an expensive regex transformation, and then uses that transformed DataFrame to compute both daily metrics `df.groupby('date').count().write(...)` and user metrics `df.groupby('user').count().write(...)`. The regex runs twice. What action should be taken after transformation? | pass→pass | 12,767 | 7,202 | -44% | 1 | 1 | 0% | 1,388 | 1,679 | +21% | 0 | 0 | — |
▸case-16 We run a PySpark batch job that shuffles 500MB of data on a local test cluster with 4 CPU cores. Spark defaults to 200 output shuffle partitions, resulting in 200 tiny tasks processing < 3MB each and high scheduling overhead. Which configuration setting lowers the default shuffle partition count? | pass→pass | 10,556 | 6,979 | -34% | 1 | 1 | 0% | 1,048 | 1,592 | +52% | 0 | 0 | — |
▸case-17 We are building an incremental ingestion job into a Delta Lake table where new record updates must overwrite matching existing records by `order_id` and insert non-matching records. What standard Delta SQL/DataFrame syntax handles this upsert operation? | pass→pass | 9,005 | 7,915 | -12% | 1 | 1 | 0% | 1,816 | 1,850 | +2% | 0 | 0 | — |
▸case-18 To debug completed Spark jobs in a web UI after the cluster terminates, where must event logs be recorded and which setting enables event logging? | pass→pass | 10,354 | 12,078 | +17% | 1 | 1 | 0% | 974 | 1,621 | +66% | 0 | 0 | — |
▸case-19 In a Spark Structured Streaming pipeline reading from Event Hubs, we want the streaming query to process available micro-batches every 5 minutes rather than continuously running in tight loops. How should the trigger be configured in PySpark? | pass→pass | 8,163 | 15,192 | +86% | 1 | 1 | 0% | 1,442 | 2,164 | +50% | 0 | 0 | — |
▸case-20 I need an Apache Airflow DAG that fetches JSON payloads from an external REST endpoint using the requests library every morning and posts the raw output to a Slack Webhook. I do not want any Spark or distributed computing involvement. How should I structure the PythonOperator task? | pass→pass | 13,879 | 19,906 | +43% | 1 | 1 | 0% | 2,778 | 3,424 | +23% | 0 | 0 | — |
▸case-21 How do I use the standard Apache Kafka CLI tools (`kafka-configs.sh`) to change the topic message retention time to 7 days (604800000 ms) for an existing topic named `user-clicks` on broker `localhost:9092`? | pass→pass | 12,188 | 9,325 | -23% | 1 | 1 | 0% | 1,505 | 1,243 | -17% | 0 | 0 | — |
▸case-22 We have a monolithic PostgreSQL database table `orders` with 50 million rows. Queries filtering by `customer_id` and `created_at` are performing full table scans. Write the SQL DDL statement to create a non-blocking composite B-Tree index on these columns. | pass→pass | 10,828 | 9,537 | -12% | 1 | 1 | 0% | 904 | 1,183 | +31% | 0 | 0 | — |