▸case-01 I am building a custom text normalization step in PySpark ML. I started by subclassing `object` and overriding `transform(self, dataset)`, but the pipeline throws type errors when calling `fit` or `transform`. How should the class definition and base classes be declared? | pass→pass | 17,241 | 19,734 | +14% | 1 | 1 | 0% | 2,425 | 3,318 | +37% | 0 | 0 | — |
▸case-02 I defined a custom Transformer subclass in PySpark ML and implemented my logic directly inside `def transform(self, dataset, params=None):`. However, parameter inheritance and validation are completely bypassed. Which internal method should be overridden instead? | pass→pass | 14,681 | 14,295 | -3% | 1 | 1 | 0% | 1,806 | 2,130 | +18% | 0 | 0 | — |
▸case-03 I want to add a configurable threshold parameter to my custom PySpark Transformer so it can be tuned or set via `setThreshold()`. Setting `self.threshold = threshold` in `__init__` fails when PySpark tries to copy or serialize the stage. How should custom parameters be declared? | pass→pass | 18,660 | 18,603 | -0% | 1 | 1 | 0% | 2,758 | 3,032 | +10% | 0 | 0 | — |
▸case-04 In my custom PySpark Transformer, I defined a `sensitivity` Param object. When calling `stage.setSensitivity(0.8)`, Python throws an `AttributeError`. How should setter and getter methods be implemented for custom Params? | pass→pass | 15,801 | 16,264 | +3% | 1 | 1 | 0% | 2,102 | 2,494 | +19% | 0 | 0 | — |
▸case-05 I am creating a PySpark Transformer that requires an input column of type `StringType`. If a user passes an `IntegerType` column, I want the pipeline to fail fast during schema validation before running the distributed job. Which method must be implemented to check column presence and data types? | pass→pass | 14,537 | 17,699 | +22% | 1 | 1 | 0% | 1,939 | 2,416 | +25% | 0 | 0 | — |
▸case-06 I need to run a complex SQL query with `CASE WHEN` logic and window functions as a stage inside a PySpark `Pipeline`. Instead of writing a custom Python class, which built-in PySpark ML transformer allows executing arbitrary SQL statements against input DataFrames? | pass→pass | 12,655 | 14,011 | +11% | 1 | 1 | 0% | 1,594 | 2,185 | +37% | 0 | 0 | — |
▸case-07 My PySpark custom transformer performs string lowercase and whitespace trimming inside `_transform`. My teammate wrote a Python UDF `udf(lambda x: x.lower().strip())`, but it is slowing down job execution on a large cluster. How should this transformation be implemented natively? | pass→pass | 14,848 | 19,003 | +28% | 1 | 1 | 0% | 1,766 | 3,085 | +75% | 0 | 0 | — |
▸case-08 When running hyperparameter tuning with PySpark ML `CrossValidator`, my custom Transformer fails during model cloning because parameter values are not preserved across copies. How should the `copy` method be overridden? | pass→pass | 21,480 | 20,510 | -5% | 1 | 1 | 0% | 3,219 | 3,336 | +4% | 0 | 0 | — |
▸case-09 I need to save my custom PySpark Transformer to disk with `transformer.save('path')` and reload it in another job with `CustomTransformer.load('path')`. Calling save throws an `NotImplementedError`. Which mixin classes or read/write interfaces need to be implemented? | pass→pass | 18,199 | 16,451 | -10% | 1 | 1 | 0% | 2,713 | 2,710 | -0% | 0 | 0 | — |
▸case-10 In my custom PySpark Transformer `_transform` method, if the requested `outputCol` already exists in the incoming DataFrame, the job quietly overwrites or creates ambiguous duplicate column references. What check should be performed in `_transform` or `transformSchema` to prevent this? | pass→pass | 15,472 | 13,913 | -10% | 1 | 1 | 0% | 2,041 | 2,120 | +4% | 0 | 0 | — |
▸case-11 I have multiple numeric feature columns (`age`, `income`, `credit_score`) that need to be combined into a single vector column before feeding into a Spark ML algorithm. What standard PySpark Transformer should be added to the pipeline? | pass→pass | 5,583 | 13,024 | +133% | 1 | 1 | 0% | 1,086 | 1,691 | +56% | 0 | 0 | — |
▸case-12 I need to replace null values in continuous numeric columns with the median value of each column. Should I write a custom Transformer using `_transform` or use an existing PySpark ML component? | pass→pass | 15,836 | 18,566 | +17% | 1 | 1 | 0% | 1,971 | 2,819 | +43% | 0 | 0 | — |
▸case-13 I am using `StringIndexer` in my PySpark ML pipeline. During inference on test data, unseen categorical labels appear, causing `Unseen label` runtime exceptions. How should the transformer be configured to skip or encode unseen categories? | pass→pass | 9,511 | 16,900 | +78% | 1 | 1 | 0% | 1,808 | 2,711 | +50% | 0 | 0 | — |
▸case-14 After converting categorical strings to category indices with `StringIndexer`, I need to convert those integer indices into sparse binary vectors for a linear model. Which PySpark Transformer should be applied next? | pass→pass | 10,421 | 11,047 | +6% | 1 | 1 | 0% | 947 | 1,470 | +55% | 0 | 0 | — |
▸case-15 I need to multiply each feature vector in a PySpark DataFrame by a fixed weight vector (scaling each feature independently). Which PySpark ML transformer handles element-wise vector scaling? | pass→pass | 6,386 | 8,027 | +26% | 1 | 1 | 0% | 1,274 | 1,878 | +47% | 0 | 0 | — |
▸case-16 I am writing unit tests with `pytest` for a custom PySpark Transformer class. Creating a new `SparkSession` inside every single test function makes the test suite extremely slow. How should the PySpark session fixture be scoped in `conftest.py`? | pass→pass | 16,815 | 13,835 | -18% | 1 | 1 | 0% | 2,122 | 2,990 | +41% | 0 | 0 | — |
▸case-17 My custom PySpark Transformer performs string concatenation of two columns using `concat()`. When input values contain `NULL`, the result becomes `NULL`. I want missing values treated as empty strings `""`. How should this be handled natively inside `_transform`? | pass→pass | 18,859 | 18,169 | -4% | 1 | 1 | 0% | 2,743 | 3,062 | +12% | 0 | 0 | — |
▸case-18 My custom PySpark Transformer needs to map country codes using a large lookup dictionary (50,000 entries) inside `_transform`. Passing the dictionary to a UDF directly causes heavy serialization overhead across worker nodes. How should this lookup table be shared efficiently across workers? | pass→pass | 20,524 | 23,077 | +12% | 1 | 1 | 0% | 2,778 | 3,770 | +36% | 0 | 0 | — |
▸case-19 I am assembling a `pyspark.ml.Pipeline` with `StringIndexer`, `OneHotEncoder`, and `VectorAssembler`. In what order must these stages be listed in the `stages` parameter list? | pass→pass | 6,516 | 16,169 | +148% | 1 | 1 | 0% | 1,249 | 2,179 | +74% | 0 | 0 | — |
▸case-20 I need to train a logistic regression model in PySpark ML on training data to learn model coefficients, evaluate convergence metrics, and save the fitted model weights. How do I configure my Transformer stage to fit coefficients to data? | pass→pass | 16,190 | 14,558 | -10% | 1 | 1 | 0% | 2,254 | 3,329 | +48% | 0 | 0 | — |
▸case-21 I need to construct an Apache Airflow DAG using `EmrAddStepsOperator` and `EmrStepSensor` to submit a PySpark batch job to an AWS EMR cluster on a daily CRON schedule `0 2 * * *`. Please provide the Airflow DAG definition code. | fail→fail | 16,695 | 16,974 | +2% | 1 | 1 | 0% | 2,645 | 2,884 | +9% | 0 | 0 | — |
▸case-22 Our Delta Lake tables on Databricks are retaining unreferenced historical data files after frequent updates, causing storage costs to spike. How do I execute `VACUUM` and `OPTIMIZE` commands with retention thresholds in PySpark SQL? | fail→fail | 38,382 | 13,581 | -65% | 1 | 1 | 0% | 8,237 | 3,050 | -63% | 0 | 0 | — |
▸case-23 I am reading real-time events from Apache Kafka using `spark.readStream`. I need to configure dynamic event-time sliding windows with a 10-minute watermark to handle late-arriving data and write results to append mode. How should this streaming query be defined? | fail→fail | 20,761 | 19,378 | -7% | 1 | 1 | 0% | 3,210 | 3,264 | +2% | 0 | 0 | — |