Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →dicklesworthstone/spark-optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning.
Passed 21 of 23 test cases with the skill, and 18 of 23 without it.
3 fixed18 passed either way2 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 3 test cases the skill fixed. Same model, same prompt, with and without the skill.
We are writing a large PySpark DataFrame to S3 in Parquet format. To ensure fast reads and proper row group sizing, write the PySpark DataFrame write call configured with the optimal compression codec and block size.
| Checked | Without skill | With skill |
|---|---|---|
| Sets Parquet write compression option to snappy. | Passed | Passed |
| Configures option parquet.block.size to 128MB (134217728 bytes). | Missed | Passed |
The instructions your model reads when the skill loads.
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
| Factor | Impact | Solution |
|---|---|---|
| Shuffle | Network I/O, disk I/O | Minimize wide transformations |
| Data Skew | Uneven task duration | Salting, broadcast joins |
| Serialization | CPU overhead | Use Kryo, columnar formats |
| Memory | GC pressure, spills | Tune executor memory |
| Partitions | Parallelism | Right-size partitions |
test cases passed, vs 18 / 23 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.