Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →sickn33/spark-optimization
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning.
Passed 21 of 22 test cases with the skill, and 20 of 22 without it.
1 fixed20 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
A test case the skill fixed. Same model, same prompt, with and without the skill.
Our nightly PySpark pipeline runs extremely slowly when joining our main event log table with a smaller dimension table. Could you write a PySpark script that sets up an optimized SparkSession with appropriate adaptive and serialization configurations, performs the join using the best strategy for a small lookup dataset, and writes the final output efficiently? Please include the code snippet with comments explaining the key session properties and join mechanics.
| Checked | Without skill | With skill |
|---|---|---|
| The response applies `broadcast()` to the small dimension DataFrame before performing the join. | Passed | Passed |
| The response calls `.explain()` on the DataFrame to inspect the execution plan. | Missed | Passed |
The instructions your model reads when the skill loads.
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
resources/implementation-playbook.md.Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
test cases passed, vs 20 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.