Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →data-goblin/executing-spark
Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
Passed 21 of 21 test cases with the skill, and 9 of 21 without it.
and 18 more in the full report
13 fixed9 passed either way0 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 12 test cases the skill fixed. Same model, same prompt, with and without the skill.
When initializing a session with `POST .../sessions`, which session state property value must the caller wait for before posting statement payloads?
| Checked | Without skill | With skill |
|---|---|---|
| Polls until session state equals idle. | Missed | Passed |
The instructions your model reads when the skill loads.
Run arbitrary PySpark or Python code on Fabric Spark compute via the Livy API. No notebook artifact is created or persisted; sessions are ephemeral. Full read/write access to lakehouse Delta tables via Spark SQL.
az login)The Livy API requires a token from az account get-access-token --resource https://api.fabric.microsoft.com. Tokens from fab auth do not work for OneLake storage access inside the Spark session.
import subprocess, json
result = subprocess.run(
["az", "account", "get-access-token", "--resource", "https://api.fabric.microsoft.com"],
capture_output=True, text=True
)
token = json.loads(result.stdout)["accessToken"]
Do not output or log the token. Pass it directly to the API call.
1. Create session POST .../sessions {"kind": "pyspark"}
2. Wait for idle GET .../sessions/{id} poll until state: "idle" (~30-90s)
3. Submit code POST .../sessions/{id}/statements {"code": "...", "kind": "pyspark"}
4. Get result GET .../sessions/{id}/statements/{n} poll until state: "available"
5. Delete session DELETE .../sessions/{id} ALWAYS do this
Base URL: https://api.fabric.microsoft.com/v1/workspaces/{wsId}/lakehouses/{lhId}/livyapi/versions/2023-12-01
CRITICAL: Always delete sessions when done. Idle sessions consume Fabric capacity units (CUs). A forgotten session burns compute until it times out (default: 20 minutes). In automation, wrap cleanup in a finally block.
test cases passed, vs 9 / 21 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.