Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →awslabs/hyperpod-performance-debugger
Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation.
Passed 21 of 22 test cases with the skill, and 8 of 22 without it.
and 17 more in the full report
13 fixed8 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 13 test cases the skill fixed. Same model, same prompt, with and without the skill.
Where can an administrator locate the output logs generated by SageMaker HyperPod deep health checks both on the central logging service and locally on a cluster instance?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies /var/log/aws/clusters/sagemaker-deep-health-check.log as the local host log path for deep health check results. | Missed | Passed |
| Specifies that CloudWatch log streams store health check results under the DeepHealthCheckResults log stream prefix. | Missed | Passed |
The instructions your model reads when the skill loads.
Route findings outside the two in-scope scenarios to the owner skill below.
| Concern observed | Route to |
|---|---|
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | hyperpod-node-debugger (§ F Hardware/Auto-Repair, § G GPU) |
Cannot allocate memory at os.fork(), root volume exhausted | hyperpod-node-debugger (§ I Resource Exhaustion) |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | hyperpod-nccl |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | hyperpod-version-checker |
| EFA self-referencing security-group rule missing — single node | hyperpod-node-debugger § A (EFA / Security Group) |
| EFA self-referencing security-group rule missing — cluster-wide | hyperpod-cluster-debugger § A (EFA Health Checks) |
| Slurm node state changes (drain / resume / reboot) | hyperpod-slurm-debugger |
| Diagnostic bundle for AWS Support | hyperpod-issue-report |
| Shell access on a node | hyperpod-ssm |
hyperpod-version-checker.hyperpod-node-debugger § G.scripts/perf-snapshot.sh (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.[CONCERN] line in the script output, open the matching section below and read the supporting reference.bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
# Scope to one suspect node:
test cases passed, vs 8 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.