Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →didhd/sagemaker-benchmark
Run a managed performance benchmark against a deployed Amazon SageMaker AI endpoint using SageMaker AI inference benchmarking (part of optimized GenAI inference recommendations; NVIDIA AIPerf under the hood). Measures TTFT, ITL, request-latency percentiles, and throughput.
Passed 17 of 20 test cases with the skill, and 10 of 20 without it.
and 10 more in the full report
9 fixed11 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
In the baseline AIPerf workload specification, what are the default mean token lengths configured for prompt input tokens (`prompt_input_tokens_mean`) and output tokens (`output_tokens_mean`)?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies 500 as the prompt_input_tokens_mean value | Missed | Passed |
| Identifies 256 as the output_tokens_mean value | Missed | Passed |
The instructions your model reads when the skill loads.
Managed performance benchmarking with Amazon SageMaker AI inference benchmarking (the benchmark capability within optimized GenAI inference recommendations). SageMaker runs NVIDIA AIPerf on managed compute and writes AIPerf results to S3 — no hand-built load generator, no self-managed AIPerf.
This is the baseline benchmark (pre-optimization). Optimization (speculative decoding / EAGLE 3, quantization, kernel tuning) is the recommendation/optimization path (
create_ai_recommendation_job/create_optimization_job) and is out of scope for the baseline beat.
sagemaker-deploy).create_ai_workload_config → create_ai_benchmark_job → poll describe_ai_benchmark_job.
| Field | Value |
|---|---|
| Target | endpoint + inference component from sagemaker-deploy |
| Workload | aiperf, the bundled datasets/sharegpt-curated.jsonl (real sharegpt prompts; see note below) |
| Profile | ~500 input / ~256 output tokens, concurrency 10, 300 requests |
| Output | s3://<sagemaker-default-bucket>/benchmark-output/ (auto-detected; never hardcoded) |
| Role | the SageMaker execution role (must trust sagemaker.amazonaws.com) |
Why a bundled dataset and not public_dataset: "sharegpt"? The raw feed derives each
request's output budget from the dataset's recorded answer lengths, and a fixed handful of
turns carry budgets of only 1–3 tokens. A reasoning model cannot emit visible content within
1–3 tokens (the budget is consumed before the answer channel opens), so those requests score
invalid and trip AIPerf's ~1% validity gate — deterministically, every run, regardless of
output_tokens_mean (per-request budgets ignore it) or min_tokens (capped per-request).
datasets/sharegpt-curated.jsonl is 500 real sharegpt prompts with output budgets ≥32
tokens: same realistic traffic, but the gate measures the endpoint, not a dataset artifact.
It is the default for any model; --public-dataset restores the raw feed.
test cases passed, vs 10 / 20 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.