Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →aws-samples/sagemaker-benchmark
Run a managed performance benchmark against a deployed Amazon SageMaker AI endpoint using SageMaker AI inference benchmarking (part of optimized GenAI inference recommendations; NVIDIA AIPerf under the hood). Measures TTFT, ITL, request-latency percentiles, and throughput.
Passed 16 of 21 test cases with the skill, and 6 of 21 without it.
and 12 more in the full report
11 fixed7 passed either way5 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 11 test cases the skill fixed. Same model, same prompt, with and without the skill.
In the WorkloadSpec JSON structure for AIPerf, what exact value should be specified for custom_dataset_type when using line-by-line JSONL prompts?
| Checked | Without skill | With skill |
|---|---|---|
| Sets custom_dataset_type to single_turn. | Missed | Passed |
The instructions your model reads when the skill loads.
Managed performance benchmarking with Amazon SageMaker AI inference benchmarking (the benchmark capability within optimized GenAI inference recommendations). SageMaker runs NVIDIA AIPerf on managed compute and writes AIPerf results to S3 — no hand-built load generator, no self-managed AIPerf.
This is the baseline benchmark (pre-optimization). Optimization (speculative decoding / EAGLE 3, quantization, kernel tuning) is the recommendation/optimization path (
create_ai_recommendation_job/create_optimization_job) and is out of scope for the baseline measurement.
sagemaker-deploy).create_ai_workload_config → create_ai_benchmark_job → poll describe_ai_benchmark_job.
| Field | Value |
|---|---|
| Target | standard endpoint, or endpoint plus Inference Component |
| Workload | aiperf, the bundled datasets/sharegpt-curated.jsonl (real sharegpt prompts; see note below) |
| Profile | ~500 input / ~256 output tokens, concurrency 10, 300 requests |
| Output | s3://<sagemaker-default-bucket>/benchmark-output/ (auto-detected; never hardcoded) |
| Role | the SageMaker execution role (must trust sagemaker.amazonaws.com) |
Why a bundled dataset and not public_dataset: "sharegpt"? The raw feed derives each
request's output budget from the dataset's recorded answer lengths, and a fixed handful of
turns carry budgets of only 1–3 tokens. A reasoning model cannot emit visible content within
1–3 tokens (the budget is consumed before the answer channel opens), so those requests score
invalid and trip AIPerf's ~1% validity gate — deterministically, every run, regardless of
output_tokens_mean (per-request budgets ignore it) or min_tokens (capped per-request).
datasets/sharegpt-curated.jsonl is 500 real sharegpt prompts with output budgets ≥32
tokens: same realistic traffic, but the gate measures the endpoint, not a dataset artifact.
It is the default for any model; --public-dataset restores the raw feed.
test cases passed, vs 6 / 21 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.