Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Model Evaluation · awslabs | DecimalAI

ai mlby awslabs

Model Evaluation

awslabs/model-evaluation

Source awslabs/agent-plugins ↗License Apache-2.0

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer.

SkillScore50Maintained

Automatic

AI Engineer pack

Last verified 8/16/2026

What it changes

Passed 11 of 22 test cases with the skill, and 9 of 22 without it.

Without skill41%

With skill50%

Checks it passed that the model alone missed

Offer Custom Scorer as an alternative evaluation method.
Identify math and code as supported built-in custom scorers.
State that LLM-as-Judge is not supported for Amazon Nova models.

and 7 more in the full report

7 fixed4 passed either way6 still failing5 broke

Pass-rate lift: +9 pp
Tokens: -22%
Agent turns: 0%
Response time: -73%

Model gemini-3.6-flashMeasured 8/16/2026v111 of 22 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/awslabs-model-evaluation/SKILL.md https://app.decimal.ai/s/awslabs-model-evaluation/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add awslabs-model-evaluation, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/awslabs-model-evaluation/ and pins it in .decimal/skills.lock

decimalai skills add awslabs-model-evaluation

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/awslabs-model-evaluation/SKILL.md https://app.decimal.ai/s/awslabs-model-evaluation/SKILL.md

Raw file: /s/awslabs-model-evaluation/SKILL.md ↗ · machine-readable summary at /s/awslabs-model-evaluation.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 7 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 7

Prompt

I want to evaluate my fine-tuned Llama model. Ask me all the questions you need at once so we can generate the code immediately.

What was checked, without and with the skill
Checked	Without skill	With skill
Ask exactly one decision question in the turn rather than combining multiple questions.	Missed	Passed

case-10 · gemini-3.6-flash · 8/16/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md103 lines13 filesincludes scripts

Model Evaluation

Generate code that evaluates a SageMaker model.

Prerequisites

The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first.

Principles

One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
Confirm before proceeding. Wait for the user to agree before moving to the next step.
Don't read files until you need them. Only read reference files when you've reached the step that requires them.
Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.
No narration. Share outcomes and ask questions. Keep responses short.
No repetition. If you said something before a tool call, don't repeat it after.

Scope

This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.

Tell the user when the skill is activated:

"I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."

If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.

Evaluation Types

There are two evaluation types:

LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)
Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.

Workflow

Step 1: Determine evaluation type

Do you already know which evaluation type to use?

Track record

Benchmark

11 / 22

test cases passed, vs 9 / 22 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

awslabs / agent-plugins

plugins/sagemaker-ai/skills/model-evaluation/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1Bundled 12 files · scripts

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Model Evaluation

Prerequisites

Principles

Scope

Evaluation Types

Workflow

Step 1: Determine evaluation type

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills