Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Model Evaluation Metrics · dicklesworthstone | DecimalAI

dicklesworthstone/model-evaluation-metrics

SkillScore100Maintained

Build model evaluation metrics operations. Auto-activating skill for ML Training. Triggers on: model evaluation metrics, model evaluation metrics Part of the ML Training skill category. Use when working with model evaluation metrics functionality. Trigger with phrases like "model evaluation metrics", "model metrics", "model".

by dicklesworthstone

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/dicklesworthstone-model-evaluation-metrics/SKILL.md

decimalai skills pull dicklesworthstone-model-evaluation-metrics --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/dicklesworthstone-model-evaluation-metrics/SKILL.md https://app.decimal.ai/s/dicklesworthstone-model-evaluation-metrics/SKILL.md

Raw file: /s/dicklesworthstone-model-evaluation-metrics/SKILL.md ↗ · machine-readable summary at /s/dicklesworthstone-model-evaluation-metrics.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

22 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 22d ago

Yes — +5% lift · at higher cost+27% tokens0% turns100% of whole cases pass with skill (22/22)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-02	✗→✓	▲ Improved	7%	0%
case-01	✓→✓	＝ Same ✓	49%	0%
case-03	✓→✓	＝ Same ✓	29%	0%
case-04	✓→✓	＝ Same ✓	31%	0%
case-05	✓→✓	＝ Same ✓	26%	0%

Showing 5 of 22 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

dicklesworthstone / pi_agent_rust

tests/ext_conformance/artifacts/plugins-community/skills/07-ml-training/model-evaluation-metrics/SKILL.md

View on GitHub ↗

License MITImported 8/10/2026Version v1

Skill contents

SKILL.md

Model Evaluation Metrics

Overview

This skill provides automated assistance for model evaluation metrics tasks within the ML Training domain.

When to Use

This skill activates automatically when you:

Mention "model evaluation metrics" in your request
Ask about model evaluation metrics patterns or best practices
Need help with machine learning training skills covering data preparation, model training, hyperparameter tuning, and experiment tracking.

Instructions

Provides step-by-step guidance for model evaluation metrics
Follows industry best practices and patterns
Generates production-ready code and configurations
Validates outputs against common standards

Examples

Example: Basic Usage Request: "Help me with model evaluation metrics" Result: Provides step-by-step guidance and generates appropriate configurations

Prerequisites

Relevant development environment configured
Access to necessary tools and services
Basic understanding of ml training concepts

Output

Generated configurations and code
Best practice recommendations
Validation results

Error Handling

| Error | Cause | Solution | |-------|-------|----------| | Configuration invalid | Missing required fields | Check documentation for required parameters | | Tool not found | Dependency not installed | Install required tools per prerequisites | | Permission denied | Insufficient access | Verify credentials and permissions |

Resources

Official documentation for related tools
Best practices guides
Community examples and tutorials

Related Skills

Part of the ML Training skill category. Tags: ml, training, pytorch, tensorflow, sklearn

Illustrative Want to see what dicklesworthstone/model-evaluation-metrics would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/30/2026, 06:08 UTC · 22 cases

Lift

+5%

100% vs 95%

Agent turns

1.0 vs 1.0

Tokens

+27%

3,185 vs 2,504.18

fail → pass · 1pass → pass · 21

Per-case results (22)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 We are building a credit card fraud detection system in PyTorch where positive cases account for 0.05% of all transactions. A colleague suggests optimizing and reporting ROC-AUC score to track performance. Write a Python script using scikit-learn to compute the primary metric better suited for severe class imbalance.	pass→pass	17,767	23,110	+30%	1	1	0%	2,635	3,923	+49%	0	0	—
case-02 Our real estate price prediction model has occasional massive sales outliers (e.g., luxury mansions 100x average price) that heavily distort model training when evaluated with Mean Squared Error (MSE). Provide a scikit-learn evaluation script that calculates a robust error metric less sensitive to extreme upper-tail outliers.	fail→pass	20,573	20,638	+0%	1	1	0%	3,136	3,356	+7%	0	0	—
case-03 An e-commerce query system returns top-20 item lists scored by relevance (0 to 4). A junior dev proposes evaluating the search results using standard Precision@20. Write a Python snippet that implements position-sensitive discounted ranking evaluation.	pass→pass	16,105	20,553	+28%	1	1	0%	2,244	2,886	+29%	0	0	—
case-04 A lung cancer screening classifier produces probability outputs. Missing a positive case is ten times worse than a false alarm. The team is currently optimizing standard F1-score. Write code to calculate the correct F-beta score emphasizing recall over precision.	pass→pass	17,712	19,423	+10%	1	1	0%	2,802	3,665	+31%	0	0	—
case-05 A risk model for loan default needs accurately calibrated probabilities rather than just class boundaries. The team evaluates default decisions using classification accuracy. Write a evaluation routine in Python that measures the discrepancy between predicted probability bins and actual empirical default rates.	pass→pass	22,474	26,815	+19%	1	1	0%	3,766	4,735	+26%	0	0	—
case-06 In a 50-class document taxonomy model, 5 classes account for 80% of documents. We want an evaluation summary metric that gives equal weight to each category regardless of class frequency. Write a scikit-learn score call for this evaluation criteria.	pass→pass	11,473	12,108	+6%	1	1	0%	1,214	1,716	+41%	0	0	—
case-07 A retail inventory model forecasts daily sales per SKU, where many days have zero sales. The team attempted using Mean Absolute Percentage Error (MAPE), but encountered division-by-zero errors. Provide a Python function calculating a scale-dependent or symmetric percentage metric suitable for zero-value series.	pass→pass	23,278	23,006	-1%	1	1	0%	3,871	4,207	+9%	0	0	—
case-08 A biomedical team evaluates a UNet model for tumor segmentation. They currently report pixel-wise accuracy, which hits 98% even when the model misses small tumors entirely. Write a Python evaluation snippet using PyTorch or scikit-learn to measure spatial overlap between prediction mask and ground truth mask.	pass→pass	16,882	22,173	+31%	1	1	0%	3,483	3,916	+12%	0	0	—
case-09 We ran k-means clustering on customer embedding vectors without ground truth labels. Someone suggested using Adjusted Rand Index (ARI) to measure cluster quality. Write Python code demonstrating the correct evaluation metric when no reference labels exist.	pass→pass	17,085	18,522	+8%	1	1	0%	2,472	2,978	+20%	0	0	—
case-10 An automated translation system converts French news articles to English. We need an automated n-gram precision metric with brevity penalty to evaluate output quality against candidate references. Write a Python snippet using NLTK or evaluate library to compute this standard translation score.	pass→pass	9,171	16,699	+82%	1	1	0%	1,852	2,780	+50%	0	0	—
case-11 An abstractive text summarizer generates short bullet points from long legal documents. We want to evaluate sentence overlap focusing on recall of reference summary n-grams and longest common subsequences. Write Python code implementing the standard metric for summarization task benchmarking.	pass→pass	21,106	17,506	-17%	1	1	0%	3,388	4,086	+21%	0	0	—
case-12 A causal language model generates text predictions. We have test sequence cross-entropy loss values from PyTorch evaluation loops. Write Python code converting cross-entropy loss into the standard exponential metric used for language modeling fluency.	pass→pass	10,281	20,099	+95%	1	1	0%	2,007	3,347	+67%	0	0	—
case-13 A question-answering system retrieves candidate passages. Users only care about how high up the single correct answer passage appears in the ranked list. Write a Python function calculating the mean evaluation metric across multiple search queries for this single-hit scenario.	pass→pass	9,953	18,532	+86%	1	1	0%	2,078	3,094	+49%	0	0	—
case-14 An autonomous driving model predicts bounding boxes for vehicles. Evaluating with raw classification accuracy ignores spatial localization errors. Write Python code or specify the protocol using standard COCO metrics for bounding box evaluation across IoU thresholds.	pass→pass	16,383	16,429	+0%	1	1	0%	3,266	3,573	+9%	0	0	—
case-15 A binary disease diagnostic model operates on a dataset with 85% negative and 15% positive labels. The team wants a metric that yields 0 for random guessing and +1 for perfect prediction across all four confusion matrix quadrants. Write code calculating this metric.	pass→pass	18,342	13,846	-25%	1	1	0%	2,768	3,172	+15%	0	0	—
case-16 A multiple linear regression model uses 50 candidate features on 100 sample rows. Standard R-squared increases every time a feature is added regardless of feature relevance. Write Python code calculating the metric that penalizes additional non-informative predictor variables.	pass→pass	17,846	11,465	-36%	1	1	0%	2,824	2,749	-3%	0	0	—
case-17 A top-N movie recommendation engine outputs a list of 10 items per user. We want a simple binary evaluation metric measuring whether the user's actual interacted item appears anywhere within the top-10 recommended list. Write a Python function for this recommendation evaluation.	pass→pass	16,296	13,954	-14%	1	1	0%	2,344	3,340	+42%	0	0	—
case-18 A multi-label news tagging model predicts overlapping categories (e.g., Tech, Politics, Economy). Frequent tags dominate overall count, and we want a global F1 score computed by aggregating total true positives, false positives, and false negatives across all classes combined. Write the scikit-learn call.	pass→pass	3,743	5,291	+41%	1	1	0%	708	1,353	+91%	0	0	—
case-19 Write a pure Python implementation of Discounted Cumulative Gain (DCG@K) for a list of gain scores [3, 2, 0, 1, 2]. Ensure the position discount uses the standard logarithmic decay formula.	pass→pass	16,786	15,890	-5%	1	1	0%	2,598	4,047	+56%	0	0	—
case-20 Configure an Optuna hyperparameter study in Python using TPE sampler to tune learning rate between 1e-5 and 1e-2 and batch size among [16, 32, 64] for 50 trials.	pass→pass	11,045	9,255	-16%	1	1	0%	1,234	2,269	+84%	0	0	—
case-21 Write a scikit-learn Pipeline with SimpleImputer and StandardScaler to fit and transform tabular numeric features prior to model input.	pass→pass	8,586	14,061	+64%	1	1	0%	1,762	2,223	+26%	0	0	—
case-22 Write PyTorch code to initialize a DistributedDataParallel (DDP) process group using the nccl backend for multi-GPU training on single node.	pass→pass	17,508	19,100	+9%	1	1	0%	2,640	2,655	+1%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 22 cases were attempted. The headline lift of +5 percentage points is the difference between those two pass rates over the 22 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 95.5%
Pass rate with the skill: 100%
Cases attempted: 22
Cases counted toward lift: 22
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/30/2026, 06:08 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-02

Prompt

Our real estate price prediction model has occasional massive sales outliers (e.g., luxury mansions 100x average price) that heavily distort model training when evaluated with Mean Squared Error (MSE). Provide a scikit-learn evaluation script that calculates a robust error metric less sensitive to extreme upper-tail outliers.

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run100

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+28% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 22 days ago

Other measured skills in the registry, with their headline benchmark lift.