Clean, normalize, and prepare raw datasets for analysis or ML. Covers missing value handling, deduplication, outlier treatment, type normalization, categorical encoding, and transformation logging.
SkillScore95Maintained
Automatic
◆Data Engineer pack
Last verified 8/21/2026
What it changes
Passed 20 of 21 test cases with the skill, and 13 of 21 without it.
Without skill64%
With skill95%
Checks it passed that the model alone missed
Caps age values above 120 at 120
Saves the audit log as cleaning_audit.json
Includes rows_before key in the audit dictionary
and 8 more in the full report
7 fixed14 passed either way1 still failing0 broke
Pass-rate lift
+32 pp
Tokens
+36%
Agent turns
0%
Response time
-23%
Model gemini-3.6-flashMeasured 8/21/2026v121 of 22 cases passed with skill · 21 comparable
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
We are building an automated dataset cleaning pipeline for `financial_claims.csv`. We want to save the final audit trail containing row count changes, missingness strategy logs, and duplicate removal counts into a standard JSON file alongside the cleaned dataset. What specific filename should be used for this log?
What was checked, without and with the skill
Checked
Without skill
With skill
Saves the audit log as cleaning_audit.json
Missed
Passed
case-16 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md154 linessingle file
Dataset Cleaning
Overview
Dataset cleaning transforms raw, messy data into a consistent, analysis-ready form. Every operation must be documented: what was changed, why, and how many rows/columns were affected. This is not a one-shot script — it's a reproducible pipeline.
When to Use
Use this skill when:
Raw data has missing values, duplicates, inconsistent types, or outliers.
Data comes from multiple sources with conflicting formats or encodings.
You need a documented cleaning pipeline, not silent dropna().
The dataset will be used for modeling, analysis, or sharing.
Do not use for:
Exploratory analysis without transformation — use exploratory-data-analysis first.
Splitting strategies — use dataset-splitting.
Version tracking — use dataset-versioning.
Cleaning Workflow
1. Missing Value Strategy
Choose and document ONE strategy per column:
| Strategy | When to use | Risk |
||-------------|------|
| Drop rows | < 5% missing, rows are independent | Loss of rare cases |
| Drop column | > 40% missing and not critical | Loss of signal |
| Mean/median imputation | Continuous, symmetric distribution | Underestimates variance |
| Mode imputation | Categorical, dominant class clear | Over-represents majority |
| Constant fill | Domain-knowledge default exists | May introduce bias |
| Model-based imputation | High missingness, strong predictors | Leakage if not careful |
| Indicator flag | Missingness itself is informative | Adds dimensionality |
import pandas as pd
import numpy as np
# NEVER do this silently: