Compute and analyze embeddings for dataset quality, distribution comparison, semantic deduplication, diversity measurement, and similarity-based filtering. Covers sentence-transformers, embedding space diagnostics, and 2025-2026 literature techniques (NeMo Curator semantic dedup, embedding similarity metrics for data selection, DataRater-style quality scoring).
SkillScore82Maintained
Automatic
◆AI Engineer pack
Last verified 8/21/2026
What it changes
Passed 18 of 22 test cases with the skill, and 14 of 22 without it.
Without skill64%
With skill82%
Checks it passed that the model alone missed
The response uses 50 bins for histogram generation.
The solution instantiates NearestNeighbors with metric set to cosine.
The response uses 0.95 as the cosine similarity deduplication threshold.
and 5 more in the full report
6 fixed12 passed either way2 still failing2 broke
Pass-rate lift
+18 pp
Tokens
+77%
Agent turns
0%
Response time
-30%
Model gemini-3.6-flashMeasured 8/21/2026v118 of 22 cases passed with skill
Tokens, turns and time describe resource use, not a dollar price.
See it in action
One of 6 test cases the skill fixed. Same model, same prompt, with and without the skill.
1 of 6
Prompt
List the standard validation steps required to pass a quality gate for an embedding-based dataset curation workflow.
What was checked, without and with the skill
Checked
Without skill
With skill
The response includes comparing train, val, and test embedding distributions using JS divergence or Wasserstein distance.
Missed
Passed
The response includes computing diversity and redundancy metrics both before and after curation.
Missed
Passed
case-12 · gemini-3.6-flash · 8/21/2026 · answers are capped and redacted
What's inside
The instructions your model reads when the skill loads.
SKILL.md269 linessingle file
Embedding Analysis for Dataset Curation
Overview
Embeddings turn unstructured text/images into a vector space where quality, diversity, and redundancy become measurable. This skill covers embedding-based dataset curation using sentence-transformers, distribution comparison metrics, semantic deduplication, and quality scoring techniques from the 2025-2026 literature.
When to Use
Use this skill when:
Detecting near-duplicate or semantically redundant examples.
Comparing embedding distributions across dataset splits (train vs. val vs. test).
Measuring dataset diversity or coverage in embedding space.
Filtering examples based on embedding distance to a "gold" reference set.
Applying NeMo Curator-style semantic dedup at scale.
Computing DataRater-style quality signals from embedding neighborhoods.
Do not use for:
Raw text deduplication (exact match) — use simple hashing.
Model training — use transformers or trl skills.
Vector database persistence — use chromadb, milvus, or qdrant skills.
from sentence_transformers import SentenceTransformer
import numpy as np
# Choose model based on domain
model = SentenceTransformer("all-MiniLM-L6-v2") # fast, general
# model = SentenceTransformer("all-mpnet-base-v2") # higher quality
# model = SentenceTransformer("BAAI/bge-large-en-v1.5") # retrieval-optimized
# model = SentenceTransformer("intfloat/multilingual-e5-large") # multilingual