LLM-Assisted Dataset Curation

LLM-Assisted Dataset Curation

Overview

Modern dataset curation uses LLMs as quality filters, rewriters, labelers, and synthetic data generators. This skill covers hosting models locally with vLLM/SGLang and using them for dataset work — not for interactive chat, but for batch, structured, reproducible data operations.

When to Use

Use this skill when:

• Scoring or filtering dataset examples with an LLM quality judge.
• Rewriting noisy text (queries, answers, reasoning traces) in bulk.
• Generating synthetic examples to balance classes or fill gaps.
• Extracting structured labels from unstructured text.
• Running curriculum scoring (difficulty, complexity, educational value).
• Implementing DataRater-style learned quality scoring (2025).

Do not use for:

• Interactive chat or single-example inspection — use a UI.
• Exact-match deduplication — use hashing.
• Embedding-based dedup — use embedding-analysis skill.

Prerequisites

Requires a running vLLM or SGLang server. See vllm and sglang skills for server setup.

# vLLM (high throughput)
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --max-model-len 8192

# SGLang (structured output)
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --port 30000

Core Patterns

1. LLM-as-Judge Quality Scoring

Score each example on clarity, correctness, and usefulness.