---
name: jmagly/example-quality-assess
source: https://app.decimal.ai/s/jmagly-example-quality-assess@1/SKILL.md
source_sha256: aa1bda25ee09
---

# example-quality-assess

Apply the GRADE framework (REF-060) to rate individual training examples — not just their sources. Writes `quality_grade` into each example's metadata and emits an aggregate quality report per dataset version.

## When to Use

- After ingest, before synthesis and publication
- When filtering candidates for a high-quality subset
- As a quality gate before preference pair generation
- During dataset versioning (feeds aggregate quality into dataset manifest)

## Parameters

### `<example-id | batch-glob>` (required)
Either a single example ID or a glob matching multiple examples (e.g., `examples/raw/*`).

### `--min-grade <HIGH|MODERATE|LOW>` (optional)
Only pass examples rated at or above this grade. Default: no filter.

### `--report <path>` (optional)
Write aggregate quality report to this path. Default: `.aiwg/training/reports/quality-<timestamp>.md`.

### `--non-interactive` (optional)
Skip interactive mode (useful for batch).

## Operation

1. **Load example(s)** — via kernel `memory-ingest` consumer interface.
2. **Apply source-level GRADE** — inherited from `acquire-training-source` output (already stored in `metadata.source_refs` lineage).
3. **Apply example-level scoring rules**:

### Upgrade Factors (each +1)

| Factor | Criterion |
|---|---|
| Clear reasoning trace | `output.reasoning_trace` is present and steps are coherent |
| Diverse task type for domain | This example's task type is under-represented in its domain |
| Cross-source corroboration | `source_refs` has 2+ independent sources supporting the same claim |
| Verifiable output | Output can be validated (e.g., code compiles, math correct, citation resolves) |
| Human-written | `synthetic: false` and `synthetic_depth: 0` |

### Downgrade Factors

| Factor | Criterion | Penalty |
|---|---|---|
| Hallucinated citation | `output` cites a source that doesn't resolve | −3 |
| Out-of-distribution | Example topic diverges from declared `domain` | −2 |
| Ambiguous prompt | `input.user` can be interpreted multiple ways | −1 |
| Truncated output | `output.assistant` ends mid-sentence | −1 |
| Unsafe content | Flagged by Llama Guard (REF-443) or similar | −2 |
| Synthetic depth > 1 | Recursion beyond first generation (ADR-022 D10) | −2 |

### Baseline Mapping

Source-level GRADE sets the baseline:

- HIGH source → example starts at **HIGH**
- MODERATE source → starts at **MODERATE**
- LOW source → starts at **LOW**
- VERY LOW source → starts at **VERY LOW**

Apply upgrade / downgrade factors (each adjusts by one tier). Cap at HIGH; floor at VERY LOW.

4. **Write grade** — set `metadata.quality_grade` on the example record.
5. **Apply `--min-grade` filter** — if set, flag examples below threshold for removal or review (does NOT auto-delete per `human-authorization` rule).
6. **Emit aggregate report** — `reports/quality-<timestamp>.md` with:
   - GRADE distribution (counts per tier)
   - Worst-offending examples (top 10 by downgrade score)
   - Domain-level quality breakdown
   - Synthetic vs human quality comparison
   - Recommendations (examples to remove, patterns to investigate)
7. **Log** — `memory-log-append` with op `lint` including findings distribution.

## Integration with Pipeline

- **decontamination-check (#842)**: Low-quality examples still count for overlap. Quality gate happens before decontamination, not after.
- **preference-generator (#839)**: Prefers HIGH-graded examples as "chosen"; LOW-graded as "rejected" candidates.
- **dataset-version (#844)**: Dataset manifest declares minimum grade applied to each split.

## Examples

```bash
# Assess all raw examples with a MODERATE minimum
example-quality-assess "examples/raw/*" --min-grade MODERATE

# Assess a single example
example-quality-assess ex-550e8400

# Generate report to custom path
example-quality-assess "examples/synthesized/*" --report reports/synth-quality-v1.md
```

## Schema Reference

- `@agentic/code/frameworks/sdlc-complete/schemas/research/quality-assessment.yaml` — GRADE schema (reused)
- `@agentic/code/frameworks/training-complete/schemas/example-record.yaml` — target record format (sets `metadata.quality_grade`)

## Delegation

- GRADE framework origin: `@agentic/code/frameworks/research-complete/skills/research-quality/SKILL.md`
- Kernel lint integration: `@agentic/code/addons/semantic-memory/skills/memory-lint/SKILL.md`
- Authorization on removal decisions: `@agentic/code/addons/aiwg-utils/rules/human-authorization.md`

## References

- REF-060 GRADE Handbook — methodology basis
- REF-442 Benchmark Contamination — hallucinated citation criterion
- REF-443 Llama Guard — unsafe content detection
- REF-446 Model Collapse — synthetic recursion penalty rationale