Label Quality Audit

Label Quality Audit

Overview

Label noise is the most insidious data quality problem — it's invisible until the model learns the wrong thing. Confident learning (Northcutt et al., 2021) identifies likely mislabeled examples using out-of-sample predicted probabilities.

When to Use

Use when: training data labels come from crowd workers, automated systems, or weak supervision. Do not use on expert-validated reference data unless auditing for drift.

Confident Learning Pipeline

import numpy as np
from sklearn.model_selection import cross_val_predict
from sklearn.ensemble import RandomForestClassifier

def confident_learning_audit(X, y, n_folds=5):
    """
    Returns indices of likely mislabeled examples.
    Based on Northcutt et al. "Confident Learning: Estimating
    Uncertainty in Dataset Labels" (JMLR 2021).
    """
    n_classes = len(np.unique(y))
    
    # 1. Out-of-sample predicted probabilities
    proba = cross_val_predict(
        RandomForestClassifier(n_estimators=100, random_state=42),
        X, y, cv=n_folds, method="predict_proba"
    )
    
    # 2. Compute confident joint
    # Estimated joint distribution of noisy labels × true labels
    confident_joint = np.zeros((n_classes, n_classes))
    for i in range(len(y)):
        true_class = y[i]
        pred_class = np.argmax(proba[i])
        confidence = proba[i][pred_class]
        
        # Count if predicted class has confidence above per-class threshold