Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →jaechang-hits/nan-safe-correlation
Per-feature NaN-safe Spearman/Pearson correlation across many features (genes, proteins, variants) with missing values. Covers why bulk matrix shortcuts fail, correct pairwise deletion, degenerate input filtering, and large-dataset performance. Use statistical-analysis for test choice; shap-model-explainability for interpretability.
Passed 19 of 22 test cases with the skill, and 10 of 22 without it.
and 11 more in the full report
10 fixed10 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.
Write a Python snippet demonstrating the correct boolean mask logic for extracting non-missing sample pairs from two 1D numpy arrays `x` and `y` representing feature observations.
| Checked | Without skill | With skill |
|---|---|---|
| Constructs pairwise non-NaN boolean mask using ~(np.isnan(x) | np.isnan(y)). | Missed | Passed |
The instructions your model reads when the skill loads.
Computing correlations across many features (genes, proteins, variants) when missing values are present is error-prone. The most common mistake is using bulk matrix shortcuts that silently mishandle NaN, producing incorrect correlation values. This guide covers correct per-feature pairwise computation, degenerate input filtering, and performance optimization.
Different features have different missing value patterns across samples. Bulk methods handle this inconsistently:
| Method | Problem |
|---|---|
DataFrame.rank() then corrwith() | rank() assigns NaN ranks; corrwith() may drop globally or per-column inconsistently |
DataFrame.corrwith(method='spearman') | Implementation varies by pandas version; may use listwise deletion |
np.corrcoef on ranked data | Propagates NaN to entire result if any value is missing |
Features that produce undefined or unstable correlations:
| Type | Description | Effect |
|---|---|---|
| Constant features | All values identical (variance = 0) | Correlation undefined (division by zero) |
| Near-constant features | Very low variance | Correlation numerically unstable |
| Too few valid values | After NaN removal, fewer than min_valid pairs | Statistically unreliable |
| Single-value after filtering | Only one unique value remains post-NaN removal | Correlation undefined |
test cases passed, vs 10 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.