Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →brycewang-stanford/assessment-design-guide
Psychometrics and educational assessment design for researchers
Passed 22 of 23 test cases with the skill, and 18 of 23 without it.
and 2 more in the full report
4 fixed18 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
I have a CSV file containing binary response data (1 for correct, 0 for incorrect) from a recent exam along with each student's total score. Can you write a Python script using pandas to run classical item analysis on this data? I want a DataFrame output that computes difficulty, discrimination using upper and lower performing groups, point-biserial correlation, and includes a status column indicating whether the item requires review.
| Checked | Without skill | With skill |
|---|---|---|
| Cutoff quantiles of 0.73 and 0.27 are computed for upper and lower groups. | Missed | Passed |
| Items with difficulty below 0.2 are assigned a flag of review. | Passed | Passed |
| Items meeting acceptable difficulty and discrimination criteria are assigned a flag of ok. | Missed | Passed |
The instructions your model reads when the skill loads.
A skill for designing, validating, and analyzing educational assessments using modern psychometric methods. Covers classical test theory, item response theory, test construction, validity evidence, and computerized adaptive testing.
Classical test theory (CTT) models observed scores as the sum of a true score and error:
X = T + E
Key reliability coefficients:
| Coefficient | Method | Interpretation |
|---|---|---|
| Cronbach's alpha | Internal consistency | Homogeneity of items |
| Test-retest | Stability over time | Temporal consistency |
| Parallel forms | Equivalent test versions | Form equivalence |
| Split-half (Spearman-Brown) | Odd-even item split | Internal consistency |
| Inter-rater (Cohen's kappa) | Multiple raters | Scoring agreement |
import numpy as np
import pandas as pd
def item_analysis(responses: pd.DataFrame, total_scores: pd.Series) -> pd.DataFrame:
"""
Classical item analysis: difficulty, discrimination, point-biserial.
responses: binary DataFrame (1=correct, 0=incorrect), items as columns.
total_scores: total test score for each examinee.
"""
results = []
for item in responses.columns:
scores = responses[item]
difficulty = scores.mean() # p-value (proportion correct)
# Point-biserial correlation
test cases passed, vs 18 / 23 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.