Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →majiayu000/advanced-evaluation
This skill should be used when the user asks to "implement LLM-as-judge", "compare model outputs", "create evaluation rubrics", "mitigate evaluation bias", or mentions direct scoring, pairwise comparison, position bias, evaluation pipelines, or automated quality assessment.
Passed 21 of 22 test cases with the skill, and 17 of 22 without it.
and 3 more in the full report
4 fixed17 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
We need to evaluate an LLM's answers for a medical Q&A database to determine whether the factual medical claims match published clinical guidelines. We have drafted a prompt asking an LLM judge to run a pairwise comparison against a benchmark answer. Is pairwise comparison the right pattern here, or should we use direct scoring? Explain the selection rationale and outline the appropriate metric for ordinal scale evaluation.
| Checked | Without skill | With skill |
|---|---|---|
| Recommends direct scoring rather than pairwise comparison for objective factual evaluation. | Passed | Passed |
| Identifies Spearman rank correlation for ordinal metric evaluation. | Missed | Passed |
The instructions your model reads when the skill loads.
This skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.
Key insight: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.
Activate this skill when:
Evaluation approaches fall into two primary categories with distinct reliability profiles:
Direct Scoring: A single LLM rates one response on a defined scale.
Pairwise Comparison: An LLM compares two responses and selects the better one.
Research from the MT-Bench paper (Zheng et al., 2023) establishes that pairwise comparison achieves higher agreement with human judges than direct scoring for preference-based evaluation, while direct scoring remains appropriate for objective criteria with clear ground truth.
LLM judges exhibit systematic biases that must be actively mitigated:
test cases passed, vs 17 / 22 without
Published skills whose descriptions closely match this one, with their headline benchmark lift.
More skills from the same source and nearby in the registry, with their headline benchmark lift.