Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/openai-evals
LLM evaluation framework and registry (OpenAI Evals). Framework for evaluating LLMs and LLM-based systems with a registry of community-contributed eval templates. Supports model-graded evals, classification, simple completion matching, and custom completion functions.
Passed 21 of 22 test cases with the skill, and 17 of 22 without it.
and 1 more in the full report
4 fixed17 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 4 test cases the skill fixed. Same model, same prompt, with and without the skill.
We are benchmarking commonsense reasoning on HellaSwag using OpenAI Evals. What eval type does HellaSwag use?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies HellaSwag as using match evaluation. | Missed | Passed |
The instructions your model reads when the skill loads.
|------|--------|
| mmlu | Match | Knowledge (57 subjects) |
| hellaswag | Match | Commonsense reasoning |
| truthfulqa | Model-graded | Truthfulness |
| gsm8k | Match | Math reasoning |
| humaneval | Custom | Code generation |
| ifeval | Model-graded | Instruction following |
| bbq | Model-graded | Bias detection |
| factuality | Model-graded | Factual accuracy |
| translation | Model-graded | Translation quality |
Browse full registry: evals/registry/evals/
# CI/CD eval pipeline
def run_eval_suite(model_name, eval_names):
results = {}
for eval_name in eval_names:
cmd = f"oaieval {model_name} {eval_name} --max_samples 200"
result = subprocess.run(cmd, shell=True, capture_output=True)
results[eval_name] = parse_accuracy(result.stdout)
return results
# Regression test
previous = {"mmlu": 0.86, "gsm8k": 0.92, "hellaswag": 0.85}
current = run_eval_suite("my-finetuned-model", ["mmlu", "gsm8k", "hellaswag"])
for name, score in current.items():
if score < previous[name] - 0.02: # 2% regression threshold
alert(f"Regression in {name}: {previous[name]:.2f} → {score:.2f}")
git lfs fetch --all is required before running community evalstest cases passed, vs 17 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.