Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →nvidia/nemo-evaluator-plugin
Use when working on the Evaluator plugin CLI, jobs, SDK-backed specs, metric types, or plugin-owned Evaluator skills.
Passed 19 of 20 test cases with the skill, and 5 of 20 without it.
and 19 more in the full report
15 fixed6 passed either way1 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
One of 14 test cases the skill fixed. Same model, same prompt, with and without the skill.
Which bundled specification file in the skill assets directory provides an online generation and LLM judge evaluation spec configured for local testing using `NVIDIA_API_KEY`?
| Checked | Without skill | With skill |
|---|---|---|
| Identifies assets/specs/llm_as_judge.json as the bundled spec asset for LLM judge evaluation. | Missed | Passed |
The instructions your model reads when the skill loads.
The Plugin CLI entrypoint is uv run nemo evaluator.
Use this skill to choose an evaluation interface and metric, validate a minimal example, submit a NeMo Platform evaluation job, and retrieve its results.
Establish these inputs before building an evaluation:
Read Metric Selection before choosing a metric for a rubric, RAG workflow, or tool-calling evaluation.
| Need | Interface |
|---|---|
| Fast metric iteration without NeMo Platform | nemo_evaluator_sdk.Evaluator |
| Dataset-driven platform job | client.evaluator.submit(...) or nemo evaluator evaluate submit |
| Multiple inline/stored metric refs in one job | nemo evaluator evaluate submit with an EvaluateInputSpec |
| Task-driven platform job | nemo evaluator agent-evaluate submit |
test cases passed, vs 5 / 20 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.