Skill Eval

Skill Eval

Answer one named maintenance decision: retain, revise, remove, or insufficient evidence. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference.

This is an optional specialist. The selected runner owns execution and bounds; native results own measurements; BD and Git retain their authority. Do not add a core skill, AO evaluation command, scheduler, dashboard, second tracker, or mandatory review merely to run an experiment.

Rules that decide the answer

• Count every attempt. Keep failed, crashed, interrupted, blocked, abandoned, missing and infrastructure-invalid attempts in the all-attempt accounting. A rerun adds an attempt; it never overwrites the one that failed.
• Vary one thing. Equalize instructions, tools, environment, model and effort across arms apart from the intended variable. If one arm's task prompt repeats the skill's direction, attribute the result to the combined instructions, not the skill alone.
• Confirm the skill loaded. Before reading a zero or small delta as no benefit, check each treatment run for the skill actually being loaded or injected. A run where it never loaded measures routing, not content.
• Calibrate the grader. Before trusting scores, confirm the judge or discriminator passes a response that plainly meets each criterion and fails one that plainly does not. A weak judge can fail correct responses wholesale.
• Small samples are directional. Report uncertainty with every difference. A difference without it shows neither benefit nor equivalence, and a zero-crossing interval is not equivalence.
• Fix the stop before running. Do not add trials until the result turns positive, remove losing observations or relax acceptance.

Choose the question