Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

eval-skills · leoyeai | DecimalAI

Agent Toolingby leoyeai

eval-skills

AI Agent Skill unit testing framework. A framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills.

Never made a case worse

SkillScore

SkillScore95/100

Test results 95

Safety

3/3 safety checks passed

Code scan
Intent
Content

What it changes

18%95%of test cases passed +77 pts

Without skill4 of 22

With skill21 of 22

17 fixed4 passed either way1 still failing0 broke

Measured on gemini-3.6-flash8/8/2026v1

Track record

Agent runs: 0; All time

See one task, with and without the skill

One test taskI have an existing evaluation report stored in `./reports/eval-result.json`. What command converts this JSON report directly into an HTML formatted report at `./reports/eval-result.html`?

Without the skillMissed

With the skillPassed

Same model, same task. Answers are capped and redacted.

Use this skill

Your agent runs on

Adds every file the skill has, pinned so you can update it. No account needed.

pip install "decimalai>=0.15"
decimalai skills add leoyeai-eval-skills
claude --plugin-dir "$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.agents"

The same install from Python, for a notebook or a setup script.

Install

pip install "decimalai>=0.15"

Add the skill

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

print("Added. Start Claude Code with the command below.")

Start Claude Code with it

claude --plugin-dir "$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.agents"

Type these in a Claude Code session. No account or API key. It writes the skill and any files it bundles, and asks first when its safety rating isn't Passed.

Add the DecimalAI plugin, once

/plugin marketplace add decimal-labs/decimalai-claude-plugins
/plugin install decimalai@decimalai

Install this skill

/decimalai:install leoyeai-eval-skills

The plugin needs python3 and curl on your machine.

No Python or account. Claude Code picks it up in its next session.

Save it into your project

curl -fsSL --create-dirs -o .claude/skills/leoyeai-eval-skills/SKILL.md https://app.decimal.ai/s/leoyeai-eval-skills/SKILL.md

Copy SKILL.md

Save it in your project, as

.claude/skills/leoyeai-eval-skills/SKILL.md

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Claude Code docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[claude-agent-sdk]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

import asyncio

from claude_agent_sdk import ClaudeAgentOptions
from decimalai.claude_agent_sdk import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

async def main():
    question = "Hello!"
    async with support.run(question) as run:  # one user message, one run
        async for message in run.query(options=ClaudeAgentOptions(max_turns=6)):
            print(message)

# In a notebook or async code: await main()
asyncio.run(main())

Prefer its own loader? ClaudeAgentOptions(plugins=[skills_plugin()], skills="all") reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

The same install from Python, for a notebook or a setup script. agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[claude-agent-sdk]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import asyncio

from claude_agent_sdk import ClaudeAgentOptions
from decimalai.claude_agent_sdk import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

async def main():
    question = "Hello!"
    async with support.run(question) as run:  # one user message, one run
        async for message in run.query(options=ClaudeAgentOptions(max_turns=6)):
            print(message)

# In a notebook or async code: await main()
asyncio.run(main())

Prefer its own loader? ClaudeAgentOptions(plugins=[skills_plugin()], skills="all") reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

import asyncio
from pathlib import Path
from claude_agent_sdk import ClaudeAgentOptions, query

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
options = ClaudeAgentOptions(system_prompt=f"You are a helpful agent.\n\n{skill}", max_turns=6)

async def main():
    async for message in query(prompt="Hello!", options=options):
        print(message)

# In a notebook or async code: await main()
asyncio.run(main())

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Claude Agent SDK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[openai-agents]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

from agents import Agent, Runner
from decimalai.openai_agents import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    name=support.name,
    instructions=support.instructions,  # your prompt, then the skill list
    tools=[*support.tools],             # your tools, then load_skill
)
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

Using SandboxAgent? capabilities=[Shell(), decimalai.openai_agents.sandbox_skills()] points its own loader at the same folder.

Install

pip install "decimalai[openai-agents]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from agents import Agent, Runner
from decimalai.openai_agents import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    name=support.name,
    instructions=support.instructions,  # your prompt, then the skill list
    tools=[*support.tools],             # your tools, then load_skill
)
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

Using SandboxAgent? capabilities=[Shell(), decimalai.openai_agents.sandbox_skills()] points its own loader at the same folder.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from agents import Agent, Runner

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = Agent(name="support", instructions=f"You are a helpful agent.\n\n{skill}")
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

OpenAI Agents docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[openai]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

import json

from openai import OpenAI
from decimalai.openai import agent_config

client = OpenAI()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your tools, as Chat Completions takes them, and what answers each.
TOOLS = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    # run.system is your prompt, then the skill list.
    messages = [{"role": "system", "content": run.system}, run.message]
    while True:
        reply = client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=messages,
            tools=[*TOOLS, *run.tools],  # your tools, then load_skill
        ).choices[0].message
        messages.append(reply)
        if not reply.tool_calls:
            break
        for call in reply.tool_calls:
            # load_skill is answered by the run; your own tools by FUNCTIONS.
            function = FUNCTIONS.get(call.function.name)
            messages.append(run.tool_result(call) or {
                "role": "tool", "tool_call_id": call.id,
                "content": function(**json.loads(call.function.arguments)) if function
                else f"There is no tool named {call.function.name!r}.",
            })
print(reply.content)

# The Responses API takes run.response_tools instead of run.tools.

Install

pip install "decimalai[openai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import json

from openai import OpenAI
from decimalai.openai import agent_config

client = OpenAI()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your tools, as Chat Completions takes them, and what answers each.
TOOLS = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    # run.system is your prompt, then the skill list.
    messages = [{"role": "system", "content": run.system}, run.message]
    while True:
        reply = client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=messages,
            tools=[*TOOLS, *run.tools],  # your tools, then load_skill
        ).choices[0].message
        messages.append(reply)
        if not reply.tool_calls:
            break
        for call in reply.tool_calls:
            # load_skill is answered by the run; your own tools by FUNCTIONS.
            function = FUNCTIONS.get(call.function.name)
            messages.append(run.tool_result(call) or {
                "role": "tool", "tool_call_id": call.id,
                "content": function(**json.loads(call.function.arguments)) if function
                else f"There is no tool named {call.function.name!r}.",
            })
print(reply.content)

# The Responses API takes run.response_tools instead of run.tools.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from openai import OpenAI

client = OpenAI()
skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
reply = client.chat.completions.create(
    model="gpt-5.4-mini",
    messages=[
        {"role": "system", "content": f"You are a helpful agent.\n\n{skill}"},
        {"role": "user", "content": "Hello!"},
    ],
).choices[0].message

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

OpenAI SDK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[anthropic]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

import anthropic
from decimalai.anthropic import agent_config

client = anthropic.Anthropic()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

question = "Hello!"
with support.run(question) as run:  # one user message, one run
    answer = client.beta.messages.tool_runner(
        model="claude-haiku-5-5",
        max_tokens=4096,
        system=run.system,     # your prompt, then the skill list
        tools=[*run.tools],    # your tools, then load_skill
        messages=[{"role": "user", "content": question}],
    ).until_done()

Install

pip install "decimalai[anthropic]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import anthropic
from decimalai.anthropic import agent_config

client = anthropic.Anthropic()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

question = "Hello!"
with support.run(question) as run:  # one user message, one run
    answer = client.beta.messages.tool_runner(
        model="claude-haiku-5-5",
        max_tokens=4096,
        system=run.system,     # your prompt, then the skill list
        tools=[*run.tools],    # your tools, then load_skill
        messages=[{"role": "user", "content": question}],
    ).until_done()

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
import anthropic

client = anthropic.Anthropic()
skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
reply = client.messages.create(
    model="claude-haiku-5-5",
    max_tokens=4096,
    system=f"You are a helpful agent.\n\n{skill}",
    messages=[{"role": "user", "content": "Hello!"}],
)

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Anthropic docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[google-genai]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

from google import genai
from google.genai import types
from decimalai.google_genai import agent_config

client = genai.Client()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your functions, as declarations for the model, and what answers each.
TOOLS = [{"function_declarations": [{
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters_json_schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                               "required": ["order_id"]},
}]}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    config = types.GenerateContentConfig(
        system_instruction=run.system,     # your prompt, then the skill list
        tools=[*TOOLS, *run.tool_params],  # your functions, then load_skill
    )
    contents = [{"role": "user", "parts": [{"text": run.message}]}]
    while True:
        response = client.models.generate_content(
            model="gemini-3.5-flash", contents=contents, config=config,
        )
        if not response.function_calls:
            break
        contents.append(response.candidates[0].content)
        parts = []
        for call in response.function_calls:
            # load_skill is answered by the run; your own functions by FUNCTIONS.
            # A loop of your own rather than automatic function calling, which
            # raises KeyError when the model asks for a function you did not pass.
            function = FUNCTIONS.get(call.name)
            parts.append(run.tool_result(call) or {"function_response": {
                "name": call.name, "id": call.id,
                "response": {"result": function(**(call.args or {})) if function
                             else f"There is no function named {call.name!r}."},
            }})
        contents.append({"role": "user", "parts": parts})
print(response.text)

Install

pip install "decimalai[google-genai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from google import genai
from google.genai import types
from decimalai.google_genai import agent_config

client = genai.Client()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your functions, as declarations for the model, and what answers each.
TOOLS = [{"function_declarations": [{
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters_json_schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                               "required": ["order_id"]},
}]}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    config = types.GenerateContentConfig(
        system_instruction=run.system,     # your prompt, then the skill list
        tools=[*TOOLS, *run.tool_params],  # your functions, then load_skill
    )
    contents = [{"role": "user", "parts": [{"text": run.message}]}]
    while True:
        response = client.models.generate_content(
            model="gemini-3.5-flash", contents=contents, config=config,
        )
        if not response.function_calls:
            break
        contents.append(response.candidates[0].content)
        parts = []
        for call in response.function_calls:
            # load_skill is answered by the run; your own functions by FUNCTIONS.
            # A loop of your own rather than automatic function calling, which
            # raises KeyError when the model asks for a function you did not pass.
            function = FUNCTIONS.get(call.name)
            parts.append(run.tool_result(call) or {"function_response": {
                "name": call.name, "id": call.id,
                "response": {"result": function(**(call.args or {})) if function
                             else f"There is no function named {call.name!r}."},
            }})
        contents.append({"role": "user", "parts": parts})
print(response.text)

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from google import genai
from google.genai import types

client = genai.Client()
skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents="Hello!",
    config=types.GenerateContentConfig(system_instruction=f"You are a helpful agent.\n\n{skill}"),
)

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Gemini API docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[langchain]" langchain-openai
decimalai skills add leoyeai-eval-skills

Add it to your agent

from langchain.agents import create_agent
from decimalai.langchain import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = create_agent(
    "openai:gpt-5.4-mini",
    tools=[],                         # your tools; the middleware adds load_skill
    middleware=[support.middleware],  # your prompt, the skill list and load_skill
)
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

Install

pip install "decimalai[langchain]" langchain-openai

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from langchain.agents import create_agent
from decimalai.langchain import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = create_agent(
    "openai:gpt-5.4-mini",
    tools=[],                         # your tools; the middleware adds load_skill
    middleware=[support.middleware],  # your prompt, the skill list and load_skill
)
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from langchain.agents import create_agent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = create_agent("openai:gpt-5.4-mini", tools=[], system_prompt=f"You are a helpful agent.\n\n{skill}")
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

LangChain docs

CrewAI's own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" "crewai>=1.15.27"
decimalai skills add leoyeai-eval-skills

Add it to your agent

import decimalai.crewai
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                               # your own CrewAI LLM
    skills=decimalai.crewai.skills(),      # the installed folder
    tools=decimalai.crewai.skill_tools(),  # reads a skill's files, runs its scripts
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

The same install from Python, for a notebook or a setup script. CrewAI's own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" "crewai>=1.15.27"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import decimalai.crewai
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                               # your own CrewAI LLM
    skills=decimalai.crewai.skills(),      # the installed folder
    tools=decimalai.crewai.skill_tools(),  # reads a skill's files, runs its scripts
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

No DecimalAI package. CrewAI's own loader reads the folder, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                         # your own CrewAI LLM
    skills=[Path(".agents/skills")],
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

CrewAI docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[adk]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

from decimalai.adk import DecimalAIPlugin, agent_config
from google.adk.agents import LlmAgent
from google.adk.apps import App

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = LlmAgent(
    name="support",          # ADK needs a Python identifier
    model="gemini-3.5-flash",
    instruction=support.instruction,  # your prompt, then the skill list
    tools=[*support.tools],           # your tools, then load_skill
)
# DecimalAIPlugin makes each invocation of runners built from this App one trace.
app = App(name="support", root_agent=agent, plugins=[DecimalAIPlugin()])

Prefer ADK's own loader? SkillToolset(skills=decimalai.adk.skills(), ...) reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

Install

pip install "decimalai[adk]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from decimalai.adk import DecimalAIPlugin, agent_config
from google.adk.agents import LlmAgent
from google.adk.apps import App

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = LlmAgent(
    name="support",          # ADK needs a Python identifier
    model="gemini-3.5-flash",
    instruction=support.instruction,  # your prompt, then the skill list
    tools=[*support.tools],           # your tools, then load_skill
)
# DecimalAIPlugin makes each invocation of runners built from this App one trace.
app = App(name="support", root_agent=agent, plugins=[DecimalAIPlugin()])

Prefer ADK's own loader? SkillToolset(skills=decimalai.adk.skills(), ...) reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from google.adk.agents import LlmAgent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = LlmAgent(
    name="support",
    model="gemini-3.5-flash",
    instruction=lambda context: f"You are a helpful agent.\n\n{skill}",
)

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Google ADK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[pydantic-ai]"
decimalai skills add leoyeai-eval-skills

Add it to your agent

from pydantic_ai import Agent
from decimalai.pydantic_ai import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    "openai:gpt-4o-mini",
    name=support.name,
    tools=[],                 # your tools; the capability adds load_skill
    capabilities=[support],   # your prompt, the skill list and load_skill
)
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

On the Pydantic AI harness, decimalai.pydantic_ai_harness.skills(project) uses its own loader instead.

Install

pip install "decimalai[pydantic-ai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from pydantic_ai import Agent
from decimalai.pydantic_ai import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    "openai:gpt-4o-mini",
    name=support.name,
    tools=[],                 # your tools; the capability adds load_skill
    capabilities=[support],   # your prompt, the skill list and load_skill
)
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

On the Pydantic AI harness, decimalai.pydantic_ai_harness.skills(project) uses its own loader instead.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from pydantic_ai import Agent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = Agent("openai:gpt-4o-mini", instructions=f"You are a helpful agent.\n\n{skill}")
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Pydantic AI docs

Deep Agents' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" deepagents
decimalai skills add leoyeai-eval-skills

Add it to your agent

import decimalai.deepagents
from deepagents import create_deep_agent

agent = create_deep_agent(model=model, **decimalai.deepagents.skills())
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

The default backend reads files but has no shell. To run a skill's scripts, pass skills(backend=LocalShellBackend(...)).

The same install from Python, for a notebook or a setup script. Deep Agents' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" deepagents

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import decimalai.deepagents
from deepagents import create_deep_agent

agent = create_deep_agent(model=model, **decimalai.deepagents.skills())
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

The default backend reads files but has no shell. To run a skill's scripts, pass skills(backend=LocalShellBackend(...)).

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from deepagents import create_deep_agent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = create_deep_agent(model=model, system_prompt=f"You are a helpful agent.\n\n{skill}")
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Deep Agents docs

Its own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" agent-framework
decimalai skills add leoyeai-eval-skills

Add it to your agent

import asyncio

from agent_framework import Agent
from decimalai.agent_framework import skills

agent = Agent(client=client, instructions="You are a helpful agent.", context_providers=[skills()])
asyncio.run(agent.run("Hello!"))

It caches skills by default, so restart the process after decimalai skills update.

The same install from Python, for a notebook or a setup script. Its own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" agent-framework

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

import asyncio

from agent_framework import Agent
from decimalai.agent_framework import skills

agent = Agent(client=client, instructions="You are a helpful agent.", context_providers=[skills()])
asyncio.run(agent.run("Hello!"))

It caches skills by default, so restart the process after decimalai skills update.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

import asyncio
from pathlib import Path
from agent_framework import Agent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = Agent(client=client, instructions=f"You are a helpful agent.\n\n{skill}")
asyncio.run(agent.run("Hello!"))

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Microsoft Agent Framework docs

Strands' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" strands-agents
decimalai skills add leoyeai-eval-skills

Add it to your agent

from decimalai.strands import skills
from strands import Agent
from strands.vended_tools import file_editor, shell

agent = Agent(
    model=model,                # always pass one: the default is Amazon Bedrock
    system_prompt="You are a helpful agent.",
    tools=[file_editor, shell],
    plugins=[skills()],
)
agent("Hello!")

Strands loads an agent's skills once, so build the agent again after decimalai skills update.

The same install from Python, for a notebook or a setup script. Strands' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" strands-agents

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("leoyeai-eval-skills")
if result.exit_code:
    raise SystemExit(result.problems)

from decimalai.strands import skills
from strands import Agent
from strands.vended_tools import file_editor, shell

agent = Agent(
    model=model,                # always pass one: the default is Amazon Bedrock
    system_prompt="You are a helpful agent.",
    tools=[file_editor, shell],
    plugins=[skills()],
)
agent("Hello!")

Strands loads an agent's skills once, so build the agent again after decimalai skills update.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

Give it to your agent

from pathlib import Path
from strands import Agent

skill = Path(".agents/skills/leoyeai-eval-skills/SKILL.md").read_text()
agent = Agent(model=model, system_prompt=f"You are a helpful agent.\n\n{skill}")
agent("Hello!")

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Strands docs

Adds every file the skill has, pinned so you can update it. Cursor, Copilot, Cline and Warp read this folder by themselves.

pip install "decimalai>=0.15"
decimalai skills add leoyeai-eval-skills

No Python or account. Cursor, Copilot, Cline and Warp read this folder by themselves.

Save it into your project

curl -fsSL --create-dirs -o .agents/skills/leoyeai-eval-skills/SKILL.md https://app.decimal.ai/s/leoyeai-eval-skills/SKILL.md

Copy SKILL.md

Save it in your project, as

.agents/skills/leoyeai-eval-skills/SKILL.md

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Cursor and other editors docs

Fetch the file and give its text to your model as instructions. No account or package needed.

Get the file

curl -fsSL https://app.decimal.ai/s/leoyeai-eval-skills/SKILL.md

Read the summary as JSON

curl -fsSL https://app.decimal.ai/s/leoyeai-eval-skills.json

This gets SKILL.md only. The skill's 3 other files are not included.

Other ways to add it

Other setups docs

You don't need a DecimalAI account.

Version: v1, tested 8/8/2026
Source: LeoYeAI/openclaw-master-skills ↗
License: MIT

Manage it in DecimalAI

Saved prompts and skills, pinned versions, and traces of this skill in your runs. DecimalAI does not run your agent.

Claude Code reads files from your project, so DecimalAI config does not apply there.

On Claude Agent SDK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On OpenAI Agents: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On OpenAI SDK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Anthropic: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Gemini API: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On LangChain: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On CrewAI: set DECIMAL_API_KEY and add decimalai.init(crewai=True) to record runs and skill use. The skill version stays in your repo.

On Google ADK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Pydantic AI: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Deep Agents: set DECIMAL_API_KEY and add decimalai.init(langchain=True) to record runs and skill use. The skill version stays in your repo.

On Microsoft Agent Framework: set DECIMAL_API_KEY and add decimalai.init(otel=True, agent_name="support") to record runs and skill use. The skill version stays in your repo.

On Strands: set DECIMAL_API_KEY and add decimalai.init(otel=True, agent_name="support") to record runs and skill use. The skill version stays in your repo.

Editors read files from your project, so DecimalAI config does not apply there.

Anywhere else: record a use with decimalai.log_skill_activation() inside a trace.

Use this skill

SKILL.md

eval-skills

AI Agent Skill unit testing framework — a framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills.

This skill fills the L1 (Skill Unit Test) gap that LangSmith / DeepEval leave open: while those platforms focus on agent-level and trajectory-level evaluation (L2-L3), eval-skills targets the individual skill level, ensuring each building block meets quality standards before it ever enters an agent pipeline.

When to Use This Skill

Before deploying a new skill to production — run eval to verify it meets your quality gate.
When choosing between multiple candidate skills — run select to rank them on the same benchmark.
When a skill is upgraded — run report diff to detect regressions.
In CI/CD — use --exit-on-fail to block merges that degrade skill quality.
When bootstrapping a new skill — run create to generate a ready-to-fill skeleton.

Capabilities

1. Find Skills

Search for existing skills by keyword, tag, or adapter type.

eval-skills find \
  --query "web search" \
  --tag retrieval api \
  --adapter http \
  --min-completion 0.8 \
  --skills-dir ./skills \
  --limit 10

Option	Description	Default
`-q, --query <string>`	Keyword search (matches name, description, tags)	—
`-t, --tag <tags...>`	Filter by tags (intersection: skill must have ALL specified tags)	—
`-a, --adapter <type>`	Filter by adapter type (`http`, `subprocess`, `mcp`)	—
`--min-completion <rate>`	Minimum historical completion rate (0.0 ~ 1.0)	—
`--skills-dir <dir>`	Directory to scan for `skill.json` files	`./skills`
`--limit <n>`	Maximum number of results	`20`

Results are ranked by search relevance (when --query is provided) or by historical completion rate (descending).

2. Create Skills

Generate a skill skeleton from a template to bootstrap development.

eval-skills create \
  --name my_api_skill \
  --from-template http_request \
  --output-dir ./skills \
  --description "Fetches weather data from OpenWeather API"

Option	Description	Default
`--name <name>`	Required. Skill name	—
`--from-template <tpl>`	Template type: `http_request`, `python_script`, `mcp_tool`	`http_request`
`--output-dir <dir>`	Output directory	`./skills`
`--description <text>`	Human-readable description embedded in `skill.json`	—

Generated file structure:

skills/my_api_skill/
  skill.json            # Skill metadata (id, schemas, adapter config)
  adapter.config.json   # Adapter-specific configuration
  tests/
    basic.eval.json     # A starter benchmark with one sample task
  skill.py              # (python_script template only) JSON-RPC entrypoint

3. Evaluate Skills

Run benchmark evaluations against one or more skills. This is the core command.

eval-skills eval \
  --skills ./skills/calculator/skill.json ./skills/search/ \
  --benchmark coding-easy \
  --concurrency 4 \
  --timeout 30000 \
  --retries 2 \
  --runs 3 \
  --evaluator exact \
  --format json markdown html \
  --output-dir ./reports \
  --exit-on-fail --min-completion 0.8 \
  --store ./eval-skills.db

Option	Description	Default
`--skills <paths...>`	Required. Skill file(s) or directory(ies)	—
`--benchmark <id\|path>`	Built-in benchmark ID or path to `benchmark.json`	`coding-easy`
`--tasks <file>`	Custom tasks JSON file (replaces benchmark)	—
`--concurrency <n>`	Number of parallel task executions	`4`
`--timeout <ms>`	Per-task timeout in milliseconds	`30000`
`--retries <n>`	Retry count on task failure (with incremental backoff)	`0`
`--runs <n>`	Repeat evaluation N times for consistency scoring	`1`
`--evaluator <type>`	Default scorer type (see Scorer Types below)	`exact`
`--format <formats...>`	Output formats: `json`, `markdown`, `html`	`json markdown`
`--output-dir <dir>`	Report output directory	`./reports`
`--exit-on-fail`	Exit with code 1 if any skill falls below threshold	disabled
`--min-completion <rate>`	Threshold for `--exit-on-fail`	`0.7`
`--dry-run`	Validate configuration only; do not execute tasks	disabled
`--benchmarks-dir <dir>`	Directory containing built-in benchmarks	`./benchmarks`
`--store <path>`	SQLite database path for persistent result storage	`./eval-skills.db`
`-c, --config <path>`	Path to `eval-skills.config.yaml`	auto-detected

Evaluation flow:

Load skills from --skills paths (supports both single skill.json and directories)
Load benchmark tasks from --benchmark or --tasks
Build the cartesian product: skills x tasks x runs
Execute all task items concurrently (controlled by --concurrency, with timeout and retry)
Score each result using the appropriate scorer
Aggregate into SkillCompletionReport per skill
Write reports to --output-dir

4. Select Skills

Filter and rank skills based on evaluation reports using a multi-dimensional strategy.

eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result.json \
  --strategy ./strategy.yaml \
  --min-completion 0.8 \
  --top-k 5 \
  --output ./selected.json

Option	Description	Default
`--from <path>`	Required. Candidate skills directory or JSON file	—
`--reports <file>`	Evaluation reports JSON file	—
`--strategy <file>`	SelectStrategy YAML/JSON file	built-in default
`--min-completion <rate>`	Override minimum completion rate filter	—
`--top-k <n>`	Return only the top K results	all
`--output <file>`	Write selected skills to file	stdout

Selection pipeline: Filter (by completion rate, error rate, latency, adapter type, required tags) -> Score -> Rank (by compositeScore, completionRate, latency, or tokenCost) -> TopK

Example strategy.yaml:

filters:
  minCompletionRate: 0.8
  maxErrorRate: 0.1
  maxLatencyP95Ms: 5000
  adapterTypes: [http, subprocess]
  requiredTags: [production-ready]
sortBy: compositeScore
order: desc
topK: 5

5. Run Pipeline

Execute the full end-to-end pipeline: Find -> Eval -> Select -> Report in a single command.

eval-skills run \
  --query "math" \
  --benchmark coding-easy \
  --skills-dir ./skills \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown \
  --output-dir ./reports

This command automates the entire process:

Find — scans --skills-dir and optionally filters by --query
Eval — evaluates all candidate skills against --benchmark
Select — filters and ranks results using --min-completion, --top-k, and optional --strategy
Report — generates output files in all requested --formats

6. Generate & Compare Reports

Convert report format

eval-skills report convert \
  --input ./reports/eval-result.json \
  --format html \
  --output ./reports/eval-result.html

Supported output formats: markdown, html.

Diff two reports (regression detection)

eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "v1.0" --label-b "v2.0" \
  --output ./reports/diff.md

Generates a side-by-side delta table per skill showing changes in completion rate, error rate, P95 latency, and composite score with directional arrows.

7. Initialize Project

eval-skills init --dir .

Creates the project scaffold:

eval-skills.config.yaml — global configuration
skills/ — directory for skill definitions
benchmarks/ — directory for benchmark files
reports/ — directory for evaluation output

8. Manage Configuration

# List all current configuration values
eval-skills config list

# Get a specific value (supports dot notation)
eval-skills config get llm.model

# Set a value (persisted to ~/.eval-skills/config.yaml)
eval-skills config set concurrency 8
eval-skills config set llm.model gpt-4o
eval-skills config set llm.temperature 0

Configuration is resolved in priority order:

CLI flags (highest priority)
eval-skills.config.yaml in current directory
~/.eval-skills/config.yaml
Built-in defaults (concurrency: 4, timeoutMs: 30000, outputDir: ./reports)

Scorer Types

Each task in a benchmark specifies an evaluator type. The scorer compares the skill's actual output against the expected output.

Type	Aliases	Description	Score Range
`exact_match`	`exact`	Strict equality comparison. Supports `caseSensitive` option.	0 or 1
`contains`	—	Checks for the presence of all specified keywords in the output. Partial credit: `matched_keywords / total_keywords`.	0.0 ~ 1.0
`json_schema`	`schema`	Validates output against a JSON Schema (using Ajv).	0 or 1
`llm_judge`	—	Sends the output + expected rubric to an LLM (configurable model) for quality rating.	0.0 ~ 1.0
`custom`	—	Loads a custom scorer from `expectedOutput.customScorerPath`.	0.0 ~ 1.0

Evaluation Metrics

Every evaluation produces a SkillCompletionReport with these metrics:

Metric	Description	Formula
Completion Rate	Fraction of tasks that passed	`pass_count / total_count`
Partial Score	Mean score across all tasks	`mean(task_scores)`
Error Rate	Fraction of tasks that errored or timed out	`(error_count + timeout_count) / total_count`
Consistency Score	Stability across multiple runs (requires `--runs >= 2`)	`1 - stddev(per_run_completion_rates)`
P50 / P95 / P99 Latency	Response time percentiles	Sorted percentile of `latencyMs`
Composite Score	Weighted overall quality score	`0.5 * CR + 0.2 * (1 - latP95_norm) + 0.3 * (1 - ER)`

Built-in Benchmarks

ID	Domain	Tasks	Scoring	Description
`coding-easy`	coding	20	mean / exact_match	Math expressions, string reversal, palindrome detection
`skill-quality`	tool-use	5	mean / contains	Metadata completeness, description quality, structure checks
`web-search-basic`	web	8	mean / contains + schema	Factual queries, keyword verification, structured output validation
`gaia-v1`	general	—	mean	Placeholder for GAIA benchmark Level 1 tasks
`toolbench-lite`	tool-use	—	mean	Placeholder for ToolBench single-tool scenarios

Custom Benchmark

Create a benchmark.json file:

{
  "id": "my-benchmark",
  "name": "My Custom Benchmark",
  "version": "1.0.0",
  "domain": "general",
  "scoringMethod": "mean",
  "maxLatencyMs": 30000,
  "metadata": { "source": "internal", "lastUpdated": "2026-02-28" },
  "tasks": [
    {
      "id": "task_001",
      "description": "Test basic addition",
      "inputData": { "expression": "2+3" },
      "expectedOutput": { "type": "exact", "value": "5" },
      "evaluator": { "type": "exact" },
      "timeoutMs": 10000,
      "tags": ["math"]
    },
    {
      "id": "task_002",
      "description": "Test keyword presence",
      "inputData": { "query": "TypeScript" },
      "expectedOutput": { "type": "contains", "keywords": ["JavaScript", "Microsoft"] },
      "evaluator": { "type": "contains", "caseSensitive": false },
      "timeoutMs": 15000,
      "tags": ["search"]
    }
  ]
}

eval-skills eval --skills ./my-skill/ --benchmark ./my-benchmark.json

Adapter Types

Skills communicate through adapters. The adapter type is specified in skill.json via adapterType.

Adapter	Protocol	How it works	Key config
`http`	REST POST	Sends `POST { skillId, version, input }` to `skill.entrypoint`. Supports Bearer / API-Key auth via env vars.	`baseUrl`, `authType`, `authTokenEnvKey`
`subprocess`	JSON-RPC 2.0 over stdin/stdout	Spawns `skill.entrypoint` (e.g. `python3 skill.py`), writes JSON-RPC request to stdin, reads response from stdout.	`command`, `args`
`mcp`	MCP Protocol	(Phase 2) Native Model Context Protocol integration via `@modelcontextprotocol/sdk`.	—

Workflow Examples

Evaluating a Single Skill

# 1. Create a skill skeleton
eval-skills create --name my_calc --from-template python_script

# 2. Implement your logic in skills/my_calc/skill.py

# 3. Run evaluation against the coding-easy benchmark
eval-skills eval \
  --skills ./skills/my_calc/skill.json \
  --benchmark coding-easy \
  --runs 3 \
  --format json markdown

# 4. Review the report
cat ./reports/eval-result-*.md

Comparing Multiple Candidate Skills

# 1. Discover candidates
eval-skills find --query "weather" --skills-dir ./skills

# 2. Evaluate all candidates on the same benchmark
eval-skills eval \
  --skills ./skills/weather_v1 ./skills/weather_v2 ./skills/weather_v3 \
  --benchmark web-search-basic \
  --runs 3

# 3. Select the best
eval-skills select \
  --from ./skills \
  --reports ./reports/eval-result-*.json \
  --min-completion 0.8 \
  --top-k 2

# 4. Compare two versions
eval-skills report diff \
  ./reports/v1.json ./reports/v2.json \
  --label-a "weather_v1" --label-b "weather_v2"

Full Pipeline (One Command)

eval-skills run \
  --skills-dir ./skills \
  --benchmark coding-easy \
  --top-k 3 \
  --min-completion 0.7 \
  --format json markdown html \
  --output-dir ./reports

CI/CD Quality Gate

# In your CI pipeline — fail the build if completion rate drops below 80%
eval-skills eval \
  --skills ./skills/production_skill \
  --benchmark coding-easy \
  --exit-on-fail \
  --min-completion 0.8 \
  --format json

Regression Detection

# Compare today's evaluation against the baseline
eval-skills report diff \
  ./reports/baseline.json ./reports/latest.json \
  --label-a "baseline" --label-b "latest" \
  --output ./reports/regression-check.md

Best Practices

Always use --runs 3 or more when evaluating for production decisions. Single-run results can be noisy; the consistency score captures stability across runs.
Use --exit-on-fail in CI/CD pipelines to enforce quality gates. Set --min-completion to your acceptable threshold (recommended: 0.8 for production skills).
Create domain-specific custom benchmarks rather than relying solely on built-in ones. Your custom benchmark should reflect real-world inputs your skill will encounter.
Use report diff after every skill upgrade to catch regressions early. Compare the new evaluation against a saved baseline report.
Use --dry-run before long evaluations to validate your configuration (skill paths, benchmark resolution, task count) without actually executing tasks.
Persist results with --store to track skill quality over time. The SQLite store enables historical trend queries.
Start with --concurrency 1 when debugging a failing skill, then increase for production benchmarking.
Tag your benchmark tasks to enable per-category analysis (e.g., filter by math, string, edge-case).

Skill JSON Schema

Every skill must provide a skill.json that conforms to this structure:

{
  "id": "my_skill_v1",
  "name": "My Skill",
  "version": "1.0.0",
  "description": "Does something useful",
  "tags": ["utility", "math"],
  "inputSchema": {
    "type": "object",
    "properties": { "query": { "type": "string" } },
    "required": ["query"]
  },
  "outputSchema": {
    "type": "object",
    "properties": { "result": { "type": "string" } }
  },
  "adapterType": "subprocess",
  "entrypoint": "python3 skill.py",
  "metadata": {
    "author": "Your Name",
    "license": "MIT",
    "homepage": "https://github.com/you/my-skill"
  }
}

Validation rules:

id: lowercase alphanumeric with _ or -, non-empty
version: semver format (X.Y.Z)
adapterType: one of http, subprocess, mcp, langchain, custom
entrypoint: non-empty string (URL for http, command for subprocess)

Global Options

These options are available on all commands:

Option	Description
`-c, --config <path>`	Path to configuration file
`--json`	JSON output format (CI-friendly)
`--no-color`	Disable colored output
`-v, --verbose`	Verbose logging
`--version`	Show version
`-h, --help`	Show help

When it activates

Automatically, when a task matches. Use this skill to assess skill quality before production, compare candidate skills on the same benchmark, enforce quality gates in CI/CD, and generate human-readable evaluation reports.

About this skill

Automatic

AI Engineer pack

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/2/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

LeoYeAI / openclaw-master-skills

skills/eval-skills/SKILL.md

View on GitHub ↗

License MITImported 8/2/2026Version v1Bundled 3 files · scripts

Illustrative Want to see what leoyeai/eval-skills would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Related skills

Skills that do similar work, with their headline benchmark lift.

From the same source and nearby in the catalogue.

Ready to supercharge your AI agents?

SkillScore

Safety

What it changes

Track record

Use this skill

eval-skills

When to Use This Skill

Capabilities

1. Find Skills

2. Create Skills

3. Evaluate Skills

4. Select Skills

5. Run Pipeline

6. Generate & Compare Reports

Convert report format

Diff two reports (regression detection)

7. Initialize Project

8. Manage Configuration

Scorer Types

Evaluation Metrics

Built-in Benchmarks

Custom Benchmark

Adapter Types

Workflow Examples

Evaluating a Single Skill

Comparing Multiple Candidate Skills

Full Pipeline (One Command)

CI/CD Quality Gate

Regression Detection

Best Practices

Skill JSON Schema

Global Options

When it activates

SkillSafety

Source

Related skills

More in the registry