Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Harness Eval · tech-leads-club | DecimalAI

Agent Toolingby tech-leads-club

Harness Eval

Evaluate a repo agent harness (AGENTS.md, rules, skills, skill refs) for broken paths/commands, redundant instructions, and usefulness using a stack-agnostic dual-judge protocol with planted traps. HIGH PRIORITY questionnaires at top: Q1 optional docs, Q2 B/C budget before Track A (certainty/tokens). A always runs after Q2; B/C opt-in. ADRs/RFCs excluded from T2. Mixed apply uses 11-mixed-apply.md (KEEP/CUT).

Maintained

SkillScore

SkillScore65/100

Test results 65

Safety

3/3 safety checks passed

Code scan
Intent
Content

What it changes

30%65%of test cases passed +35 pts

Without skill7 of 23

With skill15 of 23

9 fixed6 passed either way7 still failing1 broke

Measured on gemini-3.6-flash8/29/2026v119 comparable

Track record

Agent runs: 0; All time

See one task, with and without the skill

One test taskIn Track A path verification, a claim references SPEC_FOLDER/config.yaml. Since SPEC_FOLDER/config.yaml does not exist literally on disk, should Track A mark it as BROKEN?

Without the skillMissed

With the skillPassed

Same model, same task. Answers are capped and redacted.

Use this skill

Your agent runs on

Adds every file the skill has, pinned so you can update it. No account needed.

pip install "decimalai>=0.15"
decimalai skills add tech-leads-club-harness-eval
claude --plugin-dir "$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.agents"

The same install from Python, for a notebook or a setup script.

Install

pip install "decimalai>=0.15"

Add the skill

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

print("Added. Start Claude Code with the command below.")

Start Claude Code with it

claude --plugin-dir "$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.agents"

Type these in a Claude Code session. No account or API key. It writes the skill and any files it bundles, and asks first when its safety rating isn't Passed.

Add the DecimalAI plugin, once

/plugin marketplace add decimal-labs/decimalai-claude-plugins
/plugin install decimalai@decimalai

Install this skill

/decimalai:install tech-leads-club-harness-eval

The plugin needs python3 and curl on your machine.

No Python or account. Claude Code picks it up in its next session.

Save it into your project

curl -fsSL --create-dirs -o .claude/skills/tech-leads-club-harness-eval/SKILL.md https://app.decimal.ai/s/tech-leads-club-harness-eval/SKILL.md

Copy SKILL.md

Save it in your project, as

.claude/skills/tech-leads-club-harness-eval/SKILL.md

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Claude Code docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[claude-agent-sdk]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import asyncio

from claude_agent_sdk import ClaudeAgentOptions
from decimalai.claude_agent_sdk import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

async def main():
    question = "Hello!"
    async with support.run(question) as run:  # one user message, one run
        async for message in run.query(options=ClaudeAgentOptions(max_turns=6)):
            print(message)

# In a notebook or async code: await main()
asyncio.run(main())

Prefer its own loader? ClaudeAgentOptions(plugins=[skills_plugin()], skills="all") reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

The same install from Python, for a notebook or a setup script. agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[claude-agent-sdk]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import asyncio

from claude_agent_sdk import ClaudeAgentOptions
from decimalai.claude_agent_sdk import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

async def main():
    question = "Hello!"
    async with support.run(question) as run:  # one user message, one run
        async for message in run.query(options=ClaudeAgentOptions(max_turns=6)):
            print(message)

# In a notebook or async code: await main()
asyncio.run(main())

Prefer its own loader? ClaudeAgentOptions(plugins=[skills_plugin()], skills="all") reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

import asyncio
from pathlib import Path
from claude_agent_sdk import ClaudeAgentOptions, query

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
options = ClaudeAgentOptions(system_prompt=f"You are a helpful agent.\n\n{skill}", max_turns=6)

async def main():
    async for message in query(prompt="Hello!", options=options):
        print(message)

# In a notebook or async code: await main()
asyncio.run(main())

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Claude Agent SDK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[openai-agents]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from agents import Agent, Runner
from decimalai.openai_agents import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    name=support.name,
    instructions=support.instructions,  # your prompt, then the skill list
    tools=[*support.tools],             # your tools, then load_skill
)
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

Using SandboxAgent? capabilities=[Shell(), decimalai.openai_agents.sandbox_skills()] points its own loader at the same folder.

Install

pip install "decimalai[openai-agents]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from agents import Agent, Runner
from decimalai.openai_agents import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    name=support.name,
    instructions=support.instructions,  # your prompt, then the skill list
    tools=[*support.tools],             # your tools, then load_skill
)
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

Using SandboxAgent? capabilities=[Shell(), decimalai.openai_agents.sandbox_skills()] points its own loader at the same folder.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from agents import Agent, Runner

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = Agent(name="support", instructions=f"You are a helpful agent.\n\n{skill}")
# In a notebook or async code: result = await Runner.run(agent, "Hello!")
result = Runner.run_sync(agent, "Hello!")

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

OpenAI Agents docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[openai]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import json

from openai import OpenAI
from decimalai.openai import agent_config

client = OpenAI()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your tools, as Chat Completions takes them, and what answers each.
TOOLS = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    # run.system is your prompt, then the skill list.
    messages = [{"role": "system", "content": run.system}, run.message]
    while True:
        reply = client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=messages,
            tools=[*TOOLS, *run.tools],  # your tools, then load_skill
        ).choices[0].message
        messages.append(reply)
        if not reply.tool_calls:
            break
        for call in reply.tool_calls:
            # load_skill is answered by the run; your own tools by FUNCTIONS.
            function = FUNCTIONS.get(call.function.name)
            messages.append(run.tool_result(call) or {
                "role": "tool", "tool_call_id": call.id,
                "content": function(**json.loads(call.function.arguments)) if function
                else f"There is no tool named {call.function.name!r}.",
            })
print(reply.content)

# The Responses API takes run.response_tools instead of run.tools.

Install

pip install "decimalai[openai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import json

from openai import OpenAI
from decimalai.openai import agent_config

client = OpenAI()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your tools, as Chat Completions takes them, and what answers each.
TOOLS = [{"type": "function", "function": {
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}},
                   "required": ["order_id"]},
}}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    # run.system is your prompt, then the skill list.
    messages = [{"role": "system", "content": run.system}, run.message]
    while True:
        reply = client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=messages,
            tools=[*TOOLS, *run.tools],  # your tools, then load_skill
        ).choices[0].message
        messages.append(reply)
        if not reply.tool_calls:
            break
        for call in reply.tool_calls:
            # load_skill is answered by the run; your own tools by FUNCTIONS.
            function = FUNCTIONS.get(call.function.name)
            messages.append(run.tool_result(call) or {
                "role": "tool", "tool_call_id": call.id,
                "content": function(**json.loads(call.function.arguments)) if function
                else f"There is no tool named {call.function.name!r}.",
            })
print(reply.content)

# The Responses API takes run.response_tools instead of run.tools.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from openai import OpenAI

client = OpenAI()
skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
reply = client.chat.completions.create(
    model="gpt-5.4-mini",
    messages=[
        {"role": "system", "content": f"You are a helpful agent.\n\n{skill}"},
        {"role": "user", "content": "Hello!"},
    ],
).choices[0].message

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

OpenAI SDK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[anthropic]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import anthropic
from decimalai.anthropic import agent_config

client = anthropic.Anthropic()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

question = "Hello!"
with support.run(question) as run:  # one user message, one run
    answer = client.beta.messages.tool_runner(
        model="claude-haiku-5-5",
        max_tokens=4096,
        system=run.system,     # your prompt, then the skill list
        tools=[*run.tools],    # your tools, then load_skill
        messages=[{"role": "user", "content": question}],
    ).until_done()

Install

pip install "decimalai[anthropic]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import anthropic
from decimalai.anthropic import agent_config

client = anthropic.Anthropic()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

question = "Hello!"
with support.run(question) as run:  # one user message, one run
    answer = client.beta.messages.tool_runner(
        model="claude-haiku-5-5",
        max_tokens=4096,
        system=run.system,     # your prompt, then the skill list
        tools=[*run.tools],    # your tools, then load_skill
        messages=[{"role": "user", "content": question}],
    ).until_done()

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
import anthropic

client = anthropic.Anthropic()
skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
reply = client.messages.create(
    model="claude-haiku-5-5",
    max_tokens=4096,
    system=f"You are a helpful agent.\n\n{skill}",
    messages=[{"role": "user", "content": "Hello!"}],
)

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Anthropic docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[google-genai]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from google import genai
from google.genai import types
from decimalai.google_genai import agent_config

client = genai.Client()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your functions, as declarations for the model, and what answers each.
TOOLS = [{"function_declarations": [{
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters_json_schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                               "required": ["order_id"]},
}]}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    config = types.GenerateContentConfig(
        system_instruction=run.system,     # your prompt, then the skill list
        tools=[*TOOLS, *run.tool_params],  # your functions, then load_skill
    )
    contents = [{"role": "user", "parts": [{"text": run.message}]}]
    while True:
        response = client.models.generate_content(
            model="gemini-3.5-flash", contents=contents, config=config,
        )
        if not response.function_calls:
            break
        contents.append(response.candidates[0].content)
        parts = []
        for call in response.function_calls:
            # load_skill is answered by the run; your own functions by FUNCTIONS.
            # A loop of your own rather than automatic function calling, which
            # raises KeyError when the model asks for a function you did not pass.
            function = FUNCTIONS.get(call.name)
            parts.append(run.tool_result(call) or {"function_response": {
                "name": call.name, "id": call.id,
                "response": {"result": function(**(call.args or {})) if function
                             else f"There is no function named {call.name!r}."},
            }})
        contents.append({"role": "user", "parts": parts})
print(response.text)

Install

pip install "decimalai[google-genai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from google import genai
from google.genai import types
from decimalai.google_genai import agent_config

client = genai.Client()
# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")


def lookup_order(order_id: str) -> str:
    return f"Order {order_id} shipped yesterday."


# Your functions, as declarations for the model, and what answers each.
TOOLS = [{"function_declarations": [{
    "name": "lookup_order",
    "description": "Look up the status of an order.",
    "parameters_json_schema": {"type": "object", "properties": {"order_id": {"type": "string"}},
                               "required": ["order_id"]},
}]}]
FUNCTIONS = {"lookup_order": lookup_order}

question = "Where is order A1001?"
with support.run(question) as run:  # one user message, one run
    config = types.GenerateContentConfig(
        system_instruction=run.system,     # your prompt, then the skill list
        tools=[*TOOLS, *run.tool_params],  # your functions, then load_skill
    )
    contents = [{"role": "user", "parts": [{"text": run.message}]}]
    while True:
        response = client.models.generate_content(
            model="gemini-3.5-flash", contents=contents, config=config,
        )
        if not response.function_calls:
            break
        contents.append(response.candidates[0].content)
        parts = []
        for call in response.function_calls:
            # load_skill is answered by the run; your own functions by FUNCTIONS.
            # A loop of your own rather than automatic function calling, which
            # raises KeyError when the model asks for a function you did not pass.
            function = FUNCTIONS.get(call.name)
            parts.append(run.tool_result(call) or {"function_response": {
                "name": call.name, "id": call.id,
                "response": {"result": function(**(call.args or {})) if function
                             else f"There is no function named {call.name!r}."},
            }})
        contents.append({"role": "user", "parts": parts})
print(response.text)

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from google import genai
from google.genai import types

client = genai.Client()
skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
response = client.models.generate_content(
    model="gemini-3.5-flash",
    contents="Hello!",
    config=types.GenerateContentConfig(system_instruction=f"You are a helpful agent.\n\n{skill}"),
)

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Gemini API docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[langchain]" langchain-openai
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from langchain.agents import create_agent
from decimalai.langchain import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = create_agent(
    "openai:gpt-5.4-mini",
    tools=[],                         # your tools; the middleware adds load_skill
    middleware=[support.middleware],  # your prompt, the skill list and load_skill
)
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

Install

pip install "decimalai[langchain]" langchain-openai

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from langchain.agents import create_agent
from decimalai.langchain import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = create_agent(
    "openai:gpt-5.4-mini",
    tools=[],                         # your tools; the middleware adds load_skill
    middleware=[support.middleware],  # your prompt, the skill list and load_skill
)
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from langchain.agents import create_agent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = create_agent("openai:gpt-5.4-mini", tools=[], system_prompt=f"You are a helpful agent.\n\n{skill}")
result = agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

LangChain docs

CrewAI's own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" "crewai>=1.15.27"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import decimalai.crewai
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                               # your own CrewAI LLM
    skills=decimalai.crewai.skills(),      # the installed folder
    tools=decimalai.crewai.skill_tools(),  # reads a skill's files, runs its scripts
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

The same install from Python, for a notebook or a setup script. CrewAI's own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" "crewai>=1.15.27"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import decimalai.crewai
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                               # your own CrewAI LLM
    skills=decimalai.crewai.skills(),      # the installed folder
    tools=decimalai.crewai.skill_tools(),  # reads a skill's files, runs its scripts
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

No DecimalAI package. CrewAI's own loader reads the folder, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from crewai import Agent, Crew, Task

agent = Agent(
    role="Support", goal="Help the customer", backstory="A support agent.",
    llm=llm,                         # your own CrewAI LLM
    skills=[Path(".agents/skills")],
)
task = Task(description="Hello!", expected_output="an answer", agent=agent)
Crew(agents=[agent], tasks=[task]).kickoff()

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

CrewAI docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[adk]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from decimalai.adk import DecimalAIPlugin, agent_config
from google.adk.agents import LlmAgent
from google.adk.apps import App

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = LlmAgent(
    name="support",          # ADK needs a Python identifier
    model="gemini-3.5-flash",
    instruction=support.instruction,  # your prompt, then the skill list
    tools=[*support.tools],           # your tools, then load_skill
)
# DecimalAIPlugin makes each invocation of runners built from this App one trace.
app = App(name="support", root_agent=agent, plugins=[DecimalAIPlugin()])

Prefer ADK's own loader? SkillToolset(skills=decimalai.adk.skills(), ...) reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

Install

pip install "decimalai[adk]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from decimalai.adk import DecimalAIPlugin, agent_config
from google.adk.agents import LlmAgent
from google.adk.apps import App

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = LlmAgent(
    name="support",          # ADK needs a Python identifier
    model="gemini-3.5-flash",
    instruction=support.instruction,  # your prompt, then the skill list
    tools=[*support.tools],           # your tools, then load_skill
)
# DecimalAIPlugin makes each invocation of runners built from this App one trace.
app = App(name="support", root_agent=agent, plugins=[DecimalAIPlugin()])

Prefer ADK's own loader? SkillToolset(skills=decimalai.adk.skills(), ...) reads the same folder. Use one or the other: a skill delivered both ways reaches the prompt twice.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from google.adk.agents import LlmAgent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = LlmAgent(
    name="support",
    model="gemini-3.5-flash",
    instruction=lambda context: f"You are a helpful agent.\n\n{skill}",
)

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Google ADK docs

agent_config() hands your agent the skill. With no DecimalAI key it reads your project's files and sends nothing. Needs SDK 0.27 or newer.

Install

pip install "decimalai[pydantic-ai]"
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from pydantic_ai import Agent
from decimalai.pydantic_ai import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    "openai:gpt-4o-mini",
    name=support.name,
    tools=[],                 # your tools; the capability adds load_skill
    capabilities=[support],   # your prompt, the skill list and load_skill
)
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

On the Pydantic AI harness, decimalai.pydantic_ai_harness.skills(project) uses its own loader instead.

Install

pip install "decimalai[pydantic-ai]"

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from pydantic_ai import Agent
from decimalai.pydantic_ai import agent_config

# With no DECIMAL_API_KEY this reads the skills your project lists, from disk.
support = agent_config("support", fallback="You are a helpful agent.")

agent = Agent(
    "openai:gpt-4o-mini",
    name=support.name,
    tools=[],                 # your tools; the capability adds load_skill
    capabilities=[support],   # your prompt, the skill list and load_skill
)
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

On the Pydantic AI harness, decimalai.pydantic_ai_harness.skills(project) uses its own loader instead.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from pydantic_ai import Agent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = Agent("openai:gpt-4o-mini", instructions=f"You are a helpful agent.\n\n{skill}")
# In a notebook or async code: result = await agent.run("Hello!")
result = agent.run_sync("Hello!")

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Pydantic AI docs

Deep Agents' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" deepagents
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import decimalai.deepagents
from deepagents import create_deep_agent

agent = create_deep_agent(model=model, **decimalai.deepagents.skills())
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

The default backend reads files but has no shell. To run a skill's scripts, pass skills(backend=LocalShellBackend(...)).

The same install from Python, for a notebook or a setup script. Deep Agents' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" deepagents

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import decimalai.deepagents
from deepagents import create_deep_agent

agent = create_deep_agent(model=model, **decimalai.deepagents.skills())
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

The default backend reads files but has no shell. To run a skill's scripts, pass skills(backend=LocalShellBackend(...)).

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from deepagents import create_deep_agent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = create_deep_agent(model=model, system_prompt=f"You are a helpful agent.\n\n{skill}")
agent.invoke({"messages": [{"role": "user", "content": "Hello!"}]})

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Deep Agents docs

Its own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" agent-framework
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

import asyncio

from agent_framework import Agent
from decimalai.agent_framework import skills

agent = Agent(client=client, instructions="You are a helpful agent.", context_providers=[skills()])
asyncio.run(agent.run("Hello!"))

It caches skills by default, so restart the process after decimalai skills update.

The same install from Python, for a notebook or a setup script. Its own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" agent-framework

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

import asyncio

from agent_framework import Agent
from decimalai.agent_framework import skills

agent = Agent(client=client, instructions="You are a helpful agent.", context_providers=[skills()])
asyncio.run(agent.run("Hello!"))

It caches skills by default, so restart the process after decimalai skills update.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

import asyncio
from pathlib import Path
from agent_framework import Agent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = Agent(client=client, instructions=f"You are a helpful agent.\n\n{skill}")
asyncio.run(agent.run("Hello!"))

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Microsoft Agent Framework docs

Strands' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" strands-agents
decimalai skills add tech-leads-club-harness-eval

Add it to your agent

from decimalai.strands import skills
from strands import Agent
from strands.vended_tools import file_editor, shell

agent = Agent(
    model=model,                # always pass one: the default is Amazon Bedrock
    system_prompt="You are a helpful agent.",
    tools=[file_editor, shell],
    plugins=[skills()],
)
agent("Hello!")

Strands loads an agent's skills once, so build the agent again after decimalai skills update.

The same install from Python, for a notebook or a setup script. Strands' own skill loader reads the installed folder. No DecimalAI account needed.

Install

pip install "decimalai>=0.16" strands-agents

Add it to your agent

from decimalai import skillpkg

# Writes .agents/skills and .decimal/skills.toml. Safe to run twice.
result = skillpkg.add("tech-leads-club-harness-eval")
if result.exit_code:
    raise SystemExit(result.problems)

from decimalai.strands import skills
from strands import Agent
from strands.vended_tools import file_editor, shell

agent = Agent(
    model=model,                # always pass one: the default is Amazon Bedrock
    system_prompt="You are a helpful agent.",
    tools=[file_editor, shell],
    plugins=[skills()],
)
agent("Hello!")

Strands loads an agent's skills once, so build the agent again after decimalai skills update.

No DecimalAI package. The whole skill goes in every prompt, and nothing pins or updates it.

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

Give it to your agent

from pathlib import Path
from strands import Agent

skill = Path(".agents/skills/tech-leads-club-harness-eval/SKILL.md").read_text()
agent = Agent(model=model, system_prompt=f"You are a helpful agent.\n\n{skill}")
agent("Hello!")

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Strands docs

Adds every file the skill has, pinned so you can update it. Cursor, Copilot, Cline and Warp read this folder by themselves.

pip install "decimalai>=0.15"
decimalai skills add tech-leads-club-harness-eval

No Python or account. Cursor, Copilot, Cline and Warp read this folder by themselves.

Save it into your project

curl -fsSL --create-dirs -o .agents/skills/tech-leads-club-harness-eval/SKILL.md https://app.decimal.ai/s/tech-leads-club-harness-eval/SKILL.md

Copy SKILL.md

Save it in your project, as

.agents/skills/tech-leads-club-harness-eval/SKILL.md

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Cursor and other editors docs

Fetch the file and give its text to your model as instructions. No account or package needed.

Get the file

curl -fsSL https://app.decimal.ai/s/tech-leads-club-harness-eval/SKILL.md

Read the summary as JSON

curl -fsSL https://app.decimal.ai/s/tech-leads-club-harness-eval.json

This gets SKILL.md only. The skill's 11 other files are not included.

Other ways to add it

Other setups docs

You don't need a DecimalAI account.

Version: v1, tested 8/29/2026
Source: tech-leads-club/agent-skills ↗
License: NOASSERTION

Manage it in DecimalAI

Saved prompts and skills, pinned versions, and traces of this skill in your runs. DecimalAI does not run your agent.

Claude Code reads files from your project, so DecimalAI config does not apply there.

On Claude Agent SDK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On OpenAI Agents: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On OpenAI SDK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Anthropic: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Gemini API: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On LangChain: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On CrewAI: set DECIMAL_API_KEY and add decimalai.init(crewai=True) to record runs and skill use. The skill version stays in your repo.

On Google ADK: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Pydantic AI: save the skill to an agent here, set DECIMAL_API_KEY, and the same code uses it with no command to run. Changes you save reach the next run with no redeploy.

On Deep Agents: set DECIMAL_API_KEY and add decimalai.init(langchain=True) to record runs and skill use. The skill version stays in your repo.

On Microsoft Agent Framework: set DECIMAL_API_KEY and add decimalai.init(otel=True, agent_name="support") to record runs and skill use. The skill version stays in your repo.

On Strands: set DECIMAL_API_KEY and add decimalai.init(otel=True, agent_name="support") to record runs and skill use. The skill version stays in your repo.

Editors read files from your project, so DecimalAI config does not apply there.

Anywhere else: record a use with decimalai.log_skill_activation() inside a trace.

Use this skill

SKILL.md

Harness Eval

Run a full, stack-agnostic harness evaluation and stop at reports. Do not auto-edit AGENTS.md or skills unless the user explicitly asks after reviewing Ship/Slim.

User questionnaires (HIGH PRIORITY)

Stop and ask before continuing. Do not skip these gates. Do not silently include optional docs or spawn B/C judges.

Order after inventory: Q1 (if needed) → Q2 → then Track A (A always runs) → B/C only if approved.

Q1 — Optional project docs (after inventory)

When optional-docs-candidates.md lists optional types, ask before Q2 / Track A:

Inventory found cited project docs outside the agent skill trees.

- **Always in scope:** skill-tree files (`.agents/skills`, `.cursor/skills`, `.claude/skills`)
- **Always excluded:** ADRs / RFCs / decision-record trees (never scored as T2)
- **Optional (default: omit):** see types/paths in `optional-docs-candidates.md`

Include any optional doc types or paths in this run?
Reply with: `none` (default), type ids (e.g. `docs`), and/or specific paths.

Re-run inventory with --include-doc-type / --include-doc only after the user answers. If no optional types, skip Q1.

Q2 — Tracks B and C (before Track A — budget)

Ask before Track A so the user sets spend up front. Track A always runs next (deterministic, ~0 model tokens). B/C run only if approved.

Choose eval scope for this run (before Track A).

| Track | Question | Certainty | Token consumption |
|-------|----------|-----------|-------------------|
| **A — Correctness** | Cited path/command exists? | **Highest** — script only, no LLM. Prefers false negatives over false BROKEN. | **~0 model tokens** (always runs next) |
| **B — Redundancy** | Would an agent rediscover this cheaply without the harness? | **Medium** — dual LLM + plants; Ship only if trap PASS and both agree. Disagree → Hold. Less model-sensitive than C. | **High** — 2 judges × every claim (~N in this inventory). Each may spot-check the repo. |
| **C — Usefulness** | Does this surface change behavior vs theory/demo/overlap? | **Lowest / most subjective** — dual LLM + plants + fan-in; **model-sensitive**. Slim/Mixed need gates; prefer second-model check before large deletes. | **Highest** — 2 judges × every surface (whole files; often dominates the run). |

Notes: Ship (B) ≠ Slim (C). Rediscoverable ≠ useless. A always runs; B/C are optional.

Reply with one of: `A only`, `B`, `C`, or `B+C`.

Fill claim count from claims.md when known; surface count ≈ T0+T1+T2 markdown after extract (or say “after surfaces_extract” if not run yet).

A only: run Track A; present 04; stop (no B/C judges).
B: Track A, then Steps 4–6.
C: Track A, then Steps 7–10 (C does not need B).
B+C: Track A, then Steps 4–11.

If the user already requested B/C/full eval in the triggering message, treat as approval — still show the Q2 table once so costs are visible.

Loading this skill's files

This skill is self-contained. Protocol, scripts, and judge prompts live under this skill directory (the folder that contains this SKILL.md). Resolve SKILL_DIR as that directory — never assume another install path.

Read references/PROTOCOL.md completely before the first run in a session (and again if scripts fail).
Read references/judge-prompts.md when spawning Track B or Track C judges.
Plain-language terms: references/GLOSSARY.md (also embedded at the top of 04 / 07 / 10 reports).
Claim record shape: references/claims.schema.json (for tooling; agents do not need to load it every run).
Run scripts as python3 "$SKILL_DIR/scripts/<name>.py" ....

Run outputs (not protocol) go to the target repo at .harness-eval/runs/<run-id>/.

Critical rules

Report-only by default. Judgment ≠ remediation.
README out of scope as harness surface and as rediscovery/usefulness evidence.
Stack-agnostic. Never hard-code package managers, DBs, frameworks, or folder layouts in prompts or plants. Discover manifests that exist (JS, Python, Make/Task, Rust, Go, PHP, Ruby/Rails, Java/Gradle/Maven, plus bin/*).
Doc scope. T2 always includes agent skill-tree refs (.agents/skills, .cursor/skills, .claude/skills). ADRs / RFCs (decision-record trees) are always excluded from T2 surfaces. Other cited project docs are optional — default omit; ask via Q1 at the top of this skill, then re-run with --include-doc-type / --include-doc.
Track A always runs after inventory (deterministic, high-precision). Prefer false negatives over false BROKEN. Placeholders (SPEC_FOLDER, {x}, [feature]) are never BROKEN. Never normalize paths with str.lstrip('./').
Tracks B and C require user approval via Q2 before Track A. Do not spawn B/C judges until the user opts in. User may approve B only, C only, both, or A only.
Track B needs dual judges + plants. Judge2 is blind (must not read Judge1 scores or trap-key.json). Ship only if trap gate PASS and dual REDUNDANT with Judge2 cost ≤ 1.
Track C needs dual judges + plants. Blind Judge2 must not read 08-usefulness-j1.md or usefulness-trap-key.json. Slim only if trap PASS, dual SLIM/ROUTING-ONLY, and fan-in PASS (no other harness surface hard-loads the path as SoT — merge enforces this on the full skill tree, not just --seed). Usefulness is model-sensitive — record model: <id> in both score files; prefer same model within a run; re-judge on a second model before large Slim deletes.
KEEP / KEEP-CORE plants must not be verbatim copies of claims/surfaces already in the deck.
Subagents: use an allowlisted non-fast model (prefer the same family as the parent when policy allows). Do not use *-fast models.
Do not equate tracks. Track B Ship ≠ Track C Slim. Rediscoverable ≠ useless; useful ≠ non-redundant.
Slim apply / fan-in. Never stub or delete a Slim path listed under “Slim fan-in blocked” (or when python3 "$SKILL_DIR/scripts/slim_fanin.py" --path <P> reports citers) unless those consumers are updated in the same change.
Mixed/Slim apply stays self-contained. Cutting REPO-DEMONSTRATED / THEORY means delete or compress that bulk in the harness surface. Never replace a fenced teaching snippet (or the contract it carried) with See app/... / lib/... / test/... — that swaps SoT for a code-tree pointer. Judge evidence paths stay in score tables only; if the behavior-changing contract must survive, keep a short in-skill rule or snippet.
Mixed apply is mechanical. Dual MIXED alone is not enough. Merge emits 11-mixed-apply.md with per-ID KEEP (from Keep-core columns) and CUT (from Slim columns). Apply agents must follow that file only — do not re-judge, redesign, or invent a different pattern than KEEP. Empty Keep-core/Slim cells → skip that path (Hold).

Instructions

Step 1: Resolve SKILL_DIR

Set SKILL_DIR to the directory containing this SKILL.md. Verify:

$SKILL_DIR/references/PROTOCOL.md
$SKILL_DIR/scripts/inventory_extract.py
$SKILL_DIR/scripts/track_a_correctness.py
$SKILL_DIR/scripts/merge_agreement.py
$SKILL_DIR/scripts/surfaces_extract.py
$SKILL_DIR/scripts/merge_usefulness.py
$SKILL_DIR/scripts/slim_fanin.py
$SKILL_DIR/scripts/doc_scope.py

If missing, the skill install is broken — stop.

Step 2: Inventory + claim deck

From the target repo root:

RUN_ID=$(date -u +%Y-%m-%d)-full
python3 "$SKILL_DIR/scripts/inventory_extract.py" --root . --run-id "$RUN_ID"
# Optional scope: AGENTS.md + one-hop related skills only
# python3 "$SKILL_DIR/scripts/inventory_extract.py" --root . --run-id "$RUN_ID" --seed AGENTS.md

Expected under .harness-eval/runs/$RUN_ID/: inventory.json, claims.jsonl, claims.md, trap-key.json, optional-docs-candidates.md (+ .json).

Step 2b: Optional docs — Q1 (see top)

Read optional-docs-candidates.md. If optional types exist, run Q1 from User questionnaires. Re-run inventory only after approval:

python3 "$SKILL_DIR/scripts/inventory_extract.py" --root . --run-id "$RUN_ID" \
  --include-doc-type docs   # and/or --include-doc path

Step 2c: Track budget — Q2 (see top)

Run Q2 from User questionnaires before Track A. Record the answer (A only / B / C / B+C). Do not start Steps 4+ unless B and/or C were approved.

Step 3: Track A (deterministic) — always run

python3 "$SKILL_DIR/scripts/track_a_correctness.py" --root . --run-id "$RUN_ID"

Expected: 04-correctness.md (includes term definitions at top). Spot-check that .agents/... cites resolve (not agents/...).

Summarize Track A (broken count + notable clusters). If Q2 was A only, stop. Otherwise continue to the approved B and/or C steps.

Step 4: Track B — Judge1

Read references/judge-prompts.md (Track B Judge1). Spawn an independent subagent with an allowlisted model. Point it at .harness-eval/runs/$RUN_ID/claims.md. It writes 05-redundancy-j1.md (include model: <id>).

Judge1 may read inventory.json. Must not read trap-key.json.

Step 5: Track B — Judge2 (blind)

Read references/judge-prompts.md (Track B Judge2). Spawn a second subagent. Writes 06-blind-scores.md.

Forbidden for Judge2: trap-key.json, 05-redundancy-j1.md, 07-agreement.md, prior agreement reports.

Prefer Steps 4 and 5 in parallel.

Step 6: Merge Track B agreement

python3 "$SKILL_DIR/scripts/merge_agreement.py" --run-dir .harness-eval/runs/$RUN_ID

Expected: 07-agreement.md (Ship/Review/Hold + What these words mean). On trap FAIL: fix plants per PROTOCOL, rescore P00x, re-merge — do not Ship.

Step 7: Track C — surface deck

python3 "$SKILL_DIR/scripts/surfaces_extract.py" --root . --run-id "$RUN_ID"

Expected: surfaces.md, surfaces.json, usefulness-trap-key.json.

Step 8: Track C — Usefulness Judge1

Read references/judge-prompts.md (Usefulness Judge1). Spawn subagent with allowlisted model (record same id in header). Writes 08-usefulness-j1.md.

Must not read usefulness-trap-key.json.

Step 9: Track C — Usefulness Judge2 (blind)

Read Usefulness Judge2 prompt. Prefer same model as Step 8 for agreement stability. Writes 09-usefulness-j2.md.

Forbidden: usefulness-trap-key.json, 08-usefulness-j1.md, 10-usefulness-agreement.md, and using Track B 05/06/07 to decide usefulness classes.

Prefer Steps 8 and 9 in parallel.

Step 10: Merge Track C agreement

python3 "$SKILL_DIR/scripts/merge_usefulness.py" --run-dir .harness-eval/runs/$RUN_ID

Expected: 10-usefulness-agreement.md (Slim/Keep-core/Mixed/Hold + What these words mean), 11-mixed-apply.md (KEEP/CUT per Mixed ID), plus slim-fanin.json. On trap FAIL: do not Slim. Surfaces with slim-fanin-blocked are Hold — not Slim apply candidates.

Step 11: Present results

Summarize from the agreement reports (each starts with term definitions):

Track A broken count → 04-correctness.md
Track B trap + Ship/Review/Hold → 07-agreement.md
Track C trap + fan-in + Slim/Keep-core/Mixed/Hold → 10-usefulness-agreement.md
Call out 11-mixed-apply.md when Mixed count > 0 (the only Mixed apply path)
Call out model ids used for Track C and that Slim is model-sensitive
Call out any Slim fan-in blocked rows (consumers outside seed may appear here)

Stop unless the user asks to apply Ship/Slim/Mixed. When applying:

Slim: only paths in the Slim table (fan-in PASS); never stub fan-in-blocked paths without updating citers first.
Mixed: open 11-mixed-apply.md and execute KEEP/CUT per ID only (rule 12). Never re-judge from the Mixed path list alone. Never add code-tree path pointers as substitutes for cut demos (rule 11).

Examples

Example 1: Full harness eval

User says: "run harness eval on this repo"

Actions: inventory → Q1 if needed → Q2 (B/C budget table) → Track A → if approved, Steps 4–11. Parallel B judges, then C judges. Present agreements (terms are in the files).

Example 2: Usefulness only (existing run)

User says: "run Track C usefulness on the last harness-eval run"

Actions: Steps 7–11 on that RUN_ID (inventory must already exist).

Example 3: Wrong skill

User says: "setup harness" / "init harness" → harness setup (not this skill). User says: "specify feature" → tlc-spec-driven.

Troubleshooting

Trap gate FAIL (Track B or C)

Cause: KEEP/KEEP-CORE plants were deck duplicates, or blind judge mis-family. Solution: use skill’s fixed plant templates; rescore plants; re-merge.

Track A false missing `.agents/...`

Cause: bad path normalization. Solution: skill script must use normalize_cite (strip ./ only). Re-run Track A from $SKILL_DIR/scripts/.

Subagent blocked

Cause: missing/allowlisted model or *-fast blocked. Solution: re-spawn with an allowlisted non-fast model.

Track C Slim looks wrong after model change

Expected: usefulness is model-sensitive. Re-run C1+C2 on a second model; intersection of Slim bands is the safe delete set.

Mixed apply rewrote conventions / removed modules

Cause: apply agent re-judged from the Mixed path list instead of following KEEP/CUT. Solution: apply only via 11-mixed-apply.md; if that file is missing, re-run merge_usefulness.py; if Keep-core/Slim cells are vague, re-score those IDs before apply.

T2 empty / skill `references/` missing from inventory

Cause: path normalize used lstrip("./") and turned .agents/… into agents/…. Solution: doc_scope.normalize_rel must strip only a ./ prefix (same rule as Track A).

ADRs appeared in Track C

Cause: old inventory treated all one-hop docs/** as T2. Solution: v1.7+ excludes decision-record trees; only user-approved optional doc types (never ADR/RFC) can enter T2.

Slim stub broke another skill that loads that file

Cause: content OVERLAP/Slim without fan-in — older runs, or apply skipped the gate. Solution: restore the checklist body; re-merge with merge_usefulness.py (fan-in scans full skill trees). Confirm with slim_fanin.py --path <P>.

Scripts missing

Cause: incomplete skill folder. Solution: restore $SKILL_DIR/scripts/ and references/.

When it activates

Automatically, when a task matches. Use when the user says harness eval, harness-eval, harness debug, audit AGENTS.md, audi

About this skill

Automatic

AI Engineer pack

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/10/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

tech-leads-club / agent-skills

packages/skills-catalog/skills/(development)/harness-eval/SKILL.md

View on GitHub ↗

License CC-BY-4.0Imported 9/1/2026Version v1Bundled 11 files · scripts

Illustrative Want to see what tech-leads-club/harness-eval would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Related skills

Skills that do similar work, with their headline benchmark lift.

From the same source and nearby in the catalogue.

Ready to supercharge your AI agents?

SkillScore

Safety

What it changes

Track record

Use this skill

Harness Eval

User questionnaires (HIGH PRIORITY)

Q1 — Optional project docs (after inventory)

Q2 — Tracks B and C (before Track A — budget)

Loading this skill's files

Critical rules

Instructions

Step 1: Resolve SKILL_DIR

Step 2: Inventory + claim deck

Step 2b: Optional docs — Q1 (see top)

Step 2c: Track budget — Q2 (see top)

Step 3: Track A (deterministic) — always run

Step 4: Track B — Judge1

Step 5: Track B — Judge2 (blind)

Step 6: Merge Track B agreement

Step 7: Track C — surface deck

Step 8: Track C — Usefulness Judge1

Step 9: Track C — Usefulness Judge2 (blind)

Step 10: Merge Track C agreement

Step 11: Present results

Examples

Example 1: Full harness eval

Example 2: Usefulness only (existing run)

Example 3: Wrong skill

Troubleshooting

Trap gate FAIL (Track B or C)

Track A false missing .agents/...

Subagent blocked

Track C Slim looks wrong after model change

Mixed apply rewrote conventions / removed modules

T2 empty / skill references/ missing from inventory

ADRs appeared in Track C

Slim stub broke another skill that loads that file

Scripts missing

When it activates

SkillSafety

Source

Related skills

More in the registry

Track A false missing `.agents/...`

T2 empty / skill `references/` missing from inventory