Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

Get Started Free →

Construct Validity Assessment SOP · yogsoth-ai | DecimalAI

ai mlby yogsoth-ai

Construct Validity Assessment SOP

yogsoth-ai/construct-validity-assessment

Source yogsoth-ai/de-anthropocentric-research-engine ↗License Apache-2.0

Evaluate whether benchmark measures its claimed capability

SkillScore91Maintained

Automatic

AI Engineer pack

Last verified 8/22/2026

What it changes

Passed 21 of 23 test cases with the skill, and 13 of 23 without it.

Without skill57%

With skill91%

Checks it passed that the model alone missed

Checks convergent validity against standard visual grounding datasets
Evaluates convergent validity with established multi-hop QA benchmarks
Checks convergent validity with medical diagnostic performance benchmarks

and 8 more in the full report

9 fixed12 passed either way1 still failing1 broke

Pass-rate lift: +35 pp
Tokens: 0%
Agent turns: 0%
Response time: -19%

Model gemini-3.6-flashMeasured 8/22/2026v121 of 23 cases passed with skill

Tokens, turns and time describe resource use, not a dollar price.

Use this skill

Save it into your project's skills folder. Claude Code picks it up in its next session.

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-construct-validity-assessment/SKILL.md https://app.decimal.ai/s/yogsoth-ai-construct-validity-assessment/SKILL.md

Want it pinned and updatable? pip install "decimalai>=0.15", then decimalai skills add yogsoth-ai-construct-validity-assessment, and start Claude Code with claude --plugin-dir "$(git rev-parse --show-toplevel)/.agents"

1 · One-time — install the CLI (docs ↗)

pip install "decimalai>=0.15"

2 · Get the skill — writes .agents/skills/yogsoth-ai-construct-validity-assessment/ and pins it in .decimal/skills.lock

decimalai skills add yogsoth-ai-construct-validity-assessment

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/yogsoth-ai-construct-validity-assessment/SKILL.md https://app.decimal.ai/s/yogsoth-ai-construct-validity-assessment/SKILL.md

Raw file: /s/yogsoth-ai-construct-validity-assessment/SKILL.md ↗ · machine-readable summary at /s/yogsoth-ai-construct-validity-assessment.json ↗

Safety checks: 3 of 3 passed

Code scan CleanIntent PassedContent Clean

See it in action

One of 9 test cases the skill fixed. Same model, same prompt, with and without the skill.

1 of 9

Prompt

Can you run a construct validity assessment on the benchmark 'LogicBench-v2'? The authors state that it evaluates 'multi-step logical reasoning in pure text', but I want to make sure it isn't just testing memorization or syntax matching. Here are three representative task items: [Item 1: Syllogism grid deduction; Item 2: Variable binding text puzzle; Item 3: Constraint satisfaction problem]. Please analyze these inputs and return an overall validity verdict along with supporting evidence across the key validity dimensions.

What was checked, without and with the skill
Checked	Without skill	With skill
Includes an evaluation of convergent validity checking correlation with other established multi-step reasoning benchmarks	Missed	Passed
Identifies construct-irrelevant variance such as pattern memorization or syntax clues	Passed	Passed
Provides an overall validity verdict based on evidence across the validity dimensions	Passed	Passed

case-01 · gemini-3.6-flash · 8/22/2026 · answers are capped and redacted

What's inside

The instructions your model reads when the skill loads.

SKILL.md34 linessingle file

Construct Validity Assessment SOP

Evaluate whether a benchmark actually measures the capability it claims to measure, using psychometric validity frameworks adapted for AI evaluation.

Input

benchmark_name: Name of the benchmark
claimed_capability: What the benchmark authors claim it measures
task_examples: Representative examples from the benchmark

Procedure

Define the construct (claimed capability) precisely
Analyze task requirements — what skills are actually needed to solve examples?
Assess content validity — do items representatively sample the construct?
Check convergent validity — correlation with other measures of same construct
Check discriminant validity — independence from unrelated constructs
Identify construct-irrelevant variance (confounds)

Output

Validity verdict with evidence for each validity dimension.

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

SOP	When to use
spawn-agent	Spawn a customized CC subagent with full MCP tool access. Used by SOPs that declare execution: subagent.

Track record

Benchmark

21 / 23

test cases passed, vs 13 / 23 without

Adoption

Users: 0
Times used: 0

Safety and source details

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/3/2026

View 15 scanner checks

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

yogsoth-ai / de-anthropocentric-research-engine

skills/construct-validity-assessment/SKILL.md

View on GitHub ↗

License Apache-2.0Imported 8/3/2026Version v1

More skills from the same source and nearby in the registry, with their headline benchmark lift.

Ready to supercharge your AI agents?

See it in action

What's inside

Construct Validity Assessment SOP

Input

Procedure

Output

Available SOPs

Track record

Benchmark

Adoption

SkillSafety

Source

Related skills