---
name: openmatter-network/validation-planning
source: https://app.decimal.ai/s/openmatter-network-validation-planning@1/SKILL.md
source_sha256: c5aa24aa7166
---

# Validation planning

The front end of any selection project. Validation should begin with a **clear statement of the
proposed uses and the intended interpretations and outcomes** — *then* a design is chosen to
support those inferences. Getting this wrong wastes the whole study: a poorly executed effort can
lead an employer to reject a beneficial procedure or adopt an invalid one. "Misleading, poorly
designed validation efforts should not be undertaken."

## What validity means here

Validity is the degree to which evidence and theory support the **interpretation of scores for a
proposed use**. You validate an *inference* (e.g., "scores predict job performance"), not a test.
Plan around the inference you must defend.

The *Standards* name five sources of validity evidence; for employment the first three usually
carry the argument:
1. **Relationships with other variables** (test–criterion) — see `criterion-related-validation`
2. **Content** — see `content-based-validation`
3. **Internal structure** — see `internal-structure-validation`
4. **Response processes** (often irrelevant to employment use)
5. **Consequences** (relevant to validity only when a negative consequence traces to a measurement
   property of the procedure — e.g., bias; mere subgroup-mean differences are *not*, by themselves,
   evidence against validity)

Sources are not competing "types" of validity; they are complementary evidence. Plan to combine
them where the inference is complex, novel, or high-stakes.

## Steps

1. **Define the organization's needs, objectives, and constraints.** Work collaboratively with HR,
   recruiting, legal/compliance, IT, labor relations, and affected stakeholders. Different units
   have competing objectives (rigor vs. applicant flow). Run a cost–benefit lens over candidate
   procedures.
2. **Specify the proposed use(s) precisely.** Target job(s) or job family, candidate pool
   (experienced vs. inexperienced; applicants vs. incumbents), the decision (hire, promote, place,
   certify), and how scores will be used (rank order, cutoff, band). The *use* drives the design.
3. **Characterize the setting.** One homogeneous organization or many? Stable work or rapidly
   changing? Workforce size and applicant availability — these constrain whether a local study is
   even feasible and which strategies (local criterion study, VG, synthetic, content) are open.
4. **Inventory existing evidence.** Is there sufficient accumulated validity evidence (meta-analytic
   bases for cognitive ability and a growing set of noncognitive measures; prior local studies;
   transportable studies) to support the proposed use without new data? Weigh the *informational
   value* of new evidence against its cost. Existing evidence alone may or may not suffice — judge
   the strength of the generalization to your setting. See `generalizing-validity-evidence`.
5. **Apply the requirements of sound inference.** Reliable and relevant measures, representative
   samples, appropriate analyses, controls over plausible confounds, and people qualified for the
   tasks they perform in the study.
6. **Assess feasibility.** Time, cost, sample size/statistical power, organizational disruption,
   union–management relations. Constraints may legitimately narrow the scope of defensible
   generalizations — but never justify a misleading design.
7. **Choose a validation strategy that fits objectives, constraints, and the procedures/criteria.**
   Often more than one source of evidence is valuable. Three illustrative fits:
   - Small population, strong cumulative evidence in similar settings → **validity generalization**.
   - Same job extended from one business unit to another → **transportability**.
   - Position unique to the organization, no external evidence → **content-based** strategy.
8. **Communicate the plan.** Management and workers need the purpose, the research plan, and their
   roles. Decide and communicate confidentiality (e.g., concurrent study data kept out of
   employment decisions). Document the plan.

## Decision guidance: which strategy?

| Situation | Lean toward |
|-----------|-------------|
| Adequate sample, relevant criterion obtainable, need local proof of prediction | criterion-related |
| KSAOs/behaviors sampled directly from work; content closely mirrors the job | content-based |
| Strong external cumulative evidence; small local sample | meta-analytic validity generalization |
| Same job already validated in a comparable unit | transportability |
| Many jobs share common work components | synthetic / job-component validity |
| Multidimensional procedure whose internal structure must be defended | internal structure (as support) |

## Pitfalls

- Designing the study before pinning down the inference and use.
- Treating subgroup-mean differences as automatic evidence against validity (they trigger
  *scrutiny* for bias, not a verdict).
- Underpowered local studies: adequate power while controlling Type I error often needs samples
  that are hard to obtain — consider this *before* committing to a criterion-related design.
- Letting convenience (available criterion, available sample, available software) drive design.
- Promising legal defensibility — the *Principles* address psychological method, not the law.

## Checklist

- [ ] Proposed use(s), target job(s), and candidate pool stated explicitly
- [ ] Inference(s) to be supported written down
- [ ] Existing evidence inventoried; gap (if any) justified
- [ ] Feasibility (power, cost, time, access) assessed honestly
- [ ] Strategy chosen and rationale documented
- [ ] Stakeholders identified; communication & confidentiality plan set
- [ ] Plan documented (feeds `technical-validation-report`)

## See also

`work-analysis` · `criterion-related-validation` · `content-based-validation` ·
`internal-structure-validation` · `generalizing-validity-evidence` · `technical-validation-report`

*Source: Principles (5th ed., 2018), "Overview of the Validation Process" and "Operational
Considerations → Initiating a Validation Effort / Selecting the Validation Strategy."*