---
name: openmatter-network/generalizing-validity-evidence
source: https://app.decimal.ai/s/openmatter-network-generalizing-validity-evidence@1/SKILL.md
source_sha256: 293187c3f5f0
---

# Generalizing validity evidence

When accumulated evidence is strong enough, you may justify a procedure in a **new setting without a
local criterion study** — by demonstrating generalized validity *and* making a compelling argument
for **direct applicability** to your situation. Three strategies (not mutually exclusive, not
exhaustive): **transportability**, **synthetic/job-component validity**, and **meta-analytic
validity generalization**. All require an **analysis of the work** to establish the link.

## 1. Transportability

Apply a specific procedure in a new situation based on a validation study conducted **elsewhere**,
when key similarities make that evidence applicable.

- Carefully **review the original study** for technical soundness and conceptual/empirical relevance
  to the new situation.
- Compare on **job content, job requirements, job context, and (if feasible) the applicant group**.
- Document the comparison between the original validation sample/setting and your target use.

## 2. Synthetic validity / job-component validity

Justify use based on the validity of inferences for **one or more components (job components)** of
the work, established for those components, then "synthesized" (empirically combined) for a given job
or job family.

- Requires a **detailed analysis of work** that decomposes jobs into components and identifies which
  components a job comprises.
- Powerful when **many jobs share common components** — provides a validity source where a separate
  criterion study per job isn't feasible, and reduces burdensome data collection.
- Caveat: job requirements **unique** to a job may not be well covered by components and may need
  other evidence.

## 3. Meta-analytic validity generalization (VG)

Cumulate validity findings across studies to estimate the predictor–criterion relationship for the
domains/settings of interest, and to determine how specific/generalizable the relationship is.

- VG shows that much of the observed variation in validity across settings is due to **statistical
  artifacts** (sampling error, range restriction, criterion unreliability). Well established for
  **cognitive ability**; accruing for several **noncognitive** measures.
- **Organize around constructs.** Generalization is straightforward when results are organized by
  **predictor and criterion constructs**. Method labels alone (e.g., "interview," "SJT," "biodata")
  are **not** constructs — a method can be designed to assess very different constructs, so you can
  only generalize to applications of the method that share the relevant features (content, scoring,
  meaning of scores).
- **Evaluate the meta-analysis itself:** methods and assumptions, their tenability, statistical
  artifacts that could bias results, and **moderators**. When a substantive moderator is plausible,
  consider statistical power and the precision of reported effects to detect it. Insist on full
  reporting of how studies were categorized and analyzed; missing/unreported information undermines
  the inference.

## Local study vs. generalized evidence

Generalized evidence is often **more useful** than a single small local study. **But** a competent
local study with a **large, organizationally relevant sample** using the same test for the same work
can be **more accurate and informative** than an accumulation of small, heterogeneous, or deficient
studies that don't represent your setting. Watch for **representation gaps**: if your setting (e.g.,
a managerial job) isn't represented in the meta-analytic database (e.g., limited to entry-level
jobs), a local study may be more relevant. A **Bayesian approach** can formally combine meta-analytic
priors with locally estimated coefficients (Newman, Jacobs, & Bartram, 2007).

Sole reliance on cumulative evidence may also be **insufficient for operational needs** (e.g.,
optimal placement, combining procedures in a broader system) — supplementary local or cooperative
studies may still be warranted.

## When generalization is NOT warranted

- Generalizing from a method studied for one construct to the same method used for a different
  construct (e.g., technical-knowledge interviews → interpersonal-skill interviews).
- Applying meta-analytic results from one set of procedures/settings to a new setting using
  different, unspecified procedures.
- Assuming predictor/criterion measures sharing a construct *label* are interchangeable without
  rational and empirical support.

## Pitfalls

- Skipping the **analysis of work** that links the borrowed evidence to your job(s) — required for all
  three strategies.
- Organizing meta-analytic evidence by **method label** ("interview," "SJT") instead of by construct.
- Treating generalized evidence as automatically superior, ignoring **representation gaps** (your
  setting absent from the meta-analytic database) where a local study would be more relevant.
- Transporting a study without reviewing its technical soundness and comparing job content,
  requirements, context, and applicant group.
- Relying solely on cumulative evidence when operational needs (placement, combining procedures)
  require local data.

## Checklist

- [ ] Strategy chosen (transport / synthetic / VG) with rationale
- [ ] **Analysis of work** links the borrowed evidence to your job(s)
- [ ] Transport: original study reviewed; content/requirements/context/applicant comparison documented
- [ ] Synthetic: job decomposed into components; component validities established
- [ ] VG: evidence organized by **constructs**, not method labels
- [ ] Meta-analysis methods, assumptions, artifacts, and moderators evaluated
- [ ] Representation gaps and operational-need gaps assessed; local/Bayesian supplement considered
- [ ] Direct-applicability argument to the current setting articulated

## See also

`validation-planning` · `work-analysis` (required link) · `criterion-related-validation`
(local alternative/supplement) · `fairness-and-bias-analysis` · `technical-validation-report`

*Source: Principles (5th ed., 2018), "Generalizing Validity Evidence."*