---
name: openmatter-network/content-based-validation
source: https://app.decimal.ai/s/openmatter-network-content-based-validation@1/SKILL.md
source_sha256: 3a6fa0b4d3b3
---

# Content-based validation

Evidence that **the content of a selection procedure is a representative sample of important work
behaviors, activities, and/or worker KSAOs** defined by an analysis of work. The inference rests on
the **fidelity of the procedure's content to the work** — not on an empirical predictor–criterion
correlation. Often the strategy of choice when a job is unique, samples are too small for a
criterion study, or the procedure is built directly from the work (e.g., work samples, job-knowledge
tests).

## When content evidence is and isn't the right tool

- Strong fit when procedure content is **closely linked to job content**.
- A construct labeled an *ability or personality* characteristic does **not** by itself rule out a
  content strategy — but when content is **less clearly linked to job content**, other sources of
  evidence take precedence (e.g., criterion-related, internal structure).
- Content evidence supports **construct interpretation** and generalization to job performance/
  training when the procedure samples the work well.

## SMEs define the domain — qualify them

The study's success hinges on SME qualifications. SMEs:
- define the work domain (important behaviors, activities, worker KSAOs),
- participate in the analysis of work, and
- judge the link between selection-procedure content and work/worker requirements.

SMEs must be **competent to perform the work** and knowledgeable about relevant organizational
characteristics (shift, location, equipment, software). For behavior definition, include people
fully knowledgeable about the work. **Standardize and document** the method for translating SME
judgments into the procedure; if SME ratings evaluate the content–work match, standardize the rating
procedures and criteria for each aspect.

## Process

1. **Confirm feasibility.** Are the work and worker requirements reasonably **stable**? Can
   irrelevant content be minimized? Are qualified, unbiased SMEs available? Do cost/time allow the
   needed fidelity?
2. **Define the content domain** from accurate, thorough work information (behaviors, activities,
   responsibilities, prerequisite KSAOs). The domain need **not** include everything done on the
   job. State which important elements are included, how each links to the procedure, and **why any
   parts were excluded**. Ground the definition in organizational needs, labor markets, and
   specifications.
3. **Decide what the candidate must already have vs. will be trained.** Base the procedure on the
   analysis that defines the **balance** between KSAOs expected at entry and training the
   organization provides. (Read-the-manual example: if employees are trained on the technical
   manuals, test *reading ability* to the required level — not manual-specific knowledge.)
4. **Develop or choose the procedure; sample the domain.** Random sampling of the domain is usually
   neither feasible nor appropriate. Instead, sample the **most important** behaviors/KSAOs or those
   **prerequisite** to others, or a smaller set predictive of critical outcomes. Document the
   rationale in a **test plan** specifying which KSAOs are measured by which methods.
5. **Set the level of specificity deliberately.** More fidelity to exact job components → stronger
   content evidence, but **less resilience to change** and less transferability. If the work is
   likely to change, build **less job-specific** content (e.g., software *principles/techniques*
   rather than one program). Measure skills and knowledge that are job-relevant, never irrelevant
   content, even when a full simulation isn't feasible (a single-step problem may substitute for a
   multistep work sample).
6. **Establish administration and scoring guidelines** in advance — especially when the procedure
   includes work behaviors or activities that raise administration/scoring challenges.
7. **Evaluate the content evidence** and document everything.

## Evaluating content evidence

Content evidence rests on showing the procedure **adequately samples and is linked to** the
important work behaviors/activities/KSAOs from the work analysis. The documented development methods
**are** the primary evidence. The sufficiency of the match is a matter of **professional judgment**
based on evidence collected in the effort. Determine **reliability** of performance on the procedure
when feasible, choosing a reliability type that fits the measurement design and intended use (e.g.,
pass/fail vs. rank ordering).

## Experience and training requirements as content

A procedure may include evidence of specific prior training, experience, or achievement — judged by
the **relationship between the content of that experience and the content of the work**. Superficial
resemblance is not enough: course titles and job titles may poorly indicate actual content or
proficiency. Evaluate the **similarity of behaviors/activities/processes/KSAOs**, not labels.

## Competency modeling

A competency model can found a content study **only if it is detailed and rigorous** — as rigorous
as a traditional work-analysis project. Judge that rigor before relying on it. See `work-analysis`.

## Pitfalls

- Including content not part of the work domain (standardization artifacts aside) or, worse,
  irrelevant content.
- Over-specifying content for work that is about to change.
- Resting on a thin competency model.
- Treating face validity as content evidence (acceptability ≠ validity).
- Failing to document *why* domain elements were excluded.

## Checklist

- [ ] Work/worker requirements reasonably stable; feasibility confirmed
- [ ] Content domain defined from thorough work information
- [ ] Entry-vs.-trained KSAOs distinguished
- [ ] Sampling rationale (importance/prerequisite) documented in a test plan
- [ ] Specificity level chosen against likelihood of work change
- [ ] SMEs qualified; linkage method standardized and documented
- [ ] Administration/scoring guidelines set in advance
- [ ] Reliability assessed where feasible
- [ ] Excluded domain elements and their rationale recorded

## See also

`work-analysis` (defines the domain) · `validation-planning` ·
`internal-structure-validation` · `generalizing-validity-evidence` ·
`technical-validation-report`

*Source: Principles (5th ed., 2018), "Sources of Validity Evidence → Evidence for Validity Based on
Content."*