AI selection: technologies, data, algorithms

The starting lens for the whole "scientific, legal, and ethical concerns" framework. Before asking "is this AI tool any good?", decompose it into three independent parts and evaluate each separately (the "modular approach," Lievens & Sackett, 2017). Conflating them is the root error behind most overclaiming.

The three parts

1. Technologies — independent of the constructs measured

Examples: online games, video interviews, social media, gamification, VR. Technologies are independent from the constructs being measured and should not be confused with them (Arthur & Villado, 2008; Campbell & Fiske, 1959). Most formats can measure a wide range of constructs, and most constructs can be assessed by many technologies.

Consequence (state this in any evaluation): a new technology cannot be said to be universally valid. "Our video-interview platform is validated" is a category error — validity is a property of the inferences about the constructs measured in a specific use, not of the technology. Demand evidence about which job-relevant constructs are measured, which then informs validity and fairness.

2. Data — a continuum from intentional to incidental

Data vary along a continuum (Oswald, 2020):

• Intentional — traditional, controllable responses to a prompt (a test item, an interview

answer).

• Incidental — less intentional/controllable, requiring little or no applicant effort (and

sometimes little applicant control): social-media posts, facial movements, voice characteristics in a video, mouse clicks, response times.

Less obtrusive technologies tend to collect more incidental data, in massive amounts (game data = every click/decision/scenario; video = continuous voice and facial features; "big data" pulled from resumes, emails, social media). The more incidental the data, the more the concerns about job relevance, control, consent, and fairness intensify (see ai-candidate-data-control).

3. Algorithms — AI ⊃ ML ⊃ deep learning

• AI — broad term for computer procedures that mimic human decisions/processes/outcomes closely

enough to appear intelligent.

• Machine learning (ML) — subset: the mathematical/statistical procedures underlying these tools.
• Deep learning — subset of ML: neural-network-based.
• Supervised learning predicts a criterion (the criterion "supervises" how predictors are

used — e.g., predicting supervisory performance ratings); unsupervised learning groups people/cases into clusters with no criterion (e.g., applicants like/unlike high performers).

The "learning" happens when algorithms are first exposed to a training set; the model is judged on an independent test set (a hold-out sample, k-fold folds, or newly collected data).

A key claim to carry forward

There are hundreds of ML algorithms, and different algorithms often make highly similar predictions with similar overall accuracy (Domingos, 2012). So in personnel selection, the effectiveness of ML prediction/clustering is more likely driven by the availability of high-quality data than by which ML algorithm is chosen. Whether the advantages of a large number of predictors offset the disadvantages of "messy" data must be determined case by case. Accurate, well-justified predictions depend on good measurement processes and good data — not on algorithm sophistication. Don't let a vendor's algorithm story distract from data-quality and construct questions.

How to use this skill

1. Name the three parts of the tool under review (which technology, what data on the

intentional–incidental continuum, which algorithm type — supervised/unsupervised).

1. Reject "the technology is valid" framing; redirect to what constructs are measured and is that

measurement job-relevant, reliable, valid, and fair?

1. Locate the data on the continuum — the more incidental, the more you escalate the

control/consent/fairness concerns.

1. Treat algorithm choice as secondary to data quality; probe the data-generation and

construct-relevance story.

1. Proceed to the legal frame (ai-selection-legal-landscape) and the 11 concerns.

Pitfalls

• Accepting "this technology is validated" as if validity were a property of the medium.
• Being dazzled by the algorithm while ignoring data quality and construct relevance.
• Missing that incidental data (face, voice, social media) drives the hardest legal/ethical issues.
• Assuming supervised = good and unsupervised = bad (each has distinct evaluation needs).

Checklist

• ] Technology, data type, and algorithm type named and evaluated separately
• ] "Universally valid technology" claim rejected; construct/inference framing substituted
• ] Data located on the intentional–incidental continuum; control/consent flags raised if incidental
• ] Supervised vs. unsupervised identified; training/test-set basis confirmed
• ] Data-quality story probed as primary; algorithm choice treated as secondary

See also

ai-selection-legal-landscape · all 11 concern skills · ai-validity-evidence · ai-input-data-and-design-audit (the audit counterpart) · validation-planning

Source: Tippins, Oswald & McPhail (2021), "New Forms of Assessment."