Conjoint Experiment Diagnostics

Conjoint Experiment Diagnostics

Instructions

Work through each section below for the conjoint study under review. Assess whether the study addresses each item adequately, partially, or not at all. Flag items that pose threats to inference and prioritize recommendations by severity.

Branch on input:

• If a paper or manuscript is provided, proceed through Sections 1–5 sequentially and produce a verdict per section.
• If analysis code or data is provided, verify the actual implementation rather than just what the paper claims: (a) confirm clustering specification, (b) confirm the estimand matches the reported quantity, (c) if IRR is unmeasured, compute within-respondent task-pair agreement as a function of attribute-level differences (Clayton et al. 2023 §3.3 method 2 / projoint).

For neighboring concerns, invoke sibling skills: conjoint-design (design choices), conjoint-cleaning (Qualtrics exports → long format), hypothesis-building (linking estimands to "If-Then" predictions), methods-reporting (full JARS/DA-RT compliance and replication archive), cross-national-design (multi-country / multilingual conjoints).

1. Design Diagnostics

1.1 Attribute and Level Selection

• Are attributes conceptually distinct and non-overlapping?
• Are levels realistic and mutually exclusive within each attribute?
• Is the number of attributes justified? (Bansak et al. 2021 PSRM "Beyond the Breaking Point": response quality is generally robust even at 35 filler attributes, with detectable but modest satisficing — use as a ceiling, not an endorsement of 35 attributes as optimal. Distinct from Bansak et al. 2018 on task counts below.)
• For cross-national or multilingual designs, verify construct equivalence of attribute labels across languages (see cross-national-design).
• Are there "dominant" attributes that might crowd out attention to others?
• Do the attribute levels span a meaningful range of the construct of interest?

1.2 Profile Restrictions

• Are implausible or contradictory attribute combinations allowed?
• If restrictions are imposed, are they documented and justified?
• If no restrictions, does the paper acknowledge potential odd profiles? (Bansak & Jenke 2025: odd profiles have minimal impact on first-order inferences, but should be acknowledged)

1.3 Number of Tasks and Satisficing

• How many tasks per respondent? (Bansak et al. 2018: up to 30 tasks with limited satisficing)
• Are attention checks embedded? Are pass rates reported by task position?
• Is there evidence of survey fatigue or satisficing in later tasks? Look for: (a) response-time distributions by task position (not just means), (b) always-same-side / always-left or always-right rates, (c) random-choice / trembling-hand rates relative to Bansak et al. 2021 baselines, (d) attention-check failure rates by task position.
• Have profile-order, carryover, and fatigue assumptions been tested empirically? (Ham, Imai & Janson 2024 §3.5: CRTConjoint provides conditional randomization tests for all three of Hainmueller-Hopkins-Yamamoto 2014's identifying assumptions.)

1.4 Randomization

• Is attribute-level assignment fully randomized (uniform distribution)?
• If non-uniform, is this justified and documented?
• Are profile-pair attributes independent or dependent? (Clayton et al. 2023 distinguish independent, dependent, and pair-level attributes)