Choice-Level Analysis

Why Choice-Level Analysis?

💡 Choice-level analysis is simpler, easier, and more powerful than profile-level analysis.

🧠 Conjoint designs began in market research and psychology, where respondents rated two profiles (e.g., products). Each rating was treated as an independent observation, creating the profile-level design with 2 × n rows for n respondents.
🔍 Social scientists later used conjoint designs for choices instead of ratings but kept the same structure with a single binary outcome. This shift introduced statistical and conceptual issues.

⚠️ Problems with Profile-Level Analysis

🚫 Profile-level analysis forces researchers to correct a dependence that they created themselves.

🔁 Redundant structure: Each choice task generates two rows per task for each respondent even though there is only one independent choice.
🔗 Mechanical dependence: Selecting one profile necessarily implies rejecting the other, violating the independence assumption.
🧩 Artificial complexity: Analysts must correct for this dependence using complicated statistical adjustments, even though it arises solely from how the data were organized—not from respondents’ behavior.

✅ Advantages of Choice-Level Analysis

In contrast, choice-level analysis organizes data by respondent decisions rather than profiles.

📋 Each unit of observation represents a choice for a given task per respondent.
⚡ The outcome variable reflects which profile was chosen in that task.
🧠 This data structure allows researchers to model the choice conditional on the full comparison.

🎯 Choice-level analysis directly models the respondent’s decision between two (or more) alternatives,
capturing the true structure of the conjoint task.

Key Issues and Applications

Profile-level estimands like AMCEs assume that each profile is generated independently and ignores how respondents evaluate one profile relative to another. This limits the types of questions researchers can ask.
Choice-level analysis allows researchers to explore questions that explicitly depend on the comparison between profiles, such as:

Examples of Choice-Level Research Questions

🗳️ Do voters choose a white candidate over a non-white candidate?
(The levels—white vs. Asian, Black, Hispanic—always differ between profiles.)
🌐 Do Asian Democrat respondents prefer an Asian Republican over a white Democrat?
(Profiles are intentionally designed with multiple correlated attributes.)
📊 Do voters care about electability?
(The two percentages representing win probability must sum to 100.)
⚖️ Do voters prefer the status quo over a policy proposal?
(One profile is fixed while the other varies across tasks.)
🧭 How much do voters prefer extreme left-leaning or extreme right-leaning policies?
(Attributes are consistently positioned on the ideological spectrum.)

Furthermore, when individuals compare profiles side-by-side, their evaluations are often psychologically influenced by the alternative, such as through assimilation or contrast effects
(see Horiuchi and Johnson 2025).

Why Move to Choice-Level Analysis?

🔍 Choice-level analysis models the decision between two profiles, not the evaluation of a single profile.

This structure more closely mirrors:

🧠 Real-world decision-making, where people choose among competing alternatives.
🔄 Comparative cognition, where evaluations depend on the context of available options.
🎛️ Tradeoff reasoning, where respondents weigh attribute combinations jointly.

Hence, rather than estimating the probability of selecting an isolated profile, choice-level analysis estimates the probability of choosing one profile over another, conditional on all attributes involved.

✅ Mirrors real-world behavior
✅ Captures comparative judgment and psychological context
✅ Reveals authentic tradeoffs and priorities

Summary

Profile-Level Analysis	Choice-Level Analysis
Treats profiles as independent	Models the decision between profiles
Ignores comparative context	Captures mutual influence of options
May blur or bias tradeoffs	Highlights actual tradeoffs
Can misstate uncertainty	Produces more interpretable estimates
Requires complex correction methods	Works with simple and transparent models

Key Takeaway

🚀 If your conjoint design presents respondents with two or more profiles for comparison,
then choice-level analysis is essential for valid, interpretable, and psychologically realistic inference.

It provides:

Deeper insights into human decision-making
Cleaner estimation procedures
Closer correspondence to real-world behavior

📚 References

Clayton, Horiuchi, Kaufman, King, & Komisarchik (Forthcoming).
Correcting Measurement Error Bias in Conjoint Survey Experiments.
Forthcoming, American Journal of Political Science.
Preprint available
Horiuchi & Johnson (2025).
Advancing Conjoint Analysis: Delving Further into Profile Comparisons.
Work in progress.