Author Information
Jordan Epistola, University of Maryland, United StatesPaul Hanges, University of Maryland, United States
Tiffany Hansbrough, Binghamton University, United States
Abstract
The integration of machine learning (ML) into behavioral research offers promise for advancing theory by capturing complex patterns traditional methods often overlook. However, critiques (Vowels, 2023) caution against the use of ML for theory-building, as such methods can obscure causality and misrepresent theoretical relationships. This study addressed these concerns by comparing Hierarchical Linear Modeling (HLM), a theory-driven approach, and Multilevel Logistic Lasso Regularization (MLLR), a regularization-based ML technique, in predicting episodic ("Remember") versus semantic ("Know") memory judgments in leadership assessments.
Using a multilevel dataset comprising 11,833 observations nested within 614 participants, the study evaluated predictive accuracy, coefficient stability, and the broader implications of these methods for theory-testing in organizational research. HLM consistently outperformed MLLR across key metrics, including AUC, sensitivity, specificity, and balanced accuracy, affirming its utility for theory-driven prediction. Both approaches identified key predictors such as item sentiment and leader liking with consistent directional effects, but MLLR penalized theoretically significant predictors aggressively. For example, item concreteness, the strongest predictor per HLM, and reading level, which was small but significant, were attenuated under MLLR.
These findings highlight the limitations of ML methods in multilevel research, marking one of the first comparisons of MLLR and HLM in organizational science. While MLLR shows potential for exploratory analyses and abductive research, its limitations highlight the need for advancements in computational psychometrics to better address multilevel data. Conversely, HLM’s superior performance and interpretability reinforce the value of theory-driven approaches for both theory-testing and prediction, reinforcing critiques of overreliance on ML in social science research.
Comments
Powered by WP LinkPress