We work better together: using an ensemble of natural language processing and forced choice to measure conscientiousness

Abstract

The primary aim of this thesis was to implement a research design which collected forced choice data and text data simultaneously, then examine the psychometric properties of an ensemble model generated from both the forced choice data the text data. Forced choice (FC) and language models offer psychometric advantages, as well as advantages in selection settings when compared to Likert scales – the traditional method of measuring personality traits (Ozer & Benet-Martinez, 2006). The present research provides psychometric evidence of the validity of a multi-method ensemble approach to measuring conscientiousness. Over 45 trials, participants selected one of two statements in a FC block and then provided a short text explanation of their choice. The FC data was scored using the Generalized Thurstonian Unfolding Model (Zhang et al., 2023). The text data was embedded using pre-trained BERT-base-uncased (Devlin et al., 2018) and trained to predict conscientiousness scores generated from the Chernyshenko Conscientiousness Scale (CCS; Chernyshenko, 2002) using ridge regression. FC scores and Natural Language Processing (NLP) scores were ensembled using linear regression to predict scores generated from the CCS. Construct validity was evaluated using a Multi-Trait Multi-Method table comparing the CCS scores, the NLP-derived scores, the forced choice scores, and the ensemble model scores. Incremental validity of the ensemble model above and beyond the other models was evaluated through two hierarchical regressions predicting GPA and subjective well-being. The ensemble model yielded good criterion-related validity and convergent validity, acceptable discriminant validity, but poor reliability – likely due to unstable forced choice estimates. The ensemble model did not demonstrate incremental validity above and beyond the other estimates.