The effects of careless survey responding on the fit of latent variable models: a simulation study

Abstract

Surveys are one of the most popular, useful, and efficient methods for gathering data within both academic and organizational settings. Despite the many benefits afforded by surveys, research shows that a nontrivial number of people engage in careless responding (CR) such that their responses to surveys are not effortful, accurate or valid. This is problematic because the failure to account for CR can distort research findings, result in false theoretical conclusions and lead to precarious workplace decisions. With surveys, it is common to model responses with a latent variable framework and use fit indices to makes conclusions about how well the model represents the data. Research shows that CR can be associated with poor fit, good fit, and/or unrelated to fit. To better understand how CR affects fit, the primary goal of this study was to examine the consequences of CR on the fit of latent variable models using a comprehensive, realistic and rigorous simulation paradigm. A secondary goal was to better elucidate the nature of CR and specify how CR behavior manifests within survey responses. In Study 1, participants’ survey response patterns were experimentally shaped. In Study 2, these results were used as a basis for the primary simulation. A total of 144 conditions (which varied the sample size, number of items, CR prevalence, CR severity, and CR type), two latent variable models (item response theory and confirmatory factor analysis), and six model fit indices were examined. Overall, the results of this study show that CR is frequently associated with deteriorations in model fit. These effects are, however, highly nuanced, variable, and contingent on many factors. Moreover, this study demonstrates that good fit is not necessarily indicative of careful responding nor is poor fit always emblematic of CR. Rather, model fit and CR/response validity are distinct issues that must be separately addressed. These findings can be leveraged by researchers to develop more accurate theories and practitioners to better manage survey data that is laden with CR.