A comparison in five diverse countries of commonly used consumer survey question formats: Check-All-That-Apply (CATA), Check-All-Statements (CAS), Rate-All-That-Apply (RATA), and Rate-All-Statements (RATING)

Abstract

Recent rapid advances in information technology and improved access to affordable internet across almost all countries have contributed to the increased use of online or web surveys in consumer research. Online surveys are a fast, low-cost, and far-reaching option for collecting responses from consumers. However, similar to other survey methods, online surveys are characterized by survey errors such as coverage error, sampling error, nonresponse error, and measurement error which influence the quality of response data. This research focused on measurement error and in particular the effect of question formats on consumers’ online survey responses. Some authors have pointed out problems with using certain question formats. Therefore, the purpose of the current writing was to gain more understanding and differentiation in the characteristics of response data that are collected using four question formats that are commonly used in online surveys. The four question formats that were investigated included: the Check-All-That-Apply (CATA), Check-All-Statements (CAS) (yes/no), Rate-All-That-Apply (RATA), and Rate-All-Statements (RAS) or simply RATING. With CATA, respondents select all terms or statements that apply from a given list, while, with CAS, respondents must respond (e.g., yes/no or agree/disagree) to each term or statement to show that it applies or does not apply. For RATA, consumers select all terms or statements that apply from a given list and then continue to rate those selected based on how much they apply. With Rate-All-Statements (RATING), a widely used standard format for testing, consumers are asked to rate all terms or statements according to how much they apply. Consumer motivations for eating items that belong to five food categories (dairy, desserts, fruits, protein-rich, starch-rich foods) were assessed in this study. Also, survey versions for each of the four question formats were fielded in five countries: Unites States of America (in English),India (in Hindi or English), China (in Mandarin Chinese – Simplified), Brazil (in Portuguese), and Spain (in Spanish). Results showed that fewer “apply” responses were noted and lesser discrimination occurred among attributes (in this case eating motivations) or product categories when CATA was used instead of CAS across all five countries. However, fewer incomplete responses were collected and respondents’ liking of the survey experience was higher when CATA was used. Similarly, fewer “apply” responses were found and lesser discrimination occurred among attributes or product categories when RATA was used instead of RATING across all five countries. However similar to CATA, when RATA was used, mean scores for attributes were higher, there were fewer incomplete responses, and the survey experience was liked more. Results from comparisons of demographic impacts showed that CAS questions were more discriminating among age groups and between genders than CATA. Also, although there were several similarities in associations between age group and eating motivations and between gender and eating motivations for CAS and CATA response data, some differences in the associations were found and they were inconsistent. Similarly, discrimination among age groups and between genders was higher when the RATING question format was used instead of RATA. Also, mean scores for eating motivations for age groups and gender were more consistent when RATING was used. The findings of this research suggest that while all four question formats can be used to collect “big picture information”, CAS and RATING question formats are better suited for surveys that seek more detailed responses from the consumers. Also, for consumer researchers such as product developers, and sensory scientists, CAS and RATING were found to be more discriminating among product attributes (and among products for RATING). However, surveys with CAS and RATING questions could likely cost researchers more because they require more time to complete than corresponding surveys with CATA and RATA questions respectively.