An analysis of the modeling used to determine customer satisfaction

Abstract

Many companies use surveys to establish customer satisfaction metrics. This OEM has been using surveys to analyze customer satisfaction with their products, services, and distribution channel for several decades. Satisfaction metrics are established for the brand, product, and channel partners. The product metric is derived from a question on the survey asking customers how satisfied they are with the product. There are subsequent questions thereafter inquiring about satisfaction with specific functional areas of the product. It is common practice to use Partial Least Squares (PLS) regression analysis to evaluate what impacts the functional area questions have on the overall satisfaction question. The model results are used to understand what areas of the machine should be focused on to improve customers’ experiences with the machine. These results are compared to other data sources such as warranty, field reports, customer focus groups, etc. The results from these models are sometimes questioned based on what common intuition would suggest. Typically the top three drivers to the product metric are understandable, but there are often one or two key areas that do not make logical sense. The objective of this thesis was to understand whether PLS modeling is appropriate given the nature of customer survey data. Models were estimated using existing survey data on a specific model in the tractor product line. PLS models assume data are linear with no bounds. This in itself likely makes this type of model inappropriate for analyzing customer survey data. Responses are bounded on an 11 point scale from 0-10, however, the PLS model being non-bounded assumes there can be a score under 0 or over 10. The model also assumes a linear slope that would indicate each covariate answer 0-10 has the same level of effect on the response variable. This research has found that each covariate answer is in fact non-linear. For example, a customer answering a 2 to quality of manufacturing workmanship has a different impact on the overall satisfaction score than a customer who answers 8. Finally, this research discovered that the PLS models produce negative coefficients of significant value that are not reported to the enterprise. Binary and ordered logistic (logit) models were estimated as an alternative to PLS. Logistic models are non-linear and are commonly used to evaluate bounded data. Response data were separated into two groups based on Net Promoter Score (NPS) Methodology (Reicheld 2006). Using the NPS methodology, 0-6 scores are considered detractors, 7-8 scores are considered passives, and 9-10 scores are considered promoters. The logistic models demonstrate that the top two drivers to customer satisfaction scores are still quality of manufacturing workmanship and reliability/operational availability (similar to results of the PLS model). The unresolved problems question on the survey was included in the models and demonstrated that the predicted probability of a customer being a promoter is much higher in both binary and ordered logit models if no unresolved problems exist. Finally, the model found engine oil consumption remained negative and is statistically significant suggesting that even with the alternative modeling approach there still may be data issues related to the survey. It is recommended that the OEM implement logistic modeling for analyzing customer survey data. It is also recommended that a new survey design be constructed to eliminate issues with correlated data that can lead to spurious and unexplainable results.