Validating the User-Centered Hybrid Assessment Tool (User-CHAT): a comparative usability evaluation

Abstract

Usability practitioners need effective usability assessment techniques in order to facilitate development of usable consumer products. Many usability evaluation methods have been promoted as the ideal. Few, however, fulfill expectations concerning effectiveness. Additionally, lack of empirical data forces usability practitioners to rely on personal judgments and/or anecdotal statements when deciding upon which usability method best suits their needs. Therefore the present study had two principal objectives: (1) to validate a hybrid usability technique that identifies important and ignores inconsequential usability problems, and (2) to provide empirical performance data for several usability protocols on a variety of contemporary comparative metrics. The User-Centered Hybrid Assessment Tool (User-CHAT) was developed to maximize efficient diagnosis of usability issues from a behaviorally-based perspective while minimizing time and resource limitations typically associated with usability assessment environments. Several characteristics of user-testing, the heuristic evaluation, and the cognitive walkthrough were combined to create the User-CHAT. Prior research has demonstrated that the User-CHAT supports an evaluation within 3-4 hrs, can be used by individuals with limited human factors / usability background, and requires little training to be used competently, even for complex systems. A state-of-the-art suite of avionics displays and a series of benchmark tasks provided the context where the User-CHAT’s performance was measured relative to its parent usability methods. Two techniques generated comparison lists of usability problems – user-testing data and various inclusion criteria for usability problems identified by the User-CHAT, heuristic evaluation, and cognitive walkthrough. Overall the results demonstrated that the User-CHAT attained higher effectiveness scores than the heuristic evaluation and cognitive walkthrough, suggesting that it helped evaluators identify many usability problems that actually impact users, i.e., higher thoroughness, while attenuating time and effort on issues that were not important, i.e., higher validity. Furthermore, the User-CHAT had the greatest proportion of usability problems that were rated as serious, i.e., usability issues that hinder performance and compromise safety. The User-CHAT’s performance suggests that it is an appropriate usability technique to implement into the product development lifecycle. Limitations and future research directions are discussed.