Comprehension of multiple channel messages: Are subtitles more beneficial than soundtracks?

Abstract

Memory processes have undergone extensive investigation using various experimental methods. While working memory studies have profoundly influenced the development of new cognitive theories, including the Dual-Processing Theory (Mayer & Moreno, 1998), limited research has investigated the effects of subtitled messages on multimedia comprehension. Previous eye movement research has investigated the ability of observers to attend to the multiple sensory inputs associated with multimedia viewing (D’Ydewalle, Van Rensbergen & Pollet, 1987; D’Ydewalle & De Bruycker, 2007); however, eye movement research only scratches the surface of cognitive abilities associated with multimedia learning. In order to satisfy the need for more subtitling research two studies were performed to investigate the effects of subtitled movies on the comprehension of movie content. Both investigations involved the presentation of 10-minute movie clips from A Few Good Men and See No Evil, Hear No Evil. Participants completed three types of multiple-choice recognition questions for each movie, including: pictorial-only questions, verbal-only questions and combined-information questions. Experiment 1 was designed to investigate the difference between levels of comprehension, when verbal information was presented only in the participants’ native language (i.e. English soundtrack and/or subtitles). Results of Experiment 1 indicate that participants performed significantly better on verbal-only and combined-information questions when their native language was present in the subtitles as opposed to the soundtrack. These findings confirm previous findings that reading verbal information in subtitles is more efficient than listening to the soundtrack. Comparison of performance on the pictorial-only questions across presentation formats in Experiment 1 showed participants in the English soundtrack with no subtitles condition performed significantly better than all other conditions. Although Experiment 1 provides a basic understanding of how native language soundtracks and subtitles influence comprehension of movies, subtitled media are primarily used when viewing a movie with verbal information from a foreign language. Experiment 2 built on the results of Experiment 1 by incorporating an unfamiliar language (i.e. French). The question sets used in Experiment 1 were also used in Experiment 2; however, two French vocabulary tests were also used in Experiment 2 to measure incidental foreign language acquisition. Consistent with the results of Experiment 1, participants performed significantly better on verbal-only and combined-information questions when their native language was in the subtitles. This finding extended the conclusion that native language verbal information presented visually (i.e. subtitles) yields better performance on questions requiring verbal cues than native language verbal information presented orally (i.e. soundtrack) to foreign language material. Comparison of performance on the pictorial-only questions across presentation formats in Experiment 2 showed no significant differences. Comparison across the two experiments reflected a distraction effect associated with the presence of a foreign language. Performance on the French vocabulary tests was very poor across all conditions and yielded no significant differences, suggesting that the tasks may have been too difficult.