Recognizing the setting before reporting the action: investigating how visual events are mentally constructed from scene images

Abstract

While watching a film, the viewer begins to construct mental representations of it, which are called events. During the opening scene of a film, the viewer is presented with two distinct pieces of information that can be used to construct the event, namely the setting and an action by the main character. But, which of these two constructs are first cognitively represented by the viewer? Experiment 1 examined the time-course of basic level action categorization with superordinate and basic level scene categorization using masking. The results indicated that categorization occurred in a course-to-fine manner, inconsistent with Rosch et al.’s (1976) basic level theory. Interestingly, basic level action categorization performance did not reach ceiling when it was processed for a 367 ms SOA, suggesting that additional scene information and processing time were required. Thus, Experiment 2 examined scene and action categorization performance over multiple fixations, and the scene information that was fixated for each categorization task. Both superordinate and basic level scene categorization required only a single fixation to reach ceiling performance, inconsistent with basic level primacy, whereas basic level action categorization took two to three fixations, and led to more object fixations than in either scene categorization task. Eye movements showed evidence of a person bias across all three categorization tasks. Additionally, the categorization task did produce differences in the scene information that was fixated (Yarbus, 1967). However, could basic level theory still be correct when subjects are given a different task? When the same scene images were named, basic level action terms were used more often than basic level scene category terms, while superordinate level action terms were used relatively less often, and superordinate level scene category terms were hardly ever used. This shows that linguistic categorization (naming) is sensitive to informative, middle-level categories, whereas early perceptual categorization makes use of coarse high level distinctions. Additionally, the early perceptual advantage for scene categorization over basic level action categorization suggests that the scene category is the first construct that is used to represent events in scene images, and maybe even events in visual narratives like film.