Neural mechanisms underlying the influence of sequential predictions on scene gist recognition

Abstract

Rapid scene categorization is typically argued to be a purely feed-forward process. Yet, when navigating in our environment, we usually see predictable sequences of scene categories (e.g., offices followed by hallways, parking lots followed by sidewalks, etc.). Previous work showed that scenes are easier to categorize when they are shown in ecologically valid, predictable sequences compared to when they are shown in randomized sequences (Smith & Loschky, 2019). Given the number of stages involved in constructing a scene representation, we asked a novel research question: when in the time course of scene processing do sequential predictions begin to facilitate scene categorization? We addressed this question by measuring the temporal dynamics of scene categorization with electroencephalography. Participants saw scenes in either spatiotemporally coherent sequences (first-person viewpoint of navigating, from, say, an office to a classroom) or their randomized versions. Participants saw 10 scenes, presented in rapid serial visual presentation (RSVP), on each trial, while we recorded their visually event related potentials (vERPs). They categorized 1 of the 10 scenes from an 8 alternative forced choice (AFC) array of scene category labels. We first compared event related potentials evoked by scenes in coherent and randomized sequences. In a subsequent, more detailed analysis, we constructed scene category decoders based on the temporally resolved neural activity. Using confusion matrices, we tracked how well the pattern of errors from neural decoders explain the behavioral responses over time and compared this ability when scenes were shown in coherent or randomized sequences. We found reduced vERP amplitudes for targets in coherent sequences roughly 150 milliseconds after scene onset, when vERPs first index rapid scene categorization, and during the N400 component, suggesting a reduced semantic integration cost in coherent sequences. Critically, we also found that confusions made by neural decoders and human responses correlate more strongly in coherent sequences, beginning around 100 milliseconds. Taken together, these results suggest that predictions of upcoming scene categories influence even the earliest stages of scene processing, affecting both the extraction of visual properties and meaning.