Receptive fields as prediction devices: A comparison of cell tuning to single images and to natural image sequences in temporal cortex

We experience the world as a continuous stream of events where the previous scenes help us anticipate and interpret the current scene. Visual studies, however, typically focus on the processing of individual images presented without context. To understand how processing of isolated images relates to processing of continuously changing scenes, we compared cell responses in the macaque temporal cortex to single images (of faces and hands) with responses to the same images occurring in pairs or sequences during actions. We found two phenomena affecting the responses to image pairs and sequences: (a) temporal summation, whereby responses to inputs from successive images add together, and (b) 'forward masking', where the response to one image diminishes the response to subsequent images. Masking was maximal with visually similar images and decayed over 500 ms. Masking reflects interactions between cells rather than adaptation of individual cells following heightened activity. A cell's 'receptive field' can be defined by tuning to isolated stimuli that vary along one dimension (eg position or head view). Typically, this is a bell-shaped curve. When stimuli change continuously over time (eg head rotation through different views), summation and masking skew the tuning. The first detectable response to view sequences occurs 25 ms earlier than for corresponding isolated views. Moreover, the responses to sequences peak before the most effective solitary image: the peak shift is ~1/2 the bandwidth of tuning to isolated stimuli. These changes result in activity across cells tuned to different views of the head that 'anticipate' future views in the sequence: at any moment the maximum activity is found in those cells tuned to images that are about to occur. We note that, when sensory inputs change along any dimension, summation and masking transform classical receptive field properties of cells tuned to that dimension such that they predict imminent sensations.