Sensitivity to higher-order statistics in natural scenes

Visual processing and object recognition are among the most difficult problems facing computer vision today, yet these are also the tasks at which our brain excels without any conscious effort. What are the computations performed by successive neural processing layers, after the light signals encoded by the retina travel through LGN to the primary visual cortex and higher areas, such as V2, V4 and IT? Neuroscience has traditionally approached this question from two directions. In the first, experimentally driven approach, recordings are performed on single neurons in the visual pathway while they respond to simple and controlled visual stimuli, e.g. randomly flickering light or drifting gratings. The second approach is computational: starting with the statistical properties of natural images, we ask how a neural system should process images optimally, and then compare these theoretical predictions to the observed phenomenology. Only a limited amount of work, mostly in early vision, has been done at the interface of both approaches, by using synthetic yet natural-like stimuli to probe the visual system. This presents a real problem: as we progress along the visual pathway, neurons respond less and less to simple stimuli often used in the experiments, but we lack the ability to generate rich, natural-like, and controlled stimuli to use instead. Our ability to infer how natural vision processes shapes, texture and spatio-temporal structure in real environments is thus severely limited. Here we propose to address three important aspects of this problem: First, we will study higher-order statistical structure in natural scenes. Much is known about luminance histograms and spectra (1st / 2nd order statistics), and in the past decade higher-order structure of "oriented edges" has been modeled probabilistically. Going further, we will sample and build statistical models of binary image patches and contour fragments, the building blocks of closed contours, which are thought to drive responses beyond V1. Second, we will design new statistical inference techniques to learn models of neural behavior from experimental recordings when statistically complex stimuli are used. Specifically, we will extend the general linearized model (GLM) and maximally informative dimensions (MID) approaches to extract nonlinear stimulus features that the neurons are sensitive to, and study retinal encoding using new differential reverse correlation. Third, in collaboration with experimental labs with which we have strong ties, we will design and use naturalistic yet parametrizable stimuli and the related inference techniques, to probe the behavior of both retinal ganglion cells and neurons in higher areas of the visual cortex. In sum, our goal is to address three key interlocking questions about vision: What are higher-order statistical features of natural scenes relevant for biological vision? How can we systematically probe the visual pathways for sensitivity to such features? Can the "efficient coding hypothesis" be extended to higher-order statistics and to the central visual processing?

this project, we studied how complex visual stimuli are processed by the retina and beyond. We report three sets of interrelated results, each advancing significantly the state of the field. First, in collaboration with our experimental colleagues, we studied the simultaneous patterns of activity generated by retinal ganglion cells, the output cells of the retina which send the visual information to the brain, as the retina is stimulated by naturalistic movies. We confirmed two previously proposed hypotheses: that the code of retinal neurons is highly collective, with the activity patterns organized into clusters or modes that correspond well to external stimuli; and that the code exhibits criticality, a special property formally defined within statistical physics, at which the code capacity may be maximized. We also devised mathematical probabilistic models that fully describe these properties and showed that they provide an excellent account of the retinal activity. In sum, this work demonstrated the importance of studying neural behavior in a mathematical framework that can simultaneously capture precisely the behavior of individual neurons as well as the emergent, collective dynamics. This finding is highly relevant for neural codes beyond the retina (and, indeed, stimulated by our work, has been applied in other brain areas), and opens new questions about the functional reasons for such organization. Second, we asked how retinal code can be decoded. In decoding, we are looking for a mathematical model that takes retinal neural responses as an input and predicts the image, or movie sequence, that the retina was looking at. Previously, decoding has only been performed using very simple stimuli. Here, we have demonstrated for the first time successful decoding of a complex, randomly moving stimulus, and in a follow-up work, the first, pixel-by-pixel decoding of a high-dimensional movie. Decoding teaches us how the information can be extracted from the neural code. We observe that while neural responses are strongly nonlinear, the input image can be very well reconstructed through linear operations from the neural outputs. This performance can be improved using more sophisticated non-linear techniques which we also studied extensively. Third, in two theoretical papers we extended substantially the efficient coding theory. Efficient coding postulates that neurons use limited resources so as to transmit most information about the input stimuli, and it is possible to formalize this statement mathematically in the context of information theory. In the first paper, we for the first time formulated, and tested, an efficient coding principle that applies to higher-order statistical structure in images and thus relates to processing beyond the retina, in the visual cortex, and showed that this predicts well human responses on perceptual tests. In the second paper, we provide a mathematical unifying framework that consistently brings together three theories of neural coding (efficient, sparse, and predictive coding). This we view as a major theoretical result that resolves some apparent paradoxes in the field, and extends the neural coding theory to new regimes. This work is relevant for sensory processing in and beyond the retina, and efforts to test theorys predictions are currently under way. In sum, the advances made in this project were possible by departing from the use of traditional, simple stimuli, and studying the retina under richer, more naturalisticbut still experimentally well-controlledstimulation. This has proven to be a powerful approach that we would like to pursue further, and believe that it is very useful also for sensory processing in the cortex.