Developing Deep Neural Network based Brain Computational Models from Psychophysics Data of some Simple Perceptual Phenomena: Visual as well as Auditory

Abstract

The thesis, consisting of nine chapters, explores the methodology of building testable brain computational models using Deep Neural Networks (DNN), which are trained by psychophysics data. Psychophysics is the quantitative study of perception of physical stimuli. In psychophysics experiments, one or more parameters associated with the stimuli are changed, and the human subject’s responses to the stimuli are recorded. This thesis encompasses both experimental psychophysics works, as well as computational models of the phenomena involved.

The contributory chapters of the thesis start in Chapter 2 with the perspective building of the novel methodology followed throughout this research involving psychophysics on one hand, and deep neural network based brain modeling on the other. This chapter also focuses on the possible reasons for the discrepancies that exist between the functioning of existing deep neural networks, and psychological findings.

The present thesis explores three very simple, yet intriguing perceptual phenomena viz. flicker fusion, flicker wheel illusion and sound symbolism. While the first two are purely based on visual perception, the third, i.e. sound symbolism, as the name suggests, involves both visual and auditory perception. The experiments involved in this thesis concerning all these three perceptual phenomena share one common aspect. Not only do they involve very simple stimuli for conducting the psychological experiments, but also the subject response in all the three cases is binary. For the flicker (a flicker stimulus is a visual stimulus with intermittent illumination) fusion experiment, the subject reports whether the stimulus appears flickering or steady; for the flicker wheel illusion, subjects report whether the wheel is perceived as static or flickering (illusory), while in the sound symbolism experiments, the subject assigns a sound stimuli to one of the two visual inputs, provided.

In Chapter 3 of the thesis, a Convolutional Recurrent Neural Network (CRNN) for modeling the flicker fusion phenomenon has been proposed. It is shown that the model is trainable with psychophysics data, and testable with a wide variety of flicker patterns. Next, in Chapter 4, a DNN model that takes into account the microsaccades in the eye is presented for the Flicker Wheel illusion while also building a novel dataset for this illusion. The sound symbolism phenomena is investigated in Chapter 5, for the difference in words for round and sharp objects across several natural languages. Here again, both behavioral experiments and DNN based modeling are performed.

Thus, by establishing the efficacy of DNN based brain computational models in explaining these psychological phenomena, the present thesis goes on to further investigate the flicker stimulus, already discussed in Chapter 3, to better explore the strengths and limitations of the present brain computational modeling approach.

To this end, first in Chapter 6, the CRNN model is used to probe the relation between the psychophysics and brain electrophysiology involving the flicker stimulus. The work in this chapter, interestingly, demonstrates that many of the reported features of the human electroencephalogram (EEG) response to flicker can actually be explained as being the convolution response to the stimulus, despite the fact that the model is trained with behavioral data only.

Next, to further generate more flicker data, and subsequently put the proposed DNN model to more stringent testing, Chapter 7 of the thesis describes the construction of an indigenous low-cost device that can generate mass psychophysics data on flicker fusion to train and test DNNs. Subsequently, in Chapter 8, the psychophysics data generated from this device was used to train a CRNN. The training yielded symmetric filters, as often found in biological visual systems. The predictions made by the CRNN model on complex flicker patterns were then tested through psychophysics experiments with the device, demonstrating that the model is falsifiable with scopes of further improvement through future research.