Date on Master's Thesis/Doctoral Dissertation

5-2009

Document Type

Doctoral Dissertation

Degree Name

Ph. D.

Department

Psychological and Brain Sciences

Committee Chair

Molfese, Dennis L.

Author's Keywords

Speech perception; Event-related potentials; ERP; N400; McGurk; Multi-sensory; Chinese; Audiovisual; McGurk effect; Audiovisual integration

Subject

Speech perception

Abstract

Previous research has indicated the important role of visual information in the speech perception process. These studies have elucidated the areas of the brain involved in the processing of audiovisual stimuli. The McGurk effect, an audiovisual illusion, has been demonstrated to be a useful tool in the study of audiovisual integration. Brain imaging research suggests that the McGurk effect is modulated by brain structures in the Superior Temporal Gyrus, Supplemental Temporal Cortex and Broadman area 41. Electrophysiological studies suggest that the McGurk effect generates a different brainwave than the standard audiovisual congruent condition in frontal, central and parietal regions among N1, P2, N300 and later ERP components. The magnitude of the McGurk effect is influenced by lexical information, cultural difference and language experience, although these effects have not been investigated by electrophysiological methods. The current study compares three experimental groups, native Chinese speakers who had less than three years of residency in the United States; native Chinese speakers with more than three years of residency in the United States; and native English speakers. A high density (256 channels) hydrocel system was used to collect the ERP data from the participants in three experiments. The main effect of McGurk was studied in Experiment 1, by dubbing an audio /ba/ onto a video /ga/. The behavioral responses and the characteristic brainwave of the McGurk effect were examined. The semantic and culturally related McGurk effects were investigated in Experiment 2 and 3. The three groups were contrasted in each of the experiments. The results showed brainwave differences for the McGurk and the standard audiovisual congruent condition in the left hemisphere. These results also revealed a word/nonword separation in the McGurk effect, which provides electrophysiological evidence for an interaction of the McGurk condition and semantic/cognitive processing. Contrary to hypothesis, significant group differences were not found between the two Chinese groups. Limitations and implications of this study are discussed.

Download

Share

COinS