- Gates&Hillman Centers
- McWilliams Classroom 4303
- LEILA WEHBE
- Machine Learning Department
- Carnegie Mellon University
Predicting Neural Activity during Story Reading
Adults read text at an average speed of 3 words per seconds. At this rate, they perceive letters, identify word forms, construct the meaning of sentences and interpret the undergoing narrative. Story comprehension is therefore a rich phenomenon, requiring multiple simultaneous processes. On the other hand, fMRI, a commonly used brain-imaging tool to study reading, has a time resolution of a couple seconds along with slowly varying time dynamics. Classically fMRI has been used by localizing which areas correspond to a certain element of language processing (e.g. syntax) by presenting stimuli in blocks that vary in a few critical conditions (e.g. sentences with simple syntax vs. complex syntax). We are interested in a more extensive exploration of reading that accounts for all the levels of processing that are involved when subjects read a complex non-experimental text under close-to-normal conditions. Our aim is to reveal distinct patterns of neural representation for these levels of cognitive processing. These patterns can be used to identify the cognitive roles of different brain areas in language processing and story understanding, to investigate the mechanisms of reading and other high-level tasks, and to relate individual differences in reading to neural differences.
This proposal is based on existing results from a reading experiment for which we developed a predictive model of brain activity that was able to tell with 85% accuracy which of two passages of a story a subject was reading from a segment of unlabeled fMRI data. This model learned different patterns of representation for the different levels of story processing and those patterns aligned with many of the key findings in the literature of language processing. The model replicated all these results simultaneously and using only one experiment. This dissertation will elaborate this method of studying the brain representations involved in reading along three axes: (1) it will further develop fMRI and MEG experimental paradigms and associated approaches to model the content of the stories in a detailed intermediate feature space, (2) it will develop machine learning tools to study the mapping between this story feature space and the brain activity and (3) it will develop statistical tests and classification approaches to characterize the information content of brain activity. The result will be a comprehensive, state-of-the-art computational method to study language processing and story understanding in the brain.
Tom Mitchell (Chair)
Jack Gallant (University of California, Berkeley)
Brian Murphy(Queen's University, Belfast)