Neural Information Processing Systems

Tutorial Information

• November 27.
• Abstracts and Speaker Biographies

Session I: 09:30-11:30

Information theory
Thomas M. Cover, Stanford University

Population codes: Theory and practice
Richard S. Zemel, University of Toronto
Bruce McNaughton, University of Arizona

Session II: 13:00-15:00

Variational methods: Theory and applications
Tommi S. Jaakkola, Massachusetts Institute of Technology

Linking brain to behavior: From visual cortex to global brain organization
Stephen Grossberg, Boston University

Session III: 15:30-17:30

Probability models and Markov chain Monte Carlo
Andrew Gelman, Columbia University

Attention
Harold Pashler, University of California at San Diego

Thomas M. Cover
Department of Electrical Engineering and Statistics
Stanford University
Home Page

Abstract

It should be possible to give the high points of information theory at a tutorial level in two hours. At least that is what I'm going to try to do. We'll look at fundamental examples in data compression and channel capacity and show how they lead naturally to the entropy and maximum mutual information as general answers. We'll also consider distributed information networks and the answers to those problems as well.

This tutorial is for people who have not had information theory but are interested in the main results and the fundamental intuition behind them. The emphasis will be on examples that can be solved by inspection on one hand and the theoretical counterparts on the other.

Biography

Thomas M. Cover is a professor jointly in the departments of Statistics and Electrical Engineering at Stanford University. He holds the K.T. Li endowed chair in engineering. His areas of research are information theory, communication theory, portfolio theory and statistics. He is the author of over 100 papers and two books, including the basic textbook, ``Elements of Information Theory,'' and has authored numerous papers on portfolio theory.

He received the 1990 Claude E. Shannon Award (the highest award in information theory), the 1994 Neural Net Pioneer Award, the 1997 IEEE Richard W. Hamming Medal, and two prize paper awards. He has been elected to the National Academy of Engineering and is a Fellow of the IEEE and the Institute of Mathematical Statistics.

Richard S. Zemel
Department of Computer Science
University of Toronto
Home Page

Bruce McNaughton
Department of Psychology
University of Arizona
Home Page

Abstract

The study of population codes, whose aim is to understand the information carried in populations of neurons, is one of the primary areas linking computational theory with experimental neuroscience. Numerous neurophysiological studies have revealed that many neurons within a cortical region are tuned to specific variables; a population of cells respond to and presumably encode particular values. Models of this form of coding are becoming increasingly relevant as techniques of assessing simultaneous activity of many neurons are developed, such as multi-unit recordings and fMRI.

In this tutorial, we will cover basic techniques and lessons learned from the study of population codes, including a review of how multielectrode recordings are done, and methods of reading out information from these recordings. We will then cover more advanced topics, such as the study of interacting populations, model switching, temporal dynamics, learning, and the role of noise.

Biographies

Richard S. Zemel is an Assistant Professor in the Department of Computer Science at the University of Toronto. He received his PhD from that same department in 1993, and following postdoctoral work at the Salk Institute and Carnegie Mellon University, was on faculty at the University of Arizona before returning to Toronto. He is a recipient of a McDonnell-Pew Cognitive Neuroscience Postdoctoral Award, and an Office of Naval Research Young Investigator Award. His research interests include computational theories of inference and learning in biological and artificial systems, with a current focus on representations of visual motion, multisensory integration, and probabilistic models of neural representations.

Bruce L. McNaughton is a Professor in the Departments of Psychology and Physiology at the University of Arizona, Director of the Arizona Research Laboratories Division of Neural Systems, Memory and Aging and Director of the Cognitive and Neural Systems Graduate Program in the Department of Psychology. He received his Ph.D. from Dalhousie University, Halifax, Nova Scotia in 1978 and did his postdoctoral work in the laboratories of Dr. Per Andersen at the University of Oslo, and Dr. John O'Keefe at University College London. He has received numerous honors and awards, including a NATO Postdoctoral Fellowship, Elected Associate of the Neurosciences Research Program, and a Research Scientist Award from N.I.M.H. The main focus of Dr. McNaughton's research is the physiological and computational basis of animal cognition and memory, with emphasis on the mechanisms by which internal representations of spatial relations are constructed and stored in neural networks, in particular the hippocampal formation.

Tommi S. Jaakkola
Department of Computer Science and Electrical Engineering
Massachusetts Institute of Technology
Home Page

Abstract

Modern variational techniques, broadly construed, have become indispensable tools in various fields including physics, statistics, and control theory. The classic setting, i.e., finding the extremum of an integral involving an unknown function and its derivatives, no longer serves as a defining characteristic. The purpose of this tutorial is to introduce the basic principles and ideas behind variational methods and explain how they have and can be applied to many problems of general interest in machine learning. I will provide intuition as well as formal justification for when the methods are appropriate and when they are not. Towards the end I will also briefly discuss some open problems.

The tutorial requires little background knowledge besides general familiarity with machine learning problems and a working knowledge of optimization principles.

Biography

Tommi S. Jaakkola is an Assistant Professor of Computer Science at MIT. He received his Ph.D. from MIT in computational neuroscience, 1997, and following a Sloan postdoctoral position in computational molecular biology he joined the MIT EECS faculty 1998. His research interests include theoretical aspects of machine learning, statistical inference as well as problems in computational biology and information retrieval.

Stephen Grossberg
Department of Cognitive & Neural Systems
Boston University
Home Page

Abstract

The organization of neocortex into layers is one of its most salient anatomical features. These layers include circuits that form functional columns in cortical maps. A major unsolved problem concerns how the "laminar computing" that is realized by neocortex contributes to biological intelligence? In particular, how are bottom-up, top-down, and horizontal interactions organized within cortical layers to generate adaptive behaviors? This tutorial will begin with a brief summary of how to model such complex biological systems, and a broad overview of the global organization of the brain that is emerging from such analyses, including the organization of visual cortex for processing boundary and surface information. It will then focus on a rapidly developing neural model of how laminar computing enables the visual cortex to realize: (1) the binding process whereby cortex groups distributed data into coherent object representations; (2) the attentional process whereby cortex selectively processes important events; and (3) the developmental and learning processes whereby cortex shapes its circuits to match environmental constraints. It is proposed that the mechanisms governing (3) in the infant lead to properties (1) and (2) in the adult. New computational ideas about nonlinear feedback networks suggest how neocortex develops and learns in a stable way; how the cortex maintains its sensitivity to analog properties of the environment while it coherently binds distributed information into emergent boundary groupings; and how preattentive and attentive processing are intimately linked within the cortical layers. The model is used to simulate neurophysiological, anatomical, and psychophysical data about visual cortex. Variants of the model are finding their way into new algorithms for processing complex and noisy imagery with a large dynamical range; e.g., SAR imagery.

Biography

Stephen Grossberg is Wang Professor of Cognitive and Neural Systems and Professor of Mathematics, Psychology, and Biomedical Engineering at Boston University. He is the founder and Director of the Center for Adaptive Systems, founder and Chairman the Department of Cognitive and Neural Systems, founder and first President of the International Neural Network Society, and founder and co-Editor-in-Chief of the journal Neural Networks. Grossberg is also an editor of many other journals, including Journal of Cognitive Neuroscience, Behavioral and Brain Sciences, Cognitive Brain Research, Neural Computation, and IEEE Transactions on Neural Networks. He was general chairman of the IEEE First International Conference on Neural Networks. His lecture series at MIT Lincoln Laboratory on neural network technology was instrumental in motivating the laboratory to initiate the national DARPA Study on Neural Networks. He received the 1991 IEEE Neural Network Pioneer award, the 1992 INNS Leadership Award, the 1992 Thinking Technology Award of the Boston Computer Society, and the 2000 Information Science Award of the Association for Intelligent Machinery. He was elected a Fellow of the American Psychological Association in 1994 and a member of the Society of Experimental Psychologists in 1996.

He and his colleagues have pioneered and developed a number of the fundamental principles, mechanisms, and architectures that form the foundation for contemporary neural network research, particularly those which enable individuals to adapt successfully in real-time to unexpected environmental changes. Such models have been used both to analyse and predict interdisciplinary data about mind and brain, and to suggest novel architectures for technological applications.

Grossberg received his graduate training at Stanford University and Rockefeller University, and was a Professor at MIT before assuming his present position at Boston University.

Andrew Gelman
Department of Statistics
Columbia University
Home Page

Abstract

When you have to estimate a lot of parameters from relatively sparse data, probability modeling (also called Bayesian statistics) is an effective approach for capturing uncertainty in the parameter estimates while avoiding the overfitting that would occur if the fit of parameters to data were simply optimized. The output of such an analysis is a probability distribution in parameter space, which can be approximated by a set of random draws from this distribution, generally obtained using Markov chain Monte Carlo methods. We shall present an overview of these simulation methods, starting with the Metropolis algorithm, which is a generalization of greedy optimization. In addition, we shall discuss how simulations fit into a general view of statistical data analysis which goes beyond "parameter estimation" and "inference" and more directly addresses the iterative scientific process of model building, checking, and improvement. We illustrate with logistic and Boolean regression models that are used in psychology and elsewhere.

Biography

Andrew Gelman is a professor in the Department of Statistics and director of the Quantitative Methods in Social Sciences program at Columbia University. He received his Ph.D. in statistics at Harvard University in 1990, and his coauthored book, Bayesian Data Analysis, appeared in 1995. His applied research interests include psychology, public health, natural resources, and public policy, and his methodological interests include model checking, survey sampling, decision analysis, and statistical graphics and computation.

Harold Pashler
Department of Psychology
University of California, San Diego
Home Page

Abstract

This tutorial will present an overview of the psychological phenomena of attention and discuss their possible implications for underlying information processing mechanisms. Attention is an everyday concept that alludes on the one hand to abilities of human beings (to select some stimuli and filter out others) and on the other hand to limitations on our performance (difficulty in carrying out different mental activities at the same time). Both of these aspects of cognitive functioning will be discussed, with a focus on behavioral measurements and their implications for functional architecture. I will also provide an overview of recent neurophysiological and brain-imaging results relating to attention. Topics discussed will include selection among different sensory stimuli and the extent to which "ignored" stimuli are processed, pre-cueing of stimuli and the question of noise suppression versus signal enhancement, and the broad issue of what kinds of information-processing operations the brain is capable of carrying out in parallel. I will conclude with some speculations about how the limitations on parallel processing evident from behavioral studies may reflect underlying neural information processing architecture.

Biography

Hal Pashler is Professor in the Department of Psychology and the Program in Cognitive Science at the University of California, San Diego. His research has examined many aspects of human attention, including visual search, limitations on performing multiple tasks, and the role of attention in memory storage and retrieval. He is the author of The Psychology of Attention (1997; MIT Press) and editor of Attention (1998; Taylor & Francis). Pashler received the 1999 Troland Research Prize from the National Academy of Sciences "for his many experimental breakthroughs in the study of spatial attention and central executive control and for his insightful theoretical analysis of human cognitive architecture".