11-761 Spring 2006 Course Syllabus

Special dates:

1/ 16    class starts at 10:30am
1/ 30    class cancelled
3/13     No class: Spring Break
3/15     No class: Spring Break
4/24     Exam
5/3     Final project presentaion

• We will move at whatever pace we find comfortable.  If there is time left at the end, we will either add topics (papers) or review project work.
• Brush up on college level Prob & Stats:  [mD] ch 1-5. (optional)

Syllabus is subject to change, sometimes without notice!

Topics	Dates	Required reading (due date)	Additional Background
Course Goals, Philosophy and Mechanics	1/16
Statistical Approach to Language: Overview and Historical Perspective (Lafferty's notes)	1/16		[MS] 1.1 - 1.3
Statistical Language Modeling, Computational Linguistics, Statistical Decision Making, the Source-Channel Paradigm	1/16	[MS] 2.1 (1/16)
All About Words: Types, Tokens and Vocabularies	1/18	[MS] 1.4 (1/18)	[BCH] ch. 4
Unigrams: Statistical Estimation, Maximum Likelihood Estimates;	1/18, 1/23		[mD] ch.6, esp. 6.5  [MS] 6.2.1-6.2.2
Sparseness; Smoothing	1/25, 2/1
N-grams: linear interpolation; backoff	2/6, 2/8, 2/13, 2/15	[MS] ch 6  (2/6) [sK]  (2/6)
Measuring Success: Perplexity and Entropy	2/20, 2/22, 2/27		[MS] 2.2 IT notes Entropy of English
Decision Tree Language Models	3/1		[MS] 16.1, [BBDM]
Clustering	3/6		[MS] 14.1, class LM, Lattice LM
Latent Variable Models	3/8, 3/20	Derivation of EM for Gaussian mixture, EM derivation shortcut for exponential family	[MS] 14.2 EM notes
Hidden Markov Models	3/22		[MS] ch. 9,  HMMs for speech
HMM (Cont), EM Algorithm	3/27
Maximum Entropy Modeling	3/29		[MS] 16.2, [rR], [BDD], slides
Maximum Entropy Modeling	4/3		[MS] 16.2, [rR],
Whole-sentence language models; Semantic coherence	4/5		[RCZ]
Cancelled	4/10
Cancelled	4/12
Probabilistic Model for Collaborative Filtering	4/17		Breese et al., 1998, Hofmann & Puzicha, 1999, Si & Jin, 2003,
Stochastic Grammars, the Inside-Outside algorithm	4/19	Notes on Probabilistic Context Free Grammars	[MS] 11.1-11.4
Exam	4/24
Stochastic grammars, the Inside-OutSide algorithm	4/26		ChelbaSlides 98, JelinekChelba 99
A structured language model; Dimensional reduction	5/1	Bellegarda 99;  Indexing by latent semantic analysis	Bellegarda00;  Slides
Final project presentation	5/2 1-2:30 PM Wean Hall 4625
Final project presentation	5/3

Abbreviations (in order or appearance):

• [MS] - Manning and Schutze, Foundations of statistical natural language processing.
• [BCW] - Bell, Cleary and Witten , Text Compression.
• [mD] - Morris Degroot, Probability and Statistics, 2nd edition.
• [sK] - Slava M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer". IEEE-ASSP 35 no.3 p 400-401 1987.
• [BBDM] - "A Tree-Based Statistical Language Model for Natural Language Speech Recognition", L. Bahl, P. Brown, P. deSouza and R. Mercer, IEEE Transcation on Acoustics, Speech and Signal Processing , volume 37, pages 1001--1008, 1989.
• [rR] - "A Maximum Entropy Approach to Adaptive Statistical Language Modeling", Ronald Rosenfeld, Computer, Speech and Language, volume 10, pages 187--228, 1996.