---

11-761 Spring 2008 Course Syllabus

Special dates:

3/11     No class: Spring Break
3/13     No class: Spring Break
4/17     No class.

• We will move at whatever pace we find comfortable.  If there is time left at the end, we will either add topics (papers) or review project work.
• Brush up on college level Prob & Stats:  [mD] ch 1-5. (optional)

Syllabus is subject to change, sometimes without notice!

Topics	Dates	Required reading (due date)	Additional Background
Course Goals, Philosophy and Mechanics	1/15
Statistical Approach to Language: Overview and Historical Perspective (Lafferty's notes)	1/15	[MS] 1.1 - 1.3 (1/17)
Statistical Language Modeling, Computational Linguistics, Statistical Decision Making, the Source-Channel Paradigm	1/15	[MS] 2.1 (1/17)
All About Words: Types, Tokens and Vocabularies	1/17	[MS] 1.4 (1/17)	[BCW] ch. 4
Unigrams: Statistical Estimation, Maximum Likelihood Estimates;	1/17, 1/22	[MS] 6.2.1-6.2.2 (1/17)	[mD] ch.6, esp. 6.5
Sparseness; Smoothing	1/24, 1/29
N-grams: linear interpolation; backoff	1/31, 2/5, 2/7, 2/12	[MS] ch 6  (1/31) [sK]  (1/31)	Chen & Goodman 98 (pp. 1-21)
Measuring Success: Perplexity and Entropy	2/14, 2/19, 2/21	[MS] 2.2 (2/14)	IT notes Entropy of English
Decision Tree Language Models	2/26, 2/28	[BBDM] (2/26)	[MS] 16.1
Clustering	3/4, 3/6	class LM (2/28)	[MS] 14.1, Lattice LM
Latent Variable Models, EM Algorithm	3/18, 3/20	[MS] 14.2 (3/4), notes by Guy Lebanon: Derivation of EM for Gaussian mixture, EM derivation shortcut for exponential family	more advanced EM notes (by John Lafferty)
Hidden Markov Models	3/20, 3/25	[MS] ch. 9 (3/18)	Larry Rabiner's classic HMM tutorial
Maximum Entropy Modeling	3/27	Adam Berger's online tutorial, Convexity, Maximum Likelihood, and All That	[MS] 16.2, Noah Smith's tutorial, [BDD], [rR], [rR] slides
Whole-sentence language models; Semantic coherence	4/1		[RCZ ]
Latent Semantic Analysis and Dimensionality Reduction	4/3	Bellegarda 99, Indexing by latent semantic analysis (4/16)	Bellegarda00;, Yan Liu's Slides
Probabilistic Latent Semantic Analysis and Applications	4/8	Hoffmann 99 (4/18)	Gildea and Hofmann 99, Raux and Singh 04
Exam	4/10
Probabilistic Languages: Finite-State and Otherwise (Guest lecturer: Noah Smith)	4/15
No class	4/17
Latent Semantic Analysis	4/22		[MS] 11.1-11.4
Probabilistic Context Free Grammars (PCFG)	4/24	Notes on Probabilistic Context Free Grammars (4/24), Jelinek and Chelba 99 (4/24)	Chelba Slides 98
Final project presentation	4/29
Final project presentation	5/1

Abbreviations (in order or appearance):

• [MS] - Manning and Schutze, Foundations of statistical natural language processing.
• [BCW] - Bell, Cleary and Witten , Text Compression.
• [mD] - Morris Degroot, Probability and Statistics, 2nd edition.
• [sK] - Slava M. Katz, "Estimation of probabilities from sparse data for the language model component of a speech recognizer". IEEE-ASSP 35 no.3 p 400-401 1987.
• [BBDM] - "A Tree-Based Statistical Language Model for Natural Language Speech Recognition", L. Bahl, P. Brown, P. deSouza and R. Mercer, IEEE Transcation on Acoustics, Speech and Signal Processing , volume 37, pages 1001--1008, 1989.
• [rR] - "A Maximum Entropy Approach to Adaptive Statistical Language Modeling", Ronald Rosenfeld, Computer, Speech and Language, volume 10, pages 187--228, 1996.

---

Last modified: Jan 10 EST 2008