15-212: Principles of Programming

Lecture 20: Lexing

Many applications require some form of tokenization or lexical analysis to be carried out as a preprocessing step. Examples include compiling programming languages, processing natural languages, or manipulating HTML pages to extract structure. As an example, we study a lexical analyzer for a simple language of arithmetic expressions.

Key Concepts

Tokenization
Lexical analysis
Grammar

Sample Code

Stream Code
lecture20.sml

[ CS 15-212 Home page | schedule | language | assignments | handouts ]

Michael Erdmann