Rosie Jones

This brief tutorial will touch on some basic and very common applications of perl for processing natural language documents. Some will be the code which answers the very common and very simple questions asked by corpus linguists such as

• "How do I find the frequency of word usage in my corpus?"

• "What are the most common 10 words used in English/my mail spool?"

• "How can I get a stop-list for [Slovenian/French/Tok-pisin]?" [tokenization, counting, sorting by frequency]

We will then move on to some slightly more complex tasks such as:

• "What pairs of words occur as compounds"? [mutual information]

• "How can I make Mark V. Shaney?/ How can I generate text gibberish that sounds like my boss?" [word-based Markov Models/Language Models]

• "How can I get the root form of words?" [stemming/lemmatization]

• "How can I find all the nouns in my text?" [part-of-speech tagging]

and as a bonus

• "How can I suck down all the web-pages under www.foo.org so I can process them?"

This tutorial will be in part an introduction to perl for people who have already asked themselves the above questions, and in part an introduction to the kinds of questions asked by people interested in human languages, for those already versed in perl.

A longer document, example code, and slides will be at http://www.cs.cmu.edu/~rosie/yapc/ , and a snapshot of the materials will be available at this site after the conference.