Maxine Eskenazi www.cs.cmu.edu/~max

Principal Systems Scientist Tel. 412 268-3858

Language Technologies Institute Fax 412 268-6298

Carnegie Mellon University first three letters of my first name at three initials of the University

6413 Gates Hillman Complex dot edu

5000 Forbes Ave

Pittsburgh, PA 15213 USA

A short bio

I have been chair of the SLaTE special interest group (Speech and Language Technology for Education) of ISCA (International Speech Communication Association) from 2006 to 2011. I am Director of the Dialog Research Center

GOALS INTERESTS PROJECTS STUDENTS PUBLICATIONS TEACHING CARNEGIE SPEECH COMPANY RESOURCES

- Research Goals

Understanding the variability of the speech signal. Creating automatic systems (using spoken dialogue architectures, automatic speech recognition and synthesis) that benefit from this knowledge and that in turn provide a real benefit to end users. This endeavor implies studying groups of speakers, input conditions, styles of speech and detecting the acoustic and upper-level indices that are indicative of these variants. End benefits may include language learning and information giving and gathering.

- Research Interests

I am interested in the variability of the speech signal – relating its sources to its manifestations, characterizing groups of speakers, grouping observable phenomena. Non-native speech is a specific interest within this area, as is speaking style.

Spoken dialogue systems are one of my areas of interest. I am less interested in play systems than systems that have real users and serve a real purpose, such as Let’s Go. The Let’s Go system has been answering the phone for the Port Authority of Allegheny County every evening since the beginning of March 2005. A system with real users provides the ideal platform to run experiments on spoken dialogue such as timing and lexical entrainment. When one is lucky enough to be endowed with a system that has a constant pipeline of real users, one feels compelled to share this treasure with the research community, freely distributing the data we collect and giving them access to the system to conduct experiments of their own.

I am also interested in how to teach foreign languages effectively, both by a human and by a computer. This includes curriculum creation and navigation, interface issues and issues leading to robust learning. Culture underlies all language and so is also an interest of mine. And of course I am interested in pinpointing errors in non-native speech (patented!) and then providing appropriate corrective feedback.

The system I created to detect and correct foreign speakers’ pronunciation errors in English was called Fluency and the basic algorithms developed in that project were spun off into the NativeAccent^TM product sold by the company I started, Carnegie Speech^TM. Another use of research results in use in real life! And the REAP system is also getting into the hands of many students and teachers.

I am the chair of the ISCA special interest group on Speech and Language Technology in Education (SLaTE). Please visit our website for more information (www.sigslate.org).

- Projects

Fluency – a project to use automatic speech recognition to detect pronunciation errors and to provide appropriate correction information – contact me directly for more information.

Let’s Go – a project using a spoken dialogue system to expand access to such systems to the elderly and to non-native speakers. http://www.speech.cs.cmu.edu/letsgo/

DialRC – a Center that is at the service of the Spoken Dialog community, providing data, running studies, educating and running a Spoken Dialog Challenge http://dialrc.org/

LexE – a project on lexical entrainment on both sides of the dialog with spoken dialog systems, sometimes with non-native speakers

REAP – a project to retrieve appropriate, individuated texts for students learning to read http://reap.cs.cmu.edu/ which exists in French and Portuguese as well as in English.

- Students

Present: Jose David Aguas Lopes, Rui Correira, Sukhada Palkar, Sungjin Lee, Yibin Lin, Andrew Fandrianto

Postdoc: Oscar Saz

Past: Adam Skory, Kevin Dela Rosa, Gabriel Parent, Antoine Raux, Jonathan Brown, Jie Hu, Juan Pino, Michael Heilman, Aleata Hubbard, Elizabeth Harris, Kathrin Probst, Yan Ke, Helene Maynard-Bonneau, Paul-Eric Stern, Anne Lacheret-Dujour

- Selected Publications

Spoken Dialogue Systems and Automatic Speech Recognition

Parent, G., Eskenazi, M. Speaking to the Crowd: looking at past achievements in using crowdsourcing for speech and predicting future challenges, Proceedings Interspeech 2011, special session on crowdsourcing, Florence Italy. pdf file

Parent, G., Eskenazi, M. 2010. Toward better crowdsourced transcription: Transcription of a year of the let’s go bus information system data. In Proc SLT, Berkeley, CA. pdf file

Parent, G., Eskenazi, M., 2011, Sources of variability and adaptive tasks, Proceedings CHI2011. pdf file

Alan W Black, Susanne Burger, Alistair Conkie, Helen Hastie, Simon Keizer, Oliver Lemon, Nicolas Merigaud, Gabriel Parent, Gabriel Schubiner, Blaise Thomson, Jason D. Williams, Kai Yu, Steve Young and Maxine Eskenazi, Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results, Proc. SIGDIAL2011, Portland, OR. .pdf file

Eskenazi, M., Black, A., 2010, SDC: The Spoken Dialog Challenge, invited short presentation at SIGDIAL 2010. Slides

Parent, G. and Eskenazi, M. 2010. Lexical Entrainment of Real Users in the Let's Go Spoken Dialog System. In Proceedings of ISCA Interspeech 2010, Tokyo, Japan

Black, A., and Eskenazi, M., "The Spoken Dialogue Challenge" SIGDIAL 2009, Queen Mary University, London. 2009. pdf file

Gonzalez-Brenes, J., Black, A., and Eskenazi, M. "Describing Spoken Dialogue Systems Differences" IWSDS 2009, Irsee, Germany. pdf file

Raux, A., Langner, B., Black, A. and Eskenazi, M. Building Practical Spoken Dialog Systems ACL/HLT 2008 Tutorial, Columbus, Ohio.

Eskenazi, M., Black, A., Raux, A. and Langner, B., 2008, Let's Go Lab: a platform for evaluation of spoken dialog systems with real world users, Interspeech 2008, Brisbane, Australia. pdf file

A. Raux and M. Eskenazi, 2008, Optimizing Endpointing Thresholds using Dialogue Features in a Spoken Dialogue System, SIGdial 2008, Columbus, OH, USA. pdf file

A. Raux and M. Eskenazi, A Multi-Layer Architecture for Semi-Synchronous Event-Driven Dialogue Management, ASRU 2007, Kyoto, Japan. pdf file

Ai, H., Raux, A., Bohus, D., Exkenazi, M., and Litman, D., 2007, Comparing Spoken Dialog Corpora Collected with Recruited Subjects versus Real Users, 8th SIGDial Workshop on Discourse and Dialogue, Antwerp, Belgium. pdf file

D. Bohus, A. Raux, T. Harris, M. Eskenazi, and A. Rudnicky, 2007, Olympus: an open-source framework for conversational spoken language interface research, HLT-NAACL 2007 workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technology, Rochester, NY, USA. pdf file

Raux, A., Bohus, D., Langner, B., Black, A., Eskenazi, M., 2006, Doing Research on a Deployed Spoken Dialogue System: One Year of Let’s Go! Experience, Proc. Interspeech 2006, Pittsburgh, September 2006. .pdf file

A. Raux, B. Langner, D. Bohus, A. W Black and M. Eskenazi, Let's Go Public! Taking a Spoken Dialog System to the Real World, Interspeech 2005 Lisbon, Portugal. .pdf file

Eskenazi, M., 1998, User Come Back, DARPA Communicator Compare and Contrast Meeting, June 16-17, 1998. .pdf file

Ravishankar, M. and Eskenazi, M., 1997, Automatic Generation of Context-dependent Pronunciations, Proc. Eurospeech ’97, Rhodes, Greece, p. 2467 - 2470. .ps file

Placeway, P., Chen, S., Eskenazi, M., Jain, U., Parikh, V., Raj, B., Ravishankar, M., Rosenfeld, R., Seymore, K., Siegler, M., Stern, R., Thayer, E., 1997, The 1996 HUB-4 Sphinx-3 System, Proc, DARPA Speech Recognition Workshop, Chantilly, Virginia, Morgan Kaufmann Publishers.

Seymore, K., Chen, S., Eskenazi, M., Rosenfeld, R., (1997), Language and Pronunciation Modelling in the CMU 1996 HUB-4 Evaluation, Proc, DARPA Speech Recognition Workshop, Chantilly, Virginia, Morgan Kaufmann Publishers

Computer-Assisted Language Learning

Eskenazi, M., 2009, An overview of spoken language technology for education, Speech Communication, Elsevier, vol 51 issue 10 p. 832-844.

Vocabulary Learning

Dela Rosa, K., Eskenazi, M., 2011, Effect of Word Complexity on L2 Vocabulary Learning, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies's 6th Workshop on Innovative Use of NLP for Building Educational Applications (ACL-HLT: BEA 2011). 2011. pdf file

Dela Rosa, K., Eskenazi, M., 2011, Self-Assessment of Motivation: Explicit and Implicit Indicators in L2 Vocabulary Learning, Proceedings of the 15th International Conference on Artificial Intelligence in Education (AIED 2011). pdf file

Dela Rosa, K., Eskenazi, M., 2011, Impact of Word Sense Disambiguation on Ordering Dictionary Definitions in Vocabulary Learning Tutors, Proceedings of the 24th International FLAIRS Conference (FLAIRS 2011). pdf file

Skory, A., Eskenazi, M., 2011, Generation of Educational Content through Gameplay, Proc. SLaTE2011, Venice, August 2011. .pdf file

M. Heilman, K. Collins-Thompson, M. Eskenazi, A. Juffs, L. Wilson.  2010. Personalization of Reading Passages Improves Vocabulary Acquisition.  International Journal of Artificial Intelligence in Education,  Vol. 20 (1).

Dela Rosa, K., Parent, G. ,Eskenazi, M., 2010, Multimodal learning of words: A study on the use of speech synthesis to reinforce written text in L2 language learning, Proceedings of the ISCA Workshop on Speech and Language Technology in Education (SLaTE 2010) pdf file

Skory, A. and Eskenazi, M. (2010), Predicting Cloze Quality for Vocabulary Training, Proc. of NAACL HLT Workshop on Innovative Use of NLP for Building Educational Applications, Los Angeles, California. pdf file

Skory, A. and Eskenazi, M. (2010), Automatic Selection of Collocations for Instruction, Proc. of SLaTE Workshop on Speech and Language Technology in Education, Tokyo, Japan pdf

file

Marujo, l., Lopes, J., Mamede, N., Trancoso, I, Pino, J., Eskenazi, M., Baptista, J., Viana, C., Porting REAP to European Portuguese, Proceedings ISCA Workshop on Speech and Language for Education SLaTE2009, Warwickshire England. pdf file

Pino, J., Eskenazi, M., 2009, Semi-Automatic Generation of Cloze Question Distractors: Effect of Students’ L1, Proceedings ISCA Workshop on Speech and Language for Education SLaTE2009, Warwickshire England. pdf file

Pino, J., Eskenazi, M., 2009, Measuring Hint Level in Open Cloze Questions, Proceedings FLAIRS09. pdf file

Pino, J., Eskenazi, M., 2009, An Application of Latent Semantic Analysis to Word Sense Discrimination for Words with Related and Unrelated Meanings, Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications at NAACL, Boulder Colorado. pdf file

M. Heilman and M. Eskenazi. 2008. Self-assessment in vocabulary tutoring. In Proc. of the Young Researcher's Track. Ninth International Conference on Intelligent Tutoring Systems. pdf file

J. Pino, M. Heilman, and M. Eskenazi. 2008. A selection strategy to improve cloze question quality. In Proc. of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains. Ninth International Conference on Intelligent Tutoring System. pdf file

M. Heilman, K. Collins-Thompson, and M. Eskenazi. 2008. An analysis of statistical models and features for reading difficulty prediction. In Proc. of The 3rd Workshop on Innovative Use of NLP for Building Educational Applications. pdf file

M. Heilman, L. Zhao, J. Pino, and M. Eskenazi. 2008. Retrieval of reading materials for vocabulary and reading practice. In Proc. of the 3rd Workshop on Innovative Use of NLP for Building Educational Applications. pdf file

M. Heilman and M. Eskenazi. 2007. Application of automatic thesaurus extraction for computer generation of vocabulary questions. In Proc. of the SLaTE Workshop on Speech and Language Technology in Education. pdf file

M. Heilman, A. Juffs, and M. Eskenazi. 2007. Choosing reading passages for vocabulary learning by topic to increase intrinsic motivation. In Proc. of AIED. pdf file

M. Heilman, K. Collins-Thompson, J. Callan, and M. Eskenazi. 2007. Combining lexical and grammatical features to improve readability measures for first and second language texts. In Proc. NAACL-HLT. pdf file

Callan, J., Eskenazi, M., Perfetti, C., 2006, Progress in Providing Reader-Specific lexical Practice for Improved Reading Comprehension, presented at IES 2006 research conference, June 15-16 2006, Washington DC

Juffs, A., Eskenazi, M., Wilson, L., Pelletreau, T., Sanders, J., Callan, J., Brown, J., 2006, Promoting robust learning of vocabulary through computer assisted language learning, Proc. Joint conference of AAAL and ACLA/CAAL 2006, Montreal, June 2006.

Juffs, A., Wilson, L., Eskenazi, M., Callan, J., Brown, J., Collins-Thompson, K., Heilman, M., Pelletreau, T., Sanders, J., 2006, Robust learning of vocabulary: investigating the relationship between learner behaviour and the acquisition of vocabulary (poster). At The 40th Annual TESOL Convention and Exhibit (TESOL 2006).

Brown, J., Eskenazi, M., 2006, Using Simulated Students for the Assessment of Authentic Document Retrieval, ITS2006, Taiwan, June 2006. Lecture Notes in Computer Science, Editors:  Mitsuru Ikeda, Kevin D. Ashley, Tak-Wai Chan, Publisher: Springer Berlin Heidelberg, pp. 685 – 688. http://dx.doi.org/10.1007/11774303_68

Heilman, M., Eskenazi, M., 2006, Language Learning: Challenges for Intelligent Tutoring Systems, Workshop on Ill-defined Domains in Intelligent Tutoring, Taiwan, June 2006. .pdf file

Heilman, M., Collins-Thompson, K., Callan, J., Eskenazi, M., 2006, Classroom Success of an Intelligent Tutoring System for Lexical Practice and Reading Comprehension, Proc. Interspeech2006, Pittsburgh September 2006. .pdf file

J. Brown, G. Frishkoff, and M. Eskenazi. (2005). "Automatic question generation for vocabulary assessment." In Proceedings of HLT/EMNLP 2005. Vancouver, B.C. .pdf file

J. Brown and M. Eskenazi. (2005). "Student, text and curriculum modeling for reader-specific document retrieval." In Proceedings of the IASTED International Conference on Human-Computer Interaction 2005. Phoenix, AZ. .pdf file

Brown, J., Eskenazi, M., 2004, Retrieval of Authentic Documents for Reader-Specific Lexical Practice, Proceedings INSTIL 2004, Venice Italy. .pdf file

Non-native speech

Correia, R., Pellegrini, T, Eskenazi, M., Trancoso, I, Baptisa, J., Mamede, N., 2011, Listening Comprehension Games for Portuguese: exploring the best features, Proceedings SLaTE 2011. pdf file

Eskenazi, M., Raux, A., Harris, E., 2006, “Using speech recognition for just-in-time language learning, J. Acoust. Soc. Am, vol 120, no. 5, pt.2, p.3138. THE SLIDES FROM MY TALK ARE HERE

Raux, A., Eskenazi, M., 2004, Using Task-Oriented Spoken Dialogue Systems for Language Learning: Potential, Practical Applications and Challenges, Proceedings INSTIL 2004, Venice. .pdf file

Raux, A., Eskenazi, M., 2004, Non-native users in the Let’s Go!! Spoken Dialogue System: Dealing with Linguistic Mismatch, Proceedings HLT 2004, Boston. .pdf file

Raux, A., Langner, B., Black. A., Eskenazi. M., 2003, LET’S GO: Improving Spoken Dialog Systems for the Elderly and Non-natives, Proc. Eurospeech 2003, Denver. .pdf file

Pronunciation Training

Saz, O., Eskenazi, M., Identifying Confusable Contests for Automatic Generation of Activities in Second Language Pronunciation Training, Proceedings SLaTE 2011, Venice. pdf file

Probst, K., Ke, Y., Eskenazi, M., 2002, Enhancing foreign language tutors - in search of the golden speaker, Speech Communication, 37/3-4 pp. 161-173.

Eskenazi, M., Pelton, G. 2002, Pinpointing pronunciation errors in children’s speech: examining the role of the speech recognizer, Proposed to the Pronunciation Modeling and Lexicon Adaptation for Spoken Language Technology Workshop, Sept 2002, Colorado. .pdf file

Eskenazi, M., Ke, M., Albornoz, J., Probst, K., 2000. Update on the Fluency Pronunciation Trainer, In: Proceedings of InSTIL 2000, Dundee. .pdf file

Mayfield Tomokiyo, L., Wang, L., Eskenazi, M., 2000, An Empirical Study of the Effectiveness of Speech-Recognition-based Pronunciation Training, Proc. ICSLP 2000, Beijing.

Eskenazi, M., (1999) Issues in the use of speech recognition for foreign language tutors, invited paper: Language Learning and Technology Journal (online) Vol. 2, No. 2, January 1999, pp. 62-76. http://llt.msu.edu/vol2num2/article3/index.html

Eskenazi, M., Hansma, S., 1998, The Fluency Pronunciation Trainer, Proc. STiLL Workshop on Speech Technology in Language Learning, Marhollmen, May. .pdf file

Eskenazi, M., Hansma, S., Semp, M., Warner, R., 1998, By ear and by eye - adaptive tutoring for foreign language pronunciation training – in Proc. STiLL Workshop on Speech Technology in Language Learning Marhollmen. .pdf file

Data collection and assessment

Parent, G. and Eskenazi, M. 2010. Clustering dictionary definitions using Amazon Mechanical Turk. In Proc. of the NAACL/HLT workshop on Creating Speech and Language Data With Amazon's Mechanical Turk. pdf file

Eskenazi, M., Rudnicky, A., Gregory, K., Constantinides, P., Brennan, R., Bennett, C., Allen, J., 1998, Data Collection and Processing in the Carnegie Mellon Communicator, in Proc. ESCA Eurospeech 98. . .pdf file

M. Eskenazi, 1996, KIDS: A Database of Children's Speech , in Proc. 3rd joint Meeting: Acoustical Societies of America and Japan, Honolulu.

Eskenazi, M., Hogan, C., Allen, J., Frederking, R., 1998, Issues in database design: Recording and processing speech from new populations, Proc. LREC Assessment and Database Workshop, Grenada, Spain.

Lamel, L., Gauvain, JL., Eskenazi, M., 1991, BREF, a Large Vocabulary Spoken Corpus for French, in Proc. EUROSPEECH-91

AFNOR, 1990, norme experimentale S 31-115, Evaluation de systemes de traitement automatique de la parole Partie 1: Definitions et methode d'evaluation de systemes de reconnaissance automatique de la parole - systemes de reconnaissance globale.

Elderly speech

Eskenazi. M., Black, A., Simmons, R., 2002, Elderly Perception of Speech from a Computer, Meeting of the Acoustical Society of America, Pittsburgh, June 2002. .pdf file

Eskenazi, M., Black, A., 2001. A study on speech over the telephone and aging, Proc. Eurospeech01, Aalborg, Denmark September 2001. html link

Speaking Styles

Eskenazi, M. 1993. Trends in Speaking Style Research, Keynote speech, Proceedings Eurospeech’93, Berlin. .pdf file

Eskenazi, M., 1995, Hot Topics in Speaking Style Research, in European Studies in Phonetics and Speech Communication, Bloothooft, Hazan, Huber, Llisterri, eds., OTS Publications, The Netherlands. P. 58 - 62.

Eskenazi, M., Lacheret, A., 1991, Exploration of individual strategies in continuous speech, Speech Communication, vol. 10 no. 3.

Eskenazi, M. 1992. Changing speech styles, speakers’ strategies in read speech and careful and casual spontaneous speech. Proceedings of the International Conference on Spoken Language Processing, Banff.

Cochlear Implants

Eskenazi, M., Vormes, E., Monguillot, G., Frachet, B., 1993, A new training and assessment technique for cochlear implants, in Advances in Cochlear Implants, Hochmair-Desoyer and Hochmair eds., International Science Seminars, Vienna, Austria, p. 572-577.

- Teaching

* Speech class: 11-752 Production, Prosody and Synthesis taught with Alan Black 11-752 course description

* Language Technologies: 11-717 Language Technologies for Computer-Assisted Language Learning taught with Lori Levin and Teruko Mitamura 11-717 course description

A book chapter on the material in 11-717:

Eskenazi, M., Brown, J., 2006, Teaching the Creation of Software that Uses Speech Recognition, in Teacher Education in CALL, P. Hubbard and M. Levy Eds., Language Learning and Language Teaching series, John Benjamins Publishing.

Here is an informal course I teach on how to write a scientific paper.

Session 1

- Carnegie Speech Company

In 2001, Jaime Carbonell and I started the Carnegie Speech Company. The company produces software for teaching and assessing ESL. It has received funding from Innovation Works and from the state of Pennsylvania, and it has had SBIR grants from the US Department of Education and the National Science Foundation as well as a prestigious Advanced Technology Program award from NIST at the Department of Commerce. We have people in various places around the world using our products! You can find out all about it at: www.carnegiespeech.com

And see: http://www.postgazette.com/pg/09013/941345-298.stm?cmpid=newspanel1

Resources

Here are some things that may be of interest to you.

1. THE PITTSBURGH SCIENCE OF LEARNING CENTER
2. Grapheme to Phoneme dictionary
3. Automatic Speech Recognizer
4. AN AUTOMATIC SPEECH RECOGNITION (ASR) DIALOGUE SYSTEM
5. NEW FINDINGS IN LANGUAGE LEARNING
6. AUTHORING NEW TUTORING SYSTEMS SUMMER SCHOOL
7. FAUX AMIS

1. THE PITTSBURGH SCIENCE OF LEARNING CENTER: Through an NSF SLC award, cognitive scientists, language technologists, psychologists and others work together in this center to explore robust learning. You can find out more about our Center at learnlab.org

2. GRAPHEME_TO-PHONEME DICTIONARY: I am one of the people who has worked on CMUDICT a large grapheme-to-phoneme dictionary containing over 130,000 entries. CMUDICT can be used for a variety of applications and research topics. It is distributed as open source software and can be found at: http://www.speech.cs.cmu.edu/cgi-bin/cmudict

3. AUTOMATIC SPEECH RECOGNIZER: The first time I used SPHINX II, I was astounded at how robust it could be. It is not perfect – none are, as we all know. But with understanding of the strong and weak points of the recognizer and some smart engineering, it is possible to modify it to perform nicely in well-defined applications (like Carnegie Speech^TM’s NativeAccent^TM). It is also open source software and can be found at: http://www.cmusphinx.org .

An important element in getting the recognizer to work well in a new application is to train it with data that is representative of the speakers who will use the application and the language they will use to express themselves. Carnegie Speech^TM sells licenses to YOUTH, a database of children’s speech that we put together during our Department of Education SBIR. This can be used to train the recognizer for applications for kids from about 6 to 11. Several commercial applications successfully use this data in their products.

4. AN AUTOMATIC SPEECH RECOGNITION (ASR) DIALOGUE SYSTEM: One of the precursors of our Let’s Go dialogue system and one of the best known is the Galaxy system from MIT. http://www.sls.csail.mit.edu/GALAXY.html

5. NEW FINDINGS IN LANGUAGE LEARNING: One of the most promising directions that I know of for language learning is the one that started with D. Pisoni, A. Bradlow (Northwestern) and R. Yamada (ATR). They create pairs of sounds (R and L for Japanese learners of English) and acoustically “pull them apart” until the student can hear the difference between them. Students for whom this training works often can pronounce a new sound without having pronunciation training on it.

6. AUTHORING NEW TUTORING SYSTEMS SUMMER SCHOOL: The great people who have made some of the most advanced and successful intelligent tutoring systems that exist hold a summer school each year where you can come and use their authoring tools to create your own tutor. You can find it here: http://learnlab.org/opportunities/summer/

7. FAUX AMIS: I compiled a list of words that appear to be the same, but have very different meanings in French and English. The list is available here and if you find other entries or would like to suggest modifications or corrections, please download and let me know. I will be glad to post your comments and make changes to the list for all to use. .doc file Eventually I will add the meanings!