• Hey, I'm Khyathi Chandu!

    I am a PhD student in Language Technologies Institute at Carnegie Mellon University . I am co-advised by Prof. Alan Black and Prof. Eric Nyberg .

    My research goal is to enable machines generate natural language narrative from multiple modalities as input. In particular, I have been working on vision and language as the modalities. I have been mostly approaching this problem from the language side. Towards realizing this, I am working on Multimodal narrative Intelligence for my thesis.

    Being a multilingual myself, I appreciate how code-switching between languages made my communication and expressiveness easier. I previously worked on a few language phenomena in this domain and am continuing to work on a code-switched conversational agents.



Google Scholar Profile
Storyboarding of Recipes: Grounded Contextual Generation
ACL 2019

Khyathi Chandu, Eric Nyberg, Alan Black

"My Way of Telling a Story": Persona based Grounded Story Generation
Storytelling workshop, ACL 2019

Khyathi Chandu*, Shrimai Prabhumoye*, Ruslan Salakhutdinov, Alan Black

A Survey of Code-switched Speech and Language Processing
(under review at Computer Speech & Language 2019)

Sunayana Sitaram, Khyathi Chandu, Sai Krishna Rallabandi, Alan Black

Language Informed Modeling of Code-Switched Text

Khyathi Chandu*, Thomas Manzini*, Sumeet Singh*, Alan Black

Code-Mixed Question Answering Challenge: Crowd-sourcing Data and Techniques

Khyathi Chandu, Ekaterina Loginova, ... , Manoj Chinnakotla, Eric Nyberg, Alan Black

Tackling Code-Switched NER: Participation of CMU

Parvathy Geetha*, Khyathi Chandu*, Alan Black

Comparative Analysis of Neural QA models on SQuAD
MRQA, ACL 2018

Soumya Wadhwa, Khyathi Chandu, Eric Nyberg

Extraction Meets Abstraction: Ideal Answer Generation for Biomedical Questions
BioAsq, EMNLP 2018

Yutong Li, ... Khyathi Chandu, Eric Nyberg

Ontology-Based Retrieval and Neural Approaches for BioASQ Ideal Answer Generation
BioAsq, EMNLP 2018

Ashwin Nareshkumar*, ..., Khyathi Chandu, Teruko Mitamura, Eric Nyberg

Textually Enriched Neural Module Networks for Visual Question Answering

Khyathi Chandu, Mary Arpita Pyreddy, Matthieu Felix, Narendra Nath Joshi

WebShodh: A Code Mixed Factoid Question Answering System for Web
CLEF 2017

Khyathi Chandu, Manoj ChinnakotlaAlan W. BlackManish Shrivastava

Speech Synthesis for Mixed-Language Navigation Instructions
InterSpeech 2017

Khyathi Chandu, Sai Krishna Rallabandi, Sunayana Sitaram, Alan Black

Tackling Biomedical Text Summarization: OAQA at BioASQ 5B
BioAsq, ACL 2017

Khyathi Chandu, Aakanksha Naik, Aditya Chandrasekar, Zi Yang, Niloy Gupta, Eric Nyberg

"nee intention enti?" towards dialog act recognition in code-mixed conversations
International Conference on Asian Language Processing, IALP, 2017

Jitta Divya Sai, Khyathi Chandu, Harsha Pamidipalli, Radhika Mamidi

Building CMU Magnus from User Feedback
Amazon Alexa Challenge Proceedings

Shrimai Prabhumoye*, Fadi Botros*, Khyathi Chandu*, ..., Zhou Yu, Alan Black

"Answer ka type kya he?" Learning to Classify Questions in Code-Mixed Language
WWW 2015

Khyathi Chandu, Manoj Chinnakotla, Manish Shrivastava

Domain Adaptation in Morphological Analysis
International Journal of Languages, Literature and Linguistics, 2015

Prathyusha Kuncham, Khyathi Chandu, Kovida Nelakuditi, Dipti Misra Sharma



Prof Eric Nyberg, Prof Teruko Mitamura

Spring 2018

Prof P J Narayan

  • Spring 2016
  • Spring 2014

Prof Manish Shrivastava

  • Spring 2015

Prof Manish Shrivastava

  • Spring 2015

Prof Bhimalapuram

  • Fall 2013


Multimodal Procedural Text Generation Jan 2019-present

Advisors: Alan Black, Eric Nyberg

  • Generating Textual Narratives for procedures such as cooking recipes from images and phase patterns in text.
  • Structure learnt as a scaffold and conditioned on the decoder.
  • You can get this dataset here: Storyboarding data

Conditional Questioning in Visual Dialog Jan 2019-May 2019

Advisor: Abhinav Gupta

  • Generating relevant questions in turn-taking by condiioning the Q-bot on specific topics.
  • Scope to reduce the intermediate dialog turns by determining the context from topic modeling of history.

Personality Based Visual Storytelling Jan 2019-May 2019

Advisors: Ruslan Salakhutdinov, Alan Black

  • Generating visual stories from ViST conditioned on the personalities derived from Engaging Image Chat dataset.
  • (Just for fun, I assumed these to replicate personalities from Harry Potter, slides coming soon)
  • Conditioning the decoder of the Glocal context model to influence the choice of lexicon for a given personality type.

Entities in Visual Storytelling Jan 2019-present

Advisor: Alan Black

  • Visual stories woven around entity skeletons extracted from coreference chains targeted to model where and how to refer to an entity.
  • Proposed hierarchical glocal attention model using entity skeletons that attends to words and individual sentences in a story.

Code-Mixed Dialog Apr 2019-present

Spare time project

  • Building NLP stack for different modules needed for building a code-mixed dialog agent, capable in conversing in informal scenarios.
  • Worked so far on POS Tagging, NER, Language Modeling, Question Answering

Biomedical Text Summarization Apr 2019-present

Advisor: Eric Nyberg

  • Participated in BioASQ 5B \& 6B in ideal answer generation and achieved top ROUGE scores in final test batches (approximately 0.68).
  • Developed a query oriented abstractive summarization system with an encoder-attention-decoder paradigm attending to query and relevant documents (query-focus), diversity based attention (to combat recurring sequences) and pointer mechanism (to handle rare terms). Worked on sentence ordering and fusion algorithms to improve coherence and readability of generated summary.

Multilingual Text Representation for Speech Synthesis Apr 2019-present

Advisor: Alan Black

  • Text in navigation domain contains named entities in locations that are not in the language that the TTS database is recorded in.
  • Performed character level SVM based Language Identification to classify native and English words in GPS navigational instructions collected from Google Maps API (between 20k routes for 8 languages).
  • Some examples of synthesised navigation instructions

Code-Mixed Question Answering Apr 2019-present

Advisor: Alan Black, Eric Nyberg

  • Developed an end-to-end web based factoid QA system for Code-Mixed languages
  • Due to dearth of annotated data, used lexical level resources like transliteration and translation to achieve an MRR of 0.37 and 0.32 in Hinglish (Hindi+ English) and Tenglish (Telugu+English) respectively.
  • Curated a dataset of around 5k Code-Switched factoid questions and corresponding English answers based on code-switched articles and images.
  • Co-organizer of QA challenge in code-switching


Apple (Siri Understanding) May 2019-Aug 2019

Role: PhD Intern

  • Title: Named Entity Recognition on Code-Switched Data
  • Attentive soft selection of embedding representations between phonemes and words.
  • Transfer learning approach purely monolingual resources to perform NER and we achieved comparable results to supervised learning.

Boeing Aug 2016-Dec 2017

Role: Graduate Research Assistant

University of Pennsylvania May 2015-Aug 2015

Role: Research Intern

  • Title: Medical abstracts classification for Evidence Based Medicine

IISc Bangalore May 2014

Role: Summer visiting student

  • Undergraduate Summer School Program in Department of Computer Science and Automation (CSA) at IISc Bangalore.
  • Won second place in best project selection based on Machine Learning.
Arts and Sports

In my spare time...

HTML5 Bootstrap Template by colorlib.com


My mom is the best cook in the world. No, seriously you have no idea how great my mom is. I am from Hyderabad and I love even the simplest of her rasams more compared to the famous Hyderabadi biryani. She is my friend and also the epitome of patience.

HTML5 Bootstrap Template by colorlib.com


Be it a Saturday or a Sunday, he is always there for you! I love how our own Dumbledore of LTI is witty, wise and sarcastic. There is a lot to learn from your dedication and calm nature. Of course I am not qualified enough to brag about your research capabilities here. Ranging from throught process of how to choose an interesting problem to doing systematic groundwork to pushing boundaries, I am making a novice apprentice effort in learning all this from you.

HTML5 Bootstrap Template by colorlib.com


Just like all kids, I have grown up watching Road Runner, Tom and Jerry and Flinstones. Apart from these, the only other reality shows I have watched are dance competitions. Not having a dance teacher close to my home as a kid did not give the opportunity to be trained in dance and I know that's not an excuse. That did not stop me from learning random dance compositions from my seniors in undergrad and performing. Now and then, I still go through numerous free online lessons to learn traditional dance. The feeling you get after learning a sequence of steps and what they mean is amazing. I know I should be more systematic and sincere in learning this art form...

HTML5 Bootstrap Template by colorlib.com


Well, its raining and my meeting got cancelled. Though the rain had nothing to do with the cancellation, I decided to stay back home. My brain says do something else and I opened YouTube with a movie suggestion. I know my amateur drawing does not resemble her but she happens to be Deepika Padukone.

HTML5 Bootstrap Template by colorlib.com


Sunday evening chit chats with a friend who did not know the story of Pride and Prejudice... While giving a gist of the story, I wanted to read it again myself. Juggling between other activities, I managed to finish it after around 3 days and then I am in the mood to sketch it along with an awesome remark. Misaligning dialog of Mr Darcy with the frame in the drawing I managed to get make a small sketch.

HTML5 Bootstrap Template by colorlib.com


I will let you in on a small secret. This is where I am going to admit that I started drawing a face and smudged the corners. Darkening it more and adding a theme turned out to be this. That is not it. When some of my friends saw this, they told me that they love Sachin Tendulkar. Well, who doesn't? But that's not the point. I have not tried to draw him. Turned out to be a happy accident.

HTML5 Bootstrap Template by colorlib.com


The first time I had to stay away from my home is for my undergraduation. I should not call myself a hostelite as my home was just 12kms away and I would go away to eat my mom's food and sleep in my bed every weekend. Although I still qualify as a psuedo-hostelite since I spent 5 and a half days a week in hostel. These people are my family there. I leanred the power of adjustments, understanding each other and helping each other out. The numerous night outs with both academic stuff and of course random chattering and campus walks discussing topics ranging from the life of a tadpole to deep philosophies of existence are amazing only because of you guys. Complimenting all this is the midnight juice we get from our canteen where I have no idea of the frantic number of chocolate milkshakes that made all this even more blissful.

HTML5 Bootstrap Template by colorlib.com


The term is derived from the Sanskrit name for Lord Krishna. It means 'One who is as clear as a crystal'. I am fascinated by the phonetics of the word. I have heard someone talk about this name in a temple. It also means that babies have no prejudices or hatred. Their personalities reflect from what is written on their crystal clear hearts and minds. It may be true that a soul strengthens by learning from adversities. But that need not be the case always. We are capable of learning through empathy. Placing ourselves in the shoes of another person comes naturally to us. Understanding, respecting and being welcoming of people around us strengthens us and our community that fills our clean canvas we get as babies with a beautiful painting.

HTML5 Bootstrap Template by colorlib.com


One of the early drawings I made was to gift a friend. There is no occasion but I know she loves Federer. She knows a lot of technical details of the sport which I learn from discussing with her. I am one of those who like decorating their rooms with personalized stuff that reminds us of a story. I wanted her room also to have a small amateur piece of sketch of the person she adores. Hence it all began ...

Get in Touch


5501 Gates Hillman Complex, Language Techonologies Institute, 5000 Forbes Avenue