Recognizing Rare Social Phenomena in Conversation: Empowerment Detection in Support Group ChatroomsElijah Mayfield, David Adamson, and Carolyn Penstein Rosé
To appear in the Conference of the Association for Computational Linguistics (ACL).
Comparison of Network Heuristics for Understanding Small Groups in Synchronous Collaborative LearningGregory Dyke, Sean Goggins, Elijah Mayfield, and Carolyn Penstein Rosé
In the ACM Conference on Learning Analytics and Knowledge (LAK).
LightSIDE: Open Source Machine Learning for TextElijah Mayfield and Carolyn Rosé
To appear in International Handbook of Automated Essay Assessment.
Linguistic Analysis Methods for Studying Small GroupsIris Howley, Elijah Mayfield, and Carolyn Rosé
To appear in International Handbook of Collaborative Learning.
Discovering Habits of Effective Online Support Group Chatrooms Abstract: For users of online support groups, prior research has suggested that a positive social environment is a key enabler of coping. Typically, demonstrating such claims about social interaction would be approached through the lens of sentiment analysis. In this work, we argue instead for a multifaceted view of emotional state, which incorporates both a static view of emotion (sentiment) with a dynamic view based on the behaviors present in a text. We codify this dynamic view through data annotations marking information sharing, sentiment, and coping efficacy. Through machine learning analysis of these annotations, we demonstrate that while sentiment predicts a user’s stress at the beginning of a chat, dynamic views of efficacy are stronger indicators of stress reduction.Elijah Mayfield, Miaomiao Wen, Mitch Golant, and Carolyn Penstein Rosé
In the ACM Conference on Supporting Group Work (Group).
Hierarchical Conversation Structure Prediction in Multi-Party Chat Abstract: Conversational practices do not occur at a single unit of analysis. To understand the interplay between social positioning, information sharing, and rhetorical strategy in language, various granularities are necessary. In this work we present a machine learning model for multi-party chat which predicts conversation structure across differing units of analysis. First, we mark sentence-level behavior using an information sharing annotation scheme. By taking advantage of Integer Linear Programming and a sociolinguistic framework, we enforce structural relationships between sentence-level annotations and sequences of interaction. Then, we show that clustering these sequences can effectively disentangle the threads of conversation. This model is highly accurate, performing near human accuracy, and performs analysis on-line, opening the door to real-time analysis of the discourse of conversation.Elijah Mayfield, David Adamson, and Carolyn Penstein Rosé
In the SIGDIAL Meeting on Discourse and Dialogue.
The ACODEA Framework: Developing Segmentation and Classification Schemes for Fully Automatic Analysis of Online Discussions Abstract: Research related to online discussions frequently faces the problem of analyzing huge corpora. Natural Language Processing (NLP) technologies may allow automating this analysis. However, the state-of-the-art in machine learning and text mining approaches yields models that do not transfer well between corpora related to different topics. Also, segmenting is a necessary step, but frequently, trained models are very sensitive to the particulars of the segmentation that was used when the model was trained. Therefore, in prior published research on text classification in a CSCL context, the data was segmented by hand. We discuss work towards overcoming these challenges. We present a framework for developing coding schemes optimized for automatic segmentation and context-independent coding that builds on this segmentation. The key idea is to extract the semantic and syntactic features of each single word by using the techniques of part-of-speech tagging and named-entity recognition before the raw data can be segmented and classified. Our results show that the coding on the micro-argumentation dimension can be fully automated. Finally, we discuss how fully automated analysis can enable context-sensitive support for collaborative learning.Jin Mu, Karsten Stegmann, Elijah Mayfield, Carolyn Penstein Rosé, and Frank Fischer
In the International Journal of Computer Supported Collaborative Learning (ijCSCL).
Historical Analysis of Legal Opinions with a Sparse Mixed-Effects Latent Variable Model Abstract: We propose a latent variable model to enhance historical analysis of large corpora. This work extends prior work in topic modelling by incorporating metadata, and the interactions between the components in metadata, in a general way. To test this, we collect a corpus of slavery-related United States property law judgements sampled from the years 1730 to 1866. We study the language use in these legal cases, with a special focus on shifts in opinions on controversial topics across different regions. Because this is a longitudinal data set, we are also interested in understanding how these opinions change over the course of decades. We show that the joint learning scheme of our sparse mixed-effects model improves on other state-of-the-art generative and discriminative models on the region and time period identification tasks. Experiments show that our sparse mixed-effects model is more accurate quantitatively and qualitatively interesting, and that these improvements are robust across different parameter settings.William Yang Wang, Elijah Mayfield, Suresh Naidu, and Jeremiah Dittmar
In the Proceedings of the Association for Computational Linguistics (ACL).
Group Composition and Intelligent Dialogue Tutors for Impacting Students' Academic Self-Efficacy Abstract:In this paper, we explore using an intelligent dialogue tutor to influence student academic self-efficacy, as well as its interaction with group selfefficacy composition in a dyadic learning environment. We find providing additional tutor prompts encouraging to students to participate in discussion may have unexpected negative effects on self-efficacy, especially on students with low self-efficacy scores who have partners with low self-efficacy scores.Iris Howley, David Adamson, Gregory Dyke, Elijah Mayfield, Jack Beuth, and Carolyn Penstein Rosé
In the International Conference on Intelligent Tutoring Systems (ITS).
Oh, Dear Stacy! Social Interaction, Elaboration, and Learning with Teachable Agents Abstract:Understanding how children perceive and interact with teachable agents (systems where children learn through teaching a synthetic character embedded in an intelligent tutoring system) can provide insight into the effects of social interaction on learning with intelligent tutoring systems. We describe results from a think-aloud study where children were instructed to narrate their experience teaching Stacy, an agent who can learn to solve linear equations with the student’s help. We found treating her as a partner, primarily through aligning oneself with Stacy using pronouns like you or we rather than she or it significantly correlates with student learning, as do playful face-threatening comments such as teasing, while elaborate explanations of Stacy’s behavior in the third-person and formal tutoring statements reduce learning gains. Additionally, we found that the agent’s mistakes were a significant predictor for students shifting away from alignment with the agent.Amy Ogan, Samantha Finkelstein, Elijah Mayfield, Claudia D'Adamo, Noboru Matsuda, and Justine Cassell
In the International Conference on Human Factors in Computer Systems (CHI).
Computational Representations of Discourse Practices Across Populations in Task-based Dialogue Abstract:In this work, we employ quantitative methods to describe the discourse practices observed in a direction giving task. We place a special emphasis on comparing differences in strategies between two separate populations and between successful and unsuccessful groups. We isolate differences in these strategies through several novel representations of discourse practices. We find that information sharing, instruction giving, and social feedback strategies are distinct between subpopulations in empirically identifiable ways.Elijah Mayfield, David Adamson, Alexander I. Rudnicky, and Carolyn Penstein Rosé
In the International Conference on Intercultural Collaboration (ICIC).
Data-Driven Interaction Patterns: Authority and Information Sharing in Dialogue Abstract: We explore the utility of a computational framework for social authority in dialogue, codified as utterance-level annotations. We first use these annotations at a macro level, compiling aggregate statistics and showing that the resulting features are predictive of group performance in a task-based dialogue. Then, at a micro level, we introduce the notion of an interaction pattern, a formulation of speaker interactions over multiple turns. We use these patterns to characterize situations where speakers do not share information equally. These patterns are found to be more discriminative at this task than similar patterns using standard dialogue acts.Elijah Mayfield, Michael Garbus, David Adamson, and Carolyn Penstein Rosé
In the AAAI Fall Symposium on Building Representations of Common Ground with Intelligent Agents.
Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations Abstract: This study explored the use of machine learning to automatically evaluate the accuracy of students’ written explanations of evolutionary change. Performance of the Summarization Integrated Development Environment (SIDE) program was compared to human expert scoring using a corpus of 2,260 evolutionary explanations written by 565 undergraduate students in response to two different evolution instruments (the EGALT-F and EGALT-P) that contained prompts that differed in various surface features (such as species and traits). We tested human-SIDE scoring correspondence under a series of different training and testing conditions, using Kappa inter-rater agreement values of greater than 0.80 as a performance benchmark. In addition, we examined the effects of response length on scoring success; that is, whether SIDE scoring models functioned with comparable success on short and long responses. We found that SIDE performance was most effective when scoring models were built and tested at the individual item level and that performance degraded when suites of items or entire instruments were used to build and test scoring models. Overall, SIDE was found to be a powerful and cost-effective tool for assessing student knowledge and performance in a complex science domain.Ross H. Nehm, Minsu Ha, and Elijah Mayfield
In the Journal of Science Education and Technology (JOSTE).
Missing Something? Authority in Collaborative Learning Abstract:Past research in individual learning settings has shown that student dispositions such as self-efficacy are predictive of learning and other beneficial outcomes, but the relationship is less clear in a collaborative learning environment. This paper explores authoritativeness of stance within a conversation as a social factor influencing learning and related to self-efficacy in a computer-supported collaborative learning setting. Our results indicate that this authoritativeness measure predicts learning, where an individual‟s self- efficacy does not, and that student and partner authoritativeness predicts group self-efficacy. Further research is required to better determine the relationship between conversational authoritativeness, individual dispositions, and learning.Iris Howley, Elijah Mayfield, and Carolyn Rosé
In the Proceedings of the Computer-Supported Collaborative Learning Conference (CSCL 2011).
ACODEA: A Framework for the Development of Classification Schemes for Automatic Classification of Online Discussions. Abstract: Research related to online discussions frequently faces the problem of analyzing huge corpora. Natural Language Processing technologies may allow automating the analysis. However, the state-of-the-art in machine learning and text mining approaches yields models that do not transfer well between corpora related to different topics. Also segmenting is a necessary step, but frequently, trained models are very sensitive to the particulars of the segmentation that was used when the model was trained. Therefore, in prior published research on text classification in a CSCL context, the data has been segmented by hand. We discuss work towards overcoming these challenges. We present a framework for developing coding schemes optimized for automatic segmentation and topic independent coding that builds on this segmentation. Our results show that our coding scheme can be fully automated by using a tool called SIDE. Finally, we discuss how fully automated analysis can enable context-sensitive support for collaborative learning.Jin Mu, Karsten Stegmann, Elijah Mayfield, Carolyn Rosé, and Frank Fischer
In the Proceedings of CSCL 2011.
Recognizing Authority in Dialogue with an Integer Linear Programming Constrained Model Abstract: We present a novel computational formulation of speaker authority in discourse. This notion, which focuses on how speakers position themselves relative to each other in discourse, is first developed into a reliable coding scheme (0.71 agreement between human annotators). We also provide a computational model for automatically annotating text using this coding scheme, using supervised learning enhanced by constraints implemented with Integer Linear Programming. We show that this constrained model's analyses of speaker authority correlates very strongly with expert human judgments (r2 coefficient of 0.947).Elijah Mayfield and Carolyn Penstein Rosé
In ACL 2011.
An Analysis of Perspectives in Interactive Settings Abstract: In this paper we investigate the effect of the context of interaction on the extent to which a contributor's perspective bias is displayed through their lexical choice. We present a series of experiments on political discussion data. Our experiments indicate that (i) when people quote contributors with an opposing view, they tend to quote the words that are less strongly associated with the opposing view. (ii) Nevertheless, in quoting their opponents, the displayed bias of their word distributions shifts towards that of their opponents. (iii) The personal bias of the speaker is displayed most clearly through the words that are not quoted, (iv) although characteristics of the quoted message do have a measurable effect on the words that are included in the contribution. And, finally, (v) posts are influenced by the displayed bias of previous posts in a thread.Dong Nguyen, Elijah Mayfield, and Carolyn Penstein Rosé
In the Workshop on Social Media Analysis at KDD 2010.
Using Feature Construction to Avoid Large Feature Spaces in Text Classification Abstract: Feature space design is a critical part of machine learning. This is an especially difficult challenge in the field of text classification, where an arbitrary number of features of varying complexity can be extracted from documents as a pre-processing step. A challenge for researchers has consistently been to balance expressiveness of features with the size of the corresponding feature space, due to issues with data sparsity that arise as feature spaces grow larger. Drawing on past successes utilizing genetic programming in similar problems outside of text classification, we propose and implement a technique for constructing complex features from simpler features, and adding these more complex features into a combined feature space which can then be utilized by more sophisticated machine learning classifiers. Applying this technique to a sentiment analysis problem, we show encouraging improvement in classification accuracy, with a small and constant increase in feature space size. We also show that the features we generate carry far more predictive power than any of the simple features they contain.Elijah Mayfield and Carolyn Penstein Rosé
In the Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2010).
Sentiment Classification using Automatically Extracted Subgraph Features Abstract: In this work, we propose a novel representation of text based on patterns derived from linguistic annotation graphs. We use a subgraph mining algorithm to automatically derive fea- tures as frequent subgraphs from the annotation graph. This process generates a very large number of features, many of which are highly correlated. We propose a genetic programming based approach to feature construction which creates a fixed number of strong classification predictors from these subgraphs. We evaluate the benefit gained from evolved struc- tured features, when used in addition to the bag-of-words features, for a sentiment classification task.Shilpa Arora, Elijah Mayfield, Carolyn Penstein Rosé, and Eric Nyberg
In the Workshop on Emotion in Text at the North American Association for Computational Linguistics (NAACL 2010).
An Interactive Tool for Supporting Error Analysis for Text Mining Abstract: This demo abstract presents an interactive tool for supporting error analysis for text mining, which is situated within the Summarization Integrated Development Environment (SIDE). This freely downloadable tool was designed based on repeated experience teaching text mining over a number of years, and has been successfully tested in that context as a tool for students to use in conjunction with machine learning projects.Elijah Mayfield and Carolyn Penstein Rosé
In the Demonstration Session of NAACL 2010.
Sentence diagram generation using dependency parsing Abstract: Dependency parsers show syntactic relations between words using a directed graph, but comparing dependency parsers is difficult because of differences in theoretical models. We describe a system to convert dependency models to a structural grammar used in grammar education. Doing so highlights features that are potentially overlooked in the dependency graph, as well as exposing potential weaknesses and limitations in parsing models. Our system performs automated analysis of dependency relations and uses them to populate a data structure we designed to emulate sentence diagrams. This is done by mapping dependency relations between words to the relative positions of those words in a sentence diagram. Using an original metric for judging the accuracy of sentence diagrams, we achieve precision of 85%. Multiple causes for errors are presented as potential areas for improvement in dependency parsers.Elijah Mayfield
In the Student Research Workshop at ACL 2009.
Optimizing Java Programs Using Generic Types Abstract: Our research involves improving performance of programs written in the Java programming language. By selective specialization of generic types, we enable the compiler to eliminate typecasting, and provide type information to remove dynamic method lookup at runtime. An example of this specialization using Quicksort showed performance improvement of over 20%.Eli Mayfield, J. Kyle Roth, Daniel Selifonov, Nathan Dahlberg, Elena Machkasova
In the 22nd ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2007).
Gaining Insights from Sociolinguistic Style Analysis for Redesign of Conversational Agent Based Support for Collaborative LearningIris Howley, Rohit Kumar, Elijah Mayfield, Gregory Dyke, and Carolyn Rosé
To appear in Productive Multivocality in the Analysis of Group Interactions.
A Multivocal Process Analysis of Social Positioning in Study GroupsIris Howley, Elijah Mayfield, and Carolyn Rosé
To appear in Productive Multivocality in the Analysis of Group Interactions.