Oh, Dear Stacy! Social Interaction, Elaboration, and Learning with Teachable Agents Abstract:Understanding how children perceive and interact with teachable agents (systems where children learn through teaching a synthetic character embedded in an intelligent tutoring system) can provide insight into the effects of social interaction on learning with intelligent tutoring systems. We describe results from a think-aloud study where children were instructed to narrate their experience teaching Stacy, an agent who can learn to solve linear equations with the student’s help. We found treating her as a partner, primarily through aligning oneself with Stacy using pronouns like you or we rather than she or it significantly correlates with student learning, as do playful face-threatening comments such as teasing, while elaborate explanations of Stacy’s behavior in the third-person and formal tutoring statements reduce learning gains. Additionally, we found that the agent’s mistakes were a significant predictor for students shifting away from alignment with the agent.

Computational Representations of Discourse Practices Across Populations in Task-based Dialogue Abstract:In this work, we employ quantitative methods to describe the discourse practices observed in a direction giving task. We place a special emphasis on comparing differences in strategies between two separate populations and between successful and unsuccessful groups. We isolate differences in these strategies through several novel representations of discourse practices. We find that information sharing, instruction giving, and social feedback strategies are distinct between subpopulations in empirically identifiable ways.

2011

Data-Driven Interaction Patterns: Authority and Information Sharing in Dialogue Abstract: We explore the utility of a computational framework for social authority in dialogue, codified as utterance-level annotations. We first use these annotations at a macro level, compiling aggregate statistics and showing that the resulting features are predictive of group performance in a task-based dialogue. Then, at a micro level, we introduce the notion of an interaction pattern, a formulation of speaker interactions over multiple turns. We use these patterns to characterize situations where speakers do not share information equally. These patterns are found to be more discriminative at this task than similar patterns using standard dialogue acts.

Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations Abstract: This study explored the use of machine learning to automatically evaluate the accuracy of students’ written explanations of evolutionary change. Performance of the Summarization Integrated Development Environment (SIDE) program was compared to human expert scoring using a corpus of 2,260 evolutionary explanations written by 565 undergraduate students in response to two different evolution instruments (the EGALT-F and EGALT-P) that contained prompts that differed in various surface features (such as species and traits). We tested human-SIDE scoring correspondence under a series of different training and testing conditions, using Kappa inter-rater agreement values of greater than 0.80 as a performance benchmark. In addition, we examined the effects of response length on scoring success; that is, whether SIDE scoring models functioned with comparable success on short and long responses. We found that SIDE performance was most effective when scoring models were built and tested at the individual item level and that performance degraded when suites of items or entire instruments were used to build and test scoring models. Overall, SIDE was found to be a powerful and cost-effective tool for assessing student knowledge and performance in a complex science domain.

Missing Something? Authority in Collaborative Learning Abstract:Past research in individual learning settings has shown that student dispositions such as self-efficacy are predictive of learning and other beneficial outcomes, but the relationship is less clear in a collaborative learning environment. This paper explores authoritativeness of stance within a conversation as a social factor influencing learning and related to self-efficacy in a computer-supported collaborative learning setting. Our results indicate that this authoritativeness measure predicts learning, where an individual‟s self- efficacy does not, and that student and partner authoritativeness predicts group self-efficacy. Further research is required to better determine the relationship between conversational authoritativeness, individual dispositions, and learning.

ACODEA: A Framework for the Development of Classification Schemes for Automatic Classification of Online Discussions. Abstract: Research related to online discussions frequently faces the problem of analyzing huge corpora. Natural Language Processing technologies may allow automating the analysis. However, the state-of-the-art in machine learning and text mining approaches yields models that do not transfer well between corpora related to different topics. Also segmenting is a necessary step, but frequently, trained models are very sensitive to the particulars of the segmentation that was used when the model was trained. Therefore, in prior published research on text classification in a CSCL context, the data has been segmented by hand. We discuss work towards overcoming these challenges. We present a framework for developing coding schemes optimized for automatic segmentation and topic independent coding that builds on this segmentation. Our results show that our coding scheme can be fully automated by using a tool called SIDE. Finally, we discuss how fully automated analysis can enable context-sensitive support for collaborative learning.

Recognizing Authority in Dialogue with an Integer Linear Programming Constrained Model Abstract: We present a novel computational formulation of speaker authority in discourse. This notion, which focuses on how speakers position themselves relative to each other in discourse, is first developed into a reliable coding scheme (0.71 agreement between human annotators). We also provide a computational model for automatically annotating text using this coding scheme, using supervised learning enhanced by constraints implemented with Integer Linear Programming. We show that this constrained model's analyses of speaker authority correlates very strongly with expert human judgments (r² coefficient of 0.947).

2010

An Analysis of Perspectives in Interactive Settings Abstract: In this paper we investigate the effect of the context of interaction on the extent to which a contributor's perspective bias is displayed through their lexical choice. We present a series of experiments on political discussion data. Our experiments indicate that (i) when people quote contributors with an opposing view, they tend to quote the words that are less strongly associated with the opposing view. (ii) Nevertheless, in quoting their opponents, the displayed bias of their word distributions shifts towards that of their opponents. (iii) The personal bias of the speaker is displayed most clearly through the words that are not quoted, (iv) although characteristics of the quoted message do have a measurable effect on the words that are included in the contribution. And, finally, (v) posts are influenced by the displayed bias of previous posts in a thread.

Using Feature Construction to Avoid Large Feature Spaces in Text Classification Abstract: Feature space design is a critical part of machine learning. This is an especially difficult challenge in the field of text classification, where an arbitrary number of features of varying complexity can be extracted from documents as a pre-processing step. A challenge for researchers has consistently been to balance expressiveness of features with the size of the corresponding feature space, due to issues with data sparsity that arise as feature spaces grow larger. Drawing on past successes utilizing genetic programming in similar problems outside of text classification, we propose and implement a technique for constructing complex features from simpler features, and adding these more complex features into a combined feature space which can then be utilized by more sophisticated machine learning classifiers. Applying this technique to a sentiment analysis problem, we show encouraging improvement in classification accuracy, with a small and constant increase in feature space size. We also show that the features we generate carry far more predictive power than any of the simple features they contain.

Sentiment Classification using Automatically Extracted Subgraph Features Abstract: In this work, we propose a novel representation of text based on patterns derived from linguistic annotation graphs. We use a subgraph mining algorithm to automatically derive fea- tures as frequent subgraphs from the annotation graph. This process generates a very large number of features, many of which are highly correlated. We propose a genetic programming based approach to feature construction which creates a fixed number of strong classification predictors from these subgraphs. We evaluate the benefit gained from evolved struc- tured features, when used in addition to the bag-of-words features, for a sentiment classification task.

An Interactive Tool for Supporting Error Analysis for Text Mining Abstract: This demo abstract presents an interactive tool for supporting error analysis for text mining, which is situated within the Summarization Integrated Development Environment (SIDE). This freely downloadable tool was designed based on repeated experience teaching text mining over a number of years, and has been successfully tested in that context as a tool for students to use in conjunction with machine learning projects.

2009

Sentence diagram generation using dependency parsing Abstract: Dependency parsers show syntactic relations between words using a directed graph, but comparing dependency parsers is difficult because of differences in theoretical models. We describe a system to convert dependency models to a structural grammar used in grammar education. Doing so highlights features that are potentially overlooked in the dependency graph, as well as exposing potential weaknesses and limitations in parsing models. Our system performs automated analysis of dependency relations and uses them to populate a data structure we designed to emulate sentence diagrams. This is done by mapping dependency relations between words to the relative positions of those words in a sentence diagram. Using an original metric for judging the accuracy of sentence diagrams, we achieve precision of 85%. Multiple causes for errors are presented as potential areas for improvement in dependency parsers.

2007

Optimizing Java Programs Using Generic Types Abstract: Our research involves improving performance of programs written in the Java programming language. By selective specialization of generic types, we enable the compiler to eliminate typecasting, and provide type information to remove dynamic method lookup at runtime. An example of this specialization using Quicksort showed performance improvement of over 20%.

In Press

LightSIDE: Open Source Machine Learning for Text

Gaining Insights from Sociolinguistic Style Analysis for Redesign of Conversational Agent Based Support for Collaborative Learning

A Multivocal Process Analysis of Social Positioning in Study Groups

Linguistic Analysis Methods for Studying Small Groups