Online Interventions: Summarization and Conversational Agent Technology Triggered through Automatic Discourse Analysis

With the recent press given to online education and increasing enrollment in massively open online courses, the need for scaling up quality computer-mediated educational experiences has never been so urgent. Current offerings provide excellent materials including video lectures, exercises, and some forms of discussion opportunities. The biggest limitations are related to the human side of effective educational experiences including personal contact with instructors and the cohort experience. In the past decade, special concern has centered on students’ inability to communicate effectively, negotiate ideas, or engage in other aspects of collaborative problem-solving activity. These concerns prompt my research on interventions that improve the instructional value of online collaborative learning experiences. In recognition of the value of the work my group has done in this area, I was invited to participate as part of a team of authors in the formulation of a multi-disciplinary research agenda for online education that will be completed in Summer of 2013 and submitted to the President of the United States. My participation in this process began as an invited panelist and breakout session leader at the Workshop on Multidisciplinary Research for Online Education ( MunROE), sponsored by the Computing Community Consortium, in February 2013.

Impact: Seminal Ideas and Findings. At least a decade of research, including my own, shows that students can benefit from their interactions in learning groups when automated support is provided, especially interactive and context sensitive support. Until recently, the state-of-the-art in computer supported collaborative learning has consisted of static forms of support, such as structured interfaces, prompts, and assignment of students to scripted roles. This earlier work, referred to as scripted collaboration, has been a major focus of the field of Computer Supported Collaborative Learning in the past decade, and despite its limitations, has produced numerous demonstrations of its effectiveness in improving collaborative learning. In contrast, dynamic forms of collaboration support “listen in” on student conversations in search of important events that present opportunities for discouraging dysfunctional behavior or encouraging positive behavior using automated analysis of collaborative learning processes. My group is widely recognized as playing a major role in enabling this paradigm shift, especially as a result of demonstrations that dynamic script based support leads to improvements in learning over otherwise equivalent static forms of support (Kumar et al., 2007). Its value has been recognized in award and award nominations for my group’s work at conferences such as ACM SIGCHI, AI in Education, the International Conference of the Learning Sciences, and Computer Supported Collaborative Learning. As evidence that this shift has spread beyond my research group, three workshops on the topic of dynamic support for collaborative learning have been held in the past three years, with another in progress for AI in Education this summer and an in progress special issue of the International Journal of AI in Education co-edited by my former PhD student Rohit Kumar.

Impact: Tools and Resources for Other Researchers. A major aspect of this research began with my former PhD student Rohit Kumar’s development of the Basilica architecture, which facilitates rapid development of multi-party collaboration environments. A recently published journal article (Kumar & Rosé, 2011) describes a series of collaborative environments developed through this architecture using reusable components. PhD student David Adamson has an improved version of the architecture, referred to as Bazaar, which is freely available online and includes instructional materials that enable others to learn to use it quickly. It has been used for the past two years both at the Pittsburgh Science of Learning Center summer school and the Internship Program in Technology Supported Education Winter School. Experience at both venues demonstrates that even undergraduate students are able to learn to use Bazaar and build prototype dynamic support for collaborative learning within one or two weeks.

Impact: Change in research, teaching, and assessment practices. As a demonstration of the impact of the dynamic support technology developed by my group in the Classroom Discourse arena, in collaboration with Lauren Resnick at the Learning Research and Development Center, I have been investigating the role online collaborative activities can play in preparing high school students for whole class teacher led discussions as part of a two year professional development program. A series of studies evaluating intelligent conversational agents employing teacher facilitation practices from the Classroom Discourse literature demonstrates their significant positive impact on learning and interaction during collaborative learning (e.g., Adamson et al., 2013; Dyke et al., 2013). Furthermore, using technology supported analysis of classroom discussions collected over the two year program, we have determined that preparing students prior to teacher lead discussions by means of conversational agent facilitated online collaborative learning activities has an enabling effect on teacher uptake of productive classroom facilitation practices (effect size 1.7 standard deviations) (Clarke et al., 2013). Building on these successes, I am now engaged in a partnership with Lauren Resnick and the Institute for Learning to investigate how to leverage these findings in the context of large scale dissemination of APT professional development through Coursera. A current offering of instruction on APT through Coursera has over 20,000 enrolled for participation starting in June of 2013. Based on the finding about the important role of student preparation in facilitating teacher uptake, we are exploring opportunities for dissemination of my group’s work on automated support for small group learning as part of the instruction offered in this course in a future instantiation. In the near term, an important focus of our active collaboration surrounding this Coursera course in Summer of 2013 will be using automated analysis of threaded discussions with 20,000 participants to enable human instructors to support that interaction.

As a final exemplar of real world impact of the growing area of collaborative learning supported by intelligent conversational agents, this paradigm will be used as a key component in the Collaborative Problem Solving assessment under development by the Program for International Student Assessment (PISA), with planned administration 2015. As part of this assessment development effort, a workshop has been organized for CSCL 2013 where Art Graesser, leader of the task force, will present the draft assessment plan, and a panel of experts, on which I have been invited to serve, will offer feedback. This invitation signifies that my group’s work contributes to change in thinking about international assessment practices. PISA is a worldwide study conducted by the Organization for Economic Co-operation and Development, with the purpose of informing development of educational policies worldwide. Its findings are used to investigate what causes difference in achievement across nations.

Carolyn Penstein Rose (cprose@cs.cmu.edu)/ Carnegie Mellon University