Educational games have become an established paradigm of instructional practice; however, there is still much to be learned about how to design games to be the most beneficial for learners. An important consideration when designing an educational game is whether there is good alignment between its content goals and the instructional behaviors it makes in order to reinforce those goals. Existing methods for measuring alignment are labor intensive and use complex auditing procedures making it difficult to define and evaluate this alignment in order to guide the educational game design process. This thesis explores a way to operationalize this concept of alignment and demonstrates an analysis technique that can help educational game designers measure the alignment of both current educational game designs and prototypes of future iterations.
In my work, I explore the use of Replay Analysis, a novel technique that uses in-game replays of player sessions as a data source to support analysis. This method can be used to capture gameplay experience for the evaluation of alignment as well as other forms of analysis. The majority of my work has been performed in the context of RumbleBlocks, an educational game that teaches basic structural stability and balance concepts to young children. Using Replay Analysis, I leveraged replay data during a formative evaluation of RumbleBlocks to highlight that the game likely possesses misalignment in how it teaches some concepts of stability to players. The results led to suggestions for various design iterations.
Through exploring these design iterations, I further demonstrate an extension of Replay Analysis called Projective Replay Analysis, which uses recorded student replay data in prototypes of new versions of a game to predict whether the new version would be an improvement. I implemented two forms of projective replay: Literal Projective Replay, which uses a naïve player model that replays past player actions through a new game version exactly as they were originally recorded; and Flexible Projective Replay, which augments the process with an AI player model that uses prior player actions as training data to learn to play through a new game. To assess the validity of this method of game evaluation, I performed a new replication study of the original formative evaluation to validate whether the conclusions reached through virtual methods correspond to those reached in a normal playtesting situation. Ultimately, my findings were that Literal Projective Replay was able to predict a new and unanticipated misalignment with the game, but Flexible Projective Replay, as currently implemented, has limitations in its ability to explore new game spaces.
This work makes contributions to the fields of human-computer interaction, by exploring the benefits and limitations of different replay paradigms for the evaluation of interactive systems; learning sciences, by establishing a novel operationalization of alignment for instructional moves; and educational game design, by providing a model for using Projective Replay Analysis to guide the iterative development of an educational game.
Vincent Aleven (Chair)
Jessica Hammer (HCII/ETC)
Sharon Carver (Psychology)
Jesse Schell (ETC)
In the classroom, teachers make use of different combinations of social planes (e.g., individual, collaborative) to support learning. However, little is known about the complementary strengths of individual and collaborative learning or how to combine them so that they are more effective than either social plane alone. One roadblock to this investigation is an ability to orchestrate, or manage, more complex, but theoretically effective, combinations of collaborative and individual learning in the classroom. Prior research has created orchestration tools that support the planning and real-time management of classroom activities, which reduces the cognitive load and time needed for instructors to support the activity, allowing for more complex activities to become more manageable. Current orchestration tools do not, however, support a wide range of combinations of collaborative and individual learning activities in a flexible manner. To fully investigate the combinations of collaborative and individual learning, orchestration tools need to be developed that can support the researcher in a way that can be integrated into the classroom by accounting for teachers’ values.
My thesis work addresses two related goals. First, my work addresses the questions: Do collaborative and individual learning have complementary strengths and is a combination of the two social planes better than either alone? In my work, I developed an intelligent tutoring system (ITS) to support collaborative and individual learning. Through three studies, using this ITS, with over 500 4th and 5th grade students, I demonstrate that a collaborative ITS can be used to effectively support learning with elementary school students and that a combination of collaborative and individual learning is more effective than either alone. However, my studies did not find any support for complementary strengths and many other combinations of social planes are left to investigate. Additionally, during my experiments, I encountered challenges in orchestration that, along with the need to research more complex combinations of collaborative and individual learning, informed the next steps of my research.
The second question my thesis work addresses is: How does an orchestration tool that supports researchers in exploring this space need to be designed to align with teachers’ values for easy integration in the classroom? Specifically, I aimed to support fluid transitions between social planes where students do not all have to be working in sync, which is not currently supported in existing orchestration tools. To support the orchestration tool design, I present a framework that structures the space that a researcher can explore when combining individual and collaborative learning. The framework can act as the set of requirements to be met in the orchestration tool from the point of the researcher as well as a lens to analyze and design combined social plane activities. As a first step towards supporting fluid transitions as laid out in the framework, I present a set of statistical models that extend domain-level individual modeling into the space of collaborative environments. Finally, I developed an orchestration prototype built around my framework that can be used as a research tool to further explore combined collaborative and individual spaces. To develop the tool to be successful within the classroom, I worked with teachers through a co-design process and validation of the prototype to incorporate their values into the tool.
Taken together, my dissertation has six primary contributions. My dissertation contributes to the learning sciences through advancing our knowledge of (1) the strengths of collaborative and individual learning, although I did not find any complementary strengths, and (2) if a combination is better than either alone, which I did find support for. It contributes to educational technology through (3) the design of an effective ITS that supports collaborative and individual learning for fractions and educational data mining through (4) the advancement of models that can more accurately predict individual learning within a collaborative setting than the existing individual models. Finally, it contributes to computer supported collaborative learning and human-computer interaction through (5) a framework, which provides a lens for designing and analyzing combined collaborative and individual learning spaces, and (6) an orchestration prototype that supports fluid transitions between social planes in a way that can be a useful to both researchers and teachers in the classroom.Thesis CommitteeVincent Aleven (Co-Chair)Nikol Rummel, Co-Chair (Psychology, RUB; HCII, CMU)John ZimmermanPierre Dillenbourg (School of Computer and Communication Science, EPFL)
Feedback is an essential component of the learning process, but in fields like computer science, which have rapidly increasing class sizes, it can be difficult to provide feedback to students at scale. Intelligent tutoring systems can provide personalized feedback to students automatically, but they can take large amounts of time and expert knowledge to build, especially when determining how to give students hints. Data-driven approaches can be used to provide personalized next-step hints automatically and at scale, by mining previous students’ solutions.
I have created ITAP, the Intelligent Teaching Assistant for Programming, which automatically generates next-step hints for students in basic Python programming assignments. ITAP is composed of three stages: canonicalization, where a student's code is transformed to an abstracted representation; path construction, where the closest correct state is identified and a series of edits to that goal state are generated; and reification, where the edits are transformed back into the student's original context. With these techniques, ITAP can generate next-step hints for 100% of student submissions, and can even chain these hints together to generate a worked example. Early analysis showed that hints could be used in practice problems in a real classroom environment, but also demonstrated that students' relationships with hints and help-seeking were complex and required deeper investigation.
In my thesis work, I surveyed and interviewed students about their experience with help-seeking and using feedback, and found that students wanted more detail in hints than was initially provided. To determine how hints should be structured, I ran a usability study with programmers at varying levels of knowledge, where I found that more novice students needed much higher levels of content and detail in hints than was traditionally given. I also found that examples were commonly used in the learning process, and could serve an integral role in the feedback provision process. I then ran a randomized control trial experiment to determine the effect of next-step hints on learning and time-on-task in a practice session, and found that having hints available resulted in students spending 13.7% less time during practice while achieving the same learning results as the control group. Finally, I used the data collected during these experiments to measure ITAP’s performance over time, and found that generated hints became more optimal as data was added to the system.
My dissertation has contributed to the fields of computer science education, learning science, human-computer interaction, and data-driven tutoring. For computer science education I have created ITAP, which can serve as a practice resource for future programming students during the learning process. In the learning sciences, I have replicated the expertise effect by finding that more expert programmers want less detail in hints than novice programmers; this finding is especially important as it implies that programming teachers may provide novices with less assistance than they need. I have contributed to the literature on human-computer interaction by identifying multiple possible representations of hint messages, and analyzing how users react to and learn from these different formats during program debugging. Finally, I have contributed to the new field of data-driven tutoring by establishing that it is possible to always provide students with next-step hints, even without a starting dataset beyond the instructor solution, and by demonstrating that those hints can be improved automatically over time.
Ken Koedinger (Chair, HCII/Psych)
Brad Myers (HCII/ISR)
Vincent Aleven (HCII)
Sharon Carver (Psych)
Tiffany Barnes (North Carolina State University)
Programming skills have been considered more important than ever for people to thrive in the age of the digital economy. Meanwhile, computers have become ubiquitous in our life and work, and the way they are programmed is in need of fundamental improvements. In this talk, I introduce research on creating toolkits and integrated development environments that help people to create, edit, and make use of programs, and discuss the future of programming.
Jun Kato is a Human-Computer Interaction researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. He has focused on improving Programming Experience (PX) by creating toolkits and integrated development environments. He has worked for Microsoft and Adobe Research and received a Ph.D. from The University of Tokyo under the supervision of Prof. Takeo Igarashi in 2014.
Mobile and ubiquitous computing research has led to new techniques for cheaply, accurately, and continuously collecting data on human behavior that include detailed measurements of physical activities, social interactions and conversations, sleep quality and duration and more. Continuous and unobtrusive sensing of behaviors has tremendous potential to support the lifelong management of mental health by: (1) acting as an early warning system to detect changes in mental well-being, (2) delivering context-aware, personalized micro-interventions to patients when and where they need them, and (3) by significantly accelerating patient understanding of their illness. In this presentation, I will give an overview of our work on turning sensor-enabled mobile devices into well-being monitors and instruments for administering real-time/real-place interventions.
Tanzeem Choudhury is an associate professor in Computing and Information Sciences at Cornell University and a co-founder of HealthRhythms. At Cornell, she directs the People-Aware Computing group, which works on inventing the future of technology-assisted wellbeing. Tanzeem received her PhD from the Media Laboratory at MIT. Tanzeem was awarded the MIT Technology Review TR35 award, NSF CAREER award and a TED Fellowship. Follow the group's work on twitter @pac_cornell
Despite substantial effort made by the usable security community at facilitating the use of recommended security systems and behaviors, much security advice is ignored and many security systems are underutilized. I argue that this disconnect can partially be explained by the fact that security behaviors have myriad unaccounted for social consequences. For example, by using two- factor authentication, one might be perceived as “paranoid”. By encrypting an e-mail correspondence, one might be perceived as having something to hide. Yet, to date, little theoretical work in usable security has applied theory from social psychology to understand how these social consequences affect people’s security behaviors. Likewise, little systems work in usable security has taken social factors into consideration.
To bridge these gaps in literature and practice, I begin to build a theory of social cybersecurity and apply these theoretical insights to create systems that encourage better cybersecurity behaviors. First, through a series of interviews, surveys and a large-scale analysis of how security tools diffuse through the social networks of 1.5 million Facebook users, I constructed empirical models of how social influences affect the adoption of recommended security behaviors and systems. In so doing, I provide some of the first direct empirical evidence that security behaviors are strongly driven by social influences, and that the design of a security system strongly influences its potential for social spread. Specifically, security systems that are more observable, inclusive, and stewarded are positively affected by social influence, while those that are not are negatively affected by social influence.
Based on these empirical results, I put forth two prescriptions: (i) creating socially grounded interface “nudges” that encourage better cybersecurity behaviors, and (ii) designing new, more socially intelligent end-user facing security systems. As an example of a social “nudge”, I designed a notification that informs Facebook users that their friends use optional security systems to protect their own accounts. In an experimental evaluation with 50,000 Facebook users, I found that this social notification was significantly more effective than a non-social control notification at attracting clicks to improve account security and in motivating the adoption of optional security tools. As an example of a socially intelligent cybersecurity system, I designed Thumprint: an inclusive authentication system that authenticates and identifies individual group members of a small, local group through a single, shared secret knock. Through my evaluations, I found that Thumprint is reasonably resilient to casual but motivated adversaries and that it can reliably differentiate multiple group members who share the same secret knock. Taken together, these systems point towards a future of socially intelligent cybersecurity that encourages better security behaviors.
Concretely, this thesis provides the following contributions: (i) an initial theory of social cybersecurity, developed from both observational and experimental work, that explains how social influences affect security behaviors; (ii) a set of design recommendations for creating socially intelligent security systems that encourage better cybersecurity behaviors; (iii) the design, implementation and comprehensive evaluation of two such systems that leverage these design recommendations; and (iv) a reflection on how the insights uncovered in this work can be utilized alongside broader design considerations in HCI, security and design to create an infrastructure of useful, usable and socially intelligent cybersecurity systems.
Jason Hong (Chair)
Laura Dabbish (Co-chair)
J.D. Tygar (University of California, Berkeley)
11:30 am — Introductions to the BHCI program
11:45 am — Refugees: Cultural Orientation for Refugees & Immigrants
11:57 am — ACS: Canceer Survivors Network: BUilding a New Community Interface
12:09 pm — ATV: Interface for an Off-road Autonomous Vehicle
12:21 pm — Refining RoboTutor: machine-aided design iteration of an intelligent tutor
12:33 pm — Vocab: Promoting Children's Vocabulary Growth
12:45 pm — Animal Cop: Reporting Animal Abuse
1:00 pm — HCII Lunch - Wean Patio
1:00 pm — Vocab: Promiting Children's Vocabulary Growth
1:30 pm — ACS: Cancer Survicors Network: Building a New Community Interface
2:00 pm — ATV: Interface for an Off-road Autonomous Vehicle
2:30 pm — Refugees: Cultural Orientation for Refugees & Immigrants
3:00 pm — Refining RoboTutor: Machine-aided Design Iteration of an Intelligent Tutor
3:30 pm — Animal Cop: Reporting Animal Abuse
SCS community welcomed.
Across a wide variety of digital devices, users create, consume, and disseminate large quantities of information. While data sometimes look like a spreadsheet or network diagram, more often for everyday users their data look more like an Amazon search page, the line-up for a fantasy football team, or a set of Yelp reviews. However, interpreting these kinds of data remains a difficult task even for experts since they often feature soft or unknown constraints (e.g. "I want some Thai food, but I also want a good bargain") across highly multidimensional data (i.e. rating, reviews, popularity, proximity). Existing technology is largely optimized for users with hard criteria and satisfiable constraints, and consumer systems often use representations better suited for browsing than sensemaking.
In this thesis I explore ways to support soft constraint decision-making and exploratory data analysis by giving users tools that show fine-grained features of the data while at the same time displaying useful contextual information. I describe approaches for representing collaborative content history and working behavior that reveal both individual and group/dataset level features. Using these approaches, I investigate general visualizations that utilize physics to help even inexperienced users find small and large trends in multivariate data. I describe the transition of physics-based visualization from the research space into the commercial space through a startup company, and the insights that emerged both from interviews with experts in a wide variety of industries during commercialization and from a comparative lab study. Taking one core use case from commercialization, consumer search, I develop a prototype, Fractal, which helps users explore and apply constraints to Yelp data at a variety of scales by curating and representing individual-, group-, and dataset-level features. Through a user study and theoretical model I consider how the prototype can best aide users throughout the sensemaking process.
My dissertation further investigates physics-based approaches for represent multivariate data, and explores how the user's exploration process itself can help dynamically to refine the search process and visual representation. I demonstrate that selectively representing points using clusters can extend physics-based visualizations across a variety of data scales, and help users make sense of data at scales that might otherwise overload them. My model provides a framework for stitching together a model of user interest and data features, unsupervised clustering, and visual representations for exploratory data visualization. The implications from commercialization are more broad, giving insight into why research in the visualization space is/isn't adopted by industry, a variety of real-world use cases for multivariate exploratory data analysis, an index of common data visualization needs in industry, and even some helpful tips for research-inspired startups.
Aniket Kittur (Chair)
John Stasko (Georgia Institute of Technology)