Gaming the System
Project Description
Some students who use interactive learning environments "game the system", attempting to succeed in an educational task by systematically taking advantage of properties and regularities in the system used to complete that task, rather than by thinking through the material.
Gaming the System has been observed in a variety of computerized learning environments (and non-computerized learning environments), including educational games, graded-participation newsgroups, and collaborative learning environments.
In Cognitive Tutor software, gaming the system includes systematic guessing, and repeatedly asking for an additional hint until the software gives the student the answer.
In 2003, using human observations of student behavior in cognitive tutor classrooms, my colleagues and I determined that students who game the system learn significantly less than other students. We have replicated this finding across a variety of classrooms and tutor lessons.
Data mining showed that gaming students split into two groups. One group appears to learn reasonably well. The other group shows extremely poor learning. The group that shows poor learning appears to repeatedly game the system on specific poorly-known skills. The other group appears to game the system more sporadically and on easier skills, as part of a broader pattern of focusing time and attention on skills they know poorly.
We have developed a detector of gaming which could accurately identify which students gamed, and when they gamed. We have also validated that this detector can transfer between individual students, cohorts of students, and specific curricular material.
We have also conducted several studies to determine what motivations underlying the choice to game the system. For instance, three studies by our research group showed that several student traits/characteristics are statistically significantly associated with the choice to game the system -- however, these explanations do not explain the majority of the variance in students' gaming behavior. Two data mining studies suggest that, at maximum, stable student traits explain 16-18% of the variance in student gaming behavior.
By contrast, differences between student states and differences between tutor lessons appear to be much larger explanations for why students game the system. One study by our research group indicates that students who are feeling boredom or confusion are twice as likely as other students to begin gaming in the next 60 seconds. Two data mining studies suggest that differences between tutor lessons (i.e. individual pieces of software being used) explains 18-55% of the variance in student gaming behavior.
In other words, knowing exactly where in a learning curriculum a student is (including interface factors) and knowing what a student's affective state appear to be more powerful predictors of student gaming behavior than knowing the student's stable traits (though stable traits are still, themselves, predictors of gaming behavior).
My colleagues and I have developed multiple systems which respond to gaming the system, which will be the subject of an upcoming journal submission. We have developed a system which uses a software agent, Scooter the Tutor, to respond to when students game the system. Scooter uses emotional expressions to signal to the student and their teacher that the student is gaming; Scooter also gives supplementary exercises on exactly the material that students bypass by gaming. Scooter reduces the frequency of gaming the system; Scooter's supplementary exercises have been shown to significantly improve gaming students' learning. Scooter was awarded a best paper award in 2006. My colleagues Neil Heffernan and Jason Walonoski have developed a system to reduce gaming the system and improve learning, using visualizations of student behavior over time and at-the-moment text prompts about gaming. Their work won a best poster award in 2006. My colleague Ivon Arroyo and her research group have also developed a system that reduces gaming the system, improves learning, and improves students' attitudes towards learning, through giving regular tips on how to use tutoring software more effectively, with visualizations of student gaming over time.
Publications
Responding to Gaming the System
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference
on Intelligent Tutoring Systems, 392-401. [Won Best Paper Award] [pdf]
Baker, R.S., Corbett, A.T., Koedinger, K.R. (2006) Responding
to Problem Behaviors in Cognitive Tutors: Towards Educational Systems
Which Support All Students . National Association for the
Dually Diagnosed (NADD) Bulletin, 9 (4), 70-75.
[draft pdf]
Roll, I., Aleven, V., McLaren, B.M., Ryu, E., Baker, R.S.J.d., Koedinger, K.R. (2006) The Help Tutor: Does Metacognitive Feedback Improve Students' Help-Seeking Actions, Skills, and Learning? Proceedings of the 8th International Conference on Intelligent Tutoring Systems, 360-369. [pdf]
Detecting Gaming the System
Baker, R.S.J.d., Corbett, A.T., Roll, I., Koedinger, K.R. (to appear) Developing a Generalizable Detector of When Students Game the System To appear in User Modeling and User-Adapted Interaction. [draft pdf]
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Roll, I. (2006)
Generalizing Detection of Gaming the System Across a Tutoring Curriculum. Proceedings of the 8th International Conference on Intelligent
Tutoring Systems, 402-411. [pdf]
Baker, R.S., Corbett, A.T., Koedinger, K.R. (2004)
Detecting Student Misuse of Intelligent Tutoring Systems .
Proceedings of the 7th International Conference on Intelligent Tutoring
Systems, 531-540.
[pdf]
Baker, R.S., Corbett, A., Koedinger, K., Roll, I. (2005) Detecting
When Students Game The System, Across Tutor Subjects and Classroom
Cohorts . Proceedings of User Modeling 2005, 220-224. [pdf]
Roll, R., Baker, R., Aleven, V., McLaren, B., Koedinger, K. (2005)
Modeling Students' Metacognitive Errors in Two Intelligent Tutoring
Systems. Proceedings of User Modeling 2005, 367-376.
[pdf]
Understanding Why Students Game the System
Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., Koedinger, K. (2008) Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research, 19 (2), 185-224. [draft pdf]
Baker, R.S.J.d. (2007) Is Gaming the System State-or-Trait? Educational Data Mining Through the Multi-Contextual Application of a Validated Behavioral Model. Complete On-Line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling 2007, 76-80. [pdf]
Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B. (2007) Affect and Usage Choices in Simulation Problem Solving Environments. Proceedings of Artificial Intelligence in Education 2007, 145-152. [pdf]
Baker, R.S., Roll, I., Corbett, A.T., Koedinger, K.R. (2005) Do
Performance Goals
Lead Students to Game the System? Proceedings of the International
Conference on Artificial Intelligence and Education (AIED2005), 57-64.
[pdf]
Gaming the System and Learning
Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z. (2004) Off-Task Behavior in the Cognitive Tutor Classroom: When Students
"Game The System". Proceedings of ACM CHI 2004: Computer-Human
Interaction, 383-390. [pdf]
Baker, R.S.J.d., Aleven, V. (in press) Help Abuse and Proper Use: How Helpful is On-Demand Help When It Is Used Properly? To appear in Proceedings of the 9th International Conference on Intelligent Tutoring Systems.
Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J. (2006) Adapting to When Students Game an Intelligent Tutoring System. Proceedings of the 8th International Conference
on Intelligent Tutoring Systems, 392-401. [Won Best Paper Award] [pdf]
Other Papers Involving Gaming the System
Rodrigo, M.M.T., Baker, R.S.J.d., d'Mello, S., Gonzalez, M.C.T., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sugay, J.O., Tep, S., Viehland, N.J.B. (in press) Comparing Learners' Affect While Using an Intelligent Tutoring Systems and a Simulation Problem Solving Game. To appear in Proceedings of the 9th International Conference on Intelligent Tutoring Systems.
Baker, R.S.J.d., Corbett, A.T., Wagner, A.Z. (2006) Human Classification of
Low-Fidelity Replays of Student Actions. Proceedings of the Educational Data
Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems, 29-36.
[pdf]
Baker, R.S. (2005) Designing Intelligent Tutors That Adapt to When
Students Game the System. Doctoral Dissertation. CMU Technical Report
CMU-HCII-05-104. [pdf]
Collaborators and co-authors (in this work)
Albert Corbett
Kenneth Koedinger
Ido Roll
Didith Rodrigo
Neil Heffernan
Jason Walonoski
Vincent Aleven
Angela Wagner
Tanja Mitrovic
Shelley Evenson
Ivon Arroyo
Tom Mitchell
Daniel Baker
Joseph Beck
Jay Raspat
Meghan Naim
Other researchers who have studied gaming the system
(if I've mistakenly left you off the list, please let me know)
Mingyu Feng
Janet Schofield
Bev Woolf
Jeff Johns
Carole Beal
R. Chas Murray
Vincent Aleven
Julita Vassileva
Ran Cheng
Rebecca Crowley
Michael Yudelson
Lei Qu
Hyokyeong Lee
Ari Bader-Natal
Carolyn Rose
Gahgene Gweon
Rikke Magnussen
Morten Misfeldt
Jack Mostow