New Mechanisms for Human Computation games
TagATune is an audio-based online game that aims to extract descriptions of sounds and music from human players. It serves as a case study of a new mechanism for data elicitation that is different from the mechanisms used in current human computation games.
Law, E., Mityagin, A. and Chickering, M. Intentions: A Game for Classifying Search Query Intent CHI, pp. 3805-3810, 2009. Work-in-Progress paper
Law, E., von Ahn, L. Input-agreement: A New Mechanism for Data Collection using Human Computation Games CHI, pp. 1197-1206, 2009. (Nominated for Best Paper Award) paper CHI Madness Video
Law, E., von Ahn, L., Dannenberg, R. and Crawford, M. Tagatune: A Game for Music and Sound Annotation. ISMIR, 2007. paper
N.B. This paper is about the prototype of the game only.
Evaluation Platforms for Machine Learning
As machine learning techniques are applied to increasingly complex domains, there emerges a class of problems for which measuring accuracy is not a correct evaluation approach. My work focuses on investigating the feasibility of using human computation games as a platform for the evaluation of machine learning algorithms.
Law, E. The Problem of Accuracy as an Evaluation Criterion ICML Workshop on Evaluation Methods in Machine Learning, 2008. paper
Law, E., West, K., Mandel, M., Bay, M. and Downie, S. Evaluation of Algorithms Using Games: The Case of Music Tagging ISMIR, 2009. To Appear. Related Experiment