Language Technologies Institute Colloquium
- Doherty Hall
- CHARLES CLARKE
- School of Computer Science
- University of Waterloo
Modeling User Behavior to Predict the Performance of Informaton Retrieval Systems
Progress in information retrieval requires us to quantify the performance of search engines and other information access systems. In this talk, I will motivate and describe a novel evaluation framework for information retrieval evaluation, called time-biased gain. Time-biased gain unifies and generalizes many traditional effectiveness measures (e.g., NDCG) while accommodating aspects of user behavior not captured by these measures. By using time as a basis for calibration against actual user data, time-biased gain can reflect aspects of the search process that directly impact user experience, including document length, near-duplicate documents, and summaries.
Unlike traditional measures, which must be arbitrarily normalized for averaging purposes, time-biased gain is reported in meaningful units, such as the total number of relevant documents seen by the user. As an example, I will examine one instantiation of time-biased gain applicable to the standard "ten blue links" of web search. Rather than the single number produced by traditional effectiveness measures, time-biased gain models user variability and produces a distribution of gain on a per-query basis, allowing us to accommodate different types of user behavior and increasing the realism of the results.
Charles Clarke is a Professor in the School of Computer Science at the University of Waterloo, Canada. His research interests include information retrieval, web search, and text data mining. He has published on a wide range of topics, including papers related to question answering, XML, filesystem search, user interfaces, computational advertising, statistical natural language processing, and the evaluation of information retrieval systems. He is a co-author of the book Information Retrieval: Implementing and Evaluating Search Engines, MIT Press, 2010.
Host: Yubin Kim