Progress in information retrieval requires us to quantify the performance of search engines and other information access systems. In this talk, I will motivate and describe a novel evaluation framework for information retrieval evaluation, called time-biased gain. Time-biased gain unifies and generalizes many traditional effectiveness measures (e.g., NDCG) while accommodating aspects of user behavior not captured by these measures. By using time as a basis for calibration against actual user data, time-biased gain can reflect aspects of the search process that directly impact user experience, including document length, near-duplicate documents, and summaries.
Unlike traditional measures, which must be arbitrarily normalized for averaging purposes, time-biased gain is reported in meaningful units, such as the total number of relevant documents seen by the user. As an example, I will examine one instantiation of time-biased gain applicable to the standard "ten blue links" of web search. Rather than the single number produced by traditional effectiveness measures, time-biased gain models user variability and produces a distribution of gain on a per-query basis, allowing us to accommodate different types of user behavior and increasing the realism of the results.
Charles Clarke is a Professor in the School of Computer Science at the University of Waterloo, Canada. His research interests include information retrieval, web search, and text data mining. He has published on a wide range of topics, including papers related to question answering, XML, filesystem search, user interfaces, computational advertising, statistical natural language processing, and the evaluation of information retrieval systems. He is a co-author of the book Information Retrieval: Implementing and Evaluating Search Engines, MIT Press, 2010.
Host: Yubin Kim
jlentz [atsymbol] cs ~replace-with-a-dot~ cmu ~replace-with-a-dot~ edu