Thursday, February 27, 2003 - 3:30, NSH 3001
Beyond Independent Topical Relevance: Evaluation Metrics and Methods for Aspect Retrieval
Dr. William Cohen

Abstract:
When retrieving information, users are often not interested in simply finding relevant documents: for instance, users are usually not interested in documents that are duplicates of each other, or near-duplicates. We seek to extend the traditional model of IR by taking (some) such constraints into account. Specifically, we assume that a search topic is structured as a set of subtopics or "aspects", and that a user's goal is to retrieve documents relevant to many different "aspects" of the query. We then propose an evaluation criterion for this "aspect retrieval" problem based on comparing an actual ranking to an ideal optimal ranking. We show that this criterion naturally generalizes the traditional measures of recall and precision. We then present an evaluation of several language-model based variants of "maximal marginal relevance" rankings, using benchmark data from the TREC interactive retrieval track.