Talks

Evaluating Dynamic Search

Wednesday, November 7, 2018, 11:00 am-12:00 pm

You are subscribed to this talk through .
You are watching this talk through .
You are subscribed to this talk. (unsubscribe, watch)
You are watching this talk. (unwatch, subscribe)
You are not subscribed to this talk. (watch, subscribe)

Abstract

Dynamic search is a complex Information Retrieval (IR) task. As a result, its evaluation is also complex. A great number of factors need to be considered. They include document relevance, novelty, aspect-related novelty discounting, and user’s efforts in examining the documents. Due to increased complexity, most existing dynamic search evaluation metrics are NP-hard. Consequently, the optimal value, i.e. the upper bound, of a metric highly varies with the actual search topics. In Cran field-like settings such as the Text REtrieval Conference (TREC), scores for systems are usually averaged across all search topics. With undetermined upper bound values, however, it could be unfair to compare IR systems across different topics. This paper addresses the problem by investigating the actual per topic upper bounds of existing dynamic search metrics. Through decomposing the metrics, we derive the upper bounds via mathematical optimization. We show that after being normalized by the bounds, the NP-hard metrics are then able to provide a robust comparison across search topics. The new normalized metrics experimented on official runs submitted to the TREC 2016 Dynamic Domain (DD) Track. The talk will also share what we learned in organizing the DD Track from 2015 to 2017.

Bio

Dr. Grace Hui Yang is an Associate Professor in the Department of Computer Science at Georgetown University. Dr. Yang is leading the InfoSense (Information Retrieval and Sense-Making) group at Georgetown University, Washington D.C., U.S.A. Dr. Yang obtained her Ph.D. from the Language Technologies Institute, Carnegie Mellon University in 2011. Dr. Yang’s current research interests include deep reinforcement learning, dynamic information retrieval, search engine evaluation, privacy-preserving information retrieval, internet of things, and information organization. Prior to this, she has conducted research on question answering, ontology construction, near-duplicate detection, multimedia information retrieval, and opinion and sentiment detection. Dr. Yang's research has been supported by the Defense Advanced Research Projects Agency and the National Science Foundation. Dr. Yang is a recipient of the prestigious National Science Foundation (NSF) Faculty Early Career Development Program (CAREER) Award. Dr. Yang has co-chaired SIGIR 2013 and 2014 Doctoral Consortiums, SIGIR 2017 Workshop, WSDM 2017 Workshop, ICTIR 2017 Workshop, CIKM 2015 Tutorial, ICTIR 2018 Short Paper and SIGIR 2018 Demonstration Paper Program Committees. Dr. Yang served on the editorial board of Information Retrieval Journal from 2014 to 2017. She has served as an area chair/senior program committee member for SIGIR 2014-present, WSDM 2018-present, ECIR 2017 and for ACL 2016. Dr. Yang also co-organized the Text Retrieval Conference (TREC) Dynamic Domain Track from 2015 to 2017 and led the effort for SIGIR privacy-preserving information retrieval workshops from 2014 to 2016.

This talk is organized by Marine Carpuat