WSDM 2021 Tutorial:

Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

Date: March 8, 2021

2pm-5pm GMT+2
7am-10am Eastern Time
5.30pm-8.30pm Indian Standard Time
8pm-11pm China Standard Time
(includes a 5-minute break every hour)

Presenter: Nihar B. Shah
Abstract: Peer review is the backbone of scientific research. Yet peer review is frequently called "biased,'' "broken,'' "corrupt,'' and "unscientific'' in many scientific disciplines. This problem is further compounded with the near-exponentially growing number of submissions in various computer science conferences. Due to the prevalence of the "Matthew effect'' of the rich getting richer in academia, any source of unfairness in the peer review system, such as those discussed in this tutorial, can considerably affect the entire career trajectory of (young) researchers. In this tutorial, we will discuss a number of systemic challenges in peer review such as biases, subjectivity, miscalibration, dishonest behavior, noise, as well as policy choices. For each issue, we will first present insightful experiments to understand the issue. Then we will present computational techniques designed to address these challenges. We will also highlight a number of open problems.

Slides: link

Writeup and references: link