KDD 2021 Tutorial:

Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

Date and time:
Eastern Time: August 14, noon-3pm
Pacific Time: August 14, 9am-noon
Central European Summer Time: August 14, 6-9pm
Singapore time: August 15, midnight-3am
Indian Standard Time: August 14, 9.30pm to August 15, 12.30am

Presenter: Nihar B. Shah

Abstract: Peer review is the backbone of scientific research. Yet peer review is frequently called "biased,'' "broken,'' "corrupt,'' and "unscientific'' in many scientific disciplines. We will discuss a number of systemic challenges in peer review such as biases, subjectivity, miscalibration, dishonest behavior, noise, as well as policy choices. For each issue, we will first present insightful experiments to understand the issue. Then we will present computational techniques designed to address these challenges. We will also highlight a number of important open problems.

Slides: link
Detailed writeup and references: link

Topics:

• Prologue (Sections 1 and 2 of writeup)
• Noise (Section 3 of writeup)
• Fraud
  • Lone wolf (Section 4.1 of writeup)
  • Coalitions (Section 4.2 of writeup)
• Miscalibration (Section 5 of writeup)
• Subjectivity
  • Commensuration bias (Section 6.1 of writeup)
  • Confirmation bias (Section 6.2 of writeup)
  • Dr. Fox effect (Section 6.3 of writeup)
  • Hindering novelty (Section 6.4 of writeup)
  • Interdisciplinary research (Section 6.5 of writeup)
• Bias pertaining to author identities (Section 7 of writeup)
• Norms and policies
  • Author incentives (Section 8.1 of writeup)
  • Review quality (Section 8.2 of writeup)
  • Author rebuttal (Section 8.3 of writeup)
  • Discussions and group dynamics (Section 8.4 of writeup)
• Epilogue (Section 9 of writeup)