Machine Learning Thesis Proposal
- Remote Access - Zoom
- Virtual Presentation - ET
- IVAN STELMAKH
- Ph.D. Student
- Machine Learning Department
- Carnegie Mellon University
Making Scientific Peer Review Scientific
Many important applications such as hiring, university admissions, and scientific peer review rely on collective efforts of a large number of individuals. The scale of these applications requires decision-makers to interact with various algorithms throughout the process. However, if not designed properly, such interactions may amplify unfairness and lead to other unintended consequences. With this motivation, in my research I pursue the goal of designing large human decision-making systems in a principled manner. Concretely, my work focuses on the application of academic peer review that nowadays is severely strained by the rapid growth in the number of submissions to leading AI and ML conferences.
In this talk, I will describe two groups of projects we have completed so far to understand and improve peer review:
First, I will focus on undesirable properties of human decision-making -- noise, bias, and strategic behavior -- that negatively affect both fairness and accuracy of the process. In that, we will discuss the design of the paper-reviewer assignment algorithm that accounts for the reviewer noise in a fair and accurate manner. We will also talk about our work on detecting bias and strategic behavior.
Second, I will focus on policies that inform the design of the conference peer-review process. Surprisingly, these policies are often defined in a heuristic manner and do not undergo the typical test-and-adjust loop which is essential for the progression of scientific systems. We will talk about a series of ICML 2020 experiments that break this trend and offer evidence-based guidance for policy-makers.
Finally, I will outline several directions for future work that we plan to pursue.
Nihar Shah (Chair)
Ramesh Johari (Stanford University)
Eric Horvitz (Microsoft Research)
Zoom Participation. See announcement.