Artifical Intelligence Seminar

  • Gates Hillman Centers
  • ASA Conference Room 6115
  • IVAN STELMAKH
  • Ph.D. Student
  • Mcahine Learning Department
  • Carnegie Mellon University
Seminars

PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

In this talk, we will focus on automated assignment of papers to reviewers in conference peer review. In the first part of the talk we will show that assignment procedure currently employed by NeurIPS and ICML does not guarantee fairness. Optimizing the total quality of the assignment over all papers, this procedure may discriminate against some submissions. In contrast, we will present the assignment algorithm that maximizes the review quality of the most disadvantaged paper, thus ensuring fairness. In the second part of talk, we will discuss the parallel objective of accuracy and show that under standard statistical model, our algorithm leads to the minimax optimal accuracy of the final decisions. Finally, we will present a novel experiment that allows for an objective comparison of various assignment algorithms in terms of both fairness and accuracy, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review. The results of this experiment corroborate the theoretical guarantees of our algorithm.

The AI Seminar is generously sponsored by Apple.

For More Information, Please Contact: