Ivan Stelmakh

I'm a fifth-year PhD student in the Machine Learning Department at CMU advised by Nihar Shah and Aarti Singh. You can find my CV here.

My research interests lie in the broad area of learning from people with a focus on developing tools and techniques to support human decision-making in scientific peer review. Specifically, in my thesis research, I propose a set of tools and experiments that aim at improving fairness and efficiency of academic peer review. The rapid growth of the number of submissions to leading publication venues has identified a need for automation of some parts of the peer-review pipeline. However, there is evidence that application of off-the-shelf technologies may amplify unfairness of the process and lead to other unintended consequences. In my research, I aim at identifying and overcoming such unintended consequences by designing tools and techniques to support fair, equitable and efficient peer review.

Before coming to CMU, I received a B.S. in Applied Mathematics and Physics from Moscow Institute of Physics and Technology, where I was adviced by Vladimir Vyugin.

Email: stiv [at] cs.cmu.edu

News

September 2021
I stay at Google Research for the fall semester as a student researcher.

May 2021
I'm joining Google Research for the summer internship this year.

April 2021
Our work on improving peer review is featured on sciencemag.org.

December 2020
Three papers are accepted to AAAI.

December 2020
A blog post that summarizes three experiments we conducted on the ICML 2020 peer-review process is out! Read it on hunch.net or ML@CMU.

November 2020
I am invited to the 2021 AAAI Doctoral Consortium!

September 2020
My team ended up in top 6 of the Terminal Live: CMU vs. Waterloo AI competition!

June 2020
I joined Citadel Securities for the summer internship program.

March 2020
I joined the editorial team of CMU ML Blog. If you are a CMU student, postdoc or faculty, consider featuring your work in our blog!


Publications

A Large Scale Randomized Controlled Trial on Herding in
Peer-Review Discussions

Ivan Stelmakh, Charvi Rastogi, Nihar B. Shah, Aarti Singh and Hal Daumé III

Vox Populi, VoxDIY: Benchmark Dataset for Crowdsourced Audio Transcription
Nikita Pavlichenko, Ivan Stelmakh, Dmitry Ustalov

NeurIPS 2021 D&B
A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in
Large Conferences

Ivan Stelmakh, Nihar B. Shah, Aarti Singh and Hal Daumé III
Supplemetary Materials

AAAI 2021
Prior and Prejudice: The Novice Reviewers' Bias against Resubmissions in Conference Peer Review
Ivan Stelmakh, Nihar B. Shah, Aarti Singh and Hal Daumé III
Supplemetary Materials
Data

CSCW 2021
Debiasing Evaluations That are Biased by Evaluations
Jingyan Wang, Ivan Stelmakh, Yuting Wei and Nihar B. Shah

AAAI 2021
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
Ivan Stelmakh, Nihar Shah and Aarti Singh
Supplemetary Materials
AAAI Appendix

AAAI 2021
On Testing for Biases in Peer Review
Ivan Stelmakh, Nihar Shah and Aarti Singh

NeurIPS 2019
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
Ivan Stelmakh, Nihar Shah and Aarti Singh
Code for the PR4A assignment algorithm
Dataset

ALT 2019

JMLR
Adaptive Algorithm of Tracking the Best Experts Trajectory
Vladimir Vyugin, Ivan Stelmakh and Vladimir Trunov

JCTE