Jingyan Wang

I moved to Georgia Tech in 2021.

I am a Ph.D. student in the School of Computer Science at Carnegie Mellon University, advised by Nihar Shah. Previously, I received a B.S. in EECS from UC Berkeley, where I worked with Laura Waller on computational imaging.

My research interest is in machine learning, particularly in applications to improving the process of peer review and crowdsourcing. In these applications, the goal of my research is to understand and mitigate various biases using tools from computer science and statistics, and to also have real-world impact through outreach and policies.

Email: jingyanw [at] cs.cmu.edu

Research

The goal of my research is to understand and mitigate various sources of biases in decision making problems, such as in peer review and peer grading. To understand the biases, I use tools from statistics and computer science to identify their causes and analyze their extent. To mitigate the biases, I work on outreach and policies that lead to practical impacts. Specifically, my research considers three major components that are potentially biased: people, algorithms and policies.

Human bias: People can be biased when they make evaluations (even if they try to be accurate and impartial). One type of such bias is the miscalibration of people, that is, the fact that people have different scales when expressing opinions. In past work, we designed algorithms that take into account the miscalibration of people -- for example, some people tend to be lenient and give higher scores, and some others tend to be strict. Our algorithms are provably more robust to people’s miscalibration, and hence make the evaluation more fair. [paper]

Another type of bias is introduced by the outcome that people have experienced. For example, when evaluating instructors' teaching quality, students often give higher ratings if they receive higher grades, and often give lower ratings if they receive lower grades. In past work, we formulated a model for this problem, and proposed an estimator that reduces the bias in a data-dependent fashion, without prior knowledge on the extent of the existing bias. [paper]

Algorithmic bias: An emerging line of research on algorithmic bias addresses the question: "Can algorithms amplify biases that exist in the data?" In past work, we addressed a related but different question: "Can algorithms themselves introduce a bias, even if the data is perfectly unbiased?" Specifically, for a widely-used model (the Bradley-Terry model) and algorithm (the maximum likelihood estimator) for peer-grading, we mathematically showed that even under the assumption that students give unbiased peer grades, the algorithm systematically assigns lower scores to the top students and assigns higher scores to the students at bottom. In turn, we proposed a modification to the algorithm which provably reduces such bias and is also extremely simple to be implemented in practice. [paper]

Policy bias: Going beyond technical work, I desire to extend my research to real policy improvements in practice. For example, we compiled the data and evaluated the gender distribution in award-winning papers in 16 top computer science conferences in the past 10 years, which showed prominent differences across conferences [blog post]. We also considered the biases caused by the alphabetical ordering practice in scientific publication [blog post]. Taking cognizance of the biases arising from alphabetical ordering, the Machine Learning Department at CMU has randomized the ordering of students and faculty on its webpages.

Publications

A Heuristic for Statistical Seriation

Komal Dhull, Jingyan Wang, Nihar B. Shah, Yuanzhi Li, R. Ravi
UAI, 2021
Debiasing Evaluations That are Biased by Evaluations

Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
AAAI, 2021

[code] [talk]
Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons

Jingyan Wang, Nihar B. Shah, R. Ravi
AISTATS, 2020

[talk]
Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

Jingyan Wang, Nihar B. Shah
AAMAS, 2019

Best student paper award at AAMAS 2019

Also appeared as "Ranking and Rating Rankings and Ratings" at AAAI 2020 Sister Conference Track

[slides]
The more you look, the more you see: towards general object understanding through recursive refinement

Jingyan Wang, Olga Russakovsky, Deva Ramanan
WACV, 2018

[code] [supplementary]
Having Your Cake and Eating It Too: Jointly Optimal Codes for I/O, Storage and Network-bandwidth in Distributed Storage Systems

K. V. Rashmi, Preetum Nakkiran, Jingyan Wang, Nihar B. Shah, Kannan Ramchandran
USENIX FAST, 2015

Picked as the best paper of USENIX FAST 2015 by StorageMojo
3D differential phase-contrast microscopy with computational illumination using an LED array

Lei Tian, Jingyan Wang, Laura Waller
Optics Letters, 2014

Teaching

16-720 Computer Vision
TA, Fall 2017