Long-Term Peer Reviewing Effort is Anti-Reciprocal

Yasmine Kotturi, Andrew Du, Scott Klemmer, Chinmay Kulkarni

PDF

Abstract

Many studies demonstrate that peer reviewing provides pedagogical benefits such as inspiration and developing expert vision, and changes classroom culture by encouraging reciprocity. However, much large-scale research in peer assessment has focused on MOOCs, where students have short tenures, and is unable to describe how reciprocity-oriented classroom cultures evolve over time. This short paper presents the first long-term analysis of peer reviewing with 304 students, conducted in three large physical classes in a year-long undergraduate series. Surprisingly, this analysis reveals that when students receive better reviews on their work, they write worse reviews in the future. This suggests that while students believe in the reciprocal nature of peer review, they act anti-reciprocally. Therefore, battling the emergent norm of anti-reciprocity is crucial both for system designers and practitioners who use peer assessment.

Background

This paper questions a fundamental assumption underlying many peer review systems, that participants are inherently reciprocal, i.e. they help others because others have helped them. Yasmine questioned this assumption, and found that actually, the better help students receive from their peers, the less they reciprocate in the future.

As I write this in November 2018, I'm somewhat surprised that this paper has received such little visibility, even a year after publication. Perhaps researchers really don't like to pay attention to inconvenient truths! Still, Julie Hui seems to have used this result in her work on distributed apprenticeship among entrepreneurs. Hopefully others will follow Julie's example and create systems that don't solely rely on reciprocity for peer interactions.

Frequently asked questions

Q: Is reciprocity dead??

A No. As our paper shows, students still do help other students through peer review. However, the assumption that "oh, we just need to show students great feedback on their work, so they'll want to reciprocate" is unfounded. In fact, without further scaffolding, giving students great feedback on their work makes them less likely to give great feedback in the future.

Q: Why does this happen?

A We aren't sure yet, but one possibility is that students seeing great feedback assume someone else is putting in the hard work (a little like the infamous 'by-stander effect')

Q: What other counter-intuitive things should I think about with peer review?

A Read Angela Duckworth's paper on the benefits of giving vs. receiving feedback. I continue to be amazed at how people ask ``What's in it for me?'' when the research shows that the benefits of giving advice are so much larger than receiving advice.

BibTeX

@inproceedings{kotturi2017long, title={Long-Term peer reviewing effort is anti-reciprocal}, author={Kotturi, Yasmine and Du, Andrew and Klemmer, Scott and Kulkarni, Chinmay}, booktitle={Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale}, pages={279--282}, year={2017}, organization={ACM} }