Ranking wily people who rank each other10 Dec 2017 | Published at AAAI 2017
Anson Kahng, Yasmine Kotturi, Chinmay Kulkarni, David Kurokawa, Ariel D Procaccia
We study rank aggregation algorithms that take as input the opinions of players over their peers, represented as rankings, and output a social ordering of the players (which reflects, eg, relative contribution to a project or fit for a job). To prevent strategic behavior, these algorithms must be impartial, ie, players should not be able to influence their own position in the output ranking. We design several randomized algorithms that are impartial and closely emulate given (nonimpartial) rank aggregation rules in a rigorous sense. Experimental results further support the efficacy and practicability of our algorithms.
This is our first paper as a result of a collaboration with Ariel Procaccia's research group. It's to my knowledge the first paper to solve the challenge of conflicts of interests in peer review when the output of peer review is a ranked list. This paper is also closely tied to the Juxtapeer paper, which we published the following year.
Frequently asked questions
Q: You mention an impartial peer ranking algorithm in the paper. But can an algorithm be impartial?
A As used in this paper, "impartial" means resilient to strategic manipulation, i.e. participants are unable to change their rank based on how they rank others. It doesn't mean impartial in a social sense. As such, the same biases that hamper judgements in peer review (e.g. implicit gender biases) will also exhibit themselves in this situation. We use this term because it is familiar in the game-theory and AAAI communities, but do understand it may be understood differently in HCI/CSCW worlds.
Q: Are the mechanisms you introduce resilient to voting blocs or collusion?
A No, the mechanisms we propose are not resilient to voting blocs, collusion among participants, (and as a result) to sybil attacks by participants themselves.
Q: How many participants do you need for these mechanisms to work well?
A The paper proves properties in an asymptotic setting (i.e. with an infinite number of participants.) In practice, we have found good results with 20+ participants.