My research interests lie in the areas of statistics, machine learning, information theory and game theory, with a focus on "learning from people": How to elicit high-quality data from people? How to draw inferences from such data? I am particularly interested in dealing with issues of fairness, accuracy, and robustness. This is an exciting and challenging area of research that has many important applications including hiring, admissions, crowdsourcing, A/B testing, online ratings and recommendations, peer grading, and peer review. My research aims to address these important challenges at scale, in a principled and pragmatic manner.

I am presently particularly excited about developing principled approaches towards improving peer review. Peer review is a microcosm of this general problem of learning from people, and addressing it is particularly urgent to allow for the research community to thrive. I look to formulate and solve problems from the perspective of peer review, and then generalize the inherent ideas to the various other applications of learning from people. Here are examples of a few recent works in this area:

• Miscalibration: Reviewers are often miscalibrated. For instance, some reviewer may be lenient and always provide scores greater than 5/10, while some other may be strict and never provide more than 5/10. Or one reviewer may be moderate and another extreme -- the first reviewer's 2/10 may be equivalent to the second reviewer's 1/10 whereas the first reviewer's 3/10 may bbe equivalent to the second reviewer's 9/10. If these biases are a priori unknown, how would can calibrate the reviewers (from say, just one review obtained per reviewer)? We design a novel randomized estimator that can handle arbitrary and even adversarial miscalibrations. (link)


• Noise: Reviews are often noisy due to reasons like imperfect matches between papers and reviewers. The algorithms popularly employed today for assigning reviewers to papers can be unfair to certain papers (e.g., to interdisciplinary or novel papers). We design an algorithm to assign reviewers to papers which guarantees fairness of assignment to all papers. Simultaneously, the algorithm guarantees statistical accuracy of the review procedure. (link)


• Subjectivity: It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the fact that any paper is reviewed by only a handful of reviewers leads to a high influence of subjectivity in terms of the preferences of these few reviewers. We develop a novel method to mitigate this subjectivity -- to first learn the community's preferences from the individual reviews and then judge every paper with the same yardstick. We prove that surprisingly, this is the only method which meets three natural requirements. We also provide an empirical analysis on IJCAI 2017. (link)


• Strategic behavior: Conference peer review is essentially a zero-sum game, which can incentivize strategic reviews to influence the final ranking of one's own papers by manipulating the reviews they provide. We present a framework to ensure peer review systems that are insulated from strategic manipulations, provably guaranteeing that no reviewer can influence the ranking of any paper with which they have a conflict of interest. We present positive results in terms of an algorithm and an analysis on ICLR 2017 data, as well as negative results which demonstrate the challenges in this problem. (link)


• Efficiency: Human reviewers have a limited amount of time and bandwidth. This necessitates designing the systems around them in a manner that makes most efficient use of the human effort. In this work, we optimize one such aspect -- the system of bidding on papers by the reviewers. (link)


• Evaluation: For any system, it is very important to have in place rigorous tools that can thoroughly evaluate the performance of the system. However, there is little work on tools to evaluate the quality of peer review. The challenges here include the subjective nature of reviews, absence of ground truth, and various other idiosyncrasies of peer reviewing. With this motivation, we design tests for biases in a randomized controlled environment in peer review. We also demonstrate some issues with previous approaches in that they can yield spurious results, which are provably absent in our tests. (link)


• Empirical analysis: We analyze the peer-review data from NIPS 2016. We make several surprising observations from this data, outline some key takeaways for use in future conferences, and highlight a number of challenging and useful open problems. (link)

• On Testing for Biases in Peer Review
  
  Ivan Stelmakh, Nihar B. Shah and Aarti Singh
  NeurIPS 2019.
  
  
• Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons
  
  Jingyan Wang, Nihar B. Shah and R. Ravi
  
  
• A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
  
  Tanner Fiez, Nihar B. Shah and Lillian Ratliff
  ICML workshop on Real-world Sequential Decision Making: Reinforcement Learning And Beyond 2019
  
  
• PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
  
  Ivan Stelmakh, Nihar B. Shah and Aarti Singh
  ALT 2019
  Code for the PeerReview4All paper-reviewer assignment algorithm
  Dataset
  
  
• Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings
  
  Jingyan Wang and Nihar B. Shah
  AAMAS 2019
  My PhD student Jingyan Wang won the Best Student Paper Award at AAMAS 2019
  Nominated for the Best Paper Award
  
  
• Choosing How to Choose Papers
  
  Ritesh Noothigattu, Nihar B. Shah and Ariel Procaccia
  Code
  
  
• On Strategyproof Conference Review
  
  Yichong Xu, Han Zhao, Xiaofei Shi and Nihar B. Shah
  IJCAI 2019
  
  
• Design and Analysis of the NIPS 2016 Review Process
  
  Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon and Ulrike von Luxburg
  Journal of Machine Learning Research 2018
  
  
  PAST RESEARCH ON CROWDSOURCING
  
  
• A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
  
  Nihar B. Shah, Sivaraman Balakrishnan and Martin J. Wainwright
  Code for the WAN and the OBI-WAN estimators
  Dataset
  
  
• Active Ranking from Pairwise Comparisons and when Parametric Assumptions Don’t Help
  
  Reinhard Heckel, Nihar B. Shah, Kannan Ramchandran and Martin J. Wainwright
  Annals of Statistics (to Appear)
  Code
  
  
• Low Permutation-rank Matrices: Structural Properties and Noisy Completion
  
  Nihar B. Shah, Sivaraman Balakrishnan and Martin J. Wainwright
  ISIT 2018
  
  
• Feeling the Bern: Adaptive Estimators for Bernoulli Probabilities of Pairwise Comparisons
  
  Nihar B. Shah, Sivaraman Balakrishnan and Martin J. Wainwright
  IEEE Transactions on Information Theory (Shorter version at ISIT 2016).
  Code for the CRL estimator
  
  
• Simple, Robust and Optimal Ranking from Pairwise Comparisons
  Nihar B. Shah and Martin J. Wainwright
  Journal of Machine Learning Research, 2018.
  Dataset
  
  
• Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues
  Nihar B. Shah, Sivaraman Balakrishnan, Adityanand Guntuboyina and Martin J. Wainwright
  IEEE Transactions on Information Theory 2017 (Shorter version at ICML 2016).
  
  
• No Oops, You Won't Do It Again: Mechanisms for Self-correction in Crowdsourcing
  
  Nihar B. Shah and Dengyong Zhou
  ICML 2016.
  
  
• Approval Voting and Incentives for Crowdsourcing
  
  Nihar B. Shah, Dengyong Zhou and Yuval Peres
  ICML 2015.
  Video  Slides Dataset
  
  
• Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
  
  Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright
  The Journal of Machine Learning Research, 2016.
  Dataset for cardinal vs. ordinal  Dataset for pairwise comparison topologies
  
  
• Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
  
  Nihar B. Shah and Dengyong Zhou
  Journal of Machine Learning Research 2016 (shorter version at NIPS 2015).
  Dataset
  
  
• Parametric Prection from Parametric Agents
  
  Yuan Luo, Nihar B. Shah, Jianwei Huang, Jean Walrand
  Operations Research, 2017.
  
  
  
• Truth Serums for Massively Crowdsourced Evaluation Tasks
  
  Vijay Kamble, Nihar Shah, David Marn, Abhay Parekh, Kannan Ramachandran
  SCUGC 2015: The 5th Workshop on Social Computing and User-Generated Content.
  
  
  
• On the Impossibility of Convex Inference in Human Computation
  
  Nihar B. Shah and Dengyong Zhou
  AAAI, Austin, Jan. 2015.
  
  
• A Case for Ordinal Peer-evaluation in MOOCs
  
  Nihar B. Shah, Joseph Bradley, Abhay Parekh, Martin J. Wainwright, Kannan Ramchandran
  Neural Information Processing Systems (NIPS): Workshop on Data Driven Education, Lake Tahoe, Dec. 2013.
  
  
• Regularized Minimax Conditional Entropy for Crowdsourcing
  
  Dengyong Zhou, Qiang Liu, John Platt, Christopher Meek, and Nihar B. Shah
  Dec. 2014.
  
  
  
  PAST RESEARCH ON DISTRIBUTED STORAGE
  
  (* indicates equal contribution)
  
  
• The MDS Queue: Analysing Latency Performance of Codes
  
  Nihar B. Shah, Kangwook Lee and Kannan Ramchandran
  IEEE Transactions on Information Theory, 2017.
  
  
• A Piggybacking Design Framework for Read-and Download-efficient Distributed Storage Codes
  
  K. V. Rashmi, Nihar B. Shah and Kannan Ramchandran
  IEEE Transactions on Information Theory, 2017.
  Slides from conference (ISIT) presentation
  
  
• When Do Redundant Requests Reduce Latency ?
  
  Nihar B. Shah, Kangwook Lee and Kannan Ramchandran
  IEEE Transactions on Communication, Feb. 2016.
  Slides
  
  
• Distributed Storage Codes with Repair-by-Transfer and Non-achievability of Interior Points on the Storage-Bandwidth Tradeoff
  
  Nihar B. Shah*, K. V. Rashmi*, P. Vijay Kumar and Kannan Ramchandran
  IEEE Transactions on Information Theory, March 2012.
  
  
• Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction
  
  K. V. Rashmi*, Nihar B. Shah* and P. Vijay Kumar
  IEEE Transactions on Information Theory, August 2011.
  IEEE Data Storage Best Paper and Best Student Paper Awards for the years 2011 & 2012.
  
  
• Interference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions
  
  Nihar B. Shah*, K. V. Rashmi*, P. Vijay Kumar and Kannan Ramchandran
  IEEE Transactions on Information Theory, April 2012.
  
  
• On Minimizing Data-read and Download for Storage-Node Recovery
  
  Nihar B. Shah
  IEEE Communications Letters, 2013.
  Second place in the first ACM University Student Research Competition, 2013.
  
  
• Having Your Cake and Eating It Too: Jointly Optimal Codes for I/O, Storage and Network-bandwidth In Distributed Storage Systems
  
  KV Rashmi, Preetum Nakkiran, Jingyan Wang, Nihar B. Shah, and Kannan Ramchandran
  USENIX FAST, Santa Clara, Feb. 2015.
  Picked as the best paper of USENIX FAST 2015 by StorageMojo.
  
  
  
• Fundamental Limits on Communication for Oblivious Updates in Storage Networks
  
  Preetum Nakkiran, Nihar B. Shah, K. V. Rashmi
  IEEE GLOBECOM 2014, Dec. 2014.
  
  
• A "Hitchhiker's" Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers
  
  K. V. Rashmi, Nihar B. Shah, Dikang Gu, Hairong Kuang, Dhruba Borthakur, and Kannan Ramchandran
  ACM SIGCOMM, Aug 2014.
  
  
• One Extra Bit of Download Ensures Perfectly Private Information Retrieval
  
  Nihar B. Shah, K. V. Rashmi and Kannan Ramchandran
  ISIT 2014.
  
  
  
• A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster
  
  K. V. Rashmi, Nihar B. Shah, Dikang Gu, Hairong Kuang, Dhruba Borthakur, and Kannan Ramchandran
  USENIX HotStorage, San Jose, Jun. 2013.
  
  
• Secret Sharing Across a Network with Low Communication Cost: Distributed Algorithm and Bounds
  
  Nihar B. Shah, K. V. Rashmi and Kannan Ramchandran
  IEEE International Symposium on Information Theory (ISIT), Istanbul, Jul. 2013.
  Slides  Poster
  
  
• Regenerating Codes for Errors and Erasures in Distributed Storage
  
  K. V. Rashmi*, Nihar B. Shah*, Kannan Ramchandran, and P. Vijay Kumar
  IEEE International Symposium on Information Theory (ISIT), Cambridge, Jul. 2012.
  Slides
  
  
• Information-theoretically Secure Regenerating Codes for Distributed Storage
  
  Nihar B. Shah*, K. V. Rashmi*, and P. Vijay Kumar
  Globecom 2011.
  
  
• Enabling Node Repair in Any Erasure Code for Distributed Storage
  
  K. V. Rashmi*, Nihar B. Shah* and P. Vijay Kumar
  IEEE International Symposium on Information Theory (ISIT), St. Petersburg, Jul. 2011.
  
  
• A Flexible Class of Regenerating Codes for Distributed Storage
  
  Nihar B. Shah*, K. V. Rashmi*, and P. Vijay Kumar
  IEEE International Symposium on Information Theory (ISIT), Austin, Jun. 2010.
  
  
• Explicit and Optimal Exact-Regenerating Codes for the Minimum-Bandwidth Point in Distributed Storage
  
  K. V. Rashmi*, Nihar B. Shah*, P. Vijay Kumar, and Kannan Ramchandran
  IEEE International Symposium on Information Theory (ISIT), Austin, Jun. 2010.
  
  
• Explicit Codes Minimizing Repair Bandwidth for Distributed Storage    (the complete version on Arxiv)
  
  Nihar B. Shah*, K. V. Rashmi*, P. Vijay Kumar and Kannan Ramchandran
  IEEE Information Theory Workshop (ITW), Cairo, Jan. 2010.
  
  
• Explicit Construction of Optimal Exact Regenerating Codes for Distributed Storage
  
  K. V. Rashmi*, Nihar B. Shah*, P. Vijay Kumar and Kannan Ramchandran
  Allerton Conference on Control, Computing and Communication, Urbana-Champaign, Sep. 2009.
  
  
• Regenerating Codes for Distributed Storage Networks (invited)
  
  Nihar B. Shah*, K. V. Rashmi*, P. Vijay Kumar, and Kannan Ramchandran
  International Workshop on the Arithmetic of Finite Fields (WAIFI), Istanbul, Jun. 2010.
  
  
• Network Coding
  
  K. V. Rashmi*, Nihar B. Shah* and P. Vijay Kumar
  Resonance, vol. 15, no. 7, pp. 604-621., Jul. 2010.
  (Resonance is a journal of science education published by the Indian Academy of Sciences)
  .
• Distributed Storage System for Optimal Storage Space and Network Bandwidth Utilization and A Method Thereof
  
  K. V. Rashmi*, Nihar B. Shah* and P. Vijay Kumar
  US Patent, Nov 2011.
  

• UC Berkeley
  PhD in Electrical Engineering and Computer Sciences
  Advisors: Prof. Martin J. Wainwright and Prof. Kannan Ramchandran
  Other members of thesis committee: Prof. Christos Papadimitriou and Prof. Tom Griffiths
  
  
• Indian Institute of Science (IISc), Bangalore
  M.E. in Telecommunication
  Thesis: Minimizing Repair Bandwidth in Distributed Storage Systems
  Advisor: Prof. P. Vijay Kumar
  
  
• National Institute of Technology Karnataka, Surathkal
  B. Tech. in Electronics and Communication
  

Please visit the publications page.

• David J. Sakrison Memorial Prize for a "truly outstanding piece of research" at EECS, UC Berkeley, May 2017
• Outstanding GSI (graduate student instructor) award at UC Berkeley, 2015-16
• Microsoft Research PhD Fellowship, 2014-2016.
• IEEE Data Storage Best Paper and Best Student Paper awards for years 2011 & 2012
• Second place in the first ACM University Student Research Competition, 2013.
• Berkeley Fellowship, 2011-13 (the most prestigious fellowship for incoming graduate students at UC Berkeley).
• Excellence Award for the academic year 2011-2012 at UC Berkeley.
• Prof. SVC Aiya Medal for the best master-of-engineering student in the ECE department at IISc, 2010.

• Program committee member/Senior PC member/Area Chair, NeurIPS 2019, ICML 2019, AIStats 2019, HCOMP 2018.
• Co-chair, the NIPS 2014 workshop on Crowdsourcing and Machine-learning and the ICML 2014 workshop on Crowdsourcing and Human-computation.
• Reviewer, Journal of Machine Learning Research, Journal of Artificial Intelligence Research, Proceedings of the IEEE, IEEE Transactions on Information Theory, IEEE Transactions on Signal Processing, IEEE Transactions on Communications, IEEE Transactions on Parallel and Distributed Systems, IEEE Communications Letters, IEEE Journal on Selected Areas in Communications, ACM Transactions on Intelligent Systems, Operations Research, NIPS, ICLR, ISIT, DISC, Netcod, Infocom, Globecom, ITW, MIT Press.

• Intern, Microsoft Research Redmond, May 2013 to August 2013 and May 2014 to August 2014
  Crowdsourcing algorithms.
  
  
• Project Associate, IISc-Infosys collaborative project, Bangalore, July 2010 to June 2011
  Algorithms for robust and efficient media content distribution networks.
  
  
• Member of Technical Staff, Adobe Systems, Bangalore, July 2007 to July 2008.
  Worked on Adobe Captivate, an automated e-learning authoring tool.