Please cite the following paper if you use any of the crowdsourced data:
Nihar B. Shah and Martin J. Wainwright, "A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness," arxiv:1606.09632.

Contents:

(a) Two pdf files containing the interfaces that the workers saw. The first interface pertains to a standard additive payment scheme under the standard interface of choosing among two options; the second interface pertains to a multiplicative payment scheme [“Double or nothing”, Shah and Zhou, JMLR 2016] and a skip-based interface. Any worker is chosen one of the two interfaces uniformly at random. The results plotted in the paper pertain to the standard interface.

(b) A CSV file containing the data from the workers. The CSV file has the following general format.

Rows:
The first row is the header.
Every other row represents a worker.

Columns (in the following, "X" and "Y" represent placeholders indexing the first and last question):
Answer.elicitation: The interface and mechanism seen by the worker (“additive” or “multiplicative”; see description above)
Answer.answerX to Answer.answerY: The answer given by the worker to that question. Takes value “Mild” or “Severe” as per the worker selection or takes value “Skip” if the worker chooses to skip the question.
Answer.questionX to Answer.questionY: Since the questions are shown in a random order to each worker, this set of columns depicts the order shown to the worker. Each of the 48 images have a keyword that represents that image, and the i^{th} row of Answer.questionX represents the image shown as the i^{th} question to worker X.
Answer.trueAnsX to Answer.trueAnsY: The true answer (“Mild” or “Severe”) to the question.

The original images and their ground truth annotations were obtained from the DTU/Herlev Pap Smear Database 2005 ("Pap-smear benchmark data for pattern classification", Jan Jantzen, Jonas Noru, Georgios Dounias and Beth Bjerregaard,Nature inspired Smart Information Systems 2005).