This repository contains data that was used and obtained from experiments conducted in the paper: "Approval Voting and Incentives in Crowdsourcing" by Nihar B. Shah, Dengyong Zhou and Yuval Peres, International Conference on Machine Learning (ICML) 2015.

In these files, the terminology "subset selection" refers to the approval voting interface (where the worker is allowed to select a subset of options instead of a single option).

***Experiment details***
> Three experiments were conducted: 1. Identifying animals, 2. Identifying languages, and 3. Identifying textures.
> For each experiment, four different mechanisms were tried out (see paper for details)
> The 12 pdf files in the repository contain the interfaces seen by the workers for each experiment for each mechanism.
> There were 215 workers recruited for each experiment. The recruitment was in batches of 15 spread across several days. In the three experiments, 12, 2, 12 of the workers respectively did not submit any data. Thus we had data from 203, 213, 203 workers respectively in the three experiments. Each worker in any experiment was randomly assigned to one of the four mechanisms. 
> The number of workers in the four mechanisms was:
	Identifying Animals: Single without skip, additive: 54; Single with skip, multiplicative: 49; subset, fixed: 53; subset, multiplicative: 57
	Recognizing Languages: Single without skip, additive: 54; Single with skip, multiplicative: 40; subset, fixed: 64; subset, multiplicative: 45
	Identifying Textures: Single without skip, additive: 67; Single with skip, multiplicative: 40; subset, fixed: 45; subset, multiplicative: 51
> Only those workers with more than 500 HITs previously accepted and an acceptance rate of at least 95% were allowed.
> Each worker was made a fixed payment of 10 cents and conducted the incentive mechanism via a bonus. It is easy to see that all results hold under a setting which necessitates a fixed minimum payment.

***Data***
> The three csv files contain the raw data obtained from the workers on Amazon Mechanical Turk.
> The following is the format of these files.
>> The first row is a header and every other row is the work of a worker.
>> The column Answer.INTERFACE mentions the interface seen by the worker
>> The column Answer.MECHANISM mentions the mechanism seen by the worker
>> The columns Answer.question* contain the IDs of the questions in the order they were presented to the worker
>> The columns Answer.trueAnswer* contain the true answer to the respective questions
>> The columns Answer.answer* contain the answer(s) given by the worker to that question. 
