M. Gormley, A. Gerber, M. Harper, M. Dredze. 2010. Non-Expert
Correction of Automatically Generated Relation Annotations.

This data archive consists of the results (mturk-results.csv) and HIT
design (hit-design.html) for the paper above. The HIT design was
created for and used with the Amazon Mechanical Turk Requester
website. It is a complete Design Template and was used with an input
.csv that contained column headers of the form
sentence_0,...,sentence_9 and fact_0,...,fact_9.  Thus, a
sentence/fact pair would be found in the same row under the columns
sentence_i and fact_i for some integer i. The html form submissions
(and therefore the results which populate the "Answer."  fields) are
is_correct_0,...,is_correct_9 and timeSpentQ0,...,timeSpentQ9. The
former contains the workers annotation and the latter is the number of
seconds spent on each example.

The results file is the compilation of all results from Batches 1 to
3. These results appear under the headers that begin with
"Answer.". The original input can be found in the columns with headers
that begin with "Input." Each HIT consists of 10 examples. The i^th
example is described by a number of different columns each ending in
"_i", just as with the input described above.

In addition, the results for the expert annotations are
included. Their column headers begin with "Expert.". The columns
"Expert.one_i" and "Expert.two_i" are the annotations of expert-1 and
expert-2 respectively. Note that the cell values contained in the
"Expert."  columns are an abreviated form of what is in
"Answer.is_correct_i". For convenience, we have provided additional
columns "Input.control_i" and "Answer.turker_i" which are the control
types and the turker annotations. These columns use the same
abbreviated form as the expert responses. Finally, we've added the
column "Input.example_id_i" which is an identifier for a particular
example and is the same for each of the five turkers that annotated
that example.