It would be useful if we could tell how well a new teaching strategy would do before running an experiment. It would be even better if we could determine what is the best instructional policy to implement without having to test a bunch on actual students. One way to do this is to simulate a new instructional policy using a hypothesized student model. Unfortunately, this could lead to overestimating the performance of new teaching methods. We propose an alternative approach: using many models we expect to be wrong, rather than using one model we hope to be right.

The order in which material is presented can strongly influence what is learned, how fast performance increases, and sometimes even whether the material is learned at all...as we discover the underlying principles of the order effects in learning, we move instruction away from idiosyncratic expression and closer to a controlled and predictable science.
Frank E. Ritter and Josef Nerb, In Order to Learn

We're still not there yet! I think this paper's a step in that direction!

Can we teach others how to solve a task effectively without knowing how to solve it ourselves? I think so! Our work shows that we can effectively train crowd workers in solving complex web search tasks by having them review their peers' solutions. If we only present workers with solutions that are beyond some threshold length, reviewing peer solutions can surpass the efficacy of expert examples. This work is one step towards what may hopefully become a new era of crowdwork, where workers can be trained to solve complex tasks that cannot easily be decomposed. It also suggests a way to form low-cost yet effective educational materials for other educational settings. It turns out a lot of workers find complex web search tasks pretty fun too!
I have done over 18,000 HITs on mTurk, and your HIT was, by far, the most interesting, perfectly challenging, and FUN HIT I have ever done on mTurk. While I appreciate the bonus for the HIT, what I appreciate more was the actual HIT itself. Thank you. Thank you so much for bringing it to mTurk. I clicked the link to the daily Google a day thing, and am going to do it every day. I loved it that much.

Subject to forthcoming errata due to some additional data cleaning. At a high level, the results should be the same.

Extended abstract consisting of a subset of the content in our UAI paper with the same name.

Describes a workflow for the method in our paper with a similar name.