This note refers to the following IJAIED paper and AIED short paper about the same research.
M. Heilman, K. Collins-Thompson, M. Eskenazi, A. Juffs, L. Wilson. 2010. Personalization of Reading Passages Improves Vocabulary Acquisition. International Journal of Artificial Intelligence in Education, Vol. 20 (1).
M. Heilman, A. Juffs, and M. Eskenazi. 2007. Choosing reading passages for vocabulary learning by topic to increase intrinsic motivation. In Proc. of AIED.
The papers are not accurate in their description of how participants were assigned to conditions. They state that participants were randomly split into treatment and control groups, but the process was not proper random assignment.
The process was as follows: four lists of user names, one for each of the four classes that participated in the study, were created. Each list was sorted alphabetically by students' names (by last name first), and then the four lists were concatenated. The first participant was randomly assigned to the personalization group. After that, going down the list, every other student was assigned to the personalization group or the control group. This sort of process is related to what is called "systematic random sampling." This is not, however, true random assignment: while each participant had an equal chance of being in each condition, there was not an equal probability of all the possible ways of splitting participants into two groups. It is possible with this sort of sampling that some bias can be introduced. Note that we did not observe significant differences between the two groups at the start of the study (e.g., their scores on the MTELP language profiency test, taken prior to the study, were not significantly different).