Overview

This research material was developed by the Test Group within the Radar project. It is being disseminated through this site so that other researchers can leverage our efforts into other projects and fields. Some of this content is used to train Radar components and other content is used during the Radar evaluation test.

The content describes a technical conference (ARDRA), the planning of the conference, and small changes that need to be made to the conference plan.

The most important caveat to this collection is that over 90% of the content is fabricated. As such, there are known flaws in the content. For example, characters in email may have slightly different writing styles due to the multiple authors. Also note that any reference to real persons or organizations are strictly fictional representations.

For more detail on content creation, experiment design and protocol, execution of the experiment, see the following paper:

• A. Steinfeld, R. Bennett, K. Cunningham, M. Lahut, P.-A. Quinones, D. Wexler, D. Siewiorek, P. Cohen, J. Fitzgerald, O. Hansson, J. Hayes, M. Pool, and M. Drummond, The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop (Tech Report CMU-CS-06-125, CMU-HCII-06-102), Pittsburgh, PA: Carnegie Mellon University, School of Computer Science, 2006.

Requirements for use of this content

We are pretty flexible with respect to the use of this content. The only requirements we ask you to satisfy are:

1. Do not redistribute this content. Instead, point interested parties to this site. This requirement is mostly to prevent discrepancies in the research community due to version changes.
2. Likewise, keep track of what version you are using and report the version number when disseminating your work.
3. Cite the paper listed above and a link to this site in publications/proposals if this content is used for the work in question.
4. Send us references and links for any publications that result from the use of this content (steinfeld@cmu.edu). We will add what you send to a list on this page unless you request otherwise.
5. Contact us (steinfeld@cmu.edu) if you make substantial changes to this content and want to distribute your version to others. We will provide a link to your version on this site.

Email Corpus

Readme file describing syntax (v1.0, 132 Kb)

1. Wargaming files (v1.0, 711 messages, 1 MB zip file)
2. Placeholder for backstory archive (287 messages)

Content available upon request

Some content is intentionally not provided here due to a desire to prevent accidental bias in ongoing human subjects testing.

1. ARDRA Conference Schedule: the initial schedule in PDF and Excel.
2. Static Web: this includes websites detailing room specifications, a campus map with travel directions, and a conference planning manual. These pages are meant to be reference documents for subjects and targets for website scrapers.

Measurement

Some measurement tools are intentionally not provided here due to a desire to prevent accidental bias in ongoing human subjects testing. These are also available upon request.

1. Post-test survey: A validated survey to measure subject experiences.
2. Performance measurements (future): We are actively working to identify reusable measurements that (a) have value outside the RADAR evaluation test and (b) are easily understood by a wide range of users. These will be posted here when ready.

Publications which use this material

1. Many of the publications under the Radar project utilize content for unit tests or data from the evaluation tests which use this content. Please review the Radar publications page for a full list or research under this project.
2. Faulring, A., Mohnkern, K., Steinfeld, A., & Myers, B. (2008). Successful user interfaces for RADAR. In Proc. ACM Conference on Human Factors in Computing Systems (CHI) Workshop on Usable Artificial Intelligence. PDF (149 KB) copyrighted
3. Steinfeld, A., Bennett, S. R., Cunningham, K., Lahut, M., Quinones, P.-A., Wexler, D., Siewiorek, D., Hayes, J., Cohen, P., Fitzgerald, J., Hansson, O., Pool, M., & Drummond, M. (2007). Evaluation of an Integrated Multi-Task Machine Learning System with Humans in the Loop. In Proc. NIST Performance Metrics for Intelligent Systems Workshop (PerMIS). PDF (1.1 MB) copyrighted.
4. Steinfeld, A., Quinones, P.-A., Zimmerman, J., Bennett, S. R., & Siewiorek, D. (2007). Survey measures for evaluation of cognitive assistants. In Proc. NIST Performance Metrics for Intelligent Systems Workshop (PerMIS). PDF (457 KB) copyrighted.
5. A. Steinfeld, R. Bennett, K. Cunningham, M. Lahut, P.-A. Quinones, D. Wexler, D. Siewiorek, P. Cohen, J. Fitzgerald, O. Hansson, J. Hayes, M. Pool, and M. Drummond, The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop (Tech Report CMU-CS-06-125, CMU-HCII-06-102), Pittsburgh, PA: Carnegie Mellon University, School of Computer Science, 2006.

Related (free) corpora, etc

1. Enron Enron Email Dataset

Aaron Steinfeld, steinfeld@cmu.edu