1. Personal Name Annotation
Due to privacy issues, it is very hard to get hold of large and realistic email corpora. Here you can find
a couple of email datasets, as well as a dataset of news groups text - annotated with personal names spans.
The full description of these datasets, including relevant statistics and references, is available in:
Einat Minkov, Richard C. Wang & William W. Cohen, Extracting Personal Names from Emails:
Applying Named Entity Recognition to Informal Text, in HLT/EMNLP 2005 (PDF)
Some fast details:
- The email corpora given here were extracted from the Enron corpus, made public by the Federal
Agency Regulatory commission. A version of this data was later purchased by the CALO project,
and made available for research purposes.
- The first dataset, 'Enron-Meetings', consists of all messages located in folders named "meetings"
or "calendar" (excluding a few very large files). Most of these messages are meeting related. The second
subset, 'Enron-Random', was formed by uniformly sampling a user name (out of 158 users) and then
randomly sampling an email from that user.
- As a second type of informal text, we also annotated a collection of newsgroups postings. The
'Newsgroups' dataset was extracted from the 20Newsgroups corpus, by Vitor R. Carvalho.
- These datasets are given here in a Minorthird format (plain text, with separate labels files), as well as
in a 'general' format, where the personal labels are embedded in the text using XML tags.
- The given zipped files construct a directory tree. The separation into train and test folders corresponds
to the data splits described in the abovementioned paper. Further separation is for convenience purposes.
Email Datasets: person name disambiguation and threading
Here you can download Enron corpora and datasets, used for the general problems of entity disambiguation
2. Person name disambiguation and threading
and the extraction of inter-entity relations. Email here is represented as a relational database, which includes
text. Specifically, the tasks considered in these subsets of the Enron corpus are person name disambiguation
in email and intelligent message threading.
Two variations of the data are provided:
A. row email essages, and the corresponding datasets (queries and correct answers), as used in
B. graph files (net relations and entity declarations), and the corresponding datasets, as used in
Note: the corpora files of (A) and (B) are different representation of the same data (where reply lines
have been removed in the latter). The datasets are mostly identical, with the exception that some examples
were moved from the training and test sets to a development set.