next up previous
Next: AGIR's Generation Module Up: The Anaphora-Resolution Module Previous: Evaluation of Anaphora Resolution

Evaluation of Anaphora Resolution in English

The algorithm for anaphora resolution in English is based on the one developed for Spanish, and it has been conveniently adapted for English. The main difference between the two algorithms consists in a different order of the preferences obtained after the training phase. After this phase, we extracted the following conclusions:

After the training phase, the algorithm was evaluated over the test corpus. In the evaluation phase, two experiments were carried out. In the first experiment, only lexical, morphological, and syntactic information was used. The obtained results with the SemCor and MTI corpora appear in Table 7.


Table 7: Anaphora resolution in English, evaluation phase: experiment 1
  He She It They Him Her Them Corr Total P(%)
SEMCOR 116 10 38 50 34 0 6 175 254 68.9
MTI 1 0 347 56 0 0 66 361 470 76.8


The table shows the number of pronouns (classified by type) for the different corpora. The last three columns represent the number of correctly solved pronouns, the total number of pronouns, and the obtained precision, respectively. For instance, in the MTI corpus a precision of 76.8% was obtained.

Discussion. In pronominal anaphora resolution in English, the following results were obtained in the first experiment: SemCor corpus, P = 68.9%, R = 66%; MTI corpus, P = 76.8%, R = 72.9%.

From these results, we have extracted the following conclusions:

After analyzing the results, it was observed that the precision of the SemCor corpus was approximately 8% lower than that for the MTI corpus. The errors in the SemCor corpus mainly originated with the lack of semantic information. Therefore, in order to improve the obtained results, a second experiment was carried out with the addition of semantic information.

The modifications to the second experiment were the following:

Table 8 shows the number of pronouns (classified by type) for the different corpora after these changes were incorporated.


Table 8: Anaphora resolution in English, evaluation phase: experiment 2
  He She It They Him Her Them Corr Total P(%)
SEMCOR 116 10 38 50 34 0 6 220 254 86.6
MTI 1 0 347 56 0 0 66 361 470 76.8


As shown in Table 8, the addition of the two simple semantic constraints resulted in considerable improvement in the obtained precision (approximately 18%) for the SemCor corpus. We concluded that the use of semantic information (such as new constraints and preferences) in the process of anaphora resolution will improve the results obtained.

Finally, Table 9 compares anaphora resolution using AGIR with the other approaches previously presented17. It is important to emphasize the high percentages obtained using our system and Hobbs's method in the SemCor corpus; both systems incorporate semantic information18 into their methods using semantic constraints (selectional restrictions), whereas none of the other authors incorporate semantics in their approaches.


Table 9: Anaphora resolution in English, comparison of AGIR with other approaches
  Proximity Hobbs Lappin Strube AGIR
SEMCOR 37.0 81.9 59.4 59.4 86.6
MTI 54.9 66.0 75.1 63.2 76.8



next up previous
Next: AGIR's Generation Module Up: The Anaphora-Resolution Module Previous: Evaluation of Anaphora Resolution
Jesus Peral 2002-12-13