Comparing Specification Marks Method with Maximum Entropy-based Method

The main goal of this section is to evaluate the Specification Marks Method montoyo2001senseval and the Maximum Entropy-based Method (in particular, MEfix System) on a common data set, to allow for direct comparisons. The individual evaluation of each method has been carried out on the noun set (17 nouns) of the Spanish lexical-sample task rigau2001senseval from SENSEVAL-26. Table 12 shows precision, recall and coverage of both methods.


Table: Comparison of ME and SM for nouns in SENSEVAL-2 Spanish lexical sample

Precision Recall Coverage
SM 0.464 0.464 0.941
ME 0.646 0.646 1


In order to study a possible cooperation between both methods, we count those cases that: the two methods return the correct sense for the same occurrence, at least one of the methods provides the correct sense and finally, none of both provides the correct sense. A summary of the obtained results is shown in the Table 13. These results clearly show that there is a large room for improvement when combining both system outcomes. In fact, they provide also a possible upper bound precision for this technology, which can be set to 0.798 (more than 15 percentual points higher than the current best system). Table 14 presents a complementary view: wins, ties and loses between ME and SM when each context is examined. Although ME performs better than SM, there are 122 cases (15 %) solved only by the SM method.


Table: Correct classifications of ME and SM for nouns in SENSEVAL-2 Spanish lexical sample

Contexts Percentage
Both OK 240 0.300
One OK 398 0.498
Zero OK 161 0.202



Table: Wins, ties and loses of ME and SM systems for nouns in SENSEVAL-2 Spanish lexical sample

Wins Ties Loses
ME - SM 267 240 122




Footnotes

...Senseval-26
For both Spanish and English SENSEVAL-2 corpora, when applying the Specification Marks method we used the whole example as the context window for the target noun