To find out a more realistic assessment of these models' treatment of unknown words we processed the first section of the WSJ Penn Treebank . This consists of a total of 39923 words in news text style. Using our standard OALD lexicon we find that a total of 1775 words (4.6%) are not found in the lexicon, 943 of which are unique. Of those unknown words we find the following distribution
We listened to each of the 1775 words as pronounced by a number of the models discussed above. A yes/no decision was made about acceptability. Note that a number these words have multiple acceptable pronunciations. If any of those were predicted they were deemed acceptable. For example the pronunciations of ``Reagan'' as /r ey g ah n/ and as /r iy g ah n/ were both considered acceptable.
The best results, shown above for OALD, were obtained by building the deepest possible trees. But when those models were applied to these unknown words the results showed that although the models were not over-trained for the unseen test set extracted from the lexicon itself, they were for these unknown words. The following shows the results after varying the stop value for CART building.
|Stop||Test set||Test set||size|
Looking at those words that are pronounced wrongly we find some mistakes are still recognizable (e.g. Chrysler as /k r ih s l ah er/) but many are unacceptable and unrecognizable showing there is still work to be done. Further analysis of these words shows
Further analysis of the types of names that are still unpronounceable shows a larger proportion of non-anglo-saxon origin than in those that are correctly pronounced. As many of the languages these names originate from often have a more standardized pronunciation than English (e.g. Polish, Italian, Japanese (in its romanized form)), knowing the origin of an unknown word may allow more specific rules to be applied, but we have not yet investigated this area.