Nigam et al, EAAT 2004
From ScribbleWiki: Analysis of Social Media
Contents |
Towards a Robust Metric of Opinion
In this papers, the authors attempt to formalize the definition and measurement of opinion based on this assumption: when a sentence has a topic and contains polar language then the polar language is about the topic. They designed their system in two main components: a shallow NLP module for finding polar language and a topic classification module. The details and evaluations for these modules is what follows.
They start by explaining the importance of using the unannotated and unstructured text available as user reviews and they also establish that they are working toward a system with high precision and low recall by separating extraction in two phases: topicality and polarity and then overlapping them at the end to find opinion.
The authors review some of the seminal papers in the area and conclude the most similar work (which uses the hybrid of shallow NLP parsing and machine learning) is constrained by the grammar which is not the case in this work. The major distinction of this work is again separating polarity and topicality in extraction and then combining the two.
Various classes of polar expression are interestingly summarized and categorized in this paper. Various examples are provided to show the subjective versus objective opinions, emotive language, using intuitions, sarcasm/irony, modality, conditional, using time expressions, attribution, etc.
Extracting Polar Sentences
A domain dependent lexicon is developed. The input is tokenized, POS tagged, chunked and polarity of terms are specified based on a predefined polarity lexicon. Then, another module processes this information to aggregate (or rather conflate) information to the sentence level while considering some special situations such as negation.
Extracting Topical Sentences
For topic classification, Winnow classifier is used (the algorithm is outlined and they claim that it performed better than SVM and K-nearest neighbor for them and it is more efficient than both of them). They have manually labeled documents for training. Classifier is trained at the document level and then to use the classifier at the sentence level they used this ad hoc approach: if the classifier considers the document topical, then each sentence is evaluated otherwise all sentences are judged non-topical. They discuss that while this theoretically may cause issues but empirically it lead to a high precision low recall which fits what they needed.
Results
For experimental results, they have hand labeled random 822 users reviews (from 34000 pool). Out of total of 8947 sentences, 11% (88 documents,1298 sentences) were topical and the rest were marked as non-topical. Only 1.6% (147 sentences) of topical sentence were both polar and topical.
For polarity, the precision was around 80%; very similar to human precision. But the recall was much lower (43% for positive and 16% for negative but for human near 80%). They explain the lower recall for negative reviews is due to the higher language variations.
For topicality, the precision was around 70% (lower than 88% for human but still close.)
The precision of the combination was 65% (vs. almost 80% for human). Random sample of extracted sentences using the system is shown as well.
Future Work
They list the parameters that can be used for defining a metric of the opinion in future work. The parameters are essentially the Bayesian statistics for topics and polar topics (i.e., distributions based on frequencies). They also emphasize the importance of using informative priors for estimate.
Other directions for future work mentioned at closing is improving on recall and polarity extraction.
- Bibtex
@inproceedings{Nigam2004,
author = {Kamal Nigam and Matthew Hurst},
title = {Towards a Robust Metric of Opinion},
booktitle = {Proceedings of AAAI Spring Symposium on Exploring Attitude and Affect in Text},
year = {2004},
}
Anotated by Mehrbod.