Survey Says: Tweet!

  • Research suggests Twitter may offer an accurate gauge of public opinion--on certain subjects
By Ken Chiacchia

Let's face it: It's easy to dismiss Twitter.

Sure, about 100 million users, including journalists, politicians and business people, use the medium to trade about 65 million rapid, succinct messages every day. But with those users limited to only 140 characters, how meaningful can "tweets" possibly be?

Very meaningful indeed, as it turns out.

Analysis of tweets can provide a survey of public opinion that's comparable to established polls, according to Noah Smith, assistant professor in the Language Technologies Institute at SCS, and Bryan Routledge, associate professor of finance at the Tepper School of Business. A team led by Routledge and Smith analyzed 1 billion public tweets posted in 2008 and 2009. They showed that, for topics such as consumer confidence and presidential approval, their method had better than a 70 percent correlation with established polls. The results of the research were published in May in the Proceedings of the International AAAI Conference on Weblogs and Social Media.

 "They're short," Smith says. "When the channel is restricted, a person can only talk directly, and about simple things." This simplicity offers the possibility of a straightforward analysis that, for example, more complex financial filings can't accommodate.

To sip from the fire hose of data, the researchers used simple methods indeed. They first retrieved relevant tweets by searching for topic-relevant keywords. Then they measured the frequency of "positive" vs. "negative" words in these tweets--defined by a standard lexicon--creating a "sentiment score" that's simply the ratio of positive words to negative words.

The work had obvious pitfalls, the researchers say. Their method gives equal weight to tweets by private citizens, political operatives, companies and even spammers. Subtle changes in keywords produced strikingly different results. And the method inevitably mischaracterized some tweets--sarcastic ones, for example.

The sentiment score itself, Smith says, is relatively meaningless. For example, between January 2008 and November 2009, the weekly moving average of the sentiment ratio for "jobs" never dipped much below 2.0. That's not because optimism about the job market was twice as large as pessimism--it's an artifact of the language.

"We're not interested in the precise value of the ratio," Smith says. "We're interested in its trend over time."

That trend, the researchers say, showed a striking correlation with traditional polls. When averaged over intervals comparable to the polls' frequency, the Twitter analysis showed a strong correlation with the well-respected Gallup Poll on both consumer confidence (73.1 percent correlation) and job approval polls for President Obama (72.5 percent). The method did less well with the support for Obama vs. McCain in the 2008 election cycle (44 percent correlation), hinting that the method doesn't seem to capture the more complex situation of a contest.

Of course, traditional opinion polling conducted through field research has its own limitations, albeit better known ones. And the Twitter method may be attractive in contrast to one major limitation of polls: their considerable expense. Since the Twitter method only costs programming and CPU time, it may represent a future, bargain-basement method of measuring public opinion that complements, rather than replaces, traditional polls. "There are a lot of questions of public opinion that we don't pay that much to ask," Smith says.

Professional opinion researchers view the Carnegie Mellon team's initial observations with interest--but caution. There's a "black box" nature to Smith and Routledge's work that worries Terry Madonna, a professional pollster and professor of public affairs at Franklin & Marshall College in Lancaster, Pa.

"This is certainly path-breaking research that may have enormous significance in the future for survey research," he says. But Madonna wonders how demographically representative that Twitter users are of the general public. Smith notes that just as techniques have been developed to help control for bias in, for instance, telephone surveys, a technique could also be developed for Twitter polling.

Miles Osborne, an associate professor at the University of Edinburgh in Scotland who does research into social media, thinks the method may fail when applied to smaller questions that get fewer Twitter comments. The simplicity of the analysis, the fact that it has only been tested on topics offering massive Twitter volumes, and the noise in the data may all contribute to degrading the correlation with the "reality" of larger public opinion with less-tweeted topics, Osborne says. While it might be easy to gauge the President's popularity using Twitter, he says, it's "less so for whether Russia will default on its loans."

The CMU researchers aren't trying to replace traditional opinion surveys, and they note that they aren't claiming their method is a substitute for polls like those of Gallup or Quinnipiac University in Connecticut. Routledge and Smith say their interest--as a financial theorist and a computational linguist, respectively--is in dissecting how and why the method works or doesn't, not in replacing conventional polling.

"We don't have any particular magic ball in that we can only extract the information that's in the tweets--we can't forecast the future," Routledge says. "But tweets are attractive data, since one can gather a large cross-section easily."
For More Information: 

Jason Togyer | 412-268-8721 | jt3y@cs.cmu.edu