Can Twitter Measure Public Opinion?

There's been a lot of interest lately in whether Twitter can be used successfully as a measure of public opinion.  Certainly it's cheaper than a real survey, assuming you have the expertise to gather and analyze thousands, or hundreds of thousands, and yes even millions of tweets, as some scholars have done.

This matters not only for scholars, but journalists as well.

So far the results on Twitter are mixed.  A German study found an analysis of tweets to be as good as polls in predicting a multi-party election.  A million-tweet analysis found Twitter to be both good and not so good, depending on what you were analyzing, in a comparison with traditional U.S. polling data.

There are enormous difficulties in arriving at a suitable, valid, and reliable automated measure of sentiment from tweets.  So often what we post is tongue-in-cheek, negative when we mean positive (or vice versa).

In other words, a million tweets may seem to have enormous predictive power.  But size isn't everything.

Setting aside the problems with measurement error in any program that analyzes tweets as either positive or negative, let's not forget that while a sample of 100,000 or even a million seems impressive compared to traditional surveys of 1,000 respondents, we've seen big mistakes arise when we confuse the size of our sample with the quality of our sample.  In other words, size doesn't always matter.

And for this, I remind you of the infamous 1936 Literary Digest poll. 

You can read about it here, or here or a host of other places.  The magazine was the Time of its period.  The short version: this survey of over a million people predicted Alf Landon, a Republican, would win the 1936 presidential election.  None of us remember studying the Landon administration in U.S. history class because, as you may have guessed, there never was a President Landon.  Roosevelt won.

The magazine, using a method that had worked for it in the past, sent out over 10 million ballots, got over a million back, and called the election so very wrong.  Why?  Because a big sample is not the same as a good sample.  This was 1936.  It was the Depression.  The magazine relied on its own subscription rolls, on those who owned cars, who had telephones, and host of other sources that all skewed toward folks during those terrible economic times who could afford such things -- people who tended to vote Republican.  Thus, the mag predicted a Landon victory. 

Size, then, doesn't always matter.

Twitter is like this.  While I like Twitter and use it often, relatively few Americans make use of it, fewer still actively post to the micro-blog, and even if you can glean a million tweets on an election or public issue, the resulting sample is deeply skewed toward the geeky and those who like technology or just enjoy sharing with the world their daily wisdom.  That's a lot of sampling error.  And we're not even getting into the difficulty of having a computer program decide what's positive or negative in a 140-character-or-less posting.

There are a lot of interesting uses for Twitter -- how people respond in real-time to a television program or sporting event or even breaking news.  As a measure, by itself, of public opinion?  No.  Unless, of course, you learned nothing from the Literary Digest.  Journalists and scholars alike need to keep this in mind when using Twitter as the source of all knowledge, at least when it comes to evaluating what people think.

And now for my defense of Twitter -- as a measure of public opinion.

Yes, you read that right.  While this deserves a more in-depth analysis than I can do here, Twitter in many ways resembles what we classically think of as public opinion.  Since the 1940s or so, as polling technology grew more sophisticated, we've tended to define public opinion as that which public opinion polls measure.  Circular, to be sure, a definition driven by polling methodology and not by sound theoretical reasoning. 

The classical understanding of public opinion is more nuanced.  It includes aspects of communication, something missing in our modern snapshot definition.  In other words, people in the coffee houses of the 1700s discussing the issues of the day, a fluid understanding of opinion as it moved and changed due to not only what people thought but what they said, and how it moved and shifted.  Thus -- Twitter.  It fits this classical, versus the modern, understanding of public opinion.

As I said, my argument above deserves more time and all the usual academic citations we love to layer on our work, but the thesis is a simple one -- Twitter is an imperfect measure of public opinion, as we define it in modern times, but it may very well be the perfect measure of public opinion as we classically understand the concept: messy, fluid, and full of communication.

As journalists become more sophisticated in evaluating the Twitterverse for more than mere anecdotal evidence, they need to keep in mind the limitations of even a million tweets.