Public Opinion about Public Opinion

That great philosopher, Lucifer, once said:

   Public opinion is nothing more than this,
   what people think that other people think.

Okay, Victorian playwright Alfred Austin actually put these words in the mouth of his title character, Prince Lucifer, in his 1891 play.

(yes I've mentioned this before, back in 2008)

What I'm discussing here is part of a larger work I'm fiddling with, the idea that we don't spend enough time studying people's opinions about public opinion -- or, for lack of a better word, meta-opinion.  There are theoretical traditions where this is more common (Spiral of Silence comes immediately to mind).  These traditions tend to view public opinion from a social control perspective.  Two factors come into play here: (1) we're always sensing the climate of opinion around us, and (2) a fear of isolation leads us to go with the majority, or at least not express our minority viewpoints as readily as we might.

Psychology has examined this in detail at the micro, individual level, or small-group level (my favorite, false consensus, but see also pluralistic ignorance for a more macro and contradictory approach). 

My point is related to, but separate from, what people think about public opinion polls.  In a sense, people view such polls negatively when the results disagree with their own positions, an example of the hostile media effect.  It's vital we tease out the difference between how people respond to a poll -- and who published it -- and how they think about public opinion in a broader sense.  In the latter, I'm talking about their acceptance (or rejection) of the idea of public opinion, their acceptance that other opinions have merit, their acceptance of whether such opinions should guide policy.  How do people define public opinion?  And has this definition changed in a world that has moved from mainstream few-to-many media to social media in which we have an almost instantaneous access to, as our friend Lucifer said above, "what other people think?"  I'm guessing social media, From Facebook to Twitter, have changed our ideas of what truly is public opinion.  And I'm guessing it's become something more narrow, more in tune with our "friends" and "followers."

And it's entirely possible I'm completely wrong.

This is me thinking aloud as I work on a longer, more thorough examination of the topic, hopefully with actual data gathered (or found elsewhere) to fully explore whether people think of public opinion today in a very different way than they used to -- and what that means for public policy, for elections, and ultimately for a healthy democracy.

Of Bandwagons and Underdogs

There's a new batch of polls out today and if you cherry pick with care, your guy is ahead.

And then there is the complaining.  About polls.  About their influence on journalists and pundits and -- most important -- public opinion.

So what the hell is public opinion?  Quite simply, public opinion is what public opinion polls measure.  That's as circular a definition as you'll ever find, and a topic for another day.

Here I want to touch ever so briefly on whether polls influence people.  That is, does seeing or reading or hearing about polls make people move to the majority in a poll?

Yes.

It's called the bandwagon effect.  It exists.  It's real.

But there's also an underdog effect.  It exists.  It's real.  And it often offsets any movement to the majority by a movement to the minority in a poll.

There have been careful studies about both the bandwagon and underdog effects, even a brilliant dissertation written on the topic.  I'm not going to spend my lovely Friday afternoon digging them up and breaking them down for you.  Maybe next week.  Just trust me that polls do influence a few people -- very few -- but the influence cuts both ways, at least when it comes to opinions about a candidate and the likelihood to vote for him or her in an upcoming election.

There are, obviously, more subtle effects of poll results.  Campaign contributions, for example, tend to go to the likely winner, the person ahead in a race.  Why bet on a losing horse?  More subtle but equally important, though, is how a set of positive poll results influences the news narrative.  In other words, today's polls has at least one good result for Romney (Gallup tracking) but a bunch of fairly good ones for Obama.  Expect certain news organizations and pundits to cherry pick their favorite, or you can read someone like Nate Silver, the scary smart guy at the 538 blog, who will break it down for you in a reasonable manner.

But to return to my theme here -- yes, polls can influence people, but they tend to nudge people in both directions, basically the way they were already leaning, and the results largely cancel one another out.

Bad Polling

As readers know, I'm a constant critic of bad polling.  Here's an excellent post that takes to task Maine's news media's reporting of just such a suspicious poll.

City Survey Finds, Shockingly, the City is a Good Place to Live

I love spotting surveys like this -- a survey by city officials that discovers people who live in the city really like living in the city.  Below, I get into the methodology.  But first let's look at the city, in this case Fredericksburg, Virginia.  Here's a graph:

The survey was sent to 1,200 residents and 323 completed and returned it. A whopping 77 percent of them said called Fredericksburg an "excellent" or "good" place to live. The 20-question survey covered eight categories: community, quality, community design, public safety, environmental sustainability, recreation and wellness, community inclusiveness, civic engagement and public trust. 

Whopping?  Please, don't use whopping in a news story, even if something actually, um, whops.  Say instead: three-fourths of respondents called the town an "excellent" or "good" place to live.  Save the whopping for the city's PR folks.

The full report is here (annoying pdf format).  In it you'll find a few interesting tidbits, not because I care one way or the other about a city I've never visited, but it's telling for those interested in methodology and consultants who do this sort of thing.  There's nothing wrong with the survey.  It looks like a lot of similar projects.  But still, let's look deeper for the methodologically inclined.

• It's a mail survey.  That's unusual in these times.  Mail surveys tend to have lousy return rates (that is, how many surveys you send out versus how many returned).
• But this one has a great return rate, with 1200 households surveyed and 323 returns.  That's a 27 percent return rate.  Most mail surveys are lucky if they manage 10-15 percent.  They call it a "multi-contact" survey, meaning they annoyed people to up their return.  Which is good.
• The results were statistically weighted.  That means the consultants took census numbers and corrected their results by key demographic factors.  So if you came up with too many men in your survey as compared to what the census reports you have in the city, you statistically downgrade the results of men (or increase the opinions of women).
• There's a 5 percent margin of error, which is big but not huge.
• They took out the "don't know" responses on individual items.  This is a bit odd because, depending on their distribution, this may significantly alter the results.  Some of the "don't know" results on individual questions ranged as high as 70 percent.
• In the questionnaire, "excellent" is the first choice, followed by "good" and "fair" and "poor" and dunno.  By putting "excellent" first, you bias the results higher.  If you read the frequencies in Appendix A, you discover why they collapsed "excellent" and "good" together when reporting the results -- more people merely said "good."  A good reporter would have caught this and reported it as such.

Why am I picking on this town's survey?  Because lots of cities do this sort of thing, and lots of people do this kind of consulting, and I'm not against either.  But when reporting this journalistically, you have to dig into the methodological details to really tell what people think about an issue or place.

Are Poll Details a Good Thing in a Story?

More information is better, or so goes the standard rule of thumb, especially when reporting the results of a public opinion poll.  AAPOR (full disclosure, I'm a member) and other organizations often suggest technical poll details a journalist should include in a story, such as margin of error, number of people polled, etc. 

And then comes along my latest copy of Newspaper Research Journal with two studies about polls.  Here are the titles of each:

• Regular Readers Expect More Polling Details
• Too Many Technical Details Hinder Recall of Poll Results

So with the first study, we learn from an experiment that regular readers expect details in a poll story and miss them when they're included.  But in the second study, also an experiment, we learn that lots details hinder how much people recall of a poll story.

How do we explain the differing results?

The first study examines perceptions, plus the results really work only for regular readers of a newspaper, not everyone in the experimental study.  The second study looks at recognition.  The more details, lower the knowledge.  Fascinating.

Both studies look at a species very much like human beings -- college students.  So the generalizability of the results can certainly be questioned, but I suspect the latter finding on learning has more to do with information overload, especially for this group, then anything else.  The journalism guy in me, despite these results, still favors putting as much technical information as is reasonable in a story about a poll -- particularly if the poll is about something important. 

Problems with Internet Polls?

Public opinion surveys are a key means of learning what people know or think about the issues of the day.  The growth of Net-based polling is not without critics, and this story based on new research by some Stanford folks suggests that these "opt-in" panels are nowhere near as accurate as traditional randomly drawn samples.  Read the full study here (pdf).

Polls and Health Care

The health care debate? I'd love to avoid this one, what with all the town halls and screaming and lies and damn lies and occasional statistics, but here I am, writing about it.

So I'm skimming the polls about health care and a couple of them stick out for lots of -- as you'll see -- obvious reasons.

Here they are:

• Fox News has polls that show in July, 36 percent, and in August, 34 percent, favor health care reform legislation.
• CNN has polls that show in June, 51 percent, and in July/August, 50 percent favor health care reform legislation.

Huh?

When attempts to find out what people think turn out so differently, we often look at how the sample was drawn or how the questions were asked. I've little time to get into the sampling thing (but it looks okay), so let's look at the questions.

CNN: "From everything you have heard or read so far, do you favor or oppose Barack Obama's plan to reform health care?"

FOX: "Based on what you know about the health care reform legislation being considered right now, do you favor or oppose the plan?"

No real difference that I can see, which is troubling. Is it coincidence that FOX's poll seems negative and CNN's poll seems positive toward health care reform? I dunno, but I'd love to see the order of the questions -- what came before these and how they may have primed the responses. There's something fishy here, on one side or the other.

Death of Polling?

What people know about what others think often comes from polls, but a column today describes how at least one pollster thinks telephone polling will die by 2012.