An N of 74 is a Survey?

I like to pick on bad surveys. Today's victim example comes from a newspaper just down the road -- Gainesville, Ga. -- that reports on a survey with an N of 74.  In other words, 74 respondents. Collected over a month and a half. From a web site.

Yes, 74. As in, if you do the math, at best an 11 percent margin of error. As in, useless.  Because the 74 aren't even random folks. It's a SLOP.

Here's the lede:

Two of six key projects in the Gainesville master transportation plan have solid support, while residents are divided over a project involving a new bridge over Lake Lanier, according to online survey results released by the city’s consultant in the effort.

So you're thinking okay, a dull but straightforward story. The lede's a bit long but not terrible. I'd grade it a B-, maybe a C+. Until the second graf, that is:

The survey involved 74 responses to questions posted July 1-Aug. 15 on the city’s website.

A whopping 74 responses? Over such a long period, from a city website? In the lede you tell me it's an online survey and I might be forgiven for thinking that it's a real survey, a scientific survey, one with a decent sample and credibility. Basically the second graf would be improved by saying "a completely useless piece of crap."

Indeed, read what the consultant says in the next couple of grafs. "Take it for what it is,” said a consultant. “It’s folks who chose to get on the Web page to give you a response. It is another form of input.”

That's consultant-speak for "this survey is complete bullshit, but it's kinda interesting bullshit, just another data point, but not a good one."

Journalists -- if it's bullshit, don't report it. Or if you report it, raise immediate critical concern in your lede that anyone would rely on this in deciding how to spend millions of hard-earned tax dollars. Instead the story goes at great lengths to report various percentages from the "survey" results.  Sigh.







 

Casey Anthony -- What (kinda) People Know

Merely by mentioning Casey Anthony's name will earn me additional hits, but I wanted to briefly discuss not the verdict itself, or even the media saturation coverage of the trial, but rather the remarkable interest the public has shown in this case and its response to the final verdict.  Much of that interest was fed by Nancy Grace and Headline News, and with the public response came more and more coverage.

But what's the public response to the verdict?  We don't have traditional poll data to rely on (yet), but on Twitter the results seem all one way.  Here's an excellent analysis of Twitter traffic that found 64 percent of Twitter users disagreed with the verdict, 35 percent were undecided, and only 1 percent sided with Anthony being found not guilty.  Doubt Twitter as a measure of public sentiment?  You shouldn't.  It's been found in some academic research to be a reasonably good barometer of public opinion.  Not as good as a real poll, but good.

While we don't have any real poll data yet, at least that I can find, there is this stupid Huffington Post "informal poll" that found only 50 percent thought Anthony was guilty.  I'm rather surprised by such a SLOP (self-selected opinion poll, also known in the survey business as complete bullshit).  Usually the angry folks dominate such faux polls, but apparently not in this case.  By the way, 28 percent had her guilty of a lesser charge and 17 percent thought she was not guilty.  Some might argue hey, that's over 13,000 votes.  It's gotta be more accurate than a scientific poll with a carefully drawn sample of 1,000 people.  And then you'd be wrong.  Size doesn't matter, at least when it comes to surveys.  If you doubt this, just look up the infamous Literary Digest poll debacle of 1936.  Over a million folks surveyed.  Results -- way off.  Unless, that is, Alf Landon really was elected president in 1936.  And I'm pretty sure he wasn't.

Of course the various networks, including HLN, saw remarkable increases in their TV or online traffic during the trial and with the announcement of the verdict.  Much of the response can best be described as anger (Grace, especially, has added fuel to this fire).  Negative emotions are more powerful than positive ones.  That's why talk radio and the TV talking heads who sell partisan indignation and disgust do so well.  Taking a position, it sells.

SLOPpy Polls

A SLOP is a "self-selected opinion poll," most often seen on TV where some talkinghead host asks you to vote via phone or online on some issue. As AAPOR points out, "there is no control over respondent selection because anyone can call in an opinion." So don't be surprised when watchers of Lou Dobbs, for example, all call in and skew the program's "poll" results to favor -- surprisingly -- what Dobbs "thinks."

So as a measure of what people think, it's a load of complete crap.

And that brings us to this bit seen below by funnyman Jon Stewart. Thanks to Nick Browning for pointing this one out to me. As often is the case, Stewart sums it all up and makes it funny to boot.

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Poll Bearers
www.thedailyshow.com
Daily Show Full Episodes Political Humor Healthcare Protests