Monday, September 16, 2013

An N of 74 is a Survey?

I like to pick on bad surveys. Today's victim example comes from a newspaper just down the road -- Gainesville, Ga. -- that reports on a survey with an N of 74.  In other words, 74 respondents. Collected over a month and a half. From a web site.

Yes, 74. As in, if you do the math, at best an 11 percent margin of error. As in, useless.  Because the 74 aren't even random folks. It's a SLOP.

Here's the lede:
Two of six key projects in the Gainesville master transportation plan have solid support, while residents are divided over a project involving a new bridge over Lake Lanier, according to online survey results released by the city’s consultant in the effort.
So you're thinking okay, a dull but straightforward story. The lede's a bit long but not terrible. I'd grade it a B-, maybe a C+. Until the second graf, that is:
The survey involved 74 responses to questions posted July 1-Aug. 15 on the city’s website.
A whopping 74 responses? Over such a long period, from a city website? In the lede you tell me it's an online survey and I might be forgiven for thinking that it's a real survey, a scientific survey, one with a decent sample and credibility. Basically the second graf would be improved by saying "a completely useless piece of crap."

Indeed, read what the consultant says in the next couple of grafs. "Take it for what it is,” said a consultant. “It’s folks who chose to get on the Web page to give you a response. It is another form of input.”

That's consultant-speak for "this survey is complete bullshit, but it's kinda interesting bullshit, just another data point, but not a good one."

Journalists -- if it's bullshit, don't report it. Or if you report it, raise immediate critical concern in your lede that anyone would rely on this in deciding how to spend millions of hard-earned tax dollars. Instead the story goes at great lengths to report various percentages from the "survey" results.  Sigh.


No comments: