Thursday, August 20, 2015

Survey: Exactly How Many?

I often criticize national, regional and local news media on how they handle public opinion survey stories. Today it's The Red & Black's turn.

In an otherwise fine story about how the law has changed concerning underage drinking, this shows up in the fifteenth graf of the online version:
According to an anonymous straw poll conducted  by The Red & Black with almost 150 200 underage university students, the new law will not alter student habits.*
OK. I can quibble about using an N of 150 without stating its unscientific -- which is kinda the journalistic equivalent of why bother using it in the first place. Let's set that aside for the moment. Later in the story there's this from the fifteenth graph of the print version:
According to an anonymous straw poll conducted by The Red & Black with almost 150 underage university students, the new law will not alter student habits.
Weird. Different N for different stories, but wait. This kinda makes sense. The print version no doubt went to bed far sooner than the online version, so the N of 150 for print versus an N of 200 for online kinda makes sense, if you continued to collect surveys from your "sample" (note my intentional typographic sneer of the word sample). We know, from the story, that:
Over 90 percent of the non-freshmen students surveyed said they either illegally drank before the change and plan to continue, or they did not drink before and do not have plans to start now. Roughly two-thirds of non-freshmen respondents said they planned to drink underage, while less than half of the incoming freshman respondents answered the same way.
What's bothersome to me is there are no details on how this "sample" was collected. It's a straw poll, but what the hell does that mean? Did you stand at Tate Plaza and grab people? Survey folks at your favorite watering hole? Snag people hiking up Baxter Hill? Troll the dining halls? When you report a poll, even a bad poll like this, readers deserve just a little more information on how it was done. Spend a few words if you're going to report on a poll, just like you'd spend a few words describing a source quoted in a story.

Oh, and explain the discrepancy in your online version, saying it has updated numbers.

* I screwed up earlier, repeated the 150 twice. Online has 200, print 150. What I get for a rushed blog just before class.

No comments: