Tuesday, June 14, 2016

Bad Poll Writing

Please please please, take individual poll stories with all the skepticism you can muster. Take, for example, this new poll that breathlessly reports:
Hillary Clinton's lead over Donald Trump in the U.S. presidential race has narrowed since late last week, according to the results of the first Reuters/Ipsos poll conducted since the Orlando shooting rampage on Sunday.
If you're a Clinton supporter, you're going "Holy crap." If you're behind Trump, you're cheering.If you're a journalist writing a poll story, you should know better.

Let's break it down. The difference is miniscule, from a 13-percentage-point lead to a 11.6-percentage-point lead. That's all of 1.4 percentage points. Well within a good poll's margin of error. Except they don't report a margin of error, they report something called:
The online poll included 1,063 likely voters and had a credibility interval, a measure of accuracy, of about 3.5 percentage points.
What the hell is a credibility interval? The polling firm explains it here if you're interested, and a more skeptical look by AAPOR worth the read is here, but it boils down to the fact the sample in this poll is not random. That matters. It's an opt-in online poll. Now these are being used more and more often, and are not necessarily the evil they may seem to be. The 538 poll rankings gives IPSOS an "A-," which is damn good. But even if we accept the "credibility interval" as a surrogate for "margin of error" the results are still within that interval. In other words, there's no real difference between the two polls. That makes the hed and lede wrong.

No comments: