Screwing the Pooch (NES knowledge questions)

To screw the pooch means to royally screw up. That may be a bit of an overstatement, but there are signs that the ANES seriously had intimate relations with the pooch in how it handled political knowledge questions over several years.

Background: The ANES (American National Election Studies) conducts surveys every two years and is the source of a large part of political research done in academic journals

In a 24-page memo on serious issues concerning the validity of their knowledge questions, NES notes that "ANES political knowledge coding from this period may provide misleadingly unflattering portraits of the American public’s possession of factual information on the topics covered by ANES political knowledge questions."

Oops.

This matters. It matters because journal articles have been written based on these data. It matters because whole friggin books have been written based on these data.

In fairness, the ANES folks do a helluva job and I'm sure they are deeply embarassed and baffled and probably a bit pissed over all this. And there is a silver lining, or so they argue. After a couple of scholars discovered the problem, the ANES leaders concluded:

We are grateful to Professors Gibson and Caldeira for bringing questions about the political knowledge data to our attention. We also thank the ANES and SRC staffs for facilitating our investigation of past practices. We apologize to the user community for any negative consequences that past ANES practices have for your inquiries and scholarship. We are committed to do our best to avoid such mistakes in current and future ANES endeavors.

We view these discoveries as opening up exciting opportunities for new and important scholarship on political knowledge. Public opinion watchers thought they knew how knowledgeable Americans were about political facts, but now, the extent of public knowledge is less clear. We hope the new data ANES will release in the future will inspire scholars to conduct innovative explorations of public knowledge, perhaps yielding new insights into the extent and role of information in the realm of political behavior.

The report includes a long, somewhat complicated, but useful set of appendices detailing all the factors that play a part in this methodological fiasco. I'm not sure exactly what all this means, or even how to correct for it, or even to use ANES data for political knowledge research. Gotta sit, read, think -- and probably drink.

CNN Knowledge Test

CNN's web site has a knowledge test of 10 questions.

I nailed eight of 'em. I pass!!!

That's all fine and good, but why the hell don't they have a way to see how others have done? I want downward comparison, to feel I'm smarter than all the other schmucks who visit cnn.com. It'd be fun to know how others did.

btw, I guessed on a couple -- but guessed correctly. I had no idea what actress had recently posed topless (yeah, yeah ... Hollander should know that) but I had seen her face on CNN one day but didn't catch the audio of the story. Thus, I guessed right.

Interest versus Coverage

The fine folks at Pew often compile numbers on what stories people cover, and sometimes they mesh it with what the news has actually covered. For you PhDweeb types, this sounds a lot like agenda-setting, the tired idea that the media do a terrible job of telling people what to think but do a helluva job telling people what to think about.

There's a nice match in this report between what the press covered and what people were interested in. Scroll down and see how media coverage is related to personal interst. The 2008 campaign is first in both, as is the Spitzer story. It's interesting that only 1 percent of coverage forcused on the death of the UNC student (who was from Athens, Ga., my town, a sad story), but 7 percent of Americans were interested. A slight disconnect.

Why does it matter what people think about?

It colors everything. It frames an election. It influences how you consider a candidate, a policy issue. Hell, it affect everything. Political knowledge is one thing, but how a campaign or issue is framed has more persuasive power. The PR folks know this. So do the advertising folks. And the pros who plot political campaigns.

People need to know it too.

Only Tyrants ...

I'm looking at some old books and their mention of political knowledge. Here's a great quote from Lectures on History and General Policy by Joseph Priestly, published in 1791:

Only tyrants, and the friends of arbitrary power, have ever taken umbrage at a turn for political knowledge, and political discourse, among even the lowest of the people.

This is defense of what people should know at at time when the "low" were considered incapable of playing a role. "Political knowledge, it will be said, is useful only to politicians and ministers of state," Priestly writes, then takes apart the argument.

Not all the old stuff is so thoughtful. Here's a favorite from Sketches of the History of Man by H.H. Kames in 1774:

The progress of political knowledge has unfolded many bad effects of a great city, more weighty than any urged in the proclamations. The first I shall mention is, that people born and bred in a great city are commonly weak and effeminate.

Gotta love these old guys.

Pay for Grades

Below is an abstract of a study I blogged about earlier. Basically, paying people improves how well they do in a test of political knowledge. Clearly they are motivated by the cash.

Okay ... so?

A fad of late has been paying kids to do better in school. This makes sense, especially now that I think of it in terms of the study below. If all the other ways of increasing motivation fail (family pressure, peer pressure, the hopes of a good job, the joy of learning for the sake of learning) then why not pay the little darlings to get good grades? In this era of accountability (i.e., no child left untested), this strikes me as a very American approach. Cold, hard cash. The invisible hand. Money talks.

I'm serious. Why not?

The study abstract follows.

Surveys provide widely cited measures of political knowledge. Do seemingly rbitrary features of survey interviews affect their validity? Our answer comes from experiments embedded in a representative survey of over 1200 Americans. A
control group was asked political knowledge questions in a typical survey context. Treatment groups received the questions in altered contexts. One group received a monetary incentive for answering the questions correctly. Another was given extra time. The treatments increase the number of correct answers by 11–24%. Our findings imply that conventional knowledge measures confound respondents' recall of political facts with variation in their motivation to exert effort during survey interviews. Our work also suggests that existing measures fail to capture relevant political search skills and, hence, provide unreliable assessments of what many citizens know when they make political decisions. As a result, existing knowledge measures likely underestimate people's capacities for informed decision making.

Knowing Iraq

In a new report:

Public awareness of the number of American military fatalities in Iraq has declined sharply since last August. Today, just 28% of adults are able to say that approximately 4,000 Americans have died in the Iraq war. As of March 10, the Department of Defense had confirmed the deaths of 3,974 U.S. military personnel in Iraq.

The proportion of people who could correctly name the number of military fatalities in Iraq has been as high as 55% but has never been as low as it is now. Iraq has left the front page, it's left the top of the newscast. We have Obamillary, we have the economy. Hell, we have March Madness.

Republicans dropped from 53% to 26% correct. Democrats dropped from 49% to 30%. In an odd twist, the older you were, the greater the drop in knowledge about military deaths. No idea why. The same huge drop is seen among higher educated folks than those of less education, probably in part because the less educated had a lower number to begin with. Basement effect.

The good news is, 84% know that Oprah supports Obama. That's news you can use.

Spring Break

Due to spring break, I'm not really posting or thinking of posting or doing anything that resembles posting.

Er, except for this post...