Screwing the Pooch (NES knowledge questions)

To screw the pooch means to royally screw up. That may be a bit of an overstatement, but there are signs that the ANES seriously had intimate relations with the pooch in how it handled political knowledge questions over several years.

Background: The ANES (American National Election Studies) conducts surveys every two years and is the source of a large part of political research done in academic journals

In a 24-page memo on serious issues concerning the validity of their knowledge questions, NES notes that "ANES political knowledge coding from this period may provide misleadingly unflattering portraits of the American public’s possession of factual information on the topics covered by ANES political knowledge questions."

Oops.

This matters. It matters because journal articles have been written based on these data. It matters because whole friggin books have been written based on these data.

In fairness, the ANES folks do a helluva job and I'm sure they are deeply embarassed and baffled and probably a bit pissed over all this. And there is a silver lining, or so they argue. After a couple of scholars discovered the problem, the ANES leaders concluded:

We are grateful to Professors Gibson and Caldeira for bringing questions about the political knowledge data to our attention. We also thank the ANES and SRC staffs for facilitating our investigation of past practices. We apologize to the user community for any negative consequences that past ANES practices have for your inquiries and scholarship. We are committed to do our best to avoid such mistakes in current and future ANES endeavors.

We view these discoveries as opening up exciting opportunities for new and important scholarship on political knowledge. Public opinion watchers thought they knew how knowledgeable Americans were about political facts, but now, the extent of public knowledge is less clear. We hope the new data ANES will release in the future will inspire scholars to conduct innovative explorations of public knowledge, perhaps yielding new insights into the extent and role of information in the realm of political behavior.

The report includes a long, somewhat complicated, but useful set of appendices detailing all the factors that play a part in this methodological fiasco. I'm not sure exactly what all this means, or even how to correct for it, or even to use ANES data for political knowledge research. Gotta sit, read, think -- and probably drink.