Coding Responses

The American National Election Studies released this week the redacted responses to various open-ended questions -- including four political knowledge items -- from the 2008 study (study page here if you'd like to download the Excel file). These are the standard office recognition questions that provide a name and ask respondents to identify "what political office does he/she now hold?" The political figures were John Roberts, Dick Cheney, Gordon Brown, and Nancy Pelosi.

Later, ANES will code these as correct or incorrect, but as some of you know there have been problems with earlier coding of responses and I've blogged about this in the past here and here. ANES has its own report. Basically, some answers that should have been coded as "correct" were instead coded as "incorrect," thus deflating the public's actual knowledge as reported in these influential surveys. Some scholars studying knowledge about the U.S. Supreme Court uncovered the problem.

Okay, back to the present. There's a nice Excel file with all the various responses. It's time for a coding lesson, children, as in -- how do I code this? Take, for example, the question about Nancy Pelosi, who was (and is) Speaker of the House. Is the following response correct or incorrect? I've included misspellings.

• "she is a congressmand out of California"
• "congresswoman in Ca. 8th district"
• "majority leader in congress, democrat"
• "house of representative member"
• "specker of the house"
• "governor of Alaska, John's running mate."

Okay, we can all agree that last one is definitely coded as "incorrect." Other fun incorrect responses have her as a Republican, as a Supreme Court judge, or as the secretary of defense. Many are easy to code as right or wrong. But the others above, how do you code them? She is a congresswoman. She is a member of the House. Are those wrong just because we were hoping people to specifically identify her as Speaker and not merely (but correctly) as a member of the House of Representatives?

It'll be interesting to see how ANES staff code these in the next release. When that happens, I promise to compare these responses with their correct/incorrect dichotomy and report back, because honestly I don't know how to handle them myself. I have a few ideas and I may share some in a later post, but it has to do with a specificity scale (more later). If anyone has suggestions, lemme know. I'd be happy to hear them.