More Coding Fun

I blogged here, and a day later here, about the 2008 ANES open-ended answers and the difficulties of coding them when it comes to political knowledge. I used Nancy Pelosi as an example and asked whether it's correct or incorrect to call her merely a "member of the House" as opposed to the more specific "Speaker of the House."

Today, a few of the other questions asked of respondents. Asked who was Dick Cheney, a zillion people got it right. Vice president. Favorite answer: "vp he's a jerk. don't give him a gun." I'd count that as correct, I suppose, just because of the "vp" part.

Asked "who is Gordon Brown" (The PM of UK), however, people flaked out. Favorite answers below in quote marks, my snotty comments in italics:

• "Who is Gordon Brown?" I suppose if you don't know the answer, repeat the question.
• "works in the white house? james brown relative?" I feel goood!
  
• "small business administration." Not sure where the hell that comes from.
• "FEMA" You're doing a heckuva job, Brownie. Er, wrong Brownie.
• "Governmental. Is he even a person?" Yeah, kinda, though Tories might disagree.

This item generally gets the lowest number of correct responses. Later today, or perhaps tomorrow, I'll look at the questions on John Roberts, chief justice of the U.S. Supreme Court.

A Specificity Index for Open-Ended Coding

I blogged yesterday (see below) about problems in coding open-ended responses to survey political knowledge questions. I used the Nancy Pelosi question as an example. The "correct" response, from a scholarly standpoint, would have respondents identify her as Speaker of the House, but I also argued that it was equally correct to identify her as a congresswoman, a member of the House, and a lot of other answers that in the past would have been coded as "incorrect" in the American National Election Studies dataset.

Go back to yesterday's post for links to ANES data, the newly released raw open-ended responses, and other important points of interest, especially problems with earlier coding.

We don't know exactly how ANES staff will code these answers, but when release the next version of the 2008 pre- and post-election data, I'll do a comparison then. Today I offer an alternative Specificity approach to coding. It's simple. Anything resembling "Speaker of the House," given its specificity, gets coded as the highest, most correct, response. Let's call it a "3" for the sake of argument. Identifying Pelosi as a member of the House, while correct, loses that specificity, so it gets a "2." Calling her a politician or something similar, that's correct in a vague sort of way, so it scores a "1." And getting it wrong, that's a "0."

Missing and refusals get their own special codes. Scholarly typically recode a refusal to be the same as an "incorrect" response. That's a different problem for a different day.

My specificity method provides greater data range. If someone doesn't like it, they can collapse the resulting codes into any method that strikes them as useful, especially if they're comparing answers in 2008 with some previous year.