More Coding Fun

Just adding to my post below, this time about answers to "who is John Roberts?" Of course you know he's Chief Justice of the United States. A few favorite answers from the 2008 ANES pre- and post-election surveys:

• "president of Australia"
• "Australian prime minister" (John Howard had been PM, so perhaps that explains the confusion here)
  
• "director of the cia"
• A million versions of "never heard of him" or "I don't know."
• "on Supreme Ct." This is an interesting one. Coded as correct?
• "newscater for cnn" Misspelling included, but an interesting response. John Roberts is the morning guy for CNN, so understandable. Surprised not more of these.
  

More Coding Fun

I blogged here, and a day later here, about the 2008 ANES open-ended answers and the difficulties of coding them when it comes to political knowledge. I used Nancy Pelosi as an example and asked whether it's correct or incorrect to call her merely a "member of the House" as opposed to the more specific "Speaker of the House."

Today, a few of the other questions asked of respondents. Asked who was Dick Cheney, a zillion people got it right. Vice president. Favorite answer: "vp he's a jerk. don't give him a gun." I'd count that as correct, I suppose, just because of the "vp" part.

Asked "who is Gordon Brown" (The PM of UK), however, people flaked out. Favorite answers below in quote marks, my snotty comments in italics:

• "Who is Gordon Brown?" I suppose if you don't know the answer, repeat the question.
• "works in the white house? james brown relative?" I feel goood!
  
• "small business administration." Not sure where the hell that comes from.
• "FEMA" You're doing a heckuva job, Brownie. Er, wrong Brownie.
• "Governmental. Is he even a person?" Yeah, kinda, though Tories might disagree.

This item generally gets the lowest number of correct responses. Later today, or perhaps tomorrow, I'll look at the questions on John Roberts, chief justice of the U.S. Supreme Court.

A Specificity Index for Open-Ended Coding

I blogged yesterday (see below) about problems in coding open-ended responses to survey political knowledge questions. I used the Nancy Pelosi question as an example. The "correct" response, from a scholarly standpoint, would have respondents identify her as Speaker of the House, but I also argued that it was equally correct to identify her as a congresswoman, a member of the House, and a lot of other answers that in the past would have been coded as "incorrect" in the American National Election Studies dataset.

Go back to yesterday's post for links to ANES data, the newly released raw open-ended responses, and other important points of interest, especially problems with earlier coding.

We don't know exactly how ANES staff will code these answers, but when release the next version of the 2008 pre- and post-election data, I'll do a comparison then. Today I offer an alternative Specificity approach to coding. It's simple. Anything resembling "Speaker of the House," given its specificity, gets coded as the highest, most correct, response. Let's call it a "3" for the sake of argument. Identifying Pelosi as a member of the House, while correct, loses that specificity, so it gets a "2." Calling her a politician or something similar, that's correct in a vague sort of way, so it scores a "1." And getting it wrong, that's a "0."

Missing and refusals get their own special codes. Scholarly typically recode a refusal to be the same as an "incorrect" response. That's a different problem for a different day.

My specificity method provides greater data range. If someone doesn't like it, they can collapse the resulting codes into any method that strikes them as useful, especially if they're comparing answers in 2008 with some previous year.

Coding Responses

The American National Election Studies released this week the redacted responses to various open-ended questions -- including four political knowledge items -- from the 2008 study (study page here if you'd like to download the Excel file). These are the standard office recognition questions that provide a name and ask respondents to identify "what political office does he/she now hold?" The political figures were John Roberts, Dick Cheney, Gordon Brown, and Nancy Pelosi.

Later, ANES will code these as correct or incorrect, but as some of you know there have been problems with earlier coding of responses and I've blogged about this in the past here and here. ANES has its own report. Basically, some answers that should have been coded as "correct" were instead coded as "incorrect," thus deflating the public's actual knowledge as reported in these influential surveys. Some scholars studying knowledge about the U.S. Supreme Court uncovered the problem.

Okay, back to the present. There's a nice Excel file with all the various responses. It's time for a coding lesson, children, as in -- how do I code this? Take, for example, the question about Nancy Pelosi, who was (and is) Speaker of the House. Is the following response correct or incorrect? I've included misspellings.

• "she is a congressmand out of California"
• "congresswoman in Ca. 8th district"
• "majority leader in congress, democrat"
• "house of representative member"
• "specker of the house"
• "governor of Alaska, John's running mate."

Okay, we can all agree that last one is definitely coded as "incorrect." Other fun incorrect responses have her as a Republican, as a Supreme Court judge, or as the secretary of defense. Many are easy to code as right or wrong. But the others above, how do you code them? She is a congresswoman. She is a member of the House. Are those wrong just because we were hoping people to specifically identify her as Speaker and not merely (but correctly) as a member of the House of Representatives?

It'll be interesting to see how ANES staff code these in the next release. When that happens, I promise to compare these responses with their correct/incorrect dichotomy and report back, because honestly I don't know how to handle them myself. I have a few ideas and I may share some in a later post, but it has to do with a specificity scale (more later). If anyone has suggestions, lemme know. I'd be happy to hear them.

Knowledge about the Supreme Court

I've blogged a number of times about problems the American National Election Studies had in how they coded certain of their political knowledge questions. This came up because some scholars studying people's knowledge of the U.S. Supreme Court or its justices found inconsistencies in the ANES data, at least that didn't match what their own data showed. There's a good ANES report on just this problem.

Now the study these scholars was working on has been published. Below is a key graph describing some of the coding problems. It's worth a read for the methodologically inclined:

But these are not the only limitations plaguing this approach to measuring political knowledge. Most worrisome, in one instance, the ANES required its interviewers to code the accuracy of the respondents’ answers to the knowledge question during the interview itself, apparently using quite stringent criteria. Thus, if one replies that William Rehnquist is ‘‘the Chief Justice of the U.S. Supreme Court,’’ the interviewer would, according to ANES coding rules, record the answer as ‘‘correct’’ (even though the official title of the leader of the Court is the ‘‘Chief Justice of the United States’’). Also according to the ANES rules, references to Rehnquist as a Supreme Court judge who is the head honcho or main guy or the main one are scored as incorrect. According to these strict procedures, only 10.5% of the respondents ‘‘correctly’’ identified Rehnquist in the 2000 ANES.

Basically the open-ended responses were miscoded, or coded too strictly, or at times not even accurately when considering the official title of the chief justice. This affects a small but significant number of responses, which in turn can influence not only our evaluations of what people know about the Supreme Court but the consequences of that knowledge when examining its relationship with other variables. For example, the authors examine the role knowledge has in institutional loyalty.

Ultimately, they write:

But we do assert that the image of the American people as entirely bereft of information about courts, as ignorant of their role in the American democracy and their importance as makers of public policy, and as oblivious to the nature of judicial institutions and processes, most likely undercredits ordinary people.

In other words, the "Just How Stupid Are We" question becomes "Just How Stupid is Our Methodology?" In defense of ANES, they are working hard to repair the miscoding of previous data sets (these guys do great work) and in the latest 2008 pre- and post-election release the knowledge items are not even available yet (which is kinda pissing me off given that I need them for some work I'm doing). So the Pelosi, Cheney, Brown, and Roberts identification codes all are listed as -3, as in "we dunno yet, but we're working on it." I figure we'll see 'em sometime later this summer.