Knowledge about the Supreme Court

I've blogged a number of times about problems the American National Election Studies had in how they coded certain of their political knowledge questions. This came up because some scholars studying people's knowledge of the U.S. Supreme Court or its justices found inconsistencies in the ANES data, at least that didn't match what their own data showed. There's a good ANES report on just this problem.

Now the study these scholars was working on has been published. Below is a key graph describing some of the coding problems. It's worth a read for the methodologically inclined:

But these are not the only limitations plaguing this approach to measuring political knowledge. Most worrisome, in one instance, the ANES required its interviewers to code the accuracy of the respondents’ answers to the knowledge question during the interview itself, apparently using quite stringent criteria. Thus, if one replies that William Rehnquist is ‘‘the Chief Justice of the U.S. Supreme Court,’’ the interviewer would, according to ANES coding rules, record the answer as ‘‘correct’’ (even though the official title of the leader of the Court is the ‘‘Chief Justice of the United States’’). Also according to the ANES rules, references to Rehnquist as a Supreme Court judge who is the head honcho or main guy or the main one are scored as incorrect. According to these strict procedures, only 10.5% of the respondents ‘‘correctly’’ identified Rehnquist in the 2000 ANES.

Basically the open-ended responses were miscoded, or coded too strictly, or at times not even accurately when considering the official title of the chief justice. This affects a small but significant number of responses, which in turn can influence not only our evaluations of what people know about the Supreme Court but the consequences of that knowledge when examining its relationship with other variables. For example, the authors examine the role knowledge has in institutional loyalty.

Ultimately, they write:

But we do assert that the image of the American people as entirely bereft of information about courts, as ignorant of their role in the American democracy and their importance as makers of public policy, and as oblivious to the nature of judicial institutions and processes, most likely undercredits ordinary people.

In other words, the "Just How Stupid Are We" question becomes "Just How Stupid is Our Methodology?" In defense of ANES, they are working hard to repair the miscoding of previous data sets (these guys do great work) and in the latest 2008 pre- and post-election release the knowledge items are not even available yet (which is kinda pissing me off given that I need them for some work I'm doing). So the Pelosi, Cheney, Brown, and Roberts identification codes all are listed as -3, as in "we dunno yet, but we're working on it." I figure we'll see 'em sometime later this summer.