A Grad Class -- Maybe

I have a slot for Summer 2014 to teach a special topics graduate-level class at UGA. It can be about anything. In the past I've done summer seminars on the effects of social media, the intersection of humor and politics, and religion and media. This time I'm mulling over something less fun,  a course in secondary analysis. As Wikipedia notes, secondary analysis is "data collected by someone other than the user." The entry continues:

Common sources of secondary data for social science include censuses, organisational records and data collected through qualitative methodologies or Qualitative research. Primary data, by contrast, are collected by the investigator conducting the research.

There are a lot of reasons to like secondary analysis. Someone else has done most of the work, there are no pesky IRBs to deal with, and you get these great national or international samples and therefore terrific generalizability. So what's not to like? You don't always have access to concepts you'd like, measured in ways you'd prefer them measured, and there can be a learning curve when it comes to identifying, downloading, cleaning, and analyzing the data. A steep curve.

Hence ... my possible class, if it can attract students.

There is so much data available. Interested in health? Kids and health? Try getting IRB approval for that. But there's lots of data out there, some collected by the feds, some by others, that even dips into such sensitive topics as drug use. Interested in lifestyle stuff? Internet use? And politics, interested in politics? Oh man, we got your data right here. Well, not here, but at places like GSS, ANES, Pew, Census, and a bunch of federal data best found via ICPSR.

I'm still plotting this class, but basically it'd look something like this:

•  What is secondary analysis? How it differs from other methods.
• Read studies that use secondary analysis. Break 'em down.
• Limitations and strengths of the method.
• All the different places data exist.
• Identify concepts students are interested in.
• How to find your data. 
• Downloading and cleaning.
• Your friend, SPSS, for analysis.
• Recoding variables, getting it right.
• Running analyses.
• Writing up your results as a publishable paper.

Okay, that's what I have so far. Most of my scholarly work uses existing data. I'm particularly fond of Pew and ANES, but I've used others.  In fact, I have one dataset right now that must have 10 studies I'll never get around to.

Oh, and the dirty little secret we don't teach students in research methods -- sometimes you skim an interesting dataset and the study will jump out at you. This has happened to me a number of times, including a study that will soon be published in the Journal of Broadcasting & Electronic Media, or the one I'm messing with now where I found a bunch of questions asking what language Latinos prefer to get their news. Having fun with that one and even found a good theoretical base having to do with ties to the ancestral country and political participation. Shhhh, don't mention this to our research methods instructors.

Will the class make? For the academically uninitiated, that means enough students sign up to justify holding the class. All you usually need is five butts in seats, and it's possible it might attract folks from other colleges (political science, maybe, or sociology, or public health).

And of course it all depends on my own single vocal cord getting repaired next month so I can croak out a lecture. Hell, everything rides on that. Luckily, I don't have to make a decision on the class until early Spring.

Is Political Knowledge Research in a Crisis?

"Political knowledge research faces a problem, perhaps even a crisis."

Dire words.  Scary, if like me you make your bit of bread and meat by doing political knowledge research.  But is it so?

This study points out the well-documented problems with how scholars measure political knowledge, in particular the problems found in ANES measures -- a topic I've discussed at length and others have examined for years and years.  You may not have access to the complete article.  If not, I can send you a pdf.

What's the problem?  Deciding on whether an answer is correct, in the ballpark, or incorrect.  If I ask you "Who is Barry Hollander" the correct answer is "journalism professor."  But "he teaches journalism" is damn close, in the ballpark, but sometimes answers like that were judged as incorrect.  The difficulties here were famously discovered in questions about the Chief Justice of the Supreme Court in which a lot of answers, perfectly reasonable answers, were coded as incorrect.

The authors devised an approach, time consuming, that resulted in high reliability.  The try at a machine method, an automated approach, was less successful but still shows promise. 

Coding Hell

This is a problem for many who do research in political knowledge.  What's a right answer?

This gets a bit complicated.  Hang in there.

If I ask respondents what office does John Boehner hold, the right answer is obvious -- Speaker of the House.  But what if a respondent answers congressman or member of Congress.  They're not wrong answers, but are they right enough to code as being correct?  And what if they just say Republican leader?  That's right too.  Not what I asked -- name his office -- but not so very wrong either.

I've blogged about this several times, most recently here as I discussed an excellent study that looked at "partial right" responses to surveys.  The result there?  Folks who give kinda sorta right answers are an awful lot like the folks who give the completely right answers.  This is an enduring problem in political knowledge studies, especially as the ANES wrestled with coding issues of its knowledge question about the Supreme Court (see my link above, track back to earlier posts on this one, including a technical report).

Simply put -- what's a right answer?

I say this as I fiddle with data with just these kinds of questions and where I have to go in and decide what's a right (or wrong, or sorta wrong, or kinda wrong) answer.  My initial response is to code them in multiple ways (absolutely correct, right but not what I was looking for, not right but not wrong, wrong, and no answer).  That's five different codes.  We often dump the non-responses and incorrect responses together (that's a different methodological problem for another day).  So if a respondent didn't answer, or answered incorrectly, that's be a "0" for that question, a "1" if they got it right.  Then we'd sum their correct answers on an index designed to measure, yup, political knowledge.

But a lot of "0" responses sometimes are not so very wrong.  And is an incorrect answer the same as no answer?  Not really.   There's work out there that suggests, for example, that men are more willing than women to guess on these kinds of questions.  They get some of those guesses right, those end up with higher scores since we often through coding do not punish being wrong any more than simply saying "I dunno."  In other words, we treat both the same -- and probably shouldn't.

I'll probably use the multiple coding scheme above, collapse them if I don't see significant differences, and move on.  But it's a serious issue that, to be honest, raises serious questions about a host of previous studies on the topic.

People Don't Lie (as much as we thought)

My dad always told me people lie about three things: how often they vote, how often they went to church, and the gas mileage they get on their vehicle.

It's an accepted truism in the political science literature that survey respondents overestimate how often they vote.  Studies in the past have attempted to gauge this by checking the voting records of survey respondents, with some success.

(as a matter of information for you budding public records access scholars, you can't tell how someone voted, but you can tell if they voted, at least in most states)

There's a new report out at the ANES web site in which the authors examined the voting records of their 2008 national survey sample (report pdf here).   In surveys, the proportion of folks who say they voted is notoriously higher than the actual proportion of people who cast a ballot -- hence we know some folks are, to put it nicely, fudging.  Or so we thought.  See below.

In examining public records, the authors reported four findings:

1. Official government records "contain numerous errors" that makes voter validation difficult.
2. These errors differ widely from state to state.
3. How states report their data makes it tough to compare across states.
4. And here's the biggie -- "We found that for respondents whose government records can be identified, the records and self-reports show very high levels of agreement. This finding implies that overestimation of turnout rates by surveys is attributable to factors and processes other than respondent lying."

In other words, people aren't lying as much as we thought they were lying, and the gap between what people say in surveys about their voting habits may be due, in part, to crappy government record keeping.   Sure, people still fudge, but the difference is partly attributable to lousy records.

This is comforting for scholars who use survey data to study voting turnout, though it kinds makes my dad's advice from above a bit out of date.

Dogs and Cats, Living Together. Mass Hysteria.

See my column in today's Athens Banner-Herald.

Changes Over Time

Do people continue to care about politics and public affairs compared to years ago?  Has their trust in government changed?  And do they think they can make a difference?  Here are few tables, generated thanks to the ANES web site, to get at some of these questions.  In other words, I'm gonna cut and paste.

How about general public interest.  Below is a graph of changes over time:

As you can see above, political interest increased during the 1960s and then dropped thanks to a Watergate effect in the 1970s and into the 1980s.  It's remained largely constant, with a bit of a random walk year to year, really since about 1980.  So we can't argue that the public's interest has waned.  Then again, it's not all that high either.  These are the folks who responded with very high interest and the numbers sit about one-quarter of U.S. adults.

Okay, how about trust?    Glad you asked.

Trust dropped significantly during the 1960s (Vietnam, race) and the 1970s (Watergate), inched up in the 1980s (Reagan, Morning in America), dropped again in 1994 (GOP takeover), began a nice increase during the 1990s and 2000s and then dropped like a rock in recent years (the Obama effect?).  You can come up with any number of other explanations for these shifts up and down and all around, but my favorite would probably be that pounding against government, from left or right, along with real-world events combine to damage the nation's reputation with the public.

Another concept we're often interested in is efficacy.  That is, do people think they can make a difference.  The graph below captures that by flipping the data to describe those who do think they can have an effect on politics.

  

As you can see above, the question has to do with politics being "too complicated" for one to understand.  These are the "disagree" folks above, so they represent people who feel they can understand and make a difference.  The proportion of these folks decreased during the same period that trust went down, but then remained relatively steady until recent years when there's a bit of a jumping up and down.  Smoothing the data by looking only at presidential election years takes some of this randomness out.  In general, efficacy seems to be slightly decreasing of late.

What all this mean?  Recent years both look a lot like previous ones but there are also significant changes.  If you go to the site, linked above, and play with graphs you can see for yourself across any number of variables and concepts, from attitudes about specific issues to voting.  A lot of useful stuff here.

Damn those UF Guys

A couple of University of Florida political scientists, Michael Martinez and Stephen Craig, stole one of my great research ideas.

Okay, they didn't steal it since it's kinda obvious, but they conducted a neat little study similar to what I had blogged about here and here and here.  That is, examining how people answer political knowledge questions to get a partial knowledge, answers that are kinda sorta right, all thanks to earlier work that found serious issues in the ANES coding of responses about Chief Justice William Rhenquist.  As you may remember, since I've written about this a couple of times, scholars uncovered problems with the coding.  As the Martinez and Craig conference paper notes:

On this latter point, for example, responses in 2000 that identified Rehnquist as “Chief Justice of the Supreme Court” were coded as correct – but nine people who said that he was “the Supreme Court justice in charge” (or used similar language, omitting the words “Chief Justice”), along with nearly 400 others who named him as a justice of the Supreme Court (without reference to his role as Chief) or simply as a judge, were all coded as being no better informed than those who might have identified Rehnquist (hypothetically) as Chairman of the Federal Reserve, Mayor of Phoenix, or Canadian Prime Minister. Counting the “justice” answers as partially correct, which they arguably were, would likely result in a more flattering characterization of the public’s awareness about the Supreme Court and politics in general (Krosnick et al. 2008).

In other words, kinda right should not be coded as the same as completely wrong.

In the paper, Martinez and Craig recode 2008 ANES data to see how a more careful coding scheme might change relationships among important variables.  The results?

• The old coding scheme made it appear Americans were more dense than they really are, at least when it comes to identifying significant political actors such as the Speaker of the House.
• The partial right folks resemble in many ways the traditionally correct folks in a series of political variables and sometimes those "partial right" respondents were actually higher in certain important variables.
• As the authors note, we need to think about whether "partial knowledge" may have different influences on political choice as compared to full knowledge.

As the authors note:

Like Art Linkletter’s kids, survey respondents can say the darnedest things, as some respondents apparently confused Nancy Pelosi with Sarah Palin, recalled that Dick Cheney shot someone (without noting that he was also Vice President), and guessed that Gordon Brown might be somehow related to the godfather of soul, James Brown.

For me, I'd love to see how "partial" versus "full" knowledge stacks up against news media exposure, something not analyzed in their paper.  I have my hunches as to particular media being better at predicting full knowledge (print) versus other media being better at predicting partial knowledge (broadcast news).  This would fall neatly into the recall versus recognition work I did some years back.


In full disclosure, I should point out that I knew both of these
political scientists while a mass comm grad student at UF
a million or so years ago, when dinosaurs roamed the campus.

A Government Shutdown

With a looming federal government shutdown, we've heard a lot about the last time this happened and how Clinton bested Congressional Republicans led by Newt Gingrich.  I agree with pundits who say this time is different.  Obama is no Clinton, and no one is playing the evil part as well as Gingrich, except maybe Lord Voldemort or Dick Cheney.

Recent polls put the blame game about even. For example, a Washington Post poll in March scored it 37 percent blaming the Republicans, 37 percent blaming the Obama administration, 15 percent wagging their fingers at both. 

But I'm less interested in in who wins the blame game than in how this affects people's attitudes toward government.  There are a lot of ways to look at this.  In a nod to the Tea Party folks, there's a great poll question asked for many years by USAToday/Gallup: "In your opinion, which of the following will be the biggest threat to the country in the future: big business, big labor, or big government?"

Back in 1999 and 2000, 65 percent pointed at big government.  By 2009 that had dropped to 55 percent.  No more recent data available from this poll.  Big business has gone up as a threat, from 24 percent to 32 percent.  Those evil unions?  Always around 8 to 10 percent.  Again, these data are a bit old, ending in 2009, so we have to read them with some caution.

A different poll looks at whether people think government is doing too much or too little.  Asked from 2002 to 2010, there's been little change, easily within the margin of error.

But trust in government is the big one.  A democratic government relies on the trust of the governed, it draws on that reservoir of trust (yeah, and cash) in other to operate.  Check out the graphic below, drawn from American National Elections Studies data.  Other than a few blips up (post 9/11 being the most recent), there is a steady trend toward less trust.  If you like, you can flip the chart below and look only at the rise in distrust (see here).  What's interesting to note is the steady rise in federal government trust after the 1994 takeover by Republicans (and the budget squabble with Clinton).  It's hard to tease out whether this is a function of GOP leadership, Clinton political skill, or the feds just coming off looking better as the Republicans forced a shutdown.  I'm leaning toward the latter, though I'd need to go find studies with a detailed analysis.  The ultimate point is this -- a shutdown, if it happens, might lead to people realizing how much they depend on the federal government and improve it's reputation among a cynical public.  But it's only a theory.

Frustration with Media and ANES

The ANES is putting together a study called the 2010-2010 Evaluations of Government and Society Study.  It should be great.

While you can read about it here, "The overarching theme of the surveys," according to the page, "is citizen attitudes about government and society."  Given the tea party atmosphere and distrust of government, it struck me that certain media content -- particularly watching Beck or Hannity or perhaps Fox News in general -- might be good concepts to explore in relation to people's attitudes about government in this Tea Party atmosphere.

Guess not.

My proposal got shot down, in part because I didn't do enough work at getting it through the process, in part because they mistakenly don't buy media exposure as a viable concept.  This is, in part, due to a devastating research paper from 1993 by Larry Bartels that I never found particularly convincing, but really got the attention of a political science community that runs the ANES but, let's face it, while very smart, are far from being media scholars.  Plus, Bartels was looking at the broad exposure items (TV news, newspapers, etc.) on the old 0-to-7 days scale, while I was focused on particular programs and networks, something that would significantly reduce measurement error and also provide meaningful insight into particular programs and hosts who tend to criticize government and who also tend to have significant influence on those in the public who tend to criticize government.

In other words, they screwed up.  Or at least I'd like to think they did for my own self-esteem, but it would be helpful -- theoretically and normatively -- to know if watching or listening to Beck and Hannity and their friends plays a significant role in how people respond to government or spark their protests against government, or more important, influences the way they perceive government's role in society.  We're left with damn little else in this study, at least when it comes to the role the news media may play, at least at this time.  Or we'll get the same tired media items they always use in ANES because, let's face it, political scientists are convinced the media don't matter (er, except when they get the chance to go on TV, then suddenly it does).

Do I sound bitter?  I'm not, really.  There's enough on my plate, thank you very much.  I just find it unfortunate that the quality of media items never seems to change in the national elections studies, that Pew has moved so far ahead of them.  But we do have a million ways to ask questions that date back to the 1940s, most of which are never used in anyone's analyses. 

Sigh.

Writing Questions for the ANES

I have proposed some media items for the 2010-2012 American National Election Studies latest set of data collection.  You can see all the proposals here, including mine which you can read in pdf format here.  The overall study is the ANES evaluation of government and society, so I proposed asking questions about people's consumption of what I called (for lack of a sexier term) opinion-based news.  This includes Rush Limbaugh's radio program, Glenn Beck's wackiness on Fox News, and the like. 

The ANES in general tends to, um, suck when it comes to measuring media use, asking a broad TV news exposure question, a newspaper one, etc., with little differentiation between the kinds of programs watched, or websites viewed, or radio programs listened to.  If we're going to understand how people evaluate government, we need to know more about their exposure to content that spends most of its time -- blasting government.

Will my items be included?  I suspect not.  My theoretical rationale is rather weak despite the obvious real-world ramifications and Tea Party overtones of whether people watch Beck and Hannity and the rest.  But I had to try because I think it's important.  And there are priming and framing theoretical approaches that fit here with these items, though I'm not sure I did a particularly good job getting those across.

Sympathetic, but Confused

As I scoured the web for breaking studies or news about what people know, I came across this fairly standard letter to the editor available online of the Evansville Courier & Press.  The headline says it all -- education can increase voter turnout.  And then there are these graphs:

"We the People" educates students about the Constitution and American life, and "Project Citizen" enables students to identify a public problem and solve it through a policy-focused approach.

These programs have impact. In a 2008 American National Election Studies survey, students from these programs demonstrated greater retention of civic and political knowledge than their peers and engaged in greater participation in government affairs. But it's not just about increasing voter turnout. 

I'm baffled.  Not by the idea that such programs may improve retention of civic and political knowledge, but I'm fairly certain the 2008 ANES includes nothing about these programs. I went back and skimmed the codebooks.  Indeed there are political knowledge questions available in the data, but best I can tell (correct me if I'm wrong, someone) there is no way to tell whether respondents participated in any kind of educational programs.

And here's the fun part -- the 2008 ANES hasn't even released the political knowledge questions yet, at least in how they've coded correct or incorrect responses.  You can see the questions, but not the answers coded as correct or incorrect (however, you can see the redacted raw data, if you know where to look).

Weird.

I suspect the author, an Evansville attorney, is confusing the programs and some other reported survey or published research.  There's nothing controversial about his notion, just that I'm fairly certain it's an apples and oranges use of data. 

ANES and Evaluations of Government

The ANES is seeking possible questions for its 2010-2012 Evaluations of Government and Society Study (details here).  These will be Internet-based surveys of a random sample of U.S. adults that focus on -- obviously -- how people evaluate government.  What better time, given the Tea Party environment, the growing Republican power, the overall disenchantment with government that we seem to be seeing?

According to the call for questions:

Specifically, we have in mind here attitudes about the performance of the Obama administration on the major issues of the day, evaluations of Congress and the Supreme Court, identification with and attitudes about the major political parties, and levels of interest in and engagement with national politics. This is primarily because these perceptions are unmistakably correlated with both presidential vote choice and levels of political participation. We intend to measure each of these topics at multiple points throughout the two-year period preceding the 2012 elections. However in addition to these subjects, we envision that each of these surveys would explore a particular aspect of these political perceptions.

We're talking five rolling cross-sectional waves with lots of potential to theoretically get at what's going on out there beyond the mere snapshot surveys presently available.

Of course, being the mass comm guy I am, I wonder what media questions should be included.  ANES tends to undervalue the media.  Sure, there are the basic exposure and attention items to broad-based media (TV news, newspapers, etc.), but rarely does it burrow down to the now vital network or program level.  If Glenn Beck and Sean Hannity and Rush Limbaugh are truly fueling an anti-government sentiment, then we need to get at exposure and response to those specific hosts, those programs, and even more broadly -- watching Fox News.
 
I asked what media items might be included and the answer was ... none, yet, so propose some.  So that's what I hope to do, but it's a little more difficult to generate a theoretical perspective versus a normative one, and ANES loves theory in its proposals.  So I'm gonna give this some thought because it would be nearly criminal to attempt a major undertaking to understand anti-government evaluations without taking these media folks, and their audience, into consideration. 

Suggestions welcome.

ANES 2008 ... still waiting

I respect the hard work it takes to create the ANES.  So my criticism here is mild, because I suspect they're wrestling with various difficulties.  But it's 2010 and we're still waiting for a final release of the 2008 election data.

Let's do some history.  An advanced version of the 2004 ANES data were available on Jan. 25, 2005, and the next release was April 19, 2005.  That's fast.  The advanced version of the 2008 ANES data were available on March 5, 2009 and there is a May version listed, but it's not the full data.  We've yet to see a final release of the 2008 data and, yes, it is 2010.

There are coding issues with open-ended variables.  There is the 2008-2009 panel data to contend with.  Between these two, I assume that's the holdup (though there have been no explanations on the web site, unless I'm missing something).  I didn't attend an AAPOR meeting where they were going to discuss the study but I was told by a staffer via email almost a year ago that they would put the presentation materials online so those of us without huge travel budgets could find a clue as to what's happening.  Still nothing up.

Why does this matter?

It doesn't, not for most folks unless you happen to be a data analysis dweeb like me.  I'm buried in a study right now and wouldn't have much time to play with the ANES data (though I do have one thing I'd like to explore and can only do so with the open-ended data included).  And the people who do this work at the National Election Studies are terrific, so I don't want to make this sound whiny, but updates -- a little transparency -- would be helpful.

Where's the 2008 ANES?

We had full versions of the ANES data from the 2000 and 2004 elections by April of the next calendar year (April 2001 for the 2000 data, April 2005 for the 2004 data). 

It's November 2009 and we're still waiting for the full version of the 2008 ANES.  Hell, we may be in the next election cycle before the data appear.

On a related note, a few months ago the ANES folks presented an update at a major conference.  I couldn't attend (who the hell has travel money other than overindulged senior faculty?), but staff told me the presentation materials would be put online by the next week.  I assumed these would be powerpoint slides or stuff like that.  That was three or four ago, still no presentation materials that I can see.

I'm assuming something in that presentation would explain the delay, such as issues with open-ended coding or other quality-control matters.

Consider this my whine for the day. 

I know staff are not just sitting on the data, but as a user I'd like a little more information on what's going on, why the delay, and what it may mean for those of us who love to sit around and push SPSS buttons to analyze data.  And, oh, publish research.

More Coding Fun

Just adding to my post below, this time about answers to "who is John Roberts?" Of course you know he's Chief Justice of the United States. A few favorite answers from the 2008 ANES pre- and post-election surveys:

• "president of Australia"
• "Australian prime minister" (John Howard had been PM, so perhaps that explains the confusion here)
  
• "director of the cia"
• A million versions of "never heard of him" or "I don't know."
• "on Supreme Ct." This is an interesting one. Coded as correct?
• "newscater for cnn" Misspelling included, but an interesting response. John Roberts is the morning guy for CNN, so understandable. Surprised not more of these.
  

More Coding Fun

I blogged here, and a day later here, about the 2008 ANES open-ended answers and the difficulties of coding them when it comes to political knowledge. I used Nancy Pelosi as an example and asked whether it's correct or incorrect to call her merely a "member of the House" as opposed to the more specific "Speaker of the House."

Today, a few of the other questions asked of respondents. Asked who was Dick Cheney, a zillion people got it right. Vice president. Favorite answer: "vp he's a jerk. don't give him a gun." I'd count that as correct, I suppose, just because of the "vp" part.

Asked "who is Gordon Brown" (The PM of UK), however, people flaked out. Favorite answers below in quote marks, my snotty comments in italics:

• "Who is Gordon Brown?" I suppose if you don't know the answer, repeat the question.
• "works in the white house? james brown relative?" I feel goood!
  
• "small business administration." Not sure where the hell that comes from.
• "FEMA" You're doing a heckuva job, Brownie. Er, wrong Brownie.
• "Governmental. Is he even a person?" Yeah, kinda, though Tories might disagree.

This item generally gets the lowest number of correct responses. Later today, or perhaps tomorrow, I'll look at the questions on John Roberts, chief justice of the U.S. Supreme Court.

A Specificity Index for Open-Ended Coding

I blogged yesterday (see below) about problems in coding open-ended responses to survey political knowledge questions. I used the Nancy Pelosi question as an example. The "correct" response, from a scholarly standpoint, would have respondents identify her as Speaker of the House, but I also argued that it was equally correct to identify her as a congresswoman, a member of the House, and a lot of other answers that in the past would have been coded as "incorrect" in the American National Election Studies dataset.

Go back to yesterday's post for links to ANES data, the newly released raw open-ended responses, and other important points of interest, especially problems with earlier coding.

We don't know exactly how ANES staff will code these answers, but when release the next version of the 2008 pre- and post-election data, I'll do a comparison then. Today I offer an alternative Specificity approach to coding. It's simple. Anything resembling "Speaker of the House," given its specificity, gets coded as the highest, most correct, response. Let's call it a "3" for the sake of argument. Identifying Pelosi as a member of the House, while correct, loses that specificity, so it gets a "2." Calling her a politician or something similar, that's correct in a vague sort of way, so it scores a "1." And getting it wrong, that's a "0."

Missing and refusals get their own special codes. Scholarly typically recode a refusal to be the same as an "incorrect" response. That's a different problem for a different day.

My specificity method provides greater data range. If someone doesn't like it, they can collapse the resulting codes into any method that strikes them as useful, especially if they're comparing answers in 2008 with some previous year.

Coding Responses

The American National Election Studies released this week the redacted responses to various open-ended questions -- including four political knowledge items -- from the 2008 study (study page here if you'd like to download the Excel file). These are the standard office recognition questions that provide a name and ask respondents to identify "what political office does he/she now hold?" The political figures were John Roberts, Dick Cheney, Gordon Brown, and Nancy Pelosi.

Later, ANES will code these as correct or incorrect, but as some of you know there have been problems with earlier coding of responses and I've blogged about this in the past here and here. ANES has its own report. Basically, some answers that should have been coded as "correct" were instead coded as "incorrect," thus deflating the public's actual knowledge as reported in these influential surveys. Some scholars studying knowledge about the U.S. Supreme Court uncovered the problem.

Okay, back to the present. There's a nice Excel file with all the various responses. It's time for a coding lesson, children, as in -- how do I code this? Take, for example, the question about Nancy Pelosi, who was (and is) Speaker of the House. Is the following response correct or incorrect? I've included misspellings.

• "she is a congressmand out of California"
• "congresswoman in Ca. 8th district"
• "majority leader in congress, democrat"
• "house of representative member"
• "specker of the house"
• "governor of Alaska, John's running mate."

Okay, we can all agree that last one is definitely coded as "incorrect." Other fun incorrect responses have her as a Republican, as a Supreme Court judge, or as the secretary of defense. Many are easy to code as right or wrong. But the others above, how do you code them? She is a congresswoman. She is a member of the House. Are those wrong just because we were hoping people to specifically identify her as Speaker and not merely (but correctly) as a member of the House of Representatives?

It'll be interesting to see how ANES staff code these in the next release. When that happens, I promise to compare these responses with their correct/incorrect dichotomy and report back, because honestly I don't know how to handle them myself. I have a few ideas and I may share some in a later post, but it has to do with a specificity scale (more later). If anyone has suggestions, lemme know. I'd be happy to hear them.

Knowledge about the Supreme Court

I've blogged a number of times about problems the American National Election Studies had in how they coded certain of their political knowledge questions. This came up because some scholars studying people's knowledge of the U.S. Supreme Court or its justices found inconsistencies in the ANES data, at least that didn't match what their own data showed. There's a good ANES report on just this problem.

Now the study these scholars was working on has been published. Below is a key graph describing some of the coding problems. It's worth a read for the methodologically inclined:

But these are not the only limitations plaguing this approach to measuring political knowledge. Most worrisome, in one instance, the ANES required its interviewers to code the accuracy of the respondents’ answers to the knowledge question during the interview itself, apparently using quite stringent criteria. Thus, if one replies that William Rehnquist is ‘‘the Chief Justice of the U.S. Supreme Court,’’ the interviewer would, according to ANES coding rules, record the answer as ‘‘correct’’ (even though the official title of the leader of the Court is the ‘‘Chief Justice of the United States’’). Also according to the ANES rules, references to Rehnquist as a Supreme Court judge who is the head honcho or main guy or the main one are scored as incorrect. According to these strict procedures, only 10.5% of the respondents ‘‘correctly’’ identified Rehnquist in the 2000 ANES.

Basically the open-ended responses were miscoded, or coded too strictly, or at times not even accurately when considering the official title of the chief justice. This affects a small but significant number of responses, which in turn can influence not only our evaluations of what people know about the Supreme Court but the consequences of that knowledge when examining its relationship with other variables. For example, the authors examine the role knowledge has in institutional loyalty.

Ultimately, they write:

But we do assert that the image of the American people as entirely bereft of information about courts, as ignorant of their role in the American democracy and their importance as makers of public policy, and as oblivious to the nature of judicial institutions and processes, most likely undercredits ordinary people.

In other words, the "Just How Stupid Are We" question becomes "Just How Stupid is Our Methodology?" In defense of ANES, they are working hard to repair the miscoding of previous data sets (these guys do great work) and in the latest 2008 pre- and post-election release the knowledge items are not even available yet (which is kinda pissing me off given that I need them for some work I'm doing). So the Pelosi, Cheney, Brown, and Roberts identification codes all are listed as -3, as in "we dunno yet, but we're working on it." I figure we'll see 'em sometime later this summer.

Media Variables

A coupla days ago I discussed the ANES 2008-2009 panel data and its political knowledge variables. I tried yesterday to discuss the media variables but twice Blogger decided not to save or publish anything I wrote. So I'm trying again today.

While the political knowledge items are full of promise for analysis, the media variables ... not so much.

There are four questions, all following this format:

During a typical week, how many days do you watch news on TV, not including sports?

So the response alternatives range from 0 to 7 days. The four questions cover news on TV, radio, the Internet, and "a printed newspaper." There are a few things to like here. Using "printed" is smart. Helps respondents differentiate between the dead-tree and online version. And the "not including sports" is smart too.

Now time for the but. You knew it was coming.

TV news is too general in a fragmented media market. So is radio. Respondents may confuse NPR and Rush Limbaugh and a fragment of news caught every day between Led Zeppelin songs. And mere exposure? Attention is more important, especially when studying the effects of TV news.

ANES has a tradition of mediocre media items, though there have been individual years where they surprised me, especially when a list of entertainment programming that allowed for some interesting tests to be conducted. From a media variable perspective, these are disappointing.

In a perfect world, ANES and Pew get together and marry the ANES policy and knowledge items with Pew's brilliant media questions.