Can We Do Away with Media Exposure Questions

I blogged earlier about the difficulties of getting a proposal approved because I relied on exposure to the news, a concept some find problematic due to perceived (though to me, unconvincing) measurement issues.

This raises at least a couple of questions:

• Can we do away with media exposure questions?
• And, if so, what the hell do we use as an alternative?

Exposure versus attention has received its fair share of attention, but we can't really measure attention without first establishing exposure.  The two are intertwined.  The real problem, to me, is the broad nature of the questions often found in large surveys such as ANES.  These items are historical.  They've been asked more or less the same way for decades, and by continuing to do so, we have some ability to look at changes over time.  For an example, see this table.

But to ask broadly about television news exposure, with no difference drawn between Fox News or PBS, raises all kinds of concerns. 

The issue for me is, do we now combine ideology or partisanship with media exposure ... that is, if we're not going to get specific about where people get their news?  We know from recent research, my own and that of others, that there is a partisan migration going on, especially in cable news, as conservatives shift to Fox and liberals to MSNBC and those somewhere in the middle still watching CNN.  The same is probably true, to a lesser degree, across the news spectrum, particularly on the Internet.

So if we not going to get specific in our news questions, and that's what I tried to do in my modest proposal, then perhaps it's time to combine ideology and news exposure and basically assume likeminded folks seek out likeminded news. 

We are obligated, then, to create interaction terms -- to put it in a regression frame -- for all media and partisanship/ideology measures.  The short answer to this is, ugh, ugly, and a pain when it comes to creating all these dummy terms.  But in the long term, save more specificity in our news media questions, this may be the only logical answer.

It's fun to dip into other fields to see how they approach similar theoretical questions.  In this study, people are examined as political consumers who boycott or buycott.  Fascinating stuff.

Its measure of "political cognition" includes two dimensions: "political knowledge" and "political interest."  Below are the three political knowledge items.

• As far as you know, does the federal government spend more on social security or on foreign aid?
• Would you say that one of the parties is more conservative than the other on the national level?
• How much of a majority is required for the US Senate and House to override a presidential veto?

The political interest item is a single question, pretty basic, so let's focus on above.  Is this enough?  Dunno.  As far as I can tell, no Cronbach's Alpha is reported for the three items, so we don't know how well they hold together.  And I'm nitpicking here on a study that's actually quite interesting, tying together a typology of political consumers with key variables -- but, curiously, no media factors are included.

ANES 2008 ... still waiting

I respect the hard work it takes to create the ANES.  So my criticism here is mild, because I suspect they're wrestling with various difficulties.  But it's 2010 and we're still waiting for a final release of the 2008 election data.

Let's do some history.  An advanced version of the 2004 ANES data were available on Jan. 25, 2005, and the next release was April 19, 2005.  That's fast.  The advanced version of the 2008 ANES data were available on March 5, 2009 and there is a May version listed, but it's not the full data.  We've yet to see a final release of the 2008 data and, yes, it is 2010.

There are coding issues with open-ended variables.  There is the 2008-2009 panel data to contend with.  Between these two, I assume that's the holdup (though there have been no explanations on the web site, unless I'm missing something).  I didn't attend an AAPOR meeting where they were going to discuss the study but I was told by a staffer via email almost a year ago that they would put the presentation materials online so those of us without huge travel budgets could find a clue as to what's happening.  Still nothing up.

Why does this matter?

It doesn't, not for most folks unless you happen to be a data analysis dweeb like me.  I'm buried in a study right now and wouldn't have much time to play with the ANES data (though I do have one thing I'd like to explore and can only do so with the open-ended data included).  And the people who do this work at the National Election Studies are terrific, so I don't want to make this sound whiny, but updates -- a little transparency -- would be helpful.

Paying for Knowledge

It's nice to see someone else working my territory, examining research on political knowledge.  Here's a blog entry that looks at a study I also examined some time back.  The actual article is here, which looks at how paying someone motivates them to do better on tests of political knowledge.  The conclusion of the article is simple: "As a result, existing knowledge measures likely underestimate people’s capacities for informed decision making."

How Much Attention is Attention?

There are good reasons to pay attention to news and public affairs (civic responsibility, your job requires knowing what's happening, conversational material, etc.). There are good reasons to not pay attention (busy schedule, hate the partisanship, worried about missing the next Dancing with American Idol episode, etc.).

How much attention is enough?

If you read the scholarship on political knowledge, you might come away thinking there is a magic amount that everyone should know, a certain degree of attention everyone should have. Carefully read the literature, though, and you find the nuances: people learn what they need to know based on their jobs, their social situations, their time and resources.

Scholars tend to ask the same kinds of questions to tap political knowledge. We do this in part because they seem to work from study to study and it allows us to compare -- more or less -- our finding with other studies. It all gets into the reliability and validity of our measures that, if established by other research, frees us to get at the more substantive analyses. In other words -- get past the method stuff and get to the fun results. Sometimes a study will come along that challenges the status quo of how to measure knowledge. These are acknowledged quickly by most scholars and then they move on to, yes, studying it the same way other people have done over the years.

It may be time to reconsider our measures.

So back to my question above: How much attention is attention? We know that media exposure is a lousy measure, that how much someone attends to a medium is a more fair, more valid, and at times more reliable measure. Perhaps we need individual-specific attention measures. Is a "high" attention for one person the same as a "high" for another? And do they have the same consequences? We have to tie together exposure, attention, and some other factor -- not sure what -- to get at the changes in the media landscape. One answer may be elite versus middle-brow versus low-brow sources of news.

Is high attention to Bill O'Reilly the same as high attention to NewsHour? Probably not, but they'd go down at the same in a research situation (depending on how you scale it).

But trying to scale the kinds of content people consume, that's rife with problems. Should I score Fox News lower than CNN? No, I don't think so. People versus New Yorker magazines? Yep, that's easy. It's TV where we get into trouble, and we're probably going to have to drill down to the individual program level to get at some of this, and that's a whole different problem when it comes to survey (versus experimental) research.

More Knowledge Quotations

Information is not knowledge
- Albert Einstein

Knowledge is no value unless you put it into practice
- Anton Chekhov

Take the two famous quotes above and you actually run smack into an ongoing controversy among those who study knowledge in general and political knowledge in particular. Is what people know sufficient to assume sophistication? Those who study expertise or sophistication or similar constructs would argue no, that knowledge itself is necessary but not sufficient. It's not what ya know, it's what ya know and how ya use it that matters.

This gets at how we measure knowledge. In political studies, we rely heavily on those civics class questions of who is Nancy Pelosi or what party controls the U.S. House of Representatives. We assume if you can rattle off the answers correctly, you're somehow more knowledgeable than those who cannot -- but we rarely build into our models how people use such information, if at all. Exceptions are when we see studies of how people organize their political world, often along partisan or ideological lines, and the role civics/textbook knowledge plays in all that.

Studies of sophistication, they use knowledge, but they also tend to use other factors in a multidimensional (and often muddy) construct. In other words, I'm not sure we gain all that much in our studies of "expertise" and "sophistication," which brings us back to our tired yet true measures of civics knowledge. They're simple, they hang together well (Cronbach's Alpha is usually in the .70s), and they have years of previous research to back 'em up. So we default to the conceptual safe ground.

But information, as the good professor notes above, is not knowledge.

It's an interesting problem for those of us who mess with these concepts, who try to analyze and predict based on them, who desperately need to publish in academic journals for promotion or tenure or a better raise (when there are raises and not furloughs). For me it's mostly an interesting intellectual exercise, and eventually I'd love to create to final, best, ultimate measure of sophistication that makes actual sense. Or, failing that, a knowledge measure that answers many of the criticisms I mentioned above, and the many I've not bothered here to get into (but have discussed in other posts).

Maybe it's time for a Political Wisdom construct.

Huh?

Here's my model, the old guys you'd see around the table at some diner, sipping coffee and arguing politics, and some snotty college kid sits down and can tell them all the names they get wrong, the little factoids they get confused, but when it comes to knowing -- the old guys beat the snotty kid every time (in this scenario, I was once the snotty kid...long story). It's how you use information, how you make sense of the world, that matters most. Hence my new construct -- Political Wisdom.

And no, I have absolutely no idea how to measure it, especially in a telephone survey.

Americans, Congress, and Democratic Responsiveness

I've been skimming a new book by David Jones and Monika McDermott (image to the right) that looks at Americans and Congress and how the two interact. Chapter 3 is of particular importance to me since it explores what people know about Congress and the consequences of that knowledge.

You can typically divide scholars of political knowledge into two camps. There are those who think the public fails to meet its democratic responsibilities of being informed, and there are those who think the public manages quite nicely, thank you very much, and any failure has more to do with the kinds of questions we ask or the ways we ask them.

Jones and McDermott fall into the second group.

The authors make some good points in criticizing the kinds of questions used to tap the public's knowledge, but these criticisms are nothing new. Even people who use them are critical of their limitations. I'll probably discuss the book again this week, but one point sticks out in my mind, that of the traditional which party controls Congress question often used in creating an index of political knowledge. They make a good point that the post-election surveys that ask people to remember which party controlled the House or Senate back before the election is probably an unfair test. People do lousy on these, and with good reason. "Such a retrospective fact question is a particularly challenging test for respondents," they write on page 47.

This gets into models of how people learn, remember, and retrieve information. A better question, they argue, is the prospective one that asks which party is about to control Congress. "The information is salient," they say, "having been amply covered in the media as well as very recent, and should therefore still be readily available to citizens."

Their analyses "provide at least a ray of hope" in the public's ability to perform its democratic duties, they conclude. The book's overall conclusions are equally optimistic. If time in a busy semester allows, I'll dig deeper and write more. Or you can just get the book yourself and check it out.

Youth, Participation, and Knowledge

There was greater political participation by young people in the 2008 presidential election but that did not necessarily translate into greater political knowledge, according to a news study (press release here).

First, some caveats. This study was done by graduate students. It was presented at the recent AEJMC convention (I'm a member). So I can't say a lot about the method, because I skipped this year's convention in Boston. Caveats aside, we can either read the press release or wait until the papers themselves become available online. It's basically a study of undergraduates -- who as we all know barely qualify as real people. And there's this little bit of info near the end of the release: "the study has limitations, particularly since the students were not selected from a random sample." Yup. Okay for a conference paper, harder to sell as a journal article.

Here's an interesting quote from the press release:

"We found that the students were really politically active," York said. "They talked about the campaigns with their friends, and a lot of people got online on a social networking site to talk about the campaigns. Not many wrote blogs, but a considerable amount kept up with blogs."

This gets at the difference in what is participation. We'd normally define it as attending rallies, contributing money, putting a bumpersticker on your car, that sort of traditional thing, but there is movement to include social networking, Internet-based activities as participation.

"The measure for political knowledge was similar to a current events quiz with questions like the name of the U.S. secretary of defense," according to the release. That's okay too, though as I've discussed many times before in previous blogs, it's just one form of political knowledge.

A Specificity Index for Open-Ended Coding

I blogged yesterday (see below) about problems in coding open-ended responses to survey political knowledge questions. I used the Nancy Pelosi question as an example. The "correct" response, from a scholarly standpoint, would have respondents identify her as Speaker of the House, but I also argued that it was equally correct to identify her as a congresswoman, a member of the House, and a lot of other answers that in the past would have been coded as "incorrect" in the American National Election Studies dataset.

Go back to yesterday's post for links to ANES data, the newly released raw open-ended responses, and other important points of interest, especially problems with earlier coding.

We don't know exactly how ANES staff will code these answers, but when release the next version of the 2008 pre- and post-election data, I'll do a comparison then. Today I offer an alternative Specificity approach to coding. It's simple. Anything resembling "Speaker of the House," given its specificity, gets coded as the highest, most correct, response. Let's call it a "3" for the sake of argument. Identifying Pelosi as a member of the House, while correct, loses that specificity, so it gets a "2." Calling her a politician or something similar, that's correct in a vague sort of way, so it scores a "1." And getting it wrong, that's a "0."

Missing and refusals get their own special codes. Scholarly typically recode a refusal to be the same as an "incorrect" response. That's a different problem for a different day.

My specificity method provides greater data range. If someone doesn't like it, they can collapse the resulting codes into any method that strikes them as useful, especially if they're comparing answers in 2008 with some previous year.

Coding Responses

The American National Election Studies released this week the redacted responses to various open-ended questions -- including four political knowledge items -- from the 2008 study (study page here if you'd like to download the Excel file). These are the standard office recognition questions that provide a name and ask respondents to identify "what political office does he/she now hold?" The political figures were John Roberts, Dick Cheney, Gordon Brown, and Nancy Pelosi.

Later, ANES will code these as correct or incorrect, but as some of you know there have been problems with earlier coding of responses and I've blogged about this in the past here and here. ANES has its own report. Basically, some answers that should have been coded as "correct" were instead coded as "incorrect," thus deflating the public's actual knowledge as reported in these influential surveys. Some scholars studying knowledge about the U.S. Supreme Court uncovered the problem.

Okay, back to the present. There's a nice Excel file with all the various responses. It's time for a coding lesson, children, as in -- how do I code this? Take, for example, the question about Nancy Pelosi, who was (and is) Speaker of the House. Is the following response correct or incorrect? I've included misspellings.

• "she is a congressmand out of California"
• "congresswoman in Ca. 8th district"
• "majority leader in congress, democrat"
• "house of representative member"
• "specker of the house"
• "governor of Alaska, John's running mate."

Okay, we can all agree that last one is definitely coded as "incorrect." Other fun incorrect responses have her as a Republican, as a Supreme Court judge, or as the secretary of defense. Many are easy to code as right or wrong. But the others above, how do you code them? She is a congresswoman. She is a member of the House. Are those wrong just because we were hoping people to specifically identify her as Speaker and not merely (but correctly) as a member of the House of Representatives?

It'll be interesting to see how ANES staff code these in the next release. When that happens, I promise to compare these responses with their correct/incorrect dichotomy and report back, because honestly I don't know how to handle them myself. I have a few ideas and I may share some in a later post, but it has to do with a specificity scale (more later). If anyone has suggestions, lemme know. I'd be happy to hear them.

Kinds of Knowledge

There are many ways to measure what people know. Sometimes scholars and others use them interchangeably, missing the nuances. Below are a few:

Civics Knowledge -- usually measured by questions such as "what branch of government interprets the constitution." These are aimed at understanding a person's base level of how government is structured and works. Rarely used as a dependent variable except by people who study socializing of immigrants into the U.S. or in youth learning the basics of government, it most often shows up as an independent or control variable for other kinds of political knowledge. But you'll often see this kind of question as a generic "political knowledge" measure, especially in news stories bemoaning the fact that so few people can identity some core aspect of democracy. Not a good measure of general political knowledge and lousy in combination with media effects.

Campaign Knowledge -- usually a measure of an active political campaign, most often how various candidates stand on particular issues or, sometimes, questions like "what candidate recently said xyz." A very good measure if you're looking to explain specific campaign events or factors that lead to this kind of knowledge, such as what media best predict knowledge of some recent campaign event. Good measure with media variables.

Current Events Knowledge -- Seen perhaps most often as a measure of "political knowledge," here I'm separating it from "campaign knowledge" because, what the hell, I need more categories. Current events is a wide ranging category. It could something simple like "what party controls the U.S. Senate" or "what happened last week in Iraq?" This is a really good measure if you're studying media effects.

Political Actor Knowledge -- by actors I don't mean on the silver screen but rather measures that ask such questions as, "Who is Nancy Pelosi?" We typically prompt with a name and ask for the office, though sometimes we'll prompt with the office and ask for a name: "Who is the Speaker of the House?" Methodological note: a recent study explained the gender differences (men scoring higher than women on political knowledge tests) in part because women are rarely used in name-prompt questions. So in other words, men do better because we mostly ask about men who are political actors or public figures. So-so for media studies.

Misinformation -- better known as anti-knowledge, or knowledge of incorrect facts such as the belief that Barrack Obama is Muslim. This is the type of knowledge that most readily gets mixed up with attitudes (I'll discuss this later in the week in a post about public relations challenges). We don't study this one an awful lot but it's a great one for media scholars since certain kinds of media content (talk radio, for example) tends to be associated with incorrect knowledge.

There are no doubt others I'm missing. As they come to me, I'll add 'em. Suggestions welcome.

Knowledge about the Supreme Court

I've blogged a number of times about problems the American National Election Studies had in how they coded certain of their political knowledge questions. This came up because some scholars studying people's knowledge of the U.S. Supreme Court or its justices found inconsistencies in the ANES data, at least that didn't match what their own data showed. There's a good ANES report on just this problem.

Now the study these scholars was working on has been published. Below is a key graph describing some of the coding problems. It's worth a read for the methodologically inclined:

But these are not the only limitations plaguing this approach to measuring political knowledge. Most worrisome, in one instance, the ANES required its interviewers to code the accuracy of the respondents’ answers to the knowledge question during the interview itself, apparently using quite stringent criteria. Thus, if one replies that William Rehnquist is ‘‘the Chief Justice of the U.S. Supreme Court,’’ the interviewer would, according to ANES coding rules, record the answer as ‘‘correct’’ (even though the official title of the leader of the Court is the ‘‘Chief Justice of the United States’’). Also according to the ANES rules, references to Rehnquist as a Supreme Court judge who is the head honcho or main guy or the main one are scored as incorrect. According to these strict procedures, only 10.5% of the respondents ‘‘correctly’’ identified Rehnquist in the 2000 ANES.

Basically the open-ended responses were miscoded, or coded too strictly, or at times not even accurately when considering the official title of the chief justice. This affects a small but significant number of responses, which in turn can influence not only our evaluations of what people know about the Supreme Court but the consequences of that knowledge when examining its relationship with other variables. For example, the authors examine the role knowledge has in institutional loyalty.

Ultimately, they write:

But we do assert that the image of the American people as entirely bereft of information about courts, as ignorant of their role in the American democracy and their importance as makers of public policy, and as oblivious to the nature of judicial institutions and processes, most likely undercredits ordinary people.

In other words, the "Just How Stupid Are We" question becomes "Just How Stupid is Our Methodology?" In defense of ANES, they are working hard to repair the miscoding of previous data sets (these guys do great work) and in the latest 2008 pre- and post-election release the knowledge items are not even available yet (which is kinda pissing me off given that I need them for some work I'm doing). So the Pelosi, Cheney, Brown, and Roberts identification codes all are listed as -3, as in "we dunno yet, but we're working on it." I figure we'll see 'em sometime later this summer.

What We KnowvsWhat We Can Find Out

Does political knowledge matter in a Google world? In other words, is what people know all that important as compared to what people can find out?

I've been mulling this one over. Here I am, a guy who studies how people acquire political knowledge and the role the news media play in the process, and now to make my life worse I'm wondering how much all this matters. Does it make me more knowledgeable that I can name my U.S. representative and senators, or am I knowledgeable given I can Google those names quickly and efficiently.

Is mere factual recall a good measurement of knowledge? Or is it the ability to quickly find and use that information that matters?

In part I'm splitting conceptual hairs, in part I'm tinkering with knowledge versus sophistication versus expertise versus (insert your favorite variable here). But now I wonder whether the ability to search and find information is not, in itself, a form of political knowledge. My gut says no. My head says ... hrmmm ... mebbe.

Obviously from a measurement standpoint we can't easily gauge someones ability to quickly find and use political information, especially not in a telephone survey, so those tests of political knowledge aren't going away any time soon. Just how to measure political knowledge is controversial. I've blogged on this in the past, I'll blog on it some more in the future. It's a great topic that cuts across research methods and theory and all the rest -- the kinds of questions we ask, the ways we ask them, how tinkering here and there can mess with the results.

But we return now to my original question: has what we mean by knowledge changed somehow in a Google world? I think it has, though I'm not exactly sure what that means or how you study it. I'm gonna wrestle this one down to the ground in the next few weeks, get a little dirty in the process, and decide whether it's (1) a question worth asking and (2) if so, how to go about it, and (3) is age a significant factor here.

What People See -- in Surveys

An interesting Public Opinion Quarterly article examines how people "see" a questionnaire. They use eye-tracking hardware and software to examine, among other things, how respondents view a survey question and response alternatives (our fancy way of saying the various choices you're given online when asked to answer a survey question).

According to the authors, when all choices are visible, "respondents looked at the options near the top of the list longer than they looked at those at the bottom. In addition, they were more likely to choose the options at the top."

This neat study doesn't look specifically at political knowledge, so I'm going to extrapolate and ponder here just what this tendency might mean for respondents facing questions designed to tap their knowledge of politics and public affairs.

Could this introduce random or even systematic error into responses? Yes. Could this lower our estimation of people's actual knowledge? Probably. Should be put the "right" answer near the top so they can find it? I dunno, but my hunch is no.

I'd love to see this approach used specifically on political knowledge questions and put the "correct" answer in different locations to see whether that matters. All students know that on multiple choice tests, if all else fails, choose "C" as your answer, so I wonder whether in survey items in which there is a "correct" response (versus just your attitude about some policy or issue) the respondents would be more likely to seek out that right answer. I think so, but it needs testing.

Unfortunately I don't own some eye-tracking gear. Sigh.

Summing Up IV

This the fourth in (maybe) five installments this week to sum up what we know about the relationship between media use and political knowledge. Monday it was TV news, Tuesday entertainment faux news programs, Wednesday I discussed radio.

Today -- newspapers.

You know, those smushed-tree and ink things they quaintly toss on driveways in your neighborhood. That old guy down the street, he gets one. Hobbles out every morning in his robe, stoops down and groans while farting, hobbles back to his kitchen table to suck down coffee and figure out what the hell happened in the world yesterday.

That's a newspaper -- yesterday's news tomorrow.

I love papers. I worked at three daily newspapers in my unstellar journalism career. Covered cops and courts and politics and government and schools and a bunch of other stuff. Took pix. Wrote features. Even did some obits. Had a helluva time, but that's all beside the point here, which is how do newspapers fit in with political knowledge.

If you read previous studies you find this nice, modest, statistically significant relationship between reading the newspaper and what people know. In part this can be explained by the demographics of newspaper readers: older, more interested, more educated. In part this can be explained by the way we approach print as an active process versus how we approach the screen, a more passive process. In part it's the depth of information on a newspaper's front page and inside pages, far more than you get from TV. The old saying was that you'd find as much info on the front page of a newspaper than in 20 minutes of TV, and that's kinda true and kinda false, but it's also beyond our discussion here.

BUT ... and this is a major but, the way newspaper journalists tend to write stories, via the inverted pyramid, makes it difficult for young or uninterested readers to figure out what the hell is going on. A summary lede is fine and I love it dearly, but it sucks for some readers who do not have the context or background knowledge to understand what it's talking about.

Now here I'm talking the paper version, not the online version, but there's a cool study that compared the same exact NYT story in print and on a screen. They measured knowledge from reading the same exact story. And found ... (drum roll please) ... that people learned more from the print version. Holy dominance of print, Batman! Why? I believe again that it has everything to do with the way we approach a printed product versus one on the screen (or boob tube). There is some work to support this.

Anyway, put newspaper exposure into a multivariate model, control for all the other usual suspects in predicting political knowledge, and you'll often find newspaper reading still holds some significance. Small, modest, but often statistically significant at that magical p<.05.

But as newspaper reading drops, and online grazing replaces it, we'll see these relationships disappear in future studies.

Why? Because people will say they read newspapers online, but that exposure will reflect grazing that we all tend to do online versus a deeper, longer soak we tend to do on paper, and also will reflect the seriousness we read a paper product versus a screen product. In the long run, the relationship between "reading news" and political knowledge will -- in my humble estimation -- disappear.

Watch for it.

When Knowledge Items Go Bad

I'm working with a large data set now that includes several measures of political knowledge. One set of questions gets at what people know of the two candidates (their religion, state of residence, demographic stuff). The Cronbach's Alpha for this index is not great, about .55. An index of standard political knowledge items such as how long is a U.S. senator's term, how many times can a president be elected, standard textbook civics class stuff, that is only .60 or so. Not good.

But questions about the candidates stands on issues, that alpha is awful, somewhere between .40 and .50 depending on which items I use. These are way too low and, to be honest, unexplainable.

Oh it gets better. These indices, created quick and dirty to peek under the hood of my data, don't line up at all with my key independent variable. And dammit they should. This is a great theory with the data getting in the way (I'll discuss more fully at another time, closer to subbing this thing to a journal). Basically, there should some relationship here.

I've blown two long afternoons on massive data recoding and analysis, triple checking all my recodes.

I shoulda been a plumber. Named Joe.

More on New Data

I've just started digging into the 2008-2009 panel data released over the weekend by the ANES and the political knowledge questions look quite useful. All ANES data can be found here.

Best I can tell there are six waves of data collection, but not every wave has questions tapping respondent political knowledge. But many do. The second and sixth waves, for example, include basic questions asking people how many times a person can be elected U.S. president, how many years in a full time U.S. Senate seat, and so on. Textbook civics stuff, the kind that gives you a sense of how much a respondent understands about the workings of government.

And then there are some candidate-specific questions in the fourth and sixth waves, asking the state and religion of the two major party prez candidates, plus one each on where they worked before going into politics. Interesting.

But best of all is a set of policy questions asking where Barack Obama and John McCain stood. These offer some campaign-specific ways to test respondent knowledge. Very useful.

The great thing about panel data, of course, is they provide various answers by the same people over time so you can judge change. Using these data is tricky. There are mutiple weights that have to be applied and you have to be careful in your statistical tests, but they provide the kind of power a single snapshop can never give you. I'll be playing with these data for the rest of the semester. Got a JMCQ article already in mind.

Tomorrow, a look at the media variables in the data set.

ANES and Knowledge

Quite some time back I blogged about problems in how the American National Election Studies coded responses to their political knowledge questions, a situation uncovered by a couple of scholars doing work on what people know about the U.S. Supreme Court.

A new report discusses these problems (a longer journal article is forthcoming). For those who rely on secondary data in general and ANES data in particular to examine political knowledge, this brief report is worth the read. The problem comes down to this -- coding for wrong answers included ones that should have been categorized as correct. In asking what office does William Rhenquist hold (he was chief justice of the U.S. Supreme Court at the time), listed as wrong were such responses as "supreme court justice head honcho" and "supreme court justice, head." Should have been listed as correct.

To their credit, the fine folks at ANES have worked hard to understand and address the problem, even providing access to open-ended materials so people can more dutifully check whether a political knowledge answer is indeed incorrect.

All of us should appreciate the work of the report authors, James L. Gibson and Gregory A. Caldeira.

Discouraging "Don't know" Responses

A Methodological Moment

An interesting Public Opinion Quarterly piece follows a line of research about the measurement of political knowledge and whether we should encourage or discourage "don't know" responses. Using an online survey, the authors argue that eliminating the "DK" response works fine and it removes self confidence and other psychological factors that influence results.

It's a fine analysis, excellent discussion. Given the holiday season and grades to be recorded, I'm gonna talk only about a few key points.

"With the DK-omitted strategy, confident risk takers are no longer privileged relative to those who lack confidence or are risk averse."

First, the writer in me absolutely hates the use of privileged in this instance. Setting that aside, basically people willing to guess often get items right even when they don't know. Removing DK takes this out of the equation.

"While estimates of political knowledge have been consistently low, it is clear that question design may be a confounding factor. As web surveys become more ubiquitous, the DK omitted strategy is an increasingly viable option for measuring political knowledge."

I agree, but worry how this applies to traditional telephone or F2F (face to face) surveys. "I dunno" is an easy out, true, but in those people-based interactions not giving them an out might create survey difficulties. Hard feelings. Frustration. Guessing. All bad, all adding random error or perhaps even systematic error. In most surveys, the "don't knows" are often collapsed with incorrect answers to be scored as a "0" and correct responses getting a "1" in some kind of summed index. In other words, "don't know" often is the same as being wrong.

(As an aside, I prefer a -1 for getting it wrong, 0 for dunno, 1 for being right, but I've not used it very often because the results look just about the same as the traditional approach.)

But as we move more and more to online use of surveys, the arguments here are well taken and important for those who research what people know.

Creativity in Measurement

Measuring political knowledge can be easy. Which political party controls the House of Representatives? Who is the vice president? What candidate is against abortion? We can quibble with specific questions or wording, but agree that these more or less tap what we call political knowledge.

And we sometimes get creative.

Here are a few, using American National Elections Studies data:

• Gives response other than "don't know" to questions asking how well a U.S. representative has done his or her job
• Correctly identifies the incumbent in a district
• Gives any kind of answer other than "don't know" on a feeling thermometer.
• Remembers anything about an incumbent in a district

It's the feeling thermometer one that sometimes makes me wonder if we're not being a bit too creative. It's a standard question asking a respondent to rate a politician from 0 to 100 on how "warm" they feel. Answer a "100" and you'd have that person's baby, answer "0" and you'd eat them and their babies. But if you say "don't know" or do not provide an answer at all, it's often judged that you do not recognize the person and, thus, are less knowledgeable.

It's a bit of a stretch, but as I said above, sometimes we're forced to be creative when working with data in secondary analysis. If the items can be defended, if they all hang together, either in factor analysis or measured by Cronbach's Alpha, we're usually okay with the result.

I'm discussing this because I am about to plot out a study stretching from 2000 to 2008 on political knowledge that takes, best I can tell, a unique twist. As I progress and get closer to actual submission, I'll discuss it more openly, but right now the idea sounds cool. But as we all know, sometimes data get in the way of good theory.

Damn those data.