What Republicans Know -- Part 2

Tuesday I wrote about an interesting survey that examined, in part, how political knowledge is associated with who Republicans favor in the nominating process.  I mentioned that I couldn't find the actual knowledge questions.  Thankfully Dan Cassino, lead investigator of the study, saw my post and pointed me to the right place to find the info.

WARNING: Long post below.

You have to scroll down a bit, to page 4, to find the beginning of the questions.  Let's take a closer look.

[First, a brief aside.  In writing political knowledge questions, you want some to be more difficult than others to discriminate among respondents.  But -- and this is important -- you want to avoid the impossible-to-answer questions.  Finally, you want the questions to be unloaded, especially along partisan lines, unless of course that's what you're studying.  Simply put, you don't want questions Republicans or Democrats might be naturally better at answering.]

Back to the survey questions.  I'm not picking at this survey in particular, which is fine with some neat results, but it uses enough different kinds of questions that we can learn something from it.

• The first two questions are "yes-no" types that ask whether the leaders of Egypt and Syria have been removed.  Straightforward, simple, and roughly equivalent in difficulty.  No odd partisan breakdowns in the results.  About half of respondents got these right.

• The third question asks what country has done the most to financially bail out fellow European countries. It's difficult for me to tell if this is a recognition question (the list of countries provided) or a recall question (have to generate the correct answer -- Germany -- off the top of your head).  Yes, the difference can matter (I'm finishing a paper on just this topic).  This looks like free recall to me, given the other response alternatives. Only a quarter of respondents nailed this one.

• The fourth question asks what the sanctions about Iran are supposed to accomplish.  Again, probably free recall given the "Anything about Nuclear Program or Uranium enrichment or WMDs…" is listed as an accurate response.  This is the fourth international question in a row (though I assume the questions were asked in the survey in random order, so no problem). Nearly half got this one correct.

• Question five asks what party controls the U.S. House of Representatives, a traditional question asked in most academic surveys (ANES, for example).   I'm frankly surprised the Republicans didn't outperform Democrats even more on this one.  Two-thirds of respondents knew this.

• The sixth question requires I cut-and-paste.  It asks: "In December, House Republicans agreed to a short-term extension of a payroll tax cut, but only if President Obama agreed to do what? [Open-Ended]"  By noting this is "open-ended" it makes me wonder if I was wrong above in my guess as to the previous questions being cued or free recall.  This is a particularly difficult question, with 9 percent getting it right and 72 percent answering some form of "don't know."  That's high.

• The next two questions ask specifically about the Republican primaries (who won Iowa and New Hampshire).  In other words, GOPers are a lot more likely to answer these two correctly.  Indeed this was the case by 12 and 17 percentage points.

• The final question is a tricky one.  It asks what percentage of Americans are unemployed.  Why tricky?  Do we rely on the official number or the, arguably, real number -- which is significantly higher?  This was also open-ended.  Republicans did better than Democrats (on every question above, including this one) but their wrong answers skewed higher than Democratic wrong answers, no doubt reflecting GOP pessimism about President Barack Obama and the economy and ... if you listen to Fox News or Limbaugh or Hannity, the higher number is always stressed.

So what can we take away from the questions above?  Half are domestic, half international; that's a nice balance.  Republicans tended to do better than Democrats on all the questions, which is not surprising.  If we controlled statistically for education, those differences would largely disappear.

At least one question is probably too difficult (#6).

Overall, the public didn't perform particularly well.  Twelve percent got no question correct and only 3 percent got all eight correct.  It's that last number that worries me.  While there's no hard and fast rule about this sort of thing, I'd like to see that higher -- both because we're prefer in a democracy for more people to know what's happening in their political world, but also for simple methodological reasons.  I think a relatively fair test of current events knowledge should result in at least 10 percent getting all eight items correct, but I invented that number out of thin air.  I'd want to go back and check the Pew current event quizzes to see what percentage gets all their questions correct and use that as a guideline.

Finally, an interesting (to me) statistical bit of trivia.  Democrat "leaners" did better on the questions than pure Democrats.  What's a "leaner?"  Traditionally we ask respondents for their party identification and if they name Republican or Democrat right off, they go to the end of the continuum (far left and far right).  If they didn't name either party, we then follow up by asking which party they lean toward. These folks are categorized just to the inside of our hard-core Dems and GOPers.  In the middle go our true Independents who held out after two (or sometimes more) questions and probes designed to force them to inch one way or the other.  Why would leaners do better than party purists?  Lots of reasons, but that's probably a post for another day.

Coding Hell

This is a problem for many who do research in political knowledge.  What's a right answer?

This gets a bit complicated.  Hang in there.

If I ask respondents what office does John Boehner hold, the right answer is obvious -- Speaker of the House.  But what if a respondent answers congressman or member of Congress.  They're not wrong answers, but are they right enough to code as being correct?  And what if they just say Republican leader?  That's right too.  Not what I asked -- name his office -- but not so very wrong either.

I've blogged about this several times, most recently here as I discussed an excellent study that looked at "partial right" responses to surveys.  The result there?  Folks who give kinda sorta right answers are an awful lot like the folks who give the completely right answers.  This is an enduring problem in political knowledge studies, especially as the ANES wrestled with coding issues of its knowledge question about the Supreme Court (see my link above, track back to earlier posts on this one, including a technical report).

Simply put -- what's a right answer?

I say this as I fiddle with data with just these kinds of questions and where I have to go in and decide what's a right (or wrong, or sorta wrong, or kinda wrong) answer.  My initial response is to code them in multiple ways (absolutely correct, right but not what I was looking for, not right but not wrong, wrong, and no answer).  That's five different codes.  We often dump the non-responses and incorrect responses together (that's a different methodological problem for another day).  So if a respondent didn't answer, or answered incorrectly, that's be a "0" for that question, a "1" if they got it right.  Then we'd sum their correct answers on an index designed to measure, yup, political knowledge.

But a lot of "0" responses sometimes are not so very wrong.  And is an incorrect answer the same as no answer?  Not really.   There's work out there that suggests, for example, that men are more willing than women to guess on these kinds of questions.  They get some of those guesses right, those end up with higher scores since we often through coding do not punish being wrong any more than simply saying "I dunno."  In other words, we treat both the same -- and probably shouldn't.

I'll probably use the multiple coding scheme above, collapse them if I don't see significant differences, and move on.  But it's a serious issue that, to be honest, raises serious questions about a host of previous studies on the topic.

Don't Know means, apparently, Don't Know

In the latest Journal of Politics, Robert Luskin and John Bullock examine whether "don't know" as an available response to questions about political knowledge can affect the results.  The study itself is here (assuming you have the same access as I do).

In survey research, we often talk about DK (don't know) and how to present this option, and whether presenting it in a certain way encourages people to take the easy road and simply say they don't know versus coming up, with a little more mental effort, an answer.  This matters, at least to those of us who study the knowledge of the American electorate.  Encouraging "don't know" as a response can lead to a more disappointing portrait of the public's knowledge.  Or, to flip it, discouraging this response would paint a better picture.

The authors used to national survey experiments to explore whether discouraging DKs will matter.  As they write:

Discouraging DKs does paint a more comforting picture of the public’s knowledge of politics—but, as the foregoing shows, only slightly so in the openended case and spuriously so in the closed-ended one. Anyone searching for large caches of hidden knowledge, it appears, should look elsewhere.

Given most of the time we rely on close-ended questions to tap into the public's political knowledge, the results basically tell us that fiddling with "don't know" as a response, usually by discouraging it or not offering it as an easily available response alternative in a survey scenario, won't really improve the results.  Indeed, the authors argue that DKs should not be discouraged.  The only question, they argue, is whether to encourage them.

Yes, this gets all manner of methodologically geeky, but for those who study what people know, and who craft surveys or other instruments designed to measure it, these results are important.

Full journal cite: The Journal of Politics, Vol. 73, No. 2, April 2011, Pp. 547–557.

Women Don't Know No Politics

It's a fairly standard, and often undiscussed for obvious reasons, finding in political knowledge research -- women do poorly as compared to men in tests of what people know.  I've blogged about this before.

I bravely return to this topic.

Why?  Because there's a study published in Journal of Politics that attempts to explain why gender differences occur.  It uses a strategy I've blogged about before, one previous studies have used, to help explain why women tend to do less well than men.  A number of theories have been posited to explain this gap.  Men guess more than women on such questions and the way we construct a political knowledge index tends to score "no answer" and an incorrect answer the same.  In other words, a of their few guesses will be right, thus giving men the advantage versus women who if they don't know, will simply say they don't know.  But a more plausible explanation is the kinds of questions we ask.

Or, as Kathleen Dolan writes in her JOP piece:

When we include political knowledge measures that ask for information on the present state of women in American politics, we see women’s traditional gender disadvantage wiped out. Women in the survey reported here hold as much information about women’s place in politics as do men. This offers support for the notion that knowledge levels are, in part, a reflection of the content of the items we employ.

To put it simply, methodology matters. Unfortunately there are no media variables in the study and I'd be curious to know if exposure to the news plays any role in creating differences, or evening the results, depending on the kinds of questions one asks to men and women.  It's an interesting question, one worth exploring.

Damn those UF Guys

A couple of University of Florida political scientists, Michael Martinez and Stephen Craig, stole one of my great research ideas.

Okay, they didn't steal it since it's kinda obvious, but they conducted a neat little study similar to what I had blogged about here and here and here.  That is, examining how people answer political knowledge questions to get a partial knowledge, answers that are kinda sorta right, all thanks to earlier work that found serious issues in the ANES coding of responses about Chief Justice William Rhenquist.  As you may remember, since I've written about this a couple of times, scholars uncovered problems with the coding.  As the Martinez and Craig conference paper notes:

On this latter point, for example, responses in 2000 that identified Rehnquist as “Chief Justice of the Supreme Court” were coded as correct – but nine people who said that he was “the Supreme Court justice in charge” (or used similar language, omitting the words “Chief Justice”), along with nearly 400 others who named him as a justice of the Supreme Court (without reference to his role as Chief) or simply as a judge, were all coded as being no better informed than those who might have identified Rehnquist (hypothetically) as Chairman of the Federal Reserve, Mayor of Phoenix, or Canadian Prime Minister. Counting the “justice” answers as partially correct, which they arguably were, would likely result in a more flattering characterization of the public’s awareness about the Supreme Court and politics in general (Krosnick et al. 2008).

In other words, kinda right should not be coded as the same as completely wrong.

In the paper, Martinez and Craig recode 2008 ANES data to see how a more careful coding scheme might change relationships among important variables.  The results?

• The old coding scheme made it appear Americans were more dense than they really are, at least when it comes to identifying significant political actors such as the Speaker of the House.
• The partial right folks resemble in many ways the traditionally correct folks in a series of political variables and sometimes those "partial right" respondents were actually higher in certain important variables.
• As the authors note, we need to think about whether "partial knowledge" may have different influences on political choice as compared to full knowledge.

As the authors note:

Like Art Linkletter’s kids, survey respondents can say the darnedest things, as some respondents apparently confused Nancy Pelosi with Sarah Palin, recalled that Dick Cheney shot someone (without noting that he was also Vice President), and guessed that Gordon Brown might be somehow related to the godfather of soul, James Brown.

For me, I'd love to see how "partial" versus "full" knowledge stacks up against news media exposure, something not analyzed in their paper.  I have my hunches as to particular media being better at predicting full knowledge (print) versus other media being better at predicting partial knowledge (broadcast news).  This would fall neatly into the recall versus recognition work I did some years back.


In full disclosure, I should point out that I knew both of these
political scientists while a mass comm grad student at UF
a million or so years ago, when dinosaurs roamed the campus.

What Men Think Women Know

I've written extensively about the relationship between what people know and what people think they know -- that is, actual knowledge versus perceived knowledge.  And, fascinated as I am with research methodology, I've also blogged about studies that consistently find men do better on tests of political knowledge than women (lots of reasons for this, some methodological, some cultural).

And now, a new study comes along that kinda combines the two.

Published in a recent Political Research Quarterly, the study finds that both men and women perceive women to be less knowledgeable about politics -- regardless of the actual knowledge each person has.

Here's an interesting bit, written in socialsciencespeak:

Though Huckfeldt (2001) asserts people can judge the expertise level of those with whom they discuss politics accurately, the perception effects we find here suggest otherwise; there is distortion within the judgment of expertise according to one’s gender. We suspect that even though discussion continues to take place at comparable levels between men and women main respondents and men or women discussants, the assessment that women know less about politics, even if that is not truly the case, may affect the dynamics of discussion within the dyad.

In other words, men assume women are not knowledgeable about politics without actually assessing what they know about politics.  The study goes on to discuss a number of potential consequences, all of them important.  A useful study if you're interested not only in political knowledge, but perceptions of knowledge and gender differences.

Men and Woman

I've written a number of times about the persistent difference found in studies of political knowledge when comparing the scores of men and women.  Guys, it seems, always seem to beat the gals.  And I've also written about some interesting approaches to explain (away) this difference.  One is in the kinds of questions asked.  When female politicians are used in knowledge questions, for example, the gap lessens. Also, men guess more and, given how knowledge surveys are often scored, that helps men.

Here I report on a study that shows it's not just a U.S. thing.  A survey in China, summarized here, finds the same difference between the sexes. 

Me, I avoid this argument, in part because my major professor from grad school, Mary Ann Ferguson, always told me to avoid attempting to explain sex differences in social science research.  She also hated content analysis, calling it the great intellectual cul-de-sac.

The Power of the Probe

A probe in survey lingo is a follow-up question prompted by a respondent's failure to answer a previous question.  For example, in the 2008 ANES, respondents were asked to identify Nancy Pelosi.  If they could not or did not answer, the interviewer was trained to prompt with "Well, what's your best guess?"

That's a probe.

Not all surveys probe an initial lack of response, and this can make a significant difference if we're studying something like political knowledge.  For example, the ANES made available a redacted version of the open-ended responses to certain questions, including those measuring political knowledge (download a zip file here of the Excel document).  It's interesting.  You can see the various open-ended responses, which I've blogged about previously, but they also include a column of the probes.  If a respondent gave an initial answer--right or wrong--the probe code is 5 (no probe) or one of the other codes that signify other stuff. 

A "1" in the probe meant they asked for a respondent's best guess.  And sometimes, the probe resulted in a respondent "guessing" correctly.  How often?  There were 1,294 instances of probes of the Pelosi question.  Roughly counting, I estimate at least a hundred instances, perhaps more, where the probe resulted in a correct answer.  And that's only looking at the Pelosi question. 

In other words, a probe can definitely influence the results, which I suspect has some bearing in analyses.  And this doesn't even get into what is considered a "correct" versus an "incorrect" response.

If I was so inclined, I'd do a paper on the power of probes to elicit a correct versus an incorrect response, and then position these competing approaches to political knowledge against key variables to see whether the probe improves results.  Honest, I'd do it, except I don't know where the heck I'd publish something like this.  Public Opinion Quarterly?  Dunno, cause I'm not sure I'm smart enough to successfully publish there.  It's full of folks far brighter than myself.