What Republicans Know -- Part 2

Tuesday I wrote about an interesting survey that examined, in part, how political knowledge is associated with who Republicans favor in the nominating process.  I mentioned that I couldn't find the actual knowledge questions.  Thankfully Dan Cassino, lead investigator of the study, saw my post and pointed me to the right place to find the info.

WARNING: Long post below.

You have to scroll down a bit, to page 4, to find the beginning of the questions.  Let's take a closer look.

[First, a brief aside.  In writing political knowledge questions, you want some to be more difficult than others to discriminate among respondents.  But -- and this is important -- you want to avoid the impossible-to-answer questions.  Finally, you want the questions to be unloaded, especially along partisan lines, unless of course that's what you're studying.  Simply put, you don't want questions Republicans or Democrats might be naturally better at answering.]

Back to the survey questions.  I'm not picking at this survey in particular, which is fine with some neat results, but it uses enough different kinds of questions that we can learn something from it.

• The first two questions are "yes-no" types that ask whether the leaders of Egypt and Syria have been removed.  Straightforward, simple, and roughly equivalent in difficulty.  No odd partisan breakdowns in the results.  About half of respondents got these right.

• The third question asks what country has done the most to financially bail out fellow European countries. It's difficult for me to tell if this is a recognition question (the list of countries provided) or a recall question (have to generate the correct answer -- Germany -- off the top of your head).  Yes, the difference can matter (I'm finishing a paper on just this topic).  This looks like free recall to me, given the other response alternatives. Only a quarter of respondents nailed this one.

• The fourth question asks what the sanctions about Iran are supposed to accomplish.  Again, probably free recall given the "Anything about Nuclear Program or Uranium enrichment or WMDs…" is listed as an accurate response.  This is the fourth international question in a row (though I assume the questions were asked in the survey in random order, so no problem). Nearly half got this one correct.

• Question five asks what party controls the U.S. House of Representatives, a traditional question asked in most academic surveys (ANES, for example).   I'm frankly surprised the Republicans didn't outperform Democrats even more on this one.  Two-thirds of respondents knew this.

• The sixth question requires I cut-and-paste.  It asks: "In December, House Republicans agreed to a short-term extension of a payroll tax cut, but only if President Obama agreed to do what? [Open-Ended]"  By noting this is "open-ended" it makes me wonder if I was wrong above in my guess as to the previous questions being cued or free recall.  This is a particularly difficult question, with 9 percent getting it right and 72 percent answering some form of "don't know."  That's high.

• The next two questions ask specifically about the Republican primaries (who won Iowa and New Hampshire).  In other words, GOPers are a lot more likely to answer these two correctly.  Indeed this was the case by 12 and 17 percentage points.

• The final question is a tricky one.  It asks what percentage of Americans are unemployed.  Why tricky?  Do we rely on the official number or the, arguably, real number -- which is significantly higher?  This was also open-ended.  Republicans did better than Democrats (on every question above, including this one) but their wrong answers skewed higher than Democratic wrong answers, no doubt reflecting GOP pessimism about President Barack Obama and the economy and ... if you listen to Fox News or Limbaugh or Hannity, the higher number is always stressed.

So what can we take away from the questions above?  Half are domestic, half international; that's a nice balance.  Republicans tended to do better than Democrats on all the questions, which is not surprising.  If we controlled statistically for education, those differences would largely disappear.

At least one question is probably too difficult (#6).

Overall, the public didn't perform particularly well.  Twelve percent got no question correct and only 3 percent got all eight correct.  It's that last number that worries me.  While there's no hard and fast rule about this sort of thing, I'd like to see that higher -- both because we're prefer in a democracy for more people to know what's happening in their political world, but also for simple methodological reasons.  I think a relatively fair test of current events knowledge should result in at least 10 percent getting all eight items correct, but I invented that number out of thin air.  I'd want to go back and check the Pew current event quizzes to see what percentage gets all their questions correct and use that as a guideline.

Finally, an interesting (to me) statistical bit of trivia.  Democrat "leaners" did better on the questions than pure Democrats.  What's a "leaner?"  Traditionally we ask respondents for their party identification and if they name Republican or Democrat right off, they go to the end of the continuum (far left and far right).  If they didn't name either party, we then follow up by asking which party they lean toward. These folks are categorized just to the inside of our hard-core Dems and GOPers.  In the middle go our true Independents who held out after two (or sometimes more) questions and probes designed to force them to inch one way or the other.  Why would leaners do better than party purists?  Lots of reasons, but that's probably a post for another day.