Coding Hell

This is a problem for many who do research in political knowledge.  What's a right answer?

This gets a bit complicated.  Hang in there.

If I ask respondents what office does John Boehner hold, the right answer is obvious -- Speaker of the House.  But what if a respondent answers congressman or member of Congress.  They're not wrong answers, but are they right enough to code as being correct?  And what if they just say Republican leader?  That's right too.  Not what I asked -- name his office -- but not so very wrong either.

I've blogged about this several times, most recently here as I discussed an excellent study that looked at "partial right" responses to surveys.  The result there?  Folks who give kinda sorta right answers are an awful lot like the folks who give the completely right answers.  This is an enduring problem in political knowledge studies, especially as the ANES wrestled with coding issues of its knowledge question about the Supreme Court (see my link above, track back to earlier posts on this one, including a technical report).

Simply put -- what's a right answer?

I say this as I fiddle with data with just these kinds of questions and where I have to go in and decide what's a right (or wrong, or sorta wrong, or kinda wrong) answer.  My initial response is to code them in multiple ways (absolutely correct, right but not what I was looking for, not right but not wrong, wrong, and no answer).  That's five different codes.  We often dump the non-responses and incorrect responses together (that's a different methodological problem for another day).  So if a respondent didn't answer, or answered incorrectly, that's be a "0" for that question, a "1" if they got it right.  Then we'd sum their correct answers on an index designed to measure, yup, political knowledge.

But a lot of "0" responses sometimes are not so very wrong.  And is an incorrect answer the same as no answer?  Not really.   There's work out there that suggests, for example, that men are more willing than women to guess on these kinds of questions.  They get some of those guesses right, those end up with higher scores since we often through coding do not punish being wrong any more than simply saying "I dunno."  In other words, we treat both the same -- and probably shouldn't.

I'll probably use the multiple coding scheme above, collapse them if I don't see significant differences, and move on.  But it's a serious issue that, to be honest, raises serious questions about a host of previous studies on the topic.