Is Political Knowledge Research in a Crisis?

"Political knowledge research faces a problem, perhaps even a crisis."

Dire words.  Scary, if like me you make your bit of bread and meat by doing political knowledge research.  But is it so?

This study points out the well-documented problems with how scholars measure political knowledge, in particular the problems found in ANES measures -- a topic I've discussed at length and others have examined for years and years.  You may not have access to the complete article.  If not, I can send you a pdf.

What's the problem?  Deciding on whether an answer is correct, in the ballpark, or incorrect.  If I ask you "Who is Barry Hollander" the correct answer is "journalism professor."  But "he teaches journalism" is damn close, in the ballpark, but sometimes answers like that were judged as incorrect.  The difficulties here were famously discovered in questions about the Chief Justice of the Supreme Court in which a lot of answers, perfectly reasonable answers, were coded as incorrect.

The authors devised an approach, time consuming, that resulted in high reliability.  The try at a machine method, an automated approach, was less successful but still shows promise.