Creativity in Measurement

Measuring political knowledge can be easy. Which political party controls the House of Representatives? Who is the vice president? What candidate is against abortion? We can quibble with specific questions or wording, but agree that these more or less tap what we call political knowledge.

And we sometimes get creative.

Here are a few, using American National Elections Studies data:

• Gives response other than "don't know" to questions asking how well a U.S. representative has done his or her job
• Correctly identifies the incumbent in a district
• Gives any kind of answer other than "don't know" on a feeling thermometer.
• Remembers anything about an incumbent in a district

It's the feeling thermometer one that sometimes makes me wonder if we're not being a bit too creative. It's a standard question asking a respondent to rate a politician from 0 to 100 on how "warm" they feel. Answer a "100" and you'd have that person's baby, answer "0" and you'd eat them and their babies. But if you say "don't know" or do not provide an answer at all, it's often judged that you do not recognize the person and, thus, are less knowledgeable.

It's a bit of a stretch, but as I said above, sometimes we're forced to be creative when working with data in secondary analysis. If the items can be defended, if they all hang together, either in factor analysis or measured by Cronbach's Alpha, we're usually okay with the result.

I'm discussing this because I am about to plot out a study stretching from 2000 to 2008 on political knowledge that takes, best I can tell, a unique twist. As I progress and get closer to actual submission, I'll discuss it more openly, but right now the idea sounds cool. But as we all know, sometimes data get in the way of good theory.

Damn those data.