Measuring Political Knowledge via the ANES

Warning -- this is one my more PhDweeby posts. I'll skip the math, but in this post I will discuss a paper soon to be published in Political Analysis, a journal full of scary math.  It's about the usefulness of questions from the American National Election Studies, the dominant source of data in most political science, in measuring political knowledge.

Before we get to the meat of the thing, there's a nice section on why political knowledge matters, how it "shapes the behavior of citizens in a democracy," and a nice discussion of the different ways we measure the concept.  I may come back to that in a moment.

The authors took data from the 1992, 1996, 2000, and 2004 ANES.  In these data are various measures:

• Correct placement of the candidates and parties ideologically from one another.
• Identification of candidate and party positions on major policies.
• Identification of the positions held by major political actors

The authors took all these questions (there are a lot of them), summed them, divided by the number of items, and came up with a score.  Fairly common procedure.  After cranking the data through analyses to test for invariance, etc., see the graph below.  It's a little long, but worth the time:

Our results help explain why researchers have been frustrated in their attempt to measure and explain apparent knowledge gaps between various grouping variables including participation, media use, educational attainment, income, and age. Models seeking to explain observed differences have been unsuccessful because the construct of political knowledge is apparently qualitatively different between subgroups based on these grouping variables—excluding media use. Attempts to explain these differences will thus be unsuccessful, if political knowledge is measured using the full battery, because the measures are not sufficiently invariant to permit valid comparisons. The inconsistent results may also stem from a lack of conceptual clarity regarding the measurement model underpinning political knowledge. Our results suggest that the use of indexes assumed to be measured without error cannot be supported.

Not the "excluding media use."  That's nice to know for you budding political communication scholars out there and indeed, if you look at especially Figure 3, the media use items (in particular, newspaper use) really pop compared to others.

The authors argue that political knowledge scales must not be "established by fiat," a shot across the academic bow to most of us who use them.