Al Jazeera

With a big splash, but with less audience, Al Jazeera premiered recently as the kinda sorta next big thing in U.S. cable news. The network vowed to focus on hard news, as opposed to the crap often seen on other cable networks (I'm talking about you, CNN, Fox, and MSNBC).

But what do people think about the network? Not much, if you believe this survey of over 8,000 respondents, and I especially like how they broke it down by state, including red states versus blue states, which you can pull off when your N is so big. Scroll down the page to find the maps.

Warning -- best I can tell, this is not a random sample. I can't tell at all how they got their respondents and that is, indeed, worrisome. That said, the numbers make sense.

Oh, you can see the raw data yourself on a Google docs page. Download it, if ya like, and crank away, though there's nothing there they don't show you on their graphs and maps.

Is Political Knowledge Research in a Crisis?

"Political knowledge research faces a problem, perhaps even a crisis."

Dire words.  Scary, if like me you make your bit of bread and meat by doing political knowledge research.  But is it so?

This study points out the well-documented problems with how scholars measure political knowledge, in particular the problems found in ANES measures -- a topic I've discussed at length and others have examined for years and years.  You may not have access to the complete article.  If not, I can send you a pdf.

What's the problem?  Deciding on whether an answer is correct, in the ballpark, or incorrect.  If I ask you "Who is Barry Hollander" the correct answer is "journalism professor."  But "he teaches journalism" is damn close, in the ballpark, but sometimes answers like that were judged as incorrect.  The difficulties here were famously discovered in questions about the Chief Justice of the Supreme Court in which a lot of answers, perfectly reasonable answers, were coded as incorrect.

The authors devised an approach, time consuming, that resulted in high reliability.  The try at a machine method, an automated approach, was less successful but still shows promise. 

Newspaper MeasuresA Methodological Moment

I blogged a day or so ago about an interesting study that looks at local versus national political knowledge, the role media play, and the important socio-demographic differences that emerge depending on which kind of knowledge you examine.  Now I'd like to spend a few words discussing a key, yet odd, variable in that study -- newspaper delivery.

Permit me a methodological moment.

In studies such as this we typically see newspaper exposure, traditionally measured on that old, tired, yet sturdy 0-to-7 days a week a respondent says he or she reads the newspaper.  An improvement on this exposure measure is attention.  That is, for those who report at least some exposure, how much do you attend to it.  This is particularly helpful in studying television news, less so for print.  And yet the study I linked to above uses, instead, newspaper delivery.

What are the advantages or disadvantages?

• Advantage I -- a respondent is less likely to exaggerate.  In other words, we often say we do some socially-acceptable task, like reading the paper or watching PBS, when perhaps we don't, or at least we don't do it as many days as we brag.  So this "delivery" measure helps.
• Disadvantage I -- but, we end up with a dichotomous variable versus a continuous one.  A 0-to-7 scale is richer than a 0,1 scale in so many ways.
• Disadvantage II -- and, many people read a newspaper at work, at the office.  The pass-along number boasted by the newspaper industry is about 2.3, as high as 2.5, meaning two or three people read a single issue of the paper.  The "delivery" item misses at least some of those folks.
• Advantage II -- but, there's less error (statistically speaking) in that 0,1 measure.  Cleaner results, I suspect.  No weird distributions.
• Disadvantage III -- and yet, and yet ... no one uses "delivery" so it's difficult to say how the results here fit with previous studies.  And use of key secondary data sets, such as Pew or ANES, will not give you a similar measure; those folks use some version of exposure and/or attention.  Hard to reproduce.

In all, I'm not enamored of the "deliver" item, and it loses above in a close game, 3-2. 

The "delivery" item has SES issues, I believe.  It also misses a lot of folks who may read the paper elsewhere.  But it does reduce some of the error from people overinflating their reading of the newspaper.  So you win something, you lose something.  But I believe, overall, you lose a bit too much to make it a viable replacement for exposure (even with all its flaws).

ANES 2008 ... still waiting

I respect the hard work it takes to create the ANES.  So my criticism here is mild, because I suspect they're wrestling with various difficulties.  But it's 2010 and we're still waiting for a final release of the 2008 election data.

Let's do some history.  An advanced version of the 2004 ANES data were available on Jan. 25, 2005, and the next release was April 19, 2005.  That's fast.  The advanced version of the 2008 ANES data were available on March 5, 2009 and there is a May version listed, but it's not the full data.  We've yet to see a final release of the 2008 data and, yes, it is 2010.

There are coding issues with open-ended variables.  There is the 2008-2009 panel data to contend with.  Between these two, I assume that's the holdup (though there have been no explanations on the web site, unless I'm missing something).  I didn't attend an AAPOR meeting where they were going to discuss the study but I was told by a staffer via email almost a year ago that they would put the presentation materials online so those of us without huge travel budgets could find a clue as to what's happening.  Still nothing up.

Why does this matter?

It doesn't, not for most folks unless you happen to be a data analysis dweeb like me.  I'm buried in a study right now and wouldn't have much time to play with the ANES data (though I do have one thing I'd like to explore and can only do so with the open-ended data included).  And the people who do this work at the National Election Studies are terrific, so I don't want to make this sound whiny, but updates -- a little transparency -- would be helpful.

Our Personal Networks: A Methodology Moment

We get our political information in lots of ways -- the news media, obviously, but also from the people we talk to every day, our social network.

It's the latter that is the focus of an article in the most recent issue of Public Opinion Quarterly. The authors look at two different ways of generating survey data about people's social networks.  As the authors write:

"One method asks respondents to provide information about the individuals with whom they discuss 'important matters,' while the other asks respondents to provide information about the individuals with whom they discuss politics.  We find that respondents provide more or less the same data on their political discussion partners regardless of which name generator procedure is used."

My first thought was -- wow, cool, you can get published with a null finding.

My second thought was -- there's leeway in how we study social networks, allowing us to confidently compare studies with slightly different methods.  Useful.

The authors conclude that our "core and political discussion networks comprise the same individuals."  In other words, we talk to the same people ... politics, or not.

That's good to know from a methodological standpoint.  I assumed people carve their larger social networks into smaller chunks.  This chunk is made up of people you talk about politics with, that chunk is people you talk music with, that other chunk is for religion, etc.  Maybe some overlap, but I always assumed there were differences.  This research suggests not.

I'm a bit skeptical. Maybe I'm generalizing from an N of 1, but maybe the lack of a result here has more to do with a methodological quirk in the wording ("important matters" versus "politics") than in how we consciously or unconsciously create our interpersonal networks.  I don't know.  My gut says the method here is too mild to truly capture differences in our interpersonal networks.  The lack of results are useful, especially if you want to combine data from different approaches in a larger analysis, but I don't think we can say quite yet that we have the same interpersonal network for everything.

Slate's News Experiment

Michael Kinsley of Slate poses this experiment: two journalists will spend one hour a day reading print newspapers, two will spend one hour a day getting their news online, but not shovelware or aggregators, so that leaves out Google News or Huffingtonpost but does include sites like Politico.

This goes for three days, then he gathers them back to discuss their experience. They're really doing this.

The headline for this piece reads Who's Better Informed, Newspaper Readers or Web Surfers? so I assume political learning is one of the examined outcomes, along with the overall experience, the frustrations, the good and bad, and of course the ugly, of getting news in more or less one fashion.

Let's set the record straight. This "highly unscientific experiment," as Kinsley calls it, is highly unscientific to the point of being silly. It's more of a vaguely interesting gimmick than anything else.

There's one published study that looked at the differences between people who read the New York Times online versus those who read the paper version. This controlled experiment randomly assigned people to one of two groups (print vs. online of the Times) and examined differences in learning and other factors. Print came out ahead (I can't lay my hands on the link, but I'm fairly certain it ran in Journalism and Mass Communication Quarterly).

Any differences have to do with the way we read, the way we approach online versus ink on paper. It's more than mere differences of medium, it's differences of motivation. We simply approach a screen differently, almost at an unconscious level, and subtle differences emerge.

This is a bit more PhDweeby than Kinsley wants, of course. The gimmick study will be fun for the discussion that emerges, but beyond that the results are meaningless. After all, they're looking at very different sources AND very different media. That, budding methodologists, is a confound of confounding proportions.

The Mess of Methodology

Doing social science is fun and frustrating, but nothing is as difficult as nailing your methodology. I work in media effects and political knowledge. It's messy stuff trying to tease out relationships, so when I saw this New York Times piece today about drinking and health, it struck me as eerily familiar. As you may know, drinking in moderation is often associated with health benefits. But ... moderate drinkers also tend to not smoke, to exercise, and either avoid or do all the stuff you're supposed to avoid or do to stay healthy.

So does booze really help? I like to think so. Helps me rationalize my evenings with wine, beer, bourbon, or of course the daddy of all drinks -- single-malt Scotch.

But I'm talking research methodology. Or as a sociologist quoted in the article says:

“The moderate drinkers tend to do everything right — they exercise, they don’t smoke, they eat right and they drink moderately,” said Kaye Middleton Fillmore, a retired sociologist from the University of California, San Francisco, who has criticized the research. “It’s very hard to disentangle all of that, and that’s a real problem.”

Yup. The same can be said for studying what people know. Knowledgeable folks in the first place tend to learn more from the same material than those with little knowledge. Motivation is a huge factor, as is cognitive ability (often measured as formal education, but that's problematic as well).

Disentangling all that equals heartache or a lifetime job. I'm going with the latter, though I often feel the former -- which usually means I need a drink.

Does Media Exposure Mean Anything Today?

A couple of days ago I blogged about a new POQ study that found people overestimate in surveys how much they consume news media (in this instance, TV news). Young people in particular were found to significantly overestimate their use of TV news.

So what?

From a data analysis standpoint this causes all kinds of problems. Imagine you're studying how media exposure is related to political knowledge. Now imagine you're particularly interested in how young people use the news media to keep up with politics and public affairs. For young people, though, you find no significant relationship between their media consumption and what they know.

If indeed young people overestimate their media use, the typical relationship between news exposure and political knowledge, for them, may well disappear when actually it exists. In other words, their overestimation of news media suggests there ought to be a stronger relationship than actually exists because, well, they said they use the news media more than they really do. A usual positive association between news media exposure and political knowledge, for young people, may well turn negative. And that's bad news.

Can we correct for this statistically? I don't think so. Can we preface our news exposure questions to make it okay for people to admit they don't consume as much news as they might want to answer to make themselves sound better? Yup. Can we go to some measure other than mere exposure? Attention helps some, but it also rides on top of exposure and that may create similar problems.

Is time spent with the media even meaningful? In our ADD culture, skimming and grazing and getting bits and pieces of news from here and there, I'm not sure time spent with the media even works today. ANES tried some experimental media exposure items in the 2008 pre- and post-election surveys, but I'm saving that for another post on another day, but my overall sense is they're no better than the original ones. I'll do actual analysis of the two and report back.

Don't Know

The latest issue of Public Opinion Quarterly includes an article asking a simple question: should survey questions about political knowledge include a "don't know" response, or should respondents be forced to guess if they do not know the answer.

A series of articles by Jeffery Mondak and his colleagues suggest that perhaps it is best to create a forced choice. Encouraging "don't knows" "invites a guessing response set" perhaps linked to personality differences, they argue, and thus is biased.

Most measures of political knowledge include a "don't know" or "no answer" alternative. Is there a systemmatic bias? Can this screw up the results in some underlying way? A POQ piece by Patrick Sturgis, Nick Allum, and Patten Smith that showed up in my mailbox today suggests that "don't know" is, at least from what can tell today, a perfectly fine approach to questions of knowledge. Their experiment finds no obvious distortions, although they recommend further work to tease out potential issues.

My own gut feeling is that "don't know" is a good alternative, that we can't really tell what are guesses, even good guesses, versus bad ones that by sheer dumb luck turn out to be correct. I've also wondered whether the 1 = correct, 0 = otherwise is a good coding scheme for knowledge items. Shouldn't an incorrect response be coded as -1, a 0 reserved for no answer or don't know, and then a 1 for correct responses? An incorrect response seems to me to be qualitatively different than no answer at all. And yet like many others, I use the 1 if correct, 0 otherwise approach.

Old habits, even old methodological habits, die hard.

Screwing the Pooch (NES knowledge questions)

To screw the pooch means to royally screw up. That may be a bit of an overstatement, but there are signs that the ANES seriously had intimate relations with the pooch in how it handled political knowledge questions over several years.

Background: The ANES (American National Election Studies) conducts surveys every two years and is the source of a large part of political research done in academic journals

In a 24-page memo on serious issues concerning the validity of their knowledge questions, NES notes that "ANES political knowledge coding from this period may provide misleadingly unflattering portraits of the American public’s possession of factual information on the topics covered by ANES political knowledge questions."

Oops.

This matters. It matters because journal articles have been written based on these data. It matters because whole friggin books have been written based on these data.

In fairness, the ANES folks do a helluva job and I'm sure they are deeply embarassed and baffled and probably a bit pissed over all this. And there is a silver lining, or so they argue. After a couple of scholars discovered the problem, the ANES leaders concluded:

We are grateful to Professors Gibson and Caldeira for bringing questions about the political knowledge data to our attention. We also thank the ANES and SRC staffs for facilitating our investigation of past practices. We apologize to the user community for any negative consequences that past ANES practices have for your inquiries and scholarship. We are committed to do our best to avoid such mistakes in current and future ANES endeavors.

We view these discoveries as opening up exciting opportunities for new and important scholarship on political knowledge. Public opinion watchers thought they knew how knowledgeable Americans were about political facts, but now, the extent of public knowledge is less clear. We hope the new data ANES will release in the future will inspire scholars to conduct innovative explorations of public knowledge, perhaps yielding new insights into the extent and role of information in the realm of political behavior.

The report includes a long, somewhat complicated, but useful set of appendices detailing all the factors that play a part in this methodological fiasco. I'm not sure exactly what all this means, or even how to correct for it, or even to use ANES data for political knowledge research. Gotta sit, read, think -- and probably drink.

Pay for Grades

Below is an abstract of a study I blogged about earlier. Basically, paying people improves how well they do in a test of political knowledge. Clearly they are motivated by the cash.

Okay ... so?

A fad of late has been paying kids to do better in school. This makes sense, especially now that I think of it in terms of the study below. If all the other ways of increasing motivation fail (family pressure, peer pressure, the hopes of a good job, the joy of learning for the sake of learning) then why not pay the little darlings to get good grades? In this era of accountability (i.e., no child left untested), this strikes me as a very American approach. Cold, hard cash. The invisible hand. Money talks.

I'm serious. Why not?

The study abstract follows.

Surveys provide widely cited measures of political knowledge. Do seemingly rbitrary features of survey interviews affect their validity? Our answer comes from experiments embedded in a representative survey of over 1200 Americans. A
control group was asked political knowledge questions in a typical survey context. Treatment groups received the questions in altered contexts. One group received a monetary incentive for answering the questions correctly. Another was given extra time. The treatments increase the number of correct answers by 11–24%. Our findings imply that conventional knowledge measures confound respondents' recall of political facts with variation in their motivation to exert effort during survey interviews. Our work also suggests that existing measures fail to capture relevant political search skills and, hence, provide unreliable assessments of what many citizens know when they make political decisions. As a result, existing knowledge measures likely underestimate people's capacities for informed decision making.