Can We Do Away with Media Exposure Questions

I blogged earlier about the difficulties of getting a proposal approved because I relied on exposure to the news, a concept some find problematic due to perceived (though to me, unconvincing) measurement issues.

This raises at least a couple of questions:

• Can we do away with media exposure questions?
• And, if so, what the hell do we use as an alternative?

Exposure versus attention has received its fair share of attention, but we can't really measure attention without first establishing exposure.  The two are intertwined.  The real problem, to me, is the broad nature of the questions often found in large surveys such as ANES.  These items are historical.  They've been asked more or less the same way for decades, and by continuing to do so, we have some ability to look at changes over time.  For an example, see this table.

But to ask broadly about television news exposure, with no difference drawn between Fox News or PBS, raises all kinds of concerns. 

The issue for me is, do we now combine ideology or partisanship with media exposure ... that is, if we're not going to get specific about where people get their news?  We know from recent research, my own and that of others, that there is a partisan migration going on, especially in cable news, as conservatives shift to Fox and liberals to MSNBC and those somewhere in the middle still watching CNN.  The same is probably true, to a lesser degree, across the news spectrum, particularly on the Internet.

So if we not going to get specific in our news questions, and that's what I tried to do in my modest proposal, then perhaps it's time to combine ideology and news exposure and basically assume likeminded folks seek out likeminded news. 

We are obligated, then, to create interaction terms -- to put it in a regression frame -- for all media and partisanship/ideology measures.  The short answer to this is, ugh, ugly, and a pain when it comes to creating all these dummy terms.  But in the long term, save more specificity in our news media questions, this may be the only logical answer.

Media Coverage of HIV/AIDS

There is a strong relationship between HIV/AIDS prevalence in a country and coverage of the disease in its media, according to this recent study (abstract only, sorry).

It's interesting.  The researchers used Google News archives to find news articles in 168 countries.  What makes this interesting is a bit of methodological geekiness -- it's unclear how well, or complete, or   comprehensive, Google News truly is as a way to find articles for content analysis.  There's a study for someone right there -- probably in information science -- whether web-based aggregators make for methodologically sound samplers of news content.  I suspect yes, given how Google scours the world of the web, but it's one that still needs to be done.  Be interesting to compare Google News with Nexis or others to see what gets missed, if anything.

I Never Listen to Celine

This New York Times story gets to the heart of methodology.

American men have a naughty little secret. Sometimes, they like to relax with a little Céline Dion. Professed classical music fans have one, too: as it turns out, they don’t tune into classical radio nearly as much as they claim.

Turns out, new measures of listening -- away from surveys and more to boxes that measure actual audience behavior -- finds out something we've always known.

People lie.

Okay, not so much lie as fib.  My dad always told me people lie about three things: did you go to church, did you vote, and what gas mileage does your car get.  My corollary on gas mileage is, the higher the cost of gas, the more SUV owners lie about their mileage.  When gas was $4 a gallon in the U.S., people with fat rolling boxes lied through their teeth about their mileage.

Back to radio listening, or to be honest, any media consumption habits.

As someone who got tenure thanks to research on talk radio, I found the next part interesting.

Talk radio, a largely conservative format, turns out to have fewer fans than previously thought. Talk radio’s market share declined 2.6 percent in the study of areas where the meters were used.

There's a lot more here to examine, but the basic methodological point is simple: self-reported behaviors in surveys are often the best measure we have, but they're blunt instruments at best, with lots of random (and non-random) error.  We often answer in ways that make us feel good about ourselves, which has classical music fans a bit worried since the new numbers show fewer listeners to this genre of music than we thought.

There's even a race card.  You knew there had to be.

The makeup and size of Arbitron’s sample is an issue for some Hispanic and urban broadcasters, who say metered readings undercount minority audiences and hurt their stations disproportionately. Mr. Adams of Arbitron said the company was responding to concerns by adding more panelists who had cellphones rather than landlines, and investing in in-person coaching to make sure all panel members use the devices correctly.

In other words, the new (better) system finds fewer viewers and listeners of "ethnic" programming, which includes everything from AM radio to BET programs.  It's fascinating stuff, proof that methodology can have real-world implications.

Oh, and I never listen to Celine. Ever. Measure it any way ya want, buckos, and it ain't gonna happen.

Paying for Knowledge

It's nice to see someone else working my territory, examining research on political knowledge.  Here's a blog entry that looks at a study I also examined some time back.  The actual article is here, which looks at how paying someone motivates them to do better on tests of political knowledge.  The conclusion of the article is simple: "As a result, existing knowledge measures likely underestimate people’s capacities for informed decision making."

Stewart vs. Colbert

There is an emerging bit of scholarly work that suggests those two popular comedy faux news programs -- Jon Stewart's The Daily Show and Stephen Colbert's The Colbert Report -- have very different effects on their audiences.

Stewart's show is more straightforward.  He slams Republicans, occasionally Democrats, but basically you know where's coming from. Colbert is more sneaky.  He plays a Republican blowhard to make fun of Republicans and blowhards.  But his approach confuses some.  In other words, he nudges people to like Republicans and those he satirizes, according to one study.

Why mention this?  In part the issue is methodological.  You wouldn't want to measure exposure to the two programs and then collapse this into a single index called Exposure to Late Night Comedy.  There are enough differences, at least suggested in some published studies or a few under consideration at journals, that combining the two might, if not cancel one another out, create measurement error issues.  And I think I broke a personal record for the number of commas in a single sentence.  Apologies.

Lots of people mention such programs as a source of their news and information, as places that influence what people know.  If true, we must be careful in how we take these questions and combine them into a single variable.  In fact, best I can tell from some early data, this would be a mistake.

What dependent variables would this most likely influence?  Probably not knowledge so much as attitudinal items, such as like/dislike of candidates or even attitudes about major institutions such as Congress or the Press.  I don't see it influencing trust in government but it might influence internal or external efficacy.  In general, we need to carefully examine differences between these two programs on key criterion variables before simply combining them in a single index.

How Much Attention is Attention?

There are good reasons to pay attention to news and public affairs (civic responsibility, your job requires knowing what's happening, conversational material, etc.). There are good reasons to not pay attention (busy schedule, hate the partisanship, worried about missing the next Dancing with American Idol episode, etc.).

How much attention is enough?

If you read the scholarship on political knowledge, you might come away thinking there is a magic amount that everyone should know, a certain degree of attention everyone should have. Carefully read the literature, though, and you find the nuances: people learn what they need to know based on their jobs, their social situations, their time and resources.

Scholars tend to ask the same kinds of questions to tap political knowledge. We do this in part because they seem to work from study to study and it allows us to compare -- more or less -- our finding with other studies. It all gets into the reliability and validity of our measures that, if established by other research, frees us to get at the more substantive analyses. In other words -- get past the method stuff and get to the fun results. Sometimes a study will come along that challenges the status quo of how to measure knowledge. These are acknowledged quickly by most scholars and then they move on to, yes, studying it the same way other people have done over the years.

It may be time to reconsider our measures.

So back to my question above: How much attention is attention? We know that media exposure is a lousy measure, that how much someone attends to a medium is a more fair, more valid, and at times more reliable measure. Perhaps we need individual-specific attention measures. Is a "high" attention for one person the same as a "high" for another? And do they have the same consequences? We have to tie together exposure, attention, and some other factor -- not sure what -- to get at the changes in the media landscape. One answer may be elite versus middle-brow versus low-brow sources of news.

Is high attention to Bill O'Reilly the same as high attention to NewsHour? Probably not, but they'd go down at the same in a research situation (depending on how you scale it).

But trying to scale the kinds of content people consume, that's rife with problems. Should I score Fox News lower than CNN? No, I don't think so. People versus New Yorker magazines? Yep, that's easy. It's TV where we get into trouble, and we're probably going to have to drill down to the individual program level to get at some of this, and that's a whole different problem when it comes to survey (versus experimental) research.