Can We Measure Journalism's Impact?

Here's a fascinating column at the Nieman Journalism Lab that asks a simple yet compelling question:

If democracy would be poorer without journalism, then journalism must have some effect. Can we measure those effects in some way? 

Good question.  Plus I'm happy to write about something that doesn't involve my University's student newspaper.

Why worry about such a measure?  For me, it's important that journalism define its role in democracy.  Yes, theoretically we know the role of the press is essential to an informed electorate, a way for interests to converse an ideas to be vetted, a way for powerful institutions to be checked when they go astray.  But how do you measure that impact?  The column, by Jonathan Stray, explores some of the possibilities.  Read the column yourself.  No need for me to repeat his words.

However, at the end of the column he does bring up a topic near and dear to my heart, whether what people know, their knowledge, is a good measure of impact.

In one of the most limited, narrow senses of what journalism is supposed to do — inform voters about key election issues — American journalism failed in 2010. Or perhaps it actually did better than in 2008 — without comparable metrics, we’ll never know.

As I've written about extensively here for more years than I really want to remember, the measurement of political or current events knowledge is a tricky thing.  Without getting too PhDweebish about it, such measures are rife with conceptual and methodological difficulties, and interpreting those results can be even trickier.  Often, we're not measuring what we think we're measuring.

Okay, that aside, is knowledge a good measure?  No.  It's a terrible measure of journalism's impact for a host of reasons -- unless it's done well, with considerable caution.  It can be done, but with care.  No, I'm not positioning myself for a really lucrative consulting job (though hey, I'm available NYTimes!), I'm merely saying that tossing some current events questions at a random sample of Americans, which can be great fun and makes for terrific party talk, is a lousy way to measure journalism's impact.  It may measure to some degree the public's knowledge (and I even have my doubts there), but it's unconnected to the practice of journalism, at least in how such questions are usually framed.  Is there a way to do it well?  Absolutely.  But it'll require multiple methods, not merely surveys, to get at the real answer.

Quick examples:  Men guess on questions more than women, thus inflating their knowledge scores.  Also, how you ask the question (multiple choice, free response) gets different results with different kinds of people.  Rely on TV for news?  You'll struggle with free response.

In an era of Big Data, measuring journalism's impact seems more possible than ever.  Surveys, yes, but also analysis of Twitter and Facebook posts, experimental studies, and other ways are needed to really get at answering whether journalism has an impact.  I'd operate under a working hypothesis that it does, but to truly test that hypothesis, it's gotta be done right.  And we have to be open to being wrong, that the data tell us something else entirely.

The answer?  It's never been more important, at least to journalism folks.  I hope it gets done well.

Eyewitnesses: The courts, and journalists

The courts are turning a skeptical eye toward the "troubling lack of reliability" in eyewitness testimony.  Should journalists do the same?

The New Jersey Supreme Court, for example, is making it easier for defendants to challenge eyewitness accounts.  Here's a telling quote from a judge:

"From social science research to the review of actual police lineups, from laboratory experiments to DNA exonerations, the record proves that the possibility of mistaken identification is real. Indeed, it is now widely known that eyewitness misidentification is the leading cause of wrongful convictions across the country."

The U.S. Supreme Court has its own doubts.  "Every year, more than 75,000 eyewitnesses identify suspects in criminal investigations," according to this NYTimes lede. "Those identifications are wrong about a third of the time, a pile of studies suggest."

If you're a reporter, your first response should be:  "Oh, crap."

Journalists rush to crime scenes, to tragedies, to bars at happy hour.  Eyewitnesses are sought out, quotes dutifully recorded, stories carefully crafted.  Nothing beats being there, I always tell my journalism students, but if you can't be there nothing beats talking to someone who actually was there when an event occurred.

If the courts have some doubts about the quality of eyewitness accounts, shouldn't journalists share these doubts?  Absolutely.

The easy solution is for a reporter to judge whether a description sounds right, smells right, and is corroborated by others on the scene.  Even then, groupthink can kick in.

Cop reporters know this one.  Rush to a crime scene in a less-than-savory neighborhood where maybe a cop shot someone, say a kid.  All the witnesses share a similar story, and sometimes they're right, but sometimes they tell the story they want to believe or the story they think they saw -- versus what their eyes actually soaked up.

We're all flawed cognitive processors.  We often see what we want to see in a situation, a finding confirmed by hundreds, if not thousands, of published psychology studies.  Not only do we see what we want to see, we remember it in ways that fits our predispositions (racial, ethnic, religious, partisan, and all the rest).

If roughly a third of eyewitness accounts in court cases are wrong, is it possible roughly a third of our reported eyewitness accounts in news stories are also wrong?  Perhaps.  And that's enough to give most journalists pause.  Take the extra time to make sure other eyewitnesses share the same story.  Even then you may be wrong.  But at least you made a good-faith effort to provide the best obtainable version of the truth.

Trick Questions

I caught a bit of video this morning of Newt Gingrich holding up a Newt 2012 t-shirt.  He was set up, and nicely.    The ABC reporter got him to hold up the shirt, which he gladly did for the camera, probably figuring it for a harmless puff pic.  Gingrich is all about American jobs.  Turns out, though, the shirt was made not in the U.S. -- but in El Salvador.



Newt, you've been punk'd.

"I'll have to ask the folks who ordered this," Gingrich responded. "I don't order it and I don't do it." Campaign spokesperson Michelle Selesky said "That was a rush order made by some of the volunteers." Selesky noted the print work on the shirts was done in Atlanta. 

Okay, so Atlanta is kinda like America.  So that's something.

This is nothing new.  The traditional journalistic punking involves milk.

You may not know the routine, but reporters used to follow presidential candidates around as they try to portray themselves as caring about the common man.  And then you ask him or her: "So, what's a gallon of milk cost?"

They struggle for an answer.  They've been punk'd.

There more here than just catching a high rolling politico looking like a fool.  They can do this by themselves quite well (remember Michael Dukakis doing the bobble-head doll routine in a tank, or John Kerry in hunting gear?).  And then there was President George H. W. Bush being "amazed" by a grocery store scanner, something Americans see every day.  Turns out that story is false, but it fit the narrative of a well-heeled Bush who didn't really understand Americans in a time of economic downturn.

My point?

In what people know, perception is powerful.  We tend to see what we want to see and believe what we want to believe, but a powerful narrative that paints a candidate in a certain way, such as hypocritical -- as in the Gingrich t-shirt -- or as historically challenged -- as in Michelle Bachmann or Sarah Palin, has lasting impact on the public mind.  And they are hard to reverse.  Chevy Chase's biting imitation of Gerald Ford on Saturday Night Live may not have lost Ford the 1976 presidential race, but it certainly didn't help.

Research shows we tend to vote for presidential candidates based on two major themes: competence, and character.  Other factors in how a race is framed, such as war or the economy, also play a huge role.  But being perceived as a hypocrite or fool, those also can influence how people think about a candidate, and making every subsequent slip-up, even the most minor, seem huge by comparison.

Political Efficacy and Journalism

I have of late read a lot about how readers and viewers of the news will either seek out confirming information or, through motivated reasoning, see what they want to see and hear what they want to hear in a news story to fit their political predispositions.  The evidence is compelling, especially when you consider how people cling to misperceptions (Obama is a Muslim, etc.), but even corrections by the media of an incorrect fact leads some -- due to partisanship -- to move even further in their misperception.  People believe, it seems, what they prefer to believe.
 
And now along comes this study in the February 2011 Journal of Communication that sorta kinds follows along the same lines.

The study looks at neutral reporting of contradictory factual claims, the basic stuff of everyday journalism, and finds experimentally that exposure to such content influences what they call epistemic political efficacy, defined here as "confidence in one's own ability to determine the truth about factual political disputes."  Very interesting concept, one I wish I'd invented, particularly in these times of "death panels" and other invented political factoids.  They also consider adjudication, the notion of coming in a news story to some kind of conclusion about the truth or falsity of a disputed fact.

As the article states:

...the two experiments conducted thus far are consistent in finding that at least for readers who are high in prior interest in an issue, encountering even a single news story about that issue that contains adjudication significantly increases their epistemic political efficacy. This suggests that journalistic adjudication may be a point of leverage for increasing the quantity and quality of citizen engagement: goals that may seem unattainable when one assumes that their only important antecedents are motivational constructs such as political interest.

Some interesting ideas here that deserve more time than I can afford to give them, at least at the moment.  If I'm lucky I may revisit this study on another day and delve into the whole concept of EPE and how it relates to more traditional approaches to political efficacy.  But I do think they're on to something in these hyper-partisan times.

The News, Cancer, and a Counter-Intuitive Result

It's a given -- we learn a lot about disease and illness from the media, either in the news, or from those annoying pharmaceutical commercials, or in entertainment programs.  Me, I get all my medical knowledge (and behavioral modeling) from House.

Cancer in particular gets lots of media attention, as it should given the number of people who suffer and die from the disease every year.  But is it accurate media attention?

This study suggests not, at least when it comes to the news.  Only the abstract is available (I bet Gregory House gets access for free).  But what's really interesting is not that the news media isn't as accurate as it could be -- surprise -- but the direction of that inaccuracy.  Stay with me here.  It's kinda sorta fascinating.

The content analysis of cancer coverage by eight newspapers and five magazines reaches a counter-intuitive conclusion.  News reports often discuss aggressive treatment and survival, according to the study, but rarely touch on the bad news.  All that emphasis is mine because I love doing it.  Think about it.  How's this result for turning the traditional criticism of journalism on its ear?  We (the royal we, as in journalists) focus too much on the bad, or so goes the criticism by just about everyone on the planet who isn't a journalist, but according to this study, the news "may give patients an inappropriately optimistic view of cancer treatment, outcomes, and prognosis."

Wow.  Journalists optimistic?  Something's deeply wrong here.

Unfortunately we can't judge the quality of the research, given all we have a measly abstract.  And when non-media scholars attempt a content analysis, they often screw the pooch -- as in no inter-coder reliabilities, no cleanly drawn categories, no sense of the difficulties of analyzing a text or constructing units of measurement.  Sure, it's The Archives of Internal Medicine, a pretty damn good medical journal, but too often medical folks conduct really bad behavioral research.  For them, a media study can seem like slumming it.  That attitude often shows in the work.

But if we assume for the moment the study was done halfway well, then this result does surprise. Clearly someone in journalism is asleep at the switch.  Get back to that bad news!

Newspapers vs TelevisionJournalism vs Public Relations

Okay, this chart below says it all.  I trended newspapers and television in books scoured by Google's new ngram viewer.  It's not a great reproduction, this image, but as you can see the red (television) intersects with newspapers (blue) in the late 1960s, at least in mentions in books scanned by Google. 

Okay, that was fun.  What else can we do?  My colleagues down the hall will love this one.  Below I look at journalism (blue) versus public relations (red).  I find it fascinating that PR cranked it up in the 1950s and 1960s, in book mentions, and then slipped back again.  Why?  Some smart PR professor will no doubt let us know, or maybe do some research to explain this odd burst of attention that later slips back.

Not all "Journalisms" Are in Crisis

The title above I shamelessly borrow from a research paper (abstract here, full pdf here) that argues while traditional news approaches are in crisis, "there are many other emergent forms of journalism that simultaneously succeeded during this period, and which continue to flourish."

Here's a key point:

Whereas journalism used to be largely homogenous, thinking about journalism in the plural presents a much greater challenge to those who wish to maintain control over what the public sees and understands about politics.

This is a qualitative analysis of news. It argues that thinking of "journalisms" will help us better understand the role news plays and how people make use of news.  Or, as the author says, "I propose we now think about journalism not in the singular, but a range of journalisms which operate in different ways, fulfil different requirements, and appeal to different niche audience groups."

It's a point well made.

Kicking Journalism's Ass

I love Ira Glass and This American Life.  In a talk this week, he offered: "Opinion in all its forms is kicking the ass of journalism."

Ouch.

He especially blasted TV news, and he especially praised Jon Stewart (and, in an offhanded way, Glenn Beck, who he called "fascinating" without further elaboration).  A story about the talk is here.

He's on to something, of course.  Opinion is kicking journalism's ass, at least when it comes to ratings and attention.  That is, if you don't consider "opinion" as "journalism," which gets a bit confusing.  Is an editorial opinion?  Yes.  Is it journalism?  Kinda yes.  But what he's really doing is comparing the delivery of standard television news with the delivery of the host of TV talkmeisters, from Maddow to O'Reilly to Colbert to, well, all the people doing pretty damn well in the ratings.

Oh, and he praises NPR too.

What People Know ... about Torture

Is waterboarding torture? 

Yes, at least yes for a jillion years, as this writer points out.  It's part of a report on how journalism, which had called waterboarding torture from the dawn of time until, oh, about 2004, suddenly stopped describing it as such.  He calls journalists cowardly for suddenly not calling it torture, a technique used by evil dictators throughout history.  The full report, in annoying pdf format, is here. 

Do people think it's torture?  Back in 2009, 60 percent of Americans in a poll described it as torture.  A little over a third said it was not torture.  Three percent, having never undergone the treatment, were unsure.  In another poll the same year, 71 percent described it as torture.

Yeah, it's torture.  Of course it is -- except, perhaps for some, when we're doing it.  Then it's not so much torture as an enhanced interrogation technique, which is a nice way of saying ... torture.

Hell Freezes Over

Hijacking the blog. 

My department (Journalism) voted Tuesday to get rid of its three majors (newspapers, magazines, and publication management) and consolidate them into a single major called ... journalism.  This simple, seemingly logical move took years and cost thousands of lives.

But don't worry, academe fans.  While we have one major, we now have sequences coincidentally called -- wait for it -- magazines, publication management, public affairs (newspapers) and visual journalism (which we always had, the sequence formerly known as photojournalism).

In academe, even when hell freezes over, curiously nothing changes.

I have absolutely no way to connect this to the traditional theme of the blog.  Then again, I don't have to. It's my blog.

What Bloggers Know ... er, Kinda

Bloggers and journalists?  The same?  Different?  In terms of what they focus on, you bet there are differences.  Below are some results from a Pew Research report on what each group "covered" in the first week of February.  First, the bloggers, next the journalists.

Bloggers (left, right, moderate, whatever) seemed fascinated by the Tim Tebow Super Bowl advertisement, something that got exactly the amount of journalism coverage it deserved (a little, but c'mon, no more than that).  As far as bloggers go, the Haiti earthquake apparently never happened, and Toyota cars have never been recalled.

And bloggers are going to replace journalism?

Flacks and Flak and Journalism

I love The Atlantic.  I get an actual hard copy in the mail every month -- ya know, the kind with ink on paper.  Brief piece out on what'll happen to journalists fleeing a dying biz, often into PR, and whether this a good thing.

Science Writing

Newspaper science writing is "going out of existence," according to one report, and that's bad news because soon what people know about science will come from such brainiac experts as Sean Hannity and Glenn Beck.  If you haven't felt fear before, you should now.

Several mainstream news organizations in recent years have let go of their science reporters and done away with their science sections altogether. The science section of The New York Times, which is one of the few left in the country, features more health-related stories and fewer hard-science stories than it used to, said [Natalie] Angier, a Pulitzer Prize winner.

In a nation that struggles with basic science knowledge, that means what people know about science will only get worse. For those who drank the Kool-Aid and think the new world of blogs and various online sources will make things better -- pfffft.  It won't.  Not for most everyday people.  Instead we'll have pseudo-science ruling the day, and we'll all be worse off for it.

Yeah, the public likes science, but affect (emotion) and knowledge are very different bowls of red syrupy sugary drink.  After all, only 47 percent of Americans know electrons are smaller than atoms.  Yup, I'm reeeaaal optimistic, as science writing disappears.

Changing Journalism -- and Knowledge

Leonard Witt, who does cool stuff down the road from me on sustainable journalism, is conducting a series of interviews about the future of journalism.  So there's this interview with Jay Rosen, esteemed NYU j-prof and Master of the Universe when it comes to changing media.  The text is here or you can listen to the interview here.

Below, the part that touches on this blog's theme:

Witt:  So do you think the people will be better informed than they are now? The public and the public’s fear,  or less informed or you just don’t have a clue?
Rosen: I think it’s going to be better.
Witt: Why is that?
Rosen: Because we don’t have to depend on  a single elite for our information.
Witt: So do you think that’s already happening? Or do you think….
Rosen: Yeah it is already happening.

My gut reaction?  This is absolute crap, the idea that people will be better informed because they no longer "depend on a single elite" for information.  But it's possible my gut reaction comes from bad sushi or my previous life as a journalist or my years of studying and doing research on how people learn from the media.

In fairness, the bulk of this interview is not about how and what people learn from the news.  Neither guy is an expert on the topic.  It's more about where journalism is going and what it all means, and neither dislike the old mainstream media, they're just pointing out the obvious changes driven in a large part by technology.  So let's give 'em both some leeway here.

And then again, let's not, otherwise I'd have nothing to write about.

First, there is some evidence that political knowledge has been decreasing, not increasing, as all these new media non-elite sources come to fruition. Cause-and-effect?  Too early to say.  A lot of political knowledge comes from incidental learning, the inadvertent exposure to information, mostly from TV for people with little interest in public affairs.  In this brave new world, with so many choices, these are folks most likely to go elsewhere for entertainment rather than consume -- even if half-heartedly and accidentally -- the news.

So our brave new media world in which a million information sources bloom, for a large chunk of society that might as well be happening on the Moon.

Now, if we're talking about the chattering class, the political junky, that sliver of Americana who find politics not only interesting but as necessary as oxygen, yeah I think you've got something here.  But this is a mistake these folks often make, of generalizing from their own interests or those they hang out with over lattes.  A metro-centric viewpoint.  My own reading of the vast literature of how people learn from the media and what they do with that knowledge suggests the more you fragment the media world, the less learning that will occur, and an awful lot of that learning will be dysfunctional and misinformed.

People will feel they are informed, even when they are not -- the empty-calorie hypothesis.  

So I'm not sure exactly where Rosen comes up with the idea that people are, or will be, better informed in this new media landscape.  You could fashion an argument that younger people might be more drawn into this fragmented news environment.  Given they tend to be the least informed, we might see some improvement there, and any improvement among young citizens will help.  But I'll be honest, other than a blip in the 2008 election, the data don't really support this conclusion.  Yet.

You might also argue that people will be become better informed about what they're truly interested in, which is a riff on the V.O. Key/Doris Graber/etc. argument about the public's knowledge being greater than what we give it credit for, if we only asked the right questions.  But such balkanized knowledge and issue publics (which may or may not exist) do not serve a democracy well, so maybe new media will increase some knowledge for some people about some things, but the outcome isn't pretty, at least according to most political thinkers.

Of course, it may mean no change at all.  The public's knowledge about public affairs remained largely unchanged for decades.  And while it seems to be inching down a bit of late, that may merely be a statistical blip and nothing to do with the changing news landscape.  But it most certainly does not mean that people will be better informed, not from what we're seeing so far.

Is Journalism Dying?

Thursday's Tidbits

A few tidbits, the odds and ends off the net, of stuff vaguely related to how people use the media or learn from it.

• Google wave could mean significant changes in journalism -- for the better.
  
• What divides us?  The Pew Center says it's immigration and income, not race, in this report.
• People want TV everywhere?  Yes, according to this press release.
• But people want privacy, not tailored ads, says this site.

Wednesday Stuff

For the price of a single visit, multiple bits of information below that -- more or less -- can be related to what people know.

• Accountability journalism is at risk, says speaker at Harvard's Shorenstein Center.  One consequence is that many smaller towns risk sinking "into casual endemic civic corruption" as public officials are no longer held accountable by journalists.
• Thursday is National Punctuation Day, so says the Chicago Trib in this piece.  I. Am. Excited.   And I should probably include some "unnecessary quotations."
• Newspapers don't want a bailout.  In related news, journalism schools still popular.  Go figure.
  
• Brilliant piece in The Atlantic about how some stories show up on TV at the same time, fed by political spindoctors and hucksters and hacks.  I read it in the paper version of the mag, which I get at home, making me morally superior to all who read it online. Or at least help paying the bills for good journalism.

Rats vs Health Care

Sigh.  The top blog story of last week?  Health care, you might think.  "You Lie!" screamed by some obnoxious congressman in need of a good spanking?

Nope.  A giant rat.

The graphic to the right says it all, from the Pew Center's count of the blogosphere (above) and the evil mainstream press (below).  Go here for the report.

Again, sigh.  Blogs are gonna replace news?  Can culture death be far behind?

Even better, go further and find the Twitter Makes You Dumber controversy, which I (like a jillion others) blogged about last week. 

I now feel a member of the blogosphere, no doubt losing a precious few IQ points along the way.  I really can't spare 'em.

Reasons for Paying Attention

I wrote yesterday about what the reasons might be for people to pay attention to public affairs.

Attention to the news is necessary, according to the traditional argument, in order for people to know what's going on. An informed public, so goes the theory, is vital in a democracy. Let's assume all that (though it makes an interesting argument for a later date as to whether it matters if a public is informed or not).

What are the reasons for paying attention?

There are a lot of reasons to not pay attention.

• There are a zillion TV channels now. You don't have to watch TV news unless you absolutely want to. Many have chosen to do just that.
  
• The news itself -- especially on cable TV -- is often petty, partisan, and particularly focused on the inane.
• I get my news off Facebook ... whatever friends happen to link to.
  
• The source of real news -- newspapers -- is often presented in a dull, dry manner.
• It's so easy to just be entertained. Or if you want to feel informed while being entertained, there's always The Daily Show or Colbert Report.

I'm sure there are other reasons to not pay attention. But what are the reasons to pay attention?

• It's your responsibility in a democracy
• Take your medicine; it's good for you
• Being informed is fun!

Yeah, the reasons to pay attention, they kinda suck. Responsibility? Pfffft. I'm sure there are other reasons people should pay attention, and they do when a major story breaks, when things get serious (and when that happens, funny how CNN's audience shoots up compared to Fox News. Just saying).

So Jon Stewart has taught us an important lesson. For many, the news has to be entertaining. And Bill O'Reilly has taught us that, for many, the news has to be highly partisan. And Rush Limbaugh has taught us that, for many, the news can even be on radio. And the Internet has taught us that, for many, the net is for porn.

I'm being a bit of a pessimist here. The NYTimes and Wall Street Journal prove that, for many, quality serious news will sell. At the hyper-local level, people will pay attention if you cover your community in a way that rings true to them, stories that seem real and not artificially distanced from the communities they strive to describe. People will also pay attention if you somehow manage to work sex into the story, but that doesn't always work, so it's a challenge for many types of news stories.

Ultimately, getting people to pay attention to what's happening the world, that's not so hard. Getting them to pay attention to News, that's a bit harder. Getting them to pay for News, that's even more difficult. And thus we come to the core of the problem -- that journalism is expensive, at least the way we've been doing it for decades, and the revenue models are unraveling in some ways.

We've got to come up with ways to convince people to not only pay attention, but to pay for paying attention.

Knowing Your Town

I had my public affairs reporting students read this article about how disconnected many journalists are from the places they live and cover for their news organizations -- and how the news they produce can suffer as a consequence. Then I rattled off in class some places in Athens and asked if they'd visited or knew where they were, like Memorial Park, which sits less than a mile off the edge of campus. Only about two of 16 knew where the park was.

In a society where people move around a lot, that sense of place can suffer, especially if you don't join community organizations and link yourself up somehow with where you live (ala Bowling Alone and the whole social capital argument).

My point? What people know about their homes, their neighborhoods and their communities, must somehow ring true in the kinds of stories we craft as journalists. That's damn hard to do as a reporter if all you know is where you live, the concrete-and-brick buildings of your beat, your news organization's office, and the bar with the cheapest drinks at happy hour. I worked for three daily newspapers in three different states, and I admit to not being a joiner, not knowing much about the places I covered beyond my beat. Yeah, I could tell you which local politicians hated each other, or where most crimes happened, but I didn't know the places. I suspect my stories suffered as a consequence. And what people knew was, my stories -- good as they were technically -- didn't always ring true.