Can We Measure Journalism's Impact?

Here's a fascinating column at the Nieman Journalism Lab that asks a simple yet compelling question:

If democracy would be poorer without journalism, then journalism must have some effect. Can we measure those effects in some way? 

Good question.  Plus I'm happy to write about something that doesn't involve my University's student newspaper.

Why worry about such a measure?  For me, it's important that journalism define its role in democracy.  Yes, theoretically we know the role of the press is essential to an informed electorate, a way for interests to converse an ideas to be vetted, a way for powerful institutions to be checked when they go astray.  But how do you measure that impact?  The column, by Jonathan Stray, explores some of the possibilities.  Read the column yourself.  No need for me to repeat his words.

However, at the end of the column he does bring up a topic near and dear to my heart, whether what people know, their knowledge, is a good measure of impact.

In one of the most limited, narrow senses of what journalism is supposed to do — inform voters about key election issues — American journalism failed in 2010. Or perhaps it actually did better than in 2008 — without comparable metrics, we’ll never know.

As I've written about extensively here for more years than I really want to remember, the measurement of political or current events knowledge is a tricky thing.  Without getting too PhDweebish about it, such measures are rife with conceptual and methodological difficulties, and interpreting those results can be even trickier.  Often, we're not measuring what we think we're measuring.

Okay, that aside, is knowledge a good measure?  No.  It's a terrible measure of journalism's impact for a host of reasons -- unless it's done well, with considerable caution.  It can be done, but with care.  No, I'm not positioning myself for a really lucrative consulting job (though hey, I'm available NYTimes!), I'm merely saying that tossing some current events questions at a random sample of Americans, which can be great fun and makes for terrific party talk, is a lousy way to measure journalism's impact.  It may measure to some degree the public's knowledge (and I even have my doubts there), but it's unconnected to the practice of journalism, at least in how such questions are usually framed.  Is there a way to do it well?  Absolutely.  But it'll require multiple methods, not merely surveys, to get at the real answer.

Quick examples:  Men guess on questions more than women, thus inflating their knowledge scores.  Also, how you ask the question (multiple choice, free response) gets different results with different kinds of people.  Rely on TV for news?  You'll struggle with free response.

In an era of Big Data, measuring journalism's impact seems more possible than ever.  Surveys, yes, but also analysis of Twitter and Facebook posts, experimental studies, and other ways are needed to really get at answering whether journalism has an impact.  I'd operate under a working hypothesis that it does, but to truly test that hypothesis, it's gotta be done right.  And we have to be open to being wrong, that the data tell us something else entirely.

The answer?  It's never been more important, at least to journalism folks.  I hope it gets done well.