Engaging News and Knowledge

There's a story out today about the Engaging News Project and its study of "traditional" versus "contemporary" news presentations. Here's the lede:

There is a significant increase in page views when people browse a news website with a contemporary design compared to a website with a classic design, according to a new report from the Engaging News Project. People also learn more from the articles when they view a contemporary site.

Which is quite interesting if you're into news online. The part I was curious about, at least for this blog, was the "people also learn more" from contemporary sites. We're talking knowledge. We're talking what people know. So I went to the full report to understand how they measured this and whether it stacks up as rigorous research.

First off, how do the designs differ? I'm not a design guy, but here's a screen grab below.

To repeat, I'm setting aside the stuff on which design people liked more, spent more time with, etc., and I'm sticking to differences in learning. Read the report yourself if you're into design. Below is a key graph on knowledge:

The Engaging News Project team also found that study participants’ retention of details from the articles, though low across the board, nonetheless increased by at least 50 percent when participants viewed the contemporary homepage compared to the classic one.

Wow, a 50 percent increase. That's a lot, right? Check out the graphic from the report below.

So the differences are statistically significant in two of the three studies, though you can also argue they're not substantively different. 0.4 versus 0.6, that ain't much of a difference. Why is recall so slight? It's free recall. Subjects were asked to write in a box (all online, mind you) what they could remember from the stories. Measuring recall versus, say, recognition (think multiple choice question) is a very different type of knowledge -- something I've published research on, thank you very much. Recall favors certain kinds of people. I would have included some closed-ended questions. In my study of a real actual random national sample, I found Internet news favored recall knowledge.

Interesting sidenote: they also measured generic political knowledge but found it had no relationship in two of the three studies in which it was used. That's odd, because usually prior knowledge is a significant predictor of gaining new knowledge. Curious.

Endnote #38 has some details on how they constructed their measure that explains the low "average details" recalled.

Okay, this is all well and good, but let's assume for the moment that a "contemporary" design leads to better recall than a "classic" design. Why? In part it's how they measured recall -- open recall -- and in part it's the kinds of folks who participated in the study (not a random sample and, it's unclear to me in my reading, whether people were randomly assigned to conditions. I assume so, just didn't see it). All that aside, why would one design work better than the other? I have no friggin idea. Then again, we can give subjects the same exact story on paper and on a screen and they'll remember more from the paper version. Same story. Just different medium. So another factor is how you approach a medium, though I can't see how that plays a role here in design. I think the classic design may turn people off compared to the contemporary design, and that influences (mildly) recall. All that text on the splash page is too much, perhaps. I don't really know without giving it more thought.