Are Religious People Patriotic?

Fun with 2012 data. Here's the question:

How well does the word 'patriotic' describe most non-religious people [Extremely well, very well, moderately well, slightly well, or not at all /  not at all, slightly well, moderately well, very well, or extremely well]?

The answer? No surprise, really. Those non-religious people, they ain't patriotic, at least according to a national sample of U.S. adults. In fact, 2-out-of-5 see non-religious people as slightly or not at all patriotic.  Below, the numbers (data weighted to be representative of the U.S. population).

• Extremely Well -- 3.8 percent
• Very Well -- 13.8 percent
• Moderately Well -- 35.7 percent
• Slightly -- 23.6 percent
• Not at All -- 13.5 percent

Okay, but how about the patriotism of other groups? Glad you asked. We've got Protestants (7.9 percent "extremely well"), we've got Mormons (6.1 percent "extremely well"), we've got Catholics (8.5 percent "extremely well") and best of all, we've got Muslims (7.9 percent "extremely well," the same as Prods). 

That last one is interesting and heartening except, as you'll see below, not as much as you'd like.

The real differences seem to emerge in the "not at all" classification, the bottom of the patriotic pile. The folks seen not at all patriotic.  For Muslims, it's 33.8 percent. As a comparison, for Catholics, it's only 10.5 percent and for non-religious, it's 13.5 percent.  In other words, one-third of U.S. adults see Muslims as not at all patriotic, a number far and away above the other religious categories or even those damned-to-Hell non-religious folks.

By the way, on a separate question of all the above groups asking how violent they are, 7.9 percent see violent as describing Muslims "very well." About 1 percent see violent as describing Protestants "very well."

"Ordinary" Matters, At Least in Poll Wording

Two polls this week on the NSA mess offer a nice case study of the power of a single word in survey research.

I offer to you, dear reader, that simplest of words -- ordinary.

An ordinary word, ordinary.  It means "commonly encountered," according to this dictionary.  Or, more to the point in describing me: "Of no exceptional ability, degree, or quality; average."  Yup, that about sums me up.  Ask anyone.

But here ordinary carries extraordinary meaning.  A recent CBS news poll asked:

In order to reduce the threat of terrorism, do you approve or disapprove of federal government agencies collecting phone records of ordinary Americans?

Thirty-eight percent approved of collecting phone records, while 58 percent disapproved. I underlined ordinary because, in this instance, it's anything but.  A Pew poll conducted at about the same time asked a similar question:

As you may know, it has been reported that the National Security Agency has been getting secret court orders to track telephone call records of millions of Americans in an effort to investigate terrorism. Would you consider this access to telephone call records an acceptable or unacceptable way for the federal government to investigate terrorism?

See that millions?  Sure you do, because I cleverly emphasized it using all the typographical tools at my disposal.  The results are different.  Fifty-six percent find this acceptable in the Pew poll, significantly higher than the ordinary CBS wording.  That's a helluva difference.  Yes, the questions differ in other ways, but I'm certain the use of ordinary versus millions makes all the difference in the world.

So, which is the more accurate question?  Glad you asked.  In this case, I find ordinary to be an extraordinarily loaded word, to make it sound like it's something closer to home.  I like the millions language better.  More abstract, less threatening.  Less loaded.

 

The More You Know, the More You Don't Know?

The more Republicans know about the politics, the more likely they believe in political conspiracy theories.This conclusion comes from an article in Mother Jones and relies on a survey conducted last month.  More on that survey in a moment.

Here's the crux of the article:

Among Democrats and independents, having a higher level of political knowledge was correlated with decreased belief in conspiracies. But precisely the opposite was the case for Republicans, where knowledge actually made the problem worse. For each political knowledge question that they answered correctly, Republicans' belief in at least one conspiracy theory tended to increase by 2 percentage points.

The article's author, Chris Mooney, knows his stuff.  He suggests the reason may in part have to do with a conservative's greater need for "cognitive closure" and a need for certainty. A reasonable hypothesis, but I'd counter it has more to do with the nature of the conspiracies examined in the survey.  The two "conservative" conspiracies are about Obama specifically, the two "liberal" conspiracies have to do with Bush stealing the 2000 election and "The Truther" movement about 9/11.

Equivalent?  I'm not convinced.  But let's turn to the survey itself, a national poll of 814 registered voters conducted in December 2012.  First, despite the claim, it's really a survey of people who say they're registered voters, but that's a methodological quibble.  Political scientist Dan Cassino, who runs the Fairleigh Dickinson University poll, says "birther" conspiracy theory is likely to believed more than others because "it's been discussed so often." 

Yup.  Two myths are anchored on Obama, are more recent, and received lots of mainstream media play -- mostly attempts to debunk them.  I'd add here that there's a growing body of research that suggests attempts to debunk myths actually adds to their believability for some people, so I'd be damned cautious about arguing politically knowledgeable Republicans are more likely to believe "their" favorite myths compared to politically knowledgeable Democrats and "their" favorite myths.

And here's a point from Cassino's press release that doesn't comfortably fit the Republicans-are-evil-and-dumb narrative -- young African-Americans are more likely than whites to believe in conspiracy theories.  Not sure how the hell you explain that one.  Would you even want to try?  Not me.

If we buy this set of conspiracy theories are equivalent (debatable) and if we buy into how the survey measures political knowledge (can't tell from the release), then the results raise a number of questions. 

Allow me to get a bit PhDweebish below. 

Greater political knowledge should result in less belief in conspiracies, but hold on.  The theory of motivated reasoning suggests partisans, even knowledgeable ones, will be more likely to believe such myths.  That's all partisans, not just one side of the divide.  Essentially, people believe what they want to believe, especially when it's negative about someone they disagree with or dislike.  So I maintain the Obama-anchored questions are not equivalent to the "truther" and "stolen election" questions.  The latter are from several years ago, myths that received less mainstream news play by comparison -- and certainly damn little near the time the survey was conducted. This can skew your results.

Perhaps, and only perhaps, the Mother Jones article is a textbook instance of motivated reasoning, of seeing and believing what you want to believe -- in this case, about Republicans.

People Lie

I've written about this extensively, research into how people lie (or fib, or fudge, or exaggerate) in surveys.  The most recent issue of Public Opinion Quarterly has a study that examines viewing of U.S. presidential debates. 

The finding? 

People lie.  Or, at least, they exaggerate about their viewing, not unlike previous studies that find people say they voted when they didn't, or that they inflate how often they consume the news and so on.

In the POQ study, Marcus Prior compared survey questions about debate exposure with, as a benchmark, Nielsen ratings of the vice presidential and presidential debates from 2000 through 2008.  Between 47 and 63 percent of voting-age residents said they watched the presidential debates, about double what the Nielsen ratings say is the more likely number.

But is Nielsen a good benchmark?  Probably so, and Prior addresses this by examining data in different ways to make clear his point -- that surveys make for a lousy way to evaluate exposure to a major political event.  It may be all we have, but one filled with error and inflation, so much so that it may affect studies that seek to understand the consequences of viewing on opinions and voting preferences.


 

Booze in the News

Here's a shocking lede:

A new study published in the journal Drug and Alcohol Review reveals that young people do not possess the knowledge or skills required to adhere to government guidelines for responsible alcohol consumption. 

Okay, not shocking if, like me, you happen to work work at a university whose students tend to be stagger their way to near the top of the annual "top party school" list.  The story above, based on survey work in England, found that for five of the seven knowledge questions, fewer than half of respondents gave the correct response.

The actual study is here.  A couple of things to note.  First, the authors do an excellent job with their title, following Hollander's Academic Title Rule.  That is, using titular colinicity, you put in something clever, followed by a colon, followed by what the study is about.  In this case:

My cup runneth over: Young people's lack of knowledge of low-risk drinking guidelines

See how it's done, children?

Second point, to really get at this, you have to read the full report.  Well, you don't, because I'm here to do the ugly work for you, especially in digging into how they measured (every pun intended) drinking knowledge.  It's pretty damn obscure.  Here's the main part from the study methodology:

Participants indicated what they believed to be: the volume (in mL) of pure ethanol in a ‘unit’; and government guidelines (in units) for maximum weekly intake for men and women, maximum daily intake for men and women, and binge drinking for men and women.The number of correct responses was recorded as a total knowledge score. Respondents used 5-point Likert scales anchored with the end-points ‘not at all’ and ‘extremely’ to indicate: how familiar they were with the concept of ‘units’; how useful they believed the concept of ‘units’ to be; and how useful it would be to have more information about ‘units’.

Respondents estimated the alcohol unit content of 10 drinks selected to cover different sized servings of three types of alcoholic drinks consumed by young people. Colour pictures of each drink were accompanied by brief descriptions: red wine: 250 mL large glass; red wine: 175 mL standard glass; regular strength beer: pint (568 mL); Stella Artois lager: 330 mL bottle; Stella Artois lager: 500 mL can; Carling lager: 440 mL can; Carlsberg lager: 275 mL bottle; mixed drink—for example vodka and tonic: pub measure; Smirnoff Ice mixed vodka drink: 275 mL bottle; spirit—for example whisky: 25 mL pub measure. Estimates were dichotomised as outside or within 10% of the actual alcohol unit content [9]. Participants were given a score denoting the proportion of estimates within this range.

Above sounds more complicated than it really was.  Essentially, how much booze is in these various drinks.  Get 10 percent off, you're coded as incorrect.  And a lot of answers were incorrect.

The takeaway?  When it comes to what people know, booze is one of those things they can get wrong.

Knowledge and Acceptance

Here's a survey that essentially argues the more we know about something, the more accepting we are.  The something in this case is the "smart grid."  You probably know the "smart grid" best in terms of those "smart meters" they're sticking on houses to measure and help control power costs and consumption.

(as an aside, the wingnuts out there are convinced such smart meters are part of a UN plot to take over the world)

To get to the knowledge part, scroll down the story to the Knowledge is Key subhead.  The story tells us, for example, that "there is a strong correlation between basic knowledge and willingness to change behavior patterns to meet broad goals."  What it fails to tell us, of course, is how the hell they measured knowledge.  And there are other issues.  More on those in a moment.

More fun:  "However, this pattern is reversed in issues related to privacy. Here, the more knowledge consumers had about energy, the more concerned they were with privacy issues with home energy-usage data."

In Other Words:  the more people knew about the smart grid, the more they liked it but the more they worried about it too. 

In Other Words Part 2:  the more people knew about the smart grid, the more they could answer question about it in a direction that makes, let's face it, common sense.

And finally -- irony alert.  The folks who conducted the survey, IBM, also happen to sell technologies having to do with, yes, smart grids and smart meters.  With some rooting around I found this page that gives some of the survey details (an N of 15,000 across 15 countries).  Here's my favorite.  It looks as if part of the knowledge test involved whether the respondents had heard of smart grids, which in turn is (gasp!) associated with their attitudes toward smart grids.

Our final lesson?  Surveys of this type are designed to get the results you want, which in turn become press releases and web fodder and, eventually, uninteresting blog posts by myself.

Lying About Voting Intent

When people say they're going to vote, do they really vote?  Not according to this study (warning, not yet peer reviewed).  Even worse, from a research standpoint, they find many who say they're not going to vote actually voted.

The study gets at the question a number of ways.  One is simple.  Look at who said they're gonna vote in a survey and then go to the courthouse and see if they actually voted (it's a public record whether or not you voted, but not who you voted for).  They also used an experiment and some other approaches. 

To summarize, these "surprisingly inaccurate self-predictions" are full of measurement error, which can lead to all kinds of problems when you rely on them to, oh, decide on "likely voters" in a horserace poll (this is "grossly inadequate," the authors say) or even as a measure of political interest, motivation, or intent in academic studies.  The authors argue self-prediction is "far inferior to past vote history as a predictor of actual turnout."

Again, the caveat -- this manuscript is under review somewhere and has not as yet undergone a complete peer review.  As such, take the results with a grain of salt until it's been vetted by folks who are serious about this stuff.

But, there's some fascinating discussion at the end of the paper about the psychological underpinnings of saying you're gonna vote and you don't, or to me even more fascinating, saying you're not gonna vote and you do it anyway.  That latter, they argue, may in part be a function of frustration with the political process but among regular voters, the pull of Election Day is simply to much.  So pollsters in particular may be losing important data with their traditional screening questions.

Fruity Knowledge. Veggies Too.

Yes, this blog for four years has focused on political knowledge and especially the media's role in maintaining what people know.  Sometimes, though, you gotta go elsewhere for content, dipping into the obscure and the arcane.  Especially on slow news/academic days.

And that bring us to -- what Australians know about veggies and fruits.

Told ya.  Obscure.  Arcane.  According to this story:

A new survey of young Australians has found one in two don't know how many servings of fruit and vegetables to eat in a day, and even fewer know the serving sizes of common fruit and vegetables.

Yes, the survey of a whopping 106 college students found only 54 percent "knew the recommended daily amounts of fruit and vegetables."  This is the "first concrete evidence young Australians don't know their fruit and vegetable basics."

The boldface above, that's mine.  Concrete?  A survey of 106 college kids?  I'm not saying I don't believe the results, I'm saying the methodological problems in such a small, narrow sample are enough to make me want to skip my fruit and veggies for the day.  But there is some funny stuff here.  Read below.

Some participants estimated the serving size of grapes to be just one grape, and others estimated the serving size for carrots to be the equivalent of 20 carrots, when it is 20 grapes and half a medium-sized carrot (or half a cup of chopped carrot).

Wow.  I think we can all agree this qualifies as a major nutritional fail.  Again, I agree the results are probably accurate, that college kids are clueless on this, but the way they got there, with such a small (random?  hard to say) sample makes me wonder about just how "concrete" the results really are.

And thus ends our obscure, arcane, methodological moment.

What Women Know ... About Fertility?

According to a story on what women know about fertility:

The poll of 1,000 women ages 25 to 35 who had talked to doctors about fertility found that participants could correctly answer seven out of 10 basic questions less than half the time. The Fertility IQ 2011 Survey found that women were wrong most often about how long it takes to get pregnant — and about how much fertility declines at various ages.

First, lemme speak to the story's lede which, I have to admit, kinda creeps me out.  Here it is:

From the outside, Holly Finn certainly looks fertile. 

With shoulder-length dark hair, smooth skin and a slim but curvaceous figure, the San Francisco-area writer could be any young mom with a baby on her hip. 

Wow.  If I wrote that, it'd come across as sexual harassment. Or at least like I'm hitting on her.  Or something.

I can't really find any methodology or even details about the survey, not even by the sponsoring organization's web site.  Not that I'd be qualified to judge the quality of the questions, but it shouldn't be this difficult to find.  Troublesome.

Evangelicals and Romney

Will Christian evangelicals vote for Mitt Romney, a Mormon?  A Pew report out this week notes the following:

... a May 2011 Pew Research Center poll found that about one-third (34%) of white evangelical Protestants would be less likely to vote for a Mormon, echoing reservations that evangelical Republicans expressed about voting for a Mormon during the 2008 GOP primary campaign. 

I'm not interested debating religion and politics.  At least not at the moment (buy me a beer and we'll debate all ya want).  I'm interested here in methodology.  That is, who are these "evangelical Protestants" and how do we know they're "evangelical Protestants."  For some of you, this is obvious stuff.  For others, perhaps not so much.

I downloaded the raw Pew data to peek under the methodological hood, to see how they were identifying people as "white evangelical" or "white mainline" or "white Catholic."  In the latter, the answer is obvious -- respondents answered the race question as "white" and the religious affiliation question as "Catholic" (see below).  We should all utter a collective duh.  Let me lay it out how Pew does this.  Here's the main question:

What is your present religion, if any? Are you Protestant, Roman Catholic, Mormon, Orthodox such as Greek or Russian Orthodox, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, something else, or nothing in particular?

[INTERVIEWER: IF R VOLUNTEERS “nothing in particular, none, no religion, etc.” BEFORE REACHING END OF LIST, PROMPT WITH: And would you say that’s atheist, agnostic, or just nothing in particular?]

1              Protestant (Baptist, Methodist, Non-denominational, Lutheran, Presbyterian, Pentecostal, Episcopalian, Reformed, Church of Christ, Jehovah’s Witness, etc.)

2          Roman Catholic (Catholic)

3          Mormon (Church of Jesus Christ of Latter-day Saints/LDS)

4          Orthodox (Greek, Russian, or some other orthodox church)

5          Jewish (Judaism)

6          Muslim (Islam)

7          Buddhist

8          Hindu

9          Atheist (do not believe in God)

10        Agnostic (not sure if there is a God)

11        Something else (SPECIFY:______)

12        Nothing in particular

13        Christian (VOL.)

14        Unitarian (Universalist) (VOL.)

99        Don't Know/Refused (VOL.)

Now as you can see, it's fairly obvious and on its face holds up pretty well in terms of validity.  This allows us to categorize folks.  Of interest here, of course, are Protestants.  There are a number of ways to continue.  The ANES and GSS approach to this question is often to burrow down to the level of your actual church denomination and then let scholars collapse respondents back into whatever scheme they want, deciding whether certain denominations are "evangelical" or "fundamentalist" or whatever.  They're are actual lists for this.  I have one somewhere that goes down so far and I've used it in the past.

But Pew uses a simpler method.  Shortly after this question, they ask:

ASK IF CHRISTIAN (RELIG=1-4, 13 OR CHR=1):

BORN    Would you describe yourself as a "born again" or evangelical Christian, or not?

1          Yes, would

2          No, would not

9          Don't know/Refused (VOL.)

While it's not certain, I suspect the folks at Pew take the first question I pasted in and combine it with the question just above to categorize folks as Protestant evangelicals.   Toss in the race question and, kaboom, you're ready to rock and roll (er, methodologically speaking).  But how does the Pew approach stack up with the other version?  Not so terribly bad.  There was a study that looked at the two approaches not so many years ago, I believe in POQ, though I can't for the life of me find it now, that found the straightforward question performed pretty well compared to burrowing down to the denomination level.  Not perfect, because we don't know exactly what's going on in people's minds when asked that question above.  My gut tells me the question above, the direct one, misses some people and undercounts the evangelicals.

So all in all, the Pew question isn't half bad and, to be honest, is much simpler and less expensive to ask, to code, and to analyze.

People Don't Lie (as much as we thought)

My dad always told me people lie about three things: how often they vote, how often they went to church, and the gas mileage they get on their vehicle.

It's an accepted truism in the political science literature that survey respondents overestimate how often they vote.  Studies in the past have attempted to gauge this by checking the voting records of survey respondents, with some success.

(as a matter of information for you budding public records access scholars, you can't tell how someone voted, but you can tell if they voted, at least in most states)

There's a new report out at the ANES web site in which the authors examined the voting records of their 2008 national survey sample (report pdf here).   In surveys, the proportion of folks who say they voted is notoriously higher than the actual proportion of people who cast a ballot -- hence we know some folks are, to put it nicely, fudging.  Or so we thought.  See below.

In examining public records, the authors reported four findings:

1. Official government records "contain numerous errors" that makes voter validation difficult.
2. These errors differ widely from state to state.
3. How states report their data makes it tough to compare across states.
4. And here's the biggie -- "We found that for respondents whose government records can be identified, the records and self-reports show very high levels of agreement. This finding implies that overestimation of turnout rates by surveys is attributable to factors and processes other than respondent lying."

In other words, people aren't lying as much as we thought they were lying, and the gap between what people say in surveys about their voting habits may be due, in part, to crappy government record keeping.   Sure, people still fudge, but the difference is partly attributable to lousy records.

This is comforting for scholars who use survey data to study voting turnout, though it kinds makes my dad's advice from above a bit out of date.

North Carolina and Political Knowledge

A study is out this week, reported here, press release here, about what people know in North Carolina and, namely, who knows the most.  The lede:

Here’s what you need to know about an Elon University Poll designed to test North Carolinians’ political knowledge — older over younger, men over women, and Republicans over Democrats or registered independents.

The survey of 515 adults used 10 questions to measure political knowledge.  The news story doesn't tell us what those questions are, but you can find them in a methodological report.  It's an odd group of questions.  Some of them are obvious (who is the VP? who is the speaker of the house?), but the rest are so North Carolina specific, and probably so under the radar for most regular folks, that the demographic relationships mentioned the lede are less than surprising.

Who is the North Carolina Attorney General?  Does anyone care?  Only 19 percent got that one right.  You can argue it's a question that provides good discrimination, separating the men from the boys, so to speak, in what people know.  I dunno that an item so difficult adds much.  And you'll be happy to learn 97 percent correctly identified North Carolina's capital city (isn't it Mayberry?).

So yes, I have a few quibbles with the questions.  Two ask about federal offices, one asks about the role of federal government (who interprets the constitutionality of a law), all good and fine.  The rest either ask about North Carolina officeholders (some vague, some less so), a bit of geography (state capital), and one asking folks to identify the initials SBI (it's the State Bureau of Investigation, no doubt improving the number of correct responses among felons included in the sample). 

I understand the N.C. questions.  After all, it's a N.C. poll.  But the analysis, basically crosstabs in the pdf file above, breaks voters down by party identification, by gender, by age ... but not by the single factor nearly every study in the history of the planet finds is the most powerful predictor of political knowledge -- education.  Maybe I missed it.  But if not, it's a curious omission given the power of education to explain what people know.  Often, when you break people down by education level, all those other factors mentioned above disappear.

The Power of the Probe

A probe in survey lingo is a follow-up question prompted by a respondent's failure to answer a previous question.  For example, in the 2008 ANES, respondents were asked to identify Nancy Pelosi.  If they could not or did not answer, the interviewer was trained to prompt with "Well, what's your best guess?"

That's a probe.

Not all surveys probe an initial lack of response, and this can make a significant difference if we're studying something like political knowledge.  For example, the ANES made available a redacted version of the open-ended responses to certain questions, including those measuring political knowledge (download a zip file here of the Excel document).  It's interesting.  You can see the various open-ended responses, which I've blogged about previously, but they also include a column of the probes.  If a respondent gave an initial answer--right or wrong--the probe code is 5 (no probe) or one of the other codes that signify other stuff. 

A "1" in the probe meant they asked for a respondent's best guess.  And sometimes, the probe resulted in a respondent "guessing" correctly.  How often?  There were 1,294 instances of probes of the Pelosi question.  Roughly counting, I estimate at least a hundred instances, perhaps more, where the probe resulted in a correct answer.  And that's only looking at the Pelosi question. 

In other words, a probe can definitely influence the results, which I suspect has some bearing in analyses.  And this doesn't even get into what is considered a "correct" versus an "incorrect" response.

If I was so inclined, I'd do a paper on the power of probes to elicit a correct versus an incorrect response, and then position these competing approaches to political knowledge against key variables to see whether the probe improves results.  Honest, I'd do it, except I don't know where the heck I'd publish something like this.  Public Opinion Quarterly?  Dunno, cause I'm not sure I'm smart enough to successfully publish there.  It's full of folks far brighter than myself.

Wireless Only

Doesn't really fit what I typically blog about, but this topic comes up quite often when I teach public opinion. The proportion of folks in the U.S. who are "wireless only" without a landline phone continues to rise, as can be seen by the accompanying graphic.

For an excellent report, go to this article in pollster. The federal report is here.

And yes, there is a what people know angle.  Much of our knowledge about people's opinions derive from good public opinion surveys, but they're becoming more challenging to conduct as folks shift away from landline phones to cell phones or other mobile devices.  Many worry that our understanding of what people think about key issues will suffer as a result, though so far that doesn't seem to be the case, at least according to some recent work by Pew.

I Never Listen to Celine

This New York Times story gets to the heart of methodology.

American men have a naughty little secret. Sometimes, they like to relax with a little Céline Dion. Professed classical music fans have one, too: as it turns out, they don’t tune into classical radio nearly as much as they claim.

Turns out, new measures of listening -- away from surveys and more to boxes that measure actual audience behavior -- finds out something we've always known.

People lie.

Okay, not so much lie as fib.  My dad always told me people lie about three things: did you go to church, did you vote, and what gas mileage does your car get.  My corollary on gas mileage is, the higher the cost of gas, the more SUV owners lie about their mileage.  When gas was $4 a gallon in the U.S., people with fat rolling boxes lied through their teeth about their mileage.

Back to radio listening, or to be honest, any media consumption habits.

As someone who got tenure thanks to research on talk radio, I found the next part interesting.

Talk radio, a largely conservative format, turns out to have fewer fans than previously thought. Talk radio’s market share declined 2.6 percent in the study of areas where the meters were used.

There's a lot more here to examine, but the basic methodological point is simple: self-reported behaviors in surveys are often the best measure we have, but they're blunt instruments at best, with lots of random (and non-random) error.  We often answer in ways that make us feel good about ourselves, which has classical music fans a bit worried since the new numbers show fewer listeners to this genre of music than we thought.

There's even a race card.  You knew there had to be.

The makeup and size of Arbitron’s sample is an issue for some Hispanic and urban broadcasters, who say metered readings undercount minority audiences and hurt their stations disproportionately. Mr. Adams of Arbitron said the company was responding to concerns by adding more panelists who had cellphones rather than landlines, and investing in in-person coaching to make sure all panel members use the devices correctly.

In other words, the new (better) system finds fewer viewers and listeners of "ethnic" programming, which includes everything from AM radio to BET programs.  It's fascinating stuff, proof that methodology can have real-world implications.

Oh, and I never listen to Celine. Ever. Measure it any way ya want, buckos, and it ain't gonna happen.

When Polls Mislead?

Some are suggesting that polls of everyday folks on really complicated or emotional issues are misleading at best, probably useless at worst.  Recent examples of this argument can be found by Mark Blumenthal and Joe Klein.  Some agreed with this position but focused more on how "the media" report the polls (or even the kinds of poll questions "the media" ask) rather than the polls themselves.  The topic is, of course, health care, and what the American public really thinks about a complicated piece of legislation that is longer than most Harry Potter novels, and a helluva lot less interesting to read.

Maybe most of us could just wait for something simpler, something a bit more cheesy.  A Twilight novel version?

We swerve awfully close to elitism when we ask how much the public really understands of the topics its polled on, but to be honest most research suggests the public offers answers to survey questions without really, truly, understanding what its being asked.  So elitism?  Yeah.  But also reality.

Our Personal Networks: A Methodology Moment

We get our political information in lots of ways -- the news media, obviously, but also from the people we talk to every day, our social network.

It's the latter that is the focus of an article in the most recent issue of Public Opinion Quarterly. The authors look at two different ways of generating survey data about people's social networks.  As the authors write:

"One method asks respondents to provide information about the individuals with whom they discuss 'important matters,' while the other asks respondents to provide information about the individuals with whom they discuss politics.  We find that respondents provide more or less the same data on their political discussion partners regardless of which name generator procedure is used."

My first thought was -- wow, cool, you can get published with a null finding.

My second thought was -- there's leeway in how we study social networks, allowing us to confidently compare studies with slightly different methods.  Useful.

The authors conclude that our "core and political discussion networks comprise the same individuals."  In other words, we talk to the same people ... politics, or not.

That's good to know from a methodological standpoint.  I assumed people carve their larger social networks into smaller chunks.  This chunk is made up of people you talk about politics with, that chunk is people you talk music with, that other chunk is for religion, etc.  Maybe some overlap, but I always assumed there were differences.  This research suggests not.

I'm a bit skeptical. Maybe I'm generalizing from an N of 1, but maybe the lack of a result here has more to do with a methodological quirk in the wording ("important matters" versus "politics") than in how we consciously or unconsciously create our interpersonal networks.  I don't know.  My gut says the method here is too mild to truly capture differences in our interpersonal networks.  The lack of results are useful, especially if you want to combine data from different approaches in a larger analysis, but I don't think we can say quite yet that we have the same interpersonal network for everything.

Helping People Decide

The latest Public Opinion Quarterly includes a study with direct bearing on what people know -- in this case, the idea that informing them before surveying them will influence how they answer questions.

The study is Helping Citizens Decide in Referendums (with, of course, a colon and long sentence to create titular colinicity).  The dependent variables were likelihood of voting, preference, strength of opinion, and vote consistency.  What's that last one mean?  It harkens back to some traditional  research that examines how people organize their political positions and whether there's a logical (consistent) arrangement of attitudes.  In other words, you tend to be liberal on a host of issues, or conservative.  Therefore, consistent.

Independent variables?  Various flavors of political sophistication or knowledge through something called ICQ -- which stands for Information and Choice Questionnaire.  Basically, providing people with info about a specific problem before asking them what they think.


A careful reader will note this sounds like deliberative democracy (see my post here on it).  Not quite.  No deliberation taking place, at least the way it's meant, but yeah -- kinda sorta the same idea, that informed opinion is different than public opinion.

Informing people in advance before polling them did lead to more consistent opinions, but more so for people with less education and interest.  This follows theory, to a point.  As the authors note, cognitive processing models such as ELM suggest that people high in political sophistication will be more able and motivated to carefully consider an issue, but here it's the people with lower political sophistication who "profit more." 

I suspect some of this has to do with the artificial nature of a lab experiment, but even so it raises some interesting theoretical and methodological questions.  We've always known informed opinion differs from generic public opinion.  This study takes another step closer to understanding not only how it differs, but the consequences of those differences.

Problems with Internet Polls?

Public opinion surveys are a key means of learning what people know or think about the issues of the day.  The growth of Net-based polling is not without critics, and this story based on new research by some Stanford folks suggests that these "opt-in" panels are nowhere near as accurate as traditional randomly drawn samples.  Read the full study here (pdf).

Far Out Answers

Sometimes when you ask straightforward questions to a survey respondent you get far out answers. As I was messing around with an advance release of the ANES 2008 election data, I came across these two questions

• As far as you know, what is the current unemployment rate in the United States, that is, of the adults in the United States who want to work, what percent of them would you guess are not unemployed and looking for a job?
• What is your best guess of the average price of regular unleaded gasoline across all of today?
  

As to the first, depending on which set of numbers you use, the unemployment rate at about the time of the survey at election time was probably between 5 and 7 percent (interviews happen over a couple of months). One fed number says the average for 2008 was 5.8 percent, so let's assume it's in the ballpark.

How did people answer? Wildly. First the good news. The mode, the most popular response, was 6 percent. Sweet, but that's by a mere 264 of over 2,000 respondents. Other estimates of the U.S. unemployment rate ranged from 0 (goofy optimistic) to 100 (sickly pessimistic, but shared by 25 people grasping for an answer). Thus the challenges of asking knowledge-based questions. The gas price is a bit tougher since it's state-specific and the costs differ. The average U.S. price at about the time of the survey (serious fudging here, sorry) was between $3.50 and $3.95 (prices peaked about July 2008). Most of the survey responses were significantly lower, though one person apparently paid $50 a gallon. Maybe in Europe.

The lesson? People not only guess and guess badly, they're just plain goofy and give the silliest answer they can give to some survey questions that requires them to seriously recall number-based facts. As an analyst I'd collapse all responses over a certain point -- say all $5 or more for gas put together at $5 -- or I might even toss out the outrageous ones. It depends on the purpose of my analysis, I suppose, but the point here is that you can never guess how people will answer a seemingly simple, straightforward question.