Catholic Sample Size

I'm reading a story in The Georgia Bulletin (my Catholic diocesan newspaper -- yes, I get that bored) and came across this story:

Survey finds ongoing dissatisfaction with new Missal language

I can't find the story on the paper's web site, but here's a badly written version via the Catholic News Service. So why am I blogging about this? Because it touches on two of my favorite topics: (1) public opinion surveys, and (2) stories about public opinion surveys. Oh, plus I'm (obviously) Catholic.

A bit of background for the heathen among you. The Mass language changed a year or so ago. No one likes it, not really, but it's minor stuff mostly and by now most of us have gotten used to what is apparently more accurate, though less poetic, translations from Latin to English. There, more than you wanted to know.

Here's part of the story I have in my local Catholic "newspaper."

Three quarters of the survey participants agreed that the language of the new text is "awkward and distracting," and half said the translation "urgently needs to be revised."

The survey itself was of 519 respondents drawn from 6,000 randomly selected parishes. From a methodological standpoint, not a bad process. It's of priests and lay leaders, so we're not talking everyday Catholics here.

Msgr. Richard Hilgartner raises a couple of legitimate concerns and then adds this bit of methodological criticism. According to the story: "He also raises questions about whether the number of responses represent a meaningful sample of sentiment about the translation."

A survey of 519, assuming a good sample, is adequate. You get a margin of error of about 4.3 percent, depending on the method used to calculate it. Given the big numbers seen above, that's okay. Not great. I'd like 1,000, but 519 is perfectly adequate given the population you're trying to describe. In other words, the good monsignor should probably keep to liturgy and not survey work.

Now, a word about the story. The version I see in print, and online, both suck. I mean really suck, as in there are lots of ways to report a poll's findings and there are lots of ways to not do it well, and in this case my print version includes every possible way to screw it up. We need bullets for the questions, some way of organizing the results, and don't take a priest's word at methodological criticism. I mean, he's a friggin priest, not a survey researcher (best I can tell).

The lesson here? There's a lot about this survey to like, though I wonder at the construction of some of the questions, and certainly the stories based on the results fall far short of what a pro should do. Plus here's the kicker -- the new wording does suck, but I figure as more of us older Catholics die off the new generations will shrug and accept it.