A Climate of Comfort at UGA

There's a story in today's Athens Banner-Herald on a campus climate survey UGA did last year, the results finally being released. I emphasize finally because I'd requested this very study earlier in the summer -- as well as the raw data -- but was told it wasn't ready. Apparently it is. I've asked for my own copy and for the data so I can look it over, but for now we'll have to be satisfied with the ABH story.

Read it yourself. UGA seems quite happy with itself. Buried at the bottom are some methodological caveats that deserve note.

[The consultants] cautioned about over-generalizing the results, pointing out two shortcomings in the voluminous data. One is that people who take the time to participate in an online survey of this sort may be “self-selecting,” different from those who didn’t choose to take the survey.

In other words, this is a SLOP, a self-selected opinion poll. In other words, not generalizable, not really applicable in any way.  Barely useful. Why do it this way rather than a professional survey based on a random, generalizable sample? It costs less, so maybe it's just money. Or, just maybe, you're afraid of the results from a truly random and professional survey.

This also deserves mention:

The group most likely to participate was administrators and staff — about a third.

That alone stands out as either a damning methodological fuck up or a clever way to cook the data, to get a positive result (and the results are largely positive for UGA). Is it merely a concidence? Conspiracists, assemble.

Again, I want a copy of the report myself (I've requested it, again) and the actual data (they'll fight me on this one, but I love a good spat). UGA is usually quite good at filling public records requests. Very professional, very timely. But because a consultant did this job they may try to stiff me on the actual data. We'll see.

Finally I'm a bit baffled by the numbers in this paragraph.

Another limitation, the consultants said, was a low response rate to the survey. More than 10,500 workers and staff completed the online survey last fall from Oct. 20 to Nov. 20, out of about 46,500 eligible — about 23 percent of the total UGA and worker count.

This demonstrates a lack of understanding when it comes to surveys. If you do it right, with a random sample, you don't need a high response rate. You need a good sized sample. UGA was shooting for 30 percent participation (there's nothing magic about that number from a scientific standpoint, by the way). We got 23 percent participation. That's not bad. It's the makeup of that 23 percent, its non-randomness, that makes for questionable results.