About That UGA Climate Survey

Last year UGA did a "climate" survey. Not climate as in weather, as in warming, but climate as in diversity and that sort of thing. I wrote about it a few days ago, based on this Athens Banner-Herald story. Now I can provide you the link to the actual study, which the ABH story did not do for some reason, and discuss it in some small detail.

First, here is the link to the study page. You can read the full report (all 447 pages of it), a summary, or look at specific parts. Enjoy.

Back to my points. First, I requested from UGA a copy of the report and it provided me with the link, which is terrific. I also asked for a copy of the actual data. That request was denied because the data is part of ongoing research. Here's the open records line quoted to me. My discussion follows, but I boldface the part I find of interest.

This specific exemption applicable to those records is O.C.G.A. § 50-18-72 (a)(36). That exemption applies to: Any data, records, or information developed, collected, or received by or on behalf of faculty, staff, employees, or students of an institution of higher education or any public or private entity supporting or participating in the activities of an institution of higher education in the conduct of, or as a result of, study or research on medical, scientific, technical, scholarly, or artistic issues, whether sponsored by the institution alone or in conjunction with a governmental body or private entity, until such information is published, patented, otherwise publicly disseminated, or released to an agency whereupon the request must be made to the agency. This paragraph shall apply to, but shall not be limited to, information provided by participants in research, research notes and data, discoveries, research projects, methodologies, protocols, and creative works.

By providing me a link to a published study and summary, isn't that published? My read of the open records law says, by publishing the full study, the data is now available. I've asked for a clarification on exactly that point, but I'm hardly in the mood to fight it out in court or anything like that. The stakes are too low.

Back to the report itself. I don't care so much for the results except how the methodology skewed them. See my earlier post linked above, but basically this is a self-selected survey, meaning anyone could participate, and it turns out that of the potential respondents among students, staff, faculty, and administrators, one-third of the same are admins and staff. That's so off as to make the results questionable.

So why not a professional job with a random sample? Here's the answer:

The goal of the survey was to hear as many voices as possible. Rankin and Associates Consulting recommended not using random sampling because that methodology may inadvertently exclude populations where numbers are very small (e.g., Native American faculty). The survey was open to all faculty, staff and students to include the broadest array of perspectives possible.

First, the goal of research is to provide a useful, a generalizable answer. This fails. That said, they've done the same thing at other places, such as this Dartmouth study. According to Rankin & Associates, the consulting firm, for response rates less than 30 percent "caution is recommended when generalizing the results to the entire constituent group." The UGA study fell below that threshold, plus I'm unfamiliar with any magic assigned to the 30 percent response rate. It may be true, I've just never heard it before.

Second, oversampling certain populations is accepted practice, if done via random sampling.

When you have a SLOP (a self-selected survey), you get far more responses from people either very pissed about something or those who need the results to look good (i.e., admins and their staff). In other words, the results are deeply biased, deeply flawed.

On another day I want to break down the respondents. It's surprising. For example, 224 of the respondents work in the vice president of student affairs office, 84 for the VP of instruction, 44 of the respondents work in the provost's office, and so on ... including 19 from the office of the UGA president. Make of all this what you will.