The Power of the Probe

A probe in survey lingo is a follow-up question prompted by a respondent's failure to answer a previous question.  For example, in the 2008 ANES, respondents were asked to identify Nancy Pelosi.  If they could not or did not answer, the interviewer was trained to prompt with "Well, what's your best guess?"

That's a probe.

Not all surveys probe an initial lack of response, and this can make a significant difference if we're studying something like political knowledge.  For example, the ANES made available a redacted version of the open-ended responses to certain questions, including those measuring political knowledge (download a zip file here of the Excel document).  It's interesting.  You can see the various open-ended responses, which I've blogged about previously, but they also include a column of the probes.  If a respondent gave an initial answer--right or wrong--the probe code is 5 (no probe) or one of the other codes that signify other stuff. 

A "1" in the probe meant they asked for a respondent's best guess.  And sometimes, the probe resulted in a respondent "guessing" correctly.  How often?  There were 1,294 instances of probes of the Pelosi question.  Roughly counting, I estimate at least a hundred instances, perhaps more, where the probe resulted in a correct answer.  And that's only looking at the Pelosi question. 

In other words, a probe can definitely influence the results, which I suspect has some bearing in analyses.  And this doesn't even get into what is considered a "correct" versus an "incorrect" response.

If I was so inclined, I'd do a paper on the power of probes to elicit a correct versus an incorrect response, and then position these competing approaches to political knowledge against key variables to see whether the probe improves results.  Honest, I'd do it, except I don't know where the heck I'd publish something like this.  Public Opinion Quarterly?  Dunno, cause I'm not sure I'm smart enough to successfully publish there.  It's full of folks far brighter than myself.

Where's the 2008 ANES?

We had full versions of the ANES data from the 2000 and 2004 elections by April of the next calendar year (April 2001 for the 2000 data, April 2005 for the 2004 data). 

It's November 2009 and we're still waiting for the full version of the 2008 ANES.  Hell, we may be in the next election cycle before the data appear.

On a related note, a few months ago the ANES folks presented an update at a major conference.  I couldn't attend (who the hell has travel money other than overindulged senior faculty?), but staff told me the presentation materials would be put online by the next week.  I assumed these would be powerpoint slides or stuff like that.  That was three or four ago, still no presentation materials that I can see.

I'm assuming something in that presentation would explain the delay, such as issues with open-ended coding or other quality-control matters.

Consider this my whine for the day. 

I know staff are not just sitting on the data, but as a user I'd like a little more information on what's going on, why the delay, and what it may mean for those of us who love to sit around and push SPSS buttons to analyze data.  And, oh, publish research.

A Specificity Index for Open-Ended Coding

I blogged yesterday (see below) about problems in coding open-ended responses to survey political knowledge questions. I used the Nancy Pelosi question as an example. The "correct" response, from a scholarly standpoint, would have respondents identify her as Speaker of the House, but I also argued that it was equally correct to identify her as a congresswoman, a member of the House, and a lot of other answers that in the past would have been coded as "incorrect" in the American National Election Studies dataset.

Go back to yesterday's post for links to ANES data, the newly released raw open-ended responses, and other important points of interest, especially problems with earlier coding.

We don't know exactly how ANES staff will code these answers, but when release the next version of the 2008 pre- and post-election data, I'll do a comparison then. Today I offer an alternative Specificity approach to coding. It's simple. Anything resembling "Speaker of the House," given its specificity, gets coded as the highest, most correct, response. Let's call it a "3" for the sake of argument. Identifying Pelosi as a member of the House, while correct, loses that specificity, so it gets a "2." Calling her a politician or something similar, that's correct in a vague sort of way, so it scores a "1." And getting it wrong, that's a "0."

Missing and refusals get their own special codes. Scholarly typically recode a refusal to be the same as an "incorrect" response. That's a different problem for a different day.

My specificity method provides greater data range. If someone doesn't like it, they can collapse the resulting codes into any method that strikes them as useful, especially if they're comparing answers in 2008 with some previous year.