Word Order in Surveys

I was messing with data and came across this very simple wording experiment in the 2012 ANES time series survey.  Respondents were randomly assigned to get one or two versions of the same question in which they're asked if voting is "mainly a duty" or "mainly a choice," the difference being the order in which these responses were presented.  Did it make a difference? Kinda. And it kinda matters depending on whether respondents were also randomly assigned to the face-to-face or web versions of the survey. What I present below are weighted results.

First, here's the question itself:

Different people feel differently about voting. For some, voting is a duty - they feel they should vote in every election no matter how they feel about the candidates and parties. For others voting is a choice - they feel free to vote or not to vote, depending on how they feel about the candidates and parties. For you personally, is voting mainly a duty, mainly a choice, or neither a duty nor a choice?

That's the "duty first" version above (obviously). The "choice first" simply changes the order. Now, the results in which I combine them from both kinds of surveys:

                           Duty First   Choice First

Voting is a Duty        22.1            23.6
Voting is a Choice    21.2            19.7

As you can see, there's a recency effect of sorts. Giving the "voting is a choice" potential response first, followed by the "duty" response, results in a slightly higher "voting is a duty" answer.

Okay, but what about F2F survey versus one on the web? One argument might be that in a face-to-face survey, social desirability might lead one to be more likely to choose the "duty" answer regardless of response order. That does happen, and mostly on the web version, and especially on the web version in which "duty" is offered second.

Our takeaway? If there is one, it's that subtle difference do occur if you fail to randomize key questions, especially the order in which responses are provided.

I'll report more of these as I come across them because, dammit, they interest me if no one else.