Face-To-Face vs Online Surveys

The general idea is controversial or sensitive survey questions get very different results if asked to respondents face-to-face or on the phone versus a more impersonal approach, such as online. Let's look to see if that's the case using fresh ANES 2016 election data and whether a survey was online versus face-to-face.

• Is Obama a Muslim? Percent who say Yes:
• F2F: 31.1%
• Online: 30.3%

• Voted for Trump
• F2F: 41.3%
• Online: 39.2%

OK, let's look at these first two above. Clearly the Obama Muslim question has no mode effect, in other words asking it face-to-face versus online makes no difference given it's 31.1 versus 30.3 percent thinking he's Muslim. On voting for Trump there's no real difference either. We're talking a couple of percentage points, nothing significant or at least not substantive. OK, let's try another.

• Should transgender people have to use their birth sex bathroom?
• F2F: 50.2%
• Online: 52.6%

Slight difference above, but again it's slim, too slim to have any real meaning. How about something more mundane.

• Do you attend religious services?
• F2F: 64.5%
• Online: 58.2%

There's something going on in the question above, one that fits theory. Usually a socially desirable response (attending church, etc.) gets more "yes" responses in a phone or face-to-face survey, and that's the case here by several percentage points, enough that I'd argue survey mode matters. Here's another that's kinda interesting below:

• Favor a wall on Mexican border
• F2F: 29.5%
• Online: 33.4%

That's a decent spread above, enough that I'd argue we have a small mode effect. Respondents were a little more willing to favor building a wall in the online group versus the face-to-face group.

I can do this all day, but I'm running out of time. As we can see above, there are mode effects and they do matter, but sometimes they don't matter at all.