Far Out Answers

Sometimes when you ask straightforward questions to a survey respondent you get far out answers. As I was messing around with an advance release of the ANES 2008 election data, I came across these two questions

• As far as you know, what is the current unemployment rate in the United States, that is, of the adults in the United States who want to work, what percent of them would you guess are not unemployed and looking for a job?
• What is your best guess of the average price of regular unleaded gasoline across all of today?
  

As to the first, depending on which set of numbers you use, the unemployment rate at about the time of the survey at election time was probably between 5 and 7 percent (interviews happen over a couple of months). One fed number says the average for 2008 was 5.8 percent, so let's assume it's in the ballpark.

How did people answer? Wildly. First the good news. The mode, the most popular response, was 6 percent. Sweet, but that's by a mere 264 of over 2,000 respondents. Other estimates of the U.S. unemployment rate ranged from 0 (goofy optimistic) to 100 (sickly pessimistic, but shared by 25 people grasping for an answer). Thus the challenges of asking knowledge-based questions. The gas price is a bit tougher since it's state-specific and the costs differ. The average U.S. price at about the time of the survey (serious fudging here, sorry) was between $3.50 and $3.95 (prices peaked about July 2008). Most of the survey responses were significantly lower, though one person apparently paid $50 a gallon. Maybe in Europe.

The lesson? People not only guess and guess badly, they're just plain goofy and give the silliest answer they can give to some survey questions that requires them to seriously recall number-based facts. As an analyst I'd collapse all responses over a certain point -- say all $5 or more for gas put together at $5 -- or I might even toss out the outrageous ones. It depends on the purpose of my analysis, I suppose, but the point here is that you can never guess how people will answer a seemingly simple, straightforward question.