« Previous | Main | Next »

The Key, or the Chimera?

November 10, 2008 10:39 AM

(See end of post for afternoon addition.)

One of the items that prompted my post on the chimera of perfect polls was a Fox News piece quoting Brian Schaffner, a political science professor at the University of Massachusetts. I found the quotes and paraphrases alarming:

“All the polls predicted Obama would win, but the degree to which they got the margin is the key to successful polling, said Brian Schaffner…

“‘A real bad prediction is if you picked the wrong guy. And then it really comes down to – for a pollster – it comes down to bragging rights. …There’s a significant difference between a 5 to 6-point win and a 10-point win.’

“‘In 2004, Gallup did the best in terms of pegging the final outcome.’

“‘Pollsters who did call cell phone-only respondents tended to overstate Obama’s support somewhat.’”

The chief point of my post was to dispute directly the notion that nailing the horse race, much less the margin, is “the key” to successful polling. While a good estimate is necessary, pinpoint precision is a fiction. The notion of “bragging rights” and “pegging the final outcome” deflects us from the real aim of serious research. (And I’d add that given the sample sizes at hand there is not a “significant difference” between, let’s say, 52-46 percent and 54-44 percent, a 6-point vs. a 10-point margin.)

Nor is it correct that polls that took the trouble to include cell-only respondents “tended to overstate” Obama. Among the final polls that included cell-phone respondents, ABC/Post had Obama at 53, Gallup 53, CBS 51, Pew 49 – none an overstatement. (I am referring to polls, not projections. Gallup projected Obama at 55, Pew at 52.)

Since in the Fox posting Schaffner sure sounded like a proponent of micrometer-level evaluations of horse race estimates to establish the “best poll,” I invited him to reply to my posting. His comment almost speaks for itself, with two closing thoughts to follow.

Schaffner:

“I think the point you make in your column is absolutely correct, and it is one that I stress whenever I teach my students about the science of polling. The true value of public opinion polls – good public opinion polls – will always be as a way to understand why, not predict how, Americans cast their votes.

“However, I do think there is value in stepping back after an election and using the actual vote outcome to help us assess what we can learn about the science of polling. Indeed, I did not plot the performances of different survey organizations at Pollster.com in order to serve as referee for some pollster bragging rights contest. Rather, I did so in an effort to better understand the challenge that the cell phone only population poses to our ability to draw valid inferences about the American public.

“Elections provide one of the few moments when we get a census of public opinion that we can use as a benchmark with which to compare our survey estimates. Thus, pollsters and political scientists should absolutely use these moments to learn how to improve the science of polling. Why did polls interviewing cell phone only respondents fare about the same as those excluding that population?  Why did every poll released in the last weekend of the campaign understate McCain’s support? These are just two issues that pollsters need to evaluate in the coming months and only by comparing the polls to the actual results are we able to identify such questions. You are correct that we shouldn’t obsess over which pollster came closest to pegging the actual result, but that doesn’t mean that we shouldn’t evaluate why some polls came closer than others.”

My final comments start with a correction. McCain’s support was not understated in all the final polls, and indeed exactly as many “understated” Obama as did McCain (playing the game in which 1 or 2 points is an understatement). That’s chiefly a function of the level of undecideds they chose to report, itself a function of polling technique (and, to my taste, a too-convenient route to fiddling with allocations).

Of greater concern is Schaffner’s suggestion that including cell-only respondents is problematic because it didn’t improve estimates. In a non-significant way this is so; without cell-onlies our final Obama-McCain estimate was 52-45 percent rather than 53-44 percent, as I reported in our final pre-election analysis. Examining why is a fine thing. But taking it to the next step – arguing that noncoverage is desirable if it “improves” the allegedly precious horse-race estimate - would be precisely the wrong direction to take.

Afternoon addition:

Prof. Schaffner asked to add the following "with a note that my original statement was written when I thought you were referring to my Pollster.com post" (linked above):

"I was called by a FoxNews.com reporter on Thursday morning after the election. The reporter asked some questions about the accuracy of the pre-election polls, which I answered to the best of my knowledge. In my discussion with the reporter, I continually emphasized that assessing the accuracy of the polls was particularly important for helping us determine whether there was a "Bradley Effect" or a "Cell Phone Only" effect in the polling. At the end of the phone call, the reporter told me that she was going to try to post the story within an hour. I was immediately concerned because this suggested to me that my comments were not going to be used to help her formulate a story on polling, but rather to fit into a story she already had in mind (if not on paper).

"During my conversation with the reporter, I did discuss how the polling industry often judges which pollsters performed best in predicting the outcome. However, I was not making any normative judgement as to whether I thought that the concept of competing for bragging rights based on predictive accuracy was desirable. As I noted in my earlier response to Gary, I generally find the question of predictive accuracy to be more novelty than anything.

"Finally, during the interview I did say that it seemed as though pollsters calling cell phone only respondents overstated Obama's support. However, I had not looked at the data closely at that point (which I admitted to her) and it is now clear to me that there was no such pattern. I should not have speculated about the pattern before I had investigated it more closely.

"Ultimately, Gary and I may have some disagreement about the value of using election results as a benchmark for evaluating survey methods. However, my guess is that if Gary had been privvy to my entire interview with the reporter rather than simply the remarks that made it into the story, he would have found far less to quibble with."

November 10, 2008 in 2008 General Election | Permalink | User Comments (3)

User Comments

Posted by: Bonnie Kimberly | Nov 10, 2008 11:00:57 AM

Posted by: Mortimer Snerd | Nov 10, 2008 11:09:36 AM

Posted by: Jwench | Nov 10, 2008 11:29:00 AM

Post a comment

Name:

Email Address:

URL:

Remember personal info?

Comments: