The Numbers
A Run at the Latest Data from ABC's Poobah of Polling, Gary Langer
Gary Langer is director of polling at ABC News, where he's covered the beat of public opinion for nearly 20 years - conducting and analyzing ABC News polls, evaluating data from other sources and setting the news division's standards for poll reporting. Langer is a two-time Emmy award winner, both for ABC's reporting of public opinion polls in Iraq.
FAVORITES
RECENT POSTS
MONTHLY ARCHIVES
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | |
| 7 | 8 | 9 | 10 | 11 | 12 | 13 |
| 14 | 15 | 16 | 17 | 18 | 19 | 20 |
| 21 | 22 | 23 | 24 | 25 | 26 | 27 |
| 28 | 29 | 30 | 31 |
« Previous | Main | Next »
MOE and Mojo
December 03, 2007 10:58 AM
The Des Moines Register’s new poll, released Sunday, has Barack Obama 3 points ahead of Hillary Clinton in Iowa, which it characterizes as a “lead” for Obama. Our own ABC/Post poll two weeks ago had Obama 4 points ahead – and we called it “close,” not a lead.
What gives?
The answer is that it’s all about how far you’re willing to push the envelope. To the Register, we’re probably being too conservative. To us, the Register is going overboard. But there’ll be more of this to come in the weeks ahead, so it’s worth understanding how we get here.
A poll is not laser surgery; it’s an estimate. The reliability of the estimate is (in part) a function of its sample size. This is expressed as the margin of sampling error, and it’s customarily given at the 95 percent confidence level. For a poll of 500 likely voters, which was the sample size in both the ABC/Post and the DMR polls, that’s plus or minus 4.5 points. This means a candidate would need a lead of 9 points or more for us to say with 95 percent confidence that it’s statistically significant. Neither poll comes close.
However, the estimate a poll gets is in fact the likeliest true value, and the likelihood decreases as we move toward the extreme ends of sampling error. In fact we can calculate the level of confidence we can have that Obama really leads in either of these polls.
The answer: With Obama +4 vs. Clinton in the ABC/Post poll, we could have said with 77 percent confidence that – all else equal – he really had a lead. In the DMR poll, with Obama +3, the confidence level is 64 percent. Apparently 64 percent confidence is good enough for the Register to call it a “lead.” It’s not for us; nor was the 77 percent probability in our last poll.
Why not? One reason is that the customary confidence level in survey research is 95 percent, not 77, or 64. Another is that these probabilities only hold if all else is equal – and it isn’t. These estimates also are subject to non-sampling error, the likeliest cause of which is their estimate of who in fact qualifies as a likely voter. In our tighter likely voter model, more closely approximating caucus turnout in 2004, we didn’t have Obama +4, we had him +2, with 28 percent to Clinton’s 26 percent. The probability of that being a statistically significant lead was only 37 percent. Thus the prudent course was to call it close, which is what we did, and what we think it is.
Now on the Republican side, with 400 interviews, we had Mitt Romney with 28 percent support, Mike Huckabee with 24 percent. Huckabee had all the mojo – he was the guy making the move, as our analysis two weeks ago made clear. The probabilities still had us call the race close.
That Huckabee mojo looks to be continuing; the Register now has him at 29 percent support, to Romney’s 24 percent. The Register, again, calls Huckabee the leader. The confidence level 88 percent. Is that enough to call it a “lead”? It’s tempting. But before going there we’d want to see what turnout their likely voter model anticipates and what their other models (if any) show. Meanwhile, in our book, this one, too, is close.
As you can see, there's a bit of judgment in all this. The shorthand approach used by the AP is to say that when a candidate's numerical advantage doesn’t exceed sampling error, but is at least half of what sampling error demands, it can be called a "slight" lead. That's of course not what it is; it's really a possible lead in which we cannot be wholly confident.
All this underscores one of the fundamental points about pre-election polls: They are estimates. Even with good-quality methodology, the notion of pinpoint accuracy is a myth. And the reason we do them is not simply to try to puzzle out who's ahead – but to understand how and why the voters are coming to their choices.
December 3, 2007 in Favorite Posts | Permalink | User Comments (3)
TrackBack URL for this entry:
http://www.typepad.com/t/trackback/433071/23895394
Listed below are links to weblogs that reference MOE and Mojo:
But this discussion treads every poll as independent. When multiple polls show the same "nonsignificant" lead, the cumulative judgment of a "lead" may well be significant--and justified.
Unfortunately, most news organizations still report their own polls as if they were the only ones out there. Understandable, given the financial investment, but this habit deprives the reader of access to the full realm of information available.
Michael O'Neil, PhD
Posted by: Mike O'Neil | Dec 3, 2007 1:31:34 PM
Mike, yes, you could average polls to boost the N and improve confidence. Especially if they're polls that used similar methodology in a similar time period. But that's very often not the case. There are many methodological differences in the polls we see - some random-digit dialed, some registration-based, using different (or in some cases no!) respondent selection, different sample management, different weighting, with different levels of reported undecideds, different likely voter modeling, done at different times. Which do we combine? Which do we leave out? Do we throw them all into the mix, including questionable data along with approaches that are more defensible yet still very different? Some would, but I'm not so sure.
We can use all these polls - individually evaluated on their own merits - to inform our judgment, going well beyond the horse race to the issue and attribute questions, strength of support and the like. But when it comes to calculating the confidence level associated with a 28-25 percent horse race in a particular poll, it seems sensible to me to stick with that poll. Or if aggregating and averaging, at least to say so, with some sense of the thinking behind it.
Posted by: Gary Langer | Dec 3, 2007 3:34:35 PM
I am well aware of the range of methodologies used in pre-election polling. Indeed, we are so much of a "science" that no two organizations use exactly the same method. This is especially true in the case of hyper-slippery estimates for low-turnout caucuses.
But, realizing that the turnout projections (as opposed to the initial sampling algorithms) are as much art as science, it is precisely the knowledge that when different approaches to turnout yield the same result that a reader's confidence should increase.
That said, with Obama having about a 3% "lead" in two Iowa polls of likely caucus attendees, I think you are right in pronouncing this less than a definitive lead. Caucus projections are more slippery than primary projections, which in turn are more slippery than general election predictions. And these are far from perfect.
Still, I do find a second instance of a "lead" worth note. Even more so, I think it is clear that there is definite MOVEMENT in his direction, even if it is not clear that he currently has a lead.
(And, as a result, I'd take an even-money bet that he has a higher percentage than Hillary in the next publicly-reported poll. I could be wrong on this one, but I think the odds of that are slightly below 50%).
Mike O'Neil
Posted by: Mike ONeil | Dec 3, 2007 6:03:36 PM
Post a comment
