The Numbers

A Run at the Latest Data from ABC's Poobah of Polling, Gary Langer

Gary Langer is director of polling at ABC News, where he's covered the beat of public opinion for nearly 20 years - conducting and analyzing ABC News polls, evaluating data from other sources and setting the news division's standards for poll reporting. Langer is a two-time Emmy award winner, both for ABC's reporting of public opinion polls in Iraq.

ARCHIVES

FAVORITES

June 2009
Sun Mon Tue Wed Thu Fri Sat
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30        

« Previous | Main | Next »

New Hampshire's Polling Fiasco

January 09, 2008 12:05 AM

There will be a serious, critical look at the final pre-election polls in the Democratic presidential primary in New Hampshire; that is essential. It is simply unprecedented for so many polls to have been so wrong. We need to know why.

But we need to know it through careful, empirically based analysis. There will be a lot of claims about what happened - about respondents who reputedly lied, about alleged difficulties polling in biracial contests. That may be so. It also may be a smokescreen - a convenient foil for pollsters who'd rather fault their respondents than own up to other possibilities - such as their own failings in sampling and likely voter modeling.

There have been previous races that misstated support for black candidates in biracial races. But most of those were long ago, and there have been plenty of polls in biracial races that were accurate. (For more on past problems with polls in biracial races, see this blog I wrote for Freakonomics last May.) And there was no overstatement of Obama in Iowa polls.

On the other hand, the pre-election polls in the New Hampshire Republican race were accurate. The problem was isolated to the Democratic side - where, it should be noted, we have not just one groundbreaking candidate in Barack Obama, but also another, in Hillary Clinton.

A starting point for this analysis will be to look at every significant Democratic subgroup in the New Hampshire pre-election polls, and see how those polls did in estimating the size of those groups and their vote choices. The polls' estimates of turnout overall will be relevant as well.

In the end there may be no smoking gun. Those polls may have been accurate, but done in by a superior get-out-the-vote effort, or by very late deciders whose motivations may or may not ever be known. They may have been inaccurate because of bad modeling, compromised sampling, or simply an overabundance of enthusiasm for Obama on the heels of his Iowa victory that led his would-be supporters to overstate their propensity to turn out. (A function, perhaps, of youth.)

Prof. Jon Krosnick of Stanford University has another argument: That the order of names on the New Hampshire ballot - in which, by random draw, Clinton was toward the top, Obama at the bottom - netted her about 3 percentage points more than she'd have gotten otherwise. That's not enough to explain the gap in some of the polls, which presumably randomized candidate names, but it might hold part of the answer.

The data may tell us; it may not. What's beyond question is that it is incumbent on us - and particularly on the producers of the New Hampshire pre-election polls - to look at the data, and to look closely, and to do it without prejudging.

...Wednesday afternoon p.s.: Some folks are suggesting that "late deciders" made the difference - a common explanation for poor estimates. But the exit poll doesn't support the notion. Remove voters who decided on Tuesday and the New Hampshire exit poll result is Clinton +2 exactly her actual margin. (Among those who decided "just today" it was Clinton +3.) Next theory.

January 9, 2008 in Favorite Posts | Permalink | User Comments (887)

User Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Or you could save a lot of time and effort and do away with polls altogether. They serve no useful purpose other than to give the media pundits thousands of hours/pages of material to waste people's time with.

Posted by: Joe | Jan 9, 2008 12:51:26 AM

Polls are worthless. And once again, we are reminded why we shouldn't trust the media, or media hype for that matter.

Posted by: Doug | Jan 9, 2008 12:55:10 AM

I am glad somebody is talking about this! Here's the problem. The polls accurately predicted to within 1% error EVERY SINGLE CANDIDATE EXCEPT OBAMA/CLINTON. If you look at the pre-vote polls for ALL other candidates, they match up exactly. I mean exactly. Then, it is as if the Clinton/Obama results are reversed. They're both off by 5+% each. The statistical odds of this happening by chance must be astronomically small. This should cause a serious investigation into potential vote fraud. You must be able to explain this discrepancy and rule out fraud. Otherwise, fraud must be the prime suspect if we hope to have fair elections in the future!

Posted by: Eric | Jan 9, 2008 12:55:27 AM

There is no problem with the polls. They were correct, however, things changed the night before. Hillary's near crying event the day before was played over and over and showed her to be "finally" authentic. It got her the advantage.

Posted by: Jason | Jan 9, 2008 12:58:31 AM

I couldn't agree more, just get rid of polling and let the people vote...there is such overkill with the news coverage and emphasis of polls, not to mention the nauseating phone calls soliciting even more info...maybe Obama could use this as his first change, since we haven;t heard about any of the others.

Posted by: hitchiehoogle | Jan 9, 2008 12:58:50 AM

I would like to know how many people are registered to vot in new hampshire... I suspect that more folks voted democrat than are registered at all. Then add in the legal republican votes and you have half of vermont, and a bunch of busloads of mass. morons...the ones who bus around voting thrice for ed kennedy.... But as to the uofficial polls the media uses to try to sway opinions of folks to believe liar elections, well, sometimes you think you are succedding, when you are not - to wit: 2000, 2004. Mccain feingold helped you guys a lot.

Posted by: charlie | Jan 9, 2008 12:58:52 AM

How about another possibility: Vote fraud. I would put nothing past the Clintons. polls can be wrong, but they are rarely that wrong. Running out of ballots and running out to get more was a bit suspicious,

Posted by: Tom | Jan 9, 2008 12:59:14 AM

I'm starting to wonder if what we're seeing is the same type of thing we've seen in past elections-- electronic voting fraud-- which clearly helped Bush in the past two elections-- could it be that the Republicans WANT Hillary to get the nomination and are desperate enough to make it happen?

Awfully coincidental....

Posted by: george | Jan 9, 2008 12:59:50 AM

Polls are worthless -- to us. They are only worth it to you pundits, without them, all of you will be out of a job. This goes for sports as well. Why don't you predict the weather tomorrow based on what it was a year ago on that date. Geez...

Posted by: daniel Cooney | Jan 9, 2008 1:00:55 AM

Since the only poll that counts is the final poll in the election booth, why do we take these polls anyway?

They serve no particular public good. And, in fact do a public disservice by seeming to encourage an un healthy thing in a democracy, herdlike behavior.

All they really do is give the media a lot to talk about which is the wrong thing to talk about: handicapping the race rather than discussing the issues and the cndidatres' qualities.

Posted by: ChanRobt | Jan 9, 2008 1:00:56 AM

Call me crazy, but I seem to recall this isn't the first time in recent election history that pollsters and pundits got things completely wrong. And this isn't the first time that you media types have wrung your hands and called for some serious introspection in the wake of another screw-up. Blah, blah, blah. Whatever.

Posted by: Nick | Jan 9, 2008 1:01:03 AM

Nice. Obama pulls ahead in Iowa, and everyone celebrates. Hillary comes out on top in NH, and immediately people start "investigating" what went wrong. I am so tired of the pile-up against her. She is no more ambitious or calculating or staged than Reagan was. Just more female.

Posted by: Kevin | Jan 9, 2008 1:01:34 AM

Hillary the victim, the shed tear, the hitching voice, one day before the election. Just in time to get the typical sympathy vote as she did in New York. Hillary only wins, it appears, when she plays the victim.

Go Obama.

Posted by: Joel | Jan 9, 2008 1:01:45 AM

I am going with 'as of yet unexplained phenomena', meaning I don't put anything beyond the realm of possibility where the Clintons are concerned. I think we will know more in the coming days.

Posted by: Tim | Jan 9, 2008 1:01:47 AM

Why would I ever tell tell pollsters what I voted, my vote is secret

Posted by: John V Carey | Jan 9, 2008 1:03:06 AM

I live in New Hampshire. We were getting several phone calls a day..both from candidates and polls. I started picking a candidate of the day and that would be my answer to the pollsters. I even had one pollster giggle on Kucinich day.

Posted by: smitch | Jan 9, 2008 1:03:06 AM

Another vote for doing away with the polls. Some one should tell the reporters and analysts it's 2008. It is an era of all day information and statistics.

Also, no one is mentioning that the caucuses allow for candidates to go with a second choice if their first is non-viable. Thats approximately 16,000 votes that could've gone either way. Important?

Posted by: Doug | Jan 9, 2008 1:03:19 AM

When dealing with the Clintons fraud should be right at the top of the list of possibilities.

Posted by: Rich Border | Jan 9, 2008 1:03:36 AM

I totally agree with Eric. I would not put anything past the Clintons. They cannot be trusted.

Posted by: Linda | Jan 9, 2008 1:04:00 AM

For so many polls showing similar results in the Democratic Primary end up being so completely wrong just smells fishy to me when the Clintons are involved. Combined with the fact that Drudge ran a report that New Hampshire was running short of ballots for Democrats because so many people were turning out, makes me wonder if there isn't some shenanigans going on with busing in illegal voters. Sorry, but since it's a Clinton win when she was supposed to lose in almost every poll by at least 10 percentage points, I can't help but smell a rat considering the nasty history of this couple.

Posted by: Michael | Jan 9, 2008 1:04:58 AM

Post a comment